refactor(importers): track reimport finding buckets by id, not instance - #15600
Draft
valentijnscholten wants to merge 1 commit into
Draft
refactor(importers): track reimport finding buckets by id, not instance#15600valentijnscholten wants to merge 1 commit into
valentijnscholten wants to merge 1 commit into
Conversation
DefaultReImporter kept every original/new/reactivated/unchanged Finding instance alive for the whole run just so notify_scan_added() and update_import_history() could read a few scalar fields at the end -- on a large reimport that pins the full result set in memory. new_items, reactivated_items, unchanged_items and original_items now hold ids; the two consumers that need real rows (close_old_findings, the JIRA finding-group push) requery deliberately and in bounded chunks at the point they need them, sharing self.test instead of copying the parent chain per row. notify_scan_added also gains NOTIFICATION_SCAN_ADDED_MAX_FINDINGS (default 100) so a reimport touching thousands of findings no longer templates all of them into a single notification body.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #15599.
DefaultReImporter._process_findings_internalkept everyoriginal/new/reactivated/unchanged
Findinginstance alive in memory for thewhole run, purely so
notify_scan_added()andupdate_import_history()couldread a few scalar fields off them at the very end. On a large reimport that
pins the full result set in memory for no reason.
original_items,new_items,reactivated_items,unchanged_itemsnowhold ids, not instances.
close_old_findings()and the JIRA finding-group push are the onlyconsumers that need real rows; both now requery deliberately, in bounded
chunks, at the point they need them -- and share
self.testrather thaneach row carrying its own copy of the
test -> engagement -> productchain, the same pattern reimport matching already uses.
notify_scan_added()is rewritten to requery a capped, ordered slice(
NOTIFICATION_SCAN_ADDED_MAX_FINDINGS, default 100) instead of templatingevery touched finding into a notification body.
dojo.finding.helper.filter_findings_by_existence(instance-based, onecaller) is replaced by
filter_finding_ids_by_existence(id-based).Test plan
unittests.test_reimport_batch_flush-- updated assertions for the newid-based buckets, passes.
unittests.test_importers_importer-- no new failures vs baseline(pre-existing 301-redirect failures in
FlexibleImportTestAPI/FlexibleReimportTestAPIreproduce identically with this branchstashed out, confirmed unrelated).
ProReImporter) -- 2391 passed, 119 skipped, 0 failed after fixing alocal dev-stack env misconfiguration (
DD_V3_FEATURE_LOCATIONS) thathad produced 46 false failures unrelated to this change.
test_importers_performance.pyassertNumQueriesbaselinesupdated; the refactor is a net reduction in query count on every step
except one that initially regressed by 21 queries (an N+1 from
freshly-hydrated
close_old_findingscandidates not sharingself.test) -- fixed at the root rather than absorbed into the pinnedcount, which now shows a net decrease vs the pre-refactor baseline.
unittests/test_tag_inheritance_perf.py's reimport-with-new-findingsbaselines were hand-recomputed while rebasing onto a
devcommit thatindependently changed the same constants (two additive deltas from a
common ancestor, combined arithmetically:
EXPECTED_ZAP_REIMPORT_WITH_NEW_V3193 -> 194). Could not run locally -- this dev stack's Pro integration
permanently disables
watson, which this test class's fixture needs.Flagging for CI to confirm.