Skip to content

refactor(importers): track reimport finding buckets by id, not instance - #15600

Draft
valentijnscholten wants to merge 1 commit into
feat/persist-new-findings-seamfrom
feat/track-finding-ids-not-instances
Draft

refactor(importers): track reimport finding buckets by id, not instance#15600
valentijnscholten wants to merge 1 commit into
feat/persist-new-findings-seamfrom
feat/track-finding-ids-not-instances

Conversation

@valentijnscholten

Copy link
Copy Markdown
Member

Summary

Stacked on #15599. DefaultReImporter._process_findings_internal kept every
original/new/reactivated/unchanged Finding instance alive in memory for the
whole run, purely so notify_scan_added() and update_import_history() could
read a few scalar fields off them at the very end. On a large reimport that
pins the full result set in memory for no reason.

  • original_items, new_items, reactivated_items, unchanged_items now
    hold ids, not instances.
  • close_old_findings() and the JIRA finding-group push are the only
    consumers that need real rows; both now requery deliberately, in bounded
    chunks, at the point they need them -- and share self.test rather than
    each row carrying its own copy of the test -> engagement -> product
    chain, the same pattern reimport matching already uses.
  • notify_scan_added() is rewritten to requery a capped, ordered slice
    (NOTIFICATION_SCAN_ADDED_MAX_FINDINGS, default 100) instead of templating
    every touched finding into a notification body.
  • dojo.finding.helper.filter_findings_by_existence (instance-based, one
    caller) is replaced by filter_finding_ids_by_existence (id-based).

Test plan

  • unittests.test_reimport_batch_flush -- updated assertions for the new
    id-based buckets, passes.
  • unittests.test_importers_importer -- no new failures vs baseline
    (pre-existing 301-redirect failures in FlexibleImportTestAPI /
    FlexibleReimportTestAPI reproduce identically with this branch
    stashed out, confirmed unrelated).
  • Full Pro importers/connectors suite (which exercises this OSS code via
    ProReImporter) -- 2391 passed, 119 skipped, 0 failed after fixing a
    local dev-stack env misconfiguration (DD_V3_FEATURE_LOCATIONS) that
    had produced 46 false failures unrelated to this change.
  • Pro's test_importers_performance.py assertNumQueries baselines
    updated; the refactor is a net reduction in query count on every step
    except one that initially regressed by 21 queries (an N+1 from
    freshly-hydrated close_old_findings candidates not sharing
    self.test) -- fixed at the root rather than absorbed into the pinned
    count, which now shows a net decrease vs the pre-refactor baseline.
  • unittests/test_tag_inheritance_perf.py's reimport-with-new-findings
    baselines were hand-recomputed while rebasing onto a dev commit that
    independently changed the same constants (two additive deltas from a
    common ancestor, combined arithmetically: EXPECTED_ZAP_REIMPORT_WITH_NEW_V3
    193 -> 194). Could not run locally -- this dev stack's Pro integration
    permanently disables watson, which this test class's fixture needs.
    Flagging for CI to confirm.

DefaultReImporter kept every original/new/reactivated/unchanged Finding
instance alive for the whole run just so notify_scan_added() and
update_import_history() could read a few scalar fields at the end -- on a
large reimport that pins the full result set in memory. new_items,
reactivated_items, unchanged_items and original_items now hold ids; the two
consumers that need real rows (close_old_findings, the JIRA finding-group
push) requery deliberately and in bounded chunks at the point they need them,
sharing self.test instead of copying the parent chain per row.

notify_scan_added also gains NOTIFICATION_SCAN_ADDED_MAX_FINDINGS (default
100) so a reimport touching thousands of findings no longer templates all of
them into a single notification body.
@valentijnscholten valentijnscholten added this to the 3.3.0 milestone Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant