Skip to content

refactor(importers): extract persist_new_findings as a bulk-write seam - #15599

Open
valentijnscholten wants to merge 2 commits into
devfrom
feat/persist-new-findings-seam
Open

refactor(importers): extract persist_new_findings as a bulk-write seam#15599
valentijnscholten wants to merge 2 commits into
devfrom
feat/persist-new-findings-seam

Conversation

@valentijnscholten

Copy link
Copy Markdown
Member

Add a hook in OS to allow Pro to perform extra processing on newly created / new to be created findings during import/reimport.

Description

Extracts BaseImporter.persist_new_findings() as an overridable seam for how new
findings get written during import/reimport, without changing any observable behavior
in this repo.

Today, DefaultImporter/DefaultReImporter write each new finding one at a time via
save_no_options(), interleaved with per-finding post-processing that needs a real
primary key (locations, vulnerability IDs, tags, request/response pairs). That leaves
no seam for a downstream edition to swap in a batched write strategy without copying
the whole loop — this PR adds that seam, without adding a batched writer itself.

Changes:

  • Adds persist_new_findings(prepared_findings) -> list[Finding] to BaseImporter.
    Default implementation is the existing per-instance save_no_options() loop, just
    called once per batch instead of inline per finding.
  • Restructures DefaultImporter._process_findings_internal to prepare a batch, call
    persist_new_findings, then run the existing per-finding post-processing — same SQL,
    same order, same batch boundaries as before.
  • Restructures DefaultReImporter._process_findings_internal similarly. Reimport is
    more involved because finding_post_processing (vulnerability-ID/CWE reconciliation,
    file attachment) queries relations that require a primary key, so it's deferred as a
    unit for new findings (matched findings are unaffected, unchanged). New findings are
    persisted and post-processed once per matching batch (not the larger dedupe-batch
    boundary), so a same-report duplicate is a real, queryable row before the next
    matching batch's candidate query runs.
  • match_finding_to_candidate_reimport no longer re-sorts candidates by .id. Once a
    new finding only gets its primary key at the batch boundary, a same-report duplicate
    queued earlier in the same matching batch has no id yet to sort by. The candidate
    lists are already in priority order by construction (existing candidates fetched
    pre-sorted, same-report ones appended afterward in processing order), so returning
    them as-is is correct and avoids sorting a list that can contain a None id. The
    unique_id_from_tool_or_hash_code merge de-duplicates by object identity instead of
    .id for the same reason.
  • A follow-up commit fixes a same-report duplicate still being unsaved when it's picked
    as a reimport match target (_finalize_specific_pending_new_finding() persists it on
    demand instead), and updates the two reimport-with-new-findings query-count baselines
    for the one-fewer-query effect of batching the drain.

Test results

No behavior change intended. Verified against unmodified dev by diffing test output
before/after this change on:

  • unittests/test_importers_importer.py
  • unittests/test_importers_performance.py
  • unittests/test_import_reimport.py
  • unittests/test_reimport_batch_flush.py
  • unittests/test_importers_deduplication.py

The diff is byte-for-byte identical in both directions: same set of failing tests, and
where a test asserts an exact query count, the same exact number on both sides. No new
tests were added for the seam extraction itself, since it intentionally makes no
behavior change for this repo to cover — the seam is unused here. The follow-up fix
commit does add/update tests (see its own message).

Documentation

Not applicable — no user-facing behavior changes.

Checklist

  • Rebased against the latest dev.
  • Submitted against dev (refactor, not a bug fix).
  • Ruff compliant.
  • No new tests added for the seam extraction — see Test results above for why.

Both DefaultImporter and DefaultReImporter created new findings one at a time
via save_no_options(), interleaved with the per-finding post-processing that
needs a real primary key (locations, vulnerability ids, tags, request/response
pairs). That leaves no seam for a downstream edition to swap in a bulk write.

Extract BaseImporter.persist_new_findings() as an overridable batch write step,
called once per batch with the OSS default being the same per-instance save
loop as before -- zero behavior change, same SQL, same order, same batch
boundaries (proven via byte-for-byte identical failures and query-count
assertions against unmodified dev on test_importers_importer.py,
test_importers_performance.py, test_import_reimport.py,
test_reimport_batch_flush.py, and test_importers_deduplication.py).

Reimport also needed match_finding_to_candidate_reimport to stop re-sorting
candidates by `.id`: once a batch's new findings only get their primary key at
the batch boundary rather than immediately, a same-report duplicate queued by
add_new_finding_to_candidates earlier in the same matching batch has no id yet
to sort by. The candidate lists are already in priority order by construction
(existing candidates fetched pre-sorted, same-report ones appended afterward in
processing order), so returning them as-is is both correct and avoids sorting a
list that can contain a None id. The unique_id_from_tool_or_hash_code merge now
de-duplicates by object identity instead of `.id` for the same reason.
… it is used as a reimport match target

A same-report duplicate queued by add_new_finding_to_candidates during a
matching batch has no primary key until _drain_pending_new_findings runs at
the end of that batch. If a later finding in the same batch matches against
it first, process_matched_finding/finding_post_processing/location_handler
all require a real pk -- crashing with "Finding instance needs to have a
primary key value" (reconcile_cwes), "cannot use Finding as a dict key"
(location recording), or surfacing as a 500 from the reimport API.
_finalize_specific_pending_new_finding() now persists and post-processes
that one pending finding on demand, out of its normal per-batch drain order,
the moment it is picked as a match target.

Also updates the two reimport-with-new-findings query-count baselines: the
_drain_pending_new_findings batching (persist once per matching batch instead
of inline per finding) nets one fewer query on both the V2 and V3 paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant