IPO-012: skip finished IPOs, and make SerpAPI failures diagnosable by DoRmAmMu1997 · Pull Request #113 · DoRmAmMu1997/Streamlit-Scanner-App

DoRmAmMu1997 · 2026-08-24T18:57:44Z

Investigating the warnings from the 2026-08-24 run surfaced three separate problems. Two are real bugs; one is pure waste against a hard quota.

1. Finished IPOs were being scanned on every run

SEBI's category → status mapping is DRHP → drhp_filed, RHP → rhp_filed, final_offer → closed. A final offer document is filed after the issue completes, so closed means the IPO is over — yet ACTIVE_ISSUE_STATUSES included it.

That is expensive in a way nothing in the code hinted at. Enrichment spends 8 SerpAPI searches per issue against a hard 250/month cap. The live database holds 51 issues: 21 drhp_filed, 13 rhp_filed, 17 closed. So 136 searches per run — over half the monthly quota — were being spent on IPOs nobody can act on any more.

ACTIVE_ISSUE_STATUSES → UPCOMING_ISSUE_STATUSES, dropping CLOSED. Renamed rather than silently redefined so every use site went through review.
The filter is applied once, at issue selection, not per stage. Previously only enrichment checked status, so downloads and re-scores covered finished issues regardless. Now downloads, enrichment, extraction and scoring inherit it together.
An explicitly named issue outranks the filter. Without that escape hatch the documented --force-extract --issue-id N workflow could never reach a closed issue. The per-stage check in the enrichment loop was removed rather than kept — leaving it would have silently overridden the operator on the one path where they asked for something specific.

There is deliberately no date comparison, despite "listing date in the future" being the natural phrasing. The issue row has no listing date column, and open_date/close_date are NULL for all 51 rows because ingestion never populates them. A date filter would match everything or nothing. Lifecycle stage is the only signal that exists.

Effect: eligible issues 51 → 34; a full CLI run 408 → 272 searches.

2. A benign empty search was reported as a provider failure

SerpAPI answers a query Google found nothing for with HTTP 200 and {"error": "Google hasn't returned any results for this query."}, and the client raised on any error key. The caller's continue then skipped persistence, so the issue silently lost that signal type — indistinguishable from an outage.

This is the likely explanation for the observed clustering on peer_discovery, brokerage_review and subscription_demand: the thin-coverage queries for small IPOs.

Now matched narrowly and returned as an empty list. The match is conservative on purpose — mistaking a real error for "no results" would hide it.

3. Failures were undiagnosable by construction

SerpApiSearchError collapsed eight distinct conditions into one flat class, and the caller logs only type(exc).__name__ — a compile-time constant for that call site. All eleven warnings carried exactly as much information as "a search failed".

Worse, raise_for_status() ran before the body was read. SerpAPI reports plan exhaustion as HTTP 429 carrying a JSON body naming the cause, so the one field explaining why every remaining search would also fail was discarded. The body is now read first (already bounded to 1 MiB) and the status checked after.

The fix follows the precedent this repo already set: when SEBI's edge began returning 530, the answer was a distinct exception subclass, because the job logs a class name and never a message (upstream text is untrusted).

Type	Condition	Nature
`SerpApiQuotaError`	body says the account is out of searches	permanent for the billing period
`SerpApiRateLimitError`	HTTP 429, no quota wording	transient
`SerpApiAuthError`	HTTP 401 / 403	configuration fault
`SerpApiSearchError`	transport, timeout, 5xx, oversize, non-JSON	unchanged catch-all, still the base

All inherit the base, so every existing except SerpApiSearchError keeps working. Each carries status_code, logged alongside the class — a status code is safe metadata; the body is not, and stays redacted.

A bare 429 with no explanatory body is classified as the transient reading. Claiming exhaustion on thin evidence would stop a run that could have continued; the reverse merely lets it finish.

On quota exhaustion the collector stops the batch and the job stops enriching, mirroring the existing missing-key behaviour. That turns ~20 minutes of identical warnings into one actionable line, and it is the only behaviour change here rather than pure diagnostics.

Investigated and ruled out

The 3 ipo_document_download_failed | error_code=network_error lines are not SEBI WAF blocks. network_error is raised only from except requests.RequestException; a 530 would take the 500 <= status <= 599 branch and surface as http_error. Those three are genuine read/connect timeouts after a full 4-attempt ladder.

The downloader does lack the Referer the SEBI listing client now sends, and a 530 there would still burn the full 17s ladder and report a bare http_error — both real latent gaps, both deliberately out of scope here since neither caused the observed failures.

Testing

The entire HTTP-status path was previously uncovered — no test constructed a response with status_code >= 400 at all. Added 429+quota body, bare 429, 401/403, 5xx, the no-results shape, and a real provider error to guard the narrow match. Corrected the existing empty-results test, which asserted {"organic_results": []} — a shape SerpAPI does not send for an empty search.

On the filter side: a closed and a listed issue are skipped by download, enrichment and scoring; an explicitly named closed issue is still processed. Both were confirmed to fail with the filter disabled before being kept.

Known limitation

272 searches for a full CLI run is still above the 250/month plan. The Streamlit default (max_issues=25, 200 searches) fits. The remaining lever is a filing-date staleness cutoff — filing_date is 100% populated (60/60) — deliberately left out of this change.

Gates

pytest 1939 passed / 1 skipped, coverage 89.81%; ruff; mypy clean on 257 files; compileall; bandit; pip-audit; pre-commit config. No constraints.txt / pyproject.toml drift.

🤖 Generated with Claude Code

Codex follow-up review and remediation (2026-09-01)

Follow-up commit: c5bf3e7 (fix(ipo): harden SerpAPI failure boundaries), co-authored by Codex. Final composed head: a5f0bf7, which merges current main (4ab857f) before final verification.

Provider JSON error prose is now classification-only and never becomes an exception/log/model message; the 67-ka consumer also quarantines error as defense in depth.
The no-results exception applies only to the exact known message on HTTP success. Auth status outranks body prose, while explicit quota bodies remain terminal even on HTTP 200.
Rejected credentials stop after one request, stop later issue enrichment, emit enrichment=auth_failed, and keep the scheduler exit nonzero.
Headless enrichment defaults to 25 issues (200 searches); --max-enrichment-issues 0 explicitly removes the cap. Download/extract/score still cover the complete selected set, and enrichment_skipped_budget is visible.
Streamlit retains the durable only_active_issues key but renders Only upcoming IPOs with help text. IPO screener provenance is bumped to 1.1.0.
Architecture/operations docs now agree on eight signal queries and document every failure/budget outcome. The implementation plan is checked in at docs/superpowers/plans/2026-08-31-pr113-review-remediation.md.

Verification on final tree:

TDD regressions were observed red before implementation.
Focused integration: 279 passed before the final review round; pre-merge full suite: 1971 passed. Final composed suite after merging current main: 1980 passed, 1 skipped, 89.87% coverage (required 89%).
Ruff, compileall, mypy (257 source files), Bandit, pre-commit, and pip-audit passed; no dependency, constraints, ORM, or migration change.
Codex Security diff scan a046d6b8-c8ad-4604-b9ab-54e594e50d8e: complete coverage, zero reportable diff-introduced findings. Three boundary candidates were reproduced; all relevant hardening/correctness defects were fixed.
Independent follow-up code review: no unresolved critical, important, or minor issues.
Hosted run 33512985314: Python 3.11, Python 3.12, and Docker passed.
Hosted CodeQL run 33512981521: Actions and Python analysis passed.
Rendered Streamlit QA was attempted, but the in-app browser trusted local bridge was unavailable; deterministic registry/parameter/screener adapter tests cover the changed control contract.

SEBI's filing categories map DRHP -> drhp_filed, RHP -> rhp_filed, and final offer -> closed. A final offer document is filed AFTER the issue completes, so "closed" means the IPO is over - yet ACTIVE_ISSUE_STATUSES included it, and every stage happily processed those issues on every run. That is expensive in a way nothing in the code hinted at. Enrichment spends 8 SerpAPI searches per issue against a hard 250/month cap. The live database holds 51 issues: 21 drhp_filed, 13 rhp_filed, and 17 closed. So 136 searches per run - over half the monthly quota - were being spent on IPOs nobody can act on any more. ACTIVE_ISSUE_STATUSES becomes UPCOMING_ISSUE_STATUSES and drops CLOSED. Renamed rather than silently redefined so every use site went through review and the name states what it actually selects. Two design points worth keeping: - The filter is applied ONCE, at issue selection, rather than per stage. Only enrichment checked status before, so downloads and re-scores covered finished issues regardless. Filtering at selection means downloads, enrichment, extraction and scoring inherit it together and cannot drift apart. - An explicitly named issue outranks the filter. Without that escape hatch the documented "--force-extract --issue-id N" workflow could never reach a closed issue, and no finished offer could be deliberately re-scored after a rule change. The per-stage status check in the enrichment loop was therefore removed rather than kept: leaving it would have silently overridden the operator on the one path where they had asked for something specific. There is deliberately NO date comparison here, despite "listing date in the future" being the natural way to phrase the intent. The issue row has no listing date column at all, and open_date/close_date are NULL for all 51 rows because ingestion never populates them and IPO-011 left date extraction out of scope. A date filter would therefore match everything or nothing. Lifecycle stage is the only signal that actually exists; the constant's comment says so in case someone later reaches for the dates. Left alone on purpose: _weak_qib_demand_near_close judges issues whose status is OPEN or CLOSED, so its evidence stops refreshing for closed ones. That is correct - the vocabulary overloads "closed", and the value ingestion produces means "final offer filed", long past the book close that flag is about. Effect: eligible issues 51 -> 34, a full CLI run 408 -> 272 searches. Still above 250/month for an uncapped run; the Streamlit default (max_issues=25, 200 searches) fits. Tests: a closed and a listed issue are skipped by download, enrichment AND scoring; an explicitly named closed issue is still processed. Both were confirmed to fail with the filter disabled before being kept. Gates: full pytest (1930 passed, 89.82%), ruff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… failure A live run produced eleven of these: ipo_enrichment_failed | issue_id=43 signal_type=news error_type=SerpApiSearchError They were undiagnosable by construction. SerpApiSearchError collapsed EIGHT distinct conditions - transport timeout, 429, 401/403, 5xx, oversize body, non-JSON body, a 200-with-error payload, and a cleanup failure - into one flat class, and the caller logs only type(exc).__name__. That field is a compile-time constant for the call site, so every warning carried exactly as much information as "a search failed". Two real defects sat behind those lines. 1) A benign empty search was reported as a provider failure. SerpAPI answers a query Google found nothing for with HTTP 200 and {"error": "Google hasn't returned any results for this query."}, and the client raised on ANY error key. The caller's `continue` then skipped persistence entirely, so the issue silently lost that signal type - which read identically to an outage. This is the likely explanation for the observed clustering on peer_discovery, brokerage_review and subscription_demand: the thin-coverage queries for small IPOs. Now matched narrowly and returned as an empty list, so the signal flows down the success path and persists an honest empty observation. The match is deliberately conservative: mistaking a real error for "no results" would hide it, so anything unrecognised still raises. 2) Quota exhaustion was structurally invisible. raise_for_status() ran BEFORE the body was read. SerpAPI reports plan exhaustion as HTTP 429 carrying a JSON body that names the cause, so the one field explaining why every remaining search would also fail was thrown away. The body is now read first - it was already bounded to 1 MiB, so this costs nothing - and the status is checked after. The fix follows the precedent this repo already set for exactly this problem. When SEBI's edge began returning 530, the answer was a distinct exception subclass, because the job logs a class name and never a message (upstream text is untrusted). Same reasoning here: SerpApiQuotaError - plan spent; permanent for the billing period SerpApiRateLimitError - throttled; transient SerpApiAuthError - 401/403; configuration fault SerpApiSearchError - unchanged catch-all, and still the base class All inherit the base, so every existing `except SerpApiSearchError` keeps catching what it used to. Each carries status_code, which is logged alongside the class - a status code is safe metadata; the body is not, and stays redacted. A bare 429 with no explanatory body is classified as the TRANSIENT of the two readings. Claiming exhaustion on thin evidence would stop a run that could have continued; the reverse merely lets it finish. On quota exhaustion the collector stops the batch and the job stops enriching altogether, the same way the missing-key case already worked. That turns ~20 minutes of identical warnings into one actionable line, and it is the only behaviour change here rather than pure diagnostics. Tests: the entire HTTP-status path was previously uncovered - no test constructed a response with status_code >= 400 at all. Added 429+quota body, bare 429, 401/403, 5xx, the no-results shape, and a real provider error to guard the narrow match. Corrected the existing empty-results test, which asserted {"organic_results": []} - a shape SerpAPI does not actually send for an empty search. Gates: full pytest (1939 passed, 89.81%), ruff, mypy, compileall, bandit, pip-audit, pre-commit config. No constraints/pyproject drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

DoRmAmMu1997 · 2026-08-24T19:06:35Z

@codex can you review this?

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60cb9b4f51

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sixteen findings across the three reviews, two of which were regressions this PR introduced that defeated its own purpose. THE TOGGLE WAS A LIE _selected_issue_ids returned None whenever the toggles happened not to narrow anything, meaning "let the pipeline choose" - and the pipeline's own default is upcoming-only. So an operator who UNTICKED only_active_issues to widen the run silently got upcoming-only processing, while the results table still listed every finished issue. That is precisely the processed/reported drift _apply_selection's docstring says it exists to prevent. It was order-dependent as well: with max_issues=25 against 51 rows the cap narrowed, an explicit list was sent, and finished issues inside the cap WERE processed. Whether the toggle worked depended on whether the cap bit. The selection is now always explicit, so what the table reports is exactly what ran. The job's `if issue_ids:` becomes `is not None`, because an empty list was falsy and silently meant "every issue" - which under the new filter would have turned a deliberately empty selection into a run over everything upcoming. This also closed a scope divergence: auto-approval and the follow-up rescore now receive the same concrete list, so proposals can no longer be approved for issues the rescore pass will skip. NON-JSON ERROR BODIES BYPASSED THE WHOLE TAXONOMY Reading the body before the status is what makes a 429 quota message readable, but it also meant a response whose body is NOT JSON raised during decoding, before the status was ever inspected. Any CDN, proxy or WAF in front of SerpAPI answers a 401/403/429 with an HTML page, so the most common error responses stayed exactly as undiagnosable as before the taxonomy existed - and an HTML quota page never triggered the short-circuit. Every test written for the taxonomy used a JSON body, which is why it slipped through. Decode failures are now re-raised through the classifier. "SEARCH LIMIT" WAS TOO LOOSE TO BE A QUOTA MARKER "You have exceeded your hourly search limit" is a throttle that clears on its own, and matching the bare substring classified it as a spent plan and aborted the whole enrichment stage - the exact outcome _classify_status's own docstring argues against. Markers are now unambiguously terminal only; anything else falls through to the transient reading. QUOTA EXHAUSTION WAS INVISIBLE AND ALARMED THE SCHEDULER The state was reported only by a print to `out`, and the Streamlit path passes io.StringIO() - so the one actionable line went into a throwaway buffer. It is now a field on the outcome and a key in the completion event, mirroring enrichment_skipped_no_key. It also drove exit_code nonzero, so every scheduled run for the rest of the billing period would have alarmed identically to a real outage, while the same class of "optional feature unavailable" state (a missing key) exits 0. Quota and throttle no longer count as failures. SUSTAINED THROTTLING STILL GROUND ON Only quota broke the loop, so a bare-429 throttle still issued 8 queries across every issue and emitted hundreds of identical warnings. A streak of consecutive rate limits now stops the stage; any success resets the counter, so intermittent throttling never trips it. Nothing sleeps - this refuses to grind, it does not pace. OPERATIONAL GAPS - New --include-finished. Without it a closed issue whose download failed could only be retried by naming its id by hand, and back-applying a scoring change across finished issues meant enumerating them all. - Runs now report skipped_finished=N, so dropping a third of the inventory is never silent. - Dropped the defensive getattr on fields that are always defined. It defeated mypy and let the test fake omit quota_exhausted entirely - meaning a rename would have disabled the short-circuit with no test failure. A shared contract-complete enrichment fake replaces it. - test_finished_issues_are_skipped_by_every_stage now passes extract=True. It claimed "every stage" while never entering the extraction block, so Claude credit could have been spent on finished issues without failing. - Docs: "Every instance carries status_code" was false for the transport-failure path; corrected. The 67-ka-funda LLD still documented the flat two-exception contract even though backend/sixty_seven/agent.py is a live second consumer that now sees [] where a no-results query used to raise. Security review found nothing clearing its bar. It cleared the most promising candidate - a hostile body steering the substring classification - because that requires controlling an HTTPS body, and such an attacker could already return {"organic_results": []} pre-PR for an identical effect. It also noted the PR REDUCES key exposure: a body-carrying error short-circuits before raise_for_status(), whose HTTPError message embeds the URL including api_key. Both regressions were confirmed to fail before their fix: the non-JSON tests reproduce the bare base class, and the toggle test reproduces the narrow run. Gates: pytest 1948 passed (89.85%), ruff, mypy clean, compileall, bandit, pip-audit, pre-commit config. No constraints/pyproject drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

DoRmAmMu1997

PR #113 review — changes requested

Reviewed exact range a154f24…eecf030. The current focused suite is green
(219 tests), and the completed Codex Security diff scan found no reportable
new vulnerability because the strongest injection path already existed for
HTTP-200 provider errors at the base revision. The review still reproduced six
defects/improvements that should be fixed before merge:

[P1] The default headless run still exceeds the stated provider plan.
The PR reduces 51 issues to 34 but documents 8 searches per issue: 272 calls
against a 250/month plan. Add a safe default enrichment cap (25 issues / 200
searches), an explicit uncapped override, and a visible skipped-by-budget
count while continuing free download/extract/score work for the other issues.
[P1] Rejected credentials do not short-circuit. SerpApiAuthError
explicitly says every later request will fail, but the collector attempts all
eight signal types and the job repeats that batch for every issue. A
non-network reproducer observed auth_attempts=8 for one issue. Stop on the
first auth rejection, expose a typed job outcome, and keep the exit code
nonzero because an invalid key is a configuration fault rather than an
intentional no-key mode.
[P2] “No results” can override a failing HTTP status. A 401 response with
Google hasn't returned any results... currently returns [] before status
enforcement. Require the exact known no-results shape on a successful status;
substring lookalikes and 401/403/429/5xx must retain their typed failures.
[P2] Provider error prose crosses the model boundary unquarantined. The
changed body-first path carries payload.error in SerpApiSearchError; the
67-ka consumer copies str(exc) into its tool payload, while its injection
scanner excludes the error field. A reproducer observed
hostile_error_detected=False and hostile_error_exposed=True. Raise fixed,
app-owned exception messages after using provider text only for internal
classification, and scan error as defense in depth.
[P2] UI wording and provenance do not describe the new behavior. The
sidebar still renders the persisted key only_active_issues, although the
new rule means upcoming only, and IpoScreener.SCREENER_VERSION remains
1.0.0 despite a material selection change. Preserve the storage key but add
an explicit display label/help contract and bump provenance to 1.1.0.
[P3] Documentation disagrees with the executable contract. IPO-009 still
says seven query templates while the enum and operations budget use eight.
Reconcile the architecture/runbook and document auth termination, the
headless budget cap, and generic provider-error messages.

Implementation should remain dependency-free and test-first, with detailed
Google-style docstrings and beginner notes on every non-obvious new boundary.

Security scan report: report.md under scan
a046d6b8-c8ad-4604-b9ab-54e594e50d8e (complete coverage, zero reportable
findings). Rendered Streamlit QA could not run because the in-app browser's
trusted local bridge was unavailable; adapter and parameter-control tests will
serve as the deterministic regression layer, with hosted checks as the final
runtime gate.

Stop permanent credential failures and cap default paid enrichment, keep provider prose out of shared exceptions and AI tool payloads, require exact successful no-results responses, and make the upcoming-IPO UI/provenance contract explicit. Add full regressions, operational documentation, and the review remediation plan. Co-authored-by: Codex <codex@openai.com>

DoRmAmMu1997 · 2026-09-01T13:16:17Z

Follow-up remediation is complete on final composed head a5f0bf7 (c5bf3e7 plus current main at 4ab857f). All six review areas were addressed. Final local verification: 1980 passed / 1 skipped / 89.87% coverage; pre-commit, Ruff, compileall, mypy, Bandit, and pip-audit passed. Hosted run 33512985314 passed Python 3.11, Python 3.12, and Docker; CodeQL run 33512981521 passed Actions and Python analysis. Both review threads remain resolved.

Compose the SEC-003 ReDoS hardening now on main with the reviewed IPO-012 SerpAPI and lifecycle remediation before final verification. Co-authored-by: Codex <codex@openai.com>

Bring PR #112 onto the current main branch after PRs #113 and #115 landed. The incoming changes do not overlap the DATA-003 loader, tests, or documentation. Co-authored-by: Codex <codex@openai.com>

DoRmAmMu1997 and others added 2 commits August 24, 2026 20:25

chatgpt-codex-connector Bot reviewed Aug 24, 2026

View reviewed changes

Comment thread backend/jobs/run_ipo_screener.py Outdated

Comment thread backend/sixty_seven/search_client.py Outdated

DoRmAmMu1997 commented Aug 31, 2026

View reviewed changes

Merge origin/main into IPO-012 follow-up

a5f0bf7

Compose the SEC-003 ReDoS hardening now on main with the reviewed IPO-012 SerpAPI and lifecycle remediation before final verification. Co-authored-by: Codex <codex@openai.com>

DoRmAmMu1997 merged commit 2833bcf into main Sep 1, 2026
6 of 7 checks passed

DoRmAmMu1997 deleted the fix/ipo-012-upcoming-only-and-serpapi-taxonomy branch September 1, 2026 13:27

DoRmAmMu1997 mentioned this pull request Sep 2, 2026

fix(DATA-003): stop the scan re-dirtying the candle cache it just repaired #112

Open

Navigation Menu

IPO-012: skip finished IPOs, and make SerpAPI failures diagnosable - #113

IPO-012: skip finished IPOs, and make SerpAPI failures diagnosable#113
DoRmAmMu1997 merged 5 commits into
mainfrom
fix/ipo-012-upcoming-only-and-serpapi-taxonomy

DoRmAmMu1997 commented Aug 24, 2026 •

edited

Loading

Uh oh!

DoRmAmMu1997 commented Aug 24, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

DoRmAmMu1997 left a comment

Uh oh!

DoRmAmMu1997 commented Sep 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

DoRmAmMu1997 commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Finished IPOs were being scanned on every run

2. A benign empty search was reported as a provider failure

3. Failures were undiagnosable by construction

Investigated and ruled out

Testing

Known limitation

Gates

Codex follow-up review and remediation (2026-09-01)

Uh oh!

DoRmAmMu1997 commented Aug 24, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

DoRmAmMu1997 left a comment

Choose a reason for hiding this comment

PR #113 review — changes requested

Uh oh!

DoRmAmMu1997 commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

DoRmAmMu1997 commented Aug 24, 2026 •

edited

Loading

DoRmAmMu1997 commented Sep 1, 2026 •

edited

Loading