Parity with tango-python: new resources, full filter surface, shape overlay, conformance gates, cassette tests

.github/workflows/ci.yml

@@ -2,12 +2,15 @@ name: CI
 
 # Lint + typecheck + test gate runs on every PR and push to main.
 #
-# The SDK filter/shape conformance check needs the canonical manifest from the
-# private makegov/tango repo, which requires a TANGO_API_REPO_ACCESS_TOKEN secret
-# the public CI does not have. The conformance job SKIPS cleanly when the token
-# is absent (rather than failing on an empty token) and becomes a hard gate the
-# moment the secret is configured. The lint + test gate below is self-contained
-# and blocks the PR on failure.
+# The SDK filter/shape conformance check and the reverse shape-coverage check
+# are HARD gates that run offline against the vendored contract at
+# contracts/filter_shape_contract.json — no secrets needed, so forks and
+# tokenless runs get the full check instead of a silent skip. When
+# TANGO_API_REPO_ACCESS_TOKEN is available, the same two checks ALSO run as
+# hard gates against the fresh contract at makegov/tango HEAD, plus a warning
+# annotation when the vendored copy has drifted (re-vendor reminder). Refresh
+# the vendored contract by copying contracts/filter_shape_contract.json from
+# makegov/tango.
 on:
   push:
     branches: [ main ]
@@ -31,12 +34,12 @@ jobs:
         uses: actions/setup-node@v4
         with:
           node-version: ${{ matrix.node-version }}
+          cache: npm
 
       - name: Install dependencies
-        # No lockfile is committed (package-lock.json is gitignored), and the
-        # "prepare" script runs a build that needs tsc — so ignore scripts here
+        # The "prepare" script runs a build that needs tsc — so ignore scripts here
         # and build explicitly below.
-        run: npm install --ignore-scripts --no-audit --no-fund
+        run: npm ci --ignore-scripts --no-audit --no-fund
 
       - name: Lint
         run: npm run lint
@@ -48,17 +51,46 @@ jobs:
         run: npm run build
 
       - name: Test
-        # `vitest run` forces a single non-watch pass in CI.
-        run: npx vitest run
+        # `vitest run` forces a single non-watch pass in CI; the Node 20 leg adds `--coverage` so the suite runs exactly once per leg.
+        # No coverage fail-under gate — parity with tango-python, which has none.
+        # Integration tests replay the committed cassettes offline; production smoke stays excluded (env-gated on TANGO_LIVE_TESTS, never set here).
+        run: npx vitest run ${{ matrix.node-version == '20' && '--coverage' || '' }}
 
   conformance:
-    # Requires the canonical filter_shape manifest from the private makegov/tango
-    # repo. When TANGO_API_REPO_ACCESS_TOKEN is not configured, every real step
-    # is skipped and the job passes (rather than failing on an empty token).
-    # Configure the secret to turn this into a hard gate automatically.
+    # Hard gate against the vendored contract (contracts/filter_shape_contract.json).
+    # Runs unconditionally — no secrets required, so forks and tokenless runs
+    # get the full check instead of a silent skip.
     runs-on: ubuntu-latest
 
     steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+          cache: npm
+
+      - name: Install dependencies
+        run: npm ci --ignore-scripts --no-audit --no-fund
+
+      - name: Check SDK filter/shape conformance (vendored contract)
+        run: npx tsx scripts/check-filter-shape-conformance.ts
+
+      - name: Check reverse shape coverage (Tango exposes -> SDK captures)
+        # Complements the conformance check with the OTHER direction: fails when
+        # Tango's shape trees expose a field/expand the SDK schema doesn't capture
+        # and it isn't in contracts/shape_coverage_baseline.json. Also offline
+        # against the vendored contract — no secrets, works on forks.
+        run: npx tsx scripts/check-shape-coverage.ts
+
+      - name: Check generated overlay is current
+        # Regenerates the overlay and fails on drift, so a contract or curated-schema change that alters generator output can't land without `npm run generate-shape-overlay`.
+        run: |
+          npm run generate-shape-overlay
+          git diff --exit-code src/shapes/generatedOverlay.ts
+
+      # --- Fresh-contract gates (token-gated hard checks against tango HEAD) --
       - name: Determine token availability
         id: gate
         env:
@@ -68,30 +100,34 @@ jobs:
             echo "ready=true" >> "$GITHUB_OUTPUT"
           else
             echo "ready=false" >> "$GITHUB_OUTPUT"
-            echo "::notice::Skipping SDK conformance check — TANGO_API_REPO_ACCESS_TOKEN not configured."
+            echo "::notice::Fresh-contract gates skipped — TANGO_API_REPO_ACCESS_TOKEN not configured."
           fi
 
-      - uses: actions/checkout@v4
-        if: steps.gate.outputs.ready == 'true'
-
-      - name: Checkout tango API repo (manifest source)
+      - name: Checkout tango API repo (contract source)
         if: steps.gate.outputs.ready == 'true'
         uses: actions/checkout@v4
         with:
           repository: makegov/tango
           path: tango-api
           token: ${{ secrets.TANGO_API_REPO_ACCESS_TOKEN }}
 
-      - name: Set up Node.js
+      - name: Warn when the vendored contract has drifted from tango HEAD
         if: steps.gate.outputs.ready == 'true'
-        uses: actions/setup-node@v4
-        with:
-          node-version: "20"
+        run: |
+          if ! diff -q contracts/filter_shape_contract.json tango-api/contracts/filter_shape_contract.json >/dev/null; then
+            echo "::warning::Vendored contract differs from makegov/tango HEAD. Re-vendor contracts/filter_shape_contract.json and regenerate the overlay."
+          else
+            echo "Vendored contract matches makegov/tango HEAD."
+          fi
 
-      - name: Install dependencies
+      - name: Check SDK filter/shape conformance (fresh contract, hard gate)
         if: steps.gate.outputs.ready == 'true'
-        run: npm install --ignore-scripts --no-audit --no-fund
+        env:
+          TANGO_CONTRACT_PATH: tango-api/contracts/filter_shape_contract.json
+        run: npx tsx scripts/check-filter-shape-conformance.ts
 
-      - name: Check SDK filter/shape conformance
+      - name: Check reverse shape coverage (fresh contract, hard gate)
         if: steps.gate.outputs.ready == 'true'
-        run: npx tsx scripts/check-filter-shape-conformance.ts --manifest tango-api/contracts/filter_shape_contract.json
+        env:
+          TANGO_CONTRACT_PATH: tango-api/contracts/filter_shape_contract.json
+        run: npx tsx scripts/check-shape-coverage.ts

.github/workflows/publish.yml

@@ -22,9 +22,8 @@ jobs:
           registry-url: "https://registry.npmjs.org"
 
       - name: Install dependencies
-        # No lockfile is committed (package-lock.json is gitignored), so npm ci will fail.
-        # We also ignore scripts to avoid running "prepare" during install; build is explicit below.
-        run: npm install --ignore-scripts --no-audit --no-fund
+        # Ignore scripts to avoid running "prepare" during install; build is explicit below.
+        run: npm ci --ignore-scripts --no-audit --no-fund
 
       - name: Lint
         run: npm run lint

.gitignore

@@ -4,7 +4,6 @@ npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
 pnpm-debug.log*
-package-lock.json
 pnpm-lock.yaml
 yarn.lock
 
@@ -47,4 +46,3 @@ yoni/
 .zed/
 .idea/
 # <<< mg-tools <<<
-CLAUDE.md

CHANGELOG.md

@@ -7,6 +7,39 @@ This project follows [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Added
+- **DIBBS, exclusions, and SBIR/STTR endpoint support** (parity with tango-python v1.3.0). Six endpoint families had no SDK support at all — no models, no methods. Added `listDibbsRfqs`/`getDibbsRfq`, `listDibbsRfps`/`getDibbsRfp`, `listDibbsAwards`/`getDibbsAward`, `listExclusions`/`getExclusion`, `listSbirTopics`/`getSbirTopic`, and `listSbirSolicitations`/`getSbirSolicitation`, with every filter param in the API contract exposed as a typed option, explicit shape schemas (including the nested organization/awardee/topic/document expands), `ShapeConfig` defaults, and async iterators (`iterateDibbsRfqs`, `iterateDibbsRfps`, `iterateDibbsAwards`, `iterateExclusions`, `iterateSbirTopics`, `iterateSbirSolicitations`, plus the matching `IterableListMethod` entries for the generic `iterate()`). New model interfaces: `DibbsRfq`, `DibbsRfp`, `DibbsAward`, `Exclusion`, `SbirTopic`, `SbirSolicitation`.
+
+  Two API behaviors are worth knowing. `is_open` (DIBBS) and `is_currently_excluded` (exclusions) are derived at query time, so filter with the `open` / `active` options rather than shaping on those fields. And DIBBS `total_contract_price` is the *order* total repeated on every line item — never sum it across rows; deduplicate on award + delivery-order number first.
+- **Full typed filter surface on `listBudgetAccounts`** (parity with tango-python and the API contract). `ListBudgetAccountsOptions` now declares every `budget/accounts` filter param — the exact / `__gte` / `__lte` triplet for all 26 numeric lifecycle, ratio, and trend fields (`requested_ba`, `enacted_ba`, `apportioned`, `obligated_total`, `outlayed_total`, `unobligated_balance`, the contract/assistance breakdowns, the `*_pct` / `*_capped` ratios, YoY + 5-year-CAGR trends, and `actual_vs_requested_contract`), plus the `__in` / `__icontains` variants of the categorical filters (`federal_account_symbol`, `fiscal_year`, `agency_code`, `bureau_name`, `bea_category`, `subfunction_code`, `account_title__icontains`).
+- `getGsaElibraryContract(uuid, options)` for `/api/gsa_elibrary_contracts/{uuid}/` (parity with tango-python), with the standard `shape` / `flat` / `flatLists` / `joiner` options and the `GSA_ELIBRARY_CONTRACTS_MINIMAL` default shape.
+- Typed filter options that previously worked only through the index-signature escape hatch: `key` on `listContracts` / `listIdvs` / `listOtas` / `listOtidvs`, `cage` on `listEntities`, `id` on `listForecasts`, `opportunity_id` on `listOpportunities`, `previous_uii` on `listItDashboard`, `naics_code` on `listProtests` (sent verbatim, not remapped to `naics`), and `has_awards` on `listPsc`. The filter-shape conformance gate now reports zero index-signature warnings.
+- **Generated shape-coverage overlay** (parity with tango-python v1.4.0): `src/shapes/generatedOverlay.ts`, machine-generated by the new `scripts/generate-shape-overlay.ts` from the vendored contract plus `contracts/observed_shape_types.json` (live-API type observations vendored from tango-python). `SchemaRegistry` merges the overlay over the curated explicit schemas, so the typed shape API now accepts every field and expand the API returns — including entity `relationships(type, source)`, previously-unmapped models (`Naics`, `PSC`, `MasSin`, `BudgetAccount`, `AssistanceListing`, `BusinessType`), and all the code/description expands that were flattened to scalars. The reverse shape-coverage gate now reports **zero** gaps and `contracts/shape_coverage_baseline.json` is empty (416 → 0).
+- **Agency-filter diagnostics on `PaginatedResponse`** (parity with tango-python v1.5.0). Every list method now surfaces the API's `meta` payload, plus three parsed views: `agencyWarnings` (human-readable notes about dropped or loosely-matched agency tokens), `unresolvedAgencyTokens` (tokens that matched no organization, keyed by filter name), and `resolvedAgencies` (the organizations each token actually resolved to — the only way to catch a token fuzzy-matching an agency you did not intend). All three are total: absent or malformed `meta` yields empty values, never a throw. On the `PaginatedResponse` type, `meta` and the three parsed views are declared as optional properties — responses built by the client always populate them, but existing code that constructs the type without them keeps compiling.
+- **Structured shape errors on `TangoValidationError`** (parity with tango-python's `.issues` / `.available_fields`): new `issues` and `availableFields` getters expose the API's structured 400 payload — entries like `{"path": "tradeoff_process", "reason": "unknown_field"}` and the endpoint's valid field set — instead of leaving callers to parse `responseData` by hand.
+- Vendored the canonical API filter/shape contract at `contracts/filter_shape_contract.json` (API 4.22.0), so conformance checking is fully offline — no token, no sibling checkout.
+- New reverse shape-coverage gate `scripts/check-shape-coverage.ts` (npm script `check-shape-coverage`): walks every resource's shape tree in the vendored contract against the SDK's explicit schema registry and fails on any field or expand the SDK does not capture, unless recorded in `contracts/shape_coverage_baseline.json` as tracked backlog.
+- Accepted-gaps baselines: `contracts/conformance_baseline.json` (missing filters + unimplemented resources) and `contracts/shape_coverage_baseline.json` (known shape-coverage gaps). Baselined gaps report as warnings; anything new is an error.
+- **Recorded integration test layer** (parity with tango-python's VCR-cassette suite): a record/replay harness (`tests/integration/harness.ts`) around the SDK's injectable `fetchImpl`, with JSON cassettes in `tests/cassettes/` recorded against the live API. Default runs replay offline — a missing cassette is a hard failure so drift is loud, while an absent cassettes directory (a fork without the corpus) skips the suite with a warning. `TANGO_REFRESH_CASSETTES=true` re-records serially against the live API; `TANGO_USE_LIVE_API=true` bypasses cassettes. Cassettes never store request headers, keep only an allowlisted response-header subset, and the recorder throws rather than serialize API-key material anywhere in an interaction (asserted by unit tests in `tests/unit/integration-harness.test.ts`).
+- Per-resource integration tests (`tests/integration/*.test.ts`, 44 tests / 44 cassettes) covering contracts (including cursor pagination and shaping), entities, IDVs, vehicles, opportunities, notices, grants, forecasts, agencies + organizations, protests, budget accounts (fiscal-year range round-trip), DIBBS, exclusions, SBIR, NAICS/PSC reference data, subawards, and edge cases (404, invalid ordering, empty result page).
+- **Env-gated production smoke suite** (`tests/production/smoke.test.ts`, the node port of tango-python's `tests/production/`): runs only with `TANGO_LIVE_TESTS=true` plus `TANGO_API_KEY`, asserting light live-API invariants (pagination shape, shaping, rate-limit header parsing). Excluded from default runs and CI by `vitest.config.ts`.
+
+### Changed
+- Both conformance baselines shrank with the new resources: `dibbs/*`, `exclusions`, and `sbir/*` left `unimplemented_resources` in `contracts/conformance_baseline.json`, and their `unmapped_resource` entries left `contracts/shape_coverage_baseline.json` (422 → 416 known gaps, then 416 → 0 with the generated overlay above).
+- `scripts/check-filter-shape-conformance.ts` now defaults to the vendored contract instead of a checked-out tango API repo (`TANGO_CONTRACT_PATH` or `--manifest` still point it at one), covers every resource in the 4.22.0 contract in its resource map, and treats an unimplemented resource as an error unless baselined.
+- Removed the dead legacy `.eslintrc.cjs` — the flat `eslint.config.js` has been the operative ESLint config since the flat-config migration, and the leftover file only invited divergent edits.
+
+### Fixed
+- The six new list methods (`listDibbsRfqs`, `listDibbsRfps`, `listDibbsAwards`, `listExclusions`, `listSbirTopics`, `listSbirSolicitations`) leaked a caller-supplied `joiner` to the server as a bare query param and ignored it when unflattening `flat: true` responses (always unflattening on the default `.`). `joiner` is now threaded the same way as `listIdvs`: sent only alongside `flat=true`, and used as the unflatten separator.
+- `listBudgetAccounts`: the `fiscal_year_gte`, `fiscal_year_lte`, and `account_title` options were sent verbatim, which the API silently ignores. They are kept as legacy aliases and now remapped to the forms the API understands (`fiscal_year__gte`, `fiscal_year__lte`, `account_title__icontains`); an explicitly passed dunder param wins over its alias.
+- Docs: removed the stale `getIdvSummary` / `listIdvSummaryAwards` sections from `README.md` and `docs/API_REFERENCE.md` — those methods were removed from the SDK in 1.1.0. Documented the full new surface (DIBBS/exclusions/SBIR, budget accounts and their filter surface, `getGsaElibraryContract`, the newly typed filters, `PaginatedResponse` meta diagnostics, structured `TangoValidationError` details), completed the `ShapeConfig` preset table in `docs/SHAPES.md`, and rewrote the maintainer half of `docs/DEVELOPERS.md` around the conformance gates and the cassette record/replay workflow (the old text still claimed the SDK had no cassette mechanism).
+
+### CI
+- `package-lock.json` is now committed, and the CI + publish workflows install with `npm ci --ignore-scripts` instead of `npm install` — installs are reproducible from the lockfile instead of re-resolving dependency ranges on every run.
+- The test job's Node 20 leg runs the suite once with `--coverage` (instead of a second full pass), the default `npx vitest run` now includes the integration suite replayed offline from the committed cassettes, and both setup-node steps cache the npm store off the committed lockfile. No coverage fail-under gate, matching tango-python.
+- The `conformance` job is now a hard gate that runs both conformance directions offline against the vendored contract on every PR — it no longer needs `TANGO_API_REPO_ACCESS_TOKEN` and no longer skips silently without it. When the token is configured, both checks additionally run as hard gates against the fresh contract at makegov/tango HEAD, with a re-vendor warning when the vendored copy has drifted.
+- The `conformance` job also regenerates `src/shapes/generatedOverlay.ts` and fails on any diff, so a contract or curated-schema change that alters generator output can't land without a regenerated overlay. `scripts/generate-shape-overlay.ts` honors the same `TANGO_CONTRACT_PATH` / `--contract` override as the two check scripts.
+
 ## [1.1.0] - 2026-05-29
 
 ### Changed (breaking)