Skip to content

test(integration): add multi-entity variable ingestion test dataset - #263

Open
dwnoble wants to merge 3 commits into
mainfrom
test/multi-entity-financial-trade
Open

test(integration): add multi-entity variable ingestion test dataset#263
dwnoble wants to merge 3 commits into
mainfrom
test/multi-entity-financial-trade

Conversation

@dwnoble

@dwnoble dwnoble commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Problem

The test suite lacked end-to-end integration coverage for multi-entity statistical variables where observations link multiple entity dimensions (such as source and destination countries). We also lacked test coverage for MCF observationProperties parsing and SDMX 3.0 multi-dimension queries.

Solution

  • tests/integration/test_data/financial_trade/: Added a new test dataset modeling trade flows between countries using fictional provenance (FooBarTrade), custom dimension properties (sourceCountry, destinationCountry), and SDMX 3.0 data and availability queries.
  • test_mcf_parser.py: Added unit test verifying MCF parsing of nodes with observationProperties and custom property definitions.
  • test_config_schema.py: Added unit test verifying loading of multi-entity dataset manifests.
  • test_ingestion.py: Allowed test_02_cli_init_db to pass when targeting an already-initialized database.

Verification

  • Unit tests pass (uv run pytest packages/datacommons-schema/tests/parsers/test_mcf_parser.py tests/integration/unit_tests/test_config_schema.py, 24 passed).
  • Linter passes (uv run ruff check).
  • Full end-to-end integration test run against testbed-1 succeeded (run_e2e_tests.py --instance testbed-1 --test-config financial_trade, 13 passed in 1150.75s, Dataflow job testbed-1-financial-trade-1788932800).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new integration test dataset (financial_trade) to verify the ingestion and serving of multi-entity statistical variables with custom observation properties. It includes corresponding MCF, CSV, configuration, and test specification files, along with unit tests for MCF parsing and configuration schema loading. The feedback suggests improving the MCF definition by explicitly prefixing observationProperties with dcid: to prevent namespace resolution issues, and expanding the integration test assertions in test_spec.yaml to verify that these properties are correctly ingested as edges in the database.

Comment thread tests/integration/test_data/financial_trade/financial_trade.mcf Outdated
Comment thread tests/integration/test_data/financial_trade/test_spec.yaml
@dwnoble
dwnoble marked this pull request as ready for review September 9, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant