test(integration): add multi-entity variable ingestion test dataset - #263
Open
dwnoble wants to merge 3 commits into
Open
test(integration): add multi-entity variable ingestion test dataset#263dwnoble wants to merge 3 commits into
dwnoble wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new integration test dataset (financial_trade) to verify the ingestion and serving of multi-entity statistical variables with custom observation properties. It includes corresponding MCF, CSV, configuration, and test specification files, along with unit tests for MCF parsing and configuration schema loading. The feedback suggests improving the MCF definition by explicitly prefixing observationProperties with dcid: to prevent namespace resolution issues, and expanding the integration test assertions in test_spec.yaml to verify that these properties are correctly ingested as edges in the database.
dwnoble
marked this pull request as ready for review
September 9, 2026 06:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The test suite lacked end-to-end integration coverage for multi-entity statistical variables where observations link multiple entity dimensions (such as source and destination countries). We also lacked test coverage for MCF
observationPropertiesparsing and SDMX 3.0 multi-dimension queries.Solution
tests/integration/test_data/financial_trade/: Added a new test dataset modeling trade flows between countries using fictional provenance (FooBarTrade), custom dimension properties (sourceCountry,destinationCountry), and SDMX 3.0 data and availability queries.test_mcf_parser.py: Added unit test verifying MCF parsing of nodes withobservationPropertiesand custom property definitions.test_config_schema.py: Added unit test verifying loading of multi-entity dataset manifests.test_ingestion.py: Allowedtest_02_cli_init_dbto pass when targeting an already-initialized database.Verification
uv run pytest packages/datacommons-schema/tests/parsers/test_mcf_parser.py tests/integration/unit_tests/test_config_schema.py, 24 passed).uv run ruff check).testbed-1succeeded (run_e2e_tests.py --instance testbed-1 --test-config financial_trade, 13 passed in 1150.75s, Dataflow jobtestbed-1-financial-trade-1788932800).