Skip to content

OECDRegionalDemography_Population| Fixed missing data issue - #2152

Merged
balit-raibot merged 18 commits into
datacommonsorg:masterfrom
Ashwani725:OECDRegionalDemography_Population_missing_data
Sep 8, 2026
Merged

OECDRegionalDemography_Population| Fixed missing data issue#2152
balit-raibot merged 18 commits into
datacommonsorg:masterfrom
Ashwani725:OECDRegionalDemography_Population_missing_data

Conversation

@Ashwani725

@Ashwani725 Ashwani725 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

This PR is raised to fix validation failure in OECDRegionalDemography_Population job due to change in working age cohorts in upstream source.

Support broad working-age cohorts (20-64, 25-54, 55-64) in OECD Regional Demography Population

Add support for OECD regional demography broad working-age population cohorts:

  • Added StatVar mappings for 20-64, 25-54, and 55-64 years (Total, Male, and Female variants).
  • Added schema definitions for new non-canonical StatVars in OECD_population_stat_vars.mcf (deduplicated against existing canonical Data Commons schema nodes).
  • Configured date freshness validation rules via validation_config.json to ensure broad cohorts have data through at least 2025 and standard 5-year cohorts through at least 2023.
  • Updated sample data and golden fixtures. Note: Upstream OECD currently reports data exclusively for Total (_T) for broad age groups; gender-stratified mappings are preserved for future-proofing and remain unpopulated in current outputs.

Test result:
Test Cloud Batch Job
Test GCS Bucket
CL link

@google-cla

google-cla Bot commented Aug 10, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces new statistical variables for age groups 20-64, 25-54, and 55-64, including gender-specific breakdowns, and updates the associated column mappings and golden data. Additionally, a new validation configuration file is added to the manifest. The reviewer recommended updating the rule_id in the validation configuration for consistency and noted that the expected test files must be updated to reflect these schema changes to prevent unit test failures.

Comment thread scripts/oecd/regional_demography/population/validation_config.json Outdated
Comment thread scripts/oecd/regional_demography/population/columns.py
@Ashwani725 Ashwani725 changed the title fixed missing data issue OECDRegionalDemography_Population| Fixed missing data issue Aug 11, 2026
Removed validation rules related to golden records checks.
Comment thread scripts/oecd/regional_demography/population/manifest.json
@saanikaaa

Copy link
Copy Markdown
Contributor

Can we pls run differ to validated our changes

@Ashwani725 Ashwani725 closed this Sep 2, 2026
@Ashwani725 Ashwani725 reopened this Sep 2, 2026
@Ashwani725

Copy link
Copy Markdown
Contributor Author

Can we pls run differ to validated our changes

Please find the attached differ_summary report
differ_summary report

@saanikaaa

Copy link
Copy Markdown
Contributor

/gcbrun

Comment thread scripts/oecd/regional_demography/population/manifest.json Outdated
@balit-raibot

Copy link
Copy Markdown
Contributor

/gcbrun

@balit-raibot
balit-raibot enabled auto-merge (squash) September 8, 2026 13:05
@balit-raibot
balit-raibot merged commit 1c3f47b into datacommonsorg:master Sep 8, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants