oecd_regional_education |Increasing deletion threshold - #2209
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request updates the OECD regional education import manifest to include a node MCF file and a validation configuration file, and introduces a new validation configuration file to check the percentage of deleted records. The reviewer noted a mismatch in the validation configuration where the rule description specifies a 5% threshold, but the parameter is set to 4.
| "description": "Allow up to 5% deleted records due to OECD dataflow 2.5 NUTS 2024 regional restructuring and historical series revisions.", | ||
| "validator": "DELETED_RECORDS_PERCENT", | ||
| "params": { | ||
| "threshold": 4 |
There was a problem hiding this comment.
There is a mismatch between the rule description and the configured threshold. The description states "Allow up to 5% deleted records", but the threshold is set to 4. Please update the threshold to 5 to align with the description and the PR's intent of increasing the deletion threshold.
| "threshold": 4 | |
| "threshold": 5 |
The PR is raised to Allow up to 5% deleted records in OECD regional education validation.
Root Cause
OECD transitioned to dataflow 2.5 (OECD.CFE.EDS:DSD_REG_EDU@DF_ATTAIN(2.5)), introducing Eurostat NUTS 2024 regional restructuring (retiring obsolete territorial units such as PT16, PT17, NL31, and NL33) and pruning historical zero-value estimates. This resulted in 9,531 legitimate record deletions (~3.90% of total observations).