CaliforniaSchoolPerformance : Automation update - #2198
Conversation
Code fix unenergy
- Add download.py to fetch and normalize multi-year CAASPP research files (2015-2019, 2021-2025) - Add PV map, metadata config, and StatVar MCF in config/ - Add manifest.json with annual cron schedule - Add golden summary report in golden_data/ - Add unit tests in test_data/ - Add README documentation and run_download_process.sh
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an automated Data Commons import pipeline for California School Performance (CAASPP) academic data, including download scripts, configuration mappings, orchestration bash scripts, and unit tests. The review feedback highlights several areas for improvement: adding robust error handling in download.py to raise exceptions on failed downloads and prevent silent pipeline failures, wrapping year-range parsing in try-except blocks to handle malformed CLI inputs, making the state/county entity filtering logic more resilient to padding variations, and removing redundant individual-year processing in run_download_process.sh to optimize execution time.
- Add validation_config.json with 0.1 deletion threshold and golden check rules - Add golden_observations.csv generated via validator_goldens.py against top 100k places - Update golden_summary_report.csv with standardized stable schema property sets - Add counters/california_school_performance_counters.csv and configure in manifest.json - Register node_mcf in manifest.json for generated MCF staging to GCS
… filtering, and script optimization
- Raise RuntimeError if no files were successfully downloaded in download.py
- Add try-except to handle malformed year range format in parse_years
- Make state/county entity filtering robust to zero-padding variations using lstrip('0')
- Remove redundant individual-year processing loop in run_download_process.sh
There was a problem hiding this comment.
we don't push mcf files in git
There was a problem hiding this comment.
removed any .MCF file pushed.
There was a problem hiding this comment.
move this out of test_data folder
There was a problem hiding this comment.
done. also renamed it as california_school_performance_test.py as per naming conventions
There was a problem hiding this comment.
delete this as not required
2 participants
Summary
Converts the
CaliforniaSchoolPerformanceimport from a legacy manual script to an automated StatVar import with multi-year data support (2015–2019, 2021–2025; 2020 was cancelled due to COVID-19).Key Changes
download.py: Downloads and unifies CAASPP research files across three different historical format eras (comma-delimited 2015–2019, caret-delimited 2021–2023, and updated 69-column caret format 2024–2025). Filters for state (00000/0000000) and county levels.config/): PV mapping, metadata configuration, and StatVar MCF covering grades 3–8, 11, 13 for ELA & Math across 55 demographic subgroups.manifest.json): Configured with an annual cron schedule (0 0 1 10 *) matching the CAASPP October release cadence.test_california_school_performance.pyandgolden_summary_report.csv.README.mdandrun_download_process.shscript.Verification & Testing
python3 -m unittest test_data/test_california_school_performance.pyRelated CL
Internal Google3 CL updating
.textprotomanifest, provenance.mcf, and import group test.Code Review And Troubleshooting Report