diff --git a/statvar_imports/us_education/california_school_performance/README.md b/statvar_imports/us_education/california_school_performance/README.md new file mode 100644 index 0000000000..3bd06974b7 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/README.md @@ -0,0 +1,113 @@ +# California School Performance (CAASPP) Data Commons Import + +## Overview +This directory contains the automated Data Commons import pipeline for California public school academic performance data from the **California Assessment of Student Performance and Progress (CAASPP)** Smarter Balanced Summative Assessments. + +The data covers student achievement in **English Language Arts/Literacy (ELA)** and **Mathematics** across California public schools, school districts, counties, and statewide aggregates from 2015 to the present. + +- **Primary Source**: California Department of Education (CDE) / Educational Testing Service (ETS) +- **Research Portal**: [https://caaspp-elpac.ets.org/caaspp/ResearchFileListSB](https://caaspp-elpac.ets.org/caaspp/ResearchFileListSB) +- **Import Tier**: Automated StatVar Import + +--- + +## Directory Structure + +``` +california_school_performance/ +├── config/ +│ ├── california_school_performance_metadata.csv # Processor configurations (delimiters, headers, output columns) +│ └── california_school_performance_pvmap.csv # Property-Value mappings for subjects, grades, demographics, and metrics +├── counters/ +│ └── california_school_performance_counters.csv # Transformation counter summary +├── golden_data/ +│ ├── golden_observations.csv # Critical golden entity/place observations +│ └── golden_summary_report.csv # Golden summary report for statvar validation +├── test_data/ # Test fixtures for E2E validation +│ ├── sample_input.txt # Sample raw CAASPP input records +│ ├── sample_state_output.csv # Expected sample CSV observations +│ └── sample_state_output.tmcf # Expected sample TMCF mapping +├── california_school_performance_test.py # Unit tests for download and stream normalization +├── download.py # Automated data fetch and extraction script +├── manifest.json # Data Commons import specification +├── README.md # Documentation and usage guide +└── validation_config.json # Production import validation rules +``` + + +--- + +## Statistical Variables (StatVars) + +The pipeline maps raw test records into **5,966 distinct Statistical Variables** conforming to the Data Commons education schema: + +- **Population Type**: `Student` +- **School Subjects**: + - `EnglishLanguageArts` (Test ID: 1) + - `Mathematics` (Test ID: 2) +- **Grade Levels**: + - Grades 3, 4, 5, 6, 7, 8, 11 (`dcid:SchoolGrade3` – `dcid:SchoolGrade11`) + - Grade 13 represents "All Grades combined", mapped to unconstrained student population StatVars without grade constraints (e.g. `Count_Student_EnglishLanguageArts`) +- **Demographic Subgroups (55 groups)**: + - Gender (`Male`, `Female`) + - Race / Ethnicity (`White`, `BlackOrAfricanAmericanAlone`, `Asian`, `Filipino`, `HispanicOrLatino`, `AmericanIndianOrAlaskaNative`, `NativeHawaiianOrOtherPacificIslanderAlone`, `TwoOrMoreRaces`) + - Socioeconomic Status (`EconomicallyDisadvantaged`, `NotEconomicallyDisadvantaged`) + - Intersection of Race × Socioeconomic Status (16 groups) + - Disability Status (`WithDisability`, `NoDisability`) + - English Learner & Fluency Status (`InitialFluentProficient`, `ReclassifiedFluentProficient`, `OnlyEnglish`, `CurrentLearner`, `EverLearned`, `Adult`, `ToBeDetermined`, duration `< 12 months`, duration `>= 12 months`) + - Parent Education Level (`LessThanHighSchoolGraduate`, `HighSchoolGraduateIncludesEquivalency`, `SomeCollegeNoDegree`, `CollegeGraduate`, `GraduateSchoolOrPostGraduate`, `CA_DeclinedToState`) + - Special Student Populations (`Homeless`, `HavingHome`, `Foster`, `NotFoster`, `Migrant`, `NotMigrant`, `FamilyOfArmedForces`, `NotFamilyOfArmedForces`) +- **Metrics**: + - `Total Students Tested with Scores`: Count of students (`measuredProperty: count`) + - `Mean Scale Score`: Mean assessment score (`measuredProperty: assessmentScore`, `statType: meanValue`) + - `Percentage Standard Exceeded`: Educational achievement level `CA_StandardExceeded` (`scalingFactor: 100`, `unit: Percent`) + - `Percentage Standard Met`: Educational achievement level `CA_StandardMet` (`scalingFactor: 100`, `unit: Percent`) + - `Percentage Standard Met and Above`: Educational achievement level `CA_StandardMetAndAbove` (`scalingFactor: 100`, `unit: Percent`) + - `Percentage Standard Nearly Met`: Educational achievement level `CA_StandardNearlyMet` (`scalingFactor: 100`, `unit: Percent`) + - `Percentage Standard Not Met`: Educational achievement level `CA_StandardNotMet` (`scalingFactor: 100`, `unit: Percent`) + +--- + +## How to Run + +### 1. Download and Process the Entire Dataset (All Available Years: 2015–2025) +To download and process all years present at source (skipping 2020 when CAASPP was cancelled statewide due to COVID-19): +```bash +# Step 1: Download and normalize data +python3 download.py --years=all + +# Step 2: Generate TMCF and CSV observations using stat_var_processor +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + --input_data=input_files/sb_ca_all_years_normalized.txt \ + --pv_map=config/california_school_performance_pvmap.csv \ + --config_file=config/california_school_performance_metadata.csv \ + --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \ + --output_path=output_files/california_school_performance_all_years_output \ + --output_counters=counters/california_school_performance_counters.csv +``` +This automatically fetches each year, normalizes differences across formats, and runs `stat_var_processor.py` over the consolidated multi-year dataset (`sb_ca_all_years_normalized.txt`), generating **64,000+ observations** across all 10 years. + +### 2. Download and Process Specific Years +To fetch and process specific years: +```bash +# Specific range +python3 download.py --years=2015-2024 + +# Specific single year +python3 download.py --years=2024 +``` + +### 3. Quick Test Run +To download a lightweight test sample and verify the pipeline: +```bash +python3 download.py --test_mode +``` + +--- + +## Output Files +The pipeline produces standard Data Commons import artifacts in `output_files/`: +- `california_school_performance_all_years_output.csv`: Complete multi-year observations table (~64,000 rows across 2015–2025) +- `california_school_performance_all_years_output.tmcf`: Template MCF linking columns to Data Commons schema nodes +- `california_school_performance_{YEAR}_output.csv`: Per-year individual observation tables + diff --git a/statvar_imports/us_education/california_school_performance/california_school_performance_test.py b/statvar_imports/us_education/california_school_performance/california_school_performance_test.py new file mode 100644 index 0000000000..a7c95b4f7e --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/california_school_performance_test.py @@ -0,0 +1,119 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Unit tests for California School Performance (CAASPP) download and normalization.""" + +import io +import os +import sys +import unittest +from unittest.mock import patch + +# Ensure import directory is in sys.path +_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +if _SCRIPT_DIR not in sys.path: + sys.path.insert(0, _SCRIPT_DIR) + +import download + + +class CaliforniaSchoolPerformanceDownloadTest(unittest.TestCase): + + def test_discover_year_urls_known_years(self): + """Verify known year URL mapping retrieval.""" + urls_2015 = download.discover_year_urls(2015) + self.assertIsNotNone(urls_2015) + self.assertEqual(urls_2015, + ('sb_ca2015_all_csv_v3.zip', 'sb_ca2015_1_csv_v3.zip')) + + urls_2024 = download.discover_year_urls(2024) + self.assertIsNotNone(urls_2024) + self.assertEqual(urls_2024, + ('sb_ca2024_all_csv_v1.zip', 'sb_ca2024_1_csv_v1.zip')) + + @patch.object(download, 'probe_url_exists', return_value=False) + def test_download_parse_years(self, _): + """Verify year string parsing logic for single, list, range, and 'all'.""" + self.assertEqual(download.parse_years('2023'), [2023]) + self.assertEqual(download.parse_years('2023, 2024'), [2023, 2024]) + self.assertEqual(download.parse_years('2021-2023'), [2021, 2022, 2023]) + all_years = download.parse_years('all') + self.assertIn(2015, all_years) + self.assertIn(2024, all_years) + + def test_normalize_and_filter_stream_caret_delimited(self): + """Verify record normalization and entity filtering for caret-delimited format.""" + sample_caret = ( + 'County Code^District Code^School Code^Type ID^Test Year^Test ID^Student Group ID^Grade^' + 'Total Students Tested with Scores^Mean Scale Score^Percentage Standard Exceeded^' + 'Percentage Standard Met^Percentage Standard Met and Above^Percentage Standard Nearly Met^Percentage Standard Not Met\n' + '00^00000^0000000^4^2024^1^1^3^100^2400.0^20.0^25.0^45.0^25.0^30.0\n' + '01^00000^0000000^5^2024^1^1^3^60^2410.0^22.0^28.0^50.0^20.0^30.0\n' + '01^12345^6789012^7^2024^1^1^3^50^2350.0^10.0^20.0^30.0^30.0^40.0\n' + ) + stream = io.StringIO(sample_caret) + rows = list( + download.normalize_and_filter_stream(stream, + keep_all_entities=False)) + # School-level entity (Type ID 7) should be filtered out; State (4) and County (5) kept + self.assertEqual(len(rows), 2) + # State record + self.assertEqual(rows[0][0], '00') + self.assertEqual(rows[0][1], '2024') + self.assertEqual(rows[0][2], '1') # Student Group ID + self.assertEqual(rows[0][3], '3') # Grade + self.assertEqual(rows[0][4], '1') # Test ID + self.assertEqual(rows[0][5], '100') # Tested with scores + # County record + self.assertEqual(rows[1][0], '01') + self.assertEqual(rows[1][1], '2024') + + def test_normalize_and_filter_stream_comma_delimited_legacy_headers(self): + """Verify record normalization for comma-delimited data with older header aliases.""" + sample_comma = ( + '"County Code","District Code","School Code","Type ID","Test Year","Test Id","Subgroup ID","Grade",' + '"Students with Scores","Mean Scale Score","Percentage Standard Exceeded",' + '"Percentage Standard Met","Percentage Standard Met and Above","Percentage Standard Nearly Met","Percentage Standard Not Met"\n' + '"00","00000","0000000","4","2016","1","1","4","120","2450.0","25.0","30.0","55.0","20.0","25.0"\n' + ) + stream = io.StringIO(sample_comma) + rows = list( + download.normalize_and_filter_stream(stream, + keep_all_entities=False)) + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0][0], '00') + self.assertEqual(rows[0][1], '2016') + self.assertEqual(rows[0][2], '1') # Mapped Subgroup ID -> index 2 + self.assertEqual(rows[0][3], '4') # Grade + self.assertEqual(rows[0][4], '1') # Mapped Test Id -> index 4 + self.assertEqual(rows[0][5], + '120') # Mapped Students with Scores -> index 5 + + def test_normalize_and_filter_stream_keep_all_entities(self): + """Verify keep_all_entities=True retains school and district level entities.""" + sample_caret = ( + 'County Code^District Code^School Code^Type ID^Test Year^Test ID^Student Group ID^Grade^' + 'Total Students Tested with Scores^Mean Scale Score^Percentage Standard Exceeded^' + 'Percentage Standard Met^Percentage Standard Met and Above^Percentage Standard Nearly Met^Percentage Standard Not Met\n' + '00^00000^0000000^4^2024^1^1^3^100^2400.0^20.0^25.0^45.0^25.0^30.0\n' + '01^12345^6789012^7^2024^1^1^3^50^2350.0^10.0^20.0^30.0^30.0^40.0\n' + ) + stream = io.StringIO(sample_caret) + rows = list( + download.normalize_and_filter_stream(stream, + keep_all_entities=True)) + self.assertEqual(len(rows), 2) + + +if __name__ == '__main__': + unittest.main() diff --git a/statvar_imports/us_education/california_school_performance/config/california_school_performance_metadata.csv b/statvar_imports/us_education/california_school_performance/config/california_school_performance_metadata.csv new file mode 100644 index 0000000000..1bfebd9871 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/config/california_school_performance_metadata.csv @@ -0,0 +1,5 @@ +key,value +header_rows,1 +word_delimiter,"""""" +input_delimiter,^ +output_columns,"observationDate,observationAbout,variableMeasured,value,unit,scalingFactor" diff --git a/statvar_imports/us_education/california_school_performance/config/california_school_performance_pvmap.csv b/statvar_imports/us_education/california_school_performance/config/california_school_performance_pvmap.csv new file mode 100644 index 0000000000..37574830f7 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/config/california_school_performance_pvmap.csv @@ -0,0 +1,174 @@ +key,prop,val,prop2,val2,prop3,val3,prop4,val4,prop5,val5,prop6,val6,prop7,val7,prop8,val8 +Test Year,observationDate,{Number},,,,,,,,,,,,,, +County Code:00,observationAbout,geoId/06,,,,,,,,,,,,,, +County Code:0,observationAbout,geoId/06,,,,,,,,,,,,,, +County Code:1,observationAbout,geoId/06001,,,,,,,,,,,,,, +County Code:01,observationAbout,geoId/06001,,,,,,,,,,,,,, +County Code:2,observationAbout,geoId/06003,,,,,,,,,,,,,, +County Code:02,observationAbout,geoId/06003,,,,,,,,,,,,,, +County Code:3,observationAbout,geoId/06005,,,,,,,,,,,,,, +County Code:03,observationAbout,geoId/06005,,,,,,,,,,,,,, +County Code:4,observationAbout,geoId/06007,,,,,,,,,,,,,, +County Code:04,observationAbout,geoId/06007,,,,,,,,,,,,,, +County Code:5,observationAbout,geoId/06009,,,,,,,,,,,,,, +County Code:05,observationAbout,geoId/06009,,,,,,,,,,,,,, +County Code:6,observationAbout,geoId/06011,,,,,,,,,,,,,, +County Code:06,observationAbout,geoId/06011,,,,,,,,,,,,,, +County Code:7,observationAbout,geoId/06013,,,,,,,,,,,,,, +County Code:07,observationAbout,geoId/06013,,,,,,,,,,,,,, +County Code:8,observationAbout,geoId/06015,,,,,,,,,,,,,, +County Code:08,observationAbout,geoId/06015,,,,,,,,,,,,,, +County Code:9,observationAbout,geoId/06017,,,,,,,,,,,,,, +County Code:09,observationAbout,geoId/06017,,,,,,,,,,,,,, +County Code:10,observationAbout,geoId/06019,,,,,,,,,,,,,, +County Code:11,observationAbout,geoId/06021,,,,,,,,,,,,,, +County Code:12,observationAbout,geoId/06023,,,,,,,,,,,,,, +County Code:13,observationAbout,geoId/06025,,,,,,,,,,,,,, +County Code:14,observationAbout,geoId/06027,,,,,,,,,,,,,, +County Code:15,observationAbout,geoId/06029,,,,,,,,,,,,,, +County Code:16,observationAbout,geoId/06031,,,,,,,,,,,,,, +County Code:17,observationAbout,geoId/06033,,,,,,,,,,,,,, +County Code:18,observationAbout,geoId/06035,,,,,,,,,,,,,, +County Code:19,observationAbout,geoId/06037,,,,,,,,,,,,,, +County Code:20,observationAbout,geoId/06039,,,,,,,,,,,,,, +County Code:21,observationAbout,geoId/06041,,,,,,,,,,,,,, +County Code:22,observationAbout,geoId/06043,,,,,,,,,,,,,, +County Code:23,observationAbout,geoId/06045,,,,,,,,,,,,,, +County Code:24,observationAbout,geoId/06047,,,,,,,,,,,,,, +County Code:25,observationAbout,geoId/06049,,,,,,,,,,,,,, +County Code:26,observationAbout,geoId/06051,,,,,,,,,,,,,, +County Code:27,observationAbout,geoId/06053,,,,,,,,,,,,,, +County Code:28,observationAbout,geoId/06055,,,,,,,,,,,,,, +County Code:29,observationAbout,geoId/06057,,,,,,,,,,,,,, +County Code:30,observationAbout,geoId/06059,,,,,,,,,,,,,, +County Code:31,observationAbout,geoId/06061,,,,,,,,,,,,,, +County Code:32,observationAbout,geoId/06063,,,,,,,,,,,,,, +County Code:33,observationAbout,geoId/06065,,,,,,,,,,,,,, +County Code:34,observationAbout,geoId/06067,,,,,,,,,,,,,, +County Code:35,observationAbout,geoId/06069,,,,,,,,,,,,,, +County Code:36,observationAbout,geoId/06071,,,,,,,,,,,,,, +County Code:37,observationAbout,geoId/06073,,,,,,,,,,,,,, +County Code:38,observationAbout,geoId/06075,,,,,,,,,,,,,, +County Code:39,observationAbout,geoId/06077,,,,,,,,,,,,,, +County Code:40,observationAbout,geoId/06079,,,,,,,,,,,,,, +County Code:41,observationAbout,geoId/06081,,,,,,,,,,,,,, +County Code:42,observationAbout,geoId/06083,,,,,,,,,,,,,, +County Code:43,observationAbout,geoId/06085,,,,,,,,,,,,,, +County Code:44,observationAbout,geoId/06087,,,,,,,,,,,,,, +County Code:45,observationAbout,geoId/06089,,,,,,,,,,,,,, +County Code:46,observationAbout,geoId/06091,,,,,,,,,,,,,, +County Code:47,observationAbout,geoId/06093,,,,,,,,,,,,,, +County Code:48,observationAbout,geoId/06095,,,,,,,,,,,,,, +County Code:49,observationAbout,geoId/06097,,,,,,,,,,,,,, +County Code:50,observationAbout,geoId/06099,,,,,,,,,,,,,, +County Code:51,observationAbout,geoId/06101,,,,,,,,,,,,,, +County Code:52,observationAbout,geoId/06103,,,,,,,,,,,,,, +County Code:53,observationAbout,geoId/06105,,,,,,,,,,,,,, +County Code:54,observationAbout,geoId/06107,,,,,,,,,,,,,, +County Code:55,observationAbout,geoId/06109,,,,,,,,,,,,,, +County Code:56,observationAbout,geoId/06111,,,,,,,,,,,,,, +County Code:57,observationAbout,geoId/06113,,,,,,,,,,,,,, +County Code:58,observationAbout,geoId/06115,,,,,,,,,,,,,, +Grade:3,schoolGradeLevel,SchoolGrade3,,,,,,,,,,,,,, +Grade:03,schoolGradeLevel,SchoolGrade3,,,,,,,,,,,,,, +Grade:4,schoolGradeLevel,SchoolGrade4,,,,,,,,,,,,,, +Grade:04,schoolGradeLevel,SchoolGrade4,,,,,,,,,,,,,, +Grade:5,schoolGradeLevel,SchoolGrade5,,,,,,,,,,,,,, +Grade:05,schoolGradeLevel,SchoolGrade5,,,,,,,,,,,,,, +Grade:6,schoolGradeLevel,SchoolGrade6,,,,,,,,,,,,,, +Grade:06,schoolGradeLevel,SchoolGrade6,,,,,,,,,,,,,, +Grade:7,schoolGradeLevel,SchoolGrade7,,,,,,,,,,,,,, +Grade:07,schoolGradeLevel,SchoolGrade7,,,,,,,,,,,,,, +Grade:8,schoolGradeLevel,SchoolGrade8,,,,,,,,,,,,,, +Grade:08,schoolGradeLevel,SchoolGrade8,,,,,,,,,,,,,, +Grade:11,schoolGradeLevel,SchoolGrade11,,,,,,,,,,,,,, +Test ID:1,schoolSubject,EnglishLanguageArts,,,,,,,,,,,,,, +Test ID:01,schoolSubject,EnglishLanguageArts,,,,,,,,,,,,,, +Test ID:2,schoolSubject,Mathematics,,,,,,,,,,,,,, +Test ID:02,schoolSubject,Mathematics,,,,,,,,,,,,,, +Student Group ID:3,gender,Male,,,,,,,,,,,,,, +Student Group ID:003,gender,Male,,,,,,,,,,,,,, +Student Group ID:4,gender,Female,,,,,,,,,,,,,, +Student Group ID:004,gender,Female,,,,,,,,,,,,,, +Student Group ID:6,languageFluencyLevel,InitialFluentProficient__ReclassifiedFluentProficient__OnlyEnglish,,,,,,,,,,,,,, +Student Group ID:006,languageFluencyLevel,InitialFluentProficient__ReclassifiedFluentProficient__OnlyEnglish,,,,,,,,,,,,,, +Student Group ID:7,languageFluencyLevel,InitialFluentProficient,,,,,,,,,,,,,, +Student Group ID:007,languageFluencyLevel,InitialFluentProficient,,,,,,,,,,,,,, +Student Group ID:8,languageFluencyLevel,ReclassifiedFluentProficient,,,,,,,,,,,,,, +Student Group ID:008,languageFluencyLevel,ReclassifiedFluentProficient,,,,,,,,,,,,,, +Student Group ID:28,socialVulnerabilityStatus,Migrant,,,,,,,,,,,,,, +Student Group ID:028,socialVulnerabilityStatus,Migrant,,,,,,,,,,,,,, +Student Group ID:29,socialVulnerabilityStatus,NotMigrant,,,,,,,,,,,,,, +Student Group ID:029,socialVulnerabilityStatus,NotMigrant,,,,,,,,,,,,,, +Student Group ID:31,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,,,, +Student Group ID:031,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,,,, +Student Group ID:50,armedForcesStatus,FamilyOfArmedForces,,,,,,,,,,,,,, +Student Group ID:050,armedForcesStatus,FamilyOfArmedForces,,,,,,,,,,,,,, +Student Group ID:51,armedForcesStatus,NotFamilyOfArmedForces,,,,,,,,,,,,,, +Student Group ID:051,armedForcesStatus,NotFamilyOfArmedForces,,,,,,,,,,,,,, +Student Group ID:52,socialVulnerabilityStatus,Homeless,,,,,,,,,,,,,, +Student Group ID:052,socialVulnerabilityStatus,Homeless,,,,,,,,,,,,,, +Student Group ID:53,socialVulnerabilityStatus,HavingHome,,,,,,,,,,,,,, +Student Group ID:053,socialVulnerabilityStatus,HavingHome,,,,,,,,,,,,,, +Student Group ID:74,race,BlackOrAfricanAmericanAlone,,,,,,,,,,,,,, +Student Group ID:074,race,BlackOrAfricanAmericanAlone,,,,,,,,,,,,,, +Student Group ID:75,race,AmericanIndianOrAlaskaNative,,,,,,,,,,,,,, +Student Group ID:075,race,AmericanIndianOrAlaskaNative,,,,,,,,,,,,,, +Student Group ID:76,race,Asian,,,,,,,,,,,,,, +Student Group ID:076,race,Asian,,,,,,,,,,,,,, +Student Group ID:77,race,Filipino,,,,,,,,,,,,,, +Student Group ID:077,race,Filipino,,,,,,,,,,,,,, +Student Group ID:78,race,HispanicOrLatino,,,,,,,,,,,,,, +Student Group ID:078,race,HispanicOrLatino,,,,,,,,,,,,,, +Student Group ID:79,race,NativeHawaiianOrOtherPacificIslanderAlone,,,,,,,,,,,,,, +Student Group ID:079,race,NativeHawaiianOrOtherPacificIslanderAlone,,,,,,,,,,,,,, +Student Group ID:80,race,White,,,,,,,,,,,,,, +Student Group ID:080,race,White,,,,,,,,,,,,,, +Student Group ID:90,parentEducationStatus,LessThanHighSchoolGraduate,,,,,,,,,,,,,, +Student Group ID:090,parentEducationStatus,LessThanHighSchoolGraduate,,,,,,,,,,,,,, +Student Group ID:91,parentEducationStatus,HighSchoolGraduateIncludesEquivalency,,,,,,,,,,,,,, +Student Group ID:091,parentEducationStatus,HighSchoolGraduateIncludesEquivalency,,,,,,,,,,,,,, +Student Group ID:92,parentEducationStatus,SomeCollegeNoDegree,,,,,,,,,,,,,, +Student Group ID:092,parentEducationStatus,SomeCollegeNoDegree,,,,,,,,,,,,,, +Student Group ID:93,parentEducationStatus,CollegeGraduate,,,,,,,,,,,,,, +Student Group ID:093,parentEducationStatus,CollegeGraduate,,,,,,,,,,,,,, +Student Group ID:94,parentEducationStatus,GraduateSchoolOrPostGraduate,,,,,,,,,,,,,, +Student Group ID:094,parentEducationStatus,GraduateSchoolOrPostGraduate,,,,,,,,,,,,,, +Student Group ID:99,disabilityStatus,NoDisability,,,,,,,,,,,,,, +Student Group ID:099,disabilityStatus,NoDisability,,,,,,,,,,,,,, +Student Group ID:111,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,,,, +Student Group ID:120,languageLearned,EnglishLearner,languageLearningDuration,[- 12 Month],,,,,,,,,,,, +Student Group ID:121,parentEducationStatus,CA_DeclinedToState,,,,,,,,,,,,,, +Student Group ID:128,disabilityStatus,WithDisability,,,,,,,,,,,,,, +Student Group ID:142,languageLearned,EnglishLearner,languageLearningDuration,[12 - Month],,,,,,,,,,,, +Student Group ID:144,race,TwoOrMoreRaces,,,,,,,,,,,,,, +Student Group ID:160,languageLearned,English,languageLearningStatus,CurrentLearner,,,,,,,,,,,, +Student Group ID:170,languageLearned,English,languageLearningStatus,EverLearned,,,,,,,,,,,, +Student Group ID:180,languageLearned,OnlyEnglish,,,,,,,,,,,,,, +Student Group ID:190,languageLearned,English,languageLearningStatus,ToBeDetermined,,,,,,,,,,,, +Student Group ID:200,race,BlackOrAfricanAmericanAlone,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:201,race,AmericanIndianOrAlaskaNative,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:202,race,Asian,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:203,race,Filipino,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:204,race,HispanicOrLatino,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:205,race,NativeHawaiianOrOtherPacificIslanderAlone,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:206,race,White,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:207,race,TwoOrMoreRaces,socialVulnerabilityStatus,EconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:220,race,BlackOrAfricanAmericanAlone,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:221,race,AmericanIndianOrAlaskaNative,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:222,race,Asian,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:223,race,Filipino,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:224,race,HispanicOrLatino,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:225,race,NativeHawaiianOrOtherPacificIslanderAlone,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:226,race,White,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:227,race,TwoOrMoreRaces,socialVulnerabilityStatus,NotEconomicallyDisadvantaged,,,,,,,,,,,, +Student Group ID:240,socialVulnerabilityStatus,Foster,,,,,,,,,,,,,, +Student Group ID:241,socialVulnerabilityStatus,NotFoster,,,,,,,,,,,,,, +Student Group ID:243,languageLearned,English,languageLearningAgeGroup,Adult,,,,,,,,,,,, +Total Students Tested with Scores,value,{Number},populationType,Student,statType,measuredValue,measuredProperty,count,,,,,,,, +Mean Scale Score,value,{Number},populationType,Student,statType,meanValue,measuredProperty,assessmentScore,,,,,,,, +Percentage Standard Exceeded,value,{Number},populationType,Student,statType,measuredValue,measuredProperty,count,educationalAchievementLevel,CA_StandardExceeded,measurementDenominator,-educationalAchievementLevel,scalingFactor,100,unit,Percent +Percentage Standard Met,value,{Number},populationType,Student,statType,measuredValue,measuredProperty,count,educationalAchievementLevel,CA_StandardMet,measurementDenominator,-educationalAchievementLevel,scalingFactor,100,unit,Percent +Percentage Standard Met and Above,value,{Number},populationType,Student,statType,measuredValue,measuredProperty,count,educationalAchievementLevel,CA_StandardMetAndAbove,measurementDenominator,-educationalAchievementLevel,scalingFactor,100,unit,Percent +Percentage Standard Nearly Met,value,{Number},populationType,Student,statType,measuredValue,measuredProperty,count,educationalAchievementLevel,CA_StandardNearlyMet,measurementDenominator,-educationalAchievementLevel,scalingFactor,100,unit,Percent +Percentage Standard Not Met,value,{Number},populationType,Student,statType,measuredValue,measuredProperty,count,educationalAchievementLevel,CA_StandardNotMet,measurementDenominator,-educationalAchievementLevel,scalingFactor,100,unit,Percent diff --git a/statvar_imports/us_education/california_school_performance/counters/california_school_performance_counters.csv b/statvar_imports/us_education/california_school_performance/counters/california_school_performance_counters.csv new file mode 100644 index 0000000000..845b3bf0d5 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/counters/california_school_performance_counters.csv @@ -0,0 +1,86 @@ +"key","value" +"1:process_input_generated-svobs",64016 +"1:process_input_generated-svobs-sb_ca_all_years_normalized.txt",64016 +"1:process_input_input-data-rows",9402 +"1:process_input_input-files-processed",1 +"1:process_input_input-header-rows",1 +"1:process_input_input-rows-processed",9429 +"1:process_input_input-sections",1 +"1:process_input_num-rows-input_files/sb_ca_all_years_normalized.txt",12201 +"1:process_input_process-mem",4146962432 +"1:process_input_process-mem-rss",1300381696 +"1:process_input_process-time-sys-secs",6.64 +"1:process_input_process-time-user-secs",100.32 +"1:process_input_process_elapsed_time",54.71908008300014 +"1:process_input_process_remaining_time",86.89243674650133 +"1:process_input_processed",9429 +"1:process_input_processing-input-rows-rate",172.31772023188162 +"1:process_input_processing-input-rows-rate_input_files/sb_ca_all_years_normalized.txt",172.339548077672 +"1:process_input_processing-time-seconds",54.71803651300024 +"1:process_input_processing-time-seconds_input_files/sb_ca_all_years_normalized.txt",54.71803651300024 +"1:process_input_processing_rate",172.31649336388162 +"1:process_input_start_time",2028.288000167 +"1:process_input_total",24402 +"1:process_input_warning-unresolved-value-ref",6030 +"2:prepare_output_process-mem",4149059584 +"2:prepare_output_process-mem-rss",1302745088 +"2:prepare_output_process-time-sys-secs",6.74 +"2:prepare_output_process-time-user-secs",103.85 +"2:prepare_output_process_elapsed_time",22.978151400999902 +"2:prepare_output_processed",0 +"2:prepare_output_start_time",2083.007294716 +"3:write_statvar_mcf_process-mem",4151156736 +"3:write_statvar_mcf_process-mem-rss",1305145344 +"3:write_statvar_mcf_process-time-sys-secs",6.76 +"3:write_statvar_mcf_process-time-user-secs",104.3 +"3:write_statvar_mcf_process_elapsed_time",3.2742244069998176 +"3:write_statvar_mcf_processed",0 +"3:write_statvar_mcf_start_time",2105.985730112 +"4:write_svobs_csv_process-mem",4151156736 +"4:write_svobs_csv_process-mem-rss",1305145344 +"4:write_svobs_csv_process-time-sys-secs",6.78 +"4:write_svobs_csv_process-time-user-secs",105.71 +"4:write_svobs_csv_process_elapsed_time",1.4304956909998305 +"4:write_svobs_csv_processed",0 +"4:write_svobs_csv_start_time",2109.260218805 +"PVs-matched",826 +"dropped-output-statvars-mcf",0 +"error-node-merge-conflict",98 +"existing-nodes-from-api",98 +"existing-statvars-from-mcf",97531 +"generated-statvars",110511 +"generated-unique-statvars",110 +"input-nodes-with-additions",98 +"nodes-with-diff",98 +"output-new-schema-mcf",0 +"output-statvars-mcf",110 +"output-svobs-csv-rows",64016 +"output-svobs-unique-observationAbout",59 +"output-svobs-unique-observationDate",10 +"output-svobs-unique-scalingFactor",1 +"output-svobs-unique-typeOf",1 +"output-svobs-unique-unit",1 +"output-svobs-unique-value",13532 +"output-svobs-unique-variableMeasured",110 +"process-mem",4100825088 +"process-mem-rss",1254895616 +"process-time-sys-secs",6.53 +"process-time-user-secs",45.73 +"process_elapsed_time",0.012378501000082542 +"processed",110 +"pvs-added",98 +"pvs-added-definition",98 +"pvs-modified",98 +"pvs-modified-name",98 +"recon-lookup-schema",13762 +"recon-unmodified-nodes",64126 +"spell-allowlist-file",1 +"spell-allowlist-words",24 +"spell-check-ignored-pvs",110 +"spell-check-nodes",110 +"spell-check-pvs",815 +"start_time",2028.287452481 +"svobs-added",64016 +"total",110 +"urls-allowlist",0 +"warning-missing-property-key",1966010 diff --git a/statvar_imports/us_education/california_school_performance/download.py b/statvar_imports/us_education/california_school_performance/download.py new file mode 100755 index 0000000000..d7800232ff --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/download.py @@ -0,0 +1,484 @@ +#!/usr/bin/env python3 +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Download, normalization, and extraction script for California School Performance (CAASPP) data. + +This script fetches CAASPP research data files directly from the official +California Department of Education / Educational Testing Service (ETS) research portal +(https://caaspp-elpac.ets.org/caaspp/ResearchFileListSB) for all available years +(2015–2019, 2021–present; 2020 was cancelled statewide due to COVID-19). + +It automatically handles differences in delimiter (comma vs caret) and column naming across +eras, filters State and County aggregate records, preserves raw source archives in raw_files/, +and streams normalization into a unified format ready for processing by Data Commons stat_var_processor.py. + +Usage: + python3 download.py --years=all + python3 download.py --years=2015-2024 + python3 download.py --years=2024 + python3 download.py --test_mode +""" + +import csv +from datetime import datetime +import io +import os +import shutil +import sys +import tempfile +import time +import urllib.error +import urllib.request +import zipfile +from absl import app +from absl import flags +from absl import logging + +FLAGS = flags.FLAGS + +flags.DEFINE_string( + 'years', + 'all', + 'Years to download: "all" (2015-present, skipping 2020), range e.g. "2015-2024", or comma-separated "2023,2024".', +) +flags.DEFINE_enum( + 'data_mode', + 'all_students', + ['all_students', 'all_groups'], + 'Data scope: "all_students" downloads group 1 only (~5MB/yr); "all_groups" downloads all 55 demographic subgroups (~100MB/yr).', +) +flags.DEFINE_string( + 'output_dir', + '', + 'Directory to extract files into. Defaults to input_files/ under the script directory.', +) +flags.DEFINE_string( + 'raw_dir', + '', + 'Directory to store downloaded raw zip files. Defaults to raw_files/ under the script directory.', +) +flags.DEFINE_boolean( + 'test_mode', + False, + 'If true, downloads a small test sample for pipeline verification.', +) +flags.DEFINE_boolean( + 'clean', + False, + 'If true, deletes existing files in output_dir before downloading.', +) +flags.DEFINE_boolean( + 'keep_all_entities', + False, + 'If true, keeps all entities (including schools and districts). Defaults to False (State & County only).', +) + +BASE_URL = 'https://caaspp-elpac.ets.org/caaspp/researchfiles/' +USER_AGENT = 'Mozilla/5.0 (DataCommons Ingestion Bot)' + +# Known historical official zip filename per year: (all_subgroups_zip, all_students_zip) +KNOWN_YEAR_URL_MAP = { + 2015: ('sb_ca2015_all_csv_v3.zip', 'sb_ca2015_1_csv_v3.zip'), + 2016: ('sb_ca2016_all_csv_v3.zip', 'sb_ca2016_1_csv_v3.zip'), + 2017: ('sb_ca2017_all_csv_v2.zip', 'sb_ca2017_1_csv_v2.zip'), + 2018: ('sb_ca2018_all_csv_v3.zip', 'sb_ca2018_1_csv_v3.zip'), + 2019: ('sb_ca2019_all_csv_v4.zip', 'sb_ca2019_1_csv_v4.zip'), + 2021: ('sb_ca2021_all_csv_v2.zip', 'sb_ca2021_1_csv_v2.zip'), + 2022: ('sb_ca2022_all_csv_v1.zip', 'sb_ca2022_1_csv_v1.zip'), + 2023: ('sb_ca2023_all_csv_v1.zip', 'sb_ca2023_1_csv_v1.zip'), + 2024: ('sb_ca2024_all_csv_v1.zip', 'sb_ca2024_1_csv_v1.zip'), + 2025: ('sb_ca2025_all_csv_v1.zip', 'sb_ca2025_1_csv_v1.zip'), +} + +UNIFIED_HEADER = [ + 'County Code', + 'Test Year', + 'Student Group ID', + 'Grade', + 'Test ID', + 'Total Students Tested with Scores', + 'Mean Scale Score', + 'Percentage Standard Exceeded', + 'Percentage Standard Met', + 'Percentage Standard Met and Above', + 'Percentage Standard Nearly Met', + 'Percentage Standard Not Met', +] + + +def probe_url_exists(url: str, timeout: int = 10, max_retries: int = 3) -> bool: + """Checks if a remote URL exists using an HTTP HEAD probe with retries for transient errors.""" + for attempt in range(max_retries): + try: + req = urllib.request.Request(url, + headers={'User-Agent': USER_AGENT}, + method='HEAD') + with urllib.request.urlopen(req, timeout=timeout) as resp: + return resp.status == 200 + except urllib.error.HTTPError as e: + if e.code == 404: + return False + logging.warning('HTTP error %d probing %s (attempt %d/%d)', e.code, + url, attempt + 1, max_retries) + if attempt < max_retries - 1 and e.code in (429, 500, 502, 503, + 504): + time.sleep(2**attempt) + continue + return False + except urllib.error.URLError as e: + logging.warning('Network error probing %s (attempt %d/%d): %s', url, + attempt + 1, max_retries, e) + if attempt < max_retries - 1: + time.sleep(2**attempt) + continue + return False + except Exception as e: + logging.warning('Unexpected error probing %s: %s', url, e) + return False + return False + + +def download_url_with_retries(url: str, + target_file: str, + timeout: int = 180, + max_retries: int = 3) -> None: + """Downloads a remote URL to target_file atomically with retry and exponential backoff.""" + target_dir = os.path.dirname(os.path.abspath(target_file)) + os.makedirs(target_dir, exist_ok=True) + + for attempt in range(max_retries): + temp_file = tempfile.NamedTemporaryFile(dir=target_dir, + suffix='.zip.tmp', + delete=False) + try: + req = urllib.request.Request(url, + headers={'User-Agent': USER_AGENT}) + with urllib.request.urlopen(req, timeout=timeout) as resp: + shutil.copyfileobj(resp, temp_file) + temp_file.flush() + os.fsync(temp_file.fileno()) + temp_file.close() + os.replace(temp_file.name, target_file) + return + except Exception as e: + temp_file.close() + if os.path.exists(temp_file.name): + os.unlink(temp_file.name) + logging.warning('Download attempt %d/%d failed for %s: %s', + attempt + 1, max_retries, url, e) + if attempt < max_retries - 1: + time.sleep(2**attempt) + else: + raise + + +def discover_year_urls(year: int) -> tuple[str, str] | None: + """Discovers or resolves zip filenames for a given year.""" + if year in KNOWN_YEAR_URL_MAP: + return KNOWN_YEAR_URL_MAP[year] + + # For subsequent calendar years, probe standard version naming (v1 through v4) + for v in range(1, 5): + all_groups = f'sb_ca{year}_all_csv_v{v}.zip' + all_students = f'sb_ca{year}_1_csv_v{v}.zip' + if probe_url_exists(BASE_URL + all_students): + logging.info('Discovered CAASPP data files for year %d: %s, %s', + year, all_groups, all_students) + return (all_groups, all_students) + return None + + +def get_all_available_years() -> list[int]: + """Returns all known historical years plus any dynamically discovered newer years.""" + available = set(KNOWN_YEAR_URL_MAP.keys()) + current_year = datetime.now().year + # Probe any years beyond known historical years up to current year + 1 + for y in range(max(available) + 1, current_year + 2): + if discover_year_urls(y): + available.add(y) + return sorted(available) + + +def parse_years(years_str: str) -> list[int]: + """Parses year specification into a sorted list of ints.""" + all_years = get_all_available_years() + if years_str.strip().lower() in ('all', '*'): + return all_years + + years = set() + for part in years_str.split(','): + part = part.strip() + if '-' in part: + try: + start, end = part.split('-', 1) + for y in range(int(start.strip()), int(end.strip()) + 1): + if y in all_years or discover_year_urls(y): + years.add(y) + except ValueError: + logging.warning('Invalid year range format: %s', part) + elif part.isdigit(): + y = int(part) + if y in all_years or discover_year_urls(y): + years.add(y) + else: + logging.warning('Year %d not available from source.', y) + return sorted(years) + + +def normalize_and_filter_stream(text_stream, keep_all_entities: bool = False): + """Yields normalized CAASPP records row-by-row from a text stream.""" + first_line = text_stream.readline() + if not first_line: + return + + delim = '^' if '^' in first_line else ',' + text_stream.seek(0) + reader = csv.DictReader(text_stream, delimiter=delim) + + for r in reader: + # Strip quotes and whitespace from all keys/values + cleaned = { + k.strip().strip('"'): (v.strip().strip('"') if v else '') + for k, v in r.items() + if k + } + + # Check entity level: + # In CAASPP research files, Type ID 4 = State, 5 = County. + # State and County aggregates have District Code 00000 and School Code 0000000. + if not keep_all_entities: + type_id = cleaned.get('Type ID', '').strip() + if type_id: + if type_id not in ('4', '5'): + continue + else: + d_code = cleaned.get('District Code', '').strip() + s_code = cleaned.get('School Code', '').strip() + if not (d_code and s_code and set(d_code) == {'0'} and + set(s_code) == {'0'}): + continue + + # Handle subgroup ID variations across eras + gid = cleaned.get('Student Group ID') or cleaned.get( + 'Subgroup ID') or '' + # Handle test ID variations + tid = cleaned.get('Test ID') or cleaned.get('Test Id') or '' + # Handle count of tested with scores variations + scores = cleaned.get('Total Students Tested with Scores' + ) or cleaned.get('Students with Scores') or '' + + row = [ + cleaned.get('County Code', ''), + cleaned.get('Test Year', ''), + gid, + cleaned.get('Grade', ''), + tid, + scores, + cleaned.get('Mean Scale Score', ''), + cleaned.get('Percentage Standard Exceeded', ''), + cleaned.get('Percentage Standard Met', ''), + cleaned.get('Percentage Standard Met and Above', ''), + cleaned.get('Percentage Standard Nearly Met', ''), + cleaned.get('Percentage Standard Not Met', ''), + ] + yield row + + +def normalize_and_filter_records(raw_bytes: bytes, + keep_all_entities: bool = False + ) -> list[list[str]]: + """Normalizes raw CAASPP records across all eras into unified columns.""" + text_io = io.StringIO(raw_bytes.decode('latin1')) + return list( + normalize_and_filter_stream(text_io, + keep_all_entities=keep_all_entities)) + + +def download_and_process_year( + year: int, + data_mode: str, + output_dir: str, + keep_all_entities: bool, + raw_dir: str = '', + out_filename: str = '', +) -> str: + """Downloads and preserves raw zip, streams normalized records, and saves output atomically.""" + urls = discover_year_urls(year) + if not urls: + raise RuntimeError(f'Year {year} is not available from CAASPP portal.') + + all_groups_file, all_students_file = urls + filename = all_groups_file if data_mode == 'all_groups' else all_students_file + url = BASE_URL + filename + + if not raw_dir: + raw_dir = os.path.join(os.path.dirname(os.path.abspath(output_dir)), + 'raw_files') + os.makedirs(raw_dir, exist_ok=True) + raw_zip_path = os.path.join(raw_dir, filename) + + if not os.path.exists(raw_zip_path) or os.path.getsize(raw_zip_path) == 0: + logging.info('Fetching Year %d raw zip: %s', year, url) + download_url_with_retries(url, raw_zip_path) + else: + logging.info('Year %d: Using preserved raw archive: %s', year, + raw_zip_path) + + out_name = out_filename or f'sb_ca{year}_normalized.txt' + out_file = os.path.join(output_dir, out_name) + temp_out = tempfile.NamedTemporaryFile('w', + dir=output_dir, + delete=False, + encoding='utf-8', + newline='', + suffix='.tmp') + row_count = 0 + + try: + with zipfile.ZipFile(raw_zip_path, 'r') as z: + candidates = [ + f for f in z.namelist() + if f.startswith(f'sb_ca{year}') and 'entities' not in f + ] + if not candidates: + raise RuntimeError( + f'No matching data file found in zip for year {year}') + + writer = csv.writer(temp_out, delimiter='^') + writer.writerow(UNIFIED_HEADER) + with z.open(candidates[0]) as member_file: + text_stream = io.TextIOWrapper(member_file, encoding='latin1') + for row in normalize_and_filter_stream( + text_stream, keep_all_entities=keep_all_entities): + writer.writerow(row) + row_count += 1 + + temp_out.flush() + os.fsync(temp_out.fileno()) + temp_out.close() + os.replace(temp_out.name, out_file) + except Exception: + temp_out.close() + if os.path.exists(temp_out.name): + os.unlink(temp_out.name) + raise + + logging.info('Year %d: Extracted %d normalized rows -> %s', year, row_count, + out_file) + return out_file + + +def main(argv): + del argv # Unused. + + script_dir = os.path.dirname(os.path.abspath(__file__)) + output_dir = FLAGS.output_dir or os.path.join(script_dir, 'input_files') + raw_dir = FLAGS.raw_dir or os.path.join(script_dir, 'raw_files') + os.makedirs(output_dir, exist_ok=True) + os.makedirs(raw_dir, exist_ok=True) + + if FLAGS.clean: + logging.info('Cleaning output directory: %s', output_dir) + for item in os.listdir(output_dir): + p = os.path.join(output_dir, item) + if os.path.isfile(p) or os.path.islink(p): + os.unlink(p) + elif os.path.isdir(p): + shutil.rmtree(p) + + if FLAGS.test_mode: + logging.info('Test mode: downloading 2024 sample...') + test_file = download_and_process_year( + 2024, + 'all_students', + output_dir, + FLAGS.keep_all_entities, + raw_dir=raw_dir, + out_filename='sb_ca_test_normalized.txt', + ) + logging.info('Test mode completed. Created test file: %s', test_file) + logging.info('Master all-years file was not modified in test mode.') + return + + years = parse_years(FLAGS.years) + logging.info('Target years to download and process: %s', years) + + all_normalized_files = [] + failed_years = [] + for year in years: + try: + res = download_and_process_year( + year, + FLAGS.data_mode, + output_dir, + FLAGS.keep_all_entities, + raw_dir=raw_dir, + ) + if res: + all_normalized_files.append(res) + except Exception as e: + logging.error('Error processing year %d: %s', year, e) + failed_years.append((year, str(e))) + + if failed_years: + raise RuntimeError( + f'Import aborted due to download failures for years: {failed_years}' + ) + + if not all_normalized_files: + raise RuntimeError( + 'No files were successfully downloaded and processed.') + + # Only combine into master multi-year file if all years were requested. + # Single-year or subset downloads must NOT overwrite the consolidated master dataset. + if FLAGS.years.strip().lower() in ('all', '*'): + master_file = os.path.join(output_dir, 'sb_ca_all_years_normalized.txt') + temp_master = tempfile.NamedTemporaryFile('w', + dir=output_dir, + delete=False, + encoding='utf-8', + newline='', + suffix='.tmp') + total_master_rows = 0 + try: + writer = csv.writer(temp_master, delimiter='^') + writer.writerow(UNIFIED_HEADER) + for nf in all_normalized_files: + with open(nf, 'r', encoding='utf-8') as f_in: + reader = csv.reader(f_in, delimiter='^') + next(reader) # Skip header + for row in reader: + writer.writerow(row) + total_master_rows += 1 + temp_master.flush() + os.fsync(temp_master.fileno()) + temp_master.close() + os.replace(temp_master.name, master_file) + except Exception: + temp_master.close() + if os.path.exists(temp_master.name): + os.unlink(temp_master.name) + raise + + logging.info( + 'Created master multi-year file: %s (%d records across %d years)', + master_file, total_master_rows, len(all_normalized_files)) + else: + logging.info( + 'Individual year processing completed for: %s. Master multi-year file was not modified.', + [os.path.basename(f) for f in all_normalized_files]) + + +if __name__ == '__main__': + app.run(main) diff --git a/statvar_imports/us_education/california_school_performance/golden_data/golden_observations.csv b/statvar_imports/us_education/california_school_performance/golden_data/golden_observations.csv new file mode 100644 index 0000000000..af2d7bd54d --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/golden_data/golden_observations.csv @@ -0,0 +1,55 @@ +"observationAbout" +"geoId/06" +"geoId/06001" +"geoId/06005" +"geoId/06007" +"geoId/06009" +"geoId/06011" +"geoId/06013" +"geoId/06015" +"geoId/06017" +"geoId/06019" +"geoId/06021" +"geoId/06023" +"geoId/06025" +"geoId/06027" +"geoId/06029" +"geoId/06031" +"geoId/06033" +"geoId/06035" +"geoId/06037" +"geoId/06039" +"geoId/06041" +"geoId/06043" +"geoId/06045" +"geoId/06047" +"geoId/06053" +"geoId/06055" +"geoId/06057" +"geoId/06059" +"geoId/06061" +"geoId/06063" +"geoId/06065" +"geoId/06067" +"geoId/06069" +"geoId/06071" +"geoId/06073" +"geoId/06075" +"geoId/06077" +"geoId/06079" +"geoId/06081" +"geoId/06083" +"geoId/06085" +"geoId/06087" +"geoId/06089" +"geoId/06093" +"geoId/06095" +"geoId/06097" +"geoId/06099" +"geoId/06101" +"geoId/06103" +"geoId/06107" +"geoId/06109" +"geoId/06111" +"geoId/06113" +"geoId/06115" diff --git a/statvar_imports/us_education/california_school_performance/golden_data/golden_summary_report.csv b/statvar_imports/us_education/california_school_performance/golden_data/golden_summary_report.csv new file mode 100644 index 0000000000..da62925466 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/golden_data/golden_summary_report.csv @@ -0,0 +1,111 @@ +"Units","ScalingFactors","MeasurementMethods","MinDate","NumPlaces","StatVar","observationPeriods" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade6_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMet_In_Count_Student_SchoolGrade11_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade7_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade7_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade7_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMet_In_Count_Student_SchoolGrade8_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade7_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade5_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade6_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts","[]" +"[]","[]","[]","2015","58","Mean_AssessmentScore_Student_SchoolGrade7_Mathematics","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade5_Mathematics","[]" +"[]","[]","[]","2015","58","Mean_AssessmentScore_Student_SchoolGrade8_Mathematics","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade4_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade6_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade6_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade8_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade7_Mathematics","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade8_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade8_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade4_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade8_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade3_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade7_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade5_Mathematics","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade11_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade4_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade6_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardMetAndAbove_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMet_In_Count_Student_SchoolGrade7_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade5_Mathematics","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade3_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade3_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade4_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade3_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade4_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade11_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardNotMet_Mathematics_AsAFractionOf_Count_Student_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardMet_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade6_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade5_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade3_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade4_EnglishLanguageArts","[]" +"[]","[]","[]","2015","58","Mean_AssessmentScore_Student_SchoolGrade7_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade11_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade3_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_Mathematics","[]" +"[]","[]","[]","2015","58","Mean_AssessmentScore_Student_SchoolGrade11_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardExceeded_Mathematics_AsAFractionOf_Count_Student_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade8_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade11_Mathematics","[]" +"[]","[]","[]","2015","58","Mean_AssessmentScore_Student_SchoolGrade8_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade11_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade7_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade5_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardMetAndAbove_Mathematics_AsAFractionOf_Count_Student_Mathematics","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade7_Mathematics","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade6_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade11_Mathematics","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade4_Mathematics","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade5_Mathematics","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade4_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade3_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade8_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts","[]" +"[]","[]","[]","2015","58","Mean_AssessmentScore_Student_SchoolGrade11_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade6_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardExceeded_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade11_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardNearlyMet_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardNearlyMet_Mathematics_AsAFractionOf_Count_Student_Mathematics","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade6_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade11_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade3_EnglishLanguageArts","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade8_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade5_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade4_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade5_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMet_In_Count_Student_SchoolGrade6_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMet_In_Count_Student_SchoolGrade8_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade3_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade11_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMet_In_Count_Student_SchoolGrade7_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade4_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardMet_Mathematics_AsAFractionOf_Count_Student_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade7_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Count_Student_CAStandardNotMet_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade6_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade8_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade6_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade5_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade7_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade8_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade3_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade4_Mathematics","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade8_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade11_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade6_Mathematics","[]" +"[]","[]","[]","2015","59","Count_Student_SchoolGrade6_EnglishLanguageArts","[]" +"[Percent]","[100]","[]","2015","58","Percent_CA_StandardMet_In_Count_Student_SchoolGrade11_Mathematics","[]" +"[Percent]","[100]","[]","2015","59","Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade5_Mathematics","[]" +"[]","[]","[]","2015","59","Mean_AssessmentScore_Student_SchoolGrade3_EnglishLanguageArts","[]" diff --git a/statvar_imports/us_education/california_school_performance/manifest.json b/statvar_imports/us_education/california_school_performance/manifest.json new file mode 100644 index 0000000000..ba0b4c6f89 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/manifest.json @@ -0,0 +1,32 @@ +{ + "import_specifications": [ + { + "import_name": "CaliforniaSchoolPerformance", + "curator_emails": [ + "support@datacommons.org" + ], + "provenance_url": "https://caaspp-elpac.ets.org/caaspp/ResearchFileListSB", + "provenance_description": "Official public assessment data from the California Department of Education (CDE) and ETS, covering California Assessment of Student Performance and Progress (CAASPP) Smarter Balanced Summative Assessments for English Language Arts/Literacy (ELA) and Mathematics across California public schools, districts, counties, and statewide aggregates.", + "scripts": [ + "download.py", + "../../../tools/statvar_importer/stat_var_processor.py --input_data=input_files/sb_ca_all_years_normalized.txt --pv_map=config/california_school_performance_pvmap.csv --config_file=config/california_school_performance_metadata.csv --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --output_path=output_files/california_school_performance_all_years_output --output_counters=counters/california_school_performance_counters.csv" + ], + "import_inputs": [ + { + "template_mcf": "output_files/california_school_performance_all_years_output.tmcf", + "cleaned_csv": "output_files/california_school_performance_all_years_output.csv", + "node_mcf": "output_files/*.mcf" + } + ], + "source_files": [ + "raw_files/*.zip", + "input_files/*.txt", + "golden_data/*.csv", + "counters/*.csv", + "config/*.csv" + ], + "cron_schedule": "0 0 1 10 *", + "validation_config_file": "validation_config.json" + } + ] +} diff --git a/statvar_imports/us_education/california_school_performance/test_data/sample_input.txt b/statvar_imports/us_education/california_school_performance/test_data/sample_input.txt new file mode 100644 index 0000000000..4d3acd85d4 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/test_data/sample_input.txt @@ -0,0 +1,20 @@ +County Code^District Code^District Name^School Code^School Name^Type ID^Filler^Test Year^Test Type^Test ID^Student Group ID^Grade^Total Students Enrolled^Total Students Tested^Total Students Tested with Scores^Mean Scale Score^Percentage Standard Exceeded^Count Standard Exceeded^Percentage Standard Met^Count Standard Met^Percentage Standard Met and Above^Count Standard Met and Above^Percentage Standard Nearly Met^Count Standard Nearly Met^Percentage Standard Not Met^Count Standard Not Met^Overall Total^Area 1 Percentage Above Standard^Area 1 Count Above Standard^Area 1 Percentage Near Standard^Area 1 Count Near Standard^Area 1 Percentage Below Standard^Area 1 Count Below Standard^Area 1 Total^Area 2 Percentage Above Standard^Area 2 Count Above Standard^Area 2 Percentage Near Standard^Area 2 Count Near Standard^Area 2 Percentage Below Standard^Area 2 Count Below Standard^Area 2 Total^Area 3 Percentage Above Standard^Area 3 Count Above Standard^Area 3 Percentage Near Standard^Area 3 Count Near Standard^Area 3 Percentage Below Standard^Area 3 Count Below Standard^Area 3 Total^Area 4 Percentage Above Standard^Area 4 Count Above Standard^Area 4 Percentage Near Standard^Area 4 Count Near Standard^Area 4 Percentage Below Standard^Area 4 Count Below Standard^Area 4 Total^Composite Area 1 Percentage Above Standard^Composite Area 1 Count Above Standard^Composite Area 1 Percentage Near Standard^Composite Area 1 Count Near Standard^Composite Area 1 Percentage Below Standard^Composite Area 1 Count Below Standard^Composite Area 1 Total^Composite Area 2 Percentage Above Standard^Composite Area 2 Count Above Standard^Composite Area 2 Percentage Near Standard^Composite Area 2 Count Near Standard^Composite Area 2 Percentage Below Standard^Composite Area 2 Count Below Standard^Composite Area 2 Total +00^00000^^0000000^^4^^2024^B^1^1^3^415082^403798^403589^2409.9^23.23^93740^19.57^78987^42.80^172727^22.61^91261^34.59^139601^403589^16.85^68020^58.77^237195^24.37^98364^403579^16.40^66155^49.91^201272^33.69^135867^403294^11.39^45964^71.01^286541^17.60^71029^403534^16.32^65882^60.54^244313^23.14^93389^403584^19.76^79759^50.82^205115^29.41^118705^403579^21.25^85753^42.23^170398^36.53^147396^403547 +00^00000^^0000000^^4^^2024^B^1^1^4^424534^413723^413493^2452.0^24.21^100087^19.69^81427^43.90^181514^19.08^78903^37.02^153076^413493^18.62^76974^59.29^245136^22.10^91358^413468^14.98^61900^51.74^213785^33.28^137510^413195^11.80^48791^70.26^290479^17.94^74157^413427^15.52^64177^64.13^265149^20.35^84144^413470^21.87^90428^50.19^207526^27.94^115514^413468^20.44^84509^43.99^181886^35.57^147048^413443 +00^00000^^0000000^^4^^2024^B^1^1^5^429109^418641^418427^2490.7^22.20^92894^25.17^105328^47.37^198222^19.41^81202^33.22^139003^418427^17.89^74864^58.96^246708^23.14^96828^418400^18.83^78751^52.17^218194^28.99^121253^418198^14.00^58555^69.00^288695^17.00^71124^418374^18.63^77943^59.68^249700^21.69^90765^418408^22.17^92756^50.27^210324^27.56^115320^418400^25.28^105771^42.41^177446^32.31^135169^418386 +00^00000^^0000000^^4^^2024^B^1^1^6^431881^420698^420406^2515.3^18.10^76103^27.72^116547^45.82^192650^24.30^102145^29.88^125611^420406^18.33^77021^51.31^215636^30.36^127585^420242^16.87^70888^50.55^212363^32.58^136853^420104^13.18^55380^70.60^296727^16.22^68159^420266^18.22^76576^62.88^264291^18.91^79468^420335^21.98^92390^46.10^193720^31.92^134135^420245^23.68^99528^42.27^177673^34.05^143102^420303 +00^00000^^0000000^^4^^2024^B^1^1^7^438526^424815^424319^2536.2^17.30^73402^29.90^126854^47.19^200256^22.01^93408^30.79^130655^424319^17.82^75577^57.44^243556^24.74^104889^424022^22.33^94583^47.17^199804^30.51^129225^423612^11.89^50449^71.95^305162^16.15^68511^424122^19.67^83457^59.27^251420^21.06^89342^424219^20.49^86888^50.26^213096^29.25^124039^424023^27.90^118293^38.79^164467^33.31^141262^424022 +00^00000^^0000000^^4^^2024^B^1^1^8^441160^425307^424707^2548.7^15.96^67763^30.00^127402^45.95^195165^24.31^103254^29.74^126288^424707^17.84^75718^52.76^223932^29.41^124823^424473^18.17^77089^49.85^211474^31.97^135634^424197^14.16^60113^69.63^295612^16.21^68799^424524^20.72^87968^62.47^265216^16.81^71383^424567^21.23^90125^46.92^199146^31.85^135202^424473^25.97^110236^40.83^173363^33.20^140953^424552 +00^00000^^0000000^^4^^2024^B^1^1^11^463265^436275^435296^2590.5^26.23^114183^29.50^128398^55.73^242581^20.89^90940^23.38^101775^435296^25.54^111114^53.64^233327^20.82^90548^434989^29.03^126117^45.96^199637^25.01^108632^434386^15.33^66705^70.86^308274^13.81^60074^435053^23.96^104257^60.54^263402^15.50^67438^435097^28.44^123727^47.39^206163^24.16^105100^434990^35.39^153958^37.51^163156^27.10^117876^434990 +00^00000^^0000000^^4^^2024^B^1^1^13^3043557^2943257^2940237^^21.02^618172^26.02^764943^47.04^1383115^21.80^641113^31.15^916009^2940237^19.03^559288^55.98^1645490^24.99^734395^2939173^19.59^575483^49.59^1456529^30.81^904974^2936986^13.13^385957^70.48^2071490^16.39^481853^2939300^19.06^560260^61.35^1803491^19.59^575929^2939680^22.32^656073^48.83^1435090^28.85^848015^2939178^25.79^758048^41.11^1208389^33.10^972806^2939243 +00^00000^^0000000^^4^^2024^B^2^1^3^415082^408760^408582^2422.9^20.75^84801^24.88^101660^45.64^186461^21.88^89412^32.48^132709^408582^26.14^106803^43.35^177118^30.51^124634^408555^21.97^89770^46.10^188364^31.93^130448^408582^20.85^85198^56.17^229519^22.97^93864^408581^0.00^0^0.00^0^0.00^0^0^26.14^106803^43.35^177118^30.51^124634^408555^27.20^111118^35.90^146690^36.90^150774^408582 +00^00000^^0000000^^4^^2024^B^2^1^4^424534^418359^418155^2461.5^18.98^79358^22.22^92929^41.20^172287^28.22^117993^30.58^127875^418155^23.42^97908^42.54^177863^34.05^142366^418137^18.16^75936^47.92^200381^33.92^141835^418152^19.72^82465^53.41^223353^26.86^112330^418148^0.00^0^0.00^0^0.00^0^0^23.42^97908^42.54^177863^34.05^142366^418137^24.09^100719^35.41^148059^40.51^169372^418150 +00^00000^^0000000^^4^^2024^B^2^1^5^429109^422728^422525^2483.3^19.76^83484^15.26^64474^35.02^147958^24.61^103967^40.38^170600^422525^19.86^83894^41.11^173713^39.03^164900^422507^15.67^66215^48.69^205741^35.64^150566^422522^14.10^59557^56.30^237891^29.60^125076^422524^0.00^0^0.00^0^0.00^0^0^19.86^83894^41.11^173713^39.03^164900^422507^19.02^80374^36.25^153163^44.73^188987^422524 +00^00000^^0000000^^4^^2024^B^2^1^6^431881^424045^423789^2499.8^18.48^78301^15.84^67119^34.31^145420^25.06^106190^40.63^172179^423789^17.94^76007^39.20^166084^42.86^181620^423711^14.44^61207^47.46^201149^38.09^161433^423789^15.48^65588^56.17^238039^28.35^120161^423788^0.00^0^0.00^0^0.00^0^0^17.94^76007^39.20^166084^42.86^181620^423711^18.93^80243^36.13^153097^44.94^190448^423788 +00^00000^^0000000^^4^^2024^B^2^1^7^438526^427282^426918^2513.0^17.08^72901^16.75^71526^33.83^144427^24.63^105153^41.54^177338^426918^18.76^80065^39.47^168439^41.77^178293^426797^15.03^64155^49.52^211403^35.45^151359^426917^14.08^60123^59.29^253120^26.63^113675^426918^0.00^0^0.00^0^0.00^0^0^18.76^80065^39.47^168439^41.77^178293^426797^18.28^78041^37.90^161821^43.82^187056^426918 +00^00000^^0000000^^4^^2024^B^2^1^8^441160^427346^426863^2524.3^18.15^77493^13.41^57253^31.57^134746^21.88^93405^46.55^198712^426863^17.79^75926^40.66^173521^41.55^177300^426747^15.44^65925^50.53^215690^34.03^145239^426854^14.58^62250^57.93^247297^27.48^117313^426860^0.00^0^0.00^0^0.00^0^0^17.79^75926^40.66^173521^41.55^177300^426747^18.55^79182^36.85^157311^44.60^190367^426860 +00^00000^^0000000^^4^^2024^B^2^1^11^463265^435333^434711^2547.8^12.29^53441^15.61^67856^27.90^121297^21.15^91936^50.95^221478^434711^15.13^65773^36.56^158919^48.31^209968^434660^13.23^57517^57.63^250528^29.14^126652^434697^12.12^52689^59.41^258263^28.47^123738^434690^0.00^0^0.00^0^0.00^0^0^15.13^65773^36.56^158919^48.31^209968^434660^15.91^69154^37.06^161105^47.03^204440^434699 +00^00000^^0000000^^4^^2024^B^2^1^13^3043557^2963853^2961543^^17.89^529779^17.65^522817^35.54^1052596^23.91^708056^40.55^1200891^2961543^19.80^586376^40.38^1195657^39.82^1179081^2961114^16.23^480725^49.75^1473256^34.02^1007532^2961513^15.80^467870^56.98^1687482^27.22^806157^2961509^0.00^0^0.00^0^0.00^0^0^19.80^586376^40.38^1195657^39.82^1179081^2961114^20.22^598831^36.51^1081246^43.27^1281444^2961521 +01^00000^^0000000^^5^^2024^B^1^1^3^15433^14909^14898^2428.5^31.68^4719^19.77^2946^51.45^7665^18.69^2785^29.86^4448^14898^23.47^3496^55.82^8316^20.71^3086^14898^22.13^3293^48.44^7209^29.43^4380^14882^14.96^2229^68.74^10239^16.29^2427^14895^22.18^3305^57.38^8548^20.44^3045^14898^27.01^4024^47.44^7067^25.55^3807^14898^28.50^4245^39.74^5920^31.76^4731^14896 +01^00000^^0000000^^5^^2024^B^1^1^4^15692^15211^15200^2474.4^33.86^5146^19.42^2952^53.28^8098^15.91^2419^30.81^4683^15200^26.16^3976^55.12^8377^18.72^2846^15199^21.99^3337^49.98^7585^28.03^4253^15175^15.41^2342^68.40^10395^16.19^2460^15197^20.73^3151^61.71^9379^17.55^2668^15198^30.22^4593^46.08^7004^23.70^3602^15199^28.80^4377^41.34^6283^29.85^4537^15197 +01^00000^^0000000^^5^^2024^B^1^1^5^15926^15426^15417^2516.7^33.10^5103^24.51^3779^57.61^8882^14.91^2299^27.48^4236^15417^25.73^3967^55.28^8522^18.99^2927^15416^28.13^4333^47.95^7387^23.92^3685^15405^18.09^2788^67.38^10387^14.53^2240^15415^26.13^4029^55.21^8512^18.65^2876^15417^31.43^4845^45.45^7006^23.13^3565^15416^36.44^5617^36.83^5678^26.73^4120^15415 diff --git a/statvar_imports/us_education/california_school_performance/test_data/sample_state_output.csv b/statvar_imports/us_education/california_school_performance/test_data/sample_state_output.csv new file mode 100644 index 0000000000..f4ff19e3f4 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/test_data/sample_state_output.csv @@ -0,0 +1,132 @@ +observationDate,observationAbout,variableMeasured,value,unit,scalingFactor +2024,geoId/06,dcid:Count_Student_SchoolGrade3_EnglishLanguageArts,403589,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade3_EnglishLanguageArts,2409.9,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade3_EnglishLanguageArts,23.23,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts,19.57,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade3_EnglishLanguageArts,42.8,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts,22.61,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts,34.59,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade4_EnglishLanguageArts,413493,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade4_EnglishLanguageArts,2452,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade4_EnglishLanguageArts,24.21,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts,19.69,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade4_EnglishLanguageArts,43.9,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts,19.08,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts,37.02,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade5_EnglishLanguageArts,418427,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade5_EnglishLanguageArts,2490.7,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade5_EnglishLanguageArts,22.2,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts,25.17,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade5_EnglishLanguageArts,47.37,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts,19.41,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts,33.22,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade6_EnglishLanguageArts,420406,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade6_EnglishLanguageArts,2515.3,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade6_EnglishLanguageArts,18.1,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade6_EnglishLanguageArts,27.72,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade6_EnglishLanguageArts,45.82,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade6_EnglishLanguageArts,24.3,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade6_EnglishLanguageArts,29.88,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade7_EnglishLanguageArts,424319,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade7_EnglishLanguageArts,2536.2,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade7_EnglishLanguageArts,17.3,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade7_EnglishLanguageArts,29.9,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade7_EnglishLanguageArts,47.19,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade7_EnglishLanguageArts,22.01,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade7_EnglishLanguageArts,30.79,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade8_EnglishLanguageArts,424707,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade8_EnglishLanguageArts,2548.7,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade8_EnglishLanguageArts,15.96,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade8_EnglishLanguageArts,30,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade8_EnglishLanguageArts,45.95,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade8_EnglishLanguageArts,24.31,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade8_EnglishLanguageArts,29.74,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade11_EnglishLanguageArts,435296,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade11_EnglishLanguageArts,2590.5,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade11_EnglishLanguageArts,26.23,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade11_EnglishLanguageArts,29.5,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade11_EnglishLanguageArts,55.73,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade11_EnglishLanguageArts,20.89,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade11_EnglishLanguageArts,23.38,Percent,100 +2024,geoId/06,dcid:Count_Student_EnglishLanguageArts,2940237,, +2024,geoId/06,dcid:Count_Student_CAStandardExceeded_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts,21.02,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardMet_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts,26.02,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardMetAndAbove_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts,47.04,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardNearlyMet_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts,21.8,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardNotMet_EnglishLanguageArts_AsAFractionOf_Count_Student_EnglishLanguageArts,31.15,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade3_Mathematics,408582,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade3_Mathematics,2422.9,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade3_Mathematics,20.75,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade3_Mathematics,24.88,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade3_Mathematics,45.64,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade3_Mathematics,21.88,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade3_Mathematics,32.48,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade4_Mathematics,418155,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade4_Mathematics,2461.5,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade4_Mathematics,18.98,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade4_Mathematics,22.22,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade4_Mathematics,41.2,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade4_Mathematics,28.22,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade4_Mathematics,30.58,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade5_Mathematics,422525,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade5_Mathematics,2483.3,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade5_Mathematics,19.76,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade5_Mathematics,15.26,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade5_Mathematics,35.02,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade5_Mathematics,24.61,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade5_Mathematics,40.38,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade6_Mathematics,423789,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade6_Mathematics,2499.8,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade6_Mathematics,18.48,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade6_Mathematics,15.84,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade6_Mathematics,34.31,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade6_Mathematics,25.06,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade6_Mathematics,40.63,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade7_Mathematics,426918,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade7_Mathematics,2513,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade7_Mathematics,17.08,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade7_Mathematics,16.75,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade7_Mathematics,33.83,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade7_Mathematics,24.63,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade7_Mathematics,41.54,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade8_Mathematics,426863,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade8_Mathematics,2524.3,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade8_Mathematics,18.15,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade8_Mathematics,13.41,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade8_Mathematics,31.57,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade8_Mathematics,21.88,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade8_Mathematics,46.55,Percent,100 +2024,geoId/06,dcid:Count_Student_SchoolGrade11_Mathematics,434711,, +2024,geoId/06,dcid:Mean_AssessmentScore_Student_SchoolGrade11_Mathematics,2547.8,, +2024,geoId/06,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade11_Mathematics,12.29,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade11_Mathematics,15.61,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade11_Mathematics,27.9,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade11_Mathematics,21.15,Percent,100 +2024,geoId/06,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade11_Mathematics,50.95,Percent,100 +2024,geoId/06,dcid:Count_Student_Mathematics,2961543,, +2024,geoId/06,dcid:Count_Student_CAStandardExceeded_Mathematics_AsAFractionOf_Count_Student_Mathematics,17.89,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardMet_Mathematics_AsAFractionOf_Count_Student_Mathematics,17.65,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardMetAndAbove_Mathematics_AsAFractionOf_Count_Student_Mathematics,35.54,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardNearlyMet_Mathematics_AsAFractionOf_Count_Student_Mathematics,23.91,Percent,100 +2024,geoId/06,dcid:Count_Student_CAStandardNotMet_Mathematics_AsAFractionOf_Count_Student_Mathematics,40.55,Percent,100 +2024,geoId/06001,dcid:Count_Student_SchoolGrade3_EnglishLanguageArts,14898,, +2024,geoId/06001,dcid:Mean_AssessmentScore_Student_SchoolGrade3_EnglishLanguageArts,2428.5,, +2024,geoId/06001,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade3_EnglishLanguageArts,31.68,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts,19.77,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade3_EnglishLanguageArts,51.45,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts,18.69,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade3_EnglishLanguageArts,29.86,Percent,100 +2024,geoId/06001,dcid:Count_Student_SchoolGrade4_EnglishLanguageArts,15200,, +2024,geoId/06001,dcid:Mean_AssessmentScore_Student_SchoolGrade4_EnglishLanguageArts,2474.4,, +2024,geoId/06001,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade4_EnglishLanguageArts,33.86,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts,19.42,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade4_EnglishLanguageArts,53.28,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts,15.91,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade4_EnglishLanguageArts,30.81,Percent,100 +2024,geoId/06001,dcid:Count_Student_SchoolGrade5_EnglishLanguageArts,15417,, +2024,geoId/06001,dcid:Mean_AssessmentScore_Student_SchoolGrade5_EnglishLanguageArts,2516.7,, +2024,geoId/06001,dcid:Percent_CA_StandardExceeded_In_Count_Student_SchoolGrade5_EnglishLanguageArts,33.1,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts,24.51,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardMetAndAbove_In_Count_Student_SchoolGrade5_EnglishLanguageArts,57.61,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardNearlyMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts,14.91,Percent,100 +2024,geoId/06001,dcid:Percent_CA_StandardNotMet_In_Count_Student_SchoolGrade5_EnglishLanguageArts,27.48,Percent,100 diff --git a/statvar_imports/us_education/california_school_performance/test_data/sample_state_output.tmcf b/statvar_imports/us_education/california_school_performance/test_data/sample_state_output.tmcf new file mode 100644 index 0000000000..d252ec48e1 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/test_data/sample_state_output.tmcf @@ -0,0 +1,8 @@ +Node: E:sample_state_output->E0 +observationDate: C:sample_state_output->observationDate +observationAbout: C:sample_state_output->observationAbout +variableMeasured: C:sample_state_output->variableMeasured +value: C:sample_state_output->value +unit: C:sample_state_output->unit +scalingFactor: C:sample_state_output->scalingFactor +typeOf: dcs:StatVarObservation diff --git a/statvar_imports/us_education/california_school_performance/validation_config.json b/statvar_imports/us_education/california_school_performance/validation_config.json new file mode 100644 index 0000000000..e86ca29782 --- /dev/null +++ b/statvar_imports/us_education/california_school_performance/validation_config.json @@ -0,0 +1,31 @@ +{ + "schema_version": "1.0", + "rules": [ + { + "rule_id": "check_deleted_records_percent", + "description": "Strictly enforce historical deletion average threshold of 0.1%", + "validator": "DELETED_RECORDS_PERCENT", + "params": { + "threshold": 0.1 + } + }, + { + "rule_id": "check_goldens_summary_report", + "description": "Validates summary_report.csv against the golden summary data", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": "../../../../golden_data/golden_summary_report.csv", + "input_files": "../../input0/genmcf/summary_report.csv" + } + }, + { + "rule_id": "check_goldens_output_csv", + "description": "Verifies the generated output CSV data matches established critical golden records", + "validator": "GOLDENS_CHECK", + "params": { + "golden_files": "../../../../golden_data/golden_observations.csv", + "input_files": "../../../../output_files/california_school_performance_all_years_output.csv" + } + } + ] +}