diff --git a/statvar_imports/zurich/README.md b/statvar_imports/zurich/README.md index 58df598a4b..3efca5bd6f 100644 --- a/statvar_imports/zurich/README.md +++ b/statvar_imports/zurich/README.md @@ -1,21 +1,32 @@ 1. import_name": "Zurich_Population_Number_Of_Company_Workplace_Employees" 2. Import Overview -Zurich population data at Province and City Level. -Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/wir_2552_wiki.csv) +Number of companies, workplaces and employees in Zurich city at City, District (Kreise), and Quarter (Quartiere) Level. +Source URL: [BFS WIR STATENT Data](https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552) Import Type: Fully Autorefresh -Source Data Availability: 2011 to 2022 +Source Data Availability: 2011 to 2024 Release Frequency: P1Y -3. Preprocessing Steps (No) +3. Preprocessing Steps (Yes) +Generate rollups from the downloaded dataset: +python3 wir_2552_wiki/generate_rollups.py + +Rollup Logic & Explanation (`wir_2552_wiki/generate_rollups.py`): +- **Why Filter on Value `'0'` (`RechtsformSort == 0` & `BetriebsgrSort == 0`)**: + - In the upstream dataset (`WIR255OD2552.csv`), category code **`0` represents the official aggregate total row** across all sub-categories: `RechtsformSort == 0` is *"Alle Rechtsformen"* (all legal forms combined) and `BetriebsgrSort == 0` is *"Alle Betriebsgrössen"* (all company sizes combined). Non-zero values (`1`, `2`, etc.) represent granular sub-category breakdowns (e.g., public-law vs. private-law entities, micro-businesses `<10` vs. small businesses `10–49`). + - **Matches Target StatVar Schema**: The Data Commons StatVars emitted by this import (`Count_Company`, `Count_Person_Employed`, `Count_Person_Employed_Female`, `Count_Person_Employed_Male`, `Count_Person_FullTimeEmployee`, etc.) represent overall totals per place and year without legal form or company size constraints. + - **Prevents Conflicting Duplicate Observations**: Retaining non-zero breakdown rows would cause `stat_var_processor.py` to map multiple sub-category rows to the exact same `(place, year, StatVar)` tuple, causing observation collisions. + - **Avoids Undercounting from Suppressed Cells (`'K'`)**: Individual sub-category breakdown rows frequently contain `'K'` (confidentiality suppression for small counts). Summing non-zero breakdown rows manually would undercount true totals, whereas the official `(0, 0)` row provides the complete, authoritative total published by Statistics Zurich. +- **Unknown Region Exclusion**: Filters out unknown/unassigned region rows (`RaumSort` `990` ["Kreis Unbekannt"] and `999` ["Quartier Unbekannt"]) that do not map to valid geographic places. +- **Numeric Coercion**: Coerces metric columns (`Arbeitsstaetten`, `AnzBesch`, `AnzBeschW`, `AnzBeschM`, `AnzVZA`, `AnzVZAW`, `AnzVZAM`) to numeric values, converting non-numeric markers (e.g., `'K'` for confidential/suppressed entries) to `NaN` so they are cleanly skipped during StatVar observation generation. 4. Autorefresh Type -Fully Autorefresh:"0 2 29 * * " (Runs at 2:00 AM on the 29th day of every month). +Fully Autorefresh:"0 2 1,15 * * " (Runs at 2:00 AM on the 1st and 15th day of every month). 5. Script Execution Details -" python3 stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data='../../statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv' --pv_map='../../statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv' --config_file='../../statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv' --output_path=../../statvar_imports/zurich/wir_2552_wiki/output/zurich_population_wir_2552_wiki_output " +" python3 ../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552/download/WIR255OD2552.csv --output_folder=wir_2552_wiki/input_files && python3 wir_2552_wiki/generate_rollups.py && python3 ../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=wir_2552_wiki/input_files/WIR255OD2552_rollups.csv --pv_map=wir_2552_wiki/wir_2552_wiki_pvmap.csv --config_file=wir_2552_wiki/wir_2552_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=wir_2552_wiki/output/zurich_population_wir_2552_wiki --output_counters=wir_2552_wiki/counters/zurich_population_wir_2552_wiki_counters.csv " ##### @@ -23,7 +34,7 @@ Fully Autorefresh:"0 2 29 * * " (Runs at 2:00 AM on the 29th day of every month) 1. import_name": "Zurich_Population_By_Age" 2. Import Overview -Zurich population data at Province and City Level. +Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level. Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_age10_wiki.csv) Import Type: Fully Autorefresh Source Data Availability: 1993 to 2023 @@ -45,21 +56,21 @@ Fully Autorefresh:" 0 6 29 * * " (Runs at 6:00 AM on the 29th of every month). 1. import_name": "Zurich_Population" 2. Import Overview -Zurich population data at Province and City Level. -Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv) +Total population of Zurich city by quarter and year at City, District (Kreise), and Quarter (Quartiere) Level. +Source URL: [BEV324OD3240 Dataset](https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240) Import Type: Fully Autorefresh -Source Data Availability: 1941 to 2023 +Source Data Availability: 1941 to 2025 Release Frequency: P1Y 3. Preprocessing Steps (No) 4. Autorefresh Type -Fully Autorefresh:" 30 11 29 * * " (Runs at 11:30 AM on the 29th of every month). +Fully Autorefresh:" 30 11 1,15 * * " (Runs at 11:30 AM on the 1st and 15th of every month). 5. Script Execution Details -" python3 stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data='../../statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv' --pv_map='../../statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv' --config_file='../../statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv' --output_path=../../statvar_imports/zurich/bev_3240_wiki/output/zurich_population_bev_3240_wiki.csv_output " +" python3 ../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240/download/BEV324OD3240.csv --output_folder=bev_3240_wiki/input_files && python3 ../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=bev_3240_wiki/input_files/BEV324OD3240.csv --pv_map=bev_3240_wiki/bev_3240_wiki_pvmap.csv --config_file=bev_3240_wiki/bev_3240_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=bev_3240_wiki/output/zurich_population_bev_3240_wiki --output_counters=bev_3240_wiki/counters/zurich_population_bev_3240_wiki_counters.csv " ##### @@ -67,7 +78,7 @@ Fully Autorefresh:" 30 11 29 * * " (Runs at 11:30 AM on the 29th of every month) 1. import_name": "Zurich_Population_Number_Of_Birth_By_Origin" 2. Import Overview -Zurich population data at Province and City Level. +Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level. Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_hel_wiki.csv) Import Type: Fully Autorefresh Source Data Availability: 1998 to 2023 @@ -89,7 +100,7 @@ Fully Autorefresh:" 45 15 29 * * " (Runs at 3:45 PM on the 29th of every month). 1. import_name": "Zurich_Population_Number_Of_Birth" 2. Import Overview -Zurich population data at Province and City Level. +Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level. Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_wiki.csv) Import Type: Fully Autorefresh Source Data Availability: 1998 to 2023 @@ -111,7 +122,7 @@ Fully Autorefresh:" 0 20 29 * * " (Runs at 8:00 PM on the 29th of every month). 1. import_name": "Zurich_Population_By_Origin" 2. Import Overview -Zurich population data at Province and City Level. +Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level. Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_hel_wiki.csv) Import Type: Fully Autorefresh Source Data Availability: 1993 to 2023 @@ -134,7 +145,7 @@ Fully Autorefresh:" 15 1 29 * * " (Runs at 1:15 AM on the 29th of every month). 1. import_name": "Zurich_Population_Number_Of_Birth_By_Sex" 2. Import Overview -Zurich population data at Province and City Level. +Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level. Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_sex_wiki.csv) Import Type: Fully Autorefresh Source Data Availability: 1993 to 2023 @@ -156,7 +167,7 @@ Fully Autorefresh:" 07 19 29 * * " (Runs at 7:07 PM on the 29th of every month). 1. import_name": "Zurich_Population_By_Sex" 2. Import Overview -Zurich population data at Province and City Level. +Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level. Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_sex_wiki.csv) Import Type: Fully Autorefresh Source Data Availability: 1993 to 2023 diff --git a/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv b/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv index 79237e739e..44e9c85954 100644 --- a/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv +++ b/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv @@ -1,14 +1,13 @@ parameter,value -url,https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv +url,https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240 description, -place_type,Country +place_type,City #places_within, start_date, end_date, release_frequency, process, comments, -output_columns,"observationAbout,observationDate,value,variableMeasured,unit" +output_columns,"observationAbout,observationDate,value,variableMeasured" header_rows,1 -#input_rows,10 -#Import_Name,Zurich_Population +reconcile_nodes,False diff --git a/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv b/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv index 72007e6c3a..725dd67586 100644 --- a/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv +++ b/statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv @@ -1,4 +1,37 @@ -key,prop,val,p1,v1 -StichtagDatJahr,observationDate,{Number},, -AnzBestWir,value,{Number},populationType,Person -wikiId,observationAbout,wikidataId/{Data},, \ No newline at end of file +key,prop,val,p1,v1 +StichtagDatJahr,observationDate,{Number},, +AnzBestWir,value,{Number},populationType,Person +QuarLang:Rathaus,observationAbout,wikidataId/Q692511,, +QuarLang:Hochschulen,observationAbout,wikidataId/Q39240,, +QuarLang:Lindenhof,observationAbout,wikidataId/Q10987378,, +QuarLang:City,observationAbout,wikidataId/Q1093831,, +QuarLang:Wollishofen,observationAbout,wikidataId/Q642353,, +QuarLang:Leimbach,observationAbout,wikidataId/Q691367,, +QuarLang:Enge,observationAbout,wikidataId/Q648218,, +QuarLang:Alt-Wiedikon,observationAbout,wikidataId/Q433012,, +QuarLang:Friesenberg,observationAbout,wikidataId/Q693357,, +QuarLang:Sihlfeld,observationAbout,wikidataId/Q370104,, +QuarLang:Werd,observationAbout,wikidataId/Q531899,, +QuarLang:Langstrasse,observationAbout,wikidataId/Q1805410,, +QuarLang:Hard,observationAbout,wikidataId/Q870084,, +QuarLang:Gewerbeschule,observationAbout,wikidataId/Q693413,, +QuarLang:Escher Wyss,observationAbout,wikidataId/Q687052,, +QuarLang:Unterstrass,observationAbout,wikidataId/Q656446,, +QuarLang:Oberstrass,observationAbout,wikidataId/Q693483,, +QuarLang:Fluntern,observationAbout,wikidataId/Q693269,, +QuarLang:Hottingen,observationAbout,wikidataId/Q693454,, +QuarLang:Hirslanden,observationAbout,wikidataId/Q476940,, +QuarLang:Witikon,observationAbout,wikidataId/Q392079,, +QuarLang:Seefeld,observationAbout,wikidataId/Q692773,, +QuarLang:Mühlebach,observationAbout,wikidataId/Q693397,, +QuarLang:Weinegg,observationAbout,wikidataId/Q693321,, +QuarLang:Albisrieden,observationAbout,wikidataId/Q80797,, +QuarLang:Altstetten,observationAbout,wikidataId/Q445711,, +QuarLang:Höngg,observationAbout,wikidataId/Q455496,, +QuarLang:Wipkingen,observationAbout,wikidataId/Q678030,, +QuarLang:Affoltern,observationAbout,wikidataId/Q382903,, +QuarLang:Oerlikon,observationAbout,wikidataId/Q167179,, +QuarLang:Seebach,observationAbout,wikidataId/Q276792,, +QuarLang:Saatlen,observationAbout,wikidataId/Q652455,, +QuarLang:Schwamendingen-Mitte,observationAbout,wikidataId/Q692728,, +QuarLang:Hirzenbach,observationAbout,wikidataId/Q693374,, diff --git a/statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv b/statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv index 48bb92542b..65cd6f77de 100644 --- a/statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv +++ b/statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv @@ -1,25 +1,6 @@ -StichtagDatJahr,AnzBestWir,wikiId -2023,3343,Q692511 -2022,3330,Q692511 -2021,3275,Q692511 -2020,3317,Q692511 -2019,3307,Q692511 -2018,3267,Q692511 -2017,3266,Q692511 -2016,3226,Q692511 -2015,3197,Q692511 -2014,3236,Q692511 -2013,3194,Q692511 -2012,3164,Q692511 -2011,3140,Q692511 -2010,3099,Q692511 -2009,3110,Q692511 -2008,3051,Q692511 -2007,3079,Q692511 -2006,3097,Q692511 -2005,3081,Q692511 -2004,3115,Q692511 -2003,3149,Q692511 -2002,3128,Q692511 -2001,3103,Q692511 -2000,3109,Q692511 +StichtagDatJahr,QuarSort,QuarLang,AnzBestWir +2025,11,Rathaus,3252 +2024,11,Rathaus,3260 +2023,11,Rathaus,3343 +2022,11,Rathaus,3330 +2021,11,Rathaus,3275 diff --git a/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.csv b/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.csv index af72c49c35..5708a6ee43 100644 --- a/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.csv +++ b/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.csv @@ -1,25 +1,6 @@ -observationAbout,observationDate,value,variableMeasured,unit -wikidataId/Q692511,2023,3343,dcid:Count_Person, -wikidataId/Q692511,2022,3330,dcid:Count_Person, -wikidataId/Q692511,2021,3275,dcid:Count_Person, -wikidataId/Q692511,2020,3317,dcid:Count_Person, -wikidataId/Q692511,2019,3307,dcid:Count_Person, -wikidataId/Q692511,2018,3267,dcid:Count_Person, -wikidataId/Q692511,2017,3266,dcid:Count_Person, -wikidataId/Q692511,2016,3226,dcid:Count_Person, -wikidataId/Q692511,2015,3197,dcid:Count_Person, -wikidataId/Q692511,2014,3236,dcid:Count_Person, -wikidataId/Q692511,2013,3194,dcid:Count_Person, -wikidataId/Q692511,2012,3164,dcid:Count_Person, -wikidataId/Q692511,2011,3140,dcid:Count_Person, -wikidataId/Q692511,2010,3099,dcid:Count_Person, -wikidataId/Q692511,2009,3110,dcid:Count_Person, -wikidataId/Q692511,2008,3051,dcid:Count_Person, -wikidataId/Q692511,2007,3079,dcid:Count_Person, -wikidataId/Q692511,2006,3097,dcid:Count_Person, -wikidataId/Q692511,2005,3081,dcid:Count_Person, -wikidataId/Q692511,2004,3115,dcid:Count_Person, -wikidataId/Q692511,2003,3149,dcid:Count_Person, -wikidataId/Q692511,2002,3128,dcid:Count_Person, -wikidataId/Q692511,2001,3103,dcid:Count_Person, -wikidataId/Q692511,2000,3109,dcid:Count_Person, +observationAbout,observationDate,value,variableMeasured +wikidataId/Q692511,2025,3252,dcid:Count_Person +wikidataId/Q692511,2024,3260,dcid:Count_Person +wikidataId/Q692511,2023,3343,dcid:Count_Person +wikidataId/Q692511,2022,3330,dcid:Count_Person +wikidataId/Q692511,2021,3275,dcid:Count_Person diff --git a/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.tmcf b/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.tmcf index 2cf36b0678..9b82f0f8e5 100644 --- a/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.tmcf +++ b/statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.tmcf @@ -1,7 +1,6 @@ -Node: E:bev_3240_wiki_output->E0 -observationAbout: C:bev_3240_wiki_output->observationAbout -observationDate: C:bev_3240_wiki_output->observationDate -value: C:bev_3240_wiki_output->value -variableMeasured: C:bev_3240_wiki_output->variableMeasured -unit: C:bev_3240_wiki_output->unit +Node: E:zurich_population_bev_3240_wiki_output->E0 +observationAbout: C:zurich_population_bev_3240_wiki_output->observationAbout +observationDate: C:zurich_population_bev_3240_wiki_output->observationDate +value: C:zurich_population_bev_3240_wiki_output->value +variableMeasured: C:zurich_population_bev_3240_wiki_output->variableMeasured typeOf: dcs:StatVarObservation diff --git a/statvar_imports/zurich/bev_3240_wiki/validation_config.json b/statvar_imports/zurich/bev_3240_wiki/validation_config.json new file mode 100644 index 0000000000..0effdbf0af --- /dev/null +++ b/statvar_imports/zurich/bev_3240_wiki/validation_config.json @@ -0,0 +1,28 @@ +{ + "schema_version": "1.0", + "rules": [ + { + "rule_id": "check_deleted_records_percent", + "description": "Checks that deleted records do not exceed 0.1%.", + "validator": "DELETED_RECORDS_PERCENT", + "params": { + "threshold": 0.1 + } + }, + { + "rule_id": "check_max_date_consistent", + "description": "Checks if the MaxDate is the same for all StatVars.", + "validator": "MAX_DATE_CONSISTENT", + "params": {} + }, + { + "rule_id": "check_max_date_freshness", + "description": "Verifies that MaxDate meets the expected freshness threshold for the dataset.", + "validator": "SQL_VALIDATOR", + "params": { + "query": "SELECT MAX(MaxDate) >= '2025' AS is_fresh, COUNT(*) AS row_cnt FROM stats", + "condition": "is_fresh = TRUE AND row_cnt > 0" + } + } + ] +} diff --git a/statvar_imports/zurich/manifest.json b/statvar_imports/zurich/manifest.json index 9ec063ffb1..cdcf50ab35 100644 --- a/statvar_imports/zurich/manifest.json +++ b/statvar_imports/zurich/manifest.json @@ -5,22 +5,32 @@ "curator_emails": [ "support@datacommons.org" ], - "provenance_url": "https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/wir_2552_wiki.csv", - "provenance_description": "Number of companys, workplaces and employees of Zurich city contains structure and organization of the city administration", + "provenance_url": "https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552", + "provenance_description": "Number of companies, workplaces and employees of Zurich city contains structure and organization of the city administration", "scripts": [ - "../../util/download_util_script.py --download_url=https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/wir_2552_wiki.csv --output_folder=wir_2552_wiki/input_files", - "../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=wir_2552_wiki/input_files/wir_2552_wiki.csv --pv_map=wir_2552_wiki/wir_2552_wiki_pvmap.csv --config_file=wir_2552_wiki/wir_2552_wiki_metadata.csv --output_path=wir_2552_wiki/output/zurich_population_wir_2552_wiki" + "../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552/download/WIR255OD2552.csv --output_folder=wir_2552_wiki/input_files", + "wir_2552_wiki/generate_rollups.py", + "../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=wir_2552_wiki/input_files/WIR255OD2552_rollups.csv --pv_map=wir_2552_wiki/wir_2552_wiki_pvmap.csv --config_file=wir_2552_wiki/wir_2552_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=wir_2552_wiki/output/zurich_population_wir_2552_wiki --output_counters=wir_2552_wiki/counters/zurich_population_wir_2552_wiki_counters.csv" ], "source_files": [ - "wir_2552_wiki/input_files/wir_2552_wiki.csv" + "wir_2552_wiki/input_files/WIR255OD2552.csv", + "wir_2552_wiki/input_files/WIR255OD2552_rollups.csv", + "wir_2552_wiki/counters/*.csv" ], "import_inputs": [ { "template_mcf": "wir_2552_wiki/output/zurich_population_wir_2552_wiki.tmcf", - "cleaned_csv": "wir_2552_wiki/output/zurich_population_wir_2552_wiki.csv" + "cleaned_csv": "wir_2552_wiki/output/zurich_population_wir_2552_wiki.csv", + "node_mcf": "wir_2552_wiki/output/*.mcf" } ], - "cron_schedule": "0 2 29 * *" + "resource_limits": { + "cpu": 1, + "memory": 2, + "disk": 30 + }, + "cron_schedule": "0 2 1,15 * *", + "validation_config_file": "wir_2552_wiki/validation_config.json" }, { "import_name": "Zurich_Population_By_Age", @@ -49,22 +59,30 @@ "curator_emails": [ "support@datacommons.org" ], - "provenance_url": "https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv", + "provenance_url": "https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240", "provenance_description": "Population of Zurich city contains structure and organization of the city administration", "scripts": [ - "../../util/download_util_script.py --download_url=https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv --output_folder=bev_3240_wiki/input_files", - "../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=bev_3240_wiki/input_files/bev_3240_wiki.csv --pv_map=bev_3240_wiki/bev_3240_wiki_pvmap.csv --config_file=bev_3240_wiki/bev_3240_wiki_metadata.csv --output_path=bev_3240_wiki/output/zurich_population_bev_3240_wiki" + "../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240/download/BEV324OD3240.csv --output_folder=bev_3240_wiki/input_files", + "../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=bev_3240_wiki/input_files/BEV324OD3240.csv --pv_map=bev_3240_wiki/bev_3240_wiki_pvmap.csv --config_file=bev_3240_wiki/bev_3240_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=bev_3240_wiki/output/zurich_population_bev_3240_wiki --output_counters=bev_3240_wiki/counters/zurich_population_bev_3240_wiki_counters.csv" ], "source_files": [ - "bev_3240_wiki/input_files/bev_3240_wiki.csv" + "bev_3240_wiki/input_files/BEV324OD3240.csv", + "bev_3240_wiki/counters/*.csv" ], "import_inputs": [ { "template_mcf": "bev_3240_wiki/output/zurich_population_bev_3240_wiki.tmcf", - "cleaned_csv": "bev_3240_wiki/output/zurich_population_bev_3240_wiki.csv" + "cleaned_csv": "bev_3240_wiki/output/zurich_population_bev_3240_wiki.csv", + "node_mcf": "bev_3240_wiki/output/*.mcf" } ], - "cron_schedule": "30 11 29 * *" + "resource_limits": { + "cpu": 1, + "memory": 2, + "disk": 30 + }, + "cron_schedule": "30 11 1,15 * *", + "validation_config_file": "bev_3240_wiki/validation_config.json" }, { "import_name": "Zurich_Population_Number_Of_Birth_By_Origin", @@ -178,4 +196,4 @@ "cron_schedule": "55 10 29 * *" } ] -} \ No newline at end of file +} diff --git a/statvar_imports/zurich/wir_2552_wiki/generate_rollups.py b/statvar_imports/zurich/wir_2552_wiki/generate_rollups.py new file mode 100644 index 0000000000..163f0f0c2f --- /dev/null +++ b/statvar_imports/zurich/wir_2552_wiki/generate_rollups.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Preprocesses Zurich WIR255OD2552 dataset to filter total roll-up rows. + +Filters total rows (Alle Rechtsformen and Alle Betriebsgrössen) where +RechtsformSort == 0 and BetriebsgrSort == 0, and coerces numeric metric +columns to numbers (converting markers such as 'K' to NaN). +""" + +import os +from absl import app +from absl import flags +from absl import logging +import pandas as pd + +CURRENT_DIR = os.path.dirname(os.path.abspath(__file__)) +DEFAULT_INPUT_RAW = os.path.join(CURRENT_DIR, 'input_files', 'WIR255OD2552.csv') +DEFAULT_OUTPUT_ROLLUPS = os.path.join(CURRENT_DIR, 'input_files', + 'WIR255OD2552_rollups.csv') + +FLAGS = flags.FLAGS +flags.DEFINE_string('input_csv', DEFAULT_INPUT_RAW, + 'Path to raw input CSV file.') +flags.DEFINE_string('output_csv', DEFAULT_OUTPUT_ROLLUPS, + 'Path to output rollups CSV file.') + +REQUIRED_FILTER_COLS = ['RechtsformSort', 'BetriebsgrSort'] +NUMERIC_VAL_COLS = [ + 'Arbeitsstaetten', 'AnzBesch', 'AnzBeschW', 'AnzBeschM', 'AnzVZA', + 'AnzVZAW', 'AnzVZAM' +] + + +def process_rollups(df: pd.DataFrame) -> pd.DataFrame: + """Filters total roll-up rows and coerces metric columns to numeric. + + Args: + df: Input pandas DataFrame containing raw Zurich workplace data. + + Returns: + Filtered pandas DataFrame containing only total rollup rows. + + Raises: + ValueError: If input DataFrame is empty or no rows match filter. + KeyError: If required filter columns are missing. + """ + if df.empty: + raise ValueError("Input DataFrame is empty.") + + missing_cols = [ + col for col in REQUIRED_FILTER_COLS if col not in df.columns + ] + if missing_cols: + raise KeyError( + f"Input DataFrame is missing required filter columns: {missing_cols}. " + f"Available columns: {list(df.columns)}") + + # Keep only the total rows (Alle Rechtsformen and Alle Betriebsgrössen). + # Coerce sort columns to numeric so string or numeric '0' matches reliably. + rechtsform_sort = pd.to_numeric(df['RechtsformSort'], errors='coerce') + betriebsgr_sort = pd.to_numeric(df['BetriebsgrSort'], errors='coerce') + df_rollups = df[(rechtsform_sort == 0) & (betriebsgr_sort == 0)].copy() + + # Filter out unknown regions (RaumSort 990: Kreis Unbekannt, 999: Quartier Unbekannt) + if 'RaumSort' in df_rollups.columns: + df_rollups = df_rollups[~pd.to_numeric( + df_rollups['RaumSort'], errors='coerce').isin([990, 999])].copy() + + if df_rollups.empty: + raise ValueError( + "No rows matched filter (RechtsformSort == 0 & BetriebsgrSort == 0)." + ) + + # Convert numeric columns, converting non-numeric markers (e.g. 'K') to NaN/numeric + for col in NUMERIC_VAL_COLS: + if col in df_rollups.columns: + df_rollups[col] = pd.to_numeric(df_rollups[col], errors='coerce') + + return df_rollups + + +def generate_rollups(input_csv: str, output_csv: str) -> pd.DataFrame: + """Reads raw CSV, processes rollups, and writes the output CSV file. + + Args: + input_csv: Path to input CSV file. + output_csv: Path to output CSV file. + + Returns: + Processed pandas DataFrame written to output_csv. + + Raises: + FileNotFoundError: If input_csv does not exist. + ValueError: If input_csv is empty. + """ + if not os.path.exists(input_csv): + raise FileNotFoundError(f"Input file not found: {input_csv}") + if os.path.getsize(input_csv) == 0: + raise ValueError(f"Input file is empty: {input_csv}") + + try: + df = pd.read_csv(input_csv, encoding='utf-8') + except UnicodeDecodeError: + df = pd.read_csv(input_csv, encoding='iso-8859-1') + + df_rollups = process_rollups(df) + + output_dir = os.path.dirname(output_csv) + if output_dir: + os.makedirs(output_dir, exist_ok=True) + + df_rollups.to_csv(output_csv, index=False, encoding='utf-8') + logging.info("Successfully generated %s with %d rows.", output_csv, + len(df_rollups)) + return df_rollups + + +def main(argv): + del argv # Unused. + try: + generate_rollups(FLAGS.input_csv, FLAGS.output_csv) + except Exception as e: + logging.fatal("Failed to generate rollups: %s", e, exc_info=True) + + +if __name__ == '__main__': + app.run(main) diff --git a/statvar_imports/zurich/wir_2552_wiki/generate_rollups_test.py b/statvar_imports/zurich/wir_2552_wiki/generate_rollups_test.py new file mode 100644 index 0000000000..d6407a0359 --- /dev/null +++ b/statvar_imports/zurich/wir_2552_wiki/generate_rollups_test.py @@ -0,0 +1,197 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Hermetic unit tests for Zurich wir_2552_wiki generate_rollups module.""" + +import os +import sys +import tempfile +import unittest +from unittest.mock import patch +import numpy as np +import pandas as pd + +# Add module directory to sys.path +_MODULE_DIR = os.path.dirname(os.path.abspath(__file__)) +if _MODULE_DIR not in sys.path: + sys.path.insert(0, _MODULE_DIR) + +from generate_rollups import FLAGS, generate_rollups, main, process_rollups + + +class GenerateRollupsTest(unittest.TestCase): + """Unit test suite for generate_rollups preprocessor.""" + + def setUp(self): + super().setUp() + FLAGS.mark_as_parsed() + self.sample_raw_data = pd.DataFrame({ + 'Jahr': [2020, 2020, 2020, 2020], + 'RaumSort': [1, 1, 1, 1], + 'RaumLang': [ + 'Ganze Stadt', 'Ganze Stadt', 'Ganze Stadt', 'Ganze Stadt' + ], + 'RechtsformSort': [0, 1, 0, 2], + 'RechtsformLang': [ + 'Alle Rechtsformen', 'Öffentliches Recht', 'Alle Rechtsformen', + 'Privates Recht' + ], + 'BetriebsgrSort': [0, 0, 1, 2], + 'BetriebsgrLang': [ + 'Alle Betriebsgrössen', 'Alle Betriebsgrössen', 'Mikrobetriebe', + 'Grossbetriebe' + ], + 'Arbeitsstaetten': ['39747', '500', '1200', '100'], + 'AnzBesch': ['443621', 'K', '5000', '25000'], + 'AnzBeschW': ['202970', '250', '2500', '12000'], + 'AnzBeschM': ['240651', '250', '2500', '13000'], + 'AnzVZA': ['339750', '400', '4000', '20000'], + 'AnzVZAW': ['136406', '200', '2000', '10000'], + 'AnzVZAM': ['203349', '200', '2000', '10000'] + }) + + def test_process_rollups_filters_total_rows(self): + """Verifies that only total rollup rows (RechtsformSort == 0 & BetriebsgrSort == 0) are retained.""" + result_df = process_rollups(self.sample_raw_data) + self.assertEqual(len(result_df), 1) + self.assertEqual(result_df.iloc[0]['RechtsformSort'], 0) + self.assertEqual(result_df.iloc[0]['BetriebsgrSort'], 0) + self.assertEqual(result_df.iloc[0]['RechtsformLang'], + 'Alle Rechtsformen') + self.assertEqual(result_df.iloc[0]['BetriebsgrLang'], + 'Alle Betriebsgrössen') + + def test_process_rollups_filters_unknown_regions(self): + """Verifies that unknown regions (RaumSort 990 and 999) are excluded.""" + data = pd.DataFrame({ + 'RaumSort': [1, 990, 999, '990', '999', 990.0, 10], + 'RechtsformSort': [0, 0, 0, 0, 0, 0, 0], + 'BetriebsgrSort': [0, 0, 0, 0, 0, 0, 0], + 'Arbeitsstaetten': [ + '100', '200', '300', '400', '500', '550', '600' + ] + }) + result_df = process_rollups(data) + self.assertEqual(len(result_df), 2) + self.assertListEqual(result_df['RaumSort'].tolist(), [1, 10]) + + def test_process_rollups_numeric_coercion(self): + """Verifies that metric columns are converted to numeric and markers like 'K' become NaN.""" + data = pd.DataFrame({ + 'RechtsformSort': [0, 0], + 'BetriebsgrSort': [0, 0], + 'Arbeitsstaetten': ['100', '200'], + 'AnzBesch': ['1500', 'K'], + 'AnzBeschW': [500, 'K'], + 'AnzBeschM': ['1000', 100], + 'AnzVZA': ['1200.5', 'K'], + 'AnzVZAW': ['600.25', '300'], + 'AnzVZAM': ['600.25', 'K'] + }) + result_df = process_rollups(data) + self.assertEqual(len(result_df), 2) + self.assertEqual(result_df.iloc[0]['AnzBesch'], 1500.0) + self.assertTrue(pd.isna(result_df.iloc[1]['AnzBesch'])) + self.assertTrue(pd.isna(result_df.iloc[1]['AnzBeschW'])) + self.assertEqual(result_df.iloc[0]['AnzVZA'], 1200.5) + self.assertTrue(pd.isna(result_df.iloc[1]['AnzVZA'])) + self.assertEqual(result_df.iloc[1]['AnzVZAW'], 300.0) + + def test_process_rollups_empty_dataframe_raises(self): + """Verifies that an empty DataFrame raises ValueError.""" + with self.assertRaises(ValueError): + process_rollups(pd.DataFrame()) + + def test_process_rollups_missing_columns_raises(self): + """Verifies that missing filter columns raise KeyError.""" + df_missing = pd.DataFrame({'RechtsformSort': [0]}) + with self.assertRaises(KeyError): + process_rollups(df_missing) + + def test_process_rollups_no_matching_rows_raises(self): + """Verifies that data with no total rows (0, 0) raises ValueError.""" + df_no_match = pd.DataFrame({ + 'RechtsformSort': [1, 2], + 'BetriebsgrSort': [1, 2] + }) + with self.assertRaises(ValueError): + process_rollups(df_no_match) + + def test_generate_rollups_file_io_success(self): + """Verifies end-to-end file reading, processing, and output generation.""" + with tempfile.TemporaryDirectory() as tmp_dir: + input_csv = os.path.join(tmp_dir, 'input.csv') + output_csv = os.path.join(tmp_dir, 'output_dir', + 'output_rollups.csv') + + self.sample_raw_data.to_csv(input_csv, + index=False, + encoding='utf-8') + result_df = generate_rollups(input_csv, output_csv) + + self.assertTrue(os.path.exists(output_csv)) + self.assertEqual(len(result_df), 1) + + read_back = pd.read_csv(output_csv) + self.assertEqual(len(read_back), 1) + self.assertEqual(read_back.iloc[0]['Arbeitsstaetten'], 39747) + + def test_generate_rollups_file_not_found(self): + """Verifies that non-existent input file raises FileNotFoundError.""" + with self.assertRaises(FileNotFoundError): + generate_rollups('/non/existent/path/file.csv', '/tmp/out.csv') + + def test_generate_rollups_empty_file_raises(self): + """Verifies that an empty input file raises ValueError.""" + with tempfile.NamedTemporaryFile(suffix='.csv') as tmp_file: + with self.assertRaises(ValueError): + generate_rollups(tmp_file.name, '/tmp/out.csv') + + def test_process_rollups_string_sort_columns(self): + """Verifies that sort columns formatted as strings are matched correctly.""" + data = pd.DataFrame({ + 'RechtsformSort': ['0', '1'], + 'BetriebsgrSort': ['0', '0'], + 'Arbeitsstaetten': ['100', '200'] + }) + result_df = process_rollups(data) + self.assertEqual(len(result_df), 1) + self.assertEqual(result_df.iloc[0]['Arbeitsstaetten'], 100.0) + + def test_main_success(self): + """Verifies that main executes successfully with valid flag parameters.""" + with tempfile.TemporaryDirectory() as tmp_dir: + input_csv = os.path.join(tmp_dir, 'input.csv') + output_csv = os.path.join(tmp_dir, 'output.csv') + self.sample_raw_data.to_csv(input_csv, + index=False, + encoding='utf-8') + FLAGS.input_csv = input_csv + FLAGS.output_csv = output_csv + main([]) + self.assertTrue(os.path.exists(output_csv)) + + def test_main_failure_logs_fatal(self): + """Verifies that main logs a fatal error with exc_info on exception.""" + FLAGS.input_csv = '/non/existent/path/file.csv' + FLAGS.output_csv = '/tmp/dummy_output.csv' + with patch('generate_rollups.logging.fatal') as mock_fatal: + main([]) + mock_fatal.assert_called_once() + self.assertIn('Failed to generate rollups', + mock_fatal.call_args[0][0]) + self.assertTrue(mock_fatal.call_args[1].get('exc_info')) + + +if __name__ == '__main__': + unittest.main() diff --git a/statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv b/statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv index 26640c54a0..d22a3aca8f 100644 --- a/statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv +++ b/statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv @@ -1,25 +1,6 @@ -Zeit,companies,employees,employees_f,employees_m,fte,fte_f,fte_m,wikiId -2011,1522,10180,5492,4688,7205,3493,3710,Q692511 -2011,700,13203,7025,6178,9090,4376,4714,Q39240 -2011,1193,15160,7649,7511,12213,5602,6608,Q10987378 -2011,1834,30084,14496,15588,24633,10861,13768,Q1093831 -2011,1050,7578,3396,4182,5865,2271,3594,Q642353 -2011,177,640,340,300,460,217,246,Q691367 -2011,2463,34077,15482,18595,26297,10298,16002,Q648218 -2011,1668,25737,10281,15456,20428,6962,13465,Q433012 -2011,358,4510,3113,1397,3523,2311,1209,Q693357 -2011,1690,11292,5100,6192,8196,3148,5049,Q370104 -2011,984,9023,4202,4821,6753,2834,3919,Q531899 -2011,2066,19957,8639,11318,15238,5743,9495,Q1805410 -2011,842,7081,2376,4705,5497,1455,4045,Q870084 -2011,1363,15655,7201,8454,11596,4844,6753,Q693413 -2011,1189,28501,9554,18947,24293,7144,17146,Q687052 -2011,1931,15235,7380,7855,11198,4786,6413,Q656446 -2011,1030,10501,5217,5284,7210,3253,3956,Q693483 -2011,838,14560,8961,5599,11196,6607,4590,Q693269 -2011,1701,11377,6669,4708,8268,4448,3822,Q693454 -2011,715,3824,2212,1612,2678,1383,1296,Q476940 -2011,504,2042,1257,785,1483,833,647,Q392079 -2011,1526,9963,4975,4988,7537,3393,4146,Q692773 -2011,1503,9161,4556,4605,6891,3024,3869,Q693397 -2011,646,9788,5820,3968,6925,3913,3011,Q693321 +Jahr,RaumSort,RaumLang,RechtsformSort,RechtsformLang,BetriebsgrSort,BetriebsgrLang,Arbeitsstaetten,AnzBesch,AnzBeschW,AnzBeschM,AnzVZA,AnzVZAW,AnzVZAM +2011,1,Ganze Stadt,0,Alle Rechtsformen,0,Alle Betriebsgrössen,39747,443621,202970,240651,339750,136406,203349 +2011,10,Kreis 1,0,Alle Rechtsformen,0,Alle Betriebsgrössen,5249,68627,34662,33965,53138,24338,28799 +2011,11,Rathaus,0,Alle Rechtsformen,0,Alle Betriebsgrössen,1522,10180,5492,4688,7205,3493,3710 +2011,12,Hochschulen,0,Alle Rechtsformen,0,Alle Betriebsgrössen,700,13203,7025,6178,9090,4376,4714 +2011,13,Lindenhof,0,Alle Rechtsformen,0,Alle Betriebsgrössen,1193,15160,7649,7511,12213,5602,6608 diff --git a/statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_raw_input.csv b/statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_raw_input.csv new file mode 100644 index 0000000000..d22a3aca8f --- /dev/null +++ b/statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_raw_input.csv @@ -0,0 +1,6 @@ +Jahr,RaumSort,RaumLang,RechtsformSort,RechtsformLang,BetriebsgrSort,BetriebsgrLang,Arbeitsstaetten,AnzBesch,AnzBeschW,AnzBeschM,AnzVZA,AnzVZAW,AnzVZAM +2011,1,Ganze Stadt,0,Alle Rechtsformen,0,Alle Betriebsgrössen,39747,443621,202970,240651,339750,136406,203349 +2011,10,Kreis 1,0,Alle Rechtsformen,0,Alle Betriebsgrössen,5249,68627,34662,33965,53138,24338,28799 +2011,11,Rathaus,0,Alle Rechtsformen,0,Alle Betriebsgrössen,1522,10180,5492,4688,7205,3493,3710 +2011,12,Hochschulen,0,Alle Rechtsformen,0,Alle Betriebsgrössen,700,13203,7025,6178,9090,4376,4714 +2011,13,Lindenhof,0,Alle Rechtsformen,0,Alle Betriebsgrössen,1193,15160,7649,7511,12213,5602,6608 diff --git a/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_common_output.tmcf b/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_common_output.tmcf deleted file mode 100644 index ac95a18697..0000000000 --- a/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_common_output.tmcf +++ /dev/null @@ -1,9 +0,0 @@ -Node: E:wir_2552_wiki_output->E0 -observationAbout: C:wir_2552_wiki_output->observationAbout -observationDate: C:wir_2552_wiki_output->observationDate -value: C:wir_2552_wiki_output->value -variableMeasured: C:wir_2552_wiki_output->variableMeasured -unit: C:wir_2552_wiki_output->unit -scalingFactor: C:wir_2552_wiki_output->scalingFactor -measurementMethod: C:wir_2552_wiki_output->measurementMethod -typeOf: dcs:StatVarObservation diff --git a/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output.csv b/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output.csv index fa2612349d..0e0d19456a 100644 --- a/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output.csv +++ b/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output.csv @@ -1,29 +1,36 @@ -observationAbout,observationDate,value,variableMeasured,unit,scalingFactor,measurementMethod -wikidataId/Q692511,2011,1522,dcid:Count_Company,,, -wikidataId/Q692511,2011,10180,dcid:Count_Person_Employed,,, -wikidataId/Q692511,2011,5492,dcid:Count_Person_Employed_Female,,, -wikidataId/Q692511,2011,4688,dcid:Count_Person_Employed_Male,,, -wikidataId/Q692511,2011,7205,dcid:Count_Person_FullTimeEmployee,,, -wikidataId/Q692511,2011,3493,dcid:Count_Person_FullTimeEmployee_Female,,, -wikidataId/Q692511,2011,3710,dcid:Count_Person_FullTimeEmployee_Male,,, -wikidataId/Q39240,2011,700,dcid:Count_Company,,, -wikidataId/Q39240,2011,13203,dcid:Count_Person_Employed,,, -wikidataId/Q39240,2011,7025,dcid:Count_Person_Employed_Female,,, -wikidataId/Q39240,2011,6178,dcid:Count_Person_Employed_Male,,, -wikidataId/Q39240,2011,9090,dcid:Count_Person_FullTimeEmployee,,, -wikidataId/Q39240,2011,4376,dcid:Count_Person_FullTimeEmployee_Female,,, -wikidataId/Q39240,2011,4714,dcid:Count_Person_FullTimeEmployee_Male,,, -wikidataId/Q10987378,2011,1193,dcid:Count_Company,,, -wikidataId/Q10987378,2011,15160,dcid:Count_Person_Employed,,, -wikidataId/Q10987378,2011,7649,dcid:Count_Person_Employed_Female,,, -wikidataId/Q10987378,2011,7511,dcid:Count_Person_Employed_Male,,, -wikidataId/Q10987378,2011,12213,dcid:Count_Person_FullTimeEmployee,,, -wikidataId/Q10987378,2011,5602,dcid:Count_Person_FullTimeEmployee_Female,,, -wikidataId/Q10987378,2011,6608,dcid:Count_Person_FullTimeEmployee_Male,,, -wikidataId/Q1093831,2011,1834,dcid:Count_Company,,, -wikidataId/Q1093831,2011,30084,dcid:Count_Person_Employed,,, -wikidataId/Q1093831,2011,14496,dcid:Count_Person_Employed_Female,,, -wikidataId/Q1093831,2011,15588,dcid:Count_Person_Employed_Male,,, -wikidataId/Q1093831,2011,24633,dcid:Count_Person_FullTimeEmployee,,, -wikidataId/Q1093831,2011,10861,dcid:Count_Person_FullTimeEmployee_Female,,, -wikidataId/Q1093831,2011,13768,dcid:Count_Person_FullTimeEmployee_Male,,, +observationAbout,observationDate,value,variableMeasured +wikidataId/Q72,2011,39747,dcid:Count_Company +wikidataId/Q72,2011,443621,dcid:Count_Person_Employed +wikidataId/Q72,2011,202970,dcid:Count_Person_Employed_Female +wikidataId/Q72,2011,240651,dcid:Count_Person_Employed_Male +wikidataId/Q72,2011,339750,dcid:Count_Person_FullTimeEmployee +wikidataId/Q72,2011,136406,dcid:Count_Person_FullTimeEmployee_Female +wikidataId/Q72,2011,203349,dcid:Count_Person_FullTimeEmployee_Male +wikidataId/Q445559,2011,5249,dcid:Count_Company +wikidataId/Q445559,2011,68627,dcid:Count_Person_Employed +wikidataId/Q445559,2011,34662,dcid:Count_Person_Employed_Female +wikidataId/Q445559,2011,33965,dcid:Count_Person_Employed_Male +wikidataId/Q445559,2011,53138,dcid:Count_Person_FullTimeEmployee +wikidataId/Q445559,2011,24338,dcid:Count_Person_FullTimeEmployee_Female +wikidataId/Q445559,2011,28799,dcid:Count_Person_FullTimeEmployee_Male +wikidataId/Q692511,2011,1522,dcid:Count_Company +wikidataId/Q692511,2011,10180,dcid:Count_Person_Employed +wikidataId/Q692511,2011,5492,dcid:Count_Person_Employed_Female +wikidataId/Q692511,2011,4688,dcid:Count_Person_Employed_Male +wikidataId/Q692511,2011,7205,dcid:Count_Person_FullTimeEmployee +wikidataId/Q692511,2011,3493,dcid:Count_Person_FullTimeEmployee_Female +wikidataId/Q692511,2011,3710,dcid:Count_Person_FullTimeEmployee_Male +wikidataId/Q39240,2011,700,dcid:Count_Company +wikidataId/Q39240,2011,13203,dcid:Count_Person_Employed +wikidataId/Q39240,2011,7025,dcid:Count_Person_Employed_Female +wikidataId/Q39240,2011,6178,dcid:Count_Person_Employed_Male +wikidataId/Q39240,2011,9090,dcid:Count_Person_FullTimeEmployee +wikidataId/Q39240,2011,4376,dcid:Count_Person_FullTimeEmployee_Female +wikidataId/Q39240,2011,4714,dcid:Count_Person_FullTimeEmployee_Male +wikidataId/Q10987378,2011,1193,dcid:Count_Company +wikidataId/Q10987378,2011,15160,dcid:Count_Person_Employed +wikidataId/Q10987378,2011,7649,dcid:Count_Person_Employed_Female +wikidataId/Q10987378,2011,7511,dcid:Count_Person_Employed_Male +wikidataId/Q10987378,2011,12213,dcid:Count_Person_FullTimeEmployee +wikidataId/Q10987378,2011,5602,dcid:Count_Person_FullTimeEmployee_Female +wikidataId/Q10987378,2011,6608,dcid:Count_Person_FullTimeEmployee_Male diff --git a/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output.tmcf b/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output.tmcf new file mode 100644 index 0000000000..d25521a15c --- /dev/null +++ b/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output.tmcf @@ -0,0 +1,6 @@ +Node: E:zurich_population_wir_2552_wiki_output->E0 +observationAbout: C:zurich_population_wir_2552_wiki_output->observationAbout +observationDate: C:zurich_population_wir_2552_wiki_output->observationDate +value: C:zurich_population_wir_2552_wiki_output->value +variableMeasured: C:zurich_population_wir_2552_wiki_output->variableMeasured +typeOf: dcs:StatVarObservation diff --git a/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output_stat_vars.mcf b/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output_stat_vars.mcf new file mode 100644 index 0000000000..341685132d --- /dev/null +++ b/statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output_stat_vars.mcf @@ -0,0 +1,25 @@ +# Auto generated using command: "tools/statvar_importer/stat_var_processor.py --input_data=statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv --pv_map=statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv --config_file=statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=statvar_imports/zurich/wir_2552_wiki/test_data/zurich_population_wir_2552_wiki_output" on 2026-09-08 05:11:57.684851 + +Node: dcid:Count_Person_Employed +typeOf: dcid:StatisticalVariable +populationType: dcid:Person +measuredProperty: dcid:count +statType: dcid:measuredValue +employmentStatus: dcid:Employed + +Node: dcid:Count_Person_Employed_Female +typeOf: dcid:StatisticalVariable +populationType: dcid:Person +measuredProperty: dcid:count +statType: dcid:measuredValue +employmentStatus: dcid:Employed +gender: dcid:Female + +Node: dcid:Count_Person_Employed_Male +typeOf: dcid:StatisticalVariable +populationType: dcid:Person +measuredProperty: dcid:count +statType: dcid:measuredValue +employmentStatus: dcid:Employed +gender: dcid:Male + diff --git a/statvar_imports/zurich/wir_2552_wiki/validation_config.json b/statvar_imports/zurich/wir_2552_wiki/validation_config.json new file mode 100644 index 0000000000..9a1625851c --- /dev/null +++ b/statvar_imports/zurich/wir_2552_wiki/validation_config.json @@ -0,0 +1,28 @@ +{ + "schema_version": "1.0", + "rules": [ + { + "rule_id": "check_deleted_records_percent", + "description": "Checks that deleted records do not exceed 0.1%.", + "validator": "DELETED_RECORDS_PERCENT", + "params": { + "threshold": 0.1 + } + }, + { + "rule_id": "check_max_date_consistent", + "description": "Checks if the MaxDate is the same for all StatVars.", + "validator": "MAX_DATE_CONSISTENT", + "params": {} + }, + { + "rule_id": "check_max_date_freshness", + "description": "Verifies that MaxDate meets the expected freshness threshold for the dataset.", + "validator": "SQL_VALIDATOR", + "params": { + "query": "SELECT MAX(MaxDate) >= '2024' AS is_fresh, COUNT(*) AS row_cnt FROM stats", + "condition": "is_fresh = TRUE AND row_cnt > 0" + } + } + ] +} diff --git a/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv b/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv index f554cae90e..d934295bf0 100644 --- a/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv +++ b/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv @@ -1,14 +1,14 @@ parameter,value -url,https://www.stadt-zuerich.ch/portal/de/index.html +url,https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552 description, -place_type,Country +place_type,City #places_within, start_date, end_date, release_frequency, process, comments, -output_columns,"observationAbout,observationDate,value,variableMeasured,unit,scalingFactor,measurementMethod" +output_columns,"observationAbout,observationDate,value,variableMeasured" header_rows,1 -#input_rows,10 -#Import_Name,Zurich_Population_Number_Of_Company_Workplace_Employees +#aggregate_duplicate_svobs,sum +reconcile_nodes,False diff --git a/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv b/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv index db78483403..7079ba7945 100644 --- a/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv +++ b/statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv @@ -1,10 +1,56 @@ -key,prop,val,p1,v1,,,, -Zeit,observationDate,{Number},,,,,, -employees,value,{Number},populationType,Person,employmentStatus,Employed,, -companies,value,{Number},populationType,Company,,,, -employees_f,value,{Number},populationType,Person,employmentStatus,Employed,gender,Female -employees_m,value,{Number},populationType,Person,employmentStatus,Employed,gender,Male -fte,value,{Number},populationType,Person,employmentStatus,FullTimeEmployee,, -fte_f,value,{Number},populationType,Person,employmentStatus,FullTimeEmployee,gender,Female -fte_m,value,{Number},populationType,Person,employmentStatus,FullTimeEmployee,gender,Male -wikiId,observationAbout,wikidataId/{Data},,,,,, \ No newline at end of file +key,prop,val,p1,v1,p2,v2,p3,v3 +Jahr,observationDate,{Number},,,,,, +AnzBesch,value,{Number},populationType,Person,employmentStatus,Employed,, +Arbeitsstaetten,value,{Number},populationType,Company,,,, +AnzBeschW,value,{Number},populationType,Person,employmentStatus,Employed,gender,Female +AnzBeschM,value,{Number},populationType,Person,employmentStatus,Employed,gender,Male +AnzVZA,value,{Number},populationType,Person,employmentStatus,FullTimeEmployee,, +AnzVZAW,value,{Number},populationType,Person,employmentStatus,FullTimeEmployee,gender,Female +AnzVZAM,value,{Number},populationType,Person,employmentStatus,FullTimeEmployee,gender,Male +RaumLang:Ganze Stadt,observationAbout,wikidataId/Q72,,,,,, +RaumLang:Kreis 1,observationAbout,wikidataId/Q445559,,,,,, +RaumLang:Kreis 2,observationAbout,wikidataId/Q456153,,,,,, +RaumLang:Kreis 3,observationAbout,wikidataId/Q675017,,,,,, +RaumLang:Kreis 4,observationAbout,wikidataId/Q677133,,,,,, +RaumLang:Kreis 5,observationAbout,wikidataId/Q460885,,,,,, +RaumLang:Kreis 6,observationAbout,wikidataId/Q456170,,,,,, +RaumLang:Kreis 7,observationAbout,wikidataId/Q456149,,,,,, +RaumLang:Kreis 8,observationAbout,wikidataId/Q677710,,,,,, +RaumLang:Kreis 9,observationAbout,wikidataId/Q456159,,,,,, +RaumLang:Kreis 10,observationAbout,wikidataId/Q456175,,,,,, +RaumLang:Kreis 11,observationAbout,wikidataId/Q870098,,,,,, +RaumLang:Kreis 12,observationAbout,wikidataId/Q657525,,,,,, +RaumLang:Rathaus,observationAbout,wikidataId/Q692511,,,,,, +RaumLang:Hochschulen,observationAbout,wikidataId/Q39240,,,,,, +RaumLang:Lindenhof,observationAbout,wikidataId/Q10987378,,,,,, +RaumLang:City,observationAbout,wikidataId/Q1093831,,,,,, +RaumLang:Wollishofen,observationAbout,wikidataId/Q642353,,,,,, +RaumLang:Leimbach,observationAbout,wikidataId/Q691367,,,,,, +RaumLang:Enge,observationAbout,wikidataId/Q648218,,,,,, +RaumLang:Alt-Wiedikon,observationAbout,wikidataId/Q433012,,,,,, +RaumLang:Friesenberg,observationAbout,wikidataId/Q693357,,,,,, +RaumLang:Sihlfeld,observationAbout,wikidataId/Q370104,,,,,, +RaumLang:Werd,observationAbout,wikidataId/Q531899,,,,,, +RaumLang:Langstrasse,observationAbout,wikidataId/Q1805410,,,,,, +RaumLang:Hard,observationAbout,wikidataId/Q870084,,,,,, +RaumLang:Gewerbeschule,observationAbout,wikidataId/Q693413,,,,,, +RaumLang:Escher Wyss,observationAbout,wikidataId/Q687052,,,,,, +RaumLang:Unterstrass,observationAbout,wikidataId/Q656446,,,,,, +RaumLang:Oberstrass,observationAbout,wikidataId/Q693483,,,,,, +RaumLang:Fluntern,observationAbout,wikidataId/Q693269,,,,,, +RaumLang:Hottingen,observationAbout,wikidataId/Q693454,,,,,, +RaumLang:Hirslanden,observationAbout,wikidataId/Q476940,,,,,, +RaumLang:Witikon,observationAbout,wikidataId/Q392079,,,,,, +RaumLang:Seefeld,observationAbout,wikidataId/Q692773,,,,,, +RaumLang:Mühlebach,observationAbout,wikidataId/Q693397,,,,,, +RaumLang:Weinegg,observationAbout,wikidataId/Q693321,,,,,, +RaumLang:Albisrieden,observationAbout,wikidataId/Q80797,,,,,, +RaumLang:Altstetten,observationAbout,wikidataId/Q445711,,,,,, +RaumLang:Höngg,observationAbout,wikidataId/Q455496,,,,,, +RaumLang:Wipkingen,observationAbout,wikidataId/Q678030,,,,,, +RaumLang:Affoltern,observationAbout,wikidataId/Q382903,,,,,, +RaumLang:Oerlikon,observationAbout,wikidataId/Q167179,,,,,, +RaumLang:Seebach,observationAbout,wikidataId/Q276792,,,,,, +RaumLang:Saatlen,observationAbout,wikidataId/Q652455,,,,,, +RaumLang:Schwamendingen-Mitte,observationAbout,wikidataId/Q692728,,,,,, +RaumLang:Hirzenbach,observationAbout,wikidataId/Q693374,,,,,,