Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
14dcd68
[Zurich] Update Zurich_Population and Zurich_Population_Number_Of_Com…
abhishekjaisw Sep 3, 2026
cbbe5a0
Address code review feedback on Zurich imports
abhishekjaisw Sep 3, 2026
ca1cfd4
Merge branch 'master' into zurich-population-and-company-statvar-imports
abhishekjaisw Sep 3, 2026
8a027f3
Tune disk resource limit to 30GB to prevent OS disk exhaustion
abhishekjaisw Sep 3, 2026
23e7eb8
Address adversarial code review audit findings for Zurich imports
abhishekjaisw Sep 3, 2026
260a4a5
Merge branch 'master' into zurich-population-and-company-statvar-imports
abhishekjaisw Sep 4, 2026
50747a5
Preserve traceback with exc_info in generate_rollups and add main ent…
abhishekjaisw Sep 4, 2026
3d3fc4f
Revert .gitignore modification
abhishekjaisw Sep 4, 2026
7130dee
Merge branch 'master' into zurich-population-and-company-statvar-imports
abhishekjaisw Sep 4, 2026
317d173
Address forensic audit findings: filter unknown regions, align canoni…
abhishekjaisw Sep 7, 2026
0e54f71
Revert Canonical StatVar Alignment and Unknown Region Filtering changes
abhishekjaisw Sep 7, 2026
9e8a214
Merge branch 'master' into zurich-population-and-company-statvar-imports
abhishekjaisw Sep 7, 2026
45032b4
Address [P2] code review findings: filter unknown regions and update …
abhishekjaisw Sep 7, 2026
811b2df
Address review comments from pulkeet-wq: minimize test files and add …
abhishekjaisw Sep 8, 2026
ec457dd
Merge branch 'master' into zurich-population-and-company-statvar-imports
abhishekjaisw Sep 8, 2026
89f006a
Update cron schedules to twice a month (1st and 15th) per CL review f…
abhishekjaisw Sep 8, 2026
06869ba
Address [P3] code review findings: numeric coercion in RaumSort filte…
abhishekjaisw Sep 8, 2026
322dc76
Expand README.md with detailed explanation of why rollup filters on v…
abhishekjaisw Sep 8, 2026
6adfc0d
Merge branch 'master' into zurich-population-and-company-statvar-imports
abhishekjaisw Sep 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 28 additions & 17 deletions statvar_imports/zurich/README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,40 @@
1. import_name": "Zurich_Population_Number_Of_Company_Workplace_Employees"

2. Import Overview
Zurich population data at Province and City Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/wir_2552_wiki.csv)
Number of companies, workplaces and employees in Zurich city at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [BFS WIR STATENT Data](https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552)
Import Type: Fully Autorefresh
Source Data Availability: 2011 to 2022
Source Data Availability: 2011 to 2024
Release Frequency: P1Y

3. Preprocessing Steps (No)
3. Preprocessing Steps (Yes)
Generate rollups from the downloaded dataset:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the logic and explanation for roll ups

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit 811b2dfa. Added detailed rollup logic and explanation to Section 3 (Preprocessing Steps) of statvar_imports/zurich/README.md covering total row filtering (RechtsformSort == 0 and BetriebsgrSort == 0), unknown region exclusion (RaumSort 990/999), and numeric coercion of confidential markers ('K').

python3 wir_2552_wiki/generate_rollups.py

Rollup Logic & Explanation (`wir_2552_wiki/generate_rollups.py`):
- **Why Filter on Value `'0'` (`RechtsformSort == 0` & `BetriebsgrSort == 0`)**:
- In the upstream dataset (`WIR255OD2552.csv`), category code **`0` represents the official aggregate total row** across all sub-categories: `RechtsformSort == 0` is *"Alle Rechtsformen"* (all legal forms combined) and `BetriebsgrSort == 0` is *"Alle Betriebsgrössen"* (all company sizes combined). Non-zero values (`1`, `2`, etc.) represent granular sub-category breakdowns (e.g., public-law vs. private-law entities, micro-businesses `<10` vs. small businesses `10–49`).
- **Matches Target StatVar Schema**: The Data Commons StatVars emitted by this import (`Count_Company`, `Count_Person_Employed`, `Count_Person_Employed_Female`, `Count_Person_Employed_Male`, `Count_Person_FullTimeEmployee`, etc.) represent overall totals per place and year without legal form or company size constraints.
- **Prevents Conflicting Duplicate Observations**: Retaining non-zero breakdown rows would cause `stat_var_processor.py` to map multiple sub-category rows to the exact same `(place, year, StatVar)` tuple, causing observation collisions.
- **Avoids Undercounting from Suppressed Cells (`'K'`)**: Individual sub-category breakdown rows frequently contain `'K'` (confidentiality suppression for small counts). Summing non-zero breakdown rows manually would undercount true totals, whereas the official `(0, 0)` row provides the complete, authoritative total published by Statistics Zurich.
- **Unknown Region Exclusion**: Filters out unknown/unassigned region rows (`RaumSort` `990` ["Kreis Unbekannt"] and `999` ["Quartier Unbekannt"]) that do not map to valid geographic places.
- **Numeric Coercion**: Coerces metric columns (`Arbeitsstaetten`, `AnzBesch`, `AnzBeschW`, `AnzBeschM`, `AnzVZA`, `AnzVZAW`, `AnzVZAM`) to numeric values, converting non-numeric markers (e.g., `'K'` for confidential/suppressed entries) to `NaN` so they are cleanly skipped during StatVar observation generation.

4. Autorefresh Type

Fully Autorefresh:"0 2 29 * * " (Runs at 2:00 AM on the 29th day of every month).
Fully Autorefresh:"0 2 1,15 * * " (Runs at 2:00 AM on the 1st and 15th day of every month).

5. Script Execution Details

" python3 stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data='../../statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv' --pv_map='../../statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv' --config_file='../../statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv' --output_path=../../statvar_imports/zurich/wir_2552_wiki/output/zurich_population_wir_2552_wiki_output "
" python3 ../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552/download/WIR255OD2552.csv --output_folder=wir_2552_wiki/input_files && python3 wir_2552_wiki/generate_rollups.py && python3 ../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=wir_2552_wiki/input_files/WIR255OD2552_rollups.csv --pv_map=wir_2552_wiki/wir_2552_wiki_pvmap.csv --config_file=wir_2552_wiki/wir_2552_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=wir_2552_wiki/output/zurich_population_wir_2552_wiki --output_counters=wir_2552_wiki/counters/zurich_population_wir_2552_wiki_counters.csv "

#####


1. import_name": "Zurich_Population_By_Age"

2. Import Overview
Zurich population data at Province and City Level.
Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_age10_wiki.csv)
Import Type: Fully Autorefresh
Source Data Availability: 1993 to 2023
Expand All @@ -45,29 +56,29 @@ Fully Autorefresh:" 0 6 29 * * " (Runs at 6:00 AM on the 29th of every month).
1. import_name": "Zurich_Population"

2. Import Overview
Zurich population data at Province and City Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv)
Total population of Zurich city by quarter and year at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [BEV324OD3240 Dataset](https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240)
Import Type: Fully Autorefresh
Source Data Availability: 1941 to 2023
Source Data Availability: 1941 to 2025
Release Frequency: P1Y

3. Preprocessing Steps (No)

4. Autorefresh Type

Fully Autorefresh:" 30 11 29 * * " (Runs at 11:30 AM on the 29th of every month).
Fully Autorefresh:" 30 11 1,15 * * " (Runs at 11:30 AM on the 1st and 15th of every month).

5. Script Execution Details

" python3 stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data='../../statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv' --pv_map='../../statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv' --config_file='../../statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv' --output_path=../../statvar_imports/zurich/bev_3240_wiki/output/zurich_population_bev_3240_wiki.csv_output "
" python3 ../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240/download/BEV324OD3240.csv --output_folder=bev_3240_wiki/input_files && python3 ../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=bev_3240_wiki/input_files/BEV324OD3240.csv --pv_map=bev_3240_wiki/bev_3240_wiki_pvmap.csv --config_file=bev_3240_wiki/bev_3240_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=bev_3240_wiki/output/zurich_population_bev_3240_wiki --output_counters=bev_3240_wiki/counters/zurich_population_bev_3240_wiki_counters.csv "

#####


1. import_name": "Zurich_Population_Number_Of_Birth_By_Origin"

2. Import Overview
Zurich population data at Province and City Level.
Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_hel_wiki.csv)
Import Type: Fully Autorefresh
Source Data Availability: 1998 to 2023
Expand All @@ -89,7 +100,7 @@ Fully Autorefresh:" 45 15 29 * * " (Runs at 3:45 PM on the 29th of every month).
1. import_name": "Zurich_Population_Number_Of_Birth"

2. Import Overview
Zurich population data at Province and City Level.
Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_wiki.csv)
Import Type: Fully Autorefresh
Source Data Availability: 1998 to 2023
Expand All @@ -111,7 +122,7 @@ Fully Autorefresh:" 0 20 29 * * " (Runs at 8:00 PM on the 29th of every month).
1. import_name": "Zurich_Population_By_Origin"

2. Import Overview
Zurich population data at Province and City Level.
Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_hel_wiki.csv)
Import Type: Fully Autorefresh
Source Data Availability: 1993 to 2023
Expand All @@ -134,7 +145,7 @@ Fully Autorefresh:" 15 1 29 * * " (Runs at 1:15 AM on the 29th of every month).
1. import_name": "Zurich_Population_Number_Of_Birth_By_Sex"

2. Import Overview
Zurich population data at Province and City Level.
Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_sex_wiki.csv)
Import Type: Fully Autorefresh
Source Data Availability: 1993 to 2023
Expand All @@ -156,7 +167,7 @@ Fully Autorefresh:" 07 19 29 * * " (Runs at 7:07 PM on the 29th of every month).
1. import_name": "Zurich_Population_By_Sex"

2. Import Overview
Zurich population data at Province and City Level.
Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_sex_wiki.csv)
Import Type: Fully Autorefresh
Source Data Availability: 1993 to 2023
Expand Down
10 changes: 5 additions & 5 deletions statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
parameter,value
url,https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv
url,https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240
description,
place_type,Country
place_type,City
#places_within,
start_date,
end_date,
release_frequency,
process,
comments,
output_columns,"observationAbout,observationDate,value,variableMeasured,unit"
output_columns,"observationAbout,observationDate,value,variableMeasured"
header_rows,1
#input_rows,10
#Import_Name,Zurich_Population
reconcile_nodes,False
output_counters,bev_3240_wiki/counters/zurich_population_bev_3240_wiki_counters.csv
41 changes: 37 additions & 4 deletions statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv
Original file line number Diff line number Diff line change
@@ -1,4 +1,37 @@
key,prop,val,p1,v1
StichtagDatJahr,observationDate,{Number},,
AnzBestWir,value,{Number},populationType,Person
wikiId,observationAbout,wikidataId/{Data},,
key,prop,val,p1,v1
StichtagDatJahr,observationDate,{Number},,
AnzBestWir,value,{Number},populationType,Person
QuarLang:Rathaus,observationAbout,wikidataId/Q692511,,
QuarLang:Hochschulen,observationAbout,wikidataId/Q39240,,
QuarLang:Lindenhof,observationAbout,wikidataId/Q10987378,,
QuarLang:City,observationAbout,wikidataId/Q1093831,,
QuarLang:Wollishofen,observationAbout,wikidataId/Q642353,,
QuarLang:Leimbach,observationAbout,wikidataId/Q691367,,
QuarLang:Enge,observationAbout,wikidataId/Q648218,,
QuarLang:Alt-Wiedikon,observationAbout,wikidataId/Q433012,,
QuarLang:Friesenberg,observationAbout,wikidataId/Q693357,,
QuarLang:Sihlfeld,observationAbout,wikidataId/Q370104,,
QuarLang:Werd,observationAbout,wikidataId/Q531899,,
QuarLang:Langstrasse,observationAbout,wikidataId/Q1805410,,
QuarLang:Hard,observationAbout,wikidataId/Q870084,,
QuarLang:Gewerbeschule,observationAbout,wikidataId/Q693413,,
QuarLang:Escher Wyss,observationAbout,wikidataId/Q687052,,
QuarLang:Unterstrass,observationAbout,wikidataId/Q656446,,
QuarLang:Oberstrass,observationAbout,wikidataId/Q693483,,
QuarLang:Fluntern,observationAbout,wikidataId/Q693269,,
QuarLang:Hottingen,observationAbout,wikidataId/Q693454,,
QuarLang:Hirslanden,observationAbout,wikidataId/Q476940,,
QuarLang:Witikon,observationAbout,wikidataId/Q392079,,
QuarLang:Seefeld,observationAbout,wikidataId/Q692773,,
QuarLang:Mühlebach,observationAbout,wikidataId/Q693397,,
QuarLang:Weinegg,observationAbout,wikidataId/Q693321,,
QuarLang:Albisrieden,observationAbout,wikidataId/Q80797,,
QuarLang:Altstetten,observationAbout,wikidataId/Q445711,,
QuarLang:Höngg,observationAbout,wikidataId/Q455496,,
QuarLang:Wipkingen,observationAbout,wikidataId/Q678030,,
QuarLang:Affoltern,observationAbout,wikidataId/Q382903,,
QuarLang:Oerlikon,observationAbout,wikidataId/Q167179,,
QuarLang:Seebach,observationAbout,wikidataId/Q276792,,
QuarLang:Saatlen,observationAbout,wikidataId/Q652455,,
QuarLang:Schwamendingen-Mitte,observationAbout,wikidataId/Q692728,,
QuarLang:Hirzenbach,observationAbout,wikidataId/Q693374,,
Original file line number Diff line number Diff line change
@@ -1,25 +1,6 @@
StichtagDatJahr,AnzBestWir,wikiId
2023,3343,Q692511
2022,3330,Q692511
2021,3275,Q692511
2020,3317,Q692511
2019,3307,Q692511
2018,3267,Q692511
2017,3266,Q692511
2016,3226,Q692511
2015,3197,Q692511
2014,3236,Q692511
2013,3194,Q692511
2012,3164,Q692511
2011,3140,Q692511
2010,3099,Q692511
2009,3110,Q692511
2008,3051,Q692511
2007,3079,Q692511
2006,3097,Q692511
2005,3081,Q692511
2004,3115,Q692511
2003,3149,Q692511
2002,3128,Q692511
2001,3103,Q692511
2000,3109,Q692511
StichtagDatJahr,QuarSort,QuarLang,AnzBestWir

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets keep no more than 5 records in all test files. Please try to minimise data as much as possible

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in commit 811b2dfa. Trimmed all test input CSV files (bev_3240_wiki_input.csv, wir_2552_wiki_raw_input.csv, and wir_2552_wiki_input.csv) to 5 records each (6 lines total including header) and regenerated the corresponding test output files via stat_var_processor.py.

2025,11,Rathaus,3252
2024,11,Rathaus,3260
2023,11,Rathaus,3343
2022,11,Rathaus,3330
2021,11,Rathaus,3275
Original file line number Diff line number Diff line change
@@ -1,25 +1,6 @@
observationAbout,observationDate,value,variableMeasured,unit
wikidataId/Q692511,2023,3343,dcid:Count_Person,
wikidataId/Q692511,2022,3330,dcid:Count_Person,
wikidataId/Q692511,2021,3275,dcid:Count_Person,
wikidataId/Q692511,2020,3317,dcid:Count_Person,
wikidataId/Q692511,2019,3307,dcid:Count_Person,
wikidataId/Q692511,2018,3267,dcid:Count_Person,
wikidataId/Q692511,2017,3266,dcid:Count_Person,
wikidataId/Q692511,2016,3226,dcid:Count_Person,
wikidataId/Q692511,2015,3197,dcid:Count_Person,
wikidataId/Q692511,2014,3236,dcid:Count_Person,
wikidataId/Q692511,2013,3194,dcid:Count_Person,
wikidataId/Q692511,2012,3164,dcid:Count_Person,
wikidataId/Q692511,2011,3140,dcid:Count_Person,
wikidataId/Q692511,2010,3099,dcid:Count_Person,
wikidataId/Q692511,2009,3110,dcid:Count_Person,
wikidataId/Q692511,2008,3051,dcid:Count_Person,
wikidataId/Q692511,2007,3079,dcid:Count_Person,
wikidataId/Q692511,2006,3097,dcid:Count_Person,
wikidataId/Q692511,2005,3081,dcid:Count_Person,
wikidataId/Q692511,2004,3115,dcid:Count_Person,
wikidataId/Q692511,2003,3149,dcid:Count_Person,
wikidataId/Q692511,2002,3128,dcid:Count_Person,
wikidataId/Q692511,2001,3103,dcid:Count_Person,
wikidataId/Q692511,2000,3109,dcid:Count_Person,
observationAbout,observationDate,value,variableMeasured
wikidataId/Q692511,2025,3252,dcid:Count_Person
wikidataId/Q692511,2024,3260,dcid:Count_Person
wikidataId/Q692511,2023,3343,dcid:Count_Person
wikidataId/Q692511,2022,3330,dcid:Count_Person
wikidataId/Q692511,2021,3275,dcid:Count_Person
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
Node: E:bev_3240_wiki_output->E0
observationAbout: C:bev_3240_wiki_output->observationAbout
observationDate: C:bev_3240_wiki_output->observationDate
value: C:bev_3240_wiki_output->value
variableMeasured: C:bev_3240_wiki_output->variableMeasured
unit: C:bev_3240_wiki_output->unit
Node: E:zurich_population_bev_3240_wiki_output->E0
observationAbout: C:zurich_population_bev_3240_wiki_output->observationAbout
observationDate: C:zurich_population_bev_3240_wiki_output->observationDate
value: C:zurich_population_bev_3240_wiki_output->value
variableMeasured: C:zurich_population_bev_3240_wiki_output->variableMeasured
typeOf: dcs:StatVarObservation
28 changes: 28 additions & 0 deletions statvar_imports/zurich/bev_3240_wiki/validation_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"schema_version": "1.0",
"rules": [
{
"rule_id": "check_deleted_records_percent",
"description": "Checks that the percentage of deleted points is within the threshold.",
"validator": "DELETED_RECORDS_PERCENT",
"params": {
"threshold": 0.1
}
},
{
"rule_id": "check_max_date_consistent",
"description": "Checks if the MaxDate is the same for all StatVars.",
"validator": "MAX_DATE_CONSISTENT",
"params": {}
},
{
"rule_id": "check_max_date_freshness",
"description": "Verifies that MaxDate meets the expected freshness threshold for the dataset.",
"validator": "SQL_VALIDATOR",
"params": {
"query": "SELECT MAX(MaxDate) >= '2025' AS is_fresh, COUNT(*) AS row_cnt FROM stats",
"condition": "is_fresh = TRUE AND row_cnt > 0"
}
}
]
}
Loading
Loading