Fix epacriteriaairpollutants - #2206
Conversation
- Pre-declare 47 unindexed EPA Air Quality monitoring stations in EPA_AirQuality.mcf - Standardize units and add retry logic in air_quality.py - Add node_mcf, validation_config_file, and resource limits to manifest.json - Add validation_config.json with strict zero lint error threshold
…ality.mcf to eliminate remote DC API calls
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a diagnostic runbook, configures validation rules, and updates the air_quality.py script to handle unit mapping, filter out cross-border monitors, and use a robust HTTP session with retries for downloading data. Feedback on the changes includes addressing a critical runtime AttributeError when configuring retries, relaxing the lint error threshold in validation_config.json to prevent pipeline failures from new stations, and resolving a potential KeyError and eager evaluation issue when parsing observation units.
| raw_unit = observation.get('Units of Measure', '').strip().lower() | ||
| unit = UNIT_MAP.get(raw_unit, | ||
| get_camel_case(observation['Units of Measure'])) |
There was a problem hiding this comment.
The current implementation eagerly evaluates get_camel_case(observation['Units of Measure']) as the default argument to UNIT_MAP.get(), even if raw_unit is found in UNIT_MAP. Furthermore, if 'Units of Measure' is missing from observation, this will raise a KeyError despite the defensive .get() call on line 164. Using a short-circuiting or avoids both the eager evaluation and the potential KeyError.
raw_unit = observation.get('Units of Measure', '') or ''
unit = UNIT_MAP.get(raw_unit.strip().lower()) or get_camel_case(raw_unit)…and clean up validation config
1 participant
cloud job - https://pantheon.corp.google.com/batch/jobsDetail/regions/us-west4/jobs/epacriteriaairpollutants-shouryasingh-20260907-133701/details?e=13803378&invt=Ab5y7Q&mods=-monitoring_api_staging&project=datcom-infosys-dev