diff --git a/statvar_imports/us_cdc/county_mortality/README.md b/statvar_imports/us_cdc/county_mortality/README.md new file mode 100644 index 0000000000..2d94b88e6a --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/README.md @@ -0,0 +1,83 @@ +### CDC WONDER County-Level Mortality Across All Causes (`CDC_Mortality_UnderlyingCause`) + +This import acquires and processes county-level mortality statistics across all causes of death (ICD-10 113 Cause List) for all 50 US States and Washington D.C. from the CDC WONDER database. + +- **Import Name**: `CDC_Mortality_UnderlyingCause` +- **Source Database**: CDC WONDER Underlying Cause of Death (Database D158) +- **Source URL**: `https://wonder.cdc.gov/ucd-icd10-expanded.html` +- **Geographic Granularity**: County level (all ~3,143 US counties across all states) +- **Temporal Coverage**: 2018 to 2024 (P1Y frequency) +- **Cause Coverage**: All diseases in the ICD-10 113 Cause List (e.g. Septicemia, Diabetes Mellitus, Major Cardiovascular Diseases, Alzheimer's, Malignant Neoplasms, Respiratory Diseases, etc.) + +--- + +### Workflow + +The import consists of two fully automated steps: + +#### Step 1: Download Source Data +```bash +python3 download.py +``` +or with specific flags: +```bash +python3 download.py --states=all --years=2018-2024 --output_dir=input_files +``` +The download script automatically handles CDC WONDER sessions, agreements, rate-limiting backoffs, and query partitioning. + +#### Step 2: Process Data into Cleaned SVObs and TMCF +```bash +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \ + --input_data=input_files/*.csv \ + --pv_map=county_mortality_pvmap.csv \ + --config_file=county_mortality_metadata.csv \ + --output_path=output/underlyingcauseofdeath_county \ + --output_counters=counters/underlyingcauseofdeath_county_counters.csv +``` + +#### Output Artifacts +* `output/underlyingcauseofdeath_county.csv`: Cleaned observations mapping FIPS (`geoId/{fips}`), year, StatVar (`Count_MortalityEvent_`), death count, and unit. +* `output/underlyingcauseofdeath_county.tmcf`: Template MCF mapping CSV columns to Data Commons Knowledge Graph entities. + +--- + +### Operational Notes & Downloader Architecture + +* **Rate-Limit Handling (HTTP 429)**: CDC WONDER enforces a 30-minute IP block when query thresholds are exceeded. The downloader detects HTTP 429 responses, pauses in complete silence for 31 minutes (`1,860s`), and automatically refreshes the session before resuming. Do not terminate the process during this cooldown. +* **Large State Partitioning**: High-population states (e.g. California `06`, Florida `12`, New York `36`, Texas `48`) exceed CDC WONDER's 75,000-row query cap when queried across all years at once. The downloader automatically queries these states in 2-year chunks (or 1-year chunks for Texas) to prevent HTTP 400 ("Too Much Data") and HTTP 504 timeouts. +* **Stale Partition Cleanup**: When a state is re-downloaded, the downloader purges existing chunk and combined files for that state before writing new data, preventing duplicate observations when `stat_var_processor.py` processes `input_files/*.csv`. +* **Failure Isolation**: If an individual state query encounters network or server errors, the downloader logs the failure, continues with the remaining states, and raises a summary exception at the end of the batch run to ensure non-zero exit code while preserving downloaded progress. +* **Lazy Session Initialization**: The session agreement with CDC WONDER is only initialized when at least one state actually requires downloading, avoiding unnecessary network calls during dry runs or when data is already cached. + +--- + +### Validation + +Validation is configured in `validation_config.json`: +* `check_deleted_records_percent`: Strictly enforces a historical deletion average threshold of `0.1%`. Note that on the initial import run, this rule expectedly yields `DATA_ERROR` ("Differ summary is missing required field: 'previous_obs_count'") because there is no prior version to diff against; it is configured to safeguard future recurring refreshes. Per consensus on initial imports, golden regression files are omitted from initial submission. +* `check_max_date_freshness`: Enforces date freshness (`CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 3)`) across 116 cause-of-death StatVars to ensure latest published data is ingested relative to annual publication lag. +* `check_max_date_freshness_rare_causes`: Enforces date freshness (`CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 4)`) for 2 rare historical exception causes with low observation frequency due to CDC sub-national cell suppression (< 10 deaths). + +--- + +### Important Files + +| File | Description | +|---|---| +| `download.py` | Automated live CDC WONDER downloader, session manager, and query partitioner. | +| `download_test.py` | Comprehensive unit test suite for download session management and error handling. | +| `county_mortality_metadata.csv` | Metadata specifying header row offsets, frequency, and output columns. | +| `county_mortality_pvmap.csv` | Property-Value mapping resolving county FIPS and ICD-10 113 causes of death. | +| `manifest.json` | Automation manifest declaring scripts, inputs/outputs, cron schedule, and validation config. | +| `validation_config.json` | Configuration defining import validation rules (historical deletion and date freshness). | +| `test_data/` | Trimmed sample Delaware dataset and expected outputs for offline verification. | + +--- + +### Testing + +Run the unit test suite from this directory: +```bash +python3 -m unittest discover -v -s . -p "*_test.py" +``` diff --git a/statvar_imports/us_cdc/county_mortality/county_mortality_metadata.csv b/statvar_imports/us_cdc/county_mortality/county_mortality_metadata.csv new file mode 100644 index 0000000000..6c4034eeab --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/county_mortality_metadata.csv @@ -0,0 +1,11 @@ +parameter,value +url,https://wonder.cdc.gov/controller/datarequest/D158 +description,"County-level mortality statistics across all causes of death (ICD-10 113 Cause List) for all US states" +#place_type,county level +start_date,2018 +end_date,2024 +release_frequency,P1Y +output_columns,"observationAbout,observationDate,variableMeasured,value" +mapped_rows,1 +mapped_columns,11 +header_rows,1 diff --git a/statvar_imports/us_cdc/county_mortality/county_mortality_pvmap.csv b/statvar_imports/us_cdc/county_mortality/county_mortality_pvmap.csv new file mode 100644 index 0000000000..e8c58f3d8a --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/county_mortality_pvmap.csv @@ -0,0 +1,143 @@ +key,p1,v1,p2,v2 +County Code,#Format,CountyCode1={Number:0>5},observationAbout,geoId/{CountyCode1} +Year Code,observationDate,{Number},observationPeriod,P1Y +Deaths,populationType,MortalityEvent,value,{Number} +Population,#ignore,ignore,value,{Number} +Crude Rate,#ignore,ignore,value,{Number} +GR113-001,causeOfDeath,SalmonellaInfections,, +GR113-003,causeOfDeath,CertainOtherIntestinalInfections,, +GR113-004,causeOfDeath,Tuberculosis,, +GR113-005,causeOfDeath,RespiratoryTuberculosis,, +GR113-006,causeOfDeath,OtherTuberculosis,, +GR113-009,causeOfDeath,MeningococcalInfection,, +GR113-010,causeOfDeath,Septicemia,, +GR113-015,causeOfDeath,ViralHepatitis,, +GR113-016,causeOfDeath,HumanImmunodeficiencyVirus(Hiv)Disease,, +GR113-018,causeOfDeath,OtherAndUnspecifiedInfectiousAndParasiticDiseasesAndTheirSequelae,, +GR113-019,causeOfDeath,MalignantNeoplasms,, +GR113-020,causeOfDeath,MalignantNeoplasmsOfLip_OralCavityAndPharynx,, +GR113-021,causeOfDeath,MalignantNeoplasmOfEsophagus,, +GR113-022,causeOfDeath,MalignantNeoplasmOfStomach,, +GR113-023,causeOfDeath,MalignantNeoplasmsOfColon_RectumAndAnus,, +GR113-024,causeOfDeath,MalignantNeoplasmsOfLiverAndIntrahepaticBileDucts,, +GR113-025,causeOfDeath,MalignantNeoplasmOfPancreas,, +GR113-026,causeOfDeath,MalignantNeoplasmOfLarynx,, +GR113-027,causeOfDeath,MalignantNeoplasmsOfTrachea_BronchusAndLung,, +GR113-028,causeOfDeath,MalignantMelanomaOfSkin,, +GR113-029,causeOfDeath,MalignantNeoplasmOfBreast,, +GR113-030,causeOfDeath,MalignantNeoplasmOfCervixUteri,, +GR113-031,causeOfDeath,MalignantNeoplasmsOfCorpusUteriAndUterus_PartUnspecified,, +GR113-032,causeOfDeath,MalignantNeoplasmOfOvary,, +GR113-033,causeOfDeath,MalignantNeoplasmOfProstate,, +GR113-034,causeOfDeath,MalignantNeoplasmsOfKidneyAndRenalPelvis,, +GR113-035,causeOfDeath,MalignantNeoplasmOfBladder,, +GR113-036,causeOfDeath,MalignantNeoplasmsOfMeninges_BrainAndOtherPartsOfCentralNervousSystem,, +GR113-037,causeOfDeath,MalignantNeoplasmsOfLymphoid_HematopoieticAndRelatedTissue,, +GR113-038,causeOfDeath,HodgkinsDisease,, +GR113-039,causeOfDeath,Non-HodgkinsLymphoma,, +GR113-040,causeOfDeath,Leukemia,, +GR113-041,causeOfDeath,MultipleMyelomaAndImmunoproliferativeNeoplasms,, +GR113-042,causeOfDeath,OtherAndUnspecifiedMalignantNeoplasmsOfLymphoid_HematopoieticAndRelatedTissue,, +GR113-043,causeOfDeath,AllOtherAndUnspecifiedMalignantNeoplasms,, +GR113-044,causeOfDeath,InSituNeoplasms_BenignNeoplasmsAndNeoplasmsOfUncertainOrUnknownBehavior,, +GR113-045,causeOfDeath,Anemias,, +GR113-046,causeOfDeath,DiabetesMellitus,, +GR113-047,causeOfDeath,NutritionalDeficiencies,, +GR113-048,causeOfDeath,Malnutrition,, +GR113-049,causeOfDeath,OtherNutritionalDeficiencies,, +GR113-050,causeOfDeath,Meningitis,, +GR113-051,causeOfDeath,ParkinsonsDisease,, +GR113-052,causeOfDeath,AlzheimersDisease,, +GR113-053,causeOfDeath,MajorCardiovascularDiseases,, +GR113-054,causeOfDeath,DiseasesOfHeart,, +GR113-055,causeOfDeath,AcuteRheumaticFeverAndChronicRheumaticHeartDiseases,, +GR113-056,causeOfDeath,HypertensiveHeartDisease,, +GR113-057,causeOfDeath,HypertensiveHeartAndRenalDisease,, +GR113-058,causeOfDeath,IschemicHeartDiseases,, +GR113-059,causeOfDeath,AcuteMyocardialInfarction,, +GR113-060,causeOfDeath,OtherAcuteIschemicHeartDiseases,, +GR113-061,causeOfDeath,OtherFormsOfChronicIschemicHeartDisease,, +GR113-062,causeOfDeath,AtheroscleroticCardiovascularDisease_SoDescribed,, +GR113-063,causeOfDeath,AllOtherFormsOfChronicIschemicHeartDisease,, +GR113-064,causeOfDeath,OtherHeartDiseases,, +GR113-065,causeOfDeath,AcuteAndSubacuteEndocarditis,, +GR113-066,causeOfDeath,DiseasesOfPericardiumAndAcuteMyocarditis,, +GR113-067,causeOfDeath,HeartFailure,, +GR113-068,causeOfDeath,AllOtherFormsOfHeartDisease,, +GR113-069,causeOfDeath,Essential(Primary)HypertensionAndHypertensiveRenalDisease,, +GR113-070,causeOfDeath,CerebrovascularDiseases,, +GR113-071,causeOfDeath,Atherosclerosis,, +GR113-072,causeOfDeath,OtherDiseasesOfCirculatorySystem,, +GR113-073,causeOfDeath,AorticAneurysmAndDissection,, +GR113-074,causeOfDeath,OtherDiseasesOfArteries_ArteriolesAndCapillaries,, +GR113-075,causeOfDeath,OtherDisordersOfCirculatorySystem,, +GR113-076,causeOfDeath,InfluenzaAndPneumonia,, +GR113-077,causeOfDeath,Influenza,, +GR113-078,causeOfDeath,Pneumonia,, +GR113-079,causeOfDeath,OtherAcuteLowerRespiratoryInfections,, +GR113-080,causeOfDeath,AcuteBronchitisAndBronchiolitis,, +GR113-081,causeOfDeath,UnspecifiedAcuteLowerRespiratoryInfection,, +GR113-082,causeOfDeath,ChronicLowerRespiratoryDiseases,, +GR113-083,causeOfDeath,Bronchitis_ChronicAndUnspecified,, +GR113-084,causeOfDeath,Emphysema,, +GR113-085,causeOfDeath,Asthma,, +GR113-086,causeOfDeath,OtherChronicLowerRespiratoryDiseases,, +GR113-087,causeOfDeath,PneumoconiosesAndChemicalEffects,, +GR113-088,causeOfDeath,PneumonitisDueToSolidsAndLiquids,, +GR113-089,causeOfDeath,OtherDiseasesOfRespiratorySystem,, +GR113-090,causeOfDeath,PepticUlcer,, +GR113-091,causeOfDeath,DiseasesOfAppendix,, +GR113-092,causeOfDeath,Hernia,, +GR113-093,causeOfDeath,ChronicLiverDiseaseAndCirrhosis,, +GR113-094,causeOfDeath,AlcoholicLiverDisease,, +GR113-095,causeOfDeath,OtherChronicLiverDiseaseAndCirrhosis,, +GR113-096,causeOfDeath,CholelithiasisAndOtherDisordersOfGallbladder,, +GR113-097,causeOfDeath,Nephritis_NephroticSyndromeAndNephrosis,, +GR113-098,causeOfDeath,AcuteAndRapidlyProgressiveNephriticAndNephroticSyndrome,, +GR113-099,causeOfDeath,ChronicGlomerulonephritis_NephritisAndNephritisNotSpecifiedAsAcuteOrChronic_AndRenalSclerosisUnspecified,, +GR113-100,causeOfDeath,RenalFailure,, +GR113-102,causeOfDeath,InfectionsOfKidney,, +GR113-103,causeOfDeath,HyperplasiaOfProstate,, +GR113-104,causeOfDeath,InflammatoryDiseasesOfFemalePelvicOrgans,, +GR113-105,causeOfDeath,Pregnancy_ChildbirthAndThePuerperium,, +GR113-106,causeOfDeath,PregnancyWithAbortiveOutcome,, +GR113-107,causeOfDeath,OtherComplicationsOfPregnancy_ChildbirthAndThePuerperium,, +GR113-108,causeOfDeath,CertainConditionsOriginatingInThePerinatalPeriod,, +GR113-109,causeOfDeath,CongenitalMalformations_DeformationsAndChromosomalAbnormalities,, +GR113-110,causeOfDeath,Symptoms_SignsAndAbnormalClinicalAndLaboratoryFindings_NotElsewhereClassified,, +GR113-112,causeOfDeath,Accidents(UnintentionalInjuries),, +GR113-113,causeOfDeath,TransportAccidents,, +GR113-114,causeOfDeath,MotorVehicleAccidents,, +GR113-115,causeOfDeath,OtherLandTransportAccidents,, +GR113-116,causeOfDeath,Water_AirAndSpace_AndOtherAndUnspecifiedTransportAccidentsAndTheirSequelae,, +GR113-117,causeOfDeath,NontransportAccidents,, +GR113-118,causeOfDeath,Falls,, +GR113-119,causeOfDeath,AccidentalDischargeOfFirearms,, +GR113-120,causeOfDeath,AccidentalDrowningAndSubmersion,, +GR113-121,causeOfDeath,AccidentalExposureToSmoke_FireAndFlames,, +GR113-122,causeOfDeath,AccidentalPoisoningAndExposureToNoxiousSubstances,, +GR113-123,causeOfDeath,OtherAndUnspecifiedNontransportAccidentsAndTheirSequelae,, +GR113-124,causeOfDeath,Suicide,, +GR113-125,causeOfDeath,IntentionalSelf-Harm(Suicide)ByDischargeOfFirearms,, +GR113-126,causeOfDeath,IntentionalSelf-Harm(Suicide)ByOtherAndUnspecifiedMeansAndTheirSequelae,, +GR113-127,causeOfDeath,Assault(Homicide),, +GR113-128,causeOfDeath,Assault(Homicide)ByDischargeOfFirearms,, +GR113-129,causeOfDeath,Assault(Homicide)ByOtherAndUnspecifiedMeansAndTheirSequelae,, +GR113-130,causeOfDeath,LegalIntervention,, +GR113-131,causeOfDeath,EventsOfUndeterminedIntent,, +GR113-132,causeOfDeath,DischargeOfFirearms_UndeterminedIntent,, +GR113-133,causeOfDeath,OtherAndUnspecifiedEventsOfUndeterminedIntentAndTheirSequelae,, +GR113-135,causeOfDeath,ComplicationsOfMedicalAndSurgicalCare,, +GR113-136,causeOfDeath,EnterocolitisDueToClostridiumDifficile,, +GR113-137,causeOfDeath,COVID_19,, +GR113-111,causeOfDeath,AllOtherDiseases(Residual),, +GR113-013,causeOfDeath,Arthropod-BorneViralEncephalitis,, +GR113-017,causeOfDeath,Malaria,, +GR113-134,causeOfDeath,OperationsOfWarAndTheirSequelae,, +GR113-101,causeOfDeath,OtherDisordersOfKidney,, +GR113-002,causeOfDeath,ShigellosisAndAmebiasis,, +GR113-011,causeOfDeath,Syphilis,, +GR113-007,causeOfDeath,WhoopingCough,, +GR113-008,causeOfDeath,ScarletFeverAndErysipelas,, +GR113-012,causeOfDeath,AcutePoliomyelitis,, +GR113-014,causeOfDeath,Measles,, diff --git a/statvar_imports/us_cdc/county_mortality/download.py b/statvar_imports/us_cdc/county_mortality/download.py new file mode 100644 index 0000000000..94cf8f1036 --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/download.py @@ -0,0 +1,796 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Automated live downloader for CDC WONDER County-Level Mortality Data (D158). + +This script automates downloading county-level mortality statistics across all +causes of death (ICD-10 113 Cause List) from CDC WONDER (Database D158: +Underlying Cause of Death). + +Data is broken down by: +- Year (2018 to 2024, or specified range) +- County (all US counties across 50 states + DC) +- ICD-10 113 Cause List (all diseases/causes) + +CDC WONDER imposes a hard limit of 75,000 rows per export query. This script +queries state by state, automatically detects if a state query exceeds the +75,000 row cap, and splits into smaller year chunks to download complete data. +""" + +import csv +import datetime +import io +import os +from pathlib import Path +import time +from typing import List, Optional, Tuple +from urllib.parse import urljoin + +from absl import app, flags, logging +from bs4 import BeautifulSoup +import requests +from requests.adapters import HTTPAdapter +from retry import retry +from urllib3.util import Retry + +script_dir = os.path.dirname(os.path.abspath(__file__)) +DEFAULT_INPUT_DIR = os.path.join(script_dir, "input_files") + +# Mapping of 2-digit FIPS codes to US State / Territory names +US_STATES = { + "01": "Alabama", + "02": "Alaska", + "04": "Arizona", + "05": "Arkansas", + "06": "California", + "08": "Colorado", + "09": "Connecticut", + "10": "Delaware", + "11": "District of Columbia", + "12": "Florida", + "13": "Georgia", + "15": "Hawaii", + "16": "Idaho", + "17": "Illinois", + "18": "Indiana", + "19": "Iowa", + "20": "Kansas", + "21": "Kentucky", + "22": "Louisiana", + "23": "Maine", + "24": "Maryland", + "25": "Massachusetts", + "26": "Michigan", + "27": "Minnesota", + "28": "Mississippi", + "29": "Missouri", + "30": "Montana", + "31": "Nebraska", + "32": "Nevada", + "33": "New Hampshire", + "34": "New Jersey", + "35": "New Mexico", + "36": "New York", + "37": "North Carolina", + "38": "North Dakota", + "39": "Ohio", + "40": "Oklahoma", + "41": "Oregon", + "42": "Pennsylvania", + "44": "Rhode Island", + "45": "South Carolina", + "46": "South Dakota", + "47": "Tennessee", + "48": "Texas", + "49": "Utah", + "50": "Vermont", + "51": "Virginia", + "53": "Washington", + "54": "West Virginia", + "55": "Wisconsin", + "56": "Wyoming", +} + +# High-population states that exceed CDC WONDER's row and memory limits when queried across 7 years. +# Querying directly in 2-year chunks prevents query buffer overruns and HTTP 400 errors. +LARGE_STATES: set[str] = { + "01", "05", "06", "12", "13", "17", "18", "21", "22", "26", "27", "28", "29", + "34", "36", "37", "39", "40", "42", "45", "47", "48", "51", "53", "55", +} + +# States with exceptionally large county counts (e.g. Texas with 254 counties) that frequently +# trigger HTTP 504 Gateway Timeouts when queried across multiple years during peak traffic hours. +SINGLE_YEAR_STATES: set[str] = {"48"} + +FLAGS = flags.FLAGS +CURRENT_YEAR = datetime.date.today().year + +flags.DEFINE_string( + "states", + "all", + "Comma-separated 2-digit FIPS codes of states to download (e.g. '02,48'), or 'all'.", +) +flags.DEFINE_string( + "years", + f"2018-{CURRENT_YEAR}", + f"Year range (e.g. '2018-{CURRENT_YEAR}') or comma-separated years ('2018,2019,2020').", +) +flags.DEFINE_string( + "output_dir", + DEFAULT_INPUT_DIR, + "Directory where downloaded CSV files will be saved.", +) +flags.DEFINE_float( + "delay", + 3.0, + "Politeness delay in seconds between successive HTTP queries.", +) +flags.DEFINE_integer( + "timeout", + 120, + "HTTP request timeout in seconds.", +) +flags.DEFINE_boolean( + "skip_existing", + True, + "Skip downloading states that already have existing non-empty CSV files in output_dir.", +) +flags.DEFINE_integer( + "batch_size", + 10, + "Number of states to process per session before automatically refreshing session.", +) +flags.DEFINE_float( + "batch_cooldown", + 20.0, + "Cooldown delay in seconds between session batches to prevent rate limits.", +) + + +def parse_year_list(year_str: str) -> List[str]: + """Parses a year string like '2018-2024' or '2018,2019' into a list of year strings.""" + year_str = year_str.strip() + if "-" in year_str and not year_str.startswith("-"): + parts = year_str.split("-") + start, end = int(parts[0]), int(parts[1]) + return [str(y) for y in range(start, end + 1)] + return [y.strip() for y in year_str.split(",") if y.strip()] + + +class CdcWonderCountyMortalityDownloader: + """Automates CDC WONDER sessions and queries for County-Level Mortality data.""" + + def __init__( + self, + landing_url: str = "https://wonder.cdc.gov/ucd-icd10-expanded.html", + timeout: int = 120, + delay: float = 3.0, + ): + self.landing_url = landing_url + self.timeout = timeout + self.delay = delay + self.session = self._create_session() + self.action_url: Optional[str] = None + self.base_post_data: List[Tuple[str, str]] = [] + self.available_years: List[str] = [] + + def _create_session(self) -> requests.Session: + """Creates a requests.Session with connection pooling and HTTP retries.""" + session = requests.Session() + session.headers.update({ + "User-Agent": ( + "Mozilla/5.0 (DataCommons CDC Importer; contact:" + " support@datacommons.org)" + ) + }) + retries = Retry( + total=3, + backoff_factor=1, + status_forcelist=[500, 502, 503, 504], + raise_on_status=False, + ) + adapter = HTTPAdapter(max_retries=retries) + session.mount("https://", adapter) + session.mount("http://", adapter) + return session + + @retry( + tries=3, + delay=5, + backoff=2, + exceptions=(requests.RequestException, ValueError), + ) + def init_session(self): + """Connects to landing page, agrees to terms, and stores pre-populated form state.""" + if hasattr(self, "session") and self.session: + try: + self.session.close() + except Exception: + pass + self.session = self._create_session() + self.action_url = None + self.base_post_data = [] + + logging.info("Connecting to CDC WONDER landing page: %s", self.landing_url) + res = self.session.get(self.landing_url, timeout=self.timeout) + res.raise_for_status() + + soup = BeautifulSoup(res.text, "lxml") + form = soup.find("form", id="wonderform") + if not form: + raise ValueError("Could not find initial wonderform on CDC WONDER page.") + + action = urljoin(self.landing_url, form.get("action")) + agree_inputs = [ + (inp.get("name"), inp.get("value", "")) + for inp in form.find_all("input") + if inp.get("name") + ] + agree_inputs.append(("action-I Agree", "I Agree")) + + logging.info("Submitting Data Use Agreement (I Agree)...") + res_agree = self.session.post(action, data=agree_inputs, timeout=self.timeout) + res_agree.raise_for_status() + + soup_req = BeautifulSoup(res_agree.text, "lxml") + form_req = soup_req.find("form", id="wonderform") + if not form_req: + raise ValueError("Could not find request form after agreeing to terms.") + + self.action_url = urljoin(self.landing_url, form_req.get("action")) + + # Extract pre-populated query parameters + self.base_post_data = [] + for el in form_req.find_all(["input", "select", "textarea"]): + name = el.get("name") + if not name: + continue + if el.name == "input": + itype = el.get("type", "text").lower() + if itype in ["submit", "button", "reset", "image"]: + continue + if itype in ["checkbox", "radio"]: + if el.has_attr("checked"): + self.base_post_data.append((name, el.get("value", "on"))) + else: + self.base_post_data.append((name, el.get("value", ""))) + elif el.name == "select": + selected_opts = [opt for opt in el.find_all("option") if opt.has_attr("selected")] + if selected_opts: + for opt in selected_opts: + self.base_post_data.append((name, opt.get("value", ""))) + else: + if not el.has_attr("multiple"): + first_opt = el.find("option") + if first_opt: + self.base_post_data.append((name, first_opt.get("value", ""))) + elif el.name == "textarea": + self.base_post_data.append((name, el.text or "")) + + # Discover available year options from the year selection dropdown (F_D158.V1) + self.available_years = [] + year_select = form_req.find("select", attrs={"name": "F_D158.V1"}) + if year_select: + for opt in year_select.find_all("option"): + val = opt.get("value", "").strip() + if val.isdigit(): + self.available_years.append(val) + if self.available_years: + logging.info( + "Discovered available years on CDC WONDER: %s", self.available_years + ) + + logging.info("Successfully established CDC WONDER session with action: %s", self.action_url) + + def filter_available_years(self, years: List[str]) -> List[str]: + """Filters requested years against available years discovered on CDC WONDER.""" + if not self.available_years: + return years + valid_years = [y for y in years if y in self.available_years] + if not valid_years: + logging.warning( + "None of the requested years %s exist on CDC WONDER (available: %s). " + "Falling back to all available years.", + years, + self.available_years, + ) + return list(self.available_years) + if len(valid_years) < len(years): + dropped = [y for y in years if y not in self.available_years] + logging.info( + "Filtered out unsupported years %s (not yet published on CDC WONDER). " + "Querying: %s", + dropped, + valid_years, + ) + return valid_years + + def _build_post_data( + self, state_fips: str, years: Optional[List[str]] = None + ) -> List[Tuple[str, str]]: + """Constructs query payload for county-level mortality across all causes.""" + query_data: List[Tuple[str, str]] = [] + for k, v in self.base_post_data: + # Grouping fields: + # B_1: Year + # B_2: County + # B_3: ICD-10 113 Cause List + # B_4: *None* + # B_5: *None* + if k == "B_1": + query_data.append((k, "D158.V1-level1")) + elif k == "B_2": + query_data.append((k, "D158.V9-level2")) + elif k == "B_3": + query_data.append((k, "D158.V4")) + elif k == "B_4": + query_data.append((k, "*None*")) + elif k == "B_5": + query_data.append((k, "*None*")) + elif k == "F_D158.V9": + # Filter by state FIPS code + query_data.append((k, state_fips)) + elif k == "F_D158.V1": + # Year filter - handled below + if not years: + query_data.append((k, v)) + else: + query_data.append((k, v)) + + if years: + for y in years: + query_data.append(("F_D158.V1", y)) + + query_data.append(("action-Export Results", "Export Results")) + return query_data + + def execute_query( + self, + state_fips: str, + years: Optional[List[str]] = None, + max_retries: int = 5, + ) -> str: + """Executes query with automatic 429 rate-limit backoff and session renewal.""" + if not self.action_url or not self.base_post_data: + self.init_session() + + payload = self._build_post_data(state_fips, years) + + for attempt in range(1, max_retries + 1): + try: + logging.info( + "Dispatching HTTP POST to %s for state FIPS %s, years %s (attempt %d/%d)...", + self.action_url, + state_fips, + years, + attempt, + max_retries, + ) + res = self.session.post( + self.action_url, data=payload, timeout=self.timeout + ) + + if res.status_code == 429: + retry_after = res.headers.get("Retry-After") + wait_seconds = ( + int(retry_after) + if retry_after and retry_after.isdigit() + else 1860 + ) + logging.warning( + "Encountered HTTP 429 (Too Many Requests). CDC WONDER enforces a 30-minute " + "IP block. Waiting %d seconds (%d min) in complete silence for block to " + "clear (attempt %d)...", + wait_seconds, + wait_seconds // 60, + attempt, + ) + time.sleep(wait_seconds) + logging.info("Block elapsed. Re-initializing new CDC WONDER session...") + self.init_session() + payload = self._build_post_data(state_fips, years) + continue + + if res.status_code in (401, 403) or ( + res.text + and ( + "session has expired" in res.text.lower() + or "session timeout" in res.text.lower() + ) + ): + logging.warning( + "Session expired or rejected for FIPS %s. Re-initializing session...", + state_fips, + ) + self.init_session() + payload = self._build_post_data(state_fips, years) + continue + + if res.status_code == 400 and ( + "too much data" in res.text or "simplify your filters" in res.text + ): + logging.warning( + "FIPS %s query returned 'too much data' (HTTP 400). " + "Returning response for partitioning.", + state_fips, + ) + return res.text + + res.raise_for_status() + return res.text + + except requests.RequestException as e: + logging.warning( + "Query failed for FIPS %s (attempt %d/%d): %s", + state_fips, + attempt, + max_retries, + e, + ) + if attempt == max_retries: + raise + time.sleep(self.delay * attempt) + + raise RuntimeError( + f"Failed to fetch data for state FIPS {state_fips} after {max_retries} retries." + ) + + def download_state( + self, state_fips: str, years: List[str] + ) -> List[Tuple[str, str]]: + """Downloads county mortality data for a state, automatically partitioning if needed.""" + state_name = US_STATES.get(state_fips, f"FIPS-{state_fips}") + if self.available_years: + years = self.filter_available_years(years) + results: List[Tuple[str, str]] = [] + need_partitioning = state_fips in LARGE_STATES + + if not need_partitioning: + logging.info( + "Querying full year range (%s) for state FIPS %s (%s)...", + years, + state_fips, + state_name, + ) + try: + tsv_text = self.execute_query(state_fips, years) + first_line = tsv_text.splitlines()[0] if tsv_text else "" + if "County Code" in first_line: + results.append(("all", tsv_text)) + time.sleep(self.delay) + return results + logging.warning( + "%s response not TSV (likely exceeded 75k rows). Partitioning into chunks...", + state_name, + ) + need_partitioning = True + except Exception as e: + logging.warning( + "Querying all %d years for %s encountered %s. Partitioning into year chunks...", + len(years), + state_name, + e, + ) + need_partitioning = True + + if need_partitioning: + logging.info("Partitioning %s (%s) into chunks...", state_name, state_fips) + use_single_years = (len(years) <= 2) or (state_fips in SINGLE_YEAR_STATES) + i = 0 + while i < len(years): + if not use_single_years and (i + 1 < len(years)): + year_chunk = years[i : i + 2] + chunk_label = f"{year_chunk[0]}_{year_chunk[-1]}" + logging.info("Querying chunk %s for %s...", chunk_label, state_name) + time.sleep(self.delay) + + chunk_tsv = "" + success = False + try: + chunk_tsv = self.execute_query(state_fips, year_chunk, max_retries=2) + chunk_first_line = chunk_tsv.splitlines()[0] if chunk_tsv else "" + if "County Code" in chunk_first_line: + success = True + else: + logging.warning( + "Chunk %s response for %s did not contain valid TSV data. " + "Splitting into single years.", + chunk_label, + state_name, + ) + except Exception as e: + logging.warning( + "Chunk %s for %s failed with %s. Falling back to single-year " + "queries for this state.", + chunk_label, + state_name, + e, + ) + + if success: + results.append((chunk_label, chunk_tsv)) + i += 2 + continue + else: + use_single_years = True + + single_year = years[i] + logging.info("Querying single year %s for %s...", single_year, state_name) + time.sleep(self.delay) + sy_text = self.execute_query(state_fips, [single_year]) + first_line = sy_text.splitlines()[0] if sy_text else "" + if "County Code" not in first_line: + raise ValueError( + f"Failed to query {state_name} even for single year {single_year}." + ) + results.append((single_year, sy_text)) + i += 1 + + return results + + +def save_tsv_as_csv(raw_tsv: str, output_csv_path: str) -> int: + """Parses raw CDC TSV export into clean CSV format, stripping footer caveats.""" + Path(os.path.dirname(output_csv_path)).mkdir(parents=True, exist_ok=True) + tsv_reader = csv.reader(io.StringIO(raw_tsv), delimiter="\t") + + temp_csv_path = f"{output_csv_path}.tmp" + row_count = 0 + with open(temp_csv_path, "w", newline="", encoding="utf-8") as f: + csv_writer = csv.writer(f) + for row in tsv_reader: + if not row: + continue + if row[0].startswith("---") or (len(row) > 1 and row[1].startswith("---")): + break + csv_writer.writerow(row) + row_count += 1 + + if row_count <= 1: + if os.path.exists(temp_csv_path): + os.unlink(temp_csv_path) + raise ValueError( + f"Extracted TSV for {output_csv_path} contains no observation data " + f"(row_count={row_count})." + ) + + os.replace(temp_csv_path, output_csv_path) + return row_count + + +def _has_data_rows(file_path: Path) -> bool: + """Checks whether a CSV file contains at least one observation data row beyond the header.""" + if not file_path.exists() or file_path.stat().st_size == 0: + return False + try: + with open(file_path, "r", encoding="utf-8", errors="replace") as f: + reader = csv.reader(f) + header = next(reader, None) + if not header: + return False + for row in reader: + if row and not row[0].startswith("---") and not row[0].startswith("Total"): + return True + return False + except Exception: + return False + + +def is_state_downloaded( + output_dir: str, state_fips: str, years: Optional[List[str]] = None +) -> bool: + """Checks if valid non-empty CSV files for this state already exist covering requested years.""" + pattern = f"UnderlyingCauseofDeath_County_{state_fips}*.csv" + matches = list(Path(output_dir).glob(pattern)) + if not matches: + return False + + # Check if a single combined file exists and contains data rows for all requested years + single_file = Path(output_dir) / f"UnderlyingCauseofDeath_County_{state_fips}.csv" + if single_file.exists(): + found_years = set() + try: + with open(single_file, "r", encoding="utf-8", errors="replace") as f: + reader = csv.reader(f) + header = next(reader, None) + if header: + year_col_idx = 1 + if "Year" in header: + year_col_idx = header.index("Year") + for row in reader: + if len(row) > year_col_idx: + year_val = row[year_col_idx].strip() + if years is None or year_val in years: + found_years.add(year_val) + if years is None and found_years: + return True + if years and len(found_years) == len(years): + return True + except Exception: + pass + + # If no specific years requested, check if any matching file has valid data rows + if not years: + return any(_has_data_rows(f) for f in matches) + + # Otherwise, check if every year in years is covered by at least one chunk file with data rows + for y in years: + year_covered = False + for f in matches: + if f.name == single_file.name: + continue + if (f"_{y}.csv" in f.name or f"_{y}_" in f.name) and _has_data_rows(f): + year_covered = True + break + if not year_covered: + return False + return True + + +def _should_skip_state( + output_dir: str, + state_fips: str, + years: List[str], + state_name: str, + idx: int, + total_states: int, +) -> bool: + """Checks if state files already exist on disk and logs skip message.""" + if not is_state_downloaded(output_dir, state_fips, years=years): + return False + existing_files = list( + Path(output_dir).glob( + f"UnderlyingCauseofDeath_County_{state_fips}*.csv" + ) + ) + logging.info( + "[%d/%d] Skipping %s (FIPS %s): %d existing file(s) found.", + idx, + total_states, + state_name, + state_fips, + len(existing_files), + ) + return True + + +def download_county_mortality_data( + states: List[str], + years: List[str], + output_dir: str, + delay: float = 3.0, + timeout: int = 120, + skip_existing: bool = True, + batch_size: int = 10, + batch_cooldown: float = 20.0, +): + """Downloads CDC County Mortality data for specified states and years.""" + os.makedirs(output_dir, exist_ok=True) + downloader = CdcWonderCountyMortalityDownloader(timeout=timeout, delay=delay) + + total_files = 0 + total_rows = 0 + states_in_batch = 0 + failed_states = [] + + for idx, state_fips in enumerate(states, start=1): + state_name = US_STATES.get(state_fips, f"FIPS-{state_fips}") + + if skip_existing and _should_skip_state( + output_dir, state_fips, years, state_name, idx, len(states) + ): + continue + + if downloader.action_url is None: + downloader.init_session() + years = downloader.filter_available_years(years) + if skip_existing and _should_skip_state( + output_dir, state_fips, years, state_name, idx, len(states) + ): + continue + + logging.info( + "[%d/%d] Processing %s (FIPS %s) (Session batch item %d/%d)...", + idx, + len(states), + state_name, + state_fips, + states_in_batch + 1, + batch_size, + ) + + # Remove any existing/stale chunk or combined files for this state before re-downloading + # to prevent duplicate ingestion by stat_var_processor (which globs input_files/*.csv). + for stale_file in Path(output_dir).glob(f"UnderlyingCauseofDeath_County_{state_fips}*.csv"): + try: + stale_file.unlink() + except OSError as e: + logging.warning("Could not remove stale file %s: %s", stale_file, e) + + try: + results = downloader.download_state(state_fips, years) + for chunk_label, tsv_data in results: + if chunk_label == "all": + filename = f"UnderlyingCauseofDeath_County_{state_fips}.csv" + else: + filename = f"UnderlyingCauseofDeath_County_{state_fips}_{chunk_label}.csv" + + filepath = os.path.join(output_dir, filename) + rows = save_tsv_as_csv(tsv_data, filepath) + total_files += 1 + total_rows += rows + logging.info("Saved %s with %d rows.", filename, rows) + + states_in_batch += 1 + + # Proactive session rotation after batch_size states + if states_in_batch >= batch_size and idx < len(states): + logging.info( + "Processed batch of %d states. Taking a %.1fs cooldown and refreshing " + "session...", + states_in_batch, + batch_cooldown, + ) + time.sleep(batch_cooldown) + downloader.init_session() + states_in_batch = 0 + + except Exception as e: + logging.error( + "Failed downloading state %s (FIPS %s): %s", + state_name, + state_fips, + e, + ) + failed_states.append(state_name) + + logging.info( + "Download complete: Saved %d files with %d total rows in %s.", + total_files, + total_rows, + output_dir, + ) + if failed_states: + raise RuntimeError( + f"Failed to download data for states: {', '.join(failed_states)}" + ) + + +def main(_): + years = parse_year_list(FLAGS.years) + + if FLAGS.states.lower() == "all": + states = sorted(list(US_STATES.keys())) + else: + states = [s.strip().zfill(2) for s in FLAGS.states.split(",") if s.strip()] + + logging.info( + "Starting CDC County Mortality live download for %d states, years: %s", + len(states), + years, + ) + download_county_mortality_data( + states=states, + years=years, + output_dir=FLAGS.output_dir, + delay=FLAGS.delay, + timeout=FLAGS.timeout, + skip_existing=FLAGS.skip_existing, + batch_size=FLAGS.batch_size, + batch_cooldown=FLAGS.batch_cooldown, + ) + + +if __name__ == "__main__": + app.run(main) diff --git a/statvar_imports/us_cdc/county_mortality/download_test.py b/statvar_imports/us_cdc/county_mortality/download_test.py new file mode 100644 index 0000000000..a600aec767 --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/download_test.py @@ -0,0 +1,398 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for CDC WONDER County Mortality Downloader.""" + +import os +from pathlib import Path +import sys +import tempfile +import unittest +from unittest import mock + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import requests + +import download + + +class DownloadTest(unittest.TestCase): + + def test_parse_year_list_range(self): + years = download.parse_year_list("2018-2024") + self.assertEqual(years, ["2018", "2019", "2020", "2021", "2022", "2023", "2024"]) + + def test_parse_year_list_comma(self): + years = download.parse_year_list("2018, 2020, 2022") + self.assertEqual(years, ["2018", "2020", "2022"]) + + def test_parse_year_list_single(self): + years = download.parse_year_list("2024") + self.assertEqual(years, ["2024"]) + + @mock.patch.object(download.requests, "Session") + def test_init_session_success(self, mock_session_cls): + mock_session = mock.MagicMock() + mock_session_cls.return_value = mock_session + + mock_res1 = mock.MagicMock() + mock_res1.text = """ + + +
+ +
+ + + """ + mock_res1.raise_for_status.return_value = None + + mock_res2 = mock.MagicMock() + mock_res2.text = """ + + +
+ + + +
+ + + """ + mock_res2.raise_for_status.return_value = None + + mock_session.get.return_value = mock_res1 + mock_session.post.return_value = mock_res2 + + downloader = download.CdcWonderCountyMortalityDownloader() + downloader.init_session() + + self.assertIn("controller/datarequest/D158;jsessionid=TEST1234", downloader.action_url) + self.assertTrue(len(downloader.base_post_data) > 0) + self.assertEqual(mock_session.post.call_count, 1) + + def test_build_post_data(self): + downloader = download.CdcWonderCountyMortalityDownloader() + downloader.base_post_data = [ + ("B_1", "old_val"), + ("B_2", "old_val"), + ("B_3", "old_val"), + ("B_4", "old_val"), + ("B_5", "old_val"), + ("F_D158.V9", "*All*"), + ("F_D158.V1", "*All*"), + ("other_key", "other_val"), + ] + + payload = downloader._build_post_data("10", ["2018", "2019"]) + payload_dict = dict(payload) + + # Check that County and ICD-10 113 Cause List are selected, while B_4/B_5 are *None* + self.assertEqual(payload_dict["B_1"], "D158.V1-level1") + self.assertEqual(payload_dict["B_2"], "D158.V9-level2") + self.assertEqual(payload_dict["B_3"], "D158.V4") + self.assertEqual(payload_dict["B_4"], "*None*") + self.assertEqual(payload_dict["B_5"], "*None*") + self.assertEqual(payload_dict["F_D158.V9"], "10") + self.assertEqual(payload_dict["action-Export Results"], "Export Results") + + year_params = [v for k, v in payload if k == "F_D158.V1"] + self.assertEqual(year_params, ["2018", "2019"]) + + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "execute_query") + def test_download_state_single_query(self, mock_query): + tsv_output = ( + "Notes\tYear\tCounty\tCounty Code\tICD-10 113 Cause List\tDeaths\n" + "\t2018\tKent County, DE\t10001\tSepticemia\t20\n" + ) + mock_query.return_value = tsv_output + + downloader = download.CdcWonderCountyMortalityDownloader() + results = downloader.download_state("10", ["2018", "2019"]) + + self.assertEqual(len(results), 1) + self.assertEqual(results[0][0], "all") + self.assertEqual(results[0][1], tsv_output) + mock_query.assert_called_once_with("10", ["2018", "2019"]) + + def test_save_tsv_as_csv(self): + raw_tsv = ( + "Notes\tYear\tCounty\tCounty Code\tDeaths\n" + "\t2018\tKent County, DE\t10001\t20\n" + "---\n" + "Query Parameters:\n" + "Caveats:\n" + ) + + with tempfile.TemporaryDirectory() as temp_dir: + output_csv = os.path.join(temp_dir, "test_output.csv") + rows = download.save_tsv_as_csv(raw_tsv, output_csv) + + self.assertEqual(rows, 2) + self.assertTrue(os.path.exists(output_csv)) + lines = Path(output_csv).read_text(encoding="utf-8").splitlines() + + self.assertEqual(len(lines), 2) + self.assertEqual(lines[0], "Notes,Year,County,County Code,Deaths") + self.assertEqual(lines[1], ',2018,"Kent County, DE",10001,20') + + def test_save_tsv_as_csv_empty_raises(self): + header_only_tsv = "Notes\tYear\tCounty\tCounty Code\tDeaths\n---\nCaveats:\n" + with tempfile.TemporaryDirectory() as temp_dir: + output_csv = os.path.join(temp_dir, "test_output.csv") + with self.assertRaises(ValueError): + download.save_tsv_as_csv(header_only_tsv, output_csv) + self.assertFalse(os.path.exists(output_csv)) + self.assertFalse(os.path.exists(f"{output_csv}.tmp")) + + def test_is_state_downloaded(self): + with tempfile.TemporaryDirectory() as temp_dir: + self.assertFalse(download.is_state_downloaded(temp_dir, "10")) + + # Create empty file + f = Path(temp_dir) / "UnderlyingCauseofDeath_County_10.csv" + f.write_text("") + self.assertFalse(download.is_state_downloaded(temp_dir, "10")) + + # Create file with only 2024 (missing 2018 initial year) -> should be False + f.write_text("Header,col1,col2,col3\n" + ",2024,val2,val3\n" * 10) + self.assertFalse(download.is_state_downloaded(temp_dir, "10", years=["2018", "2024"])) + + # Create file with both initial (2018) and latest (2024) -> should be True + f.write_text( + "Header,col1,col2,col3\n" + + ",2018,val2,val3\n" * 5 + + ",2024,val2,val3\n" * 5 + ) + self.assertTrue(download.is_state_downloaded(temp_dir, "10", years=["2018", "2024"])) + self.assertFalse(download.is_state_downloaded(temp_dir, "10", years=["2018", "2025"])) + + # Create file with only 2018 data, Deaths column equals 2024 (should not match) + f.write_text("Notes,Year,County,Deaths\n" + ",2018,Kent County,2024\n" * 5) + self.assertFalse(download.is_state_downloaded(temp_dir, "10", years=["2018", "2024"])) + + # Test partitioned chunk files + f.unlink() + chunk_2024 = Path(temp_dir) / "UnderlyingCauseofDeath_County_10_2024.csv" + chunk_2024.write_text("Header,col1\n" + "val1,val2\n" * 10) + # Only latest chunk present -> should be False + self.assertFalse(download.is_state_downloaded(temp_dir, "10", years=["2018", "2024"])) + + # Both initial chunk and latest chunk present -> should be True for 2018, 2024 + chunk_2018 = Path(temp_dir) / "UnderlyingCauseofDeath_County_10_2018_2019.csv" + chunk_2018.write_text("Header,col1\n" + "val1,val2\n" * 10) + self.assertTrue(download.is_state_downloaded(temp_dir, "10", years=["2018", "2024"])) + + # If intermediate year (e.g. 2021) is requested but missing chunk -> should be False + self.assertFalse( + download.is_state_downloaded( + temp_dir, "10", years=["2018", "2021", "2024"] + ) + ) + + # Add intermediate chunk covering 2021 -> should now be True + chunk_2020 = Path(temp_dir) / "UnderlyingCauseofDeath_County_10_2020_2021.csv" + chunk_2020.write_text("Header,col1\n" + "val1,val2\n" * 10) + self.assertTrue( + download.is_state_downloaded( + temp_dir, "10", years=["2018", "2021", "2024"] + ) + ) + + # If an incomplete single combined file also exists (e.g. from an aborted run), + # but chunk files cover all requested years, it should still return True + f = Path(temp_dir) / "UnderlyingCauseofDeath_County_10.csv" + f.write_text("Header,col1\n,2018,val1\n" * 5) # only contains 2018 + self.assertTrue( + download.is_state_downloaded( + temp_dir, "10", years=["2018", "2021", "2024"] + ) + ) + f.unlink() + + # Chunk file that is header-only (e.g. CDC header without data rows) -> should be False + chunk_2020.write_text("Notes,Year,Year Code,County,County Code,Deaths\n") + self.assertFalse( + download.is_state_downloaded( + temp_dir, "10", years=["2018", "2021", "2024"] + ) + ) + + @mock.patch.object(download.time, "sleep") + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "init_session") + def test_execute_query_429_backoff(self, mock_init, mock_sleep): + downloader = download.CdcWonderCountyMortalityDownloader() + downloader.action_url = "https://wonder.cdc.gov/test" + downloader.base_post_data = [("B_1", "test")] + + mock_res_429 = mock.MagicMock() + mock_res_429.status_code = 429 + mock_res_429.headers = {"Retry-After": "1"} + + mock_res_200 = mock.MagicMock() + mock_res_200.status_code = 200 + mock_res_200.text = "Notes\tCounty Code\n" + mock_res_200.raise_for_status.return_value = None + + downloader.session.post = mock.MagicMock(side_effect=[mock_res_429, mock_res_200]) + + result = downloader.execute_query("10", ["2018"], max_retries=2) + self.assertEqual(result, "Notes\tCounty Code\n") + self.assertEqual(downloader.session.post.call_count, 2) + mock_sleep.assert_called_with(1) + mock_init.assert_called_once() + + @mock.patch.object(download.time, "sleep") + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "execute_query") + def test_download_state_timeout_fallback_to_single_years(self, mock_query, mock_sleep): + def query_side_effect(state_fips, years, **kwargs): + if len(years) > 1: + raise requests.exceptions.HTTPError("504 Server Error: Gateway Time-out") + return f"Notes\tYear\tCounty Code\n\t{years[0]}\t01001\n" + + mock_query.side_effect = query_side_effect + downloader = download.CdcWonderCountyMortalityDownloader() + results = downloader.download_state("01", ["2018", "2019", "2020"]) + + self.assertEqual(len(results), 3) + self.assertEqual([r[0] for r in results], ["2018", "2019", "2020"]) + + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "init_session") + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "download_state") + def test_download_county_mortality_data_continues_and_raises_on_failure( + self, mock_download, mock_init + ): + with tempfile.TemporaryDirectory() as temp_dir: + def side_effect(state_fips, years): + if state_fips == "10": + raise requests.exceptions.ConnectionError("Connection dropped") + return [("all", "Notes\tCounty Code\tDeaths\n\t11001\t50\n")] + + mock_download.side_effect = side_effect + + with self.assertRaises(RuntimeError) as ctx: + download.download_county_mortality_data( + states=["10", "11"], + years=["2024"], + output_dir=temp_dir, + skip_existing=False, + ) + + # Assert error message contains the failed state name + self.assertIn("Delaware", str(ctx.exception)) + # Assert state 11 was still attempted and saved despite state 10 failure + state_11_csv = Path(temp_dir) / "UnderlyingCauseofDeath_County_11.csv" + self.assertTrue(state_11_csv.exists()) + + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "init_session") + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "download_state") + def test_download_county_mortality_data_lazy_session_skipped( + self, mock_download, mock_init + ): + with tempfile.TemporaryDirectory() as temp_dir: + # Pre-populate state 10 so it is skipped + f = Path(temp_dir) / "UnderlyingCauseofDeath_County_10.csv" + f.write_text("Notes,Year,County,Deaths\n" + ",2024,Kent County,10\n" * 5) + + download.download_county_mortality_data( + states=["10"], + years=["2024"], + output_dir=temp_dir, + skip_existing=True, + ) + + # Assert init_session was NOT called because all states were skipped + mock_init.assert_not_called() + mock_download.assert_not_called() + + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "init_session") + @mock.patch.object(download.CdcWonderCountyMortalityDownloader, "download_state") + def test_download_county_mortality_data_purges_stale_files( + self, mock_download, mock_init + ): + with tempfile.TemporaryDirectory() as temp_dir: + # Pre-create an old chunk file that should be purged when re-downloading + old_chunk = Path(temp_dir) / "UnderlyingCauseofDeath_County_10_2018_2019.csv" + old_chunk.write_text("Notes,Year,County,Deaths\n" + ",2018,Kent County,10\n" * 5) + self.assertTrue(old_chunk.exists()) + + mock_download.return_value = [("all", "Notes\tCounty Code\tDeaths\n\t10001\t25\n")] + + download.download_county_mortality_data( + states=["10"], + years=["2018", "2024"], + output_dir=temp_dir, + skip_existing=False, + ) + + # Assert the old chunk file was deleted and new combined file was written + self.assertFalse(old_chunk.exists()) + combined_file = Path(temp_dir) / "UnderlyingCauseofDeath_County_10.csv" + self.assertTrue(combined_file.exists()) + + def test_filter_available_years(self): + downloader = download.CdcWonderCountyMortalityDownloader() + downloader.available_years = ["2018", "2019", "2020", "2021", "2022", "2023", "2024"] + + # Filters out unreleased future years + requested = ["2018", "2019", "2020", "2021", "2022", "2023", "2024", "2025", "2026"] + filtered = downloader.filter_available_years(requested) + self.assertEqual( + filtered, + ["2018", "2019", "2020", "2021", "2022", "2023", "2024"], + ) + + # Subset passes through untouched + subset = ["2020", "2021"] + self.assertEqual(downloader.filter_available_years(subset), ["2020", "2021"]) + + # No available_years means no filtering + downloader.available_years = [] + self.assertEqual(downloader.filter_available_years(requested), requested) + + @mock.patch.object(download.requests, "Session") + def test_init_session_discovers_available_years(self, mock_session_cls): + mock_session = mock.MagicMock() + mock_session_cls.return_value = mock_session + + mock_res1 = mock.MagicMock() + mock_res1.text = '
' + mock_res1.raise_for_status.return_value = None + + mock_res2 = mock.MagicMock() + mock_res2.text = """ +
+ +
+ """ + mock_res2.raise_for_status.return_value = None + + mock_session.get.return_value = mock_res1 + mock_session.post.return_value = mock_res2 + + downloader = download.CdcWonderCountyMortalityDownloader() + downloader.init_session() + self.assertEqual(downloader.available_years, ["2018", "2019", "2020"]) + + +if __name__ == "__main__": + unittest.main() + + diff --git a/statvar_imports/us_cdc/county_mortality/manifest.json b/statvar_imports/us_cdc/county_mortality/manifest.json new file mode 100644 index 0000000000..2a3312039f --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/manifest.json @@ -0,0 +1,31 @@ +{ + "import_specifications": [ + { + "import_name": "CDC_Mortality_UnderlyingCause", + "curator_emails": [ + "support@datacommons.org" + ], + "provenance_url": "https://wonder.cdc.gov/ucd-icd10-expanded.html", + "provenance_description": "County-level mortality statistics for all causes of death (ICD-10 113 Cause List) across all US states from CDC WONDER.", + "scripts": [ + "download.py", + "../../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=input_files/*.csv --pv_map=county_mortality_pvmap.csv --config_file=county_mortality_metadata.csv --output_path=output/underlyingcauseofdeath_county --output_counters=counters/underlyingcauseofdeath_county_counters.csv" + ], + "import_inputs": [ + { + "template_mcf": "output/underlyingcauseofdeath_county.tmcf", + "cleaned_csv": "output/underlyingcauseofdeath_county.csv", + "node_mcf": "output/*.mcf" + } + ], + "source_files": [ + "input_files/*.csv", + "counters/*.csv", + "validation_config.json" + ], + "cron_schedule": "0 5 * * 5", + "validation_config_file": "validation_config.json" + } + ] +} + diff --git a/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_input.csv b/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_input.csv new file mode 100644 index 0000000000..504fe6f140 --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_input.csv @@ -0,0 +1,71 @@ +Notes,Year,Year Code,County,County Code,ICD-10 113 Cause List,ICD-10 113 Cause List Code,Deaths,Population,Crude Rate,Crude Rate Lower 95% Confidence Interval,Crude Rate Upper 95% Confidence Interval +,2023,2023,"Kent County, DE",10001,#Septicemia (A40-A41),GR113-010,26,189789,13.7,8.9,20.1 +,2023,2023,"Kent County, DE",10001,"Other and unspecified infectious and parasitic diseases and their sequelae (A00,A05,A20-A36,A42-A44,A48-A49,A54-A79,A81-A82,A85.0-A85.1,A85.8,A86-B04,B06-B09,B25-B49,B55-B99,U07.1)",GR113-018,42,189789,22.1,15.9,29.9 +,2023,2023,"Kent County, DE",10001,#Malignant neoplasms (C00-C97),GR113-019,407,189789,214.4,193.6,235.3 +,2023,2023,"Kent County, DE",10001,"Malignant neoplasms of colon, rectum and anus (C18-C21)",GR113-023,28,189789,14.8,9.8,21.3 +,2023,2023,"Kent County, DE",10001,Malignant neoplasms of liver and intrahepatic bile ducts (C22),GR113-024,18,189789,9.5,5.6,15.0 +,2023,2023,"Kent County, DE",10001,Malignant neoplasm of pancreas (C25),GR113-025,31,189789,16.3,11.1,23.2 +,2023,2023,"Kent County, DE",10001,"Malignant neoplasms of trachea, bronchus and lung (C33-C34)",GR113-027,100,189789,52.7,42.4,63.0 +,2023,2023,"Kent County, DE",10001,Malignant neoplasm of breast (C50),GR113-029,36,189789,19.0,13.3,26.3 +,2023,2023,"Kent County, DE",10001,Malignant neoplasm of prostate (C61),GR113-033,14,189789,7.4,4.0,12.4 +,2023,2023,"Kent County, DE",10001,Malignant neoplasm of bladder (C67),GR113-035,20,189789,10.5,6.4,16.3 +,2023,2023,"Kent County, DE",10001,"Malignant neoplasms of meninges, brain and other parts of central nervous system (C70-C72)",GR113-036,10,189789,5.3,2.5,9.7 +,2023,2023,"Kent County, DE",10001,"Malignant neoplasms of lymphoid, hematopoietic and related tissue (C81-C96)",GR113-037,38,189789,20.0,14.2,27.5 +,2023,2023,"Kent County, DE",10001,Non-Hodgkin lymphoma (C82-C85),GR113-039,15,189789,7.9,4.4,13.0 +,2023,2023,"Kent County, DE",10001,Leukemia (C91-C95),GR113-040,18,189789,9.5,5.6,15.0 +,2023,2023,"Kent County, DE",10001,"All other and unspecified malignant neoplasms (C17,C23-C24,C26-C31,C37-C41,C44-C49,C51-C52,C57-C60,C62-C63,C66,C68-C69,C73-C80,C97)",GR113-043,57,189789,30.0,22.7,38.9 +,2023,2023,"Kent County, DE",10001,"#In situ neoplasms, benign neoplasms and neoplasms of uncertain or unknown behavior (D00-D48)",GR113-044,12,189789,6.3,3.3,11.0 +,2023,2023,"Kent County, DE",10001,#Diabetes mellitus (E10-E14),GR113-046,78,189789,41.1,32.5,51.3 +,2023,2023,"Kent County, DE",10001,#Nutritional deficiencies (E40-E64),GR113-047,11,189789,5.8,2.9,10.4 +,2023,2023,"Kent County, DE",10001,Malnutrition (E40-E46),GR113-048,11,189789,5.8,2.9,10.4 +,2023,2023,"Kent County, DE",10001,#Parkinson disease (G20-G21),GR113-051,16,189789,8.4,4.8,13.7 +,2023,2023,"Kent County, DE",10001,#Alzheimer disease (G30),GR113-052,72,189789,37.9,29.7,47.8 +,2023,2023,"Kent County, DE",10001,Major cardiovascular diseases (I00-I78),GR113-053,612,189789,322.5,296.9,348.0 +,2023,2023,"Kent County, DE",10001,"#Diseases of heart (I00-I09,I11,I13,I20-I51)",GR113-054,416,189789,219.2,198.1,240.3 +,2023,2023,"Kent County, DE",10001,Hypertensive heart disease (I11),GR113-056,51,189789,26.9,20.0,35.3 +,2023,2023,"Kent County, DE",10001,Ischemic heart diseases (I20-I25),GR113-058,223,189789,117.5,102.1,132.9 +,2023,2023,"Kent County, DE",10001,Acute myocardial infarction (I21-I22),GR113-059,33,189789,17.4,12.0,24.4 +,2023,2023,"Kent County, DE",10001,"Other forms of chronic ischemic heart disease (I20,I25)",GR113-061,187,189789,98.5,84.4,112.7 +,2023,2023,"Kent County, DE",10001,"Atherosclerotic cardiovascular disease, so described (I25.0)",GR113-062,67,189789,35.3,27.4,44.8 +,2023,2023,"Kent County, DE",10001,"All other forms of chronic ischemic heart disease (I20,I25.1-I25.9)",GR113-063,120,189789,63.2,51.9,74.5 +,2023,2023,"Kent County, DE",10001,Other heart diseases (I26-I51),GR113-064,128,189789,67.4,55.8,79.1 +,2023,2023,"Kent County, DE",10001,Heart failure (I50),GR113-067,37,189789,19.5,13.7,26.9 +,2023,2023,"Kent County, DE",10001,"All other forms of heart disease (I26-I28,I34-I38,I42-I49,I51)",GR113-068,91,189789,47.9,38.6,58.9 +,2023,2023,"Kent County, DE",10001,"#Essential hypertension and hypertensive renal disease (I10,I12,I15)",GR113-069,17,189789,9.0,5.2,14.3 +,2023,2023,"Kent County, DE",10001,#Cerebrovascular diseases (I60-I69),GR113-070,167,189789,88.0,74.6,101.3 +,2023,2023,"Kent County, DE",10001,Other diseases of circulatory system (I71-I78),GR113-072,11,189789,5.8,2.9,10.4 +,2024 ,2024,"Kent County, DE",10001,#Septicemia (A40-A41),GR113-010,37,192690,19.2,13.5,26.5 +,2024 ,2024,"Kent County, DE",10001,"Other and unspecified infectious and parasitic diseases and their sequelae (A00,A05,A20-A36,A42-A44,A48-A49,A54-A79,A81-A82,A85.0-A85.1,A85.8,A86-B04,B06-B09,B25-B49,B55-B99,U07.1)",GR113-018,38,192690,19.7,14.0,27.1 +,2024 ,2024,"Kent County, DE",10001,#Malignant neoplasms (C00-C97),GR113-019,420,192690,218.0,197.1,238.8 +,2024 ,2024,"Kent County, DE",10001,"Malignant neoplasms of lip, oral cavity and pharynx (C00-C14)",GR113-020,12,192690,6.2,3.2,10.9 +,2024 ,2024,"Kent County, DE",10001,Malignant neoplasm of esophagus (C15),GR113-021,11,192690,5.7,2.8,10.2 +,2024 ,2024,"Kent County, DE",10001,"Malignant neoplasms of colon, rectum and anus (C18-C21)",GR113-023,32,192690,16.6,11.4,23.4 +,2024 ,2024,"Kent County, DE",10001,Malignant neoplasms of liver and intrahepatic bile ducts (C22),GR113-024,20,192690,10.4,6.3,16.0 +,2024 ,2024,"Kent County, DE",10001,Malignant neoplasm of pancreas (C25),GR113-025,38,192690,19.7,14.0,27.1 +,2024 ,2024,"Kent County, DE",10001,"Malignant neoplasms of trachea, bronchus and lung (C33-C34)",GR113-027,96,192690,49.8,40.4,60.8 +,2024 ,2024,"Kent County, DE",10001,Malignant neoplasm of breast (C50),GR113-029,29,192690,15.1,10.1,21.6 +,2024 ,2024,"Kent County, DE",10001,"Malignant neoplasms of corpus uteri and uterus, part unspecified (C54-C55)",GR113-031,13,192690,6.7,3.6,11.5 +,2024 ,2024,"Kent County, DE",10001,Malignant neoplasm of prostate (C61),GR113-033,12,192690,6.2,3.2,10.9 +,2024 ,2024,"Kent County, DE",10001,Malignant neoplasms of kidney and renal pelvis (C64-C65),GR113-034,11,192690,5.7,2.8,10.2 +,2024 ,2024,"Kent County, DE",10001,Malignant neoplasm of bladder (C67),GR113-035,12,192690,6.2,3.2,10.9 +,2024 ,2024,"Kent County, DE",10001,"Malignant neoplasms of meninges, brain and other parts of central nervous system (C70-C72)",GR113-036,11,192690,5.7,2.8,10.2 +,2024 ,2024,"Kent County, DE",10001,"Malignant neoplasms of lymphoid, hematopoietic and related tissue (C81-C96)",GR113-037,46,192690,23.9,17.5,31.8 +,2024 ,2024,"Kent County, DE",10001,Non-Hodgkin lymphoma (C82-C85),GR113-039,16,192690,8.3,4.7,13.5 +,2024 ,2024,"Kent County, DE",10001,Leukemia (C91-C95),GR113-040,19,192690,9.9,5.9,15.4 +,2024 ,2024,"Kent County, DE",10001,"Multiple myeloma and immunoproliferative neoplasms (C88,C90)",GR113-041,11,192690,5.7,2.8,10.2 +,2024 ,2024,"Kent County, DE",10001,"All other and unspecified malignant neoplasms (C17,C23-C24,C26-C31,C37-C41,C44-C49,C51-C52,C57-C60,C62-C63,C66,C68-C69,C73-C80,C97)",GR113-043,50,192690,25.9,19.3,34.2 +,2024 ,2024,"Kent County, DE",10001,"#In situ neoplasms, benign neoplasms and neoplasms of uncertain or unknown behavior (D00-D48)",GR113-044,14,192690,7.3,4.0,12.2 +,2024 ,2024,"Kent County, DE",10001,#Diabetes mellitus (E10-E14),GR113-046,75,192690,38.9,30.6,48.8 +,2024 ,2024,"Kent County, DE",10001,#Parkinson disease (G20-G21),GR113-051,30,192690,15.6,10.5,22.2 +,2024 ,2024,"Kent County, DE",10001,#Alzheimer disease (G30),GR113-052,65,192690,33.7,26.0,43.0 +,2024 ,2024,"Kent County, DE",10001,Major cardiovascular diseases (I00-I78),GR113-053,650,192690,337.3,311.4,363.3 +,2024 ,2024,"Kent County, DE",10001,"#Diseases of heart (I00-I09,I11,I13,I20-I51)",GR113-054,444,192690,230.4,209.0,251.9 +,2024 ,2024,"Kent County, DE",10001,Hypertensive heart disease (I11),GR113-056,44,192690,22.8,16.6,30.7 +,2024 ,2024,"Kent County, DE",10001,Ischemic heart diseases (I20-I25),GR113-058,245,192690,127.1,111.2,143.1 +,2024 ,2024,"Kent County, DE",10001,Acute myocardial infarction (I21-I22),GR113-059,34,192690,17.6,12.2,24.7 +,2024 ,2024,"Kent County, DE",10001,"Other forms of chronic ischemic heart disease (I20,I25)",GR113-061,207,192690,107.4,92.8,122.1 +,2024 ,2024,"Kent County, DE",10001,"Atherosclerotic cardiovascular disease, so described (I25.0)",GR113-062,64,192690,33.2,25.6,42.4 +,2024 ,2024,"Kent County, DE",10001,"All other forms of chronic ischemic heart disease (I20,I25.1-I25.9)",GR113-063,143,192690,74.2,62.0,86.4 +,2024 ,2024,"Kent County, DE",10001,Other heart diseases (I26-I51),GR113-064,144,192690,74.7,62.5,86.9 +,2024 ,2024,"Kent County, DE",10001,Heart failure (I50),GR113-067,45,192690,23.4,17.0,31.2 +,2024 ,2024,"Kent County, DE",10001,"All other forms of heart disease (I26-I28,I34-I38,I42-I49,I51)",GR113-068,96,192690,49.8,40.4,60.8 diff --git a/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_output.csv b/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_output.csv new file mode 100644 index 0000000000..5efdba34ed --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_output.csv @@ -0,0 +1,71 @@ +observationAbout,observationDate,variableMeasured,value +geoId/10001,2023,dcid:Count_MortalityEvent_Septicemia,26 +geoId/10001,2023,dcid:Count_MortalityEvent_OtherAndUnspecifiedInfectiousAndParasiticDiseasesAndTheirSequelae,42 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasms,407 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmsOfColonRectumAndAnus,28 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmsOfLiverAndIntrahepaticBileDucts,18 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmOfPancreas,31 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmsOfTracheaBronchusAndLung,100 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmOfBreast,36 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmOfProstate,14 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmOfBladder,20 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmsOfMeningesBrainAndOtherPartsOfCentralNervousSystem,10 +geoId/10001,2023,dcid:Count_MortalityEvent_MalignantNeoplasmsOfLymphoidHematopoieticAndRelatedTissue,38 +geoId/10001,2023,dcid:Count_MortalityEvent_Non-HodgkinsLymphoma,15 +geoId/10001,2023,dcid:Count_MortalityEvent_Leukemia,18 +geoId/10001,2023,dcid:Count_MortalityEvent_AllOtherAndUnspecifiedMalignantNeoplasms,57 +geoId/10001,2023,dcid:Count_MortalityEvent_InSituNeoplasmsBenignNeoplasmsAndNeoplasmsOfUncertainOrUnknownBehavior,12 +geoId/10001,2023,dcid:Count_MortalityEvent_DiabetesMellitus,78 +geoId/10001,2023,dcid:Count_MortalityEvent_NutritionalDeficiencies,11 +geoId/10001,2023,dcid:Count_MortalityEvent_Malnutrition,11 +geoId/10001,2023,dcid:Count_MortalityEvent_ParkinsonsDisease,16 +geoId/10001,2023,dcid:Count_MortalityEvent_AlzheimersDisease,72 +geoId/10001,2023,dcid:Count_MortalityEvent_MajorCardiovascularDiseases,612 +geoId/10001,2023,dcid:Count_MortalityEvent_DiseasesOfHeart,416 +geoId/10001,2023,dcid:Count_MortalityEvent_HypertensiveHeartDisease,51 +geoId/10001,2023,dcid:Count_MortalityEvent_IschemicHeartDiseases,223 +geoId/10001,2023,dcid:Count_MortalityEvent_AcuteMyocardialInfarction,33 +geoId/10001,2023,dcid:Count_MortalityEvent_OtherFormsOfChronicIschemicHeartDisease,187 +geoId/10001,2023,dcid:Count_MortalityEvent_AtheroscleroticCardiovascularDiseaseSoDescribed,67 +geoId/10001,2023,dcid:Count_MortalityEvent_AllOtherFormsOfChronicIschemicHeartDisease,120 +geoId/10001,2023,dcid:Count_MortalityEvent_OtherHeartDiseases,128 +geoId/10001,2023,dcid:Count_MortalityEvent_HeartFailure,37 +geoId/10001,2023,dcid:Count_MortalityEvent_AllOtherFormsOfHeartDisease,91 +geoId/10001,2023,dcid:Count_MortalityEvent_Essential(Primary)HypertensionAndHypertensiveRenalDisease,17 +geoId/10001,2023,dcid:Count_MortalityEvent_CerebrovascularDiseases,167 +geoId/10001,2023,dcid:Count_MortalityEvent_OtherDiseasesOfCirculatorySystem,11 +geoId/10001,2024,dcid:Count_MortalityEvent_Septicemia,37 +geoId/10001,2024,dcid:Count_MortalityEvent_OtherAndUnspecifiedInfectiousAndParasiticDiseasesAndTheirSequelae,38 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasms,420 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfLipOralCavityAndPharynx,12 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmOfEsophagus,11 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfColonRectumAndAnus,32 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfLiverAndIntrahepaticBileDucts,20 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmOfPancreas,38 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfTracheaBronchusAndLung,96 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmOfBreast,29 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfCorpusUteriAndUterusPartUnspecified,13 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmOfProstate,12 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfKidneyAndRenalPelvis,11 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmOfBladder,12 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfMeningesBrainAndOtherPartsOfCentralNervousSystem,11 +geoId/10001,2024,dcid:Count_MortalityEvent_MalignantNeoplasmsOfLymphoidHematopoieticAndRelatedTissue,46 +geoId/10001,2024,dcid:Count_MortalityEvent_Non-HodgkinsLymphoma,16 +geoId/10001,2024,dcid:Count_MortalityEvent_Leukemia,19 +geoId/10001,2024,dcid:Count_MortalityEvent_MultipleMyelomaAndImmunoproliferativeNeoplasms,11 +geoId/10001,2024,dcid:Count_MortalityEvent_AllOtherAndUnspecifiedMalignantNeoplasms,50 +geoId/10001,2024,dcid:Count_MortalityEvent_InSituNeoplasmsBenignNeoplasmsAndNeoplasmsOfUncertainOrUnknownBehavior,14 +geoId/10001,2024,dcid:Count_MortalityEvent_DiabetesMellitus,75 +geoId/10001,2024,dcid:Count_MortalityEvent_ParkinsonsDisease,30 +geoId/10001,2024,dcid:Count_MortalityEvent_AlzheimersDisease,65 +geoId/10001,2024,dcid:Count_MortalityEvent_MajorCardiovascularDiseases,650 +geoId/10001,2024,dcid:Count_MortalityEvent_DiseasesOfHeart,444 +geoId/10001,2024,dcid:Count_MortalityEvent_HypertensiveHeartDisease,44 +geoId/10001,2024,dcid:Count_MortalityEvent_IschemicHeartDiseases,245 +geoId/10001,2024,dcid:Count_MortalityEvent_AcuteMyocardialInfarction,34 +geoId/10001,2024,dcid:Count_MortalityEvent_OtherFormsOfChronicIschemicHeartDisease,207 +geoId/10001,2024,dcid:Count_MortalityEvent_AtheroscleroticCardiovascularDiseaseSoDescribed,64 +geoId/10001,2024,dcid:Count_MortalityEvent_AllOtherFormsOfChronicIschemicHeartDisease,143 +geoId/10001,2024,dcid:Count_MortalityEvent_OtherHeartDiseases,144 +geoId/10001,2024,dcid:Count_MortalityEvent_HeartFailure,45 +geoId/10001,2024,dcid:Count_MortalityEvent_AllOtherFormsOfHeartDisease,96 diff --git a/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_output.tmcf b/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_output.tmcf new file mode 100644 index 0000000000..684cf5999b --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/test_data/underlyingcauseofdeath_county_output.tmcf @@ -0,0 +1,7 @@ +Node: E:underlyingcauseofdeath_county_output->E0 +observationAbout: C:underlyingcauseofdeath_county_output->observationAbout +observationDate: C:underlyingcauseofdeath_county_output->observationDate +variableMeasured: C:underlyingcauseofdeath_county_output->variableMeasured +value: C:underlyingcauseofdeath_county_output->value +typeOf: dcs:StatVarObservation +observationPeriod: P1Y diff --git a/statvar_imports/us_cdc/county_mortality/validation_config.json b/statvar_imports/us_cdc/county_mortality/validation_config.json new file mode 100644 index 0000000000..08a3dec889 --- /dev/null +++ b/statvar_imports/us_cdc/county_mortality/validation_config.json @@ -0,0 +1,32 @@ +{ + "schema_version": "1.0", + "rules": [ + { + "rule_id": "check_deleted_records_percent", + "description": "Strictly enforce historical deletion average threshold of 0.1%", + "validator": "DELETED_RECORDS_PERCENT", + "params": { + "threshold": 0.1 + } + }, + { + "rule_id": "check_max_date_freshness", + "description": "Verify MaxDate is within allowable 3-year publication lag for all StatVars except rare historical exceptions", + "validator": "SQL_VALIDATOR", + "params": { + "query": "SELECT StatVar, MaxDate FROM stats WHERE StatVar NOT IN ('Count_MortalityEvent_ChronicGlomerulonephritisNephritisAndNephritisNotSpecifiedAsAcuteOrChronicAndRenalSclerosisUnspecified', 'Count_MortalityEvent_AccidentalDischargeOfFirearms')", + "condition": "CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 3)" + } + }, + { + "rule_id": "check_max_date_freshness_rare_causes", + "description": "Verify MaxDate is within allowable 4-year publication lag for rare historical exception causes", + "validator": "SQL_VALIDATOR", + "params": { + "query": "SELECT StatVar, MaxDate FROM stats WHERE StatVar IN ('Count_MortalityEvent_ChronicGlomerulonephritisNephritisAndNephritisNotSpecifiedAsAcuteOrChronicAndRenalSclerosisUnspecified', 'Count_MortalityEvent_AccidentalDischargeOfFirearms')", + "condition": "CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 4)" + } + } + ] +} +