CDC_Mortality_UnderlyingCause

statvar_imports/us_cdc/county_mortality/README.md

@@ -0,0 +1,83 @@
+### CDC WONDER County-Level Mortality Across All Causes (`CDC_Mortality_UnderlyingCause`)
+
+This import acquires and processes county-level mortality statistics across all causes of death (ICD-10 113 Cause List) for all 50 US States and Washington D.C. from the CDC WONDER database.
+
+- **Import Name**: `CDC_Mortality_UnderlyingCause`
+- **Source Database**: CDC WONDER Underlying Cause of Death (Database D158)
+- **Source URL**: `https://wonder.cdc.gov/ucd-icd10-expanded.html`
+- **Geographic Granularity**: County level (all ~3,143 US counties across all states)
+- **Temporal Coverage**: 2018 to 2024 (P1Y frequency)
+- **Cause Coverage**: All diseases in the ICD-10 113 Cause List (e.g. Septicemia, Diabetes Mellitus, Major Cardiovascular Diseases, Alzheimer's, Malignant Neoplasms, Respiratory Diseases, etc.)
+
+---
+
+### Workflow
+
+The import consists of two fully automated steps:
+
+#### Step 1: Download Source Data
+```bash
+python3 download.py
+```
+or with specific flags:
+```bash
+python3 download.py --states=all --years=2018-2024 --output_dir=input_files
+```
+The download script automatically handles CDC WONDER sessions, agreements, rate-limiting backoffs, and query partitioning.
+
+#### Step 2: Process Data into Cleaned SVObs and TMCF
+```bash
+python3 ../../../tools/statvar_importer/stat_var_processor.py \
+  --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
+  --input_data=input_files/*.csv \
+  --pv_map=county_mortality_pvmap.csv \
+  --config_file=county_mortality_metadata.csv \
+  --output_path=output/underlyingcauseofdeath_county \
+  --output_counters=counters/underlyingcauseofdeath_county_counters.csv
+```
+
+#### Output Artifacts
+* `output/underlyingcauseofdeath_county.csv`: Cleaned observations mapping FIPS (`geoId/{fips}`), year, StatVar (`Count_MortalityEvent_<Cause>`), death count, and unit.
+* `output/underlyingcauseofdeath_county.tmcf`: Template MCF mapping CSV columns to Data Commons Knowledge Graph entities.
+
+---
+
+### Operational Notes & Downloader Architecture
+
+* **Rate-Limit Handling (HTTP 429)**: CDC WONDER enforces a 30-minute IP block when query thresholds are exceeded. The downloader detects HTTP 429 responses, pauses in complete silence for 31 minutes (`1,860s`), and automatically refreshes the session before resuming. Do not terminate the process during this cooldown.
+* **Large State Partitioning**: High-population states (e.g. California `06`, Florida `12`, New York `36`, Texas `48`) exceed CDC WONDER's 75,000-row query cap when queried across all years at once. The downloader automatically queries these states in 2-year chunks (or 1-year chunks for Texas) to prevent HTTP 400 ("Too Much Data") and HTTP 504 timeouts.
+* **Stale Partition Cleanup**: When a state is re-downloaded, the downloader purges existing chunk and combined files for that state before writing new data, preventing duplicate observations when `stat_var_processor.py` processes `input_files/*.csv`.
+* **Failure Isolation**: If an individual state query encounters network or server errors, the downloader logs the failure, continues with the remaining states, and raises a summary exception at the end of the batch run to ensure non-zero exit code while preserving downloaded progress.
+* **Lazy Session Initialization**: The session agreement with CDC WONDER is only initialized when at least one state actually requires downloading, avoiding unnecessary network calls during dry runs or when data is already cached.
+
+---
+
+### Validation
+
+Validation is configured in `validation_config.json`:
+* `check_deleted_records_percent`: Strictly enforces a historical deletion average threshold of `0.1%`. Note that on the initial import run, this rule expectedly yields `DATA_ERROR` ("Differ summary is missing required field: 'previous_obs_count'") because there is no prior version to diff against; it is configured to safeguard future recurring refreshes. Per consensus on initial imports, golden regression files are omitted from initial submission.
+* `check_max_date_freshness`: Enforces date freshness (`CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 3)`) across 116 cause-of-death StatVars to ensure latest published data is ingested relative to annual publication lag.
+* `check_max_date_freshness_rare_causes`: Enforces date freshness (`CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 4)`) for 2 rare historical exception causes with low observation frequency due to CDC sub-national cell suppression (< 10 deaths).
+
+---
+
+### Important Files
+
+| File | Description |
+|---|---|
+| `download.py` | Automated live CDC WONDER downloader, session manager, and query partitioner. |
+| `download_test.py` | Comprehensive unit test suite for download session management and error handling. |
+| `county_mortality_metadata.csv` | Metadata specifying header row offsets, frequency, and output columns. |
+| `county_mortality_pvmap.csv` | Property-Value mapping resolving county FIPS and ICD-10 113 causes of death. |
+| `manifest.json` | Automation manifest declaring scripts, inputs/outputs, cron schedule, and validation config. |
+| `validation_config.json` | Configuration defining import validation rules (historical deletion and date freshness). |
+| `test_data/` | Trimmed sample Delaware dataset and expected outputs for offline verification. |
+
+---
+
+### Testing
+
+Run the unit test suite from this directory:
+```bash
+python3 -m unittest discover -v -s . -p "*_test.py"
+```

statvar_imports/us_cdc/county_mortality/county_mortality_metadata.csv

@@ -0,0 +1,11 @@
+parameter,value
+url,https://wonder.cdc.gov/controller/datarequest/D158
+description,"County-level mortality statistics across all causes of death (ICD-10 113 Cause List) for all US states"
+#place_type,county level
+start_date,2018
+end_date,2024
+release_frequency,P1Y
+output_columns,"observationAbout,observationDate,variableMeasured,value"
+mapped_rows,1
+mapped_columns,11
+header_rows,1

statvar_imports/us_cdc/county_mortality/county_mortality_pvmap.csv

@@ -0,0 +1,143 @@
+key,p1,v1,p2,v2
+County Code,#Format,CountyCode1={Number:0>5},observationAbout,geoId/{CountyCode1}
+Year Code,observationDate,{Number},observationPeriod,P1Y
+Deaths,populationType,MortalityEvent,value,{Number}
+Population,#ignore,ignore,value,{Number}
+Crude Rate,#ignore,ignore,value,{Number}
+GR113-001,causeOfDeath,SalmonellaInfections,,
+GR113-003,causeOfDeath,CertainOtherIntestinalInfections,,
+GR113-004,causeOfDeath,Tuberculosis,,
+GR113-005,causeOfDeath,RespiratoryTuberculosis,,
+GR113-006,causeOfDeath,OtherTuberculosis,,
+GR113-009,causeOfDeath,MeningococcalInfection,,
+GR113-010,causeOfDeath,Septicemia,,
+GR113-015,causeOfDeath,ViralHepatitis,,
+GR113-016,causeOfDeath,HumanImmunodeficiencyVirus(Hiv)Disease,,
+GR113-018,causeOfDeath,OtherAndUnspecifiedInfectiousAndParasiticDiseasesAndTheirSequelae,,
+GR113-019,causeOfDeath,MalignantNeoplasms,,
+GR113-020,causeOfDeath,MalignantNeoplasmsOfLip_OralCavityAndPharynx,,
+GR113-021,causeOfDeath,MalignantNeoplasmOfEsophagus,,
+GR113-022,causeOfDeath,MalignantNeoplasmOfStomach,,
+GR113-023,causeOfDeath,MalignantNeoplasmsOfColon_RectumAndAnus,,
+GR113-024,causeOfDeath,MalignantNeoplasmsOfLiverAndIntrahepaticBileDucts,,
+GR113-025,causeOfDeath,MalignantNeoplasmOfPancreas,,
+GR113-026,causeOfDeath,MalignantNeoplasmOfLarynx,,
+GR113-027,causeOfDeath,MalignantNeoplasmsOfTrachea_BronchusAndLung,,
+GR113-028,causeOfDeath,MalignantMelanomaOfSkin,,
+GR113-029,causeOfDeath,MalignantNeoplasmOfBreast,,
+GR113-030,causeOfDeath,MalignantNeoplasmOfCervixUteri,,
+GR113-031,causeOfDeath,MalignantNeoplasmsOfCorpusUteriAndUterus_PartUnspecified,,
+GR113-032,causeOfDeath,MalignantNeoplasmOfOvary,,
+GR113-033,causeOfDeath,MalignantNeoplasmOfProstate,,
+GR113-034,causeOfDeath,MalignantNeoplasmsOfKidneyAndRenalPelvis,,
+GR113-035,causeOfDeath,MalignantNeoplasmOfBladder,,
+GR113-036,causeOfDeath,MalignantNeoplasmsOfMeninges_BrainAndOtherPartsOfCentralNervousSystem,,
+GR113-037,causeOfDeath,MalignantNeoplasmsOfLymphoid_HematopoieticAndRelatedTissue,,
+GR113-038,causeOfDeath,HodgkinsDisease,,
+GR113-039,causeOfDeath,Non-HodgkinsLymphoma,,
+GR113-040,causeOfDeath,Leukemia,,
+GR113-041,causeOfDeath,MultipleMyelomaAndImmunoproliferativeNeoplasms,,
+GR113-042,causeOfDeath,OtherAndUnspecifiedMalignantNeoplasmsOfLymphoid_HematopoieticAndRelatedTissue,,
+GR113-043,causeOfDeath,AllOtherAndUnspecifiedMalignantNeoplasms,,
+GR113-044,causeOfDeath,InSituNeoplasms_BenignNeoplasmsAndNeoplasmsOfUncertainOrUnknownBehavior,,
+GR113-045,causeOfDeath,Anemias,,
+GR113-046,causeOfDeath,DiabetesMellitus,,
+GR113-047,causeOfDeath,NutritionalDeficiencies,,
+GR113-048,causeOfDeath,Malnutrition,,
+GR113-049,causeOfDeath,OtherNutritionalDeficiencies,,
+GR113-050,causeOfDeath,Meningitis,,
+GR113-051,causeOfDeath,ParkinsonsDisease,,
+GR113-052,causeOfDeath,AlzheimersDisease,,
+GR113-053,causeOfDeath,MajorCardiovascularDiseases,,
+GR113-054,causeOfDeath,DiseasesOfHeart,,
+GR113-055,causeOfDeath,AcuteRheumaticFeverAndChronicRheumaticHeartDiseases,,
+GR113-056,causeOfDeath,HypertensiveHeartDisease,,
+GR113-057,causeOfDeath,HypertensiveHeartAndRenalDisease,,
+GR113-058,causeOfDeath,IschemicHeartDiseases,,
+GR113-059,causeOfDeath,AcuteMyocardialInfarction,,
+GR113-060,causeOfDeath,OtherAcuteIschemicHeartDiseases,,
+GR113-061,causeOfDeath,OtherFormsOfChronicIschemicHeartDisease,,
+GR113-062,causeOfDeath,AtheroscleroticCardiovascularDisease_SoDescribed,,
+GR113-063,causeOfDeath,AllOtherFormsOfChronicIschemicHeartDisease,,
+GR113-064,causeOfDeath,OtherHeartDiseases,,
+GR113-065,causeOfDeath,AcuteAndSubacuteEndocarditis,,
+GR113-066,causeOfDeath,DiseasesOfPericardiumAndAcuteMyocarditis,,
+GR113-067,causeOfDeath,HeartFailure,,
+GR113-068,causeOfDeath,AllOtherFormsOfHeartDisease,,
+GR113-069,causeOfDeath,Essential(Primary)HypertensionAndHypertensiveRenalDisease,,
+GR113-070,causeOfDeath,CerebrovascularDiseases,,
+GR113-071,causeOfDeath,Atherosclerosis,,
+GR113-072,causeOfDeath,OtherDiseasesOfCirculatorySystem,,
+GR113-073,causeOfDeath,AorticAneurysmAndDissection,,
+GR113-074,causeOfDeath,OtherDiseasesOfArteries_ArteriolesAndCapillaries,,
+GR113-075,causeOfDeath,OtherDisordersOfCirculatorySystem,,
+GR113-076,causeOfDeath,InfluenzaAndPneumonia,,
+GR113-077,causeOfDeath,Influenza,,
+GR113-078,causeOfDeath,Pneumonia,,
+GR113-079,causeOfDeath,OtherAcuteLowerRespiratoryInfections,,
+GR113-080,causeOfDeath,AcuteBronchitisAndBronchiolitis,,
+GR113-081,causeOfDeath,UnspecifiedAcuteLowerRespiratoryInfection,,
+GR113-082,causeOfDeath,ChronicLowerRespiratoryDiseases,,
+GR113-083,causeOfDeath,Bronchitis_ChronicAndUnspecified,,
+GR113-084,causeOfDeath,Emphysema,,
+GR113-085,causeOfDeath,Asthma,,
+GR113-086,causeOfDeath,OtherChronicLowerRespiratoryDiseases,,
+GR113-087,causeOfDeath,PneumoconiosesAndChemicalEffects,,
+GR113-088,causeOfDeath,PneumonitisDueToSolidsAndLiquids,,
+GR113-089,causeOfDeath,OtherDiseasesOfRespiratorySystem,,
+GR113-090,causeOfDeath,PepticUlcer,,
+GR113-091,causeOfDeath,DiseasesOfAppendix,,
+GR113-092,causeOfDeath,Hernia,,
+GR113-093,causeOfDeath,ChronicLiverDiseaseAndCirrhosis,,
+GR113-094,causeOfDeath,AlcoholicLiverDisease,,
+GR113-095,causeOfDeath,OtherChronicLiverDiseaseAndCirrhosis,,
+GR113-096,causeOfDeath,CholelithiasisAndOtherDisordersOfGallbladder,,
+GR113-097,causeOfDeath,Nephritis_NephroticSyndromeAndNephrosis,,
+GR113-098,causeOfDeath,AcuteAndRapidlyProgressiveNephriticAndNephroticSyndrome,,
+GR113-099,causeOfDeath,ChronicGlomerulonephritis_NephritisAndNephritisNotSpecifiedAsAcuteOrChronic_AndRenalSclerosisUnspecified,,
+GR113-100,causeOfDeath,RenalFailure,,
+GR113-102,causeOfDeath,InfectionsOfKidney,,
+GR113-103,causeOfDeath,HyperplasiaOfProstate,,
+GR113-104,causeOfDeath,InflammatoryDiseasesOfFemalePelvicOrgans,,
+GR113-105,causeOfDeath,Pregnancy_ChildbirthAndThePuerperium,,
+GR113-106,causeOfDeath,PregnancyWithAbortiveOutcome,,
+GR113-107,causeOfDeath,OtherComplicationsOfPregnancy_ChildbirthAndThePuerperium,,
+GR113-108,causeOfDeath,CertainConditionsOriginatingInThePerinatalPeriod,,
+GR113-109,causeOfDeath,CongenitalMalformations_DeformationsAndChromosomalAbnormalities,,
+GR113-110,causeOfDeath,Symptoms_SignsAndAbnormalClinicalAndLaboratoryFindings_NotElsewhereClassified,,
+GR113-112,causeOfDeath,Accidents(UnintentionalInjuries),,
+GR113-113,causeOfDeath,TransportAccidents,,
+GR113-114,causeOfDeath,MotorVehicleAccidents,,
+GR113-115,causeOfDeath,OtherLandTransportAccidents,,
+GR113-116,causeOfDeath,Water_AirAndSpace_AndOtherAndUnspecifiedTransportAccidentsAndTheirSequelae,,
+GR113-117,causeOfDeath,NontransportAccidents,,
+GR113-118,causeOfDeath,Falls,,
+GR113-119,causeOfDeath,AccidentalDischargeOfFirearms,,
+GR113-120,causeOfDeath,AccidentalDrowningAndSubmersion,,
+GR113-121,causeOfDeath,AccidentalExposureToSmoke_FireAndFlames,,
+GR113-122,causeOfDeath,AccidentalPoisoningAndExposureToNoxiousSubstances,,
+GR113-123,causeOfDeath,OtherAndUnspecifiedNontransportAccidentsAndTheirSequelae,,
+GR113-124,causeOfDeath,Suicide,,
+GR113-125,causeOfDeath,IntentionalSelf-Harm(Suicide)ByDischargeOfFirearms,,
+GR113-126,causeOfDeath,IntentionalSelf-Harm(Suicide)ByOtherAndUnspecifiedMeansAndTheirSequelae,,
+GR113-127,causeOfDeath,Assault(Homicide),,
+GR113-128,causeOfDeath,Assault(Homicide)ByDischargeOfFirearms,,
+GR113-129,causeOfDeath,Assault(Homicide)ByOtherAndUnspecifiedMeansAndTheirSequelae,,
+GR113-130,causeOfDeath,LegalIntervention,,
+GR113-131,causeOfDeath,EventsOfUndeterminedIntent,,
+GR113-132,causeOfDeath,DischargeOfFirearms_UndeterminedIntent,,
+GR113-133,causeOfDeath,OtherAndUnspecifiedEventsOfUndeterminedIntentAndTheirSequelae,,
+GR113-135,causeOfDeath,ComplicationsOfMedicalAndSurgicalCare,,
+GR113-136,causeOfDeath,EnterocolitisDueToClostridiumDifficile,,
+GR113-137,causeOfDeath,COVID_19,,
+GR113-111,causeOfDeath,AllOtherDiseases(Residual),,
+GR113-013,causeOfDeath,Arthropod-BorneViralEncephalitis,,
+GR113-017,causeOfDeath,Malaria,,
+GR113-134,causeOfDeath,OperationsOfWarAndTheirSequelae,,
+GR113-101,causeOfDeath,OtherDisordersOfKidney,,
+GR113-002,causeOfDeath,ShigellosisAndAmebiasis,,
+GR113-011,causeOfDeath,Syphilis,,
+GR113-007,causeOfDeath,WhoopingCough,,
+GR113-008,causeOfDeath,ScarletFeverAndErysipelas,,
+GR113-012,causeOfDeath,AcutePoliomyelitis,,
+GR113-014,causeOfDeath,Measles,,