Update CDC500_states query

scripts/us_cdc/cdc500_state/README.md

@@ -5,37 +5,99 @@ Author: Padma Gundapaneni @padma-g
 ## Table of Contents
 1. [About the Dataset](#about-the-dataset)
     1. [Overview](#overview)
+    2. [Data Sources and Tables](#data-sources-and-tables)
+    3. [Aggregation Methodology](#aggregation-methodology)
 2. [About the Import](#about-the-import)
     1. [Artifacts](#artifacts)
     2. [Import Procedure](#import-procedure)
 
 ## About the Dataset
 
 ### Overview
-The state level data is aggragated from city level data coming from CDC500 import. 
+The state-level dataset calculates aggregated health indicator prevalence estimates for US states from the city-level CDC 500 Cities (`CDC500`) project data, weighted by corresponding Census ACS 5-Year population counts.
 
-To get the data for this import run:
-```bash
-$ python3 process.py
-```
+### Data Sources and Tables
+
+The aggregation script queries Google Cloud BigQuery graph tables in dataset `datcom-store.spanner_dc_graph_prod_DEFAULT`:
+
+1. **`TimeSeries`**:
+   - **CDC 500 Series**: Identifies CDC 500 Statistical Variables (`provenance = 'dc/base/CDC500'` and `variable_measured LIKE 'Percent_%'`) and extracts their measurement methods (`measurement_method`). It maps each percentage health metric to its appropriate denominator demographic cohort StatVar (e.g., `Count_Person_18OrMoreYears`, `Count_Person_18To64Years`, `Count_Person_65OrMoreYears`, `Count_Person`, etc.).
+   - **Census ACS 5-Year Series**: Filters and joins population counts from Census ACS 5-Year Survey (`provenance = 'dc/base/CensusACS5YearSurvey'`).
+
+2. **`Observation`**:
+   - **Health Indicator Percentages**: Fetches city-level percentage values (`value AS percent`), observation dates (`date`), and city geoIds (`entity1 LIKE 'geoId/%' AND LENGTH(entity1) = 13`) for CDC 500 StatVars.
+   - **City Cohort Populations**: Fetches city-level population counts (`value AS population`) for the corresponding demographic cohort StatVars.
+
+### Aggregation Methodology
+
+For each state, indicator StatVar, and observation date:
+- City observations are joined with their corresponding demographic population counts.
+- City geoIds (`geoId/XXXXXXX`) are mapped to state geoIds (`geoId/XX`) using the first 8 characters (including the prefix).
+- State-level prevalence percentages are computed as a population-weighted average:
+
+$$\text{State Percent} = \frac{\sum (\text{City Population} \times \text{City Percent})}{\sum \text{City Population}}$$
+
+The output measurement method is prefixed with `dcAggregate/` (e.g., `dcAggregate/CrudePrevalence`).
+
+#### Excluded Indicators
+
+The following age-bracketed cancer screening indicators are omitted from state-level aggregation:
+- `Percent_Person_50To74Years_Female_ReceivedMammography`
+- `Percent_Person_21To65Years_Female_ReceivedCervicalCancerScreening`
+- `Percent_Person_21To65Years_Female_ReceivedPapSmearTest`
+- `Percent_Person_50To75Years_ReceivedColorectalCancerScreening`
+
+**Rationale**: The Census ACS 5-Year Survey does not publish single composite population StatVars for these non-standard multi-year age brackets (`50To74Years`, `21To65Years`, `50To75Years`). Rather than applying arbitrary proxy weights or risking silent row omission, these indicators are explicitly excluded from state aggregation.
 
 ## About the Import
 
 ### Artifacts
 
 #### Scripts
-[`process.py`](https://github.com/datacommonsorg/data/blob/master//scripts/us_cdc/cdc500_state/process.py)
+[`process.py`](https://github.com/datacommonsorg/data/blob/master/scripts/us_cdc/cdc500_state/process.py)
 
+#### Unit Tests
+[`process_test.py`](https://github.com/datacommonsorg/data/blob/master/scripts/us_cdc/cdc500_state/process_test.py)
 
-#### tMCFs
+#### tMCF Template
 [`cdc500_state.tmcf`](https://github.com/datacommonsorg/data/blob/master/scripts/us_cdc/cdc500_state/cdc500_state.tmcf)
 
+#### Validation Config
+[`validation_config.json`](https://github.com/datacommonsorg/data/blob/master/scripts/us_cdc/cdc500_state/validation_config.json)
+
 ### Import Procedure
 
-#### Data Download and Processing Steps
+#### Prerequisites
+
+Ensure Google Cloud authentication is configured with access to BigQuery dataset `datcom-store.spanner_dc_graph_prod_DEFAULT`:
+
+```bash
+$ gcloud auth application-default login
+```
+
+#### Running the Script
+
+To run the BigQuery aggregation and write the output CSV to the default output directory (`CDC500State_Output/CDC500State_Output.csv`):
 
-To get the data for this import run:
+```bash
+$ python3 scripts/us_cdc/cdc500_state/process.py
+```
+
+To specify a custom output directory:
 
 ```bash
-$ python3 process.py
+$ python3 scripts/us_cdc/cdc500_state/process.py --output_dir=/path/to/output
 ```
+
+#### Running Unit Tests
+
+Run the test suite using Python's `unittest` runner from the repository root:
+
+```bash
+$ python3 -m unittest scripts.us_cdc.cdc500_state.process_test
+```
+
+#### Automation
+
+This import is automated via Data Commons Import Automation and scheduled to run weekly via Cloud Batch every Monday at 01:00 UTC (`cron_schedule: "0 1 * * 1"` in `manifest.json`).
+

scripts/us_cdc/cdc500_state/cdc500_state.tmcf

@@ -4,5 +4,7 @@ variableMeasured: C:CDC->statvar
 observationAbout: C:CDC->observation_about
 observationDate: C:CDC->observation_date
 value: C:CDC->percent
+unit: Percent
+scalingFactor: 100
 measurementMethod: C:CDC->measurement_method
-observationPeriod: "P1Y"
+observationPeriod: "P1Y"

scripts/us_cdc/cdc500_state/manifest.json

@@ -21,10 +21,8 @@
                 "memory": 64,
                 "disk": 100
             },
-            "source_files": [
-                "CDC500State_Output/CDC500State_Output.csv"
-            ],
-            "cron_schedule": "0 1 * * 1"
+            "cron_schedule": "0 1 * * 1",
+            "validation_config_file": "validation_config.json"
         }
     ]
-}
+}

scripts/us_cdc/cdc500_state/process.py

@@ -1,4 +1,4 @@
-# Copyright 2021 Google LLC
+# Copyright 2026 Google LLC
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -11,75 +11,149 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""Processes CDC 500 cities data into aggregated state-level health indicators."""
 
 import os
+from absl import app
+from absl import flags
 from absl import logging
 from google.cloud import bigquery
 
+_FLAGS = flags.FLAGS
 _MODULE_DIR = os.path.dirname(os.path.abspath(__file__))
-_OUTPUT_FILE_PATH = os.path.join(_MODULE_DIR + '/CDC500State_Output')
-if not os.path.exists(_OUTPUT_FILE_PATH):
-    os.mkdir(_OUTPUT_FILE_PATH)
+_DEFAULT_OUTPUT_DIR = os.path.join(_MODULE_DIR, 'CDC500State_Output')
 
-query = """
-SELECT distinct * from(
-SELECT 
-  statvar, 
-  SUBSTR(observation_about,0,8) as observation_about,
-  observation_date, 
-  CONCAT('dcAggregate/',measurement_method) as measurement_method,
-  population_statvar,
-  SUM(CAST(pop_count AS FLOAT64))*100/SUM(CAST(population AS FLOAT64)) as percent
-FROM 
-(
+flags.DEFINE_string('output_dir', _DEFAULT_OUTPUT_DIR,
+                    'Directory to write output CSV.')
+
+QUERY = """
+WITH cdc_sv AS (
   SELECT
-    SVO1.variable_measured as statvar,
-    SVO1.observation_about as observation_about,
-    SVO1.observation_date as observation_date,
-    SVO1.value as percent,
-    SVO1.measurement_method as measurement_method,
-    SVO2.variable_measured as population_statvar,
-    SVO2.value as population,
-    CAST(SVO2.value AS FLOAT64) * CAST(SVO1.value AS FLOAT64) / 100 as pop_count
-  FROM `datcom-store.dc_kg_latest.StatVarObservation` as SVO1
-  JOIN `datcom-store.dc_kg_latest.StatVarObservation` as SVO2 ON TRUE
-  JOIN (
-    # Get the statvars and corresponding population statvar
-    # with ‘Percent_’ replaced with ‘Count_’ and
-    # dropping the non-age, non-gender constraints.
-    SELECT
-      SVO.variable_measured as CDC500,
-      CONCAT('Count_', REGEXP_SUBSTR(SVO.variable_measured, '(Person_.*ale|Person_.*Years|Person)')) as pop_statvar
-      FROM `datcom-store.dc_kg_latest.StatVarObservation` as SVO
-    WHERE
-      SVO.prov_id = 'dc/base/CDC500'
-      AND SVO.variable_measured like 'Percent_%'
-      GROUP BY CDC500, pop_statvar
-   ) AS CDC_SV ON TRUE
-  WHERE
-    SVO1.prov_id = 'dc/base/CDC500'
-    AND SVO1.variable_measured LIKE 'Percent%'
-    AND SVO1.observation_about = SVO2.observation_about
-    AND SVO1.observation_date = SVO2.observation_date
-    AND SVO1.variable_measured = CDC_SV.CDC500
-    AND SVO2.variable_measured = CDC_SV.pop_statvar
-    AND SVO1.observation_about like "geoId/%"
-) group by 1,2,3,4,5
+    variable_measured AS cdc500,
+    CASE
+      WHEN REGEXP_CONTAINS(variable_measured, r'65OrMoreYears.*Female|Female.*65OrMoreYears') THEN 'Count_Person_65OrMoreYears_Female'
+      WHEN REGEXP_CONTAINS(variable_measured, r'65OrMoreYears.*Male|Male.*65OrMoreYears') THEN 'Count_Person_65OrMoreYears_Male'
+      WHEN variable_measured LIKE '%65OrMoreYears%' THEN 'Count_Person_65OrMoreYears'
+      WHEN variable_measured LIKE '%18To64Years%' THEN 'Count_Person_18To64Years'
+      WHEN variable_measured LIKE '%18OrMoreYears%' THEN 'Count_Person_18OrMoreYears'
+      ELSE 'Count_Person'
+    END AS pop_statvar
+  FROM `datcom-store.spanner_dc_graph_prod_DEFAULT.TimeSeries`
+  WHERE provenance = 'dc/base/CDC500'
+    AND variable_measured LIKE 'Percent_%'
+    AND variable_measured NOT IN (
+      'Percent_Person_50To74Years_Female_ReceivedMammography',
+      'Percent_Person_21To65Years_Female_ReceivedCervicalCancerScreening',
+      'Percent_Person_21To65Years_Female_ReceivedPapSmearTest',
+      'Percent_Person_50To75Years_ReceivedColorectalCancerScreening'
+    )
+  GROUP BY cdc500, pop_statvar
+),
+
+svo_percent AS (
+  SELECT
+    O.variable_measured AS statvar,
+    O.entity1 AS observation_about,
+    O.date AS observation_date,
+    O.value AS percent,
+    T.measurement_method AS measurement_method,
+    cdc_sv.pop_statvar
+  FROM `datcom-store.spanner_dc_graph_prod_DEFAULT.Observation` AS O
+  INNER JOIN `datcom-store.spanner_dc_graph_prod_DEFAULT.TimeSeries` AS T
+    ON O.variable_measured = T.variable_measured
+    AND O.entity1 = T.entity1
+    AND O.facet_id = T.facet_id
+    AND T.provenance = 'dc/base/CDC500'
+    AND T.variable_measured LIKE 'Percent_%'
+  INNER JOIN cdc_sv
+    ON O.variable_measured = cdc_sv.cdc500
+  WHERE O.entity1 LIKE 'geoId/%'
+    AND LENGTH(O.entity1) = 13
+    AND O.variable_measured LIKE 'Percent_%'
+    AND O.variable_measured NOT IN (
+      'Percent_Person_50To74Years_Female_ReceivedMammography',
+      'Percent_Person_21To65Years_Female_ReceivedCervicalCancerScreening',
+      'Percent_Person_21To65Years_Female_ReceivedPapSmearTest',
+      'Percent_Person_50To75Years_ReceivedColorectalCancerScreening'
+    )
+  QUALIFY ROW_NUMBER() OVER (
+    PARTITION BY O.variable_measured, O.entity1, O.date, T.measurement_method
+    ORDER BY O.last_update_timestamp DESC
+  ) = 1
+),
+
+svo_count AS (
+  SELECT
+    O.variable_measured AS population_statvar,
+    O.entity1 AS observation_about,
+    O.date AS observation_date,
+    O.value AS population
+  FROM `datcom-store.spanner_dc_graph_prod_DEFAULT.Observation` AS O
+  INNER JOIN `datcom-store.spanner_dc_graph_prod_DEFAULT.TimeSeries` AS T
+    ON O.variable_measured = T.variable_measured
+    AND O.entity1 = T.entity1
+    AND O.facet_id = T.facet_id
+    AND T.provenance = 'dc/base/CensusACS5YearSurvey'
+  INNER JOIN (
+    SELECT DISTINCT pop_statvar
+    FROM cdc_sv
+  ) AS pop
+    ON O.variable_measured = pop.pop_statvar
+  WHERE O.entity1 LIKE 'geoId/%'
+    AND LENGTH(O.entity1) = 13
+  QUALIFY ROW_NUMBER() OVER (
+    PARTITION BY O.variable_measured, O.entity1, O.date
+    ORDER BY O.last_update_timestamp DESC
+  ) = 1
 )
+
+SELECT 
+  p.statvar, 
+  SUBSTR(p.observation_about, 1, 8) AS observation_about,
+  p.observation_date, 
+  CONCAT('dcAggregate/', p.measurement_method) AS measurement_method,
+  p.pop_statvar AS population_statvar,
+  SAFE_DIVIDE(
+    SUM(CAST(c.population AS FLOAT64) * CAST(p.percent AS FLOAT64)),
+    SUM(CAST(c.population AS FLOAT64))
+  ) AS percent
+FROM svo_percent AS p
+INNER JOIN svo_count AS c
+  ON p.observation_about = c.observation_about
+  AND p.observation_date = c.observation_date
+  AND p.pop_statvar = c.population_statvar
+GROUP BY 1, 2, 3, 4, 5
 """
 
-client = bigquery.Client()
-try:
-    logging.info("Running the query")
-    query_job = client.query(query)
-except Exception as e:
-    logging.fatal(f"Error faced while running the query {e}")
-try:
-    logging.info("Converting to dataframe")
-    results = query_job.to_dataframe()
-except Exception as e:
-    logging.info(f"Error faced while fetching results: {e}")
-
-logging.info("Writing output to CSV")
-output_file = os.path.join(_OUTPUT_FILE_PATH + "/CDC500State_Output.csv")
-results.to_csv(output_file, index=False)
+
+def run_process(client: bigquery.Client, output_file: str) -> bool:
+    """Executes the BigQuery query and writes the resulting DataFrame to output_file."""
+    logging.info("Running BigQuery aggregation query...")
+    query_job = client.query(QUERY)
+
+    logging.info("Fetching query results into dataframe...")
+    df = query_job.to_dataframe()
+
+    if df.empty:
+        raise RuntimeError("BigQuery query returned 0 rows.")
+
+    output_dir = os.path.dirname(output_file)
+    if output_dir:
+        os.makedirs(output_dir, exist_ok=True)
+    logging.info("Writing %d rows to %s", len(df), output_file)
+    temp_file = output_file + ".tmp"
+    df.to_csv(temp_file, index=False)
+    os.replace(temp_file, output_file)
+    return True
+
+
+def main(argv):
+    del argv  # Unused.
+    client = bigquery.Client()
+    output_file = os.path.join(_FLAGS.output_dir, 'CDC500State_Output.csv')
+    run_process(client, output_file)
+
+
+if __name__ == '__main__':
+    app.run(main)
+