US_CDC_PRAMS : Import Automation

[kartik-s21]

Summary

Automates the manual script-based import US_CDC_PRAMS (scripts/cdc_prams) to enable automated scheduling, regression testing, and pipeline execution in the Data Commons importer.

Context

• Dataset: US CDC Pregnancy Risk Assessment Monitoring System (PRAMS) MCH Indicators (2016–2020).
• Format: Script-based import extracting 508-compliant multi-page PDFs using Tabula and generating Data Commons CSV, MCF, and TMCF.
• Coverage: 33,180 observations across 168 StatisticalVariables for 48 states/territories/cities + US national.

Key Changes

1. Automation & Catalog Discovery:
   • Added manifest.json registering scripts/cdc_prams:US_CDC_PRAMS with yearly refresh (0 0 1 6 *).
   • Verified catalog discovery with list_imports.py.
2. Pipeline Fixes:
   • download.py & download_input_files.py: Added User-Agent handling to bypass CDC Akamai WAF blocks.
   • process.py: Upgraded for NumPy 2.x (np.nan) and Pandas 2.x compatibility; eliminated dynamic dtype warnings.
   • process_test.py: Fixed tuple syntax bug and streamlined test suite (runs cleanly in ~16s).
3. Validation & Goldens:
   • Added validation_config.json with standard rules (DELETED_RECORDS_PERCENT, EMPTY_IMPORT_CHECK, LINT_ERROR_COUNT, MISSING_REFS_COUNT, GOLDENS_CHECK).
   • Added golden_observations.csv (validated against gs://unresolved_mcf/import_validation/top_100k_places.csv, 49/49 places matching).
   • Added golden_summary_report.csv (canonical GenMCF schema summary covering all 168 StatVars).
4. Environment & Documentation:
   • Added tabula-py to import-automation/executor/requirements.txt to support PDF extraction across executor container environments.
   • Updated README.md with complete indicator topics, layout, and execution instructions.

Verification

• Unit Tests: python3 -m unittest scripts/cdc_prams/process_test.py -> 2/2 tests passed (OK).
• Code Style: Formatted with yapf --style=google (0 diffs).
• Golden Validation:
  • validator_goldens.py on summary report: 168/168 goldens matched (100%).
  • validator_goldens.py on observations: 49/49 golden places matched (100%).
• Cloud Batch Job: Successfully executed end-to-end; GenMCF generated and resolved all 33,180 rows across 168 nodes with 0 errors.

[gemini-code-assist]

Code Review

This pull request establishes import automation for the US CDC Pregnancy Risk Assessment Monitoring System (PRAMS) dataset. The changes introduce a manifest, validation configurations, and golden datasets, while refactoring the download and processing scripts to use robust HTTP sessions and modern pandas methods. Unit tests have also been updated to use class-level setup and teardown. The review feedback suggests making the confidence interval string-splitting logic more robust against variable spacing and improving exception handling and caching during ZIP file downloads.

[gemini-code-assist]

Splitting the CI column using r"\s+|-" is fragile when there are spaces around the hyphen (e.g., 39.3 (36.4 - 42.2)). This results in empty columns in the split DataFrame, shifting the upper limit out of the first 3 columns and causing it to be lost. Normalizing the string by removing parentheses and replacing hyphens with spaces first, then splitting by whitespace, is much more robust and handles variable spacing gracefully.

Suggested change

	df[i] = df[i].fillna('').astype(str).replace({
	'<NA>': '',
	'nan': '',
	'None': ''
	})
	# Splitting the column based on space and "-"
	df_split = df[i].str.split(r"\s+|-", expand=True)
	df[i] = df[i].fillna('').astype(str).replace({
	'\u003cNA\u003e': '',
	'nan': '',
	'None': ''
	})
	# Normalize separators by removing parentheses and replacing hyphens with spaces
	normalized = df[i].str.replace(r'[\\(\\)]', '', regex=True).str.replace('-', ' ', regex=False).str.strip()
	df_split = normalized.str.split(r'\\s+', expand=True)

[gemini-code-assist]

When downloading and processing ZIP files, ensure that exception handling covers both network-related errors (e.g., requests.exceptions.RequestException) and ZIP-specific errors (e.g., zipfile.BadZipFile) to prevent the script from crashing on corrupted or invalid downloads. Additionally, writing the downloaded content to out_file first before extracting it ensures the caching mechanism works correctly and avoids redundant downloads.

Suggested change

	if download_file_url.endswith(".zip"):
	with zipfile.ZipFile(io.BytesIO(req.content)) as zipfileout:
	zipfileout.extractall(path)
	else:
	with open(out_file, 'wb') as file:
	file.write(req.content)
	try:
	with open(out_file, 'wb') as file:
	file.write(req.content)
	if download_file_url.endswith(".zip"):
	with zipfile.ZipFile(out_file) as zipfileout:
	zipfileout.extractall(path)
	except (requests.exceptions.RequestException, zipfile.BadZipFile) as e:
	raise RuntimeError(f"Failed to download or extract zip file: {e}")

References

1. When downloading and processing ZIP files, ensure that exception handling covers both network-related errors (e.g., requests.exceptions.RequestException) and ZIP-specific errors (e.g., zipfile.BadZipFile) to prevent the script from crashing on corrupted or invalid downloads.