Updated mapping and respective files with download script

statvar_imports/us_nces/nces_employed_college_grads/README.md

@@ -1,27 +1,101 @@
-NCSES_Employed_College_Grads_Import
+# NCSES Employed College Graduates Import
 
-how to download data: Download script (download_util_script.py). To download the data, you'll need to use the provided download script,download_util_script.py. This script will automatically create an "input" folder where you should place the file to be processed.
+**Import Name:** `NCSES_Employed_College_Grads_Import`  
+**Dataset Provider:** National Center for Science and Engineering Statistics (NCSES), National Science Foundation (NSF)  
+**Survey:** National Survey of College Graduates (NSCG)  
+**Target Table:** Table 6-2 – *"Employed college graduates, by sex, ethnicity, race, and major occupation: 2003–23"*  
+**Provenance URL:** [NCSES Data Explorer](https://ncsesdata.nsf.gov/explorer/datatables?term=race&exactMatch=no&page=1&filterSuperTopic=Demographics&filterTopic=Sex&datatablespage=2)  
+**Survey Landing Page:** [NSCG Overview & Data Tables](https://ncses.nsf.gov/surveys/national-survey-college-graduates)  
+**Licensing & Terms:** U.S. Federal Government Public Domain ([NSF Open Data Policy](https://www.nsf.gov/policies/open-government))
 
-type of place: State.
+---
 
-statvars: Demographics
+## 1. Dataset Overview & Coverage
 
-years: 2004 to 2021.
+This import pipeline ingests longitudinal counts of employed U.S. college graduates holding a bachelor's degree or higher across demographics and occupation groups:
 
-place_resolution: Places resolved to wikidataId in metadata sheet itself.
+- **Geographic Coverage:** National level (`country/USA`).
+- **Temporal Coverage:** Biennial survey cycles: 2003, 2010, 2013, 2015, 2017, 2019, 2021, 2023 and so on.
+- **Demographics:**
+  - Gender: `Female`, `Male`, and Total (`Both sexes`).
+  - Ethnicity: `HispanicOrLatino`.
+  - Race: `WhiteAlone`, `BlackOrAfricanAmericanAlone`, `Asian`, `AmericanIndianOrAlaskaNative`, `OtherPacificIslander`, `TwoOrMoreRaces`.
+- **Occupations:**
+  - S&E Occupations (Biological/agricultural/life scientists, Computer/mathematical scientists, Physical/related scientists, Social/related scientists, Engineers).
+  - S&E-Related Occupations.
+  - Non-S&E Occupations.
+- **Statistical Variables:** 209 canonical StatVars under `dcs:StatVarObservation`.
 
-Command to download using the download util.
+### Survey Footnote '2023a' & Header Normalization
+In Table 6-2 (NSF 25-322), the 2023 column header is labeled `2023a`:
+- **What 'a' Represents:** According to official NCSES survey documentation, footnote `a` documents a survey questionnaire wording change for sex:
+  > *"The 2023 estimates by sex were based on responses to the question, 'What sex were you assigned at birth, on your original birth certificate? 1. Male, 2. Female,' which was a change from prior survey cycles."*
+  (Prior survey cycles asked *"What is your sex?"* in 2021 and *"Are you ... 1. Male, 2. Female"* in 2019 and earlier).
+- **Not Provisional Data:** Footnote `a` does **not** indicate preliminary, unverified, or provisional data. NCSES/NSCG does not publish provisional data in these final analytic data releases; all published figures represent final survey estimates.
+- **Why It Is Removed:** `stat_var_processor.py` matches year column headers via exact string matching against keys defined in `pv_map.csv` (`2023`). Leaving the footnote suffix intact causes `stat_var_processor.py` to fail to recognize the column, dropping all 2023 observations. Because the numbers represent final official estimates for the 2023 survey cycle, `download.py` safely normalizes `2023a` to `2023` in the header rows (rows 1–4) when generating `source_files/cleaned_<filename>.xlsx`.
 
-python3 download_util_script.py --download_url="https://ncses.nsf.gov/pubs/nsf23306/assets/data-tables/tables/nsf23306-tab006-002.xlsx" --output_folder="source_files" --unzip=False
+---
 
+## 2. Directory Layout
 
-#### Processing the data:
+```text
+nces_employed_college_grads/
+├── README.md                 # Dataset and operational documentation
+├── download.py               # Dynamic scraper & header normalizer
+├── manifest.json             # Pipeline automation & Cloud Batch execution spec
+├── metadata.csv              # Global metadata config (place resolution to country/USA)
+├── pv_map.csv                # Property-value mapping rules & #Header directives
+├── validation_config.json    # Validation rules (0.1% deletions threshold)
+├── source_files/             # Downloaded files (raw nsf*.xlsx and cleaned_*.xlsx)
+├── output/                   # Transformed output CSV and TMCF
+├── counters/                 # Counter logs produced by stat_var_processor
+└── test_data/                # Minimal sample input and expected golden output
+```
 
+---
 
-How to run:
-python3 stat_var_processor.py --input_data='../../statvar_imports/us_nces/nces_employed_college_grads/test_data/<filename>.xlsx' --pv_map="../../statvar_imports/us_nces/nces_employed_college_grads/<filename>.csv" --output_path='../../statvar_imports/us_nces/nces_employed_college_grads/test_data/<filename>' --config="../../statvar_imports/us_nces/nces_employed_college_grads/<filename>.csv"
+## 3. Execution Workflow
 
+### Step 1: Download & Preprocess Source Data
+Scrapes the NSCG landing page to locate the latest Table 6-2 Excel file, saves the raw file to `source_files/` to preserve data provenance, and generates an atomically written `source_files/cleaned_<filename>.xlsx` with normalized year headers:
 
+```bash
+python3 download.py
+```
 
-Example
-python3 stat_var_processor.py --input_data='../../statvar_imports/us_nces/nces_employed_college_grads/test_data/sample_input.xlsx' --pv_map="../../statvar_imports/us_nces/nces_employed_college_grads/pv_map.csv" --output_path='../../statvar_imports/us_nces/nces_employed_college_grads/test_data/sample_output' --config="../../statvar_imports/us_nces/nces_employed_college_grads/metadata.csv"
+### Step 2: Transform & Generate StatisticalVariable Observations
+Executes the standard Data Commons `stat_var_processor`:
+
+```bash
+python3 ../../../tools/statvar_importer/stat_var_processor.py \
+  --input_data=source_files/cleaned_*.xlsx \
+  --pv_map=pv_map.csv \
+  --config_file=metadata.csv \
+  --output_path=output/nces_college \
+  --output_counters=counters/nces_college.csv \
+  --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf
+```
+
+### Generated Outputs
+- **Cleaned Data:** `output/nces_college.csv` (1,571 observations).
+- **Template MCF:** `output/nces_college.tmcf`.
+- **Counters:** `counters/nces_college.csv`.
+
+---
+
+## 4. Testing & Validation
+
+### Lint & Formatting Check
+Run lint via java jar tool and verify compliance with Google style via YAPF:
+
+```bash
+yapf --diff --style=google download.py
+```
+
+---
+
+## 5. Pipeline Automation & Refresh Cadence
+
+- **Automation:** Scheduled via Cloud Batch in `manifest.json`.
+- **Cron Cadence:** Twice monthly on the 4th and 18th at 07:00 UTC (`0 07 4,18 * *`).
+- **Validation:** Enforced via `validation_config.json` with `DELETED_RECORDS_PERCENT` capped at `0.1`.

statvar_imports/us_nces/nces_employed_college_grads/download.py

@@ -0,0 +1,200 @@
+#!/usr/bin/env python3
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#           https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Download and preprocess NSCG employed college graduates tables.
+
+This script scrapes the NCSES National Survey of College Graduates (NSCG)
+landing page to discover the latest publication table (Table 6-2), downloads
+the raw Excel file into source_files/, and creates an atomically written
+cleaned copy (cleaned_<filename>.xlsx) where year headers with survey
+methodology footnote suffixes (e.g. '2023a' -> '2023') are normalized strictly
+within header rows 1-4.
+"""
+
+import os
+import re
+import sys
+from typing import Optional
+from urllib.parse import urlparse
+from absl import app
+from absl import logging
+import openpyxl
+
+# Add data/util to sys.path so we can import shared wrapper functions
+_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
+_PROJECT_ROOT = os.path.abspath(os.path.join(_SCRIPT_DIR, '../../..'))
+_UTIL_DIR = os.path.join(_PROJECT_ROOT, 'util')
+if _UTIL_DIR not in sys.path:
+    sys.path.insert(0, _UTIL_DIR)
+
+try:
+    from download_util import request_url
+    from download_util_script import download_file
+except ImportError:
+    logging.fatal("Could not import download utilities from 'util/'.")
+
+LANDING_PAGE_URL = (
+    "https://ncses.nsf.gov/surveys/national-survey-college-graduates")
+FILE_PATTERN = (
+    r'/pubs/[^/]+/assets/data-tables/tables/[^/]*tab006-002\.(?:xlsx|csv)')
+OUTPUT_FOLDER = os.path.join(_SCRIPT_DIR, "source_files")
+
+# Pattern to strip footnote markers (e.g. '2023a' -> '2023').
+# Note: In Table 6-2 (NSF 25-322), footnote 'a' on '2023a' denotes a survey
+# question change: "The 2023 estimates by sex were based on responses to the
+# question, 'What sex were you assigned at birth, on your original birth
+# certificate? 1. Male, 2. Female,' which was a change from prior survey cycles."
+YEAR_HEADER_PATTERN = re.compile(r'^\s*(\d{4})([a-zA-Z*#].*|\s+.*)?$',
+                                 re.DOTALL)
+
+
+def resolve_url(landing_url: str = LANDING_PAGE_URL,
+                file_pattern: str = FILE_PATTERN,
+                headers: Optional[dict] = None,
+                tries: int = 3,
+                delay: int = 5) -> Optional[str]:
+    """Scrapes landing page HTML to dynamically find matching table URL.
+
+    Args:
+        landing_url: URL of the webpage containing table links.
+        file_pattern: Regex pattern to match the target link.
+        headers: Optional dictionary of HTTP headers to send with the request.
+        tries: Number of retry attempts.
+        delay: Initial delay for retries in seconds.
+
+    Returns:
+        Absolute URL of the target file, or None if not found or on error.
+    """
+    logging.info("Attempting to resolve target URL from landing page: %s",
+                 landing_url)
+
+    try:
+        content = request_url(landing_url,
+                              headers=headers or {},
+                              output='text',
+                              retries=tries,
+                              retry_secs=delay)
+    except Exception as e:
+        logging.error("Failed to fetch landing page '%s': %s", landing_url, e)
+        return None
+
+    if not content:
+        logging.error("Failed to fetch landing page '%s': empty response.",
+                      landing_url)
+        return None
+
+    matches = re.findall(file_pattern, content)
+    if not matches:
+        logging.error("No link matching pattern '%s' found on '%s'.",
+                      file_pattern, landing_url)
+        return None
+
+    resolved_path = matches[0]
+    parsed_landing = urlparse(landing_url)
+    base_domain = f"{parsed_landing.scheme}://{parsed_landing.netloc}"
+    resolved_url = f"{base_domain}{resolved_path}"
+    logging.info("Dynamically resolved download URL: %s", resolved_url)
+    return resolved_url
+
+
+def clean_year_headers(folder_path: str, max_header_row: int = 4) -> bool:
+    """Cleans year headers in downloaded Excel files by removing footnote suffixes.
+
+    Normalizes footnote suffixes (e.g., '2023a' -> '2023') strictly in the top
+    header rows (rows 1-4). Leaves all data rows (row 5+) completely untouched.
+    Writes the output to a separate 'cleaned_<filename>' file atomically using a
+    temporary file and rename, preserving raw data provenance and original file
+    modification timestamps.
+
+    Args:
+        folder_path: Path to directory containing downloaded Excel files.
+        max_header_row: Maximum row index to inspect for header columns.
+
+    Returns:
+        True if header cleaning succeeded, False if an error occurred.
+    """
+    if not folder_path or not os.path.exists(folder_path):
+        return True
+
+    for filename in os.listdir(folder_path):
+        if not filename.endswith('.xlsx') or filename.startswith('cleaned_'):
+            continue
+
+        file_path = os.path.join(folder_path, filename)
+        cleaned_file_path = os.path.join(folder_path, f"cleaned_{filename}")
+        temp_cleaned_path = f"{cleaned_file_path}.tmp"
+        wb = None
+        try:
+            wb = openpyxl.load_workbook(file_path)
+            sheet = wb.active
+            for row in sheet.iter_rows(max_row=max_header_row):
+                for cell in row:
+                    if isinstance(cell.value, str):
+                        new_val = YEAR_HEADER_PATTERN.sub(r'\1', cell.value)
+                        if new_val != cell.value:
+                            cell.value = new_val
+            # Atomic write: save to temporary file, then atomic rename
+            wb.save(temp_cleaned_path)
+            os.replace(temp_cleaned_path, cleaned_file_path)
+            logging.info("Successfully wrote cleaned file to '%s'",
+                         cleaned_file_path)
+        except (ValueError, OSError) as e:
+            logging.error("Error cleaning headers in file '%s': %s", file_path,
+                          e)
+            if os.path.exists(temp_cleaned_path):
+                try:
+                    os.remove(temp_cleaned_path)
+                except OSError:
+                    pass
+            return False
+        except Exception as e:
+            logging.error(
+                "An unexpected error occurred while cleaning headers in '%s': %s",
+                file_path, e)
+            if os.path.exists(temp_cleaned_path):
+                try:
+                    os.remove(temp_cleaned_path)
+                except OSError:
+                    pass
+            return False
+        finally:
+            if wb is not None:
+                wb.close()
+
+    return True
+
+
+def main(_):
+    logging.set_verbosity(logging.INFO)
+    logging.info("Script execution started...")
+
+    # Ensure output directory exists idempotently
+    os.makedirs(OUTPUT_FOLDER, exist_ok=True)
+
+    resolved_url = resolve_url(LANDING_PAGE_URL, FILE_PATTERN, None)
+    if not resolved_url:
+        logging.fatal("Failed to resolve URL from landing page.")
+
+    if not download_file(resolved_url, OUTPUT_FOLDER, False, None):
+        logging.fatal(
+            "File download or processing failed. Check logs for details.")
+
+    if not clean_year_headers(OUTPUT_FOLDER):
+        logging.fatal("Year header cleaning failed. Check logs for details.")
+
+    logging.info("Script processing completed successfully.")
+
+
+if __name__ == '__main__':
+    app.run(main)

statvar_imports/us_nces/nces_employed_college_grads/manifest.json

@@ -8,19 +8,23 @@
             "provenance_url": "https://ncsesdata.nsf.gov/explorer/datatables?term=race&exactMatch=no&page=1&filterSuperTopic=Demographics&filterTopic=Sex&datatablespage=2",
             "provenance_description": "NCSES is the U.S. agency that collects and reports data on science, engineering, and technology",
             "scripts": [
-                "../../../util/download_util_script.py --download_url=https://ncses.nsf.gov/pubs/nsf23306/assets/data-tables/tables/nsf23306-tab006-002.xlsx --output_folder=source_files",
-                "../../../tools/statvar_importer/stat_var_processor.py --input_data=source_files/*.xlsx --pv_map=pv_map.csv --config_file=metadata.csv --output_path=output/nces_college"
+                "download.py",
+                "../../../tools/statvar_importer/stat_var_processor.py --input_data=source_files/cleaned_*.xlsx --pv_map=pv_map.csv --config_file=metadata.csv --output_path=output/nces_college --output_counters=counters/nces_college.csv --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
             ],
             "source_files": [
-                "source_files/*.xlsx"
+                "source_files/*.xlsx",
+                "counters/*.csv",
+                "validation_config.json"
             ],
             "import_inputs": [
                 {
                     "template_mcf": "output/nces_college.tmcf",
-                    "cleaned_csv": "output/nces_college.csv"
+                    "cleaned_csv": "output/nces_college.csv",
+                    "node_mcf": "output/*.mcf"
                 }
             ],
-            "cron_schedule": "0 07 4,18 * *"
+            "cron_schedule": "0 07 4,18 * *",
+            "validation_config_file": "validation_config.json"
         }
     ]
-}
+}

statvar_imports/us_nces/nces_employed_college_grads/metadata.csv

@@ -2,4 +2,3 @@ param,value
 places_within,country/USA
 output_columns,"observationAbout,observationDate,value,variableMeasured,unit,scalingFactor,measurementMethod,observationPeriod"
 url,https://ncsesdata.nsf.gov/explorer/datatables?term=race&exactMatch=no&page=1&filterSuperTopic=Demographics&filterTopic=Sex&datatablespage=2
-comments,"Table nsf23306-tab006-002 (no direct link available),  Changed  Black to BlackOrAfricanAmericanAlone and added employmentStatus	: Employed"