[India_RBI_State_Statistics] Modified download script and handle deletions

[Krishnam24maheshwari]

In this PR we fixed import execution failures and data corruption issues in the India_RBI_State_Statistics import pipeline

The import was failing at step 1 in rbi_download.py and had several downstream issues preventing successful transformation:

1. Unversioned Remote Config Dependency: rbi_download.py had a hard dependency on gs://unresolved_mcf/india_rbi/state_statistics/configs.py, immediately crashing in environments without GCS access or ADC credentials.
2. Download Rejection & Brittle Execution: Requests to rbi.org.in lacked browser User-Agent headers, causing HTTP 403 / throttling rejections, and the download loop was not isolated per table.
3. Data Corruption & Pandas Incompatibility: Blank/empty cells were converted to the literal string "nan", numeric observation rows were cast to string types, and deprecated applymap calls caused runtime errors across different Pandas versions.
4. Unquoted Globs in run.sh: --input_data=$SCRIPT_PATH/input_files/agriculture/*.xlsx was unquoted, causing Bash to expand the 42 matched files as positional arguments. As a result, Python only processed the first file, silently dropping the other 41 files (and 7 of 8 files in price_and_wages).
5. Entity Resolution Cache Misses: Asterisks stripped from places during preprocessing (replace('*', '')) caused "Kolkata" and "Chennai" to miss the cache entries in rbi_places_resolver.csv (Kolkata**, Chennai**).

Changes made :

1. rbi_download.py & configs.json :
   Migrated table configuration to a local, version-controlled configs.json by default, with automatic fallback if remote GCS paths are unreachable.
   Added browser headers (User-Agent, Accept, Accept-Language) to prevent HTTP 403 rejections.
   Added caching to skip already downloaded files (input_files/).
   Isolated individual table downloads in try/except blocks so a single transient failure does not abort the entire batch.

2. run.sh
   Quoted Input Globs: Enclosed --input_data="$SCRIPT_PATH/input_files/agriculture/*.xlsx" and --input_data="$SCRIPT_PATH/input_files/price_and_wages/*.xlsx" in quotes so stat_var_processor.py correctly expands and processes all 42 agriculture and 8 price/wage tables.
   Counter Tracking: Added COUNTERS_PATH=$SCRIPT_PATH/counters and appended --output_counters to all 18 processing commands.
   Fixed Log Messages: Corrected copy-paste errors in error exit messages.

Added threshold rule for DELETED_RECORDS_PERCENT to 7. As there are up to 6.9% deletion is happening in one of the file I have added all the deletions findings and reasoning in the doc - https://docs.google.com/document/d/1BLArT3T2-2EVql0Ol8tSYw9QtjFjzCzockJBquMC4AY/edit?tab=t.0

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request updates the RBI state statistics import pipeline by refactoring the preprocessing script to write to a temporary directory, adding unit tests, updating place mappings, and introducing a validation configuration. The reviewer feedback highlights several issues: df.map incorrectly converts NaN values to 'nan' strings; mapping combined regions like 'Haryana, Delhi & Chandigarh' to Haryana's ID (Q1174) is incorrect; the Wikidata ID for NCT of Delhi should be updated to Q9357528 for AA1 level data; and using a broad wildcard pattern for node_mcf in manifest.json is inefficient and should be scoped to specific directories.

[Krishnam24maheshwari]

done