Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions scripts/earthengine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The scripts use [Google Earth Engine](https://earthengine.google.com/)
to extract flooded regions as a [geoTIFF](https://en.wikipedia.org/wiki/GeoTIFF)
file. These `.tif` files are exported to [Google Cloud
Storage](https://cloud.google.com/storage) (GCS).
The raster files are then copied over using `gsutil` to be processed into `.csv` locally.
The raster files are then copied over using `gcloud storage` to be processed into `.csv` locally.


The tools can be installed as follows:
Expand All @@ -36,7 +36,7 @@ earthengine authenticate --quiet
```

### Google Cloud SDK
Install the `gsutil` command to copy files from GCS to the local
Install the `gcloud storage` command to copy files from GCS to the local
machine.

1. Install GCS tools using the command:
Expand All @@ -55,7 +55,7 @@ gcloud auth login
To copy files on GCS, create a storage bucket with the following command or on
the cloud console.
```
gsutil mb gs://<GCS-BUCKET-NAME>/
gcloud storage buckets create gs://<GCS-BUCKET-NAME>/
```

## Extract geoTIFF from EarthEngine (EE)
Expand Down Expand Up @@ -99,7 +99,7 @@ in the previous step into a StatVarObservation MCF nodes for
following:
```
# Download the geoTiff from GCS
gsutil cp gs://<GCS_BUCKET>/<GCS_FOLDER>/*.tif .
gcloud storage cp gs://<GCS_BUCKET>/<GCS_FOLDER>/*.tif .
# Convert the geoTiff into raster
python3 raster_to_csv.py \
--input_geotiff=ee_image_dynamic_world-band_water-r_max-mask_land-s_1000-from_2022-01-01.tif \
Expand Down
2 changes: 1 addition & 1 deletion scripts/oecd/regional_demography/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ In each subdirectory, the source CSVs are saved as `REGION_DEMOGR_{subdirectory
Population raw data is stored [here](https://pantheon.corp.google.com/storage/browser/_details/datcom-source-data/oecd/regional_demography/population/REGION_DEMOGR_population.csv?authuser=0&project=datcom-204919) since it's quite big.
To copy raw CSV into the population directory, run:
```bash
gsutil cp gs://datcom-source-data/oecd/regional_demography/population/REGION_DEMOGR_population.csv .
gcloud storage cp gs://datcom-source-data/oecd/regional_demography/population/REGION_DEMOGR_population.csv .
```

Population raw CSV is missing data for region_id CL16. To add the missing data, we created manual_curated_population.csv with command
Expand Down
4 changes: 2 additions & 2 deletions scripts/us_bts/latch/download_script.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ def _download_paginated_socrata_file(
def create_download_configs(gcs_urls_config_file: str) -> List[Dict]:
"""Reads the URL config JSON from GCS and generates the download configurations."""
try:
result = subprocess.run(['gsutil', 'cat', gcs_urls_config_file],
result = subprocess.run(['gcloud', 'storage', 'cat', gcs_urls_config_file],
capture_output=True,
text=True,
check=True,
Expand All @@ -131,7 +131,7 @@ def create_download_configs(gcs_urls_config_file: str) -> List[Dict]:
)
except (subprocess.CalledProcessError, FileNotFoundError) as e:
logging.fatal(
f"Failed to read GCS config file '{gcs_urls_config_file}': {e}. Ensure 'gsutil' is in your PATH and the file exists."
f"Failed to read GCS config file '{gcs_urls_config_file}': {e}. Ensure 'gcloud' is in your PATH and the file exists."
)
return []
except json.JSONDecodeError as e:
Expand Down
2 changes: 1 addition & 1 deletion scripts/us_eia/opendata/generate_jsonl_for_bq.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

To import to bigquery:
- run this script: `python3 generate_jsonl_for_bq.py`
- copy tmp_bq_import/ to gcs: `gsutil -m cp -r bq_import gs://us_eia/`
- copy tmp_bq_import/ to gcs: `gcloud storage cp --recursive bq_import gs://us_eia/`
- load data into bigquery:
```
bq load \
Expand Down
8 changes: 4 additions & 4 deletions scripts/us_nces/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,13 @@ This dataset has Population Estimates for the National Center for Education Stat

Note: Give one year at a time for District and Public Schools as there are large number of column values.

### upload the files manually to GCP bucket for processing using gsutil command.
### upload the files manually to GCP bucket for processing using gcloud storage command.
Ex private :
gsutil cp -r /scripts/us_nces/demographics/private_school/input_files gs://unresolved_mcf/us_nces/demographics/private_school/semi_automation_input_files/
gcloud storage cp --recursive /scripts/us_nces/demographics/private_school/input_files gs://unresolved_mcf/us_nces/demographics/private_school/semi_automation_input_files/
Ex public :
gsutil cp -r /scripts/us_nces/demographics/public_school/input_files gs://unresolved_mcf/us_nces/demographics/public_school/semi_automation_input_files/
gcloud storage cp --recursive /scripts/us_nces/demographics/public_school/input_files gs://unresolved_mcf/us_nces/demographics/public_school/semi_automation_input_files/
Ex district :
gsutil cp -r /scripts/us_nces/demographics/school_district/input_files gs://unresolved_mcf/us_nces/demographics/school_district/semi_automation_input_files/
gcloud storage cp --recursive /scripts/us_nces/demographics/school_district/input_files gs://unresolved_mcf/us_nces/demographics/school_district/semi_automation_input_files/

### Note:
The only manual part here is after downloading the input files and then uploading them to gcp bucket. Once they're uploaded, Each import requires its own sh command to copy the files from Google Cloud to a local folder called gcs_folder/input_files. From there, a script automatically picks up these files to process them. Finally, it generates the output and saves it in gcs_folder/output_files
Expand Down
2 changes: 1 addition & 1 deletion scripts/who/gho/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The following are the files generated by the [import_data.py](import_data.py) sc
1. Get the raw data in one of 2 ways:
- To get the pre-downloaded data from our Google Cloud Storage, run and then unzip the file:
```
gsutil cp gs://datcom-source-data/who/gho/indicator_data.zip .
gcloud storage cp gs://datcom-source-data/who/gho/indicator_data.zip .
```
- To download data using the WHO API (this can take about 1 hour), run:
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@
"z = zipfile.ZipFile(io.BytesIO(r.content))\n",
"z.extractall(TMP_PATH)\n",
"\n",
"!gsutil -m cp -r {TMP_PATH} {GCS_BUCKET}"
"!gcloud storage cp --recursive {TMP_PATH} {GCS_BUCKET}"
],
"execution_count": 16,
"outputs": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ SCRIPT_PATH=$(realpath "$(dirname "$0")")

mkdir -p "input_files"

gsutil cp "gs://unresolved_mcf/country/brazil/VISDATA/Benefits_RuralDevelopmentProgram/latest/input_files/*.csv" "$SCRIPT_PATH/input_files"
gcloud storage cp "gs://unresolved_mcf/country/brazil/VISDATA/Benefits_RuralDevelopmentProgram/latest/input_files/*.csv" "$SCRIPT_PATH/input_files"

PROCESSOR_SCRIPT="$SCRIPT_PATH/../../../tools/statvar_importer/stat_var_processor.py"
INPUT_BASE_PATH="$SCRIPT_PATH/input_files"
Expand Down
8 changes: 4 additions & 4 deletions statvar_imports/fbi/fbigovcrime/preprocess.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,19 +59,19 @@ def copy_and_process_files_from_gcs(gcs_bucket, gcs_prefix, local_base_dir):
logging.info(f"Listing files from: {gcs_source_path_wildcard}")

try:
result = subprocess.run(['gsutil', 'ls', gcs_source_path_wildcard], capture_output=True, text=True, check=True)
result = subprocess.run(['gcloud', 'storage', 'ls', gcs_source_path_wildcard], capture_output=True, text=True, check=True)
gcs_files = result.stdout.strip().split('\n')
if not gcs_files or gcs_files == ['']:
logging.fatal(f"No .xlsx files found at '{gcs_source_path_wildcard}'")
raise RuntimeError('No .xlsx files found')
return
except subprocess.CalledProcessError as e:
logging.fatal(f"Error listing files from GCS with gsutil: {e.stderr}")
logging.fatal(f"Error listing files from GCS with gcloud storage: {e.stderr}")
raise RuntimeError('Error listing files from GCS')
return
except FileNotFoundError:
logging.fatal("gsutil command not found. Please ensure the Google Cloud SDK is installed and in your PATH.")
raise RuntimeError('gsutil command not found')
logging.fatal("gcloud storage command not found. Please ensure the Google Cloud SDK is installed and in your PATH.")
raise RuntimeError('gcloud storage command not found')
return

logging.info(f"Found {len(gcs_files)} xlsx files to copy and process.")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,6 @@ If the download script fails or logs HTTP `401 Unauthorized` / `403 Forbidden` e
* Upload the updated configuration file to GCS:
```bash
gcloud storage cp download_config.json gs://unresolved_mcf/india_ndap/NDAP_NSS_Health/latest/download_config.json
# or
gsutil cp download_config.json gs://unresolved_mcf/india_ndap/NDAP_NSS_Health/latest/download_config.json
```
3. **Verify Download**:
* Re-run `python3 download_script.py` and verify `india_nss_health_ailments.csv` downloads successfully.