RoutineMatch: East Coast Urban Intelligence Pipeline

This project implements a serverless data pipeline for processing and analyzing urban data for major East Coast cities (NYC, DC, Baltimore, Philly, Boston, Atlanta, etc.).

Architecture

1. Ingestion: Apache Beam pipeline running on Dataflow processes raw city data.
2. Storage: Raw data is landed in BigQuery (city_raw dataset).
3. Transformation: Dataform handles the ELT process, transforming raw data into intelligence tables (city_intelligence dataset).
4. CI/CD: Cloud Build automates pipeline validation and Dataform compilation.
5. Analysis: BigFrames provides a Pythonic interface for rapid data exploration.

Getting Started

Prerequisites

• Google Cloud Project with BigQuery and Dataflow APIs enabled.
• Dataform repository initialized in the GCP console.

Local Development

pip install apache-beam[gcp] bigframes
python pipeline.py --project routinematch-prod --region us-east1 --runner DirectRunner

Deploying via Cloud Build

Push changes to GitHub to trigger the cloudbuild.yaml workflow in Google Cloud.