Skip to content

util: remove the sunsetted firestore upload path - #4480

Open
openroad-ci wants to merge 1 commit into
The-OpenROAD-Project:masterfrom
The-OpenROAD-Project-staging:chore/remove-firestore-upload-path
Open

util: remove the sunsetted firestore upload path#4480
openroad-ci wants to merge 1 commit into
The-OpenROAD-Project:masterfrom
The-OpenROAD-Project-staging:chore/remove-firestore-upload-path

Conversation

@openroad-ci

Copy link
Copy Markdown
Member

What

Removes the direct Firestore write from flow/util/uploadMetadata.py, along with the --cred argument and the firebase_admin dependency. Declares google-cloud-pubsub in its place.

Depends on jenkins-ci#213, which stops passing --cred. That has to land first — see Ordering below.

Why

Metrics reports go to Pub/Sub, where a consumer service lands them in SQL. The direct Firestore write this script also performed is sunsetted, and its database has been deleted, so the code path is dead weight. The original author fenced the regions with # --- FIRESTORE (remove when deprecating) --- for exactly this moment.

Changes

flow/util/uploadMetadata.py — 161 lines removed, 0 added:

  • the four # --- FIRESTORE --- regions (imports, upload_data(), the init block, the in-loop call)
  • the --cred argument
  • from datetime import datetime, timezone, which was referenced only from upload_data() and would otherwise become an unused import

Reformatted with black 26.5.1 to match the pinned CI version.

etc/requirements-common.infirebase_admingoogle-cloud-pubsub.

Pub/Sub was never declared here. It worked only because orfsUploadMetadata.groovy runs pip install google-cloud-pubsub at build time, on top of the grpc/google-auth/protobuf stack that firebase_admin happened to pull in. Removing firebase_admin without declaring Pub/Sub would leave the only surviving upload path fetching that whole stack over the network on every report. Declaring it makes it a hash-pinned image dependency instead.

etc/requirements-common_lock.txt — regenerated with the exact command in its header, under Python 3.10 to match:

pip-compile --allow-unsafe --generate-hashes \
  --output-file=etc/requirements-common_lock.txt etc/requirements-common.in
count packages
Removed 18 firebase-admin, google-cloud-firestore, google-cloud-storage, google-cloud-core, httpx[http2], h2/hpack/hyperframe/h11/httpcore, pyjwt[crypto], cachecontrol, msgpack, anyio, …
Added 6 google-cloud-pubsub, grpc-google-iam-v1, opentelemetry-api/-sdk/-semantic-conventions, googleapis-common-protos[grpc]
Version-bumped 0 no existing pin changed

Net 62 → 50 packages.

Ordering

--cred is optional today, so jenkins-ci#213 (which stops passing it) is safe to merge on its own and is already up. This PR removes the argument entirely, so #213 must merge first — otherwise uploadMetadata.py dies on unrecognized arguments: --cred. Since the library is @Library('utils@main'), #213 takes effect on every ORFS branch at once, so there is no window where an old branch still passes it.

Verification

Ran the rewritten script against a synthetic reports/ tree with recording stubs for pubsub_v1 / service_account, and compared against the same runs on the pre-change script:

Scenario Pub/Sub payload before after
v3 (no provenance) 555 bytes, v3 555 bytes, v3
v4 (with provenance) 691 bytes, v4 691 bytes, v4

Byte-identical. Firestore/firebase calls go from 12 to 0. Passing --cred to the new script errors out as expected, confirming the ordering requirement above.

Not in this PR

  • flake.nix still lists python3Packages.firebase-admin.
  • .github/requirements/requirements.in still lists firebase-admin, for github-actions-update-rules.yml and github-actions-manual-update-rules.yml — both of which call flow/util/updateRules.py, deleted in 63cfaa332 (2025-02-25).

Metrics reports go to Pub/Sub, where a consumer service lands them in SQL.
The direct Firestore write this script also performed is sunsetted and its
database has been deleted, so the code, the --cred argument and the
firebase_admin dependency are all dead weight.

Drops the FIRESTORE-marked regions the original author left for exactly this
purpose, along with the now-unused datetime/timezone import, which was only
referenced from upload_data().

Swaps firebase_admin for google-cloud-pubsub in requirements-common.in and
regenerates the lock. Pub/Sub was never declared there; it worked only
because the Jenkins library pip-installs it at run time on top of the
grpc/google-auth stack that firebase_admin happened to pull in. Declaring it
makes the surviving path a hash-pinned image dependency instead of a
per-build network fetch. The regenerated lock bumps no existing package: 18
removed, 6 added, 62 -> 50 total.

Jenkins stopped passing --cred first, so this is safe to land now.
Verified with recording stubs that the Pub/Sub payload is byte-identical
before and after: 555 bytes for v3, 691 bytes for v4 with provenance.

Signed-off-by: SombraSoft <sombrio@sombrasoft.dev>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes the deprecated Firestore upload functionality from flow/util/uploadMetadata.py and replaces the firebase_admin dependency with google-cloud-pubsub in the requirements. This includes removing the upload_data function, the --cred command-line argument, and Firestore initialization, while updating the locked dependencies in etc/requirements-common_lock.txt. There are no review comments, and I have no feedback to provide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants