util: remove the sunsetted firestore upload path - #4480
Open
openroad-ci wants to merge 1 commit into
Open
Conversation
Metrics reports go to Pub/Sub, where a consumer service lands them in SQL. The direct Firestore write this script also performed is sunsetted and its database has been deleted, so the code, the --cred argument and the firebase_admin dependency are all dead weight. Drops the FIRESTORE-marked regions the original author left for exactly this purpose, along with the now-unused datetime/timezone import, which was only referenced from upload_data(). Swaps firebase_admin for google-cloud-pubsub in requirements-common.in and regenerates the lock. Pub/Sub was never declared there; it worked only because the Jenkins library pip-installs it at run time on top of the grpc/google-auth stack that firebase_admin happened to pull in. Declaring it makes the surviving path a hash-pinned image dependency instead of a per-build network fetch. The regenerated lock bumps no existing package: 18 removed, 6 added, 62 -> 50 total. Jenkins stopped passing --cred first, so this is safe to land now. Verified with recording stubs that the Pub/Sub payload is byte-identical before and after: 555 bytes for v3, 691 bytes for v4 with provenance. Signed-off-by: SombraSoft <sombrio@sombrasoft.dev>
Contributor
There was a problem hiding this comment.
Code Review
This pull request removes the deprecated Firestore upload functionality from flow/util/uploadMetadata.py and replaces the firebase_admin dependency with google-cloud-pubsub in the requirements. This includes removing the upload_data function, the --cred command-line argument, and Firestore initialization, while updating the locked dependencies in etc/requirements-common_lock.txt. There are no review comments, and I have no feedback to provide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Removes the direct Firestore write from
flow/util/uploadMetadata.py, along with the--credargument and thefirebase_admindependency. Declaresgoogle-cloud-pubsubin its place.Depends on jenkins-ci#213, which stops passing
--cred. That has to land first — see Ordering below.Why
Metrics reports go to Pub/Sub, where a consumer service lands them in SQL. The direct Firestore write this script also performed is sunsetted, and its database has been deleted, so the code path is dead weight. The original author fenced the regions with
# --- FIRESTORE (remove when deprecating) ---for exactly this moment.Changes
flow/util/uploadMetadata.py— 161 lines removed, 0 added:# --- FIRESTORE ---regions (imports,upload_data(), the init block, the in-loop call)--credargumentfrom datetime import datetime, timezone, which was referenced only fromupload_data()and would otherwise become an unused importReformatted with
black26.5.1 to match the pinned CI version.etc/requirements-common.in—firebase_admin→google-cloud-pubsub.Pub/Sub was never declared here. It worked only because
orfsUploadMetadata.groovyrunspip install google-cloud-pubsubat build time, on top of thegrpc/google-auth/protobufstack thatfirebase_adminhappened to pull in. Removingfirebase_adminwithout declaring Pub/Sub would leave the only surviving upload path fetching that whole stack over the network on every report. Declaring it makes it a hash-pinned image dependency instead.etc/requirements-common_lock.txt— regenerated with the exact command in its header, under Python 3.10 to match:firebase-admin,google-cloud-firestore,google-cloud-storage,google-cloud-core,httpx[http2],h2/hpack/hyperframe/h11/httpcore,pyjwt[crypto],cachecontrol,msgpack,anyio, …google-cloud-pubsub,grpc-google-iam-v1,opentelemetry-api/-sdk/-semantic-conventions,googleapis-common-protos[grpc]Net 62 → 50 packages.
Ordering
--credis optional today, so jenkins-ci#213 (which stops passing it) is safe to merge on its own and is already up. This PR removes the argument entirely, so #213 must merge first — otherwiseuploadMetadata.pydies onunrecognized arguments: --cred. Since the library is@Library('utils@main'), #213 takes effect on every ORFS branch at once, so there is no window where an old branch still passes it.Verification
Ran the rewritten script against a synthetic
reports/tree with recording stubs forpubsub_v1/service_account, and compared against the same runs on the pre-change script:555 bytes, v3555 bytes, v3691 bytes, v4691 bytes, v4Byte-identical. Firestore/firebase calls go from 12 to 0. Passing
--credto the new script errors out as expected, confirming the ordering requirement above.Not in this PR
flake.nixstill listspython3Packages.firebase-admin..github/requirements/requirements.instill listsfirebase-admin, forgithub-actions-update-rules.ymlandgithub-actions-manual-update-rules.yml— both of which callflow/util/updateRules.py, deleted in63cfaa332(2025-02-25).