docs(config): supplying a BigQuery service account key from the environment - #339
Merged
Conversation
…the environment
`serviceAccountKey` is a `json` property, and json properties take their value
literally — an `{"env": "..."}` inside one is never resolved. A server holding
its key in an environment variable therefore cannot use that property, and the
failure arrives as an error from BigQuery about a missing client_email field
rather than anything naming the config.
Document `serviceAccountKeyJson` and `serviceAccountKeyJsonBase64`
(malloydata/malloy#3039), which are `secret` strings and do take references,
with the shell commands that produce each form. Also state the json-property
exception in the Environment Variables section, which until now claimed
without qualification that any property value can be replaced with a
reference.
Signed-off-by: lloyd tabb <lloyd@malloydata.org>
lloydtabb
commented
Aug 13, 2026
| export BIGQUERY_CREDENTIALS_JSON_B64="$(base64 < service-account-key.json | tr -d '\n')" | ||
| ``` | ||
|
|
||
| Note that `serviceAccountKey` — the `json`-typed property — **cannot** take an environment variable reference. Like every `json` property, it treats `{"env": "..."}` as literal data, so that object itself becomes the credentials and BigQuery rejects it with `The incoming JSON object does not contain a client_email field`. Use one of the two `secret` properties above instead. If several are set, the order of precedence is `serviceAccountKey`, `serviceAccountKeyJson`, `serviceAccountKeyJsonBase64`. |
Contributor
Author
There was a problem hiding this comment.
Not sure we need this note @mtoy-googly-moogly, what do you think? Is it too verbose? The other is useful.
lloydtabb
commented
Aug 13, 2026
|
|
||
| You can also provide a plain string value directly — this is useful for testing but not recommended for shared or committed config files. | ||
|
|
||
| **One exception: `json` properties.** Properties whose type is `json` — `serviceAccountKey`, `ssl`, `session`, `extraCredential`, `extraHeaders` — hold structured data, and they take that data literally. A `{"env": "..."}` written inside one is not resolved; it is passed along to the driver as the object it looks like, which usually surfaces later as a confusing error from the driver rather than as a missing-variable message. Where a secret needs to come from the environment, use the string-typed property meant for it (for BigQuery, [`serviceAccountKeyJson`](#bigquery-google-bigquery)). |
Contributor
Author
There was a problem hiding this comment.
@mtoy-googly-moogly I think to kill this too, but I figured I'd discuss before doing it.
Follows the change in malloydata/malloy#3039 to a single `serviceAccountKeyJson` that takes JSON or base64 and detects which. Drops the separate base64 property, and states the true reason base64 is offered: quoting, not newlines — a downloaded key file carries its newlines as `\n` escapes, so the JSON is one line with none to mangle. Signed-off-by: lloyd tabb <lloyd@malloydata.org>
lloydtabb
commented
Aug 14, 2026
| export BIGQUERY_CREDENTIALS_JSON="$(base64 < service-account-key.json | tr -d '\n')" | ||
| ``` | ||
|
|
||
| No flag distinguishes the two — the encoding is detected, since JSON always begins with `{` and `{` is not a base64 character. Surrounding whitespace is ignored, so a trailing newline from `$(cat …)` or a here-doc is harmless. |
Contributor
Author
There was a problem hiding this comment.
Let's lose this paragraph. I don't think anyone cares
Review feedback on #339. The detection rule (JSON starts with `{`, base64 can't) is implementation detail nobody reading this page needs — both encodings work, which the examples already show. The json-property exception in the Environment Variables section is dropped too; the note in the BigQuery section covers the case a reader actually hits, and postgres and trino already carry their own. Signed-off-by: lloyd tabb <lloyd@malloydata.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs for malloydata/malloy#3039, which adds
serviceAccountKeyJsonto the BigQuery connection — onesecretstring holding the whole key file, as JSON or base64, with the encoding detected rather than declared.Two changes to
setup/config.malloynb:The BigQuery section gains the parameter, an example of pointing it at an environment variable, and the shell commands that produce each encoding (
jq -cfor JSON,base64 | tr -d '\n'for base64 — GNUbase64wraps at 76 columns, which is awkward in an environment variable even though the decoder tolerates it). It notes that the encoding needs no flag because JSON always starts with{and{is not a base64 character, and that surrounding whitespace is ignored. It also mentions that with no key configured the connection falls back to application default credentials, which is what most local development is actually using.The section says plainly that
serviceAccountKeycannot take an{"env": ...}reference, and quotes the error BigQuery gives when someone tries —The incoming JSON object does not contain a client_email field. That string is the entire symptom, and it names neither the property nor the reason, so it is worth having in a page that search will find.The Environment Variables section currently opens with "Any property value can be replaced with an environment variable reference," which isn't true of
jsonproperties. Thepostgresandtrinosections each already carry a local correction for their own json properties; this adds the general statement in the place where someone forms the expectation, and points to the string-typed property to use instead.No new pages and no anchor changes; the one new cross-link resolves to the existing
#bigquery-google-bigqueryheading, checked against the site's ownhashForHeading.