Conversation
A Zenodo record holds at most 100 files and the compaction meant to keep a study under that cap is not built yet, so session 101 fails and stays failed: the refusal maps to QUOTA_EXCEEDED, which is slow-tier and needs human action to clear. No data is lost -- the submission stays in pending storage and QueuePanel surfaces the reason -- but the researcher cannot fix it themselves, so they need to hear about the limit before they start collecting rather than after. Unconditional and offline, unlike dataverse.ts's version probe: the cap is a property of Zenodo itself rather than of an installation, so there is nothing to interrogate and no failure mode to fail open from. Both this and its tests should be deleted when compaction ships. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ten findings from a high-effort review of this branch, plus two follow-ups they surfaced. Filename identity (4 findings). The collision cache claimed the raw leaf filename while listFiles reported provider-transformed names -- Zenodo flattens slashes, Drive keeps only the leaf -- so on a rehydrated cache no claim ever matched. Zenodo's writeSessionFile is an overwriting PUT and Dataverse silently renames duplicates, so neither has a NAME_CONFLICT to fall back on: the miss silently destroyed one participant's data on Zenodo and duplicated it on Dataverse. Adds StorageProvider.storedNameFor, the cache's identity function, and routes every claim/confirm through claimNameFor(provider, uploadFilename). Two adjacent bugs fell out: api-data queued the raw leaf on its collision-cache paths, which would have dropped a metadataActive submission at the container root, and the request path and the retry worker claimed in different namespaces, so a queued retry never re-entered its own pending claim. Retry queue (2 findings). handleRetryFailure tiered on the code the doc was QUEUED with and never wrote the current attempt's back, pinning an item to whatever failed first. It now tiers on the attempt that just failed and stores it; a failure that never reached the provider clears the field and drops to the slow tier. RATE_LIMITED leaves the fast tier -- five attempts inside ~31 minutes is far short of a provider's rate-limit window, after which the item is marked failed and its cached payload deleted a week later -- and a Retry-After is clamped to MAX_BACKOFF_MS rather than the tier cap, so the provider's stated delay is no longer shortened. Metadata (2 findings). performUpdate discarded the WriteResult, so a metadataFileRef never followed Dataverse's delete-and-re-add to its new file id; every later submission then 404'd and self-healed by creating another dataset_description.json, which Dataverse renames rather than rejects. CONTENTION joins NON_HEALABLE_CODES: on Dataverse a contended update means the re-add lost a race, so re-creating immediately is a third write into the same contended container. Dataverse adapter (2 findings). The participant-supplied filename went raw into a Content-Disposition header, where a quote closed the parameter and a CRLF ended the header block; it is escaped now, and the fixed multipart boundary is per-request and random so the raw submission cannot close the part either. Missing response ids are omitted rather than stringified into the truthy "undefined", and createDataContainer rejects a 2xx body carrying no id instead of returning undefined fields Firestore refuses to store. Queue panel copy. A permanently failed row still read "it is being retried automatically". That reassurance now shows only while retries are running, failures carrying no taxonomy code are explained rather than printed raw, and Zenodo's QUOTA_EXCEEDED names its 100-file cap instead of claiming the account is out of space. PROVIDER_TOKEN_EXPIRED no longer says "Dataverse" when zenodo.ts emits the same code. Adds 36 tests, including metadata-ref-refresh.test.js, which drives blockMetadata against a fake adapter -- the existing OSF mock returns existingFileRef unconditionally and structurally cannot express a ref change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`npm run lint` ran `next lint`, which Next.js 16 removed, so the argument was read as a project DIRECTORY: "Invalid project directory provided, no such directory: <repo>/lint". Linting had in fact been inert for longer than that -- ESLint 9 uses flat config by default and only falls back to .eslintrc when ESLINT_USE_FLAT_CONFIG=false, so .eslintrc.json had already stopped being read. Replaces it with eslint.config.mjs built on eslint-config-next@16's flat exports, keeping the same rule set (next/core-web-vitals, then eslint-config-prettier, then react/no-unescaped-entities off) and the same lint surface `next lint` covered by default. Verified against the resolved config: 61 active rules, Next and rules-of-hooks at error, Prettier-owned stylistic rules off. Two rules are set to "warn" rather than the error they default to: react-hooks/set-state-in-effect and react-hooks/purity, both new in eslint-plugin-react-hooks v7, which arrived with eslint-config-next@16 and did not exist under the config this replaces. They flag seven real findings across Navbar, ChangePassword, the admin pages, reset-password and OAuthTokenStatus. Each needs a behavioral look at the effect involved, so they belong in their own change rather than riding along with a build-tooling fix -- the config comment lists them and says to promote the rules back to error once they are addressed. No CI workflow referenced the lint script, so nothing was failing on this; the command was simply unusable locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixes the seven findings the previous commit parked as warnings, then removes the downgrade so react-hooks/set-state-in-effect and react-hooks/purity are enforced at their default error severity. Derived state that was being mirrored into useState by an effect is now computed during render: ChangePassword's passwordMatch and passwordLengthSatisfied, and reset-password's view state, which is derived from the URL with only the page's own "email sent" transition kept in state. Both had a visible symptom. ChangePassword initialised passwordLengthSatisfied to true, so an empty field rendered as valid until the effect ran; reset-password rendered the "forgot password" form and swapped to the token form a beat later, so a user arriving from a reset link saw the wrong form flash by. Client-only values that cannot be read during SSR now go through useSyncExternalStore rather than a setState-in-effect flag: Navbar's hydration guard, and the admin OAuth banner's localStorage dismissal, which needs a listener set so dismissing it re-renders. The banner had the same wrong-for-one-render problem -- it defaulted to dismissed, so a researcher who had never dismissed it got no banner until the effect corrected it. The queue-resolved notice on the experiment page compares against the previous queue length during render, React's documented way to adjust state when a value changes, with the 8-second auto-hide left in an effect keyed off the flag. This also fixes a latent bug: the old effect returned its timer cleanup BEFORE recording the new count, so prevQueueCount kept a stale non-zero value after a transition fired. The two exhaustive-deps warnings turned out to be load-bearing, and the rule's suggestion is wrong here. Adding `router` to the provider-connect effect hangs its test suite outright: the useRouter mock returns a new object per call, the effect re-runs on every render, and because it dispatches, every run schedules the next -- an unbounded loop re-running the OAuth token exchange. Both callback pages now depend on `push` alone, destructured, since the query parameters are already listed individually. pages/oauth2/callback.js keeps `user?.uid` over `user` behind a documented eslint-disable: useAuthState hands back a fresh User object on every token refresh, and re-running a completed OAuth exchange because an object identity churned is not what the dependency is for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dataverse and Zenodo storage adapters
OSF is shutting down its projects feature. New experiments can no longer be created against it; experiments already collecting are untouched and keep writing until the sunset date. The gate that matters is in firestore.rules, not the UI. OSF experiments were created browser-side -- lib/experiment-creation.js wrote the document with the client SDK -- so removing the option from pages/admin/new.js only hides it. `allow create` now requires a storageProvider that is not 'osf'. It also requires the field to be PRESENT: absent used to mean OSF by default (see getProviderForExperiment in functions/src/providers/index.ts), which would otherwise have left a second, quieter way onto OSF. Updates stay legacy-tolerant so in-flight studies remain editable. firestore.rules also carries a relaxation of isAccountCreation() here rather than in its own commit: the two changes are one file and one deploy unit. It stops requiring osfToken == '' when the field is absent, which a new account no longer has. Experiment creation is now server-side for every provider. pages/osf-entry.js and lib/osf-utils.js go with the browser-driven path -- the entry point only ever created NEW OSF experiments, so it is dead the moment the rule lands. The pinned OSF-form regression test is inverted rather than deleted, so a reintroduction is caught. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds lib/auth-providers.js, a registry shaped like the existing STORAGE_PROVIDERS map so the two read alike. Every entry resolves to a Firebase AuthProvider, so all call sites are uniform regardless of provider: signInWithPopup / linkWithPopup / unlink. Adding a fourth provider later is one registry entry, one icon, and console config. No server code participates in sign-in at all. ORCID needs Identity Platform's generic OIDC support (Firebase has no native ORCID provider) and is registered as `oidc.orcid` against issuer https://orcid.org, code flow. Two details come straight from ORCID's discovery document and are load-bearing: - scopes_supported is ONLY "openid", and ORCID will not issue an id_token unless it is requested, so the adapter asks for it and nothing else. - claims_supported has NO email claim. An ORCID sign-in therefore always yields user.email === null -- not just when a researcher marks their address private. Hence providesEmail: false, and ensureUserDocument writing email: "" rather than assuming. GitHub asks for user:email, which it otherwise withholds unless the researcher has made the address public. ensureUserDocument (lib/user-bootstrap.js) creates users/{uid} on a first federated sign-in. Nothing else would: that document was only ever written by the signup form or the OSF callback, and federated sign-in goes through neither. The read-before-write is required by firestore.rules, not an optimization -- an unconditional merge over an existing document would put fields like connectedAccounts into request.resource.data and be denied. OSF sign-in is deliberately LEFT IN PLACE on the sign-in page and removed only from sign-up. Researchers who signed up through OSF still need a way in so they can link a new provider without losing their uid; no new account should be created against a platform that is closing. Deletes SignUpWithOSF, plus OneClickAuth and OSFToken, which were already unreferenced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lib/osf-sunset.js is the single source of truth for the wind-down: OSF_SUNSET_DATE (2026-11-16), a formatted label, and the two predicates that decide who sees the notices. isLegacyOsfExperiment treats an ABSENT storageProvider as OSF, mirroring getProviderForExperiment in functions/src/providers/index.ts -- if the two disagreed, the banner would go missing on exactly the oldest experiments. osfSunsetLabel formats in UTC deliberately. "2026-11-16" parses as UTC midnight, so formatting it in any US timezone -- where most of DataPipe's researchers are -- renders November 15, announcing a deadline a day earlier than the one agreed. Pinned by a test run under TZ=America/Los_Angeles. Adds OsfRelinkButton, extracted so OAuthTokenStatus can offer it too. That component previously told researchers with an expired grant to "sign out and sign back in with OSF" -- advice that stops working the day OSF sign-in is removed, silently killing any study whose token lapsed mid-wind-down. It now re-runs the authorization directly, which is the storage grant (the `linking` branch of oauth2-callback.ts) and never mints a session. Also makes the OSF sign-in button read as the legacy path it now is, and replaces two OSF-specific copy strings with provider-neutral ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Experiments are keyed by `owner: uid`, so any route that mints a fresh
uid severs a researcher from their own data. linkWithPopup attaches a
credential to the EXISTING Firebase user and leaves the uid alone, which
is why Identity Platform was chosen over a second custom-token flow: no
server-side account merge is needed at all.
AddSignInMethodBanner triggers on providerData being EMPTY. That is
exact rather than heuristic: OSF sign-in works by minting a Firebase
custom token, and a custom-token session carries no federated provider
and no password. Zero linked providers therefore means "this account is
reachable by the flow being removed and nothing else" -- precisely the
population that gets locked out. It disappears the moment they link
anything, with no flag to write or maintain, and is not dismissible.
LinkedAccounts lists methods from providerData crossed with the
registry, and refuses to unlink the last one -- otherwise a researcher
locks themselves out of an account that still owns their experiments.
The account page now decides whether to offer the password form from
providerData rather than the legacy users/{uid}.authMethod field, so a
researcher who signed up with OSF and has since added a password still
sees it. The OSF section moves below storage providers and appears only
for accounts that actually connected OSF.
scripts/backfill-osf-auth-emails.mjs is the escape hatch for researchers
who never return before the cutoff. Their Auth records were created by
createCustomToken and carry no email at all, so today they have no way
to reclaim the account; copying the address from Firestore gives them
password-reset and email-link routes back to the SAME uid. Dry-run by
default. It refuses to guess on two populations and reports them for
manual handling: synthetic user-<osfId>@osf.io placeholders (written
when OSF's emails endpoint failed, not real inboxes) and addresses
already owned by an email/password account.
Replaces the dashboard's old banner, which urged researchers to switch
between two OSF token methods -- both now legacy, so promoting either
would push people further onto the platform they need to leave.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Remove OSF: close it to new experiments, and replace OSF sign-in
The new "Sign in with ..." buttons, the OSF sign-in button, the OSF
re-authorize button and the Unlink button were all effectively
invisible: dark text on greyBackground (#1C1F22).
Cause: Chakra v3's `outline` recipe sets
`color: var(--chakra-colors-color-palette-fg)`. With no colorPalette
that resolves to the DEFAULT GRAY palette's fg = gray.800. lib/theme.js
already re-points the semantic `fg` token light, but not the gray
palette's fg, which is what unpaletted recipes actually read.
The non-obvious part is why the usual fix does not work. A plain
`color="white"` prop -- the pattern used elsewhere in this codebase --
compiles into the SAME emotion class as the recipe, and the recipe's
declaration is emitted AFTER it, so at equal specificity the recipe wins
and the override silently does nothing. `css={{ color: "white" }}`
behaves identically. Verified by reading the emitted stylesheet: the
prop rule sat at byte offset 69676 and the recipe rule at 69819 for the
identical class name.
Fixed by doubling the selector (`&&`), giving 0-2-0 against the recipe's
0-1-0, which wins whatever the emission order. Confirmed against the
rendered HTML: all four buttons on /signin and /signup now resolve to
var(--chakra-colors-white).
Shared as `outlineOnDark` from lib/theme.js so the four call sites
cannot drift.
NOTE: this same trap likely affects the pre-existing outline buttons in
components/CopyButton.js and pages/admin/index.js, which use the
color="white" prop that this change shows does not win. Not touched here
-- they are behind auth and were not part of the reported problem.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix: outline buttons rendered near-black on the dark background
Fixes the cause rather than the symptom, and reverts the per-button workaround from the previous commit. DataPipe renders on a permanently dark surface (globalCss sets body to greyBackground #1C1F22) while Chakra's colour mode is light, so every palette resolves its _light values. For the GRAY palette those are built for a white page: gray.fg = gray.800 = #27272a, which is 1.11:1 against the body. Any component that does not name a colorPalette falls back to gray, so an unstyled `variant="outline"` or `variant="ghost"` button was effectively invisible. gray.fg is now gray.200 -- 13.05:1. The whole palette moves, not just fg: variants read different tokens, so lightening fg alone would leave `subtle` painting light text on the near-white gray.subtle background. gray.border is gray.500 rather than gray.600 because WCAG 1.4.11 asks 3.0 for non-text UI boundaries and 600 measured 2.14:1 while 500 gives 3.43:1. CORRECTION to the previous commit's reasoning. It claimed a style prop cannot beat the recipe, and that the outline buttons in Footer.js, CopyButton.js and dashboard/Title.js were therefore also broken. That was wrong -- Footer's button carries color="white" and does render white. The real defect was narrower: the new auth buttons set NO colour at all and so inherited the bad default. Components that set an explicit colour were never affected and are unchanged here. The `&&` double-specificity override and the `outlineOnDark` helper it needed are removed; the four call sites are plain `variant="outline"` again and are legible from the theme alone, as is any future one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix: re-point the gray palette for this app's dark surface
Linking GitHub to an ORCID-created account failed with auth/email-already-in-use and showed "Sign in using the method you set up originally, then add GitHub from your account settings" -- advice for a visitor at the sign-in page, shown to a researcher who was already signed in and already on the account page. The underlying refusal is correct. Firebase allows one account per email address, so a GitHub credential whose email belongs to another DataPipe account cannot be attached to this one. ORCID is what makes this reachable: it returns no email claim, so an ORCID account starts with none and the first federated provider linked to it is the first chance for a collision with an account the researcher already had. messageForAuthError now takes the operation as a third argument and the three call sites pass it. Only the collision case and the fallback differ; everything else reads the same in all three modes. The linking copy says outright that there is no self-service merge -- experiments are keyed by `owner: uid`, so combining accounts is a maintainer operation -- rather than implying a retry will help. Also adds auth/no-such-provider, which unlink can raise and which was falling through to a message about sign-in. lib/auth-errors.js had no test file; it has one now, pinning the wording that went wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Linking GitHub to an ORCID-created account failed with auth/email-already-in-use and showed "Sign in using the method you set up originally, then add GitHub from your account settings" -- advice for a visitor at the sign-in page, shown to a researcher who was already signed in and already on the account page. The underlying refusal is correct. Firebase allows one account per email address, so a GitHub credential whose email belongs to another DataPipe account cannot be attached to this one. ORCID is what makes this reachable: it returns no email claim, so an ORCID account starts with none and the first federated provider linked to it is the first chance for a collision with an account the researcher already had. messageForAuthError now takes the operation as a third argument and the three call sites pass it. Only the collision case and the fallback differ; everything else reads the same in all three modes. The linking copy says outright that there is no self-service merge -- experiments are keyed by `owner: uid`, so combining accounts is a maintainer operation -- rather than implying a retry will help. Also adds auth/no-such-provider, which unlink can raise and which was falling through to a message about sign-in. lib/auth-errors.js had no test file; it has one now, pinning the wording that went wrong. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Account deletion ran client-side -- deleteUser(auth.currentUser) destroyed
the Firebase Auth record first, and cleanup hung off the onUserDeleted
trigger afterwards. That order cannot be made safe: once the auth record is
gone, a cleanup that does not run leaves the researcher's data with no owner
and no way for them to sign back in and retry.
datapipe-test carries the residue. Two users/ documents belong to uids with
no auth record, and so do two experiments -- one of them still `active`.
An orphaned active experiment is not inert: api-data.ts persists each
submission to Cloud Storage (persistPending) BEFORE it checks that the owner
still exists, so every submission to an orphan leaves a file behind and then
answers 400.
Deletion now runs in functions/src/delete-account.ts: purge first, delete
the auth record only once the purge returns. Failing now leaves the account
intact and the operation retryable.
The purge itself moves to purge-user-data.ts and fixes two gaps in what the
trigger used to do:
- It found experiments through users/{uid}.experiments, a client-maintained
array. Anything missing from it survived its owner. It now queries
`where owner == uid`, which cannot drift.
- It deleted the experiment document but not the filenameClaims
subcollection beneath it. Firestore does not cascade.
It also now clears uploadQueue entries and pending-data/ objects, neither of
which was touched before.
onUserDeleted stays as a backstop for deletions that never reach the
endpoint -- the console, the Admin SDK, a support action. purgeUserData is
idempotent, so the trigger firing after deleteAccount has already purged is
harmless.
Moving the call server-side would have quietly dropped the recent-login
requirement that client-side deleteUser enforced for us, so the endpoint
checks the token's auth_time against Firebase's own five-minute threshold and
verifies with checkRevoked. The client maps that to "sign out and sign back
in" instead of the old bare tooltip, which showed a raw Firebase error
message in red 12px text.
functions/scripts/purge-orphaned-users.mjs cleans up what the old order
already left behind: dry-run by default, and it treats an account as orphaned
only on an explicit auth/user-not-found -- any other error is reported as
undetermined and skipped.
Both admin-SDK scripts move under functions/, where firebase-admin actually
resolves. They imported it from scripts/, which has no node_modules, so
neither could ever have run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: give account-linking failures their own wording
Linking GitHub to an ORCID-created account failed with
auth/email-already-in-use and showed "Sign in using the method you set up
originally, then add GitHub from your account settings" -- advice for a
visitor at the sign-in page, shown to a researcher who was already signed
in and already on the account page.
The underlying refusal is correct. Firebase allows one account per email
address, so a GitHub credential whose email belongs to another DataPipe
account cannot be attached to this one. ORCID is what makes this reachable:
it returns no email claim, so an ORCID account starts with none and the
first federated provider linked to it is the first chance for a collision
with an account the researcher already had.
messageForAuthError now takes the operation as a third argument and the
three call sites pass it. Only the collision case and the fallback differ;
everything else reads the same in all three modes. The linking copy says
outright that there is no self-service merge -- experiments are keyed by
`owner: uid`, so combining accounts is a maintainer operation -- rather
than implying a retry will help.
Also adds auth/no-such-provider, which unlink can raise and which was
falling through to a message about sign-in.
lib/auth-errors.js had no test file; it has one now, pinning the wording
that went wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: delete accounts server-side, and purge everything they own
Account deletion ran client-side -- deleteUser(auth.currentUser) destroyed
the Firebase Auth record first, and cleanup hung off the onUserDeleted
trigger afterwards. That order cannot be made safe: once the auth record is
gone, a cleanup that does not run leaves the researcher's data with no owner
and no way for them to sign back in and retry.
datapipe-test carries the residue. Two users/ documents belong to uids with
no auth record, and so do two experiments -- one of them still `active`.
An orphaned active experiment is not inert: api-data.ts persists each
submission to Cloud Storage (persistPending) BEFORE it checks that the owner
still exists, so every submission to an orphan leaves a file behind and then
answers 400.
Deletion now runs in functions/src/delete-account.ts: purge first, delete
the auth record only once the purge returns. Failing now leaves the account
intact and the operation retryable.
The purge itself moves to purge-user-data.ts and fixes two gaps in what the
trigger used to do:
- It found experiments through users/{uid}.experiments, a client-maintained
array. Anything missing from it survived its owner. It now queries
`where owner == uid`, which cannot drift.
- It deleted the experiment document but not the filenameClaims
subcollection beneath it. Firestore does not cascade.
It also now clears uploadQueue entries and pending-data/ objects, neither of
which was touched before.
onUserDeleted stays as a backstop for deletions that never reach the
endpoint -- the console, the Admin SDK, a support action. purgeUserData is
idempotent, so the trigger firing after deleteAccount has already purged is
harmless.
Moving the call server-side would have quietly dropped the recent-login
requirement that client-side deleteUser enforced for us, so the endpoint
checks the token's auth_time against Firebase's own five-minute threshold and
verifies with checkRevoked. The client maps that to "sign out and sign back
in" instead of the old bare tooltip, which showed a raw Firebase error
message in red 12px text.
functions/scripts/purge-orphaned-users.mjs cleans up what the old order
already left behind: dry-run by default, and it treats an account as orphaned
only on an explicit auth/user-not-found -- any other error is reported as
undetermined and skipped.
Both admin-SDK scripts move under functions/, where firebase-admin actually
resolves. They imported it from scripts/, which has no node_modules, so
neither could ever have run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…tudies Zenodo allows 100 files per record and refuses the 101st, so a study simply stopped at ~session 100: the write mapped to QUOTA_EXCEEDED, a slow-tier queue failure no amount of retrying could clear. Zenodo's keyspace is also flat, so a metadataActive record showed data_raw_subject-1.json where Psych-DS calls for data/raw/subject-1.json. Both are now handled by the same mechanism. Compaction (during collection). Older sessions are sealed into a batch zip that carries the real Psych-DS tree, then the loose originals are deleted. Ordering is the safety property and is not negotiable: upload, verify the provider's reported md5, seal the claims, and only then delete. DataPipe keeps no copy of submitted data, so a delete before a verified upload is unrecoverable. Batch membership is recorded before the upload, so an interrupted pass resumes rather than sealing the same sessions into a second zip -- as hashes, not filenames, preserving the collision cache's privacy property. Archived claims lose their TTL. A confirmed claim normally expires after 90 days, which is only safe because a cold cache rehydrates from the provider's listing; an archived file is not in that listing, so an expiring claim would silently re-open a filename collected months earlier. Discovery is event-driven -- there is no cron. DataPipe is the only writer to these containers, so it already knows when one has grown. Two Firestore triggers replace what began as a 6-hour poll: `sessions` incrementing on an experiment, and upload-queue writes (which catch a draining backlog and a provider reporting the record full). An idle study now costs zero provider listings; an active one is examined on every change. A write gate closes the remaining hole. While a pass holds the lease, submissions divert to the durable upload queue instead of the provider, so the file count cannot grow mid-pass. It costs nothing: every caller already loads the experiment document. A record that fills anyway borrows .psychds-ignore's slot -- its content is a fixed constant, so giving it up loses nothing. Finalization (end of study). One merged archive holding the complete Psych-DS tree, built by streaming into Cloud Storage and uploaded from a stream, so archive size is bounded by the provider's 50 GB per-file limit rather than by function memory. A split would break the Psych-DS compatibility the archive exists to provide, so it must stay unreachable in practice. Finalization is permanent: the experiment stops accepting submissions, the retry worker refuses to write into a sealed record, and firestore.rules blocks a client from clearing the flag through the client SDK. Researcher-facing: a dashboard control with an explicit confirmation, and an FAQ entry explaining why adding files to provider storage during collection is unsupported -- it also desynchronizes the collision cache. Zenodo's setupWarnings stopgap is removed; it told researchers to stay under 100 submissions, which is no longer true. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`firebase deploy` submits index definitions and returns; the builds run asynchronously for minutes while the newly deployed functions are already serving. A query needing an index that has not finished building fails with FAILED_PRECONDITION. releaseHeldUploads runs on nearly every compaction path, including the common below-watermark one, and sat outside runCompaction's try/catch -- so during that window it would propagate out of compactExperiment, fail the Firestore trigger that called it, and be retried for up to seven days, for every experiment-document update on every capped-provider experiment near the watermark. It is an accelerator, not a correctness step: the entries it releases are already on the 60-second fast tier or will be retried regardless, and the compaction itself has committed by the time it runs. Losing it costs a slightly slower queue drain. So it is now best-effort and logs instead of throwing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
origin/test had squash-merged equivalents of this branch's two account fixes (#162, #163), which collided with the same files here. Only firebase.json needed a manual resolution: both sides appended to the hosting rewrite list, so the /api/finalize entry and the /api/deleteaccount entry were each present on one side only. Kept both. functions/src/index.ts auto-merged; verified every import has a matching export and all four new functions are still registered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat: archive compaction and end-of-study finalization for capped providers
lib/provider-config.js hardcoded https://zenodo.org and rendered no server field, so connecting Zenodo on datapipe-test created real depositions on the live service using the researcher's real account. There was no way around it from the UI either: pasting a sandbox token is rejected at connect time, because the connect endpoint validates it against zenodo.org. NEXT_PUBLIC_ZENODO_ENV now selects the host the same way NEXT_PUBLIC_OSF_ENV already does -- a prefix, "" on production and "sandbox." on the test site. Both deploy workflows set it explicitly rather than relying on the default, so production cannot inherit a sandbox value by accident. No backend change was needed: zenodo.ts already allowlists both hosts and resolves the server from the stored connection, so this only decides where NEW connections point. containerLink now takes its host from the CONTAINER rather than a constant, matching dataverse's already-tested behavior. An experiment created before a deployment was switched still lives where it was created, so its link has to follow the data rather than today's configuration. The host is resolved once at module scope. defaultServerUrl is evaluated when the module loads and containerLink when it is called, so reading process.env separately in each let them disagree -- which a test caught. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat: point the test deployment at the Zenodo sandbox
Compaction and finalization added deleteFile, writeStreamedFile and downloadFileBytes, and all three have only ever run against a local Express mock. Every bug the original Zenodo spike caught -- the 415 on content-type, the encoded-slash 404, the exceeding/exceeds regex miss -- was adapter-level and invisible to a mock built from the same assumptions as the code. F. deleteFile. Confirms a delete removes the object, and records what Zenodo returns for deleting a key that is already gone. The adapter reports that as SUCCESS deliberately, because compaction resumes an interrupted pass by re-deleting whatever is left; if Zenodo answers differently the resume path stops being idempotent. G. writeStreamedFile. The least-proven path in the feature, and finalization depends on it completely. Sends 2 MB in 64 KB chunks and checks three things separately: that undici accepts the stream at all, that the stored bytes match, and that the response still carries a checksum -- without one, compaction can never authorize deleting the originals. H. downloadFileBytes. Round-trips deliberately invalid UTF-8 and also reads the same object through downloadFile, so the corruption that motivated the second method stays visible rather than asserted. I. checksum format. The quietest failure available: a bucket PUT reports "md5:<hex>", and crash-resume compares the LISTING endpoint's checksum against a recorded md5. If the two disagree on format, every interrupted pass silently discards its archive and rebuilds instead of resuming, with nothing in the logs. F-I run before E, which fills the record to its 100-file cap and so must stay last. Noted in the header, since the first draft of this had them after E where they would have failed for lack of room. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gate F caught this on its first live run: Zenodo's bucket DELETE returns 500 INTERNAL SERVER ERROR while actually removing the object. Reproduced consistently -- 3/3 further deletes, all 500, all genuinely gone. This is a service BUG, not a contract. Zenodo documents 204 for its deposition-files delete and does not document the bucket delete this adapter uses at all, so there is no documented status to conform to. Matching reports: zenodo/zenodo#2502 and #2506. The consequences were real. sealAndDelete would have reported every file in a batch as undeleted, and worse, the saturation path deletes .psychds-ignore to free a slot for the archive and ABORTS if that reports failure -- so a full record would have refused to compact while having actually freed the slot. Special-casing 500 is wrong in both directions: hardcoding it as success would mask a genuine outage, and trusting it means believing a delete failed when it did not. So on any non-2xx, non-404 response the adapter now asks what is actually in the deposition, and reports success if the object is gone. Correct whether the bug is present, fixed, or intermittent; the check never runs once Zenodo returns 204. Gate F also under-reported -- it printed only delete.success, not the mapped status and message, which is what a diagnosis actually needs. Fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test: spike gates for the adapter methods that have never run against real Zenodo
The spike scripts verify adapter methods against a real provider; this covers the layer above them -- a full compaction cycle and finalization driven through the deployed HTTP API, including whether the Firestore triggers fire at all, which nothing local can test. Most of it needs no credentials. /api/data is public by design (participants' browsers post to it), so the load, the duplicate-rejection check and the post-finalization rejection are all observable from status codes alone. That matters most for the duplicate check: after compaction a filename exists ONLY inside the archive, so if the sealed claims were lost the resubmission is accepted as new and the duplicate quietly lands. It is the feature's quietest failure and it is fully detectable from outside. ZENODO_TOKEN/DEPOSITION_ID additionally enable the provider-side checks -- file count dropping, the batch archive appearing, and downloading it to confirm the data/raw/ paths that Zenodo's flat keyspace cannot hold. That uses the shipped readArchive, so the verification runs the same code compaction does. ID_TOKEN enables the finalize phase. Both degrade to explicit SKIPs rather than silently passing. Polls for the archive rather than assuming a submission count: how many files a submission produces depends on how many sidecar CSVs the data yields. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by the first live run. An experiment sitting at ~22 of 100 files started returning 202-queued to participants with failureReason "Compaction in progress" while nothing was being compacted at all. compactExperiment is called from a Firestore trigger on EVERY submission, and it acquired the lease before deciding whether there was anything to do -- holding it across a token resolve and a listFiles round trip to the provider. The lease is not a bookkeeping marker: it is what makes compaction-gate.ts divert live submissions into the upload queue. So every participant was paying for a question whose answer is almost always no, and under the burst profile in requirement 6 a large share of submissions would have been diverted. No data was at risk -- the queue is durable and drains on the 60-second fast tier -- but people were being told to wait on a no-op. Everything up to the decision is now read-only and lease-free. The lease is taken only when there is genuinely work, and the listing is retaken under it so no decision rests on state that can still move. The no-work path records what it saw through a new noteCheck rather than releaseLease. Reusing releaseLease would clear compactingUntil and cancel a lease this call never owned -- a worse bug than the one being fixed. Emulator tests could not have caught this: they call compactExperiment sequentially, so nothing ever races a no-op check. The regressions added here test the property that matters instead -- that the write gate stays OPEN during a check, sampled through the same field isCompactionInFlight reads, and that a real pass still closes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test: live end-to-end check against a deployed DataPipe
POST /api/data is anonymous -- participants' browsers call it directly --
and it accepted an optional metadataOptions body object that flowed
straight through persistPending/blockMetadata into produceMetadata as the
seed descriptor for the vendored @jspsych/metadata generate(). Every key
landed verbatim in dataset_description.json, which merges into the
persistent metadata/{experimentId} doc and uploads to the researcher's
storage -- so any participant could set or overwrite a dataset's name,
author, license or @context, and the value would persist into every later
merge.
Removed the parameter entirely rather than sanitizing it: there is no
legitimate participant-supplied metadata field, so the fix is to close the
channel, not filter it. A researcher-controlled equivalent on the
experiment document is separate follow-up work.
- api-data.ts / api-base64.ts: stop reading metadataOptions from the body
and stop passing it to persistPending; a body that still includes the
key is now silently ignored (dropped by destructuring), so old scripts
keep working.
- metadata-block.ts / metadata-production.ts: dropped the parameter from
blockMetadata/produceMetadata; generate() now always sees the empty seed
it already used on the no-options branch.
- persist-pending.ts: removed metadataOptions from the PendingEnvelope
writer and its positional argument list. The reader tolerates envelopes
already in the bucket that still carry the field -- readPendingEnvelope
never destructured it, so an old envelope's stray key is just ignored
until it ages out under the normal retention window.
- interfaces.ts: dropped metadataOptions from RequestBody.
- pages/docs/api.js: removed the documented parameter row.
- docs/provider-migration-design.md: noted the 2026-09-15 removal under
Metadata-file tracking.
Updated every test that called the removed parameter (metadata-ref-refresh,
scheduled-pending-recovery-emulator, payload-encryption-emulator) and
rewrote metadata-production.test.js's "provided options" case to assert
the opposite -- a second argument is now inert. Added a regression case to
metadata-emulator.test.js that POSTs metadataOptions: { name: "hijacked",
author: "attacker", license: "CC0" } against a metadata-active experiment
and asserts neither the created dataset_description.json body nor the
persisted Firestore metadata doc contains any of those values (extended
the file's mock OSF server to capture PUT bodies, not just call counts).
`cd functions && npm run build` and `npm run lint` (repo root) are both
clean. Pure suites re-run from the repo root, all passing: metadata-
production (11), metadata-update (auto), metadata-derived-files (auto) --
50 tests total across the three files.
Emulator-backed files touching this change, for the parent to run:
metadata-emulator, metadata-ref-emulator, metadata-ref-refresh,
metadata-derived-upload-emulator, skip-metadata-emulator, data-emulator,
scheduled-pending-recovery-emulator, early-persist-emulator,
payload-encryption-emulator.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
…efixes storage.rules already denied all client read/write, but neither deploy workflow ever included `storage` in its --only list, so the deny-all rules had never actually been deployed to either bucket's default rules (Firebase's own default -- allow read, write: if request.auth != null -- would apply instead). This release starts routing participant submissions through the default bucket (persist-pending.ts under pending-data/, queue-upload.ts under upload-queue/, finalization.ts's scratch object under finalization/, all read back by scheduled-pending-recovery.ts, scheduled-upload-retry.ts, api-queue-status.ts and purge-user-data.ts), so any signed-in user could otherwise enumerate and read other researchers' pending data with a plain Storage SDK call. Confirmed no client code uses the Storage SDK (grepped pages/components/lib for getStorage and firebase/storage: no hits), so the fix is deploying the existing deny-all rules, not relaxing anything. - storage.rules: unchanged rule, added a header comment listing the three known Admin-SDK-only prefixes and stating nothing here is meant to be client-reachable. - .github/workflows/firebase-deploy.yml and firebase-deploy-test.yml: added `storage` to the `--only` list, ordered before `functions` (firebase-tools deploys --only targets in the order given) so the rules are in place before functions keep writing to the bucket. `firebase deploy --only storage` requires the default bucket to already exist, which is true for both osf-relay.appspot.com and datapipe-test.appspot.com; noted in the new workflow comments. - __tests__/storage-rules.test.js: new suite, sibling of __tests__/firestore-rules.test.js and __tests__/database-rules.test.js, using @firebase/rules-unit-testing's storage option (confirmed supported by the installed 5.0.0 -- its public types export storage on both TestEnvironmentConfig and RulesTestContext) against the storage emulator on port 9199. Asserts unauthenticated and authenticated clients are both denied getBytes/list on pending-data/x/y and upload-queue/z, and denied uploadBytes anywhere. node.js.yml's `firebase emulators:exec` already starts every emulator in firebase.json, storage included, so no CI change was needed to run it. - docs/provider-migration-design.md: the gdrive deployment checklist's "Deploy order" item now spells out that "rules" includes storage, not just firestore/database. `npm run lint` is clean. Workflow YAML validated with js-yaml (all three under .github/workflows/ parse). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
…laims TTL
Two Firestore configuration gaps, both invisible to the emulator (it
neither enforces composite indexes nor TTL policies):
1. api-queue-status.ts's list form (GET /api/queuestatus with no
`download`/`downloadAll`, documented at pages/docs/api.js's "Queue
status" section) runs
.where("experimentID","==",X).where("status","in",[...]).orderBy("createdAt","desc").
None of the three existing uploadQueue composite indexes serve it --
two constrain `owner` or `providerErrorCode` instead of `createdAt`,
the third only covers the retry sweep's (status, nextRetryAt) shape.
In production this fails with FAILED_PRECONDITION, and since the
.get() sat outside any try/catch, the endpoint 500s with a non-JSON
body. Added the (experimentID ASC, status ASC, createdAt DESC) index
to firestore.indexes.json and wrapped the query in the same
try/catch-then-JSON-500 shape the rest of the file already uses for
its download and downloadAll branches. The dashboard's own query
(pages/admin/[experiment_id].js) additionally filters on `owner`
before orderBy, which is why it was never affected.
2. collision-cache.ts's filenameClaims subcollection has always relied on
Firestore TTL to expire `expiresAt`, but the policy was never declared
anywhere -- docs/provider-migration-design.md's deployment checklist
told the operator to create it by hand in the console. Commit f0aafe0
already established why that doesn't work: `firebase deploy --only
firestore --force` reconciles fieldOverrides against
firestore.indexes.json, and an undeclared TTL is something the deploy
is actively instructed to delete. Declared it as a fieldOverrides
entry (collectionGroup filenameClaims, field expiresAt, ttl: true),
copying the mail/delivery.expireAt override's exact indexes shape so
default single-field indexing on the field isn't also switched off.
Rewrote the checklist item to match.
Audited every other Firestore query in functions/src, pages, components
and lib for the same two-or-more-field gap (query location -> fields ->
serving index):
- purge-user-data.ts (4 sites): single `owner`/`datapipe.owner` equality
-> automatic single-field index, no composite needed.
- api-queue-status.ts downloadAll: experimentID==, status in [...], no
orderBy -> served as a prefix of the (experimentID, status,
providerErrorCode) index.
- compaction.ts releaseHeldUploads: experimentID==, status==,
providerErrorCode in [...] -> exact match of the (experimentID, status,
providerErrorCode) index.
- compaction.ts (2 sites), finalization.ts, upload-retention.ts:
experimentID==/status== equality-only or single-field -> covered by
existing composites or automatic indexing.
- scheduled-mail-retry.ts (2 queries): delivery.state==+delivery.retryable==
and delivery.state==+delivery.leaseExpiresAt<= -> the two existing mail
composites.
- check-email-conflict.ts, oauth2-callback.ts (3 sites): single-field
equality -> automatic indexing.
- providers/gdrive.ts: single-field range -> automatic indexing.
- providers/osf.ts: usingPersonalToken==+refreshTokenExpires<= -> the
existing `users` composite.
- scheduled-upload-retry.ts: status==+nextRetryAt<=, orderBy(nextRetryAt)
-> the existing (status, nextRetryAt) composite; the retention sweep's
single-field createdAt<=+orderBy(createdAt) -> automatic indexing.
- pages/admin/[experiment_id].js: experimentID==+owner==+status in
[...]+orderBy(createdAt desc) -> the (experimentID, owner, status,
createdAt) composite (already correct -- this is the dashboard's
working equivalent of the broken query above); its second query
(experimentID==+owner==) -> the same composite's prefix.
- pages/admin/index.js, components/account/ProviderConnections.js:
single-field `owner` equality -> automatic indexing.
No other gap found; the review's findings are confirmed.
Tests: functions/src/__tests__/firestore-indexes.test.js is a new pure
test parsing the committed JSON directly, asserting the new uploadQueue
index, the new filenameClaims TTL override, and that the mail TTL
override survives future edits. functions/src/__tests__/api-queue-status-emulator.test.js
is new emulator coverage for the list endpoint (previously untested):
seeds two queue entries and asserts 200 with both present in createdAt
descending order, and 403 for a non-owner. It cannot prove the index
exists in production -- the emulator doesn't enforce composite indexes --
which is exactly why the pure test above exists.
`cd functions && npm run build` and `npm run lint` (repo root) are both
clean. firestore-indexes.test.js passes (3/3) run from the repo root.
The parent should run the two emulator-backed files:
functions/src/__tests__/api-queue-status-emulator.test.js (new) and
functions/src/__tests__/upload-queue.test.js (pre-existing, unaffected
but exercises the same uploadQueue collection).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
Production (main) has no `concurrency: 1` on these two functions, so it gets roughly 20 x 80 = 1,600 concurrent submission slots from the default concurrency of 80. 058a1db added `concurrency: 1` here for a sound reason (memory safety under concurrent large payloads on a 512MiB instance), but left the global `maxInstances: 20` (index.ts) as the only ceiling, cutting this branch to 20 slots service-wide -- a ~98% capacity regression hiding behind an unrelated perf commit. A lecture-hall-scale burst overflows that in seconds; Cloud Run sheds the rest as 429/504 with a non-JSON body, and the jsPsych plugin's unconditional response.json() loses the data client-side before anything was persisted. Added per-function `maxInstances` (apiData: 300, apiBase64: 100 -- lower, since base64 carries larger media payloads per request) overriding the global 20, keeping `concurrency: 1` exactly as-is. maxInstances is a ceiling, not a reservation, so idle-instance cost is unaffected; Cloud Run's default per-region instance quota comfortably covers both. Also added `timeoutSeconds: 300` (up from the 60s default) to both. Neither endpoint declared one, so collision-cache.ts's rehydrate() -- called from inside the request when the collision cache is cold, listing every file a provider container holds and bulk-writing one Firestore claim per file in batches of 500 -- was killed mid-pass on a legacy experiment with thousands of files. REHYDRATION_LEASE_MS was also 60s, expiring at the same moment the instance died, so the next submission just repeated the same too-slow rehydration forever: a permanent per-experiment stall. Requests through Firebase Hosting's /api/data and /api/base64 rewrites are still capped at a fixed 60 seconds regardless of this value (documented on pages/docs/api.js's "Limits" section; firebase-tools' own MAX_V2_HTTP_TIMEOUT_SECONDS of 3600s caps only the function's own timeout, not what Hosting waits for) -- a participant whose request triggers a cold rehydration will likely still see a 504. timeoutSeconds: 300 still fixes the stall: Cloud Run keeps the instance running server-side past that client disconnect, so rehydrate() finishes writing the cache, and every submission after that one finds it warm instead of repeating the doomed attempt on every request. Raised REHYDRATION_LEASE_MS to 330s (300s timeout + 30s margin) to match -- it must outlive the longest request that could legitimately still be holding it. On its own that would make a DEAD holder (an instance hard-killed mid-rehydration, which never runs a `finally` block to clear its own lease) block every other submission to the experiment for up to 5.5 minutes instead of the previous 60 seconds. Added a heartbeat inside rehydrate() (a new REHYDRATION_HEARTBEAT_MS, 45s, renewed on a timer every 15s across the whole call, listFiles() included) so a live holder's stored lease never drifts far from 45s stale, and a dead holder's lease is picked up within about that same window instead of the full 5.5 minutes. Not implemented, left as a follow-up: capping how many files a single rehydration claims per request and resuming the rest on a later request. listFilesFn is a single non-paginated call into each provider adapter today, so bounding it would mean adding cursor/pagination support to every adapter -- a real feature, not a small change alongside this fix. Added functions/src/__tests__/function-capacity-options.test.js (pure -- imports the built api-data.js/api-base64.js and asserts maxInstances, timeoutSeconds and concurrency on their firebase-functions v2 `__endpoint`) and rehydration-lease-timing.test.js (pure -- asserts REHYDRATION_LEASE_MS/REHYDRATION_HEARTBEAT_MS's relationship to the 300s timeout). Extended collision-cache.test.js (emulator-backed) with a test that the heartbeat actually renews mid-rehydration. `cd functions && npm run build` and `npm run lint` are clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
package.json claims MIT, but the license text lives at the repository root and `files: ["dist"]` means the published tarball carried neither. npm always includes a LICENSE file from the package directory, so a copy here is what actually reaches anyone who installs it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SgDfss2zhjwMky2mBKbcBX
npm trusts one named workflow file in one named repository and trades the
OIDC token GitHub mints for that run for a credential good for a single
publish. Nothing publish-capable is stored in this repository's secrets,
and a leaked workflow log cannot be replayed into a release.
Two things this binds that a token did not:
- The workflow's FILENAME is part of the trust. Renaming or moving
release-client.yml revokes publishing until npm is updated to match,
and it fails as an authentication error rather than a configuration
one, so both the workflow and the release doc say so.
- npm 11.5.1 is the floor for OIDC and Node 22 still ships npm 10, so
the CLI is upgraded explicitly. Without it the publish falls back to
looking for a token and dies with ENEEDAUTH -- which reads like a
missing secret and sends you looking in the wrong place. Pinned
rather than @latest, since this step is all that stands between a
green build and a version number that can never be reused.
Provenance comes with trusted publishing, so NPM_CONFIG_PROVENANCE is
gone too.
docs/releasing-the-client.md carries the one-time setup, including the
part that cannot be automated: a trusted publisher is configured on a
package's settings page, so the package has to exist first. 0.1.0 has to
be published by hand, which is also what claims an unscoped name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgDfss2zhjwMky2mBKbcBX
npm has a CLI for this:
npm trust github datapipe-client \
--repo jspsych/datapipe --file release-client.yml --allow-publish
with --dry-run to preview, `npm trust list` to see what is configured and
`npm trust revoke --id=<id>` to undo one. The setup doc walked through
the website UI instead, which is more steps and leaves nothing to check
the result against.
Also softens the ordering claim it made. It asserted that a trusted
publisher cannot be attached to a package that does not exist yet, so
0.1.0 must be published by hand first. That may not hold -- the CLI
takes an optional package argument and may register ahead of the first
publish -- so the doc now says to try the trust command first and fall
back to a manual publish only if npm refuses.
npm pinned to 11.17.0, the version the command was verified against
locally, rather than the older 11.6.2 that only just cleared the 11.5.1
OIDC floor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgDfss2zhjwMky2mBKbcBX
The heartbeat added for the lease fix (previous commit) called clearInterval() at each individual exit point (the listFilesFn failure branch, the success branch, and again in an outer finally) instead of in exactly one place. Running the full suite under emulators:exec surfaced "Jest did not exit one second after the test run has completed" and then an outright hang requiring a kill -- collision-cache.test.js's new heartbeat test deliberately lets a real tick fire, and the redundant early-return clearInterval calls made it harder to be sure every exit path actually reached one. Consolidated to a single clearInterval() in one `finally` wrapping the whole rehydrate() body, so every exit -- success, the listFilesFn failure, or any other error thrown out of the batch-write loop -- passes through it. Kept `.unref()` on the interval (already present) so it can never be the reason a process fails to exit on its own. Replaced the two early clearInterval() calls with a `stopped` boolean the heartbeat callback checks before writing, preserving the original protection against a tick already in flight resurrecting a rehydratingUntil this call just deleted, without needing more than one clearInterval() call to audit. Verified: `npx jest --detectOpenHandles functions/src/__tests__/function-capacity-options.test.js functions/src/__tests__/rehydration-lease-timing.test.js` exits cleanly (neither file calls rehydrate(), so neither could have been holding a heartbeat handle open, but both are confirmed clean regardless). Re-read collision-cache.test.js's new heartbeat test (#14): it already awaits claimPromise to completion before the test ends, so it does not leave a rehydration in flight. `cd functions && npm run build` and `npm run lint` are clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
It was named three times and linked from none of them, while @jspsych/extension-pipe two paragraphs away had a link to its source. A researcher who read that a library existed had nowhere to click, and `npm install datapipe-client` appeared nowhere on the site at all -- the script tag existed only inside the dashboard's code panel. Deliberately not a full API reference page. packages/client/README.md already is one, and a hand-maintained second copy would drift from it within a release or two with nothing to catch it -- the same reason the code samples were extracted into a module a test can execute. So this covers installation and links out for the surface. The two things it does spell out are the ones a reader cannot get from the sample and that fail quietly when missed: getCondition throws while nothing else in the library does, and flush() has to precede reading sessionId or the submission goes unmatched and the staged copy comes back as a spurious .partial.json next to a complete file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SgDfss2zhjwMky2mBKbcBX
…nto raise-submission-capacity # Conflicts: # functions/src/api-data.ts
CI caught this: the app's suite failed with "Cannot find module 'vitest'" on packages/client/test/*.test.ts. Jest's testMatch is purely path-based, so adding any directory under packages/ puts its tests in the app's run regardless of what runner they belong to. Worth spelling out in the comment, because the obvious inference is wrong: this has nothing to do with npm workspaces. packages/client is deliberately not a workspace member, and testMatch does not know what a workspace is -- the file being on disk under rootDir is the whole qualification. Declining to use workspaces did not avoid this, it only made it look like it had been avoided. next/jest merges rather than replaces, so its own /node_modules/ and /.next/ entries survive; verified with --showConfig, and --listTests still collects all 99 app suites and nothing under packages/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SgDfss2zhjwMky2mBKbcBX
Extract the DataPipe browser client into its own package
Recommend the jsPsych extension for sending data
…kout
signInWithEmailAndPassword never called ensureUserDocument (lib/user-bootstrap.js),
unlike the federated path (AuthProviderButtons.js) and signup (pages/signup.js). A
password account whose users/{uid} doc was missing -- a half-failed account
deletion (delete-account.ts's Auth delete failing after the purge succeeded), or a
legacy signup write that never landed -- hit AuthCheck's contact-email gate on
every admin route. ContactEmailGate's save is a setDoc(..., {merge:true}) of only
the four contactEmail keys, which on a missing document is a CREATE that
firestore.rules' isAccountCreation() denies (it requires uid/email/experiments),
so the researcher saw "Could not save your email address" forever with no way
out.
components/SignInForm.js now calls ensureUserDocument(credential.user) after a
successful sign-in, before navigating, with the same error handling as the
federated path: a bootstrap failure surfaces as a generic sign-in error rather
than navigating into a route that will just hit the same denied write.
components/ContactEmailGate.js also gets a defense-in-depth recovery path: if
userDoc is missing when the gate saves, it calls ensureUserDocument(user) first
so the follow-up contactEmail write becomes an update instead of a doomed create.
This covers a tab left open from before this fix, and any future path that
signs a user in without bootstrapping.
Checked every other client sign-in entry point (grep -rn "signInWith" pages
components lib): pages/oauth2/callback.js's signInWithCustomToken (OSF) is
preceded by a server-side Admin SDK doc write in
functions/src/oauth2-callback.ts, which bypasses these rules entirely --
no client-side gap. pages/reset-password.js's confirmPasswordReset does not
establish a session by itself, so it is not an entry point either.
Added __tests__/sign-in-form.test.jsx (ensureUserDocument called with the
signed-in user before navigation; a bootstrap failure blocks navigation and
shows the generic sign-in error, mirroring the federated path), extended
__tests__/contact-email-gate.test.jsx (missing doc: ensureUserDocument then
the contactEmail write; present doc: ensureUserDocument not called), and
extended __tests__/firestore-rules.test.js's account-creation section to pin
both sides of the bug: ensureUserDocument's exact password-account shape is
allowed, and ContactEmailGate's bare four-key shape against a missing
document is denied.
Pure jest suites (sign-in-form, contact-email-gate) pass; npm run lint is
clean. firestore.rules is unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
Remove the unauthenticated metadataOptions write channel from /api/data
Deploy storage.rules to both projects
Add the missing uploadQueue composite index and declare the filenameClaims TTL
Raise apiData/apiBase64 instance ceiling and rehydration lease
…back Two things, found by CodeQL flagging the first on the contrib PR that carries the same function. `url.replace(/\/+$/, "")` is js/polynomial-redos, high severity. Being straight about it: not a fix for an observed problem. V8 appears to optimise the anchored case -- a 60k-slash string showed no measurable slowdown -- and nothing hostile reaches this anyway, since the value is the researcher's own baseURL rather than participant input. The loop is provably linear, reads no worse, and costs less than re-arguing the alert on every scan. The second is a real bug, surfaced by writing a test for the first. setBaseURL fell back to the default on a falsy ARGUMENT, but "/" and "///" are truthy and normalize to "". An empty base makes endpoint() return "/api/data/" -- a relative URL -- so every submission would quietly go to the experiment's own host rather than DataPipe, and the researcher would be left reading 404s from their own server. The fallback now tests what normalizing produced. 48 tests, tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SgDfss2zhjwMky2mBKbcBX
The first deploy of #233 to datapipe-test was refused for apidata: "Max instances must be set to 200 or fewer to set the requested total CPU" (Cloud Run quota, us-central1). Every other target deployed, which left the test site running new rules and functions against the previous apidata revision. 200 is the largest value that deploys without a quota increase, and still ten times the global ceiling; apiBase64 stays at 100. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
A clarity pass over every page under pages/docs/ and the getting-started
guide. Wording only: headings, section ids, code samples, error codes and
comments are unchanged, and no fact, limit or caveat was dropped.
- Em dashes: 245 down to 4. The two left in prose are the live-sessions
panel's own status labels, quoted as the dashboard shows them.
- Cut throat-clearing ("There is…", "It is worth knowing…") and filler
clauses, and gave passives with an obvious actor that actor.
- "Label — description" list items use a colon throughout, including the
FeatureItem and ProviderOption separators in getting-started.js.
- API reference status and error rows share one sentence shape.
- Two page purposes drop contractions to match the rest of the docs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bootstrap users/{uid} on password sign-in to stop a contact-email lockout
Strip trailing slashes without a backtracking regex, and fix the fallback
Cap apidata at 200 instances, the Cloud Run quota ceiling
Tighten the docs prose
The homepage specimen and the dashboard's JavaScript tabs showed raw fetch calls; they now use DataPipe.saveData, saveBase64Data and getCondition, matching the Save as you go tab. The docs lead with the client and describe calling the API directly as the alternative, and no longer claim the client streams by default: it streams once an experiment starts a session. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Use datapipe-client in the plain JavaScript examples and docs
gdrive.ts used a fixed, public multipart boundary
(datapipe-gdrive-multipart-boundary) while concatenating the
participant's raw submission verbatim between its boundary lines. A
payload containing "\r\n--datapipe-gdrive-multipart-boundary--" closed
the data part early: Drive stored a truncated file and returned 201, so
DataPipe recorded a successful upload silently missing its tail. The
old comment argued a fixed boundary was fine because the body is built
in one shot -- true for uniqueness, not for unguessability.
newMultipartBoundary() now returns datapipe-gdrive-<16 random bytes hex>
per upload, mirroring dataverse.ts's existing fix and reasoning for the
identical class of bug. buildMultipartBody returns the boundary
alongside the bytes so the caller puts the same value in the body and
the Content-Type header.
Checked every other adapter for the same shape: zenodo.ts and osf.ts
both PUT raw bytes (octet-stream / put-file-osf.ts, update-file-osf.ts)
with no multipart body, so they were never exposed. dataverse.ts already
had the random-boundary fix. gdrive.ts was the only adapter left with a
fixed one.
providers-gdrive.test.js gets two new cases: consecutive uploads use
different boundaries matching ^datapipe-gdrive-[0-9a-f]{32}$ with the
Content-Type boundary matching the one written into the body, and an
injection case that parses the body by the real (random) boundary and
confirms a payload forging the old fixed boundary's closing delimiter
no longer truncates the data part. gdrive-emulator.test.js already
parses the boundary from the Content-Type header rather than assuming
the old constant, so it needs no change; oauth-connect-emulator.test.js
doesn't parse multipart bodies at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8xf16ov88M3KojPVfQwJT
Randomize the Drive adapter's multipart boundary per request
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs
mainwith everything merged totestsince the last release sync (#149, April 2026) — 187 commits across PRs #150–#187.Highlights
Multi-backend provider migration (#154–#161, #164) — core architecture for sending data to providers beyond OSF, plus:
Required contact email (#184) — schema + Firestore rules, signup seeding, verification round trip, first-failure notification state machine, backfill migration.
Payload encryption at rest (#185) and mail delivery via SES trigger, replacing the deprecated extension.
Metadata — version-drift fix, sidecar files, discoverability, follow-ups (#150–#153).
Docs — full restructure with new nav and redirects (#186), multi-backend guides (#178), 32 MB limit documentation.
Design pass — account settings redesign (#179), light mode (#181), homepage cleanup (#182), logo animation (#180), spacing pass (#183), branded 404 (#42).
CI & deps — ESLint on every PR (#109), dependency refresh clearing all advisories, firebase-admin v14, pinned firebase-tools (#187).
🤖 Generated with Claude Code