diff --git a/.context/state/current.md b/.context/state/current.md index 02ab1d7..b616037 100644 --- a/.context/state/current.md +++ b/.context/state/current.md @@ -1,6 +1,29 @@ # Current State -Last reviewed: 2026-09-04. +Last reviewed: 2026-09-11. + +## In flight + +**A box may now contain PyPI dependencies** — branch `pypi-licence-declaration`, unreleased. + +Until now it could not, and nobody had noticed: pixi records an SPDX licence for every conda package +and **none at all** for a PyPI one, so `lockedCondaDistributions()` failed the parse on the first +PyPI entry it met. That refusal is correct — an unlicensed dependency is a legal problem, not a +reporting gap — but it left an author with nothing to do about it, because the missing licence is +not in the lock and never will be. Every box built with Scrollcase so far has been pure conda, +which is why the gap survived two format versions. + +The fix follows `bundledLicenseDeclaration` exactly: a scroll points at a reviewed +`{ name, version, declaredLicense }` array, checked against the lock in both directions so neither +an uncovered package nor a stale line can pass. conda packages stay ineligible. Declared entries +carry `licenseDeclaredBy: "project"` in the inventory, so a project whose lock declares everything +sees no change at all. + +**Additive throughout — no format break, no v4.** The scroll field is optional, the signed release, +`box.json`, the payload digest and every `kind` string are untouched, and the only entries that +change shape are ones no released version could have produced. It is a minor release. + +Found downstream, by the first project that needed a model whose dependencies are PyPI-only. ## Current focus diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a3c986..2fff253 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,40 @@ All notable changes to Scrollcase are documented here. The format follows ## [Unreleased] +### Added — a box may contain PyPI dependencies + +- **`pypiLicenseDeclaration` supplies the licences `pixi.lock` does not carry.** pixi records an + SPDX licence for every conda package and **none at all** for a PyPI one — a PyPI entry holds a + name, a version, a hash and the requirements, and nothing about terms. Since an undeclared licence + fails the parse outright, and rightly so, **no box whose lock contained a single PyPI package + could be built**: it stopped at `== lacks a declared license in pixi.lock`, with + nothing an author could do about it. + + A scroll now points at a reviewed `{ name, version, declaredLicense }` array, maintained by the + project from the distributions its own lock pins. `validateDeclaredPypiLicenses()` checks the + shape; `readDeclaredPypiLicenses()` loads it for both `audit` and `build`, so what an author + reviews is what a build signs. + +- **The declaration is checked against the lock in both directions.** Every locked package the lock + leaves unnamed must be covered, and every entry must name a package that actually needed one — so + a line left behind by a dependency that was removed, upgraded, or that started declaring its own + licence fails instead of quietly standing. A conda package is never eligible: conda-forge states a + licence for all of them, and a declaration that could restate published metadata is a declaration + that could contradict it. + +- **An entry supplied this way carries `licenseDeclaredBy: "project"`** in the inventory the box + ships, and only those do. A reader can tell an asserted licence from a recorded one, and a project + whose lock declares everything sees a byte-identical inventory. Nothing in the signed release, + `box.json`, the payload digest or any `kind` string changes. + +### Fixed + +- **A quoted version in `pixi.lock` is no longer read as part of the version.** pixi quotes any + scalar YAML would otherwise read as a number, so `version: '1.84'` was becoming the four-character + string `'1.84'` in the inventory. Only PyPI entries take name and version from those fields — + conda takes both from the package filename — so no inventory that could be produced before this + release changes. + ### Added — the `node` and `native` demo boxes are published - **`codon-demo`, `transcode-demo` and `dataset-demo` are downloadable boxes**, signed by CI under diff --git a/docs/public/schema/v3/scroll.schema.json b/docs/public/schema/v3/scroll.schema.json index 819ef0e..94e3604 100644 --- a/docs/public/schema/v3/scroll.schema.json +++ b/docs/public/schema/v3/scroll.schema.json @@ -105,6 +105,14 @@ "minLength": 1, "description": "Path to the reviewed licence inventory derived from pixi.lock, which carries an SPDX licence per package. The build fails if the lock no longer matches what was reviewed." }, + "pypiLicenseDeclaration": { + "type": "string", + "minLength": 1, + "description": "Path to the project's licences for the PyPI half of pixi.lock. pixi records an SPDX licence for every conda package and none at all for a PyPI one — it writes a name, a version, a hash and the requirements, and nothing about terms — so a lock with PyPI dependencies cannot be inventoried from the lock alone. The file is a JSON array of { name, version, declaredLicense } entries, and it is checked against the lock both ways: every package the lock leaves unnamed must be covered, and every entry must name a package the lock actually needed one for, so a stale line fails instead of quietly standing. A conda package is never eligible, because conda-forge states a licence for all of them and no declaration should restate published metadata. Entries supplied this way are marked in the inventory the box ships, so a reader can tell what the lock said from what the project asserted.", + "examples": [ + "legal/pypi-licenses.json" + ] + }, "bundledLicenseDeclaration": { "type": "string", "minLength": 1, diff --git a/docs/reference/api/node.md b/docs/reference/api/node.md index 83d2291..cf5fc86 100644 --- a/docs/reference/api/node.md +++ b/docs/reference/api/node.md @@ -402,10 +402,16 @@ Details in [Workspace Configuration](/reference/configuration). | Export | Purpose | | --- | --- | -| `createCondaDependencyLicenseAudit({ lockBytes, targetId, namespace })` | The inventory, derived from a `pixi.lock` | +| `createCondaDependencyLicenseAudit({ lockBytes, targetId, namespace, declaredLicenses })` | The inventory, derived from a `pixi.lock` | | `validateCondaDependencyLicenseAudit(reviewed, actual)` | Throw unless a reviewed audit still matches the lock exactly | -| `lockedCondaDistributions(lockBytes)` | The parsed distributions with their declared licences | +| `lockedCondaDistributions(lockBytes, declaredLicenses)` | The parsed distributions with their declared licences | | `parseCondaPackageReference(url)` | `{ name, version }` from a conda package filename | +| `readDeclaredPypiLicenses(scroll, projectRoot)` | Load a scroll's `pypiLicenseDeclaration`, or an empty map | +| `validateDeclaredPypiLicenses(declared)` | SPDX by `name==version`, or throw on a malformed declaration | + +pixi records no licence for a PyPI distribution, so a lock containing one cannot be inventoried from +the lock alone. A scroll's `pypiLicenseDeclaration` names the reviewed file that supplies the missing +half, and the two are checked against each other in both directions. ```js import { readFile } from 'node:fs/promises'; diff --git a/docs/reference/scroll.md b/docs/reference/scroll.md index e709b16..48313bb 100644 --- a/docs/reference/scroll.md +++ b/docs/reference/scroll.md @@ -216,6 +216,7 @@ build reads, and provenance records. Nothing downstream can tell which half a va | `cacheSubdir` | no | Directory relative to the box root holding model assets. Defaults to `cache/` | | `environment` | no | String environment variables required whenever Scrollcase runs the box interpreter | | `condaDependencyLicenseAudit` | no | Path (from the project root) to the reviewed licence inventory, written and declared by [`audit --write`](/reference/cli#audit). When declared, the build fails if the lock no longer matches what was reviewed | +| `pypiLicenseDeclaration` | no | Path (from the project root) to the licences of the PyPI half of `pixi.lock`, which pixi does not record. See [PyPI licences](#pypi-licences) | | `bundledLicenseDeclaration` | no | Path (from the project root) to the licences of dependencies compiled *inside* a binary this box ships. See [Bundled licences](#bundled-licences) | The dependencies themselves live in `pixi.toml`, not here: @@ -236,14 +237,58 @@ or the solve produces an environment that cannot run on the machine the box is f [`scrollcase add dep `](/reference/cli#add) writes into every target's manifest at once, so they cannot drift apart, and `--from-requirements` imports an existing pip file. -### Bundled licences +### PyPI licences `condaDependencyLicenseAudit` is **derived**: `pixi.lock` already records an SPDX licence per conda package, so Scrollcase computes the inventory and checks it against what you reviewed. -It cannot do that for a binary you supply. Whatever was linked into that binary was linked before -Scrollcase saw the file, nothing in the build records it, and reading the binary would be guessing — -which is worse than not answering. So that half is **declared**: +**It records none for a PyPI package.** A PyPI entry in the lock carries a name, a version, a hash +and the requirements, and nothing about terms. So a box with PyPI dependencies cannot be inventoried +from the lock alone, and rather than ship a package whose licence nobody has named, the build stops: + +``` +box licence audit: biopython==1.84 lacks a declared license in pixi.lock +``` + +That half is **declared**, from the distributions your own lock already pins: + +```jsonc +"pypiLicenseDeclaration": "legal/pypi-licenses.json" +``` + +pointing at a JSON array your project reviews and keeps up to date: + +```jsonc +[ + { "name": "biopython", "version": "1.84", "declaredLicense": "LicenseRef-Biopython" }, + { "name": "click", "version": "8.1.7", "declaredLicense": "BSD-3-Clause" } +] +``` + +All three fields are required, and extra fields of your own are ignored — the record of *how* you +determined a licence belongs in your file, and is not something Scrollcase needs to read. + +The declaration and the lock are checked against each other **in both directions**: every locked +package the lock leaves unnamed must be covered, and every entry must name a package the lock +actually needed one for. So a line left behind by a dependency that was removed, upgraded, or that +started declaring its own licence fails the build instead of quietly standing: + +``` +box licence audit: declared PyPI licence for openssl==3.6.3 matches no locked package that needs one +``` + +A conda package is never eligible. conda-forge states a licence for all of them, and a declaration +that could restate published metadata is a declaration that could contradict it. + +In the inventory the box ships, an entry supplied this way carries `licenseDeclaredBy: "project"`, +and only those do — so a reader can tell an asserted licence from a recorded one, and a project whose +lock declares everything sees no change at all. + +### Bundled licences + +The same reasoning reaches further than PyPI. Whatever was linked into a binary you supply was +linked before Scrollcase saw the file, nothing in the build records it, and reading the binary would +be guessing — which is worse than not answering. So that half is **declared** too: ```jsonc "bundledLicenseDeclaration": "legal/bundled-dependencies.json" diff --git a/docs/white-paper.md b/docs/white-paper.md index c9b4b50..e794912 100644 --- a/docs/white-paper.md +++ b/docs/white-paper.md @@ -3194,6 +3194,26 @@ Two rules keep the result honest. A package whose licence is absent or literally And names are kept raw rather than normalised, because conda filenames already carry the canonical name and normalising would mangle legitimate leading-underscore names such as `_openmp_mutex`. +There is one place the lock cannot answer. **pixi records an SPDX licence for a conda package and +none at all for a PyPI one** — a PyPI entry carries a name, a version, a hash and the requirements, +and nothing about terms. A lock with PyPI dependencies therefore cannot be inventoried from the lock +alone, and the first rule above stops the build rather than shipping a package whose licence nobody +has named. + +A scroll closes that gap with `pypiLicenseDeclaration`, a path to a reviewed +`{ name, version, declaredLicense }` array the project maintains from the distributions its own lock +pins. `validateDeclaredPypiLicenses()` checks the shape and `readDeclaredPypiLicenses()` loads it for +both `audit` and `build`, so what an author reviews is what a build signs. The declaration is checked +against the lock in both directions: every package the lock leaves unnamed must be covered, and every +entry must name a package the lock actually needed one for, so a line left behind by a removed or +upgraded dependency fails instead of quietly standing. A conda package is never eligible — +conda-forge states a licence for all of them, and a declaration that could restate published +metadata is a declaration that could contradict it. + +Entries supplied this way carry `licenseDeclaredBy: "project"` in the inventory, and only those do, +so a reader can tell an asserted licence from a recorded one, and a project whose lock declares +everything sees no change at all. + The result is sorted by name then version, which is what makes the inventory itself deterministic. @@ -3921,7 +3941,7 @@ itself: | Filesystem | `collectFiles`, `fileExists`, `sha256File` | | Identity | `boxReleaseObjectPrefix`, `boxReleaseStem`, `builderVersionFields` | | Launchers | `repairPosixLaunchers` | -| Licences | `createCondaDependencyLicenseAudit`, `lockedCondaDistributions`, `parseCondaPackageReference`, `validateCondaDependencyLicenseAudit` | +| Licences | `createCondaDependencyLicenseAudit`, `lockedCondaDistributions`, `parseCondaPackageReference`, `readDeclaredPypiLicenses`, `validateCondaDependencyLicenseAudit`, `validateDeclaredPypiLicenses` | | pixi | `condaPackArguments`, `findCondaPack`, `findPixi`, `installAndPackPixiEnvironment`, `pixiInstallArguments`, `pixiLockArguments` | | Process | `fail`, `run`, `runResult` | | Toolchain | `CONDA_PACK_VERSION` | @@ -6603,6 +6623,7 @@ disk beside the installed package rather than something a browser can fetch. | `boxReleaseStem`, `boxReleaseObjectPrefix`, `builderVersionFields` | Release naming and builder identity | | `lockedCondaDistributions`, `parseCondaPackageReference` | Reading the lock into package identities | | `createCondaDependencyLicenseAudit`, `validateCondaDependencyLicenseAudit` | Producing and checking the licence inventory | +| `readDeclaredPypiLicenses`, `validateDeclaredPypiLicenses` | The PyPI licences a lock cannot state and a project declares | | `run`, `runResult`, `fail` | The subprocess seam and the single error path | diff --git a/src/build/audit.mjs b/src/build/audit.mjs index dda97a0..4f83f40 100644 --- a/src/build/audit.mjs +++ b/src/build/audit.mjs @@ -14,7 +14,11 @@ import { mkdir, readFile, writeFile } from 'node:fs/promises'; import { dirname, isAbsolute, join, relative, sep } from 'node:path'; import { boxTargetId } from '../contract/targets.mjs'; import { compareStableStrings, fileExists, safeRelativePath } from './filesystem.mjs'; -import { createCondaDependencyLicenseAudit, validateCondaDependencyLicenseAudit } from './licenses.mjs'; +import { + createCondaDependencyLicenseAudit, + readDeclaredPypiLicenses, + validateCondaDependencyLicenseAudit, +} from './licenses.mjs'; import { fail } from './process.mjs'; import { readScroll } from './scroll.mjs'; import { setScrollField } from './scroll-edit.mjs'; @@ -46,11 +50,13 @@ export async function auditScroll(name, { write = false, namespace } = {}) { const inventory = createCondaDependencyLicenseAudit({ lockBytes: await readFile(lockPath), targetId: boxTargetId(scroll.target), + declaredLicenses: await readDeclaredPypiLicenses(scroll, workspace.root), ...(namespace ? { namespace } : {}), }); - // A package with no declared licence never reaches here: parsing the lock rejects it outright, - // which is the point — an unlicensed dependency is a legal problem, not a reporting gap. + // A package with no licence never reaches here: the lock must name one, or the project's declared + // PyPI inventory must. That is the point — an unlicensed dependency is a legal problem, not a + // reporting gap. const licences = new Map(); for (const entry of inventory.packages) { licences.set(entry.declaredLicense, (licences.get(entry.declaredLicense) ?? 0) + 1); diff --git a/src/build/box.mjs b/src/build/box.mjs index 6a5eb4d..09212fa 100644 --- a/src/build/box.mjs +++ b/src/build/box.mjs @@ -44,6 +44,7 @@ import { assertExecutionFiles } from './execution.mjs'; import { boxReleaseObjectPrefix, boxReleaseStem, builderVersionFields } from './identity.mjs'; import { createCondaDependencyLicenseAudit, + readDeclaredPypiLicenses, validateBundledLicenses, validateCondaDependencyLicenseAudit, } from './licenses.mjs'; @@ -127,6 +128,7 @@ async function writeLicenceInventories({ scroll, lockPath, payloadDir, projectRo const actual = createCondaDependencyLicenseAudit({ lockBytes: await readFile(lockPath), targetId: boxTargetId(scroll.target), + declaredLicenses: await readDeclaredPypiLicenses(scroll, projectRoot), }); const reviewedPath = join(projectRoot, safeRelativePath(scroll.condaDependencyLicenseAudit)); const reviewed = JSON.parse(await readFile(reviewedPath, 'utf8')); diff --git a/src/build/index.d.mts b/src/build/index.d.mts index d63a4ab..24a852c 100644 --- a/src/build/index.d.mts +++ b/src/build/index.d.mts @@ -3,7 +3,7 @@ export { CONDA_PACK_VERSION } from "./toolchain.mjs"; export { createDeterministicZip, extractZipArchive, listZipEntries } from "./archive.mjs"; export { collectFiles, fileExists, payloadDigest, sha256File } from "./filesystem.mjs"; export { boxReleaseObjectPrefix, boxReleaseStem, builderVersionFields } from "./identity.mjs"; -export { createCondaDependencyLicenseAudit, lockedCondaDistributions, parseCondaPackageReference, validateCondaDependencyLicenseAudit } from "./licenses.mjs"; +export { createCondaDependencyLicenseAudit, lockedCondaDistributions, parseCondaPackageReference, readDeclaredPypiLicenses, validateCondaDependencyLicenseAudit, validateDeclaredPypiLicenses } from "./licenses.mjs"; export { condaPackArguments, findCondaPack, findPixi, installAndPackPixiEnvironment, pixiInstallArguments, pixiLockArguments } from "./pixi.mjs"; export { fail, run, runResult } from "./process.mjs"; export { DEFAULT_WORKSPACE_PATHS, SCROLLCASE_CONFIG_FILENAME, configureWorkspace, findWorkspaceConfig, getWorkspace, resolveWorkspace, workspaceOverridesFromArgv, workspaceOverridesFromFlags } from "./workspace.mjs"; diff --git a/src/build/index.mjs b/src/build/index.mjs index b1f7f24..3e2dabf 100644 --- a/src/build/index.mjs +++ b/src/build/index.mjs @@ -18,7 +18,9 @@ export { createCondaDependencyLicenseAudit, lockedCondaDistributions, parseCondaPackageReference, + readDeclaredPypiLicenses, validateCondaDependencyLicenseAudit, + validateDeclaredPypiLicenses, } from './licenses.mjs'; export { condaPackArguments, diff --git a/src/build/licenses.d.mts b/src/build/licenses.d.mts index b42eb26..114ad90 100644 --- a/src/build/licenses.d.mts +++ b/src/build/licenses.d.mts @@ -16,23 +16,59 @@ export function parseCondaPackageReference(url: string): { * by indented `key: value` fields. This scans that regular, machine-generated structure directly * rather than taking a transitive YAML dependency. * + * pixi records an SPDX licence for a conda package and none at all for a PyPI one, so a lock with + * PyPI dependencies cannot be inventoried from the lock alone. `declaredLicenses` supplies the + * missing half from what the project reviewed; where it is absent, an undeclared package still + * fails, because a dependency whose licence nobody has named is a legal problem rather than a + * reporting gap. + * * @param {Buffer} lockBytes the committed `pixi.lock` + * @param {Map} [declaredLicenses] SPDX by `name==version`, from the project's + * reviewed declaration * @returns {LockedDistribution[]} sorted by name then version - * @throws {Error} when the lock is unparseable or a package lacks a licence + * @throws {Error} when the lock is unparseable or a package's licence is nowhere to be found + */ +export function lockedCondaDistributions(lockBytes: Buffer, declaredLicenses?: Map): LockedDistribution[]; +/** + * Reads a project's declared licences for the PyPI distributions its lock does not name. + * + * The shape is checked here; whether each entry belongs is checked against the lock, by + * `createCondaDependencyLicenseAudit`, which is the only place both are in hand. + * + * @param {unknown} declared the parsed contents of the project's declaration file + * @returns {Map} SPDX expression by `name==version` + * @throws {Error} when the shape is wrong or a distribution is named twice */ -export function lockedCondaDistributions(lockBytes: Buffer): LockedDistribution[]; +export function validateDeclaredPypiLicenses(declared: unknown): Map; /** - * Builds the deterministic conda license audit bound to one pixi.lock and target. + * Loads a scroll's declared PyPI licences, or an empty map when it declares none. * - * @param {{ lockBytes: Buffer, targetId: string, namespace?: string }} options + * Both the audit command and the build read the declaration through here, so a licence the author + * reviewed with `audit` is the same licence the build signs. + * + * @param {{ pypiLicenseDeclaration?: string }} scroll + * @param {string} projectRoot + * @returns {Promise>} SPDX expression by `name==version` + * @throws {Error} when the declared path is missing or its contents are malformed + */ +export function readDeclaredPypiLicenses(scroll: { + pypiLicenseDeclaration?: string; +}, projectRoot: string): Promise>; +/** + * Builds the deterministic dependency licence audit bound to one pixi.lock and target. + * + * @param {{ lockBytes: Buffer, targetId: string, namespace?: string, + * declaredLicenses?: Map }} options * @returns {{ schemaVersion: 2, kind: string, targetId: string, dependencyLockSha256: string, * packages: LockedDistribution[] }} - * @throws {Error} when a locked package declares no licence + * @throws {Error} when a locked package's licence is nowhere to be found, or a declared licence + * names a package the lock does not need one for */ -export function createCondaDependencyLicenseAudit({ lockBytes, targetId, namespace }: { +export function createCondaDependencyLicenseAudit({ lockBytes, targetId, namespace, declaredLicenses, }: { lockBytes: Buffer; targetId: string; namespace?: string; + declaredLicenses?: Map; }): { schemaVersion: 2; kind: string; @@ -66,16 +102,21 @@ export function validateCondaDependencyLicenseAudit(reviewed: unknown, actual: R */ export function validateBundledLicenses(declared: unknown, carriedPaths: Set): Promise; /** - * One package as the lock declares it. + * One package the lock pins, with the licence that applies to it. */ export type LockedDistribution = { name: string; version: string; /** - * the SPDX expression carried by the lock + * the SPDX expression, from the lock or from the project */ declaredLicense: string; source: "conda" | "pypi"; + /** + * present only when the lock named no licence and the + * project supplied one, so a reader can tell the two apart + */ + licenseDeclaredBy?: "project"; }; /** * One dependency compiled inside a binary the box ships, as the project declared it. diff --git a/src/build/licenses.mjs b/src/build/licenses.mjs index 57aba07..223f309 100644 --- a/src/build/licenses.mjs +++ b/src/build/licenses.mjs @@ -1,12 +1,19 @@ /** - * The dependency licence inventories a box ships, and there are two of them because they are known - * in two different ways. + * The dependency licence inventories a box ships. They are separate because a licence is known in + * more than one way, and pretending otherwise is how a tool ends up guessing. * * The conda half is **derived** from the committed lock file rather than from the installed tree: * the lock already carries an SPDX licence per package, and `pixi install --frozen` guarantees the * installed set equals it. That makes the audit a pure function of a file the user reviews, so it * can be computed without a built prefix and cannot drift from what was approved. * + * The PyPI half of that same lock is not derivable, because pixi records no licence for a PyPI + * distribution — it writes a name, a version, a hash and the requirements, and nothing about terms. + * So a project **declares** those, from the distributions its own lock pins, and the declaration is + * checked against the lock both ways: every package the lock leaves unnamed must be covered, and + * every entry must be one the lock actually needed. A conda package is never eligible, so no + * declaration can restate metadata conda-forge already publishes. + * * The bundled half cannot be derived at all. A binary a scroll brings into the box was linked * before Scrollcase saw it, and no file in the build says what went into it; reading the binary * would be guessing, and guessing about a licence is worse than not answering. So that half is @@ -17,18 +24,21 @@ import { createHash } from 'node:crypto'; import { readFile } from 'node:fs/promises'; +import { join } from 'node:path'; import { DEFAULT_DOCUMENT_NAMESPACE } from '../contract/documents.mjs'; -import { compareStableStrings, safeRelativePath } from './filesystem.mjs'; +import { compareStableStrings, fileExists, safeRelativePath } from './filesystem.mjs'; import { schemaValidationError } from './schema-validation.mjs'; /** - * One package as the lock declares it. + * One package the lock pins, with the licence that applies to it. * * @typedef {object} LockedDistribution * @property {string} name * @property {string} version - * @property {string} declaredLicense the SPDX expression carried by the lock + * @property {string} declaredLicense the SPDX expression, from the lock or from the project * @property {'conda' | 'pypi'} source + * @property {'project'} [licenseDeclaredBy] present only when the lock named no licence and the + * project supplied one, so a reader can tell the two apart */ function fail(message) { @@ -60,6 +70,9 @@ export function parseCondaPackageReference(url) { return { name: parts.join('-'), version }; } +/** pixi quotes a version YAML would otherwise read as a number, so `'1.84'` means `1.84`. */ +const unquoteScalar = (value) => value.replace(/^'(.*)'$/, '$1'); + /** * Parses the exact conda + pypi distributions and their declared licenses from a pixi.lock. * @@ -67,11 +80,19 @@ export function parseCondaPackageReference(url) { * by indented `key: value` fields. This scans that regular, machine-generated structure directly * rather than taking a transitive YAML dependency. * + * pixi records an SPDX licence for a conda package and none at all for a PyPI one, so a lock with + * PyPI dependencies cannot be inventoried from the lock alone. `declaredLicenses` supplies the + * missing half from what the project reviewed; where it is absent, an undeclared package still + * fails, because a dependency whose licence nobody has named is a legal problem rather than a + * reporting gap. + * * @param {Buffer} lockBytes the committed `pixi.lock` + * @param {Map} [declaredLicenses] SPDX by `name==version`, from the project's + * reviewed declaration * @returns {LockedDistribution[]} sorted by name then version - * @throws {Error} when the lock is unparseable or a package lacks a licence + * @throws {Error} when the lock is unparseable or a package's licence is nowhere to be found */ -export function lockedCondaDistributions(lockBytes) { +export function lockedCondaDistributions(lockBytes, declaredLicenses = new Map()) { const lines = lockBytes.toString('utf8').split(/\r?\n/); const start = lines.findIndex((line) => line === 'packages:'); if (start === -1) fail('pixi.lock has no packages section'); @@ -82,12 +103,26 @@ export function lockedCondaDistributions(lockBytes) { let { name, version } = current; if (current.source === 'conda') ({ name, version } = parseCondaPackageReference(current.url)); if (!name || !version) fail(`pixi.lock package lacks a name or version: ${current.url}`); - if (!current.license || current.license.toUpperCase() === 'UNKNOWN') { + const lockLicense = current.license && current.license.toUpperCase() !== 'UNKNOWN' ? current.license : null; + // Only the PyPI half may be supplied by the project. conda-forge states a licence for every + // package, so a conda entry without one is a broken lock, not a gap a declaration should paper + // over — and keeping the door shut means no declaration can quietly restate conda's own metadata. + const declared = lockLicense === null && current.source === 'pypi' + ? declaredLicenses.get(`${name}==${version}`) + : undefined; + if (lockLicense === null && declared === undefined) { fail(`${name}==${version} lacks a declared license in pixi.lock`); } // conda/pypi filenames already carry the canonical name, so keep raw names — normalizing // would mangle legitimate leading-underscore conda names like `_openmp_mutex`. - distributions.push({ name, version, declaredLicense: current.license, source: current.source }); + distributions.push({ + name, + version, + declaredLicense: lockLicense ?? declared, + source: current.source, + // Only on the declared half, so an inventory of a lock that declares everything is unchanged. + ...(lockLicense === null ? { licenseDeclaredBy: 'project' } : {}), + }); current = null; }; for (let index = start + 1; index < lines.length; index += 1) { @@ -105,8 +140,8 @@ export function lockedCondaDistributions(lockBytes) { if (!field) continue; const [, key, value] = field; if (key === 'license' && current.license === null) current.license = value.trim(); - else if (key === 'name' && current.name === null) current.name = value.trim(); - else if (key === 'version' && current.version === null) current.version = value.trim(); + else if (key === 'name' && current.name === null) current.name = unquoteScalar(value.trim()); + else if (key === 'version' && current.version === null) current.version = unquoteScalar(value.trim()); } flush(); return distributions.sort((left, right) => @@ -114,20 +149,87 @@ export function lockedCondaDistributions(lockBytes) { } /** - * Builds the deterministic conda license audit bound to one pixi.lock and target. + * Reads a project's declared licences for the PyPI distributions its lock does not name. + * + * The shape is checked here; whether each entry belongs is checked against the lock, by + * `createCondaDependencyLicenseAudit`, which is the only place both are in hand. + * + * @param {unknown} declared the parsed contents of the project's declaration file + * @returns {Map} SPDX expression by `name==version` + * @throws {Error} when the shape is wrong or a distribution is named twice + */ +export function validateDeclaredPypiLicenses(declared) { + if (!Array.isArray(declared) || declared.length === 0) { + fail('a declared PyPI licence inventory must be a non-empty array'); + } + const licenses = new Map(); + for (const entry of /** @type {Record[]} */ (declared)) { + const { name, version, declaredLicense } = entry ?? {}; + const missing = ['name', 'version', 'declaredLicense'] + .filter((field) => typeof (entry ?? {})[field] !== 'string' || !(entry ?? {})[field]); + if (missing.length > 0) { + fail(`a declared PyPI licence entry lacks ${missing.join(', ')}: ${JSON.stringify(entry)}`); + } + const key = `${name}==${version}`; + if (licenses.has(key)) fail(`the declared PyPI licence inventory names ${key} twice`); + licenses.set(key, /** @type {string} */ (declaredLicense)); + } + return licenses; +} + +/** + * Loads a scroll's declared PyPI licences, or an empty map when it declares none. + * + * Both the audit command and the build read the declaration through here, so a licence the author + * reviewed with `audit` is the same licence the build signs. * - * @param {{ lockBytes: Buffer, targetId: string, namespace?: string }} options + * @param {{ pypiLicenseDeclaration?: string }} scroll + * @param {string} projectRoot + * @returns {Promise>} SPDX expression by `name==version` + * @throws {Error} when the declared path is missing or its contents are malformed + */ +export async function readDeclaredPypiLicenses(scroll, projectRoot) { + if (!scroll.pypiLicenseDeclaration) return new Map(); + const path = join(projectRoot, safeRelativePath(scroll.pypiLicenseDeclaration)); + if (!await fileExists(path)) { + fail(`declared PyPI licence inventory is missing: ${scroll.pypiLicenseDeclaration}`); + } + return validateDeclaredPypiLicenses(JSON.parse(await readFile(path, 'utf8'))); +} + +/** + * Builds the deterministic dependency licence audit bound to one pixi.lock and target. + * + * @param {{ lockBytes: Buffer, targetId: string, namespace?: string, + * declaredLicenses?: Map }} options * @returns {{ schemaVersion: 2, kind: string, targetId: string, dependencyLockSha256: string, * packages: LockedDistribution[] }} - * @throws {Error} when a locked package declares no licence + * @throws {Error} when a locked package's licence is nowhere to be found, or a declared licence + * names a package the lock does not need one for */ -export function createCondaDependencyLicenseAudit({ lockBytes, targetId, namespace = DEFAULT_DOCUMENT_NAMESPACE }) { +export function createCondaDependencyLicenseAudit({ + lockBytes, + targetId, + namespace = DEFAULT_DOCUMENT_NAMESPACE, + declaredLicenses = new Map(), +}) { + const packages = lockedCondaDistributions(lockBytes, declaredLicenses); + // A declaration nobody needed is a stale entry: the package left the lock, changed version, or + // started declaring its own licence. Saying so beats carrying a claim about nothing. + const supplied = new Set(packages + .filter((entry) => entry.licenseDeclaredBy) + .map((entry) => `${entry.name}==${entry.version}`)); + for (const key of declaredLicenses.keys()) { + if (!supplied.has(key)) { + fail(`declared PyPI licence for ${key} matches no locked package that needs one`); + } + } return { schemaVersion: 2, kind: `${namespace}.dependency-license-audit`, targetId, dependencyLockSha256: sha256(lockBytes), - packages: lockedCondaDistributions(lockBytes), + packages, }; } diff --git a/src/build/scroll.mjs b/src/build/scroll.mjs index 707070f..721b5d9 100644 --- a/src/build/scroll.mjs +++ b/src/build/scroll.mjs @@ -313,6 +313,7 @@ async function readExactScroll(reference) { ...(scroll.execution?.binary ? [scroll.execution.binary] : []), ...(scroll.parity ? [scroll.parity.script] : []), ...(scroll.condaDependencyLicenseAudit ? [scroll.condaDependencyLicenseAudit] : []), + ...(scroll.pypiLicenseDeclaration ? [scroll.pypiLicenseDeclaration] : []), ...(scroll.bundledLicenseDeclaration ? [scroll.bundledLicenseDeclaration] : []), ]; for (const path of payloadPaths) safeRelativePath(path); diff --git a/src/contract/schema/scroll.schema.json b/src/contract/schema/scroll.schema.json index 819ef0e..94e3604 100644 --- a/src/contract/schema/scroll.schema.json +++ b/src/contract/schema/scroll.schema.json @@ -105,6 +105,14 @@ "minLength": 1, "description": "Path to the reviewed licence inventory derived from pixi.lock, which carries an SPDX licence per package. The build fails if the lock no longer matches what was reviewed." }, + "pypiLicenseDeclaration": { + "type": "string", + "minLength": 1, + "description": "Path to the project's licences for the PyPI half of pixi.lock. pixi records an SPDX licence for every conda package and none at all for a PyPI one — it writes a name, a version, a hash and the requirements, and nothing about terms — so a lock with PyPI dependencies cannot be inventoried from the lock alone. The file is a JSON array of { name, version, declaredLicense } entries, and it is checked against the lock both ways: every package the lock leaves unnamed must be covered, and every entry must name a package the lock actually needed one for, so a stale line fails instead of quietly standing. A conda package is never eligible, because conda-forge states a licence for all of them and no declaration should restate published metadata. Entries supplied this way are marked in the inventory the box ships, so a reader can tell what the lock said from what the project asserted.", + "examples": [ + "legal/pypi-licenses.json" + ] + }, "bundledLicenseDeclaration": { "type": "string", "minLength": 1, diff --git a/src/contract/types/index.d.ts b/src/contract/types/index.d.ts index bb36493..a57cdcb 100644 --- a/src/contract/types/index.d.ts +++ b/src/contract/types/index.d.ts @@ -161,6 +161,10 @@ export interface BoxScroll { * Path to the reviewed licence inventory derived from pixi.lock, which carries an SPDX licence per package. The build fails if the lock no longer matches what was reviewed. */ condaDependencyLicenseAudit?: string; + /** + * Path to the project's licences for the PyPI half of pixi.lock. pixi records an SPDX licence for every conda package and none at all for a PyPI one — it writes a name, a version, a hash and the requirements, and nothing about terms — so a lock with PyPI dependencies cannot be inventoried from the lock alone. The file is a JSON array of { name, version, declaredLicense } entries, and it is checked against the lock both ways: every package the lock leaves unnamed must be covered, and every entry must name a package the lock actually needed one for, so a stale line fails instead of quietly standing. A conda package is never eligible, because conda-forge states a licence for all of them and no declaration should restate published metadata. Entries supplied this way are marked in the inventory the box ships, so a reader can tell what the lock said from what the project asserted. + */ + pypiLicenseDeclaration?: string; /** * Path to the project's inventory of dependencies compiled *inside* the binaries this box ships. pixi.lock declares a licence per conda package, but it cannot see what was linked into a supplied executable before the build ever started, and nothing Scrollcase can read will tell it. So this half is declared rather than derived: the file is a JSON array of { name, version, declaredLicense, linkedInto } entries, and the build checks that every path it names is really in the box before carrying the list into the signed release. What belongs in it is the project's judgement; Scrollcase transports and signs what the project reviewed and never decides what a complete inventory is. */ diff --git a/tests/unit/project-surface.test.mjs b/tests/unit/project-surface.test.mjs index 40c381f..a6ca1ce 100644 --- a/tests/unit/project-surface.test.mjs +++ b/tests/unit/project-surface.test.mjs @@ -26,6 +26,18 @@ packages: size: 1 `; +/** + * The same lock with a PyPI dependency, written exactly as pixi writes one: a name, a version, a + * hash and the requirements, and nothing about licence terms. The quoted version is pixi's too — it + * quotes any scalar YAML would otherwise read as a number. + */ +const LOCK_WITH_PYPI = `${LOCK}- pypi: https://files.pythonhosted.org/packages/ab/cd/biopython-1.84.whl + name: biopython + version: '1.84' + sha256: eee + requires_python: '>=3.9' +`; + describe('setting a project up', () => { const created = []; @@ -167,7 +179,7 @@ describe('auditing dependency licences', () => { await Promise.all(created.splice(0).map((path) => rm(path, { recursive: true, force: true }))); }); - async function projectWithLock({ auditPath = 'legal/audit.json' } = {}) { + async function projectWithLock({ auditPath = 'legal/audit.json', lock = LOCK, pypiDeclaration } = {}) { const root = await realpath(await mkdtemp(join(tmpdir(), 'scrollcase-audit-'))); created.push(root); await initProject({ root }); @@ -187,11 +199,21 @@ describe('auditing dependency licences', () => { }); const scroll = JSON.parse(await readFile(join(result.scrollDir, 'scroll.json'), 'utf8')); if (auditPath) scroll.condaDependencyLicenseAudit = auditPath; + if (pypiDeclaration) { + scroll.pypiLicenseDeclaration = 'legal/pypi-licenses.json'; + await mkdir(join(root, 'legal'), { recursive: true }); + await writeFile( + join(root, 'legal/pypi-licenses.json'), + `${JSON.stringify(pypiDeclaration, null, 2)}\n`, + ); + } await writeFile(join(result.scrollDir, 'scroll.json'), `${JSON.stringify(scroll, null, 2)}\n`); - await writeFile(join(result.scrollDir, 'pixi.lock'), LOCK); + await writeFile(join(result.scrollDir, 'pixi.lock'), lock); return { root, scrollRef: result.scrollRef }; } + const BIOPYTHON = [{ name: 'biopython', version: '1.84', declaredLicense: 'LicenseRef-Biopython' }]; + it('summarises the inventory straight from the lock, with no build', async () => { const { scrollRef } = await projectWithLock({ auditPath: null }); const { summary, inventory, reviewed } = await auditScroll(scrollRef); @@ -242,4 +264,76 @@ describe('auditing dependency licences', () => { await writeFile(join(root, 'legal/audit.json'), `${JSON.stringify(stale, null, 2)}\n`); await expect(auditScroll(scrollRef)).rejects.toThrow(/differ from the reviewed audit/); }); + + it('refuses a PyPI dependency the project has not given a licence', async () => { + const { scrollRef } = await projectWithLock({ auditPath: null, lock: LOCK_WITH_PYPI }); + await expect(auditScroll(scrollRef)) + .rejects.toThrow(/biopython==1\.84 lacks a declared license in pixi\.lock/); + }); + + it('takes the PyPI half from the project, and says which half that was', async () => { + const { scrollRef } = await projectWithLock({ + auditPath: null, + lock: LOCK_WITH_PYPI, + pypiDeclaration: BIOPYTHON, + }); + + const { inventory, summary } = await auditScroll(scrollRef); + + expect(summary.packageCount).toBe(3); + // The quoted version in the lock is YAML's, not part of the version. + expect(inventory.packages).toContainEqual({ + name: 'biopython', + version: '1.84', + declaredLicense: 'LicenseRef-Biopython', + source: 'pypi', + licenseDeclaredBy: 'project', + }); + // A lock-derived entry is untouched, so an all-conda project's inventory does not change at all. + expect(inventory.packages).toContainEqual({ + name: 'openssl', + version: '3.6.3', + declaredLicense: 'Apache-2.0', + source: 'conda', + }); + }); + + it('refuses a declared licence for a package the lock did not need one for', async () => { + const { scrollRef } = await projectWithLock({ + auditPath: null, + lock: LOCK_WITH_PYPI, + // Both are stale in the way that matters: neither names a locked package missing a licence. + pypiDeclaration: [...BIOPYTHON, { name: 'openssl', version: '3.6.3', declaredLicense: 'MIT' }], + }); + await expect(auditScroll(scrollRef)) + .rejects.toThrow(/declared PyPI licence for openssl==3\.6\.3 matches no locked package/); + }); + + it('refuses a declaration that is missing a field, or names one package twice', async () => { + const { scrollRef: incomplete } = await projectWithLock({ + auditPath: null, + lock: LOCK_WITH_PYPI, + pypiDeclaration: [{ name: 'biopython', version: '1.84' }], + }); + await expect(auditScroll(incomplete)).rejects.toThrow(/lacks declaredLicense/); + + resetWorkspace(); + const { scrollRef: duplicated } = await projectWithLock({ + auditPath: null, + lock: LOCK_WITH_PYPI, + pypiDeclaration: [...BIOPYTHON, ...BIOPYTHON], + }); + await expect(auditScroll(duplicated)).rejects.toThrow(/names biopython==1\.84 twice/); + }); + + it('refuses a declared path that is not there', async () => { + const { root, scrollRef } = await projectWithLock({ + auditPath: null, + lock: LOCK_WITH_PYPI, + pypiDeclaration: BIOPYTHON, + }); + await rm(join(root, 'legal/pypi-licenses.json')); + await expect(auditScroll(scrollRef)) + .rejects.toThrow(/declared PyPI licence inventory is missing/); + }); });