diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b042c3..7d53c12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,23 @@ All notable changes to Scrollcase are documented here. The format follows ## [Unreleased] +## [1.2.0] — 2026-09-12 + +### Added + +- **`assetArchives` accepts an uncompressed `tar`.** The format was `zip` or `tar.gz` only, so a + publisher who ships a plain tarball — the usual choice when every member inside is already + compressed, and a second pass would cost minutes to save nothing — could not be consumed at all. + The archive is listed, validated and expanded through the same code as `tar.gz`; the only thing + missing was permission to say so. + + Compression is now left to the library on read, which is what it was doing regardless: node-tar + reads the header and decompresses or not, and the `gzip` option applies to writing. A scroll's + `tar` / `tar.gz` distinction records what the author pinned, and the asset's SHA-256 holds them + to it. + + Reported by a project whose model bundles 1.86 GB of pickled molecule definitions that way. + ## [1.1.1] — 2026-09-11 ### Fixed diff --git a/docs/guides/managing-assets.md b/docs/guides/managing-assets.md index 0749384..ebc5ecd 100644 --- a/docs/guides/managing-assets.md +++ b/docs/guides/managing-assets.md @@ -74,6 +74,9 @@ When the upstream artefact is a tarball or zip, declare it as an asset and then ] ``` +- `format` is `zip`, `tar` or `tar.gz`. Reach for `tar` when the publisher ships an uncompressed + tarball, which is the usual choice when its members are already compressed; it goes through the + same listing, validation and extraction as `tar.gz`. - Entries are listed and validated **before** extraction, so a malicious archive cannot write outside its destination. - `stripComponents` drops the redundant top-level wrapper directory many published archives diff --git a/docs/public/schema/v3/scroll.schema.json b/docs/public/schema/v3/scroll.schema.json index 94e3604..7ab3c28 100644 --- a/docs/public/schema/v3/scroll.schema.json +++ b/docs/public/schema/v3/scroll.schema.json @@ -201,8 +201,10 @@ "format": { "enum": [ "zip", + "tar", "tar.gz" - ] + ], + "description": "How the downloaded file is packed. `tar` is an uncompressed archive, which is what a publisher produces when the members are already compressed and a second pass would only cost time; it is validated and extracted exactly like `tar.gz`." }, "destination": { "$ref": "#/$defs/payloadPath" diff --git a/docs/reference/scroll.md b/docs/reference/scroll.md index 48313bb..2bd54f3 100644 --- a/docs/reference/scroll.md +++ b/docs/reference/scroll.md @@ -598,7 +598,10 @@ refuses to overwrite them. ] ``` -`format` is `zip` or `tar.gz`. An archive is expanded at build time, so **it has no `embed` field**: +`format` is `zip`, `tar` or `tar.gz`. `tar` is an uncompressed tarball — what a publisher ships when +the members are already compressed and a second pass would only cost time — and it is listed, +validated and expanded exactly like `tar.gz`. An archive is expanded at build time, so **it has no +`embed` field**: "leave it out and let the caller fetch it" names nothing that could happen. Version 2 refused that combination with a cross-field check; version 3 makes it unspeakable. diff --git a/docs/white-paper.md b/docs/white-paper.md index e794912..7c11f57 100644 --- a/docs/white-paper.md +++ b/docs/white-paper.md @@ -1360,7 +1360,7 @@ The published package depends on three libraries and nothing else: | Package | Version | Role | | --- | --- | --- | -| `tar` | 7.5.22 | Reads the conda-pack tarball into the payload; validates and extracts `tar.gz` scroll assets and toolchain archives | +| `tar` | 7.5.22 | Reads the conda-pack tarball into the payload; validates and extracts `tar` and `tar.gz` scroll assets and toolchain archives | | `yauzl` | 3.4.0 | Reads and validates box ZIP archives | | `yazl` | 3.3.1 | Writes the deterministic box ZIP archive | @@ -1411,9 +1411,11 @@ Link targets are read once during validation and reused during extraction, so a rewritten archive cannot pass the check with one value and extract with another. **`tar` — reading only.** Used to extract the conda-pack output into the payload, and to validate -and extract `tar.gz` scroll assets and toolchain archives. TAR entries are validated before +and extract `tar` and `tar.gz` scroll assets and toolchain archives. TAR entries are validated before extraction and the accepted types are `File`, `OldFile` and `Directory` only: links and special -entries in a TAR are refused outright. +entries in a TAR are refused outright. Compression is left to the library, which reads the header and +decompresses or not accordingly; a scroll's `tar` / `tar.gz` distinction records what the author +pinned, and the asset's SHA-256 is what holds them to it. One subtlety belongs here because it is not obvious from the code's shape. When extracting the conda-pack tarball, links are deliberately extracted in a **second pass**, after every regular entry diff --git a/package-lock.json b/package-lock.json index 0d5b192..dc8da39 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "scrollcase", - "version": "1.1.1", + "version": "1.2.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "scrollcase", - "version": "1.1.1", + "version": "1.2.0", "license": "Apache-2.0", "dependencies": { "tar": "7.5.22", diff --git a/package.json b/package.json index 6f7b437..4e8dbb7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "scrollcase", - "version": "1.1.1", + "version": "1.2.0", "schemaVersion": "3", "description": "Pack an entire Python environment and the code it runs into a single, self-contained, portable and signed box.", "license": "Apache-2.0", diff --git a/src/build/archive.d.mts b/src/build/archive.d.mts index 73a1aa2..620aa85 100644 --- a/src/build/archive.d.mts +++ b/src/build/archive.d.mts @@ -74,9 +74,9 @@ export function extractZipArchive(archivePath: string, destination: string): Pro * Extracts scroll assets using only pinned Node archive implementations. * * @param {string} archivePath - * @param {'zip' | 'tar.gz'} format + * @param {'zip' | 'tar' | 'tar.gz'} format * @param {string} destination * @param {number} [stripComponents] * @returns {Promise} */ -export function extractScrollArchive(archivePath: string, format: "zip" | "tar.gz", destination: string, stripComponents?: number): Promise; +export function extractScrollArchive(archivePath: string, format: "zip" | "tar" | "tar.gz", destination: string, stripComponents?: number): Promise; diff --git a/src/build/archive.mjs b/src/build/archive.mjs index e34d111..4d2eb8c 100644 --- a/src/build/archive.mjs +++ b/src/build/archive.mjs @@ -353,12 +353,17 @@ export async function extractZipArchive(archivePath, destination) { await validateExtractedTree(destination, { allowLinks: true }); } -/** Lists TAR assets and rejects paths, links, and special entries before extraction. */ +/** + * Lists TAR assets and rejects paths, links, and special entries before extraction. + * + * Compression is not named here: node-tar reads the header and decompresses or not on its own, and + * the `gzip` option it takes applies to writing. The scroll's `tar` / `tar.gz` distinction is the + * author saying what they pinned, and the asset's SHA-256 is what makes that claim binding. + */ async function validateTarArchive(archivePath) { let violation; await tar.t({ file: archivePath, - gzip: true, strict: true, onentry(entry) { if (violation) return; @@ -379,7 +384,7 @@ async function validateTarArchive(archivePath) { * Extracts scroll assets using only pinned Node archive implementations. * * @param {string} archivePath - * @param {'zip' | 'tar.gz'} format + * @param {'zip' | 'tar' | 'tar.gz'} format * @param {string} destination * @param {number} [stripComponents] * @returns {Promise} @@ -389,12 +394,11 @@ export async function extractScrollArchive(archivePath, format, destination, str try { if (format === 'zip') { await extractZipArchive(archivePath, tempRoot); - } else if (format === 'tar.gz') { + } else if (format === 'tar' || format === 'tar.gz') { await validateTarArchive(archivePath); await tar.x({ file: archivePath, cwd: tempRoot, - gzip: true, preservePaths: false, strict: true, }); diff --git a/src/contract/schema/scroll.schema.json b/src/contract/schema/scroll.schema.json index 94e3604..7ab3c28 100644 --- a/src/contract/schema/scroll.schema.json +++ b/src/contract/schema/scroll.schema.json @@ -201,8 +201,10 @@ "format": { "enum": [ "zip", + "tar", "tar.gz" - ] + ], + "description": "How the downloaded file is packed. `tar` is an uncompressed archive, which is what a publisher produces when the members are already compressed and a second pass would only cost time; it is validated and extracted exactly like `tar.gz`." }, "destination": { "$ref": "#/$defs/payloadPath" diff --git a/src/contract/types/index.d.ts b/src/contract/types/index.d.ts index a57cdcb..e3241e4 100644 --- a/src/contract/types/index.d.ts +++ b/src/contract/types/index.d.ts @@ -207,7 +207,10 @@ export interface BoxScroll { */ assetArchives?: { relativePath: PayloadPath; - format: 'zip' | 'tar.gz'; + /** + * How the downloaded file is packed. `tar` is an uncompressed archive, which is what a publisher produces when the members are already compressed and a second pass would only cost time; it is validated and extracted exactly like `tar.gz`. + */ + format: 'zip' | 'tar' | 'tar.gz'; destination: PayloadPath; stripComponents?: number; removeAfterExtract?: boolean; diff --git a/tests/unit/archive-security.test.mjs b/tests/unit/archive-security.test.mjs index 1498628..a272187 100644 --- a/tests/unit/archive-security.test.mjs +++ b/tests/unit/archive-security.test.mjs @@ -130,6 +130,30 @@ describe('archive boundaries', () => { expect(await fileExists(destination)).toBe(false); }); + it('expands an uncompressed tarball, which a publisher of already-compressed members ships', async () => { + const root = await scratch(); + const staging = join(root, 'staging'); + await mkdir(join(staging, 'mols'), { recursive: true }); + await writeFile(join(staging, 'mols', 'T9E.pkl'), 'bytes'); + const archive = join(root, 'mols.tar'); + await tar.c({ file: archive, cwd: staging, gzip: false }, ['mols']); + const destination = join(root, 'destination'); + await extractScrollArchive(archive, 'tar', destination, 1); + expect(await collectFiles(destination)).toEqual(['T9E.pkl']); + }); + + it('applies the same entry rules to an uncompressed tarball as to a compressed one', async () => { + const root = await scratch(); + const staging = join(root, 'staging'); + await mkdir(staging); + await writeFile(join(staging, 'target'), 'bytes'); + await symlink('target', join(staging, 'link')); + const archive = join(root, 'link.tar'); + await tar.c({ file: archive, cwd: staging, gzip: false }, ['link']); + await expect(extractScrollArchive(archive, 'tar', join(root, 'destination'))) + .rejects.toThrow(/links and special entries/); + }); + it('orders files by raw path strings rather than host collation', async () => { const root = await scratch(); for (const name of ['a', '_', 'B']) await writeFile(join(root, name), name);