Walk the unpacked tree without following symlinks in the mtime pass - #220
Merged
Conversation
After extracting a tarball to disk, unpack refreshed every file's mtime
by listing the tree with filelib:wildcard("**"), which descends into
symlinked directories, and file:write_file_info/3, which resolves
symlinks. Packages created by old clients can contain in-tree symlink
cycles (create rejects them today, unpack accepts them), and on such a
package the pass visits every file once per cycle expansion: svx 0.2.0
holds 1,171 files but the wildcard enumerates ~47k paths, turning a
260ms extraction into multiple seconds of mtime updates.
Replace the wildcard with a read_link_info walk that updates regular
files and directories and skips symlinks entirely, computing the
timestamp once.
ericmj
marked this pull request as ready for review
August 21, 2026 12:43
ericmj
added a commit
to hexpm/hexpm
that referenced
this pull request
Aug 21, 2026
Pulls in hexpm/hex_core#220, which stops the post-extract mtime pass from following symlinks. On the svx 0.2.0 tarball this takes unpack from 22.3s to 0.3s on a prod worker, and the whole diff generation job from ~27s to 5.7s.
ericmj
added a commit
to hexpm/hexpm
that referenced
this pull request
Aug 21, 2026
…er concurrency (#1849) * Never follow symlinks when walking unpacked tarballs Diff generation and preview uploads walked unpacked tarballs with Path.wildcard, which descends into symlinked directories, and File.regular?, which resolves symlinks. Packages published by old Hex clients can contain symlinks (current hex_tarball.create rejects them, unpack still materializes them), and svx 0.2.0 contains link cycles that made its 1,186 files walk as 54,691 on a prod worker, spending 255s of the diff worker's 270s budget before piece uploads (HEXPM-D6). Walk the tree with lstat instead: symlinks are skipped both as entries and as directories, in the shared Hexpm.Utils.tree_regular_files/1. The per-side regular-file checks in the diff generator use lstat too, so a symlink on one side is treated as absent rather than resolved. * Upload diff pieces concurrently The generator uploaded each piece to the diff bucket inline while walking the file list, one blocking store put per changed file. On the prod measurement of svx 0.2.0 to 0.3.0 the sequential puts were the largest remaining cost after the symlink fix. Build pieces lazily and feed them through Task.async_stream at concurrency 10 with a 120s task timeout (a GCS put retries internally for up to ~83s), mirroring the preview upload path. Results stay ordered so piece indexes and the metadata file list keep matching, and upload exceptions reraise in the caller so Generator.generate/1 still returns {:error, {exception, stacktrace}}. A git failure now raises instead of returning {:error, {:git_diff, reason}}, which no caller matched on. * Raise GCS transfer concurrency to 32 and grow the connection pool A concurrency sweep on a prod worker (1,169 diff pieces, two rounds per level) scaled linearly to 50: 7.7-8.2s at 10, 3.9-4.1s at 20, 2.4-2.6s at 32, 1.5-1.6s at 50, tracking within 11% of ideal N-way speedup over the 63ms mean per-object latency. The limit is pool budget, not throughput, so raise the per-job concurrency to 32 everywhere a job streams objects (diff pieces, preview files, hexdocs pages, registry objects, delete_many) and size the shared GCS Finch pool for the worst case: 10 heavy jobs plus 2 registry jobs at 32 each is 384 concurrent requests, so 50 x 8 = 400 connections. Also download the two diff tarballs concurrently. Their TmpDir paths are allocated in the job process because TmpDir deletes a tracked path when the process that created it exits, and the download tasks exit immediately. * Bump hex_core for the symlink-safe unpack mtime pass Pulls in hexpm/hex_core#220, which stops the post-extract mtime pass from following symlinks. On the svx 0.2.0 tarball this takes unpack from 22.3s to 0.3s on a prod worker, and the whole diff generation job from ~27s to 5.7s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After extracting a tarball to disk, unpack refreshes every file's mtime by listing the tree with
filelib:wildcard("**")and callingfile:write_file_info/3on each path. Both follow symlinks: the wildcard descends into symlinked directories andwrite_file_inforesolves the link target. Tarballs published by old Hex clients can contain in-tree symlink cycles (createrejects them today, unpack still accepts them), and on such a package the pass visits every file once per cycle expansion. svx 0.2.0 ships an example app's_buildwith two such cycles: it holds 1,171 regular files, but the wildcard enumerates ~47k paths (bounded only by the kernel's symlink depth limit), so a 263mshex_erl_tarextraction turns into a 6.4sunpackon a fast local disk and 22s on a small cloud instance.This replaces the wildcard with a
read_link_infowalk that updates regular files and directories, never descends into or touches symlinks, and computes the timestamp once. Both call sites (package unpack and docs unpack) use it. Measured on the same svx tarball,hex_tarball:unpackdrops from 6,381ms to 326ms locally, and from 22.3s to 332ms on the cloud instance. Skipping symlinks also stopswrite_file_infofrom writing through them, which previously touched whatever the link resolved to.Found while debugging hexpm diff-worker timeouts on this package, where the same wildcard pattern existed in hexpm's own tree walk and is fixed separately.