uv: ホイールキャッシュ内の全ファイルを重複排除

Uh oh! There was an error while loading. Please reload this page. astral-sh / uv Public Notifications You must be signed in to change notification settings Fork 3.5k Star 89.3k Conversation charliermarsh commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page. Copy link Copy Markdown Member Summary On main, we support content-addressed caching, but only at the wheel-level. That is, if you download the same wheel twice from different sources, they share a cache entry. But files within or across wheels are not deduplicated at all. This PR adds deduplication at the file level: every file is now stored under its BLAKE3 hash in a files-v0 bucket. We hardlink these objects into their original locations in archive-v0, so the installation step doesn't change at all -- we're just deduping within the cache (and cache cleanup removes file objects when their hardlink count drops to one). In the prior proposal (#19694), we included the following table: File selection Additional savings Files hardlinked Distinct files-v0 objects Executables and native libraries 275.7 MiB 3,336 3,088 Any payload file ≥ 10 MiB 235.0 MiB 80 78 Any payload file ≥ 1 MiB 279.7 MiB 373 349 Any payload file ≥ 100 KiB 353.4 MiB 2,830 2,458 Any payload file ≥ 10 KiB 475.7 MiB 23,767 18,722 Any payload file ≥ 1 KiB 537.4 MiB 95,156 66,422 All payload files 545.2 MiB 134,222 87,129 So we're saving 545.2 MiB on my local machine, or about 10% of the cache. In return, the net effect seems to be something like a <4% slowdown for cold installs (and no effect on warm installs), which I think is probably worthwhile here. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. 🚀 3 ilkersigirci, madig, and PixiBixi reacted with rocket emoji All reactions 🚀 3 reactions charliermarsh temporarily deployed to automations August 27, 2026 19:40 — with GitHub Actions Inactive astral-automations-bot Bot mentioned this pull request Aug 27, 2026 Linux cargo-test setup fails when Depot runners lack Minix support astral-sh/uv-dev#896 Open charliermarsh temporarily deployed to automations August 27, 2026 22:01 — with GitHub Actions Inactive charliemarsh commented Aug 27, 2026 • edited by charliermarsh Loading Uh oh! There was an error while loading. Please reload this page. Copy link Copy Markdown Contributor N.B. Slop comment for benchmark data. We benchmarked the optimization in 85c3f7485 against the binary-only cache at a3f977ece, with --preview-features content-addressed-cache enabled on both. These are medians for the complete uv pip install process; positive changes mean slower. Wheel Cache Parent median Optimized median Change (95% CI) AnyIO 4.9.0 cold 90.4 ms 93.4 ms +3.39% (+2.87% to +3.86%) AnyIO 4.9.0 warm 29.9 ms 30.3 ms +1.14% (-1.08% to +3.85%) SymPy 1.14.0 cold 367.1 ms 381.6 ms +3.95% (+3.49% to +4.46%) SymPy 1.14.0 warm 84.6 ms 84.0 ms -0.64% (-1.04% to -0.11%) NumPy 2.2.6 cold 314.1 ms 326.5 ms +3.95% (+3.24% to +4.82%) NumPy 2.2.6 warm 62.5 ms 62.7 ms +0.28% (-0.55% to +1.25%) PyTorch 2.7.1+cpu cold 2982.6 ms 3072.9 ms +3.03% (+0.98% to +5.18%) PyTorch 2.7.1+cpu warm 404.0 ms 403.1 ms -0.21% (-0.72% to +0.11%) Before this optimization, the measured cold regressions were +4.02% for AnyIO 4.9.0, +19.41% for SymPy 1.14.0, +12.75% for NumPy 2.2.6, +15.30% for PyTorch 2.7.1+cpu. All eight median regressions are below 5%. The PyTorch cold 95% interval still reaches 5.18%, so its upper bound is not below 5%. Every other upper bound is below 5%. We retained all 1,860 measured installs, including outliers, and combined every confirmation run for this candidate. Cold measurements have 360 paired rounds for AnyIO, 120 each for SymPy and NumPy, and 90 for PyTorch; warm measurements have 60 paired rounds per wheel. Baseline/candidate order alternates, with three warmups per case. Intervals use a paired percentile bootstrap of the ratio of medians with 10,000 resamples. These measurements ran on Linux/ext4, an AMD EPYC-Milan VM pinned to eight CPUs, with Python 3.12.13. Both binaries use Rust 1.98.0 and the same profiling profile (optimized, no LTO). Each sample installs one pinned local wheel offline, without dependencies or bytecode compilation, using hardlinks into a fresh virtual environment. Cold removes the entire uv cache; warm retains a primed cache. Setup and cleanup are untimed, and the wheel OS page cache is warm. No compilation ran during the benchmarks. These results do not cover other platforms or network-inclusive installs. All-file deduplication, content/executable identities, the cache layout, complete archives, and copy fallbacks are preserved. Inode checks confirmed that every archived file shares its file-store object for all four wheels. The five targeted integration tests passed ten stress iterations (50 executions), including local and streamed wheels with one and four workers, RECORD handling, cache cleanup, and cross-filesystem installation. Formatting and Clippy with warnings denied also passed. All reactions Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. charliermarsh temporarily deployed to automations August 27, 2026 23:35 — with GitHub Actions Inactive codspeed-hq Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page. Copy link Copy Markdown Merging this PR will not alter performance ✅ 25 untouched benchmarks ⏩ 12 skipped benchmarks1 Comparing charlie/dirhash-all-files (73c621b) with main (dd85179) Footnotes 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩ All reactions Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. charliermarsh marked this pull request as ready for review August 28, 2026 00:03 charliermarsh force-pushed the charlie/dirhash-binaries branch from a3f977e to 1f28fd5 Compare August 28, 2026 00:08 charliermarsh force-pushed the charlie/dirhash-all-files branch from 0a8edc0 to a108fab Compare August 28, 2026 00:08 charliermarsh temporarily deployed to automations August 28, 2026 00:09 — with GitHub Actions Inactive charliermarsh changed the base branch from charlie/dirhash-binaries to main August 28, 2026 00:09 charliermarsh added enhancement New feature or improvement to existing functionality preview Experimental behavior labels Aug 28, 2026 charliermarsh requested review from woodruffw and zanieb August 28, 2026 00:13 charliermarsh temporarily deployed to automations August 28, 2026 00:23 — with GitHub Actions Inactive charliermarsh mentioned this pull request Aug 28, 2026 Pipeline file cache publication for streamed wheels #21333 Draft astral-sh-bot Bot commented Aug 28, 2026 Copy link Copy Markdown uv test inventory changes This PR changes the tests when compared with the main base revision. Added tests: 5 Removed tests: 0 Changed suites: 2 uv-extract: +1 / -0 Added: uv-extract::dirhash::archive::tests::extracted_file_executable_status Removed: none uv::pip_install: +4 / -0 Added: uv::pip_install::pip_install::all_files_except_record_use_archive_file_store uv::pip_install::pip_install::binary_payload_copy_fallback_uses_archive_file_store uv::pip_install::pip_install::binary_payloads_stay_in_archive_without_preview uv::pip_install::pip_install::binary_payloads_use_archive_file_store Removed: none All reactions Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. charliermarsh mentioned this pull request Aug 28, 2026 Store cached wheel permissions in archive manifests #21334 Draft zsol self-requested a review August 28, 2026 09:53 This was referenced Aug 28, 2026 Prefetch wheel central directories for streamed file publication #21338 Draft Pipeline wheel file publication with executable deferral #21339 Draft charliemarsh-oai mentioned this pull re