Load CUDA external data through pinned buffers

[Xavier Dupré (xadupre)]

Description

Load large CUDA external initializers directly from their files into two reusable 64 MiB pinned host buffers.

The CUDA Execution Provider now supplies an IExternalDataLoader. For each block, independent reads fill disjoint ranges of the next pinned buffer while the preceding buffer is transferred asynchronously to the GPU. The two buffers and CUDA streams are retained for the lifetime of the loader and synchronized before reuse and before returning.

This avoids the previous mmap -> pageable CPU memory -> pinned memory -> GPU path. CPU and other execution providers keep their existing external-data behavior, and ordinary CUDA data transfers keep the existing CUDA-managed pageable-memory staging.

The default IExternalDataLoader::LoadTensor implementation remains available but is defined inline, so each
provider shared library can emit the interface type information it needs.

The PR also includes a standalone benchmark for CUDA InferenceSession creation with targeted page-cache eviction.

Configuration

The CUDA EP provider option external_data_loader_reading_threads controls how
many independent CPU read tasks fill each 64 MiB pinned staging buffer. The
default is 4, which was the fastest setting on the benchmarked eight-disk
NVMe volume. Each task reads a disjoint range of the active buffer; once every
range is complete, the whole buffer is submitted to CUDA while the next buffer
is filled. Values from 0 through 64 are accepted because the best value depends
on the storage device, filesystem, and host:

• 0 disables the custom CUDA external-data loader and uses the existing
  framework loading path.
• 1 uses the pinned-buffer loader but reads synchronously on the calling
  thread, without creating a reader task.
• 2..64 use that many parallel read tasks per 64 MiB block.

Python example:

providers = [
    (
        "CUDAExecutionProvider",
        {"external_data_loader_reading_threads": 4},
    ),
    "CPUExecutionProvider",
]

The benchmark script exposes the same setting as --reading-threads.

Synchronization and locking

• One loader-wide mutex protects both pinned buffers and both CUDA streams for an
  entire initializer load. Initializers therefore cannot concurrently reuse the
  same staging resources.
• Reader tasks do not take a shared mutex: every task writes to a distinct,
  non-overlapping range of the active pinned buffer.
• Joining all reader futures is the barrier that guarantees the active buffer is
  completely filled before its H2D copy is submitted.
• Each buffer has its own CUDA stream. The stream is synchronized before that
  buffer is reused, so CPU readers never overwrite memory still consumed by DMA.
• Both streams are drained before a successful return and on read or copy errors.

Data path

The complete cold-cache path is:

                         External-data file on local NVMe
                                      |
                                      | NVMe DMA / block I/O
                                      v
                         Linux kernel page-cache pages
                                      |
                                      | CPU copies performed by read()
                                      | 4 independent reads per 64M block by default
                                      | direct NVMe read reference: 6.8 GB/s
                                      |
                    +-----------------+-----------------+
                    |                                   |
                    v                                   v
         +-----------------------+           +-----------------------+
         | pinned buffer 0, 64M  |           | pinned buffer 1, 64M  |
         | CPU fills block N     |           | CPU fills block N + 1 |
         +-----------------------+           +-----------------------+
                    |                                   |
                    | cudaMemcpyAsync                   | cudaMemcpyAsync
                    | pinned -> VRAM: ~55.4 GB/s        | pinned -> VRAM: ~55.4 GB/s
                    |                                   |
                    +-----------------+-----------------+
                                      |
                                      v
                      CUDA BFC Arena initializer buffer
                                      |
                                      | optional CUDA prepack,
                                      | transpose and unpack kernels
                                      | device reads + device writes
                                      v
                          Final prepared CUDA weights

  Timeline:

    CPU reads block N + 1 into buffer 1
           || concurrently with
    PCIe DMA transfers block N from buffer 0

    CPU reads block N + 2 into buffer 0
           || concurrently with
    PCIe DMA transfers block N + 1 from buffer 1

There is no intermediate mmap-backed pageable tensor and no
mmap -> pinned user-space copy. Standard buffered file I/O still necessarily
copies data from Linux page-cache pages into the pinned user-space buffer.
That CPU copy was not measured independently; the 4.6 GB/s figure covers
the complete cold buffered-read path from the NVMe file into pinned memory.
CUDA then performs one H2D DMA from that pinned buffer into the initializer's
device allocation at approximately 55.4 GB/s in the pinned-memory H2D
microbenchmark. Operators that prepack weights may subsequently read that CUDA
allocation and write a transformed CUDA allocation.

In the benchmarked default configuration, the H2D destination is the
initializer buffer planned and allocated from the CUDA BFC Arena. The new
loader writes each block directly into its final offset in that arena buffer.

This replaces the previous path:

  External-data file on NVMe
              |
              | page faults / block I/O
              v
  Linux page cache + mmap-backed pageable CPU tensor
              |
              | synchronous cudaMemcpy from pageable memory
              | CUDA driver-managed internal pinned staging
              v
  CUDA BFC Arena initializer buffer
              |
              | optional CUDA prepack/transpose/unpack
              v
  Final prepared CUDA weights

The new path removes the full mmap-backed CPU tensor and the CUDA driver's
implicit pageable-memory staging. It replaces them with controlled parallel
read() calls directly into two persistent pinned buffers followed by explicit
asynchronous H2D copies into the same CUDA BFC Arena destination.

Controlled loading benchmarks

Model: qwen3.5-35b-cuda-int4/model.onnx

• 20,895,301,632-byte external-data file
• NVIDIA H200, GPU 0
• 96 intra-op threads, spinning disabled
• fresh process for every run
• cold runs use targeted POSIX_FADV_DONTNEED for both model files
• warm runs retain the Linux page cache
• baseline: merge-base eebea690a0
• candidate: this PR at 79b8727e1f
• both variants built from source with GCC 13.2, Release, Python 3.12, CUDA 13.3, the same cuDNN 9 package, shared library, CUDA NHWC enabled, and CMAKE_CUDA_ARCHITECTURES=native

CUDA was verified as active in both builds. A held candidate session allocated approximately 39.4 GiB on the H200.

CUDA

Version	Cache	Run 1	Run 2	Run 3	Mean
Merge-base eebea690a0	Cold	30.93 s	31.42 s	30.75 s	31.03 s
This PR 79b8727e1f	Cold	26.58 s	26.59 s	26.69 s	26.62 s
Merge-base eebea690a0	Warm	28.15 s	28.40 s	26.82 s	27.79 s
This PR 79b8727e1f	Warm	24.31 s	24.67 s	24.28 s	24.42 s

The direct pinned loader reduces complete CUDA InferenceSession creation time by 14.2% cold
(4.41 seconds) and 12.1% warm (3.37 seconds).

A direct-I/O sequential read (dd, 16 MiB blocks) read the external-data file in 3.09 seconds, or
6.8 GB/s. The remaining session creation time includes graph initialization, CUDA allocation,
weight upload/preparation, and synchronization.

The earlier 10.63/7.07-second figures were invalid because that build included changes from another PR.
The earlier PyPI comparison was also removed in favor of this same-revision, same-toolchain comparison.

CUDA BF16 reading-thread scaling

Model: qwen3.5-35b-cuda-bf16/model.onnx

• 69,451,776,000 bytes of external data
• NVIDIA H200, GPU 0
• fresh process and targeted page-cache eviction for every measurement
• CUDA initialization included in the measured InferenceSession creation time
• three cold-cache repetitions per configuration
• baseline: merge-base eebea690a0
• candidate: this PR, with external_data_loader_reading_threads set to each value from 0 through 6

Configuration	Run 1	Run 2	Run 3	Mean	Difference vs. baseline
Merge-base, without PR	47.420 s	46.954 s	46.846 s	47.073 s	reference
PR, 0 reading threads	47.208 s	46.897 s	47.044 s	47.049 s	-0.1%
PR, 1 reading thread	48.723 s	48.964 s	49.416 s	49.034 s	+4.2%
PR, 2 reading threads	39.685 s	37.790 s	37.847 s	38.441 s	-18.3%
PR, 3 reading threads	34.345 s	34.364 s	34.334 s	34.348 s	-27.0%
PR, 4 reading threads	33.061 s	33.073 s	33.066 s	33.067 s	-29.8%
PR, 5 reading threads	33.230 s	33.224 s	33.255 s	33.236 s	-29.4%
PR, 6 reading threads	34.760 s	33.056 s	33.018 s	33.611 s	-28.6%

The 0 case is the existing framework implementation, not a zero-worker
variant of the new loader. Its mean is within 0.1% of the build without the PR,
confirming that it preserves the previous behavior. A single pinned-buffer
reader is 4.2% slower than the existing path, so pinned memory alone does not
provide the gain. The improvement comes from parallel reads: performance
plateaus at four to five readers, with the default of 4 producing the best
mean at 33.067 seconds, a 29.8% reduction from the baseline.

CPU control

The CUDA external-data loader is not registered in CPU-only sessions; CPU external initializers retain
the existing mmap path. The controlled CPU comparison confirms no meaningful change.

Version	Full cold	Full warm
Merge-base eebea690a0	463.86 s	468.66 s
This PR 79b8727e1f	464.51 s	466.23 s

The full cold difference is +0.14% and the warm difference is -0.52%, both within run-to-run
noise for an 8-minute CPU prepacking workload. With prepacking disabled, three-run means were
0.406/0.380 seconds (base cold/warm) and 0.418/0.376 seconds (PR cold/warm), likewise showing no
material CPU-path effect.

Validation

• Built controlled CPU and CUDA wheels for the merge-base and PR head with matching toolchains/options.
• Loaded the Qwen3.5 35B CUDA INT4 model with CUDA active in both builds.
• Added CUDA correctness coverage below/at the 16 MiB read threshold, for disabling the loader, for synchronous reads, across repeated alternating-buffer reuse, and across device restoration; all six targeted tests pass.
• Checked C++ and Python formatting.

[Copilot]

🟡 Changes recommended

Error paths can leave DMA in flight, and setup failures regress previously valid transfers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds pinned-buffer staging to accelerate large synchronous pageable host-to-CUDA transfers.

Changes:

• Alternates 64 MiB chunks across two CUDA streams.
• Retains per-device staging resources.
• Adds a cold model-loading benchmark.

File summaries

File	Description
gpu_data_transfer.cc	Implements staged CUDA transfers.
gpu_data_transfer.h	Declares staging state and synchronization.
benchmark_cuda_model_loading.py	Benchmarks CUDA session creation.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 5
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Xavier Dupré (xadupre)]

Controlled benchmark correction. The previous 10.63/7.07-second results mixed another PR, and the PyPI comparison used a different release. The final comparison uses the merge-base eebea690a0 and this PR at 79b8727e1f, both built from source with the same GCC 13.2, Release, Python 3.12, CUDA 13.3, cuDNN 9, shared-library, CUDA-NHWC, and native-H200 settings.

Model: /mnt/nvme/xadupre/models/qwen/qwen3.5-35b-cuda-int4/model.onnx (20.895 GB external data), NVIDIA H200 GPU 0, 96 intra-op threads, fresh process per run.

Version	Cache	Run 1	Run 2	Run 3	Mean
Base eebea690a0	CUDA cold	30.93 s	31.42 s	30.75 s	31.03 s
PR 79b8727e1f	CUDA cold	26.58 s	26.59 s	26.69 s	26.62 s
Base eebea690a0	CUDA warm	28.15 s	28.40 s	26.82 s	27.79 s
PR 79b8727e1f	CUDA warm	24.31 s	24.67 s	24.28 s	24.42 s

The PR reduces complete CUDA session creation by 14.2% cold and 12.1% warm. Direct sequential read throughput was 6.8 GB/s.

CPU control, using matching CPU builds: base 463.86 s cold / 468.66 s warm; PR 464.51 s cold / 466.23 s warm. The differences (+0.14% cold, -0.52% warm) are measurement noise, as expected because CPU-only sessions do not register the CUDA loader.

[Xavier Dupré (xadupre)]

Additional profiling clarified the denominator for the loading-time improvement.

The first CUDA Runtime call in the process (cudaGetDeviceCount() while parsing the CUDA EP options) triggers CUDA driver/context initialization. On this machine, that one-time initialization takes approximately 20.6-20.7 seconds. This was confirmed by explicitly calling cudaSetDevice(0) followed by cudaFree(0) before constructing the ORT session:

Phase	Time
One-time CUDA runtime/context initialization	20.63-20.73 s
ORT session creation after CUDA initialization	5.91 s
ORT session creation without prepacking	4.50 s
CUDA external-data loader	4.15-4.16 s
File reads inside the loader	3.91-3.93 s
Prepacking	~1.41 s

The ~20-second CUDA startup is model-independent, paid once per process/device, and effectively fixed/incompressible from the perspective of this PR. Nsight Systems initializes CUDA before the Python timer, which also explains why profiled runs appeared to take only 7-8 seconds while normal fresh-process runs took 26-30 seconds.

Therefore, the roughly 3-second improvement from this PR should be interpreted relative to the approximately 10-second model-loading portion, rather than relative to the full approximately 30-second fresh-process time. In other words, the relevant improvement is closer to 3 seconds out of 10, not 3 seconds out of 30.

[Copilot]

🟡 Changes recommended

Pageable fallback synchronization, stream-error cleanup, and direct-struct option validation remain incorrect.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Reader task exceptions can bypass CUDA stream cleanup and permit reuse of a pinned buffer while its prior transfer remains active.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

onnxruntime/core/providers/cuda/cuda_external_data_loader.cc:54

• Repository guidance requires InlinedVector instead of std::vector for internal C++ containers (AGENTS.md:81-90). This reader list is bounded by 64 entries, so use InlinedVector<std::future<common::Status>> and include core/common/inlined_containers.h.

This issue also appears on line 83 of the same file.

onnxruntime/core/providers/cuda/cuda_external_data_loader.cc:83

• Repository guidance requires InlinedVector instead of std::vector for internal C++ containers (AGENTS.md:81-90). Replace this pageable byte buffer with the ORT container alias as well, keeping the bounded allocation behavior unchanged.

  std::vector<uint8_t> buffer(std::min(kBufferSize, length));

• Files reviewed: 13/13 changed files
• Comments generated: 1
• Review effort level: Balanced

[Xavier Dupré (xadupre)]

This PR currently depends on #32672.

The failing webgpu_plugin_build_x64_RelWithDebInfo check is caused by the WebGPU header dependency regression already present on main after #32667, not by the CUDA external-data loader changes in this PR. #32672 fixes that regression. Once it is merged, this PR should be updated with main and the WebGPU check rerun.

[Copilot]

🟡 Changes recommended

The CUDA plugin build has an incomplete Tensor type, and setup failures are retried on subsequent loads.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

onnxruntime/core/providers/cuda/cuda_external_data_loader.cc:215

• When pinned-buffer or stream setup fails, this path falls back successfully but leaves no state recording the failure. Every later external initializer retries the 128 MiB pinned allocation and stream creation before falling back again, which can add repeated setup overhead and memory pressure for models with many initializers under the same resource limit. Cache the setup failure for this loader (and use the pageable fallback directly for subsequent loads).

    // TODO: Remember setup failures during initialization and report the first CUDA error
    // so later initializers do not repeatedly retry unavailable pinned buffers or streams.
    return LoadWithPageableBuffer(*file, data_offset, length, tensor, reading_thread_count_, reader_pool_);

• Files reviewed: 19/19 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🟡 Changes recommended

Unresolved range-overflow and staging-resource handling issues require fixes.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

onnxruntime/core/providers/cuda/cuda_external_data_loader.cc:216

• Every CUDA external initializer, including a 1-byte or otherwise sub-threshold one, calls EnsureResources() and allocates two 64 MiB pinned buffers plus two CUDA streams before reading it. That imposes a 128 MiB pinned-memory/setup cost on small models where the parallel loader provides no benefit; avoid creating the fixed staging resources for small requests (or otherwise size/lazily select the fallback path).

onnxruntime/core/framework/tensorprotoutils.cc:1846

• This new error message is missing a separator between the tensor name and the explanation, so it is emitted as TensorProto for: weightsExpected to have external data. Add a leading space before expected so failures identify the problem clearly.

              tensor_proto.name(), "Expected to have external data");

onnxruntime/core/providers/cuda/cuda_external_data_loader.cc:216

• After setup fails, buffers_[0] remains null, so each later external initializer retries both pinned-buffer/stream setup and cleanup before taking the pageable fallback. A model with many external initializers can therefore repeat a costly 128 MiB pin/unpin attempt for every tensor; cache the setup failure for this loader and use the fallback directly for subsequent loads.

    // TODO: Remember setup failures during initialization and report the first CUDA error
    // so later initializers do not repeatedly retry unavailable pinned buffers or streams.
    return LoadWithPageableBuffer(*file, data_offset, length, tensor, reading_thread_count_, reader_pool_);

• Files reviewed: 19/19 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🔵 Needs a closer look

The repeated-load regression test cannot detect a skipped or stale second load because it reuses identical source and destination contents.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

onnxruntime/test/providers/cuda/test_cases/cuda_external_data_loader_test.cc:214

• This repeated-load case writes the same bytes into the same already-populated tensor, so it still passes if a later LoadTensor call fails to refill or copy the persistent staging buffers. Verify after each load and change the source bytes (or use a fresh sentinel-filled destination) before the second call so cross-call buffer reuse is actually exercised.

• Files reviewed: 19/19 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Xavier Dupré (xadupre)]

Addressed the latest suppressed Copilot review comment in cb4dbb3f9e. The repeated-load test now verifies the tensor after every load, overwrites the GPU destination with a sentinel first, and uses a different source-byte pattern for the second call, so a skipped or stale second load fails. This feedback was emitted only in a review summary and did not create a resolvable review thread; there are currently no unresolved review threads on this PR.

[Copilot]

🔵 Needs a closer look

Cross-platform concurrent I/O, CUDA synchronization, and shared-library interface changes warrant final human review.

Review details

• Files reviewed: 19/19 changed files
• Comments generated: 0 new
• Review effort level: Balanced