Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions core/packages/gax/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,4 @@ dist/
*.tgz
**/*.tgz
test/showcase-echo-client/protos/protos.*
test/showcase-resumable-upload/fixtures/protos/protos.*
42 changes: 42 additions & 0 deletions core/packages/gax/client-libraries.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,48 @@ in the second parameter:
const [response] = await client.sampleMethod(request, options);
```

### Resumable uploads

Some APIs expose methods that upload large payloads through the resumable
upload protocol. For these methods, the client method no
longer returns the response directly; it returns a
[`ResumableUpload`](https://googleapis.dev/nodejs/google-gax/latest/classes/ResumableUpload.html)
helper. Call `start()` with a `NodeJS.ReadableStream` and await `finished()`
for the final response:

```ts
const helper = await client.createResumableUpload(request);
await helper.start({
uploadStream: dataStream,
chunkSize: 8 * 1024 * 1024, // 8MB chunks
onProgress: status => {
console.log(`Committed ${status.bytesUploaded} bytes to ${status.uploadUrl}`);
},
});
const response = await helper.finished();
Comment on lines +162 to +170

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The example for starting a resumable upload appears to be incorrect. The start method expects an uploadSource parameter of type ResumableSource, but the example uses uploadStream. Additionally, the dataStream variable is not defined, which could be confusing.

Note: Since this is an auto-generated file, please do not edit it directly as these changes will be overwritten. Instead, apply this fix upstream in the generator or templates.

Suggested change
const helper = await client.createResumableUpload(request);
await helper.start({
uploadStream: dataStream,
chunkSize: 8 * 1024 * 1024, // 8MB chunks
onProgress: status => {
console.log(`Committed ${status.bytesUploaded} bytes to ${status.uploadUrl}`);
},
});
const response = await helper.finished();
const uploadSource = client.getResumableSource(filePath);
const helper = await client.createResumableUpload(request);
await helper.start({
uploadSource,
chunkSize: 8 * 1024 * 1024, // 8MB chunks
onProgress: status => {
console.log('Committed ' + status.bytesUploaded + ' bytes to ' + status.uploadUrl);
},
});
const response = await helper.finished();
References
  1. Do not manually edit auto-generated files to fix typos or make other changes, as these edits will be overwritten during the next regeneration. Instead, apply the fixes upstream in the generator or templates.

```

The session URL is available as `helper.uploadUrl` once the upload has
started. Save it if you need to resume the upload later — for example after a
process crash or network drop. To resume, pass the saved URL to `start()` on a
new helper, along with a fresh stream of the same payload:

```ts
const helper = await client.createResumableUpload();
await helper.start({
uploadStream: dataStream,
resumeUrl: savedUploadUrl,
});
const response = await helper.finished();
Comment on lines +179 to +184

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similar to the previous example, this code snippet for resuming an upload uses the incorrect uploadStream parameter. It should be uploadSource. When resuming, you still need to provide the same payload source.

Note: Since this is an auto-generated file, please do not edit it directly as these changes will be overwritten. Instead, apply this fix upstream in the generator or templates.

Suggested change
const helper = await client.createResumableUpload();
await helper.start({
uploadStream: dataStream,
resumeUrl: savedUploadUrl,
});
const response = await helper.finished();
const uploadSource = client.getResumableSource(filePath);
const helper = await client.createResumableUpload();
await helper.start({
uploadSource,
resumeUrl: savedUploadUrl,
});
const response = await helper.finished();
References
  1. Do not manually edit auto-generated files to fix typos or make other changes, as these edits will be overwritten during the next regeneration. Instead, apply the fixes upstream in the generator or templates.

```

The current implementation requires a seekable stream (for example, a file
stream). Errors fall into three categories: transient errors (retried with
exponential backoff), state mismatches (recovered by querying the server for
the committed byte offset), and fatal errors (propagated to the caller).
The whole session is bounded by a global deadline (10 minutes by default,
scaled up for large payloads and overridable via `globalDeadlineMs`).

### Long-running operations

Some methods are expected to run longer. They return an object of type
Expand Down
2 changes: 2 additions & 0 deletions core/packages/gax/src/clientInterface.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import {
PageDescriptor,
StreamDescriptor,
} from './descriptor';
import {ResumableUploadDescriptor} from './resumableUpload';
import * as longrunning from './longRunningCalls/longrunning';
import * as operationProtos from '../protos/operations';

Expand All @@ -51,6 +52,7 @@ export interface Descriptors {
stream: {[name: string]: StreamDescriptor};
longrunning: {[name: string]: LongrunningDescriptor};
batching?: {[name: string]: BundleDescriptor};
resumableUpload?: {[name: string]: ResumableUploadDescriptor};
}

export interface Callback<
Expand Down
1 change: 1 addition & 0 deletions core/packages/gax/src/descriptor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,4 @@ export {LongRunningDescriptor as LongrunningDescriptor} from './longRunningCalls
export {PageDescriptor} from './paginationCalls/pageDescriptor';
export {StreamDescriptor} from './streamingCalls/streamDescriptor';
export {BundleDescriptor} from './bundlingCalls/bundleDescriptor';
export {ResumableUploadDescriptor} from './resumableUpload';
12 changes: 12 additions & 0 deletions core/packages/gax/src/fallback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,18 @@ export {
PageDescriptor,
StreamDescriptor,
} from './descriptor';
export {
ResumableUploadDescriptor,
ResumableUploadSession,
ResumableUploadState,
resumableUploadStub,
} from './resumableUpload';
export type {
ResumableUploadContext,
ResumableUploadProgress,
ResumableUploadStartParams,
ResumableSource,
} from './resumableUpload';

export {StreamType} from './streamingCalls/streaming';

Expand Down
14 changes: 14 additions & 0 deletions core/packages/gax/src/gax.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
*/

import type {Message} from 'protobufjs';
import type {ResumableUploadContext} from './resumableUpload';
import {warn} from './warnings';
import {GoogleError} from './googleError';
import {BundleOptions} from './bundlingCalls/bundleExecutor';
Expand Down Expand Up @@ -172,6 +173,11 @@ export interface CallOptions {
apiName?: string;
retryRequestOptions?: RetryRequestOptions;
enableTelemetryTracing?: boolean;
/**
* Internal context used by resumable upload methods. Populated by
* GAPIC-generated client libraries; do not set manually.
*/
resumableUpload?: ResumableUploadContext;
}

export class CallSettings {
Expand All @@ -189,6 +195,7 @@ export class CallSettings {
apiName?: string;
retryRequestOptions?: RetryRequestOptions;
enableTelemetryTracing?: boolean;
resumableUpload?: ResumableUploadContext;

/**
* @param {Object} settings - An object containing parameters of this settings.
Expand Down Expand Up @@ -223,6 +230,8 @@ export class CallSettings {
this.apiName = settings.apiName ?? undefined;
this.retryRequestOptions = settings.retryRequestOptions;
this.enableTelemetryTracing = settings.enableTelemetryTracing;
this.resumableUpload =
'resumableUpload' in settings ? settings.resumableUpload : undefined;
}

/**
Expand All @@ -247,6 +256,7 @@ export class CallSettings {
let apiName = this.apiName;
let retryRequestOptions = this.retryRequestOptions;
let enableTelemetryTracing = this.enableTelemetryTracing;
let resumableUpload = this.resumableUpload;

// If the user provides a timeout to the method, that timeout value will be used
// to override the backoff settings.
Expand Down Expand Up @@ -305,6 +315,9 @@ export class CallSettings {
if ('enableTelemetryTracing' in options) {
enableTelemetryTracing = options.enableTelemetryTracing;
}
if ('resumableUpload' in options) {
resumableUpload = options.resumableUpload;
}

return new CallSettings({
timeout,
Expand All @@ -318,6 +331,7 @@ export class CallSettings {
apiName,
retryRequestOptions,
enableTelemetryTracing,
resumableUpload,
});
}
}
Expand Down
13 changes: 13 additions & 0 deletions core/packages/gax/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,19 @@ export {
PageDescriptor,
StreamDescriptor,
} from './descriptor';
export {ResumableUploadDescriptor} from './resumableUpload';
export {
ResumableUploadSession,
resumableUploadStub,
ResumableUploadState,
} from './resumableUpload';
export type {
ResumableUploadContext,
ResumableUploadProgress,
ResumableUploadStartParams,
ResumableSource,
} from './resumableUpload';
export {resumableSourceFromFile} from './resumableSourceFromFile';
export {
CallOptions,
CallSettings,
Expand Down
43 changes: 43 additions & 0 deletions core/packages/gax/src/resumableSourceFromFile.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import {createReadStream} from 'fs';

Check warning on line 17 in core/packages/gax/src/resumableSourceFromFile.ts

View workflow job for this annotation

GitHub Actions / lint

'fs' imported multiple times
import {statSync} from 'fs';

Check warning on line 18 in core/packages/gax/src/resumableSourceFromFile.ts

View workflow job for this annotation

GitHub Actions / lint

'fs' imported multiple times

import {ResumableSource} from './resumableUpload';

/**
* Creates a {@link ResumableSource} backed by a local file.
*
* This factory lives outside the fallback transport entrypoint so browser
* builds do not pull in the Node.js `fs` module. Generated clients should
* delegate to this function rather than constructing file streams directly.
*/
export function resumableSourceFromFile(filePath: string): ResumableSource {
const stat = statSync(filePath);
return {
size: stat.size,
getStream: (offset?: number) => {
const start = offset ?? 0;
if (start < 0 || start > stat.size) {
throw new RangeError(
`Invalid start offset ${start} for file of size ${stat.size}.`,
);
}
return createReadStream(filePath, {start});
},
};
}
Loading
Loading