From 65dc1ad2464a342e7850313d7683fdda7aa1252a Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:32:17 -0600 Subject: [PATCH 01/12] Fix typos and a stale claim in the docs Corrects overalapping, manullay, "Streams that contains" and "If all candidates fails" in the Advanced usage patterns guide, and replaces the time-relative "developed for 3 years" claim on the overview page. Closes #591 Co-Authored-By: Claude Opus 5 --- changelog.d/591.doc.md | 1 + doc/source/advanced.rst | 6 +++--- doc/source/index.rst | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) create mode 100644 changelog.d/591.doc.md diff --git a/changelog.d/591.doc.md b/changelog.d/591.doc.md new file mode 100644 index 000000000..3f62ef354 --- /dev/null +++ b/changelog.d/591.doc.md @@ -0,0 +1 @@ +Fix typos in the Advanced usage patterns guide and remove an outdated time-relative claim about the library's age from the overview. diff --git a/doc/source/advanced.rst b/doc/source/advanced.rst index 5b78c8f9c..400e548af 100644 --- a/doc/source/advanced.rst +++ b/doc/source/advanced.rst @@ -79,7 +79,7 @@ Concatenate files of known size If one of remote source has length smaller than :term:`absoluteMinimumPartSize` then it cannot be copied into large file part. Such remote source would be downloaded and concatenated locally with local source or with other downloaded remote source. -Please note that this method only allows checksum verification for local upload sources. Checksum verification for remote sources is available only when local copy is available. In such case :meth:`b2sdk.v3.Bucket.create_file` can be used with overalapping ranges in input. +Please note that this method only allows checksum verification for local upload sources. Checksum verification for remote sources is available only when local copy is available. In such case :meth:`b2sdk.v3.Bucket.create_file` can be used with overlapping ranges in input. For more information about ``concatenate`` please see :meth:`b2sdk.v3.Bucket.concatenate` and :class:`b2sdk.v3.CopySource`. @@ -353,7 +353,7 @@ Manual continuation (streamed version) >>> large_file_id = storage.query({'name': remote_name})[0]['large_file_id'] >>> bucket.create_file_stream(input_sources, remote_name, file_info, large_file_id=large_file_id) -Streams that contains remote sources cannot be continued with :meth:`b2sdk.v3.Bucket.create_file` - internally :meth:`b2sdk.v3.Bucket.create_file` stores plan information in file info for such inputs, and verifies it before any copy/upload and :meth:`b2sdk.v3.Bucket.create_file_stream` cannot store this information. Local source only inputs can be safely continued with :meth:`b2sdk.v3.Bucket.create_file` in auto continue mode or manual continue mode (because plan information is not stored in file info in such case). +Streams that contain remote sources cannot be continued with :meth:`b2sdk.v3.Bucket.create_file` - internally :meth:`b2sdk.v3.Bucket.create_file` stores plan information in file info for such inputs, and verifies it before any copy/upload and :meth:`b2sdk.v3.Bucket.create_file_stream` cannot store this information. Local source only inputs can be safely continued with :meth:`b2sdk.v3.Bucket.create_file` in auto continue mode or manual continue mode (because plan information is not stored in file info in such case). Auto continuation ----------------- @@ -364,7 +364,7 @@ Auto continuation For local source only input, :meth:`b2sdk.v3.Bucket.create_file` would try to find matching unfinished large file. It will verify uploaded parts checksums with local sources - the most completed, having all uploaded parts matched candidate would be automatically selected as file to continue. If there is no matching candidate (even if there are unfinished files for the same file name) new large file would be started. -In other cases plan information would be generated and :meth:`b2sdk.v3.Bucket.create_file` would try to find unfinished large file with matching plan info in its file info. If there is one or more such unfinished large files, :meth:`b2sdk.v3.Bucket.create_file` would verify checksums for all locally available parts and choose any matching candidate. If all candidates fails on uploaded parts checksums verification, process is interrupted and error raises. In such case corrupted unfinished large files should be cancelled manullay and :meth:`b2sdk.v3.Bucket.create_file` should be retried, or auto continuation should be turned off with `auto_continue=False` +In other cases plan information would be generated and :meth:`b2sdk.v3.Bucket.create_file` would try to find unfinished large file with matching plan info in its file info. If there is one or more such unfinished large files, :meth:`b2sdk.v3.Bucket.create_file` would verify checksums for all locally available parts and choose any matching candidate. If all candidates fail on uploaded parts checksums verification, process is interrupted and error raises. In such case corrupted unfinished large files should be cancelled manually and :meth:`b2sdk.v3.Bucket.create_file` should be retried, or auto continuation should be turned off with `auto_continue=False` No continuation diff --git a/doc/source/index.rst b/doc/source/index.rst index f1f2f61fa..ea1e4ddda 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -35,7 +35,7 @@ When building an application which uses B2 cloud, it is possible to implement an - reporting progress of operations to an object of your choice - exception hierarchy makes it easy to display informative messages to users - interrupted transfers are automatically continued -- **b2sdk** has been developed for 3 years before it version 1.0.0 was released. It's stable and mature. +- **b2sdk** had been in development for years before its 1.0.0 release. It's stable and mature. ######################################### From a7511707cf9d328e54a93a1123aabbce52be37bc Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:32:24 -0600 Subject: [PATCH 02/12] Install the local checkout in the development install guide The Installation Guide told readers to check out the repository and then run "pip install b2sdk", which resolves the published package from PyPI and ignores local changes. Use an editable install of the checkout, and point at the Contributors Guide for the full nox/uv setup. Closes #590 Co-Authored-By: Claude Opus 5 --- changelog.d/590.doc.md | 1 + doc/source/install.rst | 13 ++++++++++--- 2 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 changelog.d/590.doc.md diff --git a/changelog.d/590.doc.md b/changelog.d/590.doc.md new file mode 100644 index 000000000..af97b4e72 --- /dev/null +++ b/changelog.d/590.doc.md @@ -0,0 +1 @@ +Correct the Installation Guide's development-install instructions, which told readers to check out the repository and then install the published package from PyPI instead of their local checkout. diff --git a/doc/source/install.rst b/doc/source/install.rst index 159ea3332..1b2d75ea1 100644 --- a/doc/source/install.rst +++ b/doc/source/install.rst @@ -15,8 +15,15 @@ In order to properly set a dependency, see :ref:`versioning chapter `_ and +`uv `_ to manage test and documentation environments. See +:ref:`contributors_guide` for the full development setup. From 2e10606eeeed33006dad854f380ba06a4923b3ee Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:32:32 -0600 Subject: [PATCH 03/12] Correct the nox session list and CI matrix in contributor docs The Sphinx contributor guide advertised a "deploy" nox session that does not exist; releases are published by the Continuous Delivery workflow on a version tag. Both contributor guides also claimed a matrix of 8 Python versions, which no longer matches PYTHON_VERSIONS; describe the matrix by reference so it does not drift again. Closes #588 Closes #589 Co-Authored-By: Claude Opus 5 --- CONTRIBUTING.md | 4 ++-- changelog.d/588.doc.md | 1 + changelog.d/589.doc.md | 1 + doc/source/contributing.rst | 8 ++++++-- 4 files changed, 10 insertions(+), 4 deletions(-) create mode 100644 changelog.d/588.doc.md create mode 100644 changelog.d/589.doc.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c561287bc..e8607ec8c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,8 +14,8 @@ In order to make it easier to contribute, core developers of this project: * maintain Continuous Integration (by using GitHub Actions) that: * runs all sorts of linters * checks if the Python distribution can be built - * runs all tests on a matrix of 8 versions of Python (including pypy) and 3 operating systems - (Linux, Mac OS X and Windows) + * runs all tests on a matrix of every supported Python version (CPython and PyPy, see + `PYTHON_VERSIONS` in [noxfile.py](noxfile.py)) across Linux, macOS and Windows * checks if the documentation can be built properly * maintain other Continuous Integration tools (coverage tracker) diff --git a/changelog.d/588.doc.md b/changelog.d/588.doc.md new file mode 100644 index 000000000..c07000e08 --- /dev/null +++ b/changelog.d/588.doc.md @@ -0,0 +1 @@ +Remove the nonexistent `deploy` Nox session from the Contributors Guide and describe how releases are actually published by the Continuous Delivery workflow. diff --git a/changelog.d/589.doc.md b/changelog.d/589.doc.md new file mode 100644 index 000000000..f4b8015d1 --- /dev/null +++ b/changelog.d/589.doc.md @@ -0,0 +1 @@ +Correct the stale CI test matrix description in the contributor guides, which claimed eight Python versions instead of the currently supported set. diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst index 03c33fe9a..ba7a1d100 100644 --- a/doc/source/contributing.rst +++ b/doc/source/contributing.rst @@ -20,7 +20,7 @@ We encourage outside contributors to perform changes on our codebase. Many such * runs all sorts of linters * checks if the Python distribution can be built - * runs all tests on a matrix of 8 versions of Python (including pypy) and 3 operating systems (Linux, Mac OS X and Windows) + * runs all tests on a matrix of every supported Python version (CPython and PyPy, see ``PYTHON_VERSIONS`` in ``noxfile.py``) across Linux, macOS and Windows * checks if the documentation can be built properly * maintain other Continuous Integration tools (coverage tracker) @@ -36,10 +36,14 @@ With ``nox``, you can run different sessions (default are ``lint`` and ``test``) * ``test`` (``test-3.10``, ``test-3.11``, ``test-3.12``, ``test-3.13``, ``test-3.14``, ``test-pypy3.10``) -> Run test suite. * ``cover`` -> Perform coverage analysis. * ``build`` -> Build the distribution. -* ``deploy`` -> Deploy the distribution to the PyPi. * ``doc`` -> Build the documentation. * ``doc_cover`` -> Perform coverage analysis for the documentation. +Releases are not published from a ``nox`` session. The *Continuous Delivery* GitHub Actions +workflow (``.github/workflows/cd.yml``) builds the distribution when a version tag is pushed, +creates a GitHub release, and uploads it to PyPI for non-prerelease versions. The release +procedure itself is described in ``README.release.md``. + For example:: $ nox -s format From 6d5ca860122f1f0652583ddfcc0d4996b2c2aa15 Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:37:28 -0600 Subject: [PATCH 04/12] Fix invalid API usage in the Quick Start examples The sync example used a nonexistent EncryptionSettings class, passed id= to EncryptionKey instead of key_id=, and gave BasicSyncEncryptionSettingsProvider one mapping where it requires both read and write mappings. Uploads documented file_infos= instead of file_info=. The file-lock examples also handed a FileRetentionSetting to update_file_legal_hold, which takes a LegalHold; those calls now use update_file_retention. Closes #584 Closes #585 Co-Authored-By: Claude Opus 5 --- changelog.d/584.doc.md | 1 + changelog.d/585.doc.md | 1 + doc/source/quick_start.rst | 26 +++++++++++++++----------- 3 files changed, 17 insertions(+), 11 deletions(-) create mode 100644 changelog.d/584.doc.md create mode 100644 changelog.d/585.doc.md diff --git a/changelog.d/584.doc.md b/changelog.d/584.doc.md new file mode 100644 index 000000000..756bbea1f --- /dev/null +++ b/changelog.d/584.doc.md @@ -0,0 +1 @@ +Fix the Quick Start examples to match the public `b2sdk.v3` interface: `EncryptionSetting` rather than `EncryptionSettings`, `EncryptionKey(key_id=...)` rather than `id=`, both required mappings for `BasicSyncEncryptionSettingsProvider`, and `file_info=` rather than `file_infos=`. diff --git a/changelog.d/585.doc.md b/changelog.d/585.doc.md new file mode 100644 index 000000000..784e6f02e --- /dev/null +++ b/changelog.d/585.doc.md @@ -0,0 +1 @@ +Use `update_file_retention` instead of `update_file_legal_hold` in the Quick Start file-lock examples that pass a `FileRetentionSetting`. diff --git a/doc/source/quick_start.rst b/doc/source/quick_start.rst index 4c74ecebc..2950ea88c 100644 --- a/doc/source/quick_start.rst +++ b/doc/source/quick_start.rst @@ -50,14 +50,18 @@ Synchronization ) >>> no_progress = False - >>> encryption_settings_provider = BasicSyncEncryptionSettingsProvider({ - 'bucket1': EncryptionSettings(mode=EncryptionMode.SSE_B2), - 'bucket2': EncryptionSettings( + >>> bucket_settings = { + 'bucket1': EncryptionSetting(mode=EncryptionMode.SSE_B2), + 'bucket2': EncryptionSetting( mode=EncryptionMode.SSE_C, - key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', id='user-generated-key-id') + key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', key_id='user-generated-key-id') ), 'bucket3': None, - }) + } + >>> encryption_settings_provider = BasicSyncEncryptionSettingsProvider( + read_bucket_settings=bucket_settings, + write_bucket_settings=bucket_settings, + ) >>> with SyncReport(sys.stdout, no_progress) as reporter: synchronizer.sync_folders( source_folder=source, @@ -176,7 +180,7 @@ Upload file >>> bucket.upload_local_file( local_file=local_file_path, file_name=b2_file_name, - file_infos=file_info, + file_info=file_info, ) @@ -194,14 +198,14 @@ Upload file encrypted with SSE-C >>> file_info = {'how': 'good-file'} >>> encryption_setting = EncryptionSetting( mode=EncryptionMode.SSE_C, - key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', id='user-generated-key-id'), + key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', key_id='user-generated-key-id'), ) >>> bucket = b2_api.get_bucket_by_name(bucket_name) >>> bucket.upload_local_file( local_file=local_file_path, file_name=b2_file_name, - file_infos=file_info, + file_info=file_info, encryption=encryption_setting, ) @@ -337,7 +341,7 @@ Update file lock configuration >>> file_id = '4_z5485a1682662eb3e60980d10_f113f963288e711a6_d20190404_m065910_c002_v0001095_t0044' >>> file_name = 'dummy.pdf' >>> b2_api.update_file_legal_hold(file_id, file_name, LegalHold.ON) - >>> b2_api.update_file_legal_hold( + >>> b2_api.update_file_retention( file_id, file_name, FileRetentionSetting(RetentionMode.GOVERNANCE, int(time.time() + 100)*1000)) @@ -497,10 +501,10 @@ Update file lock >>> # equivalent to >>> b2_api.update_file_legal_hold(file_version.id_, file_version.file_name, LegalHold.ON) >>> b2_api.update_file_legal_hold(download_version.id_, download_version.file_name, LegalHold.ON) - >>> b2_api.update_file_legal_hold( + >>> b2_api.update_file_retention( file_version.id_, file_version.file_name, FileRetentionSetting(RetentionMode.GOVERNANCE, int(time.time() + 100)*1000)) - >>> b2_api.update_file_legal_hold( + >>> b2_api.update_file_retention( download_version.id_, download_version.file_name, FileRetentionSetting(RetentionMode.GOVERNANCE, int(time.time() + 100)*1000)) From 0387d5d69f974494a6e97ddbcf24a37b7ab5bf37 Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:37:34 -0600 Subject: [PATCH 05/12] Fix stale names in the Advanced usage patterns guide The prose referred to LocalUploadSource, which is not exported; the surrounding examples already use UploadSourceLocalFile. One example also called planner.create_file, while every other example in the section calls bucket.create_file. Closes #586 Co-Authored-By: Claude Opus 5 --- changelog.d/586.doc.md | 1 + doc/source/advanced.rst | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 changelog.d/586.doc.md diff --git a/changelog.d/586.doc.md b/changelog.d/586.doc.md new file mode 100644 index 000000000..986ff1691 --- /dev/null +++ b/changelog.d/586.doc.md @@ -0,0 +1 @@ +Fix the Advanced usage patterns guide to reference `UploadSourceLocalFile` rather than the nonexistent `LocalUploadSource`, and to call `bucket.create_file` rather than an undefined `planner` object. diff --git a/doc/source/advanced.rst b/doc/source/advanced.rst index 400e548af..40373d11d 100644 --- a/doc/source/advanced.rst +++ b/doc/source/advanced.rst @@ -150,7 +150,7 @@ The assumption here is that the file has been appended to since it was last uplo >>> bucket.create_file(input_sources, remote_name, file_info) -`LocalUploadSource` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range (when concatenating, local source would have destination offset at the end of remote source) +:class:`b2sdk.v3.UploadSourceLocalFile` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range (when concatenating, local source would have destination offset at the end of remote source) For more information see :meth:`b2sdk.v3.Bucket.create_file`. @@ -179,7 +179,7 @@ Change the middle of the remote file >>> bucket.create_file(input_sources, remote_name, file_info) -`LocalUploadSource` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range. +:class:`b2sdk.v3.UploadSourceLocalFile` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range. For more information see :meth:`b2sdk.v3.Bucket.create_file`. @@ -284,7 +284,7 @@ Here the planner has only used a remote source where remote range was not availa .. code-block:: python - >>> planner.create_file(input_sources, remote_name, file_info, prioritize='remote') + >>> bucket.create_file(input_sources, remote_name, file_info, prioritize='remote') # planner parts: cloud[A, D], local[D, E] Here the planner has only used a local source where remote range was not available, minimizing uploads. From d06de5e1f15fdb3a3ff594205ecec6e81bf11d8b Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:39:29 -0600 Subject: [PATCH 06/12] Make the published Contributors Guide the canonical source CONTRIBUTING.md and doc/source/contributing.rst duplicated environment setup, nox sessions, testing and documentation commands, and had already drifted apart. Move the guidance that only existed in the Markdown copy (versioning, changelog fragments, dependency management, keyword-filtered tests) into the Sphinx guide, and reduce CONTRIBUTING.md to a short GitHub-facing entry point that links to it. Closes #599 Co-Authored-By: Claude Opus 5 --- CONTRIBUTING.md | 157 ++++++------------------------------ changelog.d/599.doc.md | 1 + doc/source/contributing.rst | 54 +++++++++++++ 3 files changed, 80 insertions(+), 132 deletions(-) create mode 100644 changelog.d/599.doc.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e8607ec8c..3c3ee9d32 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,146 +1,39 @@ # Contributing to B2 Python SDK -We encourage outside contributors to perform changes on our codebase. Many such changes have been merged already. -In order to make it easier to contribute, core developers of this project: +We encourage outside contributors to perform changes on our codebase. Many such changes have been merged +already. -* provide guidance (through the issue reporting system) -* provide tool assisted code review (through the Pull Request system) -* maintain a set of unit tests -* maintain a set of integration tests (run with a production cloud) -* maintain development automation tools using [nox](https://github.com/theacodes/nox) that can easily: - * format the code using [ruff](https://github.com/astral-sh/ruff) - * runs linters to find subtle/potential issues with maintainability - * run the test suite on multiple Python versions using [pytest](https://github.com/pytest-dev/pytest) -* maintain Continuous Integration (by using GitHub Actions) that: - * runs all sorts of linters - * checks if the Python distribution can be built - * runs all tests on a matrix of every supported Python version (CPython and PyPy, see - `PYTHON_VERSIONS` in [noxfile.py](noxfile.py)) across Linux, macOS and Windows - * checks if the documentation can be built properly -* maintain other Continuous Integration tools (coverage tracker) +**The full contributors guide is published at +[b2-sdk-python.readthedocs.io/en/master/contributing.html](https://b2-sdk-python.readthedocs.io/en/master/contributing.html)** +(source: [`doc/source/contributing.rst`](doc/source/contributing.rst)). It is the canonical reference for +environment setup, the available `nox` sessions, dependency management, testing, and building the +documentation. This file is a short entry point only — please keep detailed guidance in the published +guide rather than duplicating it here. -## Versioning +## Getting started -This package's versions adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and the versions are -established by reading git tags, i.e. no code or manifest file changes are required when working on PRs. +You'll need [nox](https://github.com/theacodes/nox) and [uv](https://docs.astral.sh/uv/): -## Changelog + pip install nox uv -Each PR needs to have at least one changelog (aka news) item added. This is done by creating files in `changelog.d`. -`towncrier` is used for compiling these files into [CHANGELOG.md](CHANGELOG.md). There are several types of changes -(news): - -1. fixed -2. changed -3. added -4. deprecated -5. removed -6. infrastructure -7. doc - - -The `changelog.d` file name convention is: - -1. If the PR closes a github issue: `{issue_number}.{type}.md` e.g. `157.fixed.md`. Note that the - change description still has to be complete, linking an issue is just there for convenience, a change like - `fixed #157` will not be accepted. -2. If the PR is not related to a github issue: `+{unique_string}.{type}.md` e.g. `+foobar.fixed.md`. - -These files can either be created manually, or using `towncrier` e.g. - - towncrier create -c 'write your description here' 157.fixed.md - -`towncrier create` also takes care of duplicates automatically (if there is more than 1 news fragment of one type -for a given github issue). - -## Developer Info - -You'll need to have [nox](https://github.com/theacodes/nox) and [uv](https://docs.astral.sh/uv/) installed: - -* `pip install nox uv` - -With `nox`, you can run different sessions (default are `lint` and `test`): - -* `format` -> Format the code. -* `lint` -> Run linters. -* `test` (`test-3.10`, `test-3.11`, `test-3.12`, `test-3.13`, `test-3.14`, `test-pypy3.10`) -> Run test suite. -* `cover` -> Perform coverage analysis. -* `build` -> Build the distribution. -* `doc` -> Build the documentation. -* `doc_cover` -> Perform coverage analysis for the documentation. - -For example: - - $ nox -s format - nox > Running session format - nox > Creating virtual environment (virtualenv) using python3.14 in .nox/format - ... - - $ nox -s format - nox > Running session format - nox > Re-using existing virtual environment at .nox/format. - ... - - $ nox --no-venv -s format - nox > Running session format - ... - -Sessions `test` ,`unit`, and `integration` can run on many Python versions, 3.10-3.14 (+ pypy3.10) by default. - -Sessions other than `test` use the last CPython version from `NOX_PYTHONS`, 3.14 by default. - -You can change it: - - export NOX_PYTHONS=3.12,3.14 - -With the above setting, session `test` will run on Python 3.12 and 3.14, and all other sessions on Python 3.14. - -Given Python interpreters should be installed in the operating system or via [pyenv](https://github.com/pyenv/pyenv). - -## Managing dependencies - -We use [uv](https://docs.astral.sh/uv/) for managing dependencies and developing locally. -If you want to change any of the project requirements (or requirement bounds) in `pyproject.toml`, -make sure that `uv.lock` file reflects those changes by using `uv add`, `uv lock` or other -commands - see [documentation](https://docs.astral.sh/uv/). You can verify that the lock file -is up to date by running `uv lock --check`. - -## Linting - -To run all available linters: +Then, to run the linters and the test suite: nox -s lint - -## Testing - -To run all tests on every available Python version: - nox -s test -To run all tests on a specific version: - - nox -s test-3.10 - -To run just unit tests: - - nox -s unit-3.10 - -To run just integration tests: - - export B2_TEST_APPLICATION_KEY=your_app_key - export B2_TEST_APPLICATION_KEY_ID=your_app_key_id - nox -s integration-3.10 - -To run tests by keyword expressions: - - nox -s unit-3.10 -- -k keyword - -## Documentation - -To build the documentation and watch for changes (including the source code): +Integration tests need real B2 credentials; see the published guide for details. - nox -s doc +## Before you open a pull request -To just build the documentation: +* **Add a changelog entry.** Every PR needs at least one news fragment in `changelog.d`, or CI will + fail. Name it `{issue_number}.{type}.md` (e.g. `157.fixed.md`) when the PR closes an issue, or + `+{unique_string}.{type}.md` (e.g. `+foobar.fixed.md`) otherwise. Valid types are `fixed`, `changed`, + `added`, `deprecated`, `removed`, `infrastructure` and `doc`. The description must stand on its own — + a change like `fixed #157` will not be accepted. [towncrier](https://towncrier.readthedocs.io/) + compiles these into [CHANGELOG.md](CHANGELOG.md). +* **Don't bump the version.** Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) + and are established by reading git tags, so no code or manifest changes are required in a PR. +* **Update `uv.lock` if you touched dependencies.** Use `uv add` or `uv lock`, and verify with + `uv lock --check`. - nox --non-interactive -s doc +Releases are cut by maintainers; see [README.release.md](README.release.md). diff --git a/changelog.d/599.doc.md b/changelog.d/599.doc.md new file mode 100644 index 000000000..deb43320b --- /dev/null +++ b/changelog.d/599.doc.md @@ -0,0 +1 @@ +Consolidate contributor guidance into the published Contributors Guide, and reduce `CONTRIBUTING.md` to a short entry point that links to it, so the two copies can no longer drift apart. diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst index ba7a1d100..dba63a811 100644 --- a/doc/source/contributing.rst +++ b/doc/source/contributing.rst @@ -25,6 +25,45 @@ We encourage outside contributors to perform changes on our codebase. Many such * maintain other Continuous Integration tools (coverage tracker) +Versioning +############# + +This package's versions adhere to `Semantic Versioning `_, and the +versions are established by reading git tags, i.e. no code or manifest file changes are required when +working on PRs. + +Changelog +############# + +Each PR needs to have at least one changelog (aka news) item added. This is done by creating files in +``changelog.d``. `towncrier `_ is used for compiling these files into +``CHANGELOG.md``. There are several types of changes (news): + +#. ``fixed`` +#. ``changed`` +#. ``added`` +#. ``deprecated`` +#. ``removed`` +#. ``infrastructure`` +#. ``doc`` + +The ``changelog.d`` file name convention is: + +#. If the PR closes a GitHub issue: ``{issue_number}.{type}.md``, e.g. ``157.fixed.md``. Note that the + change description still has to be complete; linking an issue is just there for convenience. A change + like ``fixed #157`` will not be accepted. +#. If the PR is not related to a GitHub issue: ``+{unique_string}.{type}.md``, e.g. ``+foobar.fixed.md``. + +These files can either be created manually, or using ``towncrier``, e.g.:: + + $ towncrier create -c 'write your description here' 157.fixed.md + +``towncrier create`` also takes care of duplicates automatically (if there is more than one news fragment +of one type for a given GitHub issue). + +Developer info +############## + You'll need to have `nox `_ and `uv `_ installed: * ``pip install nox uv`` @@ -72,6 +111,17 @@ With the above setting, session ``test`` will run on Python 3.12 and 3.14, and a Given Python interpreters should be installed in the operating system or via `pyenv `_. +Managing dependencies +##################### + +We use `uv `_ for managing dependencies and developing locally. If you want +to change any of the project requirements (or requirement bounds) in ``pyproject.toml``, make sure that +the ``uv.lock`` file reflects those changes by using ``uv add``, ``uv lock`` or other commands - see the +`uv documentation `_. You can verify that the lock file is up to date by +running:: + + $ uv lock --check + Linting ############# @@ -101,6 +151,10 @@ To run just integration tests:: $ export B2_TEST_APPLICATION_KEY_ID=your_app_key_id $ nox -s integration-3.10 +To run tests matching a keyword expression:: + + $ nox -s unit-3.10 -- -k keyword + Documentation ############# From b4b4ca44d74bba16c908a71f4bf6f9bc2510a6ae Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:41:00 -0600 Subject: [PATCH 07/12] Document where a new test belongs The repository keeps four numbered interfaces working at once, and the unit suite is what enforces that, but nothing explained how the apiver machinery selects tests. A test could therefore be placed where it covers only one interface even though the changed _internal behaviour is shared. Add a decision table for the shared, per-version, v_all and integration locations, document the apiver marker, and add a checklist for _internal changes. Closes #592 Co-Authored-By: Claude Opus 5 --- changelog.d/592.doc.md | 1 + doc/source/contributing.rst | 65 +++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) create mode 100644 changelog.d/592.doc.md diff --git a/changelog.d/592.doc.md b/changelog.d/592.doc.md new file mode 100644 index 000000000..0e2aba5d5 --- /dev/null +++ b/changelog.d/592.doc.md @@ -0,0 +1 @@ +Document how to choose a location for a new test in the Contributors Guide, covering the shared unit directories, the per-interface `v0`-`v3` directories, `v_all`, integration tests, the `apiver` marker, and a checklist for changes to `b2sdk/_internal`. diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst index dba63a811..746762590 100644 --- a/doc/source/contributing.rst +++ b/doc/source/contributing.rst @@ -155,6 +155,71 @@ To run tests matching a keyword expression:: $ nox -s unit-3.10 -- -k keyword +Where to put a new test +======================= + +**b2sdk** keeps several numbered interfaces (``b2sdk.v0`` through ``b2sdk.v3``) working at the same +time, and the unit suite is what holds that promise. ``nox -s unit`` therefore runs ``pytest`` once per +interface version - ``--api=v3``, then ``v2``, ``v1`` and ``v0`` - so a single shared test is executed +four times, once against each interface. + +Two mechanisms make that work: + +* ``pytest`` gains an ``--api`` option, defaulting to the newest version. It puts + ``test/unit//apiver`` on ``sys.path``, so a shared test can write + ``from apiver_deps import B2Api`` and receive whichever interface is under test. The ``apiver`` and + ``apiver_int`` fixtures expose that version as ``"v2"`` or ``2`` respectively. +* Tests living under a *different* version's directory are not collected. Running with ``--api=v3`` + skips ``test/unit/v0``, ``v1`` and ``v2`` entirely. + +Choose a location accordingly: + +.. list-table:: + :header-rows: 1 + :widths: 28 22 50 + + * - Location + - Collected for + - Use it for + * - ``test/unit//``, e.g. ``bucket``, ``api``, ``sync`` + - every interface version + - The default home. Behaviour shared by all interfaces - in practice, anything implemented in + ``b2sdk/_internal``. Import through ``apiver_deps``. + * - ``test/unit/v0`` … ``test/unit/v3`` + - only its own version + - Behaviour that exists in exactly one interface, such as a legacy constructor argument or an + older return type preserved by a compatibility adapter. + * - ``test/unit/v_all/`` + - every interface version + - Assertions about the interfaces themselves - that a name is still exported, or that two + versions agree. + * - ``test/integration/`` + - not run by ``nox -s unit`` + - End-to-end behaviour against real B2. Requires credentials, so it cannot gate every PR. + +If a shared test applies to only some versions, keep it in the shared directory and mark it rather than +copying it into each version directory:: + + @pytest.mark.apiver(1) # only v1 + @pytest.mark.apiver(1, 3) # only v1 and v3 + @pytest.mark.apiver(from_ver=2) # v2 and newer + @pytest.mark.apiver(to_ver=2) # v2 and older + +``from_ver`` and ``to_ver`` are inclusive, and a single mark may not mix positional and keyword +arguments. Marks also work on individual ``pytest.param`` entries, which is the usual way to assert that +an older interface raises a different exception than a newer one. When a module-level ``pytestmark`` and +a test-level mark disagree, the test is skipped unless *both* conditions allow it. See +``test/unit/conftest.py`` for the full behaviour. + +When you change something in ``b2sdk/_internal``, check that: + +#. the regression test sits in a shared directory, so it runs for every interface, and not in a single + version directory where it would silently cover only one; +#. it imports from ``apiver_deps`` rather than a fixed ``b2sdk.vN``; +#. any interface that deliberately keeps the old behaviour has a test asserting that, placed in that + version's directory or marked with ``apiver``; +#. ``nox -s unit`` passes - a suite that passes only for the newest interface is not enough. + Documentation ############# From 522ae636861ac240e5a57f8418a947f925afc2c4 Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:42:01 -0600 Subject: [PATCH 08/12] Expand the release guide into a maintainer runbook The guide was a single instruction to run make_release_commit and follow its output, leaving the boundaries undocumented: that the session builds the changelog but neither commits nor tags, that CI must be green before tagging because a PyPI version cannot be reused, and that CD creates a GitHub release but publishes to PyPI only when the token is present and the tag is not a prerelease. Document those, plus recovery before and after a tag is pushed. Closes #593 Co-Authored-By: Claude Opus 5 --- README.release.md | 107 ++++++++++++++++++++++++++++++++++++++++- changelog.d/593.doc.md | 1 + 2 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 changelog.d/593.doc.md diff --git a/README.release.md b/README.release.md index 9ebbd8303..81a344ee2 100644 --- a/README.release.md +++ b/README.release.md @@ -1,3 +1,108 @@ # Release Process -- Run `nox -s make_release_commit -- X.Y.Z` where `X.Y.Z` is the version you're releasing, and follow the instructions +Releasing **b2sdk** is part automated and part manual. A `nox` session prepares the changelog, a +maintainer commits and tags by hand, and the *Continuous Delivery* workflow +([`.github/workflows/cd.yml`](.github/workflows/cd.yml)) does everything from the tag onwards. + +Nothing in the release is triggered by merging to `master`. Publishing begins only when a `v*` tag is +pushed. + +## Prerequisites + +* **A clean worktree.** `make_release_commit` aborts if `git diff --stat` reports anything. Note that + it inspects tracked files only, so untracked files will not stop it. +* **Release from `master`.** The session only *warns* when you are on another branch, it does not stop + you. Releasing from a feature branch is almost never what you want. +* **A version of the form `X.Y.Z`**, with integers and no leading `v` and no suffix. Anything else is + rejected. +* **An `upstream` remote** pointing at the release repository. `noxfile.py` defines the expected URL as + `UPSTREAM_REPO_URL`; the session prints a command to add it if it is missing. +* **Changelog fragments** for everything going out. Every PR should already have added one to + `changelog.d`; review that directory before starting, because whatever is there becomes the release + notes. + +## Steps + +### 1. Build the changelog + +``` +nox -s make_release_commit -- X.Y.Z +``` + +Despite the session's name, **this does not commit and does not tag.** It runs `towncrier build`, which +folds the `changelog.d` fragments into `CHANGELOG.md` and deletes the fragment files, then prints the +commands for the remaining steps. Review the resulting `CHANGELOG.md` diff before going further — this +is the last convenient moment to fix wording. + +### 2. Commit and push + +``` +git commit -m "release X.Y.Z" +git push upstream master +``` + +### 3. Wait for CI + +Let the *Continuous Integration* workflow finish on the pushed commit. Tagging a commit whose CI is red +means publishing a broken release, and a version number cannot be reused on PyPI. + +### 4. Tag and push the tag + +``` +git tag vX.Y.Z +git push upstream vX.Y.Z +``` + +Pushing the tag is the point of no return. + +## What the CD workflow does + +Triggered by any pushed tag matching `v*`: + +1. Decides whether the release is a prerelease, based only on whether the tag's last character is a + digit. `v1.2.3` is a normal release. +2. Builds the distribution with `nox -s build`, which also asserts the built package imports from + `site-packages` rather than the checkout. The version is derived from the tag, not from any file. +3. Reads the notes for that version out of `CHANGELOG.md`. If the tag and the changelog heading do not + match, there will be no release body. +4. Creates a GitHub release and attaches the distribution. +5. Uploads to PyPI — **only** when the `B2_PYPI_PASSWORD` secret is present *and* the release is not a + prerelease. A repository without that secret still gets a GitHub release and silently publishes + nothing to PyPI, so do not treat a green CD run as proof that PyPI was updated. + +## After the release + +* Confirm the GitHub release exists, has release notes, and has the distribution attached. +* If a PyPI publish was expected, confirm the new version is actually on PyPI. +* Confirm `pip install b2sdk==X.Y.Z` resolves. + +## If something goes wrong + +### Before the tag is pushed + +Everything is still local and reversible. Remember that `towncrier build` **deleted** the changelog +fragments, so recovering means restoring them: + +* Fragments deleted but not yet committed: `git checkout -- changelog.d CHANGELOG.md`. +* Already committed: `git reset --hard HEAD~1` (only if the commit has not been pushed), or revert it. + +Then fix the problem and start again from step 1. + +### After the tag is pushed, before PyPI published + +Delete the tag locally and upstream, fix the problem, and re-tag: + +``` +git push upstream :refs/tags/vX.Y.Z +git tag -d vX.Y.Z +``` + +Delete the GitHub release too, if one was created. Re-using the version number is safe only while +nothing has reached PyPI. + +### After PyPI published + +The version is spent. PyPI does not allow re-uploading a version, even after deleting it. Do not try to +reuse the number — fix the problem and release the next patch version instead. If the published release +is actively harmful, yank it on PyPI (which hides it from new resolutions without breaking pins that +already reference it) and follow up with a fixed release. diff --git a/changelog.d/593.doc.md b/changelog.d/593.doc.md new file mode 100644 index 000000000..26594d812 --- /dev/null +++ b/changelog.d/593.doc.md @@ -0,0 +1 @@ +Expand the release guide into a maintainer runbook covering prerequisites, the changelog build, the commit/CI/tag sequence, what the Continuous Delivery workflow does, post-release verification, and recovery before and after a tag is pushed. From d89760a6b95e8ca901d859efb621b3267944755f Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 15:43:46 -0600 Subject: [PATCH 09/12] Make the public/internal namespace boundary explicit The interface policy described which modules are public but said nothing about the versioned namespace itself, which re-exports far more than the public interface - low-level HTTP classes, simulators, constants, and names leaked by star-imports. State that importability is not a promise, and give the rule for telling public, internal and unclassified names apart. The per-name classification that #597 also asks for still needs maintainer review, so this does not close the issue. Refs #597 Co-Authored-By: Claude Opus 5 --- changelog.d/+interface-boundary.doc.md | 1 + doc/source/api_types.rst | 34 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 changelog.d/+interface-boundary.doc.md diff --git a/changelog.d/+interface-boundary.doc.md b/changelog.d/+interface-boundary.doc.md new file mode 100644 index 000000000..391df3310 --- /dev/null +++ b/changelog.d/+interface-boundary.doc.md @@ -0,0 +1 @@ +State explicitly that being importable from a versioned namespace such as `b2sdk.v3` is not a compatibility promise, and document how to tell whether a name is public, internal, or unclassified. diff --git a/doc/source/api_types.rst b/doc/source/api_types.rst index 3995def72..86fc279c6 100644 --- a/doc/source/api_types.rst +++ b/doc/source/api_types.rst @@ -74,6 +74,40 @@ Those modules will generally not change in a backwards-incompatible way between .. note:: ``b2sdk.*._something`` and ``b2sdk.*.*._something``, while having a name beginning with an underscore, are **NOT** considered public interface. +Importable is not the same as public +==================================== + +The versioned namespace re-exports a considerably larger surface than the public interface. Alongside +the classes documented here, ``b2sdk.v3`` carries low-level HTTP machinery, test simulators, +transfer-manager internals, module-level constants, and a few names that are merely an artefact of +star-imports - ``annotations``, leaked from ``from __future__ import annotations``, is one. Most of the +names in the namespace are not mentioned anywhere in this documentation. + +**Being importable from** ``b2sdk.v3`` **is therefore not a compatibility promise.** To find out where a +name stands, look it up in the :doc:`API Reference `: + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - Where the name appears + - What is promised + * - :ref:`Public API ` + - Public interface. Pin the major version. + * - :ref:`Internal API ` + - Documented, but may change in a non-major release. Pin the middle version. + * - Neither + - No compatibility promise. It may be renamed or removed in any release. + +A name in the third group has not necessarily been judged internal - classifying the namespace is still +in progress. Treat the absence of documentation as the absence of a promise, rather than as a statement +about what the name is meant to be. If you depend on such a name, please file an issue so it can be +considered for the public interface instead of being relied on as an incidental re-export. + +The current contents of the namespace can be listed with:: + + python -c "import b2sdk.v3; print(sorted(n for n in vars(b2sdk.v3) if not n.startswith('_')))" + .. _internal_interface: ****************** From c1211e45c31d9787552310f52a0c0ddf067441eb Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 16:10:22 -0600 Subject: [PATCH 10/12] Add a replication guide and API reference Replication was fully exposed by the SDK but undocumented, discoverable only by guessing at names. Add a task-oriented guide covering setup via ReplicationSetupHelper and by hand, the application-key capabilities each side needs, per-file replication status, monitoring a rule, and how to read and interpret a scan report. Add reference pages for the public replication classes. Documenting ReplicationMonitor also required indenting the continuation lines of its docstring's parameter list, which docutils otherwise rejects. No executable code is changed. ReplicationReport is described in prose rather than generated, because its inherited annotations cannot be resolved from the subclass's module. Fixing that properly belongs in b2sdk/_internal/scan/scan.py. Closes #595 Co-Authored-By: Claude Opus 5 --- b2sdk/_internal/replication/monitoring.py | 10 +- changelog.d/595.doc.md | 1 + doc/source/api/replication/monitoring.rst | 19 ++ doc/source/api/replication/setting.rst | 17 ++ doc/source/api/replication/setup.rst | 8 + doc/source/api/replication/types.rst | 8 + doc/source/api_reference.rst | 4 + doc/source/index.rst | 1 + doc/source/replication.rst | 236 ++++++++++++++++++++++ 9 files changed, 299 insertions(+), 5 deletions(-) create mode 100644 changelog.d/595.doc.md create mode 100644 doc/source/api/replication/monitoring.rst create mode 100644 doc/source/api/replication/setting.rst create mode 100644 doc/source/api/replication/setup.rst create mode 100644 doc/source/api/replication/types.rst create mode 100644 doc/source/replication.rst diff --git a/b2sdk/_internal/replication/monitoring.py b/b2sdk/_internal/replication/monitoring.py index bae3a2a83..c97c87274 100644 --- a/b2sdk/_internal/replication/monitoring.py +++ b/b2sdk/_internal/replication/monitoring.py @@ -114,14 +114,14 @@ class ReplicationMonitor: :param b2sdk.v2.Bucket bucket: replication source bucket :param b2sdk.v2.ReplicationRule rule: replication rule to be monitored; - should belong to `bucket`'s replication configuration + should belong to `bucket`'s replication configuration :param b2sdk.v2.B2Api destination_api: B2Api instance for destination - bucket; if destination bucket is on the same account as source bucket, - omit this parameter and then source bucket's B2Api will be used + bucket; if destination bucket is on the same account as source bucket, + omit this parameter and then source bucket's B2Api will be used :param b2sdk.v2.ProgressReport report: instance of ProgressReport which will report - scanning progress, by default to stdout + scanning progress, by default to stdout :param b2sdk.v2.ScanPoliciesManager scan_policies_manager: a strategy to scan - files, so that several files that match some criteria may be omitted + files, so that several files that match some criteria may be omitted :rtype: b2sdk.v2.ReplicationMonitor """ diff --git a/changelog.d/595.doc.md b/changelog.d/595.doc.md new file mode 100644 index 000000000..f59ac6131 --- /dev/null +++ b/changelog.d/595.doc.md @@ -0,0 +1 @@ +Add a task-oriented replication guide covering setup, required application-key capabilities, per-file replication status, monitoring and how to read a scan report, plus API reference pages for the public replication classes. diff --git a/doc/source/api/replication/monitoring.rst b/doc/source/api/replication/monitoring.rst new file mode 100644 index 000000000..a3b74207a --- /dev/null +++ b/doc/source/api/replication/monitoring.rst @@ -0,0 +1,19 @@ +.. _replication_monitoring: + +Replication monitoring +====================== + +.. autoclass:: b2sdk.v3.ReplicationMonitor() + :members: + :exclude-members: source_folder, destination_folder + :special-members: __init__ + +.. autoclass:: b2sdk.v3.ReplicationScanResult() + :inherited-members: + :members: + +``ReplicationReport`` is a :class:`b2sdk.v3.CountAndSampleScanReport` whose results are +:class:`b2sdk.v3.ReplicationScanResult` instances. It groups scanned files by result rather than +listing them: ``counter_by_status`` counts the files sharing each result, while +``samples_by_status_first`` and ``samples_by_status_last`` hold the first and last example seen for +each. See :ref:`replication` for how to read one. diff --git a/doc/source/api/replication/setting.rst b/doc/source/api/replication/setting.rst new file mode 100644 index 000000000..ff6992d03 --- /dev/null +++ b/doc/source/api/replication/setting.rst @@ -0,0 +1,17 @@ +.. _replication_setting: + +Replication settings +==================== + +.. autoclass:: b2sdk.v3.ReplicationConfiguration() + :inherited-members: + :members: + :special-members: __init__ + +.. autoclass:: b2sdk.v3.ReplicationRule() + :inherited-members: + :members: + :special-members: __init__ + +.. autoclass:: b2sdk.v3.ReplicationConfigurationFactory() + :no-members: diff --git a/doc/source/api/replication/setup.rst b/doc/source/api/replication/setup.rst new file mode 100644 index 000000000..786f929c0 --- /dev/null +++ b/doc/source/api/replication/setup.rst @@ -0,0 +1,8 @@ +.. _replication_setup: + +Replication setup helper +======================== + +.. autoclass:: b2sdk.v3.ReplicationSetupHelper() + :members: + :special-members: __init__ diff --git a/doc/source/api/replication/types.rst b/doc/source/api/replication/types.rst new file mode 100644 index 000000000..bb28b4a23 --- /dev/null +++ b/doc/source/api/replication/types.rst @@ -0,0 +1,8 @@ +.. _replication_types: + +Replication types +================= + +.. autoclass:: b2sdk.v3.ReplicationStatus() + :inherited-members: + :members: diff --git a/doc/source/api_reference.rst b/doc/source/api_reference.rst index e449307a4..3825eb33f 100644 --- a/doc/source/api_reference.rst +++ b/doc/source/api_reference.rst @@ -37,6 +37,10 @@ Public API api/transfer/outbound/outbound_source api/encryption/setting api/encryption/types + api/replication/setting + api/replication/setup + api/replication/monitoring + api/replication/types .. _api_internal: diff --git a/doc/source/index.rst b/doc/source/index.rst index ea1e4ddda..8c66b53a4 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -48,6 +48,7 @@ Documentation index tutorial quick_start server_side_encryption + replication advanced glossary api_types diff --git a/doc/source/replication.rst b/doc/source/replication.rst new file mode 100644 index 000000000..ebfd1fd16 --- /dev/null +++ b/doc/source/replication.rst @@ -0,0 +1,236 @@ +.. _replication: + +######################## +Replication +######################## + +Replication asks B2 to copy files from one bucket to another automatically, as they are uploaded. The +bucket being copied *from* is the **source**; the bucket being copied *to* is the **destination**. One +bucket may be both at once, and a source bucket may replicate to several destinations under different +rules. + +Replication is configured on the buckets, not on individual files: you attach a +:class:`b2sdk.v3.ReplicationConfiguration` to each side, and B2 does the copying. The SDK's job is to +help you write that configuration, create the application keys it needs, and inspect the outcome. + +.. note:: + Replication only applies to files uploaded *after* a rule starts matching them. To bring across + files that already exist, set ``include_existing_files=True`` on the rule. + +********************** +What replication needs +********************** + +Both sides need an application key, and the two sides need different capabilities: + +.. list-table:: + :header-rows: 1 + :widths: 20 80 + + * - Side + - Capabilities + * - Source + - ``readFiles``, ``readFileLegalHolds``, ``readFileRetentions`` + * - Destination + - ``writeFiles``, ``writeFileLegalHolds``, ``writeFileRetentions``, ``deleteFiles`` + +The source bucket's configuration records the id of the source key. The destination bucket's +configuration maps each incoming source key id to the destination key that should be used for it - that +mapping is how a destination bucket authorises a particular source to write into it. + +If the two buckets live in different accounts, you will need a separate :class:`b2sdk.v3.B2Api` +instance for each. + +********************** +Setting up replication +********************** + +Using the setup helper +====================== + +:class:`b2sdk.v3.ReplicationSetupHelper` is the supported path. It creates both application keys, adds +the rule to the source, and registers the key mapping on the destination: + +.. code-block:: python + + >>> from b2sdk.v3 import ReplicationSetupHelper + + >>> source_bucket = b2_api.get_bucket_by_name('source-bucket') + >>> destination_bucket = b2_api.get_bucket_by_name('destination-bucket') + + >>> rsh = ReplicationSetupHelper() + >>> source_bucket, destination_bucket = rsh.setup_both( + source_bucket=source_bucket, + destination_bucket=destination_bucket, + name='my-rule', + prefix='folder/', + ) + +It returns the two updated buckets, so use the returned objects rather than the ones you passed in. + +The keys it creates are named after the buckets, suffixed ``-replisrc`` and ``-replidst``. If a rule is +already present, the helper places the new one ahead of the existing rules rather than overwriting +them. + +The two halves are also available separately, which is what you need when the buckets are on different +accounts - run :meth:`~b2sdk.v3.ReplicationSetupHelper.setup_destination` against the destination +account first, then :meth:`~b2sdk.v3.ReplicationSetupHelper.setup_source` against the source account +with the resulting key. + +Writing the configuration by hand +================================= + +When you need full control, build the configuration yourself and pass it to +:meth:`b2sdk.v3.Bucket.update` (or to :meth:`b2sdk.v3.B2Api.create_bucket` for a new bucket). You are +then responsible for creating the application keys with the capabilities listed above. + +.. code-block:: python + + >>> from b2sdk.v3 import ReplicationConfiguration, ReplicationRule + + >>> source_bucket.update( + replication=ReplicationConfiguration( + rules=[ + ReplicationRule( + destination_bucket_id=destination_bucket.id_, + name='my-rule', + file_name_prefix='folder/', + ), + ], + source_key_id=source_key.id_, + ), + ) + + >>> destination_bucket.update( + replication=ReplicationConfiguration( + source_to_destination_key_mapping={ + source_key.id_: destination_key.id_, + }, + ), + ) + +A few constraints are enforced when the objects are constructed, so you will hear about mistakes before +a request is sent: + +* a rule ``name`` must match ``[a-zA-Z0-9_-]`` and be at most 64 characters; +* ``destination_bucket_id`` is required; +* ``priority`` must be between 1 and 2\ :sup:`31`\ -1, and defaults to 128 - lower numbers are + evaluated first, and the first matching rule wins; +* a configuration that has ``rules`` must also have a ``source_key_id``. + +Inspecting what is configured +============================= + +``bucket.replication`` is either ``None`` or a :class:`b2sdk.v3.ReplicationConfiguration`. Because one +bucket can be both ends of a replication relationship, ask which role it plays: + +.. code-block:: python + + >>> bucket.replication.is_source + True + >>> bucket.replication.is_destination + False + >>> [rule.name for rule in bucket.replication.rules] + ['my-rule'] + +******************** +Checking on progress +******************** + +Per-file status +=============== + +A file version carries a :class:`b2sdk.v3.ReplicationStatus`, which is one of: + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - Status + - Meaning + * - ``PENDING`` + - On the source, and not yet copied to every destination. + * - ``COMPLETED`` + - On the source, and copied successfully. + * - ``FAILED`` + - On the source, and B2 will not retry. Needs investigation. + * - ``REPLICA`` + - This file *is* a copy - the status you see on the destination side. + +A source file whose status stays ``PENDING`` is usually still in flight. ``FAILED`` is terminal, and the +common causes are a key that no longer has the required capabilities, a destination key mapping that +does not match the source key in use, or a file the destination refuses because of its own lock +settings. + +Monitoring a rule +================= + +:class:`b2sdk.v3.ReplicationMonitor` walks one rule and reports what it finds on both sides: + +.. code-block:: python + + >>> from b2sdk.v3 import ReplicationMonitor + + >>> rule = source_bucket.replication.rules[0] + >>> monitor = ReplicationMonitor(bucket=source_bucket, rule=rule) + >>> report = monitor.scan() + +The monitor scans the source and, by default, the matching files on the destination so it can compare +them. Pass ``scan_destination=False`` to look at the source alone, which is much cheaper. + +Two optional arguments matter in practice. ``destination_api`` takes a second +:class:`b2sdk.v3.B2Api`, and is required when the destination bucket is on another account; omit it and +the source bucket's own API is used. ``scan_policies_manager`` takes a +:class:`b2sdk.v3.ScanPoliciesManager`, letting you skip files you do not care about. + +The monitor validates its arguments on construction: it raises ``ValueError`` if the bucket has no +replication configuration at all, or if the rule you passed does not belong to that bucket's +configuration. + +Reading the report +================== + +:meth:`~b2sdk.v3.ReplicationMonitor.scan` returns a :class:`b2sdk.v3.ReplicationReport`. Rather than a +flat list of files, it groups them: ``counter_by_status`` counts how many files share each distinct +:class:`b2sdk.v3.ReplicationScanResult`, and ``samples_by_status_first`` and ``samples_by_status_last`` +hold the first and last example seen for each, so you have something concrete to go and look at. + +.. code-block:: python + + >>> for result, count in report.counter_by_status.most_common(): + print(count, result.source_replication_status, result.destination_replication_status) + +A scan result describes a source/destination *pair*, and the fields fall into three groups: + +* what the source file looks like - ``source_replication_status``, ``source_has_hide_marker``, + ``source_encryption_mode``, ``source_has_large_metadata``, ``source_has_file_retention``, + ``source_has_legal_hold``; +* what the destination file looks like - ``destination_replication_status``; +* how the two compare - ``metadata_differs``, ``hash_differs``. + +Any of these may be ``None``, which means "not known" rather than "false" - typically because only one +side of the pair exists, or because the destination was not scanned. + +Interpreting partial results +============================ + +A scan is a snapshot of a system that is still moving, so mixed results are normal rather than a sign of +failure: + +* **A destination file is missing entirely.** The destination fields are ``None``. Expected while a file + is still ``PENDING``; worth investigating once the source reads ``COMPLETED``. +* **``hash_differs`` is true.** The two sides hold different content. This is briefly normal if the + source file was replaced while the scan was running, and a real problem otherwise. +* **``metadata_differs`` is true.** The content matches but the file info does not. Large metadata is the + usual cause - see ``source_has_large_metadata``, since B2 will not replicate metadata beyond a size + limit. +* **``source_has_legal_hold`` or ``source_has_file_retention`` is set.** Copying these requires the + ``*FileLegalHolds`` and ``*FileRetentions`` capabilities on *both* keys. Keys created by hand without + them are a common cause of ``FAILED``. + +.. note:: + Only the latest version of each file is inspected. Earlier versions are not represented in a report, + so a clean scan is not a statement about a bucket's whole version history. + +For the full signatures of everything above, see :ref:`replication_setting`, :ref:`replication_setup`, +:ref:`replication_monitoring` and :ref:`replication_types`. From 7a70ae1f326ddde6f46e4fae0de2e95318c4145b Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 16:15:08 -0600 Subject: [PATCH 11/12] Resolve the embedded documentation TODOs Five of the six todo directives are addressed: the "delete doc/source/b2sdk?" note is obsolete since that directory no longer exists, raw_simulator already has an autodoc page, the "Why use b2sdk?" list is now consistently phrased and links to the features it mentions, and the progress reporters and exception hierarchy have real documentation rather than a note saying they need some. The remaining directive, in advanced.rst, asks for `prioritize` to accept an enum instead of a string. That is an API design change rather than documentation, so it is left in place until an issue exists to link it to. Refs #601 Co-Authored-By: Claude Opus 5 --- changelog.d/601.doc.md | 1 + doc/source/api/exception.rst | 51 ++++++++++++++++++++++++++-- doc/source/api/progress.rst | 64 +++++++++++++++++++++++++++++++++--- doc/source/index.rst | 32 +++++++----------- 4 files changed, 121 insertions(+), 27 deletions(-) create mode 100644 changelog.d/601.doc.md diff --git a/changelog.d/601.doc.md b/changelog.d/601.doc.md new file mode 100644 index 000000000..cf88f11b4 --- /dev/null +++ b/changelog.d/601.doc.md @@ -0,0 +1 @@ +Resolve the embedded documentation TODOs: drop the obsolete ones, make the "Why use b2sdk?" list consistent and linked, and document the progress reporters and the exception hierarchy properly. diff --git a/doc/source/api/exception.rst b/doc/source/api/exception.rst index 4a0396d93..30e1bcca9 100644 --- a/doc/source/api/exception.rst +++ b/doc/source/api/exception.rst @@ -1,8 +1,55 @@ Exceptions ========== -.. todo:: - improve documentation of exceptions, automodule -> autoclass? +Every error raised by **b2sdk** derives from :class:`b2sdk.v3.exception.B2Error`, so catching that one +class is enough to catch anything the library raises deliberately. + +How to catch them +----------------- + +The hierarchy is arranged so you can be as specific as you need. Catch a named class when you intend to +handle that exact condition, and fall back to a broader one otherwise: + +.. code-block:: python + + >>> from b2sdk.v3.exception import B2Error, NonExistentBucket + + >>> try: + bucket = b2_api.get_bucket_by_name('no-such-bucket') + except NonExistentBucket: + ... # handle this specific case + except B2Error as e: + print(e) # anything else the SDK raises + +.. warning:: + The hierarchy may gain intermediate classes in a backwards-compatible release, so an exception's + direct parent is not guaranteed to stay the same. Use ``isinstance`` and ``super()`` rather than + naming a parent class explicitly. See :ref:`interface version compatibility + `. + +Deciding whether to retry +------------------------- + +Errors carry their own retry advice, so you rarely need to inspect status codes yourself: + +* ``should_retry_http()`` is true when the failed HTTP call is worth repeating. +* ``should_retry_upload()`` is true when the upload should be retried after obtaining a fresh upload URL. +* ``retry_after_seconds`` is set when the server asked the client to wait before issuing more requests. + When it is present, honour it rather than retrying immediately. + +**b2sdk** already applies this logic internally for its own transfers, so these are relevant mainly if +you are driving the lower-level API yourself. + +Two conveniences +---------------- + +Every error has a ``prefix`` property giving a readable name derived from the class name, which is what +makes the default messages presentable to end users. And +:func:`b2sdk.v3.exception.interpret_b2_error` turns a raw B2 error response into the appropriate +exception class, which is the function responsible for the hierarchy below being raised at all. + +Reference +--------- .. automodule:: b2sdk.v3.exception :members: diff --git a/doc/source/api/progress.rst b/doc/source/api/progress.rst index 95f3c4434..6a1daf800 100644 --- a/doc/source/api/progress.rst +++ b/doc/source/api/progress.rst @@ -1,13 +1,67 @@ Progress reporters ================== -.. note:: - Concrete classes described in this chapter implement methods defined in ``AbstractProgressListener`` +**b2sdk** reports progress through two separate mechanisms, and which one you want depends on what you +are doing. + +For **a single transfer** - one upload, download or copy - pass a *progress listener* to the method. +Every such method accepts a ``progress_listener`` argument, and the classes on this page are the +available implementations. + +For **a whole sync operation**, which scans folders and then transfers many files, a single listener is +not enough. Sync instead takes a :class:`b2sdk.v3.SyncReport`, which tracks the scan and comparison +phases as well as the transfers. See :ref:`sync` for how to use it. + +Choosing a listener +------------------- + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - Class + - When to use it + * - :class:`b2sdk.v3.TqdmProgressListener` + - Interactive use. Draws a progress bar, and needs the ``tqdm`` package installed. + * - :class:`b2sdk.v3.SimpleProgressListener` + - Interactive use without ``tqdm``. Prints plain percentages. + * - :class:`b2sdk.v3.DoNothingProgressListener` + - Non-interactive use - servers, scripts, tests. Discards everything. + * - :class:`b2sdk.v3.ProgressListenerForTest` + - Test support. Records the calls it receives so they can be asserted on. + +If you do not want to choose, :func:`b2sdk.v3.make_progress_listener` picks for you: it returns +``DoNothingProgressListener`` when ``quiet`` is set, ``TqdmProgressListener`` when ``tqdm`` is +importable, and ``SimpleProgressListener`` otherwise. -.. todo:: - improve documentation of progress reporters +.. code-block:: python - include info about sync progress + >>> from b2sdk.v3 import make_progress_listener + + >>> with make_progress_listener('uploading backup.tar', quiet=False) as listener: + bucket.upload_local_file( + local_file='backup.tar', + file_name='backup.tar', + progress_listener=listener, + ) + +Writing your own +---------------- + +Subclass :class:`b2sdk.v3.AbstractProgressListener` and implement ``set_total_bytes`` and +``bytes_completed``. Two details of the contract are easy to get wrong: + +* ``set_total_bytes`` is always called before the listener is entered, but **may be called again** if a + transfer is retried. +* ``bytes_completed`` receives a **running total, not a delta**, and that total **can go down**, because + a failed transfer restarts from the beginning. Do not accumulate the values you are given. + +Listeners are context managers, and ``close`` must be called exactly once - it asserts if called twice - +so prefer a ``with`` block over calling it yourself. + +.. note:: + The concrete classes on this page all implement the methods defined by + :class:`b2sdk.v3.AbstractProgressListener`. .. autoclass:: b2sdk.v3.AbstractProgressListener :inherited-members: diff --git a/doc/source/index.rst b/doc/source/index.rst index 8c66b53a4..772ad9ada 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -16,26 +16,18 @@ to provide command-line access to the B2 service, but there are many possible ap Why use b2sdk? ######################################### -.. todo:: - delete doc/source/b2sdk? - -.. todo:: - describe raw_simulator in detail - -.. todo:: - fix list consistency style in "Why use b2sdk?", add links - -When building an application which uses B2 cloud, it is possible to implement an independent B2 API client, but using **b2sdk** allows for: - -- reuse of code that is already written, with hundreds of unit tests -- use of **Synchronizer**, a high-performance, parallel rsync-like utility -- developer-friendly library :ref:`api version policy ` which guards your program against incompatible changes -- `B2 integration checklist `_ is passed automatically -- **raw_simulator** makes it easy to mock the B2 cloud for unit testing purposes -- reporting progress of operations to an object of your choice -- exception hierarchy makes it easy to display informative messages to users -- interrupted transfers are automatically continued -- **b2sdk** had been in development for years before its 1.0.0 release. It's stable and mature. +When building an application which uses B2 cloud, it is possible to implement an independent B2 API client, but using **b2sdk** gives you: + +- code that is already written and covered by hundreds of unit tests +- :ref:`Synchronizer `, a high-performance, parallel rsync-like utility +- :ref:`Replication ` support, for keeping a second bucket up to date automatically +- a developer-friendly :ref:`api version policy ` that guards your program against incompatible changes +- automatic compliance with the `B2 integration checklist `_ +- :doc:`raw_simulator `, which mocks the B2 cloud so you can unit test without network access +- :doc:`progress reporting ` to an object of your choice +- an :doc:`exception hierarchy ` that makes it easy to show users informative messages +- automatic continuation of interrupted transfers +- a stable and mature codebase, in development for years before its 1.0.0 release ######################################### From eefd84814b4d02a1ec56848ade1e25d809ab0373 Mon Sep 17 00:00:00 2001 From: Felipe Fumero Date: Wed, 9 Sep 2026 16:19:47 -0600 Subject: [PATCH 12/12] Link the remaining documentation TODO to its tracking issue The last todo directive asked for `prioritize` to accept an enum instead of a string, which is an API design change rather than documentation work. It is now tracked as issue #602 and the directive links to it, retained in place so that anyone editing that section sees it next to the documented string values. It stays out of published builds, since the todo directive only renders when B2_SPHINX_TODO is set. Closes #601 Co-Authored-By: Claude Opus 5 --- doc/source/advanced.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/source/advanced.rst b/doc/source/advanced.rst index 40373d11d..6e0d283b5 100644 --- a/doc/source/advanced.rst +++ b/doc/source/advanced.rst @@ -301,8 +301,10 @@ In `local_verification` mode the remote range was artificially split into three .. note:: `prioritize` is just a planner setting - remote parts are always verified if matching local parts exists. -.. TODO:: - prioritization should accept enum, not string +.. todo:: + Prioritization should accept an enum, not a string. Tracked in + `issue #602 `_. Kept here so that anyone + editing this section sees it alongside the documented string values. .. _continuation: