diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c561287b..3c3ee9d3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,146 +1,39 @@ # Contributing to B2 Python SDK -We encourage outside contributors to perform changes on our codebase. Many such changes have been merged already. -In order to make it easier to contribute, core developers of this project: +We encourage outside contributors to perform changes on our codebase. Many such changes have been merged +already. -* provide guidance (through the issue reporting system) -* provide tool assisted code review (through the Pull Request system) -* maintain a set of unit tests -* maintain a set of integration tests (run with a production cloud) -* maintain development automation tools using [nox](https://github.com/theacodes/nox) that can easily: - * format the code using [ruff](https://github.com/astral-sh/ruff) - * runs linters to find subtle/potential issues with maintainability - * run the test suite on multiple Python versions using [pytest](https://github.com/pytest-dev/pytest) -* maintain Continuous Integration (by using GitHub Actions) that: - * runs all sorts of linters - * checks if the Python distribution can be built - * runs all tests on a matrix of 8 versions of Python (including pypy) and 3 operating systems - (Linux, Mac OS X and Windows) - * checks if the documentation can be built properly -* maintain other Continuous Integration tools (coverage tracker) +**The full contributors guide is published at +[b2-sdk-python.readthedocs.io/en/master/contributing.html](https://b2-sdk-python.readthedocs.io/en/master/contributing.html)** +(source: [`doc/source/contributing.rst`](doc/source/contributing.rst)). It is the canonical reference for +environment setup, the available `nox` sessions, dependency management, testing, and building the +documentation. This file is a short entry point only — please keep detailed guidance in the published +guide rather than duplicating it here. -## Versioning +## Getting started -This package's versions adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and the versions are -established by reading git tags, i.e. no code or manifest file changes are required when working on PRs. +You'll need [nox](https://github.com/theacodes/nox) and [uv](https://docs.astral.sh/uv/): -## Changelog + pip install nox uv -Each PR needs to have at least one changelog (aka news) item added. This is done by creating files in `changelog.d`. -`towncrier` is used for compiling these files into [CHANGELOG.md](CHANGELOG.md). There are several types of changes -(news): - -1. fixed -2. changed -3. added -4. deprecated -5. removed -6. infrastructure -7. doc - - -The `changelog.d` file name convention is: - -1. If the PR closes a github issue: `{issue_number}.{type}.md` e.g. `157.fixed.md`. Note that the - change description still has to be complete, linking an issue is just there for convenience, a change like - `fixed #157` will not be accepted. -2. If the PR is not related to a github issue: `+{unique_string}.{type}.md` e.g. `+foobar.fixed.md`. - -These files can either be created manually, or using `towncrier` e.g. - - towncrier create -c 'write your description here' 157.fixed.md - -`towncrier create` also takes care of duplicates automatically (if there is more than 1 news fragment of one type -for a given github issue). - -## Developer Info - -You'll need to have [nox](https://github.com/theacodes/nox) and [uv](https://docs.astral.sh/uv/) installed: - -* `pip install nox uv` - -With `nox`, you can run different sessions (default are `lint` and `test`): - -* `format` -> Format the code. -* `lint` -> Run linters. -* `test` (`test-3.10`, `test-3.11`, `test-3.12`, `test-3.13`, `test-3.14`, `test-pypy3.10`) -> Run test suite. -* `cover` -> Perform coverage analysis. -* `build` -> Build the distribution. -* `doc` -> Build the documentation. -* `doc_cover` -> Perform coverage analysis for the documentation. - -For example: - - $ nox -s format - nox > Running session format - nox > Creating virtual environment (virtualenv) using python3.14 in .nox/format - ... - - $ nox -s format - nox > Running session format - nox > Re-using existing virtual environment at .nox/format. - ... - - $ nox --no-venv -s format - nox > Running session format - ... - -Sessions `test` ,`unit`, and `integration` can run on many Python versions, 3.10-3.14 (+ pypy3.10) by default. - -Sessions other than `test` use the last CPython version from `NOX_PYTHONS`, 3.14 by default. - -You can change it: - - export NOX_PYTHONS=3.12,3.14 - -With the above setting, session `test` will run on Python 3.12 and 3.14, and all other sessions on Python 3.14. - -Given Python interpreters should be installed in the operating system or via [pyenv](https://github.com/pyenv/pyenv). - -## Managing dependencies - -We use [uv](https://docs.astral.sh/uv/) for managing dependencies and developing locally. -If you want to change any of the project requirements (or requirement bounds) in `pyproject.toml`, -make sure that `uv.lock` file reflects those changes by using `uv add`, `uv lock` or other -commands - see [documentation](https://docs.astral.sh/uv/). You can verify that the lock file -is up to date by running `uv lock --check`. - -## Linting - -To run all available linters: +Then, to run the linters and the test suite: nox -s lint - -## Testing - -To run all tests on every available Python version: - nox -s test -To run all tests on a specific version: - - nox -s test-3.10 - -To run just unit tests: - - nox -s unit-3.10 - -To run just integration tests: - - export B2_TEST_APPLICATION_KEY=your_app_key - export B2_TEST_APPLICATION_KEY_ID=your_app_key_id - nox -s integration-3.10 - -To run tests by keyword expressions: - - nox -s unit-3.10 -- -k keyword - -## Documentation - -To build the documentation and watch for changes (including the source code): +Integration tests need real B2 credentials; see the published guide for details. - nox -s doc +## Before you open a pull request -To just build the documentation: +* **Add a changelog entry.** Every PR needs at least one news fragment in `changelog.d`, or CI will + fail. Name it `{issue_number}.{type}.md` (e.g. `157.fixed.md`) when the PR closes an issue, or + `+{unique_string}.{type}.md` (e.g. `+foobar.fixed.md`) otherwise. Valid types are `fixed`, `changed`, + `added`, `deprecated`, `removed`, `infrastructure` and `doc`. The description must stand on its own — + a change like `fixed #157` will not be accepted. [towncrier](https://towncrier.readthedocs.io/) + compiles these into [CHANGELOG.md](CHANGELOG.md). +* **Don't bump the version.** Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) + and are established by reading git tags, so no code or manifest changes are required in a PR. +* **Update `uv.lock` if you touched dependencies.** Use `uv add` or `uv lock`, and verify with + `uv lock --check`. - nox --non-interactive -s doc +Releases are cut by maintainers; see [README.release.md](README.release.md). diff --git a/README.release.md b/README.release.md index 9ebbd830..81a344ee 100644 --- a/README.release.md +++ b/README.release.md @@ -1,3 +1,108 @@ # Release Process -- Run `nox -s make_release_commit -- X.Y.Z` where `X.Y.Z` is the version you're releasing, and follow the instructions +Releasing **b2sdk** is part automated and part manual. A `nox` session prepares the changelog, a +maintainer commits and tags by hand, and the *Continuous Delivery* workflow +([`.github/workflows/cd.yml`](.github/workflows/cd.yml)) does everything from the tag onwards. + +Nothing in the release is triggered by merging to `master`. Publishing begins only when a `v*` tag is +pushed. + +## Prerequisites + +* **A clean worktree.** `make_release_commit` aborts if `git diff --stat` reports anything. Note that + it inspects tracked files only, so untracked files will not stop it. +* **Release from `master`.** The session only *warns* when you are on another branch, it does not stop + you. Releasing from a feature branch is almost never what you want. +* **A version of the form `X.Y.Z`**, with integers and no leading `v` and no suffix. Anything else is + rejected. +* **An `upstream` remote** pointing at the release repository. `noxfile.py` defines the expected URL as + `UPSTREAM_REPO_URL`; the session prints a command to add it if it is missing. +* **Changelog fragments** for everything going out. Every PR should already have added one to + `changelog.d`; review that directory before starting, because whatever is there becomes the release + notes. + +## Steps + +### 1. Build the changelog + +``` +nox -s make_release_commit -- X.Y.Z +``` + +Despite the session's name, **this does not commit and does not tag.** It runs `towncrier build`, which +folds the `changelog.d` fragments into `CHANGELOG.md` and deletes the fragment files, then prints the +commands for the remaining steps. Review the resulting `CHANGELOG.md` diff before going further — this +is the last convenient moment to fix wording. + +### 2. Commit and push + +``` +git commit -m "release X.Y.Z" +git push upstream master +``` + +### 3. Wait for CI + +Let the *Continuous Integration* workflow finish on the pushed commit. Tagging a commit whose CI is red +means publishing a broken release, and a version number cannot be reused on PyPI. + +### 4. Tag and push the tag + +``` +git tag vX.Y.Z +git push upstream vX.Y.Z +``` + +Pushing the tag is the point of no return. + +## What the CD workflow does + +Triggered by any pushed tag matching `v*`: + +1. Decides whether the release is a prerelease, based only on whether the tag's last character is a + digit. `v1.2.3` is a normal release. +2. Builds the distribution with `nox -s build`, which also asserts the built package imports from + `site-packages` rather than the checkout. The version is derived from the tag, not from any file. +3. Reads the notes for that version out of `CHANGELOG.md`. If the tag and the changelog heading do not + match, there will be no release body. +4. Creates a GitHub release and attaches the distribution. +5. Uploads to PyPI — **only** when the `B2_PYPI_PASSWORD` secret is present *and* the release is not a + prerelease. A repository without that secret still gets a GitHub release and silently publishes + nothing to PyPI, so do not treat a green CD run as proof that PyPI was updated. + +## After the release + +* Confirm the GitHub release exists, has release notes, and has the distribution attached. +* If a PyPI publish was expected, confirm the new version is actually on PyPI. +* Confirm `pip install b2sdk==X.Y.Z` resolves. + +## If something goes wrong + +### Before the tag is pushed + +Everything is still local and reversible. Remember that `towncrier build` **deleted** the changelog +fragments, so recovering means restoring them: + +* Fragments deleted but not yet committed: `git checkout -- changelog.d CHANGELOG.md`. +* Already committed: `git reset --hard HEAD~1` (only if the commit has not been pushed), or revert it. + +Then fix the problem and start again from step 1. + +### After the tag is pushed, before PyPI published + +Delete the tag locally and upstream, fix the problem, and re-tag: + +``` +git push upstream :refs/tags/vX.Y.Z +git tag -d vX.Y.Z +``` + +Delete the GitHub release too, if one was created. Re-using the version number is safe only while +nothing has reached PyPI. + +### After PyPI published + +The version is spent. PyPI does not allow re-uploading a version, even after deleting it. Do not try to +reuse the number — fix the problem and release the next patch version instead. If the published release +is actively harmful, yank it on PyPI (which hides it from new resolutions without breaking pins that +already reference it) and follow up with a fixed release. diff --git a/b2sdk/_internal/replication/monitoring.py b/b2sdk/_internal/replication/monitoring.py index bae3a2a8..c97c8727 100644 --- a/b2sdk/_internal/replication/monitoring.py +++ b/b2sdk/_internal/replication/monitoring.py @@ -114,14 +114,14 @@ class ReplicationMonitor: :param b2sdk.v2.Bucket bucket: replication source bucket :param b2sdk.v2.ReplicationRule rule: replication rule to be monitored; - should belong to `bucket`'s replication configuration + should belong to `bucket`'s replication configuration :param b2sdk.v2.B2Api destination_api: B2Api instance for destination - bucket; if destination bucket is on the same account as source bucket, - omit this parameter and then source bucket's B2Api will be used + bucket; if destination bucket is on the same account as source bucket, + omit this parameter and then source bucket's B2Api will be used :param b2sdk.v2.ProgressReport report: instance of ProgressReport which will report - scanning progress, by default to stdout + scanning progress, by default to stdout :param b2sdk.v2.ScanPoliciesManager scan_policies_manager: a strategy to scan - files, so that several files that match some criteria may be omitted + files, so that several files that match some criteria may be omitted :rtype: b2sdk.v2.ReplicationMonitor """ diff --git a/changelog.d/+interface-boundary.doc.md b/changelog.d/+interface-boundary.doc.md new file mode 100644 index 00000000..391df331 --- /dev/null +++ b/changelog.d/+interface-boundary.doc.md @@ -0,0 +1 @@ +State explicitly that being importable from a versioned namespace such as `b2sdk.v3` is not a compatibility promise, and document how to tell whether a name is public, internal, or unclassified. diff --git a/changelog.d/584.doc.md b/changelog.d/584.doc.md new file mode 100644 index 00000000..756bbea1 --- /dev/null +++ b/changelog.d/584.doc.md @@ -0,0 +1 @@ +Fix the Quick Start examples to match the public `b2sdk.v3` interface: `EncryptionSetting` rather than `EncryptionSettings`, `EncryptionKey(key_id=...)` rather than `id=`, both required mappings for `BasicSyncEncryptionSettingsProvider`, and `file_info=` rather than `file_infos=`. diff --git a/changelog.d/585.doc.md b/changelog.d/585.doc.md new file mode 100644 index 00000000..784e6f02 --- /dev/null +++ b/changelog.d/585.doc.md @@ -0,0 +1 @@ +Use `update_file_retention` instead of `update_file_legal_hold` in the Quick Start file-lock examples that pass a `FileRetentionSetting`. diff --git a/changelog.d/586.doc.md b/changelog.d/586.doc.md new file mode 100644 index 00000000..986ff169 --- /dev/null +++ b/changelog.d/586.doc.md @@ -0,0 +1 @@ +Fix the Advanced usage patterns guide to reference `UploadSourceLocalFile` rather than the nonexistent `LocalUploadSource`, and to call `bucket.create_file` rather than an undefined `planner` object. diff --git a/changelog.d/588.doc.md b/changelog.d/588.doc.md new file mode 100644 index 00000000..c07000e0 --- /dev/null +++ b/changelog.d/588.doc.md @@ -0,0 +1 @@ +Remove the nonexistent `deploy` Nox session from the Contributors Guide and describe how releases are actually published by the Continuous Delivery workflow. diff --git a/changelog.d/589.doc.md b/changelog.d/589.doc.md new file mode 100644 index 00000000..f4b8015d --- /dev/null +++ b/changelog.d/589.doc.md @@ -0,0 +1 @@ +Correct the stale CI test matrix description in the contributor guides, which claimed eight Python versions instead of the currently supported set. diff --git a/changelog.d/590.doc.md b/changelog.d/590.doc.md new file mode 100644 index 00000000..af97b4e7 --- /dev/null +++ b/changelog.d/590.doc.md @@ -0,0 +1 @@ +Correct the Installation Guide's development-install instructions, which told readers to check out the repository and then install the published package from PyPI instead of their local checkout. diff --git a/changelog.d/591.doc.md b/changelog.d/591.doc.md new file mode 100644 index 00000000..3f62ef35 --- /dev/null +++ b/changelog.d/591.doc.md @@ -0,0 +1 @@ +Fix typos in the Advanced usage patterns guide and remove an outdated time-relative claim about the library's age from the overview. diff --git a/changelog.d/592.doc.md b/changelog.d/592.doc.md new file mode 100644 index 00000000..0e2aba5d --- /dev/null +++ b/changelog.d/592.doc.md @@ -0,0 +1 @@ +Document how to choose a location for a new test in the Contributors Guide, covering the shared unit directories, the per-interface `v0`-`v3` directories, `v_all`, integration tests, the `apiver` marker, and a checklist for changes to `b2sdk/_internal`. diff --git a/changelog.d/593.doc.md b/changelog.d/593.doc.md new file mode 100644 index 00000000..26594d81 --- /dev/null +++ b/changelog.d/593.doc.md @@ -0,0 +1 @@ +Expand the release guide into a maintainer runbook covering prerequisites, the changelog build, the commit/CI/tag sequence, what the Continuous Delivery workflow does, post-release verification, and recovery before and after a tag is pushed. diff --git a/changelog.d/595.doc.md b/changelog.d/595.doc.md new file mode 100644 index 00000000..f59ac613 --- /dev/null +++ b/changelog.d/595.doc.md @@ -0,0 +1 @@ +Add a task-oriented replication guide covering setup, required application-key capabilities, per-file replication status, monitoring and how to read a scan report, plus API reference pages for the public replication classes. diff --git a/changelog.d/599.doc.md b/changelog.d/599.doc.md new file mode 100644 index 00000000..deb43320 --- /dev/null +++ b/changelog.d/599.doc.md @@ -0,0 +1 @@ +Consolidate contributor guidance into the published Contributors Guide, and reduce `CONTRIBUTING.md` to a short entry point that links to it, so the two copies can no longer drift apart. diff --git a/changelog.d/601.doc.md b/changelog.d/601.doc.md new file mode 100644 index 00000000..cf88f11b --- /dev/null +++ b/changelog.d/601.doc.md @@ -0,0 +1 @@ +Resolve the embedded documentation TODOs: drop the obsolete ones, make the "Why use b2sdk?" list consistent and linked, and document the progress reporters and the exception hierarchy properly. diff --git a/doc/source/advanced.rst b/doc/source/advanced.rst index 5b78c8f9..6e0d283b 100644 --- a/doc/source/advanced.rst +++ b/doc/source/advanced.rst @@ -79,7 +79,7 @@ Concatenate files of known size If one of remote source has length smaller than :term:`absoluteMinimumPartSize` then it cannot be copied into large file part. Such remote source would be downloaded and concatenated locally with local source or with other downloaded remote source. -Please note that this method only allows checksum verification for local upload sources. Checksum verification for remote sources is available only when local copy is available. In such case :meth:`b2sdk.v3.Bucket.create_file` can be used with overalapping ranges in input. +Please note that this method only allows checksum verification for local upload sources. Checksum verification for remote sources is available only when local copy is available. In such case :meth:`b2sdk.v3.Bucket.create_file` can be used with overlapping ranges in input. For more information about ``concatenate`` please see :meth:`b2sdk.v3.Bucket.concatenate` and :class:`b2sdk.v3.CopySource`. @@ -150,7 +150,7 @@ The assumption here is that the file has been appended to since it was last uplo >>> bucket.create_file(input_sources, remote_name, file_info) -`LocalUploadSource` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range (when concatenating, local source would have destination offset at the end of remote source) +:class:`b2sdk.v3.UploadSourceLocalFile` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range (when concatenating, local source would have destination offset at the end of remote source) For more information see :meth:`b2sdk.v3.Bucket.create_file`. @@ -179,7 +179,7 @@ Change the middle of the remote file >>> bucket.create_file(input_sources, remote_name, file_info) -`LocalUploadSource` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range. +:class:`b2sdk.v3.UploadSourceLocalFile` has the size determined automatically in this case. This is more efficient than :meth:`b2sdk.v3.Bucket.concatenate`, as it can use the overlapping ranges when a remote part is smaller than :term:`absoluteMinimumPartSize` to prevent downloading a range. For more information see :meth:`b2sdk.v3.Bucket.create_file`. @@ -284,7 +284,7 @@ Here the planner has only used a remote source where remote range was not availa .. code-block:: python - >>> planner.create_file(input_sources, remote_name, file_info, prioritize='remote') + >>> bucket.create_file(input_sources, remote_name, file_info, prioritize='remote') # planner parts: cloud[A, D], local[D, E] Here the planner has only used a local source where remote range was not available, minimizing uploads. @@ -301,8 +301,10 @@ In `local_verification` mode the remote range was artificially split into three .. note:: `prioritize` is just a planner setting - remote parts are always verified if matching local parts exists. -.. TODO:: - prioritization should accept enum, not string +.. todo:: + Prioritization should accept an enum, not a string. Tracked in + `issue #602 `_. Kept here so that anyone + editing this section sees it alongside the documented string values. .. _continuation: @@ -353,7 +355,7 @@ Manual continuation (streamed version) >>> large_file_id = storage.query({'name': remote_name})[0]['large_file_id'] >>> bucket.create_file_stream(input_sources, remote_name, file_info, large_file_id=large_file_id) -Streams that contains remote sources cannot be continued with :meth:`b2sdk.v3.Bucket.create_file` - internally :meth:`b2sdk.v3.Bucket.create_file` stores plan information in file info for such inputs, and verifies it before any copy/upload and :meth:`b2sdk.v3.Bucket.create_file_stream` cannot store this information. Local source only inputs can be safely continued with :meth:`b2sdk.v3.Bucket.create_file` in auto continue mode or manual continue mode (because plan information is not stored in file info in such case). +Streams that contain remote sources cannot be continued with :meth:`b2sdk.v3.Bucket.create_file` - internally :meth:`b2sdk.v3.Bucket.create_file` stores plan information in file info for such inputs, and verifies it before any copy/upload and :meth:`b2sdk.v3.Bucket.create_file_stream` cannot store this information. Local source only inputs can be safely continued with :meth:`b2sdk.v3.Bucket.create_file` in auto continue mode or manual continue mode (because plan information is not stored in file info in such case). Auto continuation ----------------- @@ -364,7 +366,7 @@ Auto continuation For local source only input, :meth:`b2sdk.v3.Bucket.create_file` would try to find matching unfinished large file. It will verify uploaded parts checksums with local sources - the most completed, having all uploaded parts matched candidate would be automatically selected as file to continue. If there is no matching candidate (even if there are unfinished files for the same file name) new large file would be started. -In other cases plan information would be generated and :meth:`b2sdk.v3.Bucket.create_file` would try to find unfinished large file with matching plan info in its file info. If there is one or more such unfinished large files, :meth:`b2sdk.v3.Bucket.create_file` would verify checksums for all locally available parts and choose any matching candidate. If all candidates fails on uploaded parts checksums verification, process is interrupted and error raises. In such case corrupted unfinished large files should be cancelled manullay and :meth:`b2sdk.v3.Bucket.create_file` should be retried, or auto continuation should be turned off with `auto_continue=False` +In other cases plan information would be generated and :meth:`b2sdk.v3.Bucket.create_file` would try to find unfinished large file with matching plan info in its file info. If there is one or more such unfinished large files, :meth:`b2sdk.v3.Bucket.create_file` would verify checksums for all locally available parts and choose any matching candidate. If all candidates fail on uploaded parts checksums verification, process is interrupted and error raises. In such case corrupted unfinished large files should be cancelled manually and :meth:`b2sdk.v3.Bucket.create_file` should be retried, or auto continuation should be turned off with `auto_continue=False` No continuation diff --git a/doc/source/api/exception.rst b/doc/source/api/exception.rst index 4a0396d9..30e1bcca 100644 --- a/doc/source/api/exception.rst +++ b/doc/source/api/exception.rst @@ -1,8 +1,55 @@ Exceptions ========== -.. todo:: - improve documentation of exceptions, automodule -> autoclass? +Every error raised by **b2sdk** derives from :class:`b2sdk.v3.exception.B2Error`, so catching that one +class is enough to catch anything the library raises deliberately. + +How to catch them +----------------- + +The hierarchy is arranged so you can be as specific as you need. Catch a named class when you intend to +handle that exact condition, and fall back to a broader one otherwise: + +.. code-block:: python + + >>> from b2sdk.v3.exception import B2Error, NonExistentBucket + + >>> try: + bucket = b2_api.get_bucket_by_name('no-such-bucket') + except NonExistentBucket: + ... # handle this specific case + except B2Error as e: + print(e) # anything else the SDK raises + +.. warning:: + The hierarchy may gain intermediate classes in a backwards-compatible release, so an exception's + direct parent is not guaranteed to stay the same. Use ``isinstance`` and ``super()`` rather than + naming a parent class explicitly. See :ref:`interface version compatibility + `. + +Deciding whether to retry +------------------------- + +Errors carry their own retry advice, so you rarely need to inspect status codes yourself: + +* ``should_retry_http()`` is true when the failed HTTP call is worth repeating. +* ``should_retry_upload()`` is true when the upload should be retried after obtaining a fresh upload URL. +* ``retry_after_seconds`` is set when the server asked the client to wait before issuing more requests. + When it is present, honour it rather than retrying immediately. + +**b2sdk** already applies this logic internally for its own transfers, so these are relevant mainly if +you are driving the lower-level API yourself. + +Two conveniences +---------------- + +Every error has a ``prefix`` property giving a readable name derived from the class name, which is what +makes the default messages presentable to end users. And +:func:`b2sdk.v3.exception.interpret_b2_error` turns a raw B2 error response into the appropriate +exception class, which is the function responsible for the hierarchy below being raised at all. + +Reference +--------- .. automodule:: b2sdk.v3.exception :members: diff --git a/doc/source/api/progress.rst b/doc/source/api/progress.rst index 95f3c443..6a1daf80 100644 --- a/doc/source/api/progress.rst +++ b/doc/source/api/progress.rst @@ -1,13 +1,67 @@ Progress reporters ================== -.. note:: - Concrete classes described in this chapter implement methods defined in ``AbstractProgressListener`` +**b2sdk** reports progress through two separate mechanisms, and which one you want depends on what you +are doing. + +For **a single transfer** - one upload, download or copy - pass a *progress listener* to the method. +Every such method accepts a ``progress_listener`` argument, and the classes on this page are the +available implementations. + +For **a whole sync operation**, which scans folders and then transfers many files, a single listener is +not enough. Sync instead takes a :class:`b2sdk.v3.SyncReport`, which tracks the scan and comparison +phases as well as the transfers. See :ref:`sync` for how to use it. + +Choosing a listener +------------------- + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - Class + - When to use it + * - :class:`b2sdk.v3.TqdmProgressListener` + - Interactive use. Draws a progress bar, and needs the ``tqdm`` package installed. + * - :class:`b2sdk.v3.SimpleProgressListener` + - Interactive use without ``tqdm``. Prints plain percentages. + * - :class:`b2sdk.v3.DoNothingProgressListener` + - Non-interactive use - servers, scripts, tests. Discards everything. + * - :class:`b2sdk.v3.ProgressListenerForTest` + - Test support. Records the calls it receives so they can be asserted on. + +If you do not want to choose, :func:`b2sdk.v3.make_progress_listener` picks for you: it returns +``DoNothingProgressListener`` when ``quiet`` is set, ``TqdmProgressListener`` when ``tqdm`` is +importable, and ``SimpleProgressListener`` otherwise. -.. todo:: - improve documentation of progress reporters +.. code-block:: python - include info about sync progress + >>> from b2sdk.v3 import make_progress_listener + + >>> with make_progress_listener('uploading backup.tar', quiet=False) as listener: + bucket.upload_local_file( + local_file='backup.tar', + file_name='backup.tar', + progress_listener=listener, + ) + +Writing your own +---------------- + +Subclass :class:`b2sdk.v3.AbstractProgressListener` and implement ``set_total_bytes`` and +``bytes_completed``. Two details of the contract are easy to get wrong: + +* ``set_total_bytes`` is always called before the listener is entered, but **may be called again** if a + transfer is retried. +* ``bytes_completed`` receives a **running total, not a delta**, and that total **can go down**, because + a failed transfer restarts from the beginning. Do not accumulate the values you are given. + +Listeners are context managers, and ``close`` must be called exactly once - it asserts if called twice - +so prefer a ``with`` block over calling it yourself. + +.. note:: + The concrete classes on this page all implement the methods defined by + :class:`b2sdk.v3.AbstractProgressListener`. .. autoclass:: b2sdk.v3.AbstractProgressListener :inherited-members: diff --git a/doc/source/api/replication/monitoring.rst b/doc/source/api/replication/monitoring.rst new file mode 100644 index 00000000..a3b74207 --- /dev/null +++ b/doc/source/api/replication/monitoring.rst @@ -0,0 +1,19 @@ +.. _replication_monitoring: + +Replication monitoring +====================== + +.. autoclass:: b2sdk.v3.ReplicationMonitor() + :members: + :exclude-members: source_folder, destination_folder + :special-members: __init__ + +.. autoclass:: b2sdk.v3.ReplicationScanResult() + :inherited-members: + :members: + +``ReplicationReport`` is a :class:`b2sdk.v3.CountAndSampleScanReport` whose results are +:class:`b2sdk.v3.ReplicationScanResult` instances. It groups scanned files by result rather than +listing them: ``counter_by_status`` counts the files sharing each result, while +``samples_by_status_first`` and ``samples_by_status_last`` hold the first and last example seen for +each. See :ref:`replication` for how to read one. diff --git a/doc/source/api/replication/setting.rst b/doc/source/api/replication/setting.rst new file mode 100644 index 00000000..ff6992d0 --- /dev/null +++ b/doc/source/api/replication/setting.rst @@ -0,0 +1,17 @@ +.. _replication_setting: + +Replication settings +==================== + +.. autoclass:: b2sdk.v3.ReplicationConfiguration() + :inherited-members: + :members: + :special-members: __init__ + +.. autoclass:: b2sdk.v3.ReplicationRule() + :inherited-members: + :members: + :special-members: __init__ + +.. autoclass:: b2sdk.v3.ReplicationConfigurationFactory() + :no-members: diff --git a/doc/source/api/replication/setup.rst b/doc/source/api/replication/setup.rst new file mode 100644 index 00000000..786f929c --- /dev/null +++ b/doc/source/api/replication/setup.rst @@ -0,0 +1,8 @@ +.. _replication_setup: + +Replication setup helper +======================== + +.. autoclass:: b2sdk.v3.ReplicationSetupHelper() + :members: + :special-members: __init__ diff --git a/doc/source/api/replication/types.rst b/doc/source/api/replication/types.rst new file mode 100644 index 00000000..bb28b4a2 --- /dev/null +++ b/doc/source/api/replication/types.rst @@ -0,0 +1,8 @@ +.. _replication_types: + +Replication types +================= + +.. autoclass:: b2sdk.v3.ReplicationStatus() + :inherited-members: + :members: diff --git a/doc/source/api_reference.rst b/doc/source/api_reference.rst index e449307a..3825eb33 100644 --- a/doc/source/api_reference.rst +++ b/doc/source/api_reference.rst @@ -37,6 +37,10 @@ Public API api/transfer/outbound/outbound_source api/encryption/setting api/encryption/types + api/replication/setting + api/replication/setup + api/replication/monitoring + api/replication/types .. _api_internal: diff --git a/doc/source/api_types.rst b/doc/source/api_types.rst index 3995def7..86fc279c 100644 --- a/doc/source/api_types.rst +++ b/doc/source/api_types.rst @@ -74,6 +74,40 @@ Those modules will generally not change in a backwards-incompatible way between .. note:: ``b2sdk.*._something`` and ``b2sdk.*.*._something``, while having a name beginning with an underscore, are **NOT** considered public interface. +Importable is not the same as public +==================================== + +The versioned namespace re-exports a considerably larger surface than the public interface. Alongside +the classes documented here, ``b2sdk.v3`` carries low-level HTTP machinery, test simulators, +transfer-manager internals, module-level constants, and a few names that are merely an artefact of +star-imports - ``annotations``, leaked from ``from __future__ import annotations``, is one. Most of the +names in the namespace are not mentioned anywhere in this documentation. + +**Being importable from** ``b2sdk.v3`` **is therefore not a compatibility promise.** To find out where a +name stands, look it up in the :doc:`API Reference `: + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - Where the name appears + - What is promised + * - :ref:`Public API ` + - Public interface. Pin the major version. + * - :ref:`Internal API ` + - Documented, but may change in a non-major release. Pin the middle version. + * - Neither + - No compatibility promise. It may be renamed or removed in any release. + +A name in the third group has not necessarily been judged internal - classifying the namespace is still +in progress. Treat the absence of documentation as the absence of a promise, rather than as a statement +about what the name is meant to be. If you depend on such a name, please file an issue so it can be +considered for the public interface instead of being relied on as an incidental re-export. + +The current contents of the namespace can be listed with:: + + python -c "import b2sdk.v3; print(sorted(n for n in vars(b2sdk.v3) if not n.startswith('_')))" + .. _internal_interface: ****************** diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst index 03c33fe9..74676259 100644 --- a/doc/source/contributing.rst +++ b/doc/source/contributing.rst @@ -20,11 +20,50 @@ We encourage outside contributors to perform changes on our codebase. Many such * runs all sorts of linters * checks if the Python distribution can be built - * runs all tests on a matrix of 8 versions of Python (including pypy) and 3 operating systems (Linux, Mac OS X and Windows) + * runs all tests on a matrix of every supported Python version (CPython and PyPy, see ``PYTHON_VERSIONS`` in ``noxfile.py``) across Linux, macOS and Windows * checks if the documentation can be built properly * maintain other Continuous Integration tools (coverage tracker) +Versioning +############# + +This package's versions adhere to `Semantic Versioning `_, and the +versions are established by reading git tags, i.e. no code or manifest file changes are required when +working on PRs. + +Changelog +############# + +Each PR needs to have at least one changelog (aka news) item added. This is done by creating files in +``changelog.d``. `towncrier `_ is used for compiling these files into +``CHANGELOG.md``. There are several types of changes (news): + +#. ``fixed`` +#. ``changed`` +#. ``added`` +#. ``deprecated`` +#. ``removed`` +#. ``infrastructure`` +#. ``doc`` + +The ``changelog.d`` file name convention is: + +#. If the PR closes a GitHub issue: ``{issue_number}.{type}.md``, e.g. ``157.fixed.md``. Note that the + change description still has to be complete; linking an issue is just there for convenience. A change + like ``fixed #157`` will not be accepted. +#. If the PR is not related to a GitHub issue: ``+{unique_string}.{type}.md``, e.g. ``+foobar.fixed.md``. + +These files can either be created manually, or using ``towncrier``, e.g.:: + + $ towncrier create -c 'write your description here' 157.fixed.md + +``towncrier create`` also takes care of duplicates automatically (if there is more than one news fragment +of one type for a given GitHub issue). + +Developer info +############## + You'll need to have `nox `_ and `uv `_ installed: * ``pip install nox uv`` @@ -36,10 +75,14 @@ With ``nox``, you can run different sessions (default are ``lint`` and ``test``) * ``test`` (``test-3.10``, ``test-3.11``, ``test-3.12``, ``test-3.13``, ``test-3.14``, ``test-pypy3.10``) -> Run test suite. * ``cover`` -> Perform coverage analysis. * ``build`` -> Build the distribution. -* ``deploy`` -> Deploy the distribution to the PyPi. * ``doc`` -> Build the documentation. * ``doc_cover`` -> Perform coverage analysis for the documentation. +Releases are not published from a ``nox`` session. The *Continuous Delivery* GitHub Actions +workflow (``.github/workflows/cd.yml``) builds the distribution when a version tag is pushed, +creates a GitHub release, and uploads it to PyPI for non-prerelease versions. The release +procedure itself is described in ``README.release.md``. + For example:: $ nox -s format @@ -68,6 +111,17 @@ With the above setting, session ``test`` will run on Python 3.12 and 3.14, and a Given Python interpreters should be installed in the operating system or via `pyenv `_. +Managing dependencies +##################### + +We use `uv `_ for managing dependencies and developing locally. If you want +to change any of the project requirements (or requirement bounds) in ``pyproject.toml``, make sure that +the ``uv.lock`` file reflects those changes by using ``uv add``, ``uv lock`` or other commands - see the +`uv documentation `_. You can verify that the lock file is up to date by +running:: + + $ uv lock --check + Linting ############# @@ -97,6 +151,75 @@ To run just integration tests:: $ export B2_TEST_APPLICATION_KEY_ID=your_app_key_id $ nox -s integration-3.10 +To run tests matching a keyword expression:: + + $ nox -s unit-3.10 -- -k keyword + +Where to put a new test +======================= + +**b2sdk** keeps several numbered interfaces (``b2sdk.v0`` through ``b2sdk.v3``) working at the same +time, and the unit suite is what holds that promise. ``nox -s unit`` therefore runs ``pytest`` once per +interface version - ``--api=v3``, then ``v2``, ``v1`` and ``v0`` - so a single shared test is executed +four times, once against each interface. + +Two mechanisms make that work: + +* ``pytest`` gains an ``--api`` option, defaulting to the newest version. It puts + ``test/unit//apiver`` on ``sys.path``, so a shared test can write + ``from apiver_deps import B2Api`` and receive whichever interface is under test. The ``apiver`` and + ``apiver_int`` fixtures expose that version as ``"v2"`` or ``2`` respectively. +* Tests living under a *different* version's directory are not collected. Running with ``--api=v3`` + skips ``test/unit/v0``, ``v1`` and ``v2`` entirely. + +Choose a location accordingly: + +.. list-table:: + :header-rows: 1 + :widths: 28 22 50 + + * - Location + - Collected for + - Use it for + * - ``test/unit//``, e.g. ``bucket``, ``api``, ``sync`` + - every interface version + - The default home. Behaviour shared by all interfaces - in practice, anything implemented in + ``b2sdk/_internal``. Import through ``apiver_deps``. + * - ``test/unit/v0`` … ``test/unit/v3`` + - only its own version + - Behaviour that exists in exactly one interface, such as a legacy constructor argument or an + older return type preserved by a compatibility adapter. + * - ``test/unit/v_all/`` + - every interface version + - Assertions about the interfaces themselves - that a name is still exported, or that two + versions agree. + * - ``test/integration/`` + - not run by ``nox -s unit`` + - End-to-end behaviour against real B2. Requires credentials, so it cannot gate every PR. + +If a shared test applies to only some versions, keep it in the shared directory and mark it rather than +copying it into each version directory:: + + @pytest.mark.apiver(1) # only v1 + @pytest.mark.apiver(1, 3) # only v1 and v3 + @pytest.mark.apiver(from_ver=2) # v2 and newer + @pytest.mark.apiver(to_ver=2) # v2 and older + +``from_ver`` and ``to_ver`` are inclusive, and a single mark may not mix positional and keyword +arguments. Marks also work on individual ``pytest.param`` entries, which is the usual way to assert that +an older interface raises a different exception than a newer one. When a module-level ``pytestmark`` and +a test-level mark disagree, the test is skipped unless *both* conditions allow it. See +``test/unit/conftest.py`` for the full behaviour. + +When you change something in ``b2sdk/_internal``, check that: + +#. the regression test sits in a shared directory, so it runs for every interface, and not in a single + version directory where it would silently cover only one; +#. it imports from ``apiver_deps`` rather than a fixed ``b2sdk.vN``; +#. any interface that deliberately keeps the old behaviour has a test asserting that, placed in that + version's directory or marked with ``apiver``; +#. ``nox -s unit`` passes - a suite that passes only for the newest interface is not enough. + Documentation ############# diff --git a/doc/source/index.rst b/doc/source/index.rst index f1f2f61f..772ad9ad 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -16,26 +16,18 @@ to provide command-line access to the B2 service, but there are many possible ap Why use b2sdk? ######################################### -.. todo:: - delete doc/source/b2sdk? +When building an application which uses B2 cloud, it is possible to implement an independent B2 API client, but using **b2sdk** gives you: -.. todo:: - describe raw_simulator in detail - -.. todo:: - fix list consistency style in "Why use b2sdk?", add links - -When building an application which uses B2 cloud, it is possible to implement an independent B2 API client, but using **b2sdk** allows for: - -- reuse of code that is already written, with hundreds of unit tests -- use of **Synchronizer**, a high-performance, parallel rsync-like utility -- developer-friendly library :ref:`api version policy ` which guards your program against incompatible changes -- `B2 integration checklist `_ is passed automatically -- **raw_simulator** makes it easy to mock the B2 cloud for unit testing purposes -- reporting progress of operations to an object of your choice -- exception hierarchy makes it easy to display informative messages to users -- interrupted transfers are automatically continued -- **b2sdk** has been developed for 3 years before it version 1.0.0 was released. It's stable and mature. +- code that is already written and covered by hundreds of unit tests +- :ref:`Synchronizer `, a high-performance, parallel rsync-like utility +- :ref:`Replication ` support, for keeping a second bucket up to date automatically +- a developer-friendly :ref:`api version policy ` that guards your program against incompatible changes +- automatic compliance with the `B2 integration checklist `_ +- :doc:`raw_simulator `, which mocks the B2 cloud so you can unit test without network access +- :doc:`progress reporting ` to an object of your choice +- an :doc:`exception hierarchy ` that makes it easy to show users informative messages +- automatic continuation of interrupted transfers +- a stable and mature codebase, in development for years before its 1.0.0 release ######################################### @@ -48,6 +40,7 @@ Documentation index tutorial quick_start server_side_encryption + replication advanced glossary api_types diff --git a/doc/source/install.rst b/doc/source/install.rst index 159ea333..1b2d75ea 100644 --- a/doc/source/install.rst +++ b/doc/source/install.rst @@ -15,8 +15,15 @@ In order to properly set a dependency, see :ref:`versioning chapter `_ and +`uv `_ to manage test and documentation environments. See +:ref:`contributors_guide` for the full development setup. diff --git a/doc/source/quick_start.rst b/doc/source/quick_start.rst index 4c74eceb..2950ea88 100644 --- a/doc/source/quick_start.rst +++ b/doc/source/quick_start.rst @@ -50,14 +50,18 @@ Synchronization ) >>> no_progress = False - >>> encryption_settings_provider = BasicSyncEncryptionSettingsProvider({ - 'bucket1': EncryptionSettings(mode=EncryptionMode.SSE_B2), - 'bucket2': EncryptionSettings( + >>> bucket_settings = { + 'bucket1': EncryptionSetting(mode=EncryptionMode.SSE_B2), + 'bucket2': EncryptionSetting( mode=EncryptionMode.SSE_C, - key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', id='user-generated-key-id') + key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', key_id='user-generated-key-id') ), 'bucket3': None, - }) + } + >>> encryption_settings_provider = BasicSyncEncryptionSettingsProvider( + read_bucket_settings=bucket_settings, + write_bucket_settings=bucket_settings, + ) >>> with SyncReport(sys.stdout, no_progress) as reporter: synchronizer.sync_folders( source_folder=source, @@ -176,7 +180,7 @@ Upload file >>> bucket.upload_local_file( local_file=local_file_path, file_name=b2_file_name, - file_infos=file_info, + file_info=file_info, ) @@ -194,14 +198,14 @@ Upload file encrypted with SSE-C >>> file_info = {'how': 'good-file'} >>> encryption_setting = EncryptionSetting( mode=EncryptionMode.SSE_C, - key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', id='user-generated-key-id'), + key=EncryptionKey(secret=b'VkYp3s6v9y$B&E)H@McQfTjWmZq4t7w!', key_id='user-generated-key-id'), ) >>> bucket = b2_api.get_bucket_by_name(bucket_name) >>> bucket.upload_local_file( local_file=local_file_path, file_name=b2_file_name, - file_infos=file_info, + file_info=file_info, encryption=encryption_setting, ) @@ -337,7 +341,7 @@ Update file lock configuration >>> file_id = '4_z5485a1682662eb3e60980d10_f113f963288e711a6_d20190404_m065910_c002_v0001095_t0044' >>> file_name = 'dummy.pdf' >>> b2_api.update_file_legal_hold(file_id, file_name, LegalHold.ON) - >>> b2_api.update_file_legal_hold( + >>> b2_api.update_file_retention( file_id, file_name, FileRetentionSetting(RetentionMode.GOVERNANCE, int(time.time() + 100)*1000)) @@ -497,10 +501,10 @@ Update file lock >>> # equivalent to >>> b2_api.update_file_legal_hold(file_version.id_, file_version.file_name, LegalHold.ON) >>> b2_api.update_file_legal_hold(download_version.id_, download_version.file_name, LegalHold.ON) - >>> b2_api.update_file_legal_hold( + >>> b2_api.update_file_retention( file_version.id_, file_version.file_name, FileRetentionSetting(RetentionMode.GOVERNANCE, int(time.time() + 100)*1000)) - >>> b2_api.update_file_legal_hold( + >>> b2_api.update_file_retention( download_version.id_, download_version.file_name, FileRetentionSetting(RetentionMode.GOVERNANCE, int(time.time() + 100)*1000)) diff --git a/doc/source/replication.rst b/doc/source/replication.rst new file mode 100644 index 00000000..ebfd1fd1 --- /dev/null +++ b/doc/source/replication.rst @@ -0,0 +1,236 @@ +.. _replication: + +######################## +Replication +######################## + +Replication asks B2 to copy files from one bucket to another automatically, as they are uploaded. The +bucket being copied *from* is the **source**; the bucket being copied *to* is the **destination**. One +bucket may be both at once, and a source bucket may replicate to several destinations under different +rules. + +Replication is configured on the buckets, not on individual files: you attach a +:class:`b2sdk.v3.ReplicationConfiguration` to each side, and B2 does the copying. The SDK's job is to +help you write that configuration, create the application keys it needs, and inspect the outcome. + +.. note:: + Replication only applies to files uploaded *after* a rule starts matching them. To bring across + files that already exist, set ``include_existing_files=True`` on the rule. + +********************** +What replication needs +********************** + +Both sides need an application key, and the two sides need different capabilities: + +.. list-table:: + :header-rows: 1 + :widths: 20 80 + + * - Side + - Capabilities + * - Source + - ``readFiles``, ``readFileLegalHolds``, ``readFileRetentions`` + * - Destination + - ``writeFiles``, ``writeFileLegalHolds``, ``writeFileRetentions``, ``deleteFiles`` + +The source bucket's configuration records the id of the source key. The destination bucket's +configuration maps each incoming source key id to the destination key that should be used for it - that +mapping is how a destination bucket authorises a particular source to write into it. + +If the two buckets live in different accounts, you will need a separate :class:`b2sdk.v3.B2Api` +instance for each. + +********************** +Setting up replication +********************** + +Using the setup helper +====================== + +:class:`b2sdk.v3.ReplicationSetupHelper` is the supported path. It creates both application keys, adds +the rule to the source, and registers the key mapping on the destination: + +.. code-block:: python + + >>> from b2sdk.v3 import ReplicationSetupHelper + + >>> source_bucket = b2_api.get_bucket_by_name('source-bucket') + >>> destination_bucket = b2_api.get_bucket_by_name('destination-bucket') + + >>> rsh = ReplicationSetupHelper() + >>> source_bucket, destination_bucket = rsh.setup_both( + source_bucket=source_bucket, + destination_bucket=destination_bucket, + name='my-rule', + prefix='folder/', + ) + +It returns the two updated buckets, so use the returned objects rather than the ones you passed in. + +The keys it creates are named after the buckets, suffixed ``-replisrc`` and ``-replidst``. If a rule is +already present, the helper places the new one ahead of the existing rules rather than overwriting +them. + +The two halves are also available separately, which is what you need when the buckets are on different +accounts - run :meth:`~b2sdk.v3.ReplicationSetupHelper.setup_destination` against the destination +account first, then :meth:`~b2sdk.v3.ReplicationSetupHelper.setup_source` against the source account +with the resulting key. + +Writing the configuration by hand +================================= + +When you need full control, build the configuration yourself and pass it to +:meth:`b2sdk.v3.Bucket.update` (or to :meth:`b2sdk.v3.B2Api.create_bucket` for a new bucket). You are +then responsible for creating the application keys with the capabilities listed above. + +.. code-block:: python + + >>> from b2sdk.v3 import ReplicationConfiguration, ReplicationRule + + >>> source_bucket.update( + replication=ReplicationConfiguration( + rules=[ + ReplicationRule( + destination_bucket_id=destination_bucket.id_, + name='my-rule', + file_name_prefix='folder/', + ), + ], + source_key_id=source_key.id_, + ), + ) + + >>> destination_bucket.update( + replication=ReplicationConfiguration( + source_to_destination_key_mapping={ + source_key.id_: destination_key.id_, + }, + ), + ) + +A few constraints are enforced when the objects are constructed, so you will hear about mistakes before +a request is sent: + +* a rule ``name`` must match ``[a-zA-Z0-9_-]`` and be at most 64 characters; +* ``destination_bucket_id`` is required; +* ``priority`` must be between 1 and 2\ :sup:`31`\ -1, and defaults to 128 - lower numbers are + evaluated first, and the first matching rule wins; +* a configuration that has ``rules`` must also have a ``source_key_id``. + +Inspecting what is configured +============================= + +``bucket.replication`` is either ``None`` or a :class:`b2sdk.v3.ReplicationConfiguration`. Because one +bucket can be both ends of a replication relationship, ask which role it plays: + +.. code-block:: python + + >>> bucket.replication.is_source + True + >>> bucket.replication.is_destination + False + >>> [rule.name for rule in bucket.replication.rules] + ['my-rule'] + +******************** +Checking on progress +******************** + +Per-file status +=============== + +A file version carries a :class:`b2sdk.v3.ReplicationStatus`, which is one of: + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - Status + - Meaning + * - ``PENDING`` + - On the source, and not yet copied to every destination. + * - ``COMPLETED`` + - On the source, and copied successfully. + * - ``FAILED`` + - On the source, and B2 will not retry. Needs investigation. + * - ``REPLICA`` + - This file *is* a copy - the status you see on the destination side. + +A source file whose status stays ``PENDING`` is usually still in flight. ``FAILED`` is terminal, and the +common causes are a key that no longer has the required capabilities, a destination key mapping that +does not match the source key in use, or a file the destination refuses because of its own lock +settings. + +Monitoring a rule +================= + +:class:`b2sdk.v3.ReplicationMonitor` walks one rule and reports what it finds on both sides: + +.. code-block:: python + + >>> from b2sdk.v3 import ReplicationMonitor + + >>> rule = source_bucket.replication.rules[0] + >>> monitor = ReplicationMonitor(bucket=source_bucket, rule=rule) + >>> report = monitor.scan() + +The monitor scans the source and, by default, the matching files on the destination so it can compare +them. Pass ``scan_destination=False`` to look at the source alone, which is much cheaper. + +Two optional arguments matter in practice. ``destination_api`` takes a second +:class:`b2sdk.v3.B2Api`, and is required when the destination bucket is on another account; omit it and +the source bucket's own API is used. ``scan_policies_manager`` takes a +:class:`b2sdk.v3.ScanPoliciesManager`, letting you skip files you do not care about. + +The monitor validates its arguments on construction: it raises ``ValueError`` if the bucket has no +replication configuration at all, or if the rule you passed does not belong to that bucket's +configuration. + +Reading the report +================== + +:meth:`~b2sdk.v3.ReplicationMonitor.scan` returns a :class:`b2sdk.v3.ReplicationReport`. Rather than a +flat list of files, it groups them: ``counter_by_status`` counts how many files share each distinct +:class:`b2sdk.v3.ReplicationScanResult`, and ``samples_by_status_first`` and ``samples_by_status_last`` +hold the first and last example seen for each, so you have something concrete to go and look at. + +.. code-block:: python + + >>> for result, count in report.counter_by_status.most_common(): + print(count, result.source_replication_status, result.destination_replication_status) + +A scan result describes a source/destination *pair*, and the fields fall into three groups: + +* what the source file looks like - ``source_replication_status``, ``source_has_hide_marker``, + ``source_encryption_mode``, ``source_has_large_metadata``, ``source_has_file_retention``, + ``source_has_legal_hold``; +* what the destination file looks like - ``destination_replication_status``; +* how the two compare - ``metadata_differs``, ``hash_differs``. + +Any of these may be ``None``, which means "not known" rather than "false" - typically because only one +side of the pair exists, or because the destination was not scanned. + +Interpreting partial results +============================ + +A scan is a snapshot of a system that is still moving, so mixed results are normal rather than a sign of +failure: + +* **A destination file is missing entirely.** The destination fields are ``None``. Expected while a file + is still ``PENDING``; worth investigating once the source reads ``COMPLETED``. +* **``hash_differs`` is true.** The two sides hold different content. This is briefly normal if the + source file was replaced while the scan was running, and a real problem otherwise. +* **``metadata_differs`` is true.** The content matches but the file info does not. Large metadata is the + usual cause - see ``source_has_large_metadata``, since B2 will not replicate metadata beyond a size + limit. +* **``source_has_legal_hold`` or ``source_has_file_retention`` is set.** Copying these requires the + ``*FileLegalHolds`` and ``*FileRetentions`` capabilities on *both* keys. Keys created by hand without + them are a common cause of ``FAILED``. + +.. note:: + Only the latest version of each file is inspected. Earlier versions are not represented in a report, + so a clean scan is not a statement about a bucket's whole version history. + +For the full signatures of everything above, see :ref:`replication_setting`, :ref:`replication_setup`, +:ref:`replication_monitoring` and :ref:`replication_types`.