restore.sh: take the standby parameter settings from pg_control - #72
Open
souravbiswassanto wants to merge 1 commit into
Open
souravbiswassanto wants to merge 1 commit into
souravbiswassanto wants to merge 1 commit into
Conversation
A PITR restore replays WAL, which makes the recovery instance a standby, so
PostgreSQL enforces that max_connections, max_worker_processes,
max_locks_per_transaction and max_prepared_transactions are each >= the value
recorded in the source's pg_control.
On a KubeDB database those are normally raised through the config secret. That
secret is mounted at /etc/config in the database pod, but the WAL-restore
sidekick gets no such mount, so the role template's
include_if_exists '/etc/config/user.conf' resolves to nothing and recovery
falls back to the built-in defaults of 100 / 8 / 64 / 0. Against any database
whose configuration raises them, replay dies immediately with
FATAL: recovery aborted because of insufficient parameter settings
DETAIL: max_connections = 100 is a lower setting than on the primary
server, where its value was 200.
and keeps retrying. Nothing surfaces it: the pods stay Running and the
Postgres object sits in Provisioning, so the only evidence is the restorer's
log.
Read the values out of the restored control file instead. pg_controldata is
already in the image and reports exactly what the check compares against, so
this needs no knowledge of how the source was configured and cannot drag in
settings the restore image cannot honour - notably shared_preload_libraries,
which this script deliberately pins to a minimal list because the restore
image does not carry the enterprise extensions. That is also why mounting the
config secret here is the wrong fix.
max_wal_senders is left alone unless the source ran with more than the 90 this
script already pins.
The block is appended after the role template so nothing downstream can lower
the values again, and it starts with an explicit newline because the template
does not end with one - without it the first setting is silently absorbed into
the template's trailing comment line.
Verified on a KubeDB cluster: a source with max_connections 200 and
max_worker_processes 24, restored to a chosen timestamp, previously needed the
values hand-appended inside the restorer pod to get past the parameter check.
With this change the same restore completes unattended and lands exactly on
the target - 3600 of 4200 rows, every row after the target correctly absent,
latest row 25 seconds before it, and the instance promoted out of recovery.
Signed-off-by: souravbiswassanto <saurov@appscode.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A PITR restore replays WAL, which makes the recovery instance a standby. PostgreSQL then enforces that
max_connections,max_worker_processes,max_locks_per_transactionandmax_prepared_transactionsare each>=the value recorded in the source’spg_control.On a KubeDB database those are normally raised through the config secret. That secret is mounted at
/etc/configin the database pod, but the WAL-restore sidekick gets no such mount —getRestoreSidekickVolumesinpostgres/pkg/controller/sidekick.gobuilds only the data PVC plus the three script volumes. So the role template’sinclude_if_exists = '/etc/config/user.conf'resolves to nothing and recovery falls back to the built-in defaults100 / 8 / 64 / 0.Against any database whose configuration raises them, replay dies immediately and keeps retrying:
Nothing surfaces this. The pods stay
Runningand thePostgresobject sits inProvisioning; the only evidence is inkubectl logs <db>-wal-restorer-0.Approach
Read the values out of the restored control file.
pg_controldatais already in the image and reports exactly what the check compares against, so this needs no knowledge of how the source was configured.Why not mount the config secret instead
I tried that first. It works for the parameter check but immediately fails differently:
The restore image is
postgres-archiver:*_16.1-bookworm, a generic Debian Postgres — not the enterprise DB image. It cannot load the extensions a real deployment names inshared_preload_libraries. That is presumably why this script already pinsshared_preload_librariesto a minimal list. Mounting the user config would undo that deliberate choice, so readingpg_controlis the safer fix.max_wal_sendersis left alone unless the source ran with more than the 90 this script already pins.Two details worth noting for review:
printf '\n'because the role template does not end with a newline. Without it the first setting is silently absorbed into the template’s trailing comment (# icu_validation_level = warningmax_connections = 200) — I hit exactly that during testing.Verification
On a KubeDB cluster with a source running
max_connections = 200,max_worker_processes = 24:in_recoveryfalsemax_connections200Unrelated observation
echo "max_replication_slots = 90" >>/tmp/postgresql.confon the line aftermv /tmp/postgresql.conf "$PGDATA/postgresql.conf"writes to a file that has already been moved, so it is a no-op. Left untouched to keep this diff focused — flagging in case it is not intentional.