Skip to content

restore.sh: take the standby parameter settings from pg_control - #72

Open
souravbiswassanto wants to merge 1 commit into
masterfrom
restore-gating-params
Open

souravbiswassanto wants to merge 1 commit into
masterfrom
restore-gating-params

Conversation

@souravbiswassanto

Copy link
Copy Markdown
Member

A PITR restore replays WAL, which makes the recovery instance a standby. PostgreSQL then enforces that max_connections, max_worker_processes, max_locks_per_transaction and max_prepared_transactions are each >= the value recorded in the source’s pg_control.

On a KubeDB database those are normally raised through the config secret. That secret is mounted at /etc/config in the database pod, but the WAL-restore sidekick gets no such mount — getRestoreSidekickVolumes in postgres/pkg/controller/sidekick.go builds only the data PVC plus the three script volumes. So the role template’s include_if_exists = '/etc/config/user.conf' resolves to nothing and recovery falls back to the built-in defaults 100 / 8 / 64 / 0.

Against any database whose configuration raises them, replay dies immediately and keeps retrying:

FATAL:  recovery aborted because of insufficient parameter settings
DETAIL: max_connections = 100 is a lower setting than on the primary server,
        where its value was 200.

Nothing surfaces this. The pods stay Running and the Postgres object sits in Provisioning; the only evidence is in kubectl logs <db>-wal-restorer-0.

Approach

Read the values out of the restored control file. pg_controldata is already in the image and reports exactly what the check compares against, so this needs no knowledge of how the source was configured.

Why not mount the config secret instead

I tried that first. It works for the parameter check but immediately fails differently:

FATAL: 58P01: could not access file "pg_stat_monitor": No such file or directory

The restore image is postgres-archiver:*_16.1-bookworm, a generic Debian Postgres — not the enterprise DB image. It cannot load the extensions a real deployment names in shared_preload_libraries. That is presumably why this script already pins shared_preload_libraries to a minimal list. Mounting the user config would undo that deliberate choice, so reading pg_control is the safer fix.

max_wal_senders is left alone unless the source ran with more than the 90 this script already pins.

Two details worth noting for review:

  • The block is appended after the role template so nothing downstream can lower the values again.
  • It starts with an explicit printf '\n' because the role template does not end with a newline. Without it the first setting is silently absorbed into the template’s trailing comment (# icu_validation_level = warningmax_connections = 200) — I hit exactly that during testing.

Verification

On a KubeDB cluster with a source running max_connections = 200, max_worker_processes = 24:

  • Before: the restore stalled indefinitely and only completed after hand-appending the settings inside the restorer pod.
  • After: the same restore completes unattended and lands exactly on the target.
restored source
total rows 3600 4200
rows written after the target 0 600
latest row 25s before the target
in_recovery false
max_connections 200 200

Unrelated observation

echo "max_replication_slots = 90" >>/tmp/postgresql.conf on the line after mv /tmp/postgresql.conf "$PGDATA/postgresql.conf" writes to a file that has already been moved, so it is a no-op. Left untouched to keep this diff focused — flagging in case it is not intentional.

A PITR restore replays WAL, which makes the recovery instance a standby, so
PostgreSQL enforces that max_connections, max_worker_processes,
max_locks_per_transaction and max_prepared_transactions are each >= the value
recorded in the source's pg_control.

On a KubeDB database those are normally raised through the config secret. That
secret is mounted at /etc/config in the database pod, but the WAL-restore
sidekick gets no such mount, so the role template's
include_if_exists '/etc/config/user.conf' resolves to nothing and recovery
falls back to the built-in defaults of 100 / 8 / 64 / 0. Against any database
whose configuration raises them, replay dies immediately with

  FATAL:  recovery aborted because of insufficient parameter settings
  DETAIL: max_connections = 100 is a lower setting than on the primary
          server, where its value was 200.

and keeps retrying. Nothing surfaces it: the pods stay Running and the
Postgres object sits in Provisioning, so the only evidence is the restorer's
log.

Read the values out of the restored control file instead. pg_controldata is
already in the image and reports exactly what the check compares against, so
this needs no knowledge of how the source was configured and cannot drag in
settings the restore image cannot honour - notably shared_preload_libraries,
which this script deliberately pins to a minimal list because the restore
image does not carry the enterprise extensions. That is also why mounting the
config secret here is the wrong fix.

max_wal_senders is left alone unless the source ran with more than the 90 this
script already pins.

The block is appended after the role template so nothing downstream can lower
the values again, and it starts with an explicit newline because the template
does not end with one - without it the first setting is silently absorbed into
the template's trailing comment line.

Verified on a KubeDB cluster: a source with max_connections 200 and
max_worker_processes 24, restored to a chosen timestamp, previously needed the
values hand-appended inside the restorer pod to get past the parameter check.
With this change the same restore completes unattended and lands exactly on
the target - 3600 of 4200 rows, every row after the target correctly absent,
latest row 25 seconds before it, and the instance promoted out of recovery.

Signed-off-by: souravbiswassanto <saurov@appscode.com>
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9c9242ea-c7db-4176-94b0-e0e1505677df


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant