Recover and roll back

Pick the correct recovery path

Situation	Supported path
New release fails immediately and an N-1 point exists	nookctl rollback from the retained pre-update backup
Host/database loss with a retained continuity pair	Full nookctl restore on the intended deployment
Missing or corrupt uploaded files	Restore file storage separately while API remains stopped
Need a copy at a new identity/origin	Deployment fork/migration procedure, not restore
Lost Portal/commercial access	Customer Portal account recovery; do not edit local entitlement rows
Lost local user access	Local account/admin recovery; do not rotate the deployment identity

Restore is not migration. It must preserve the deployment identity and canonical API origin in the continuity manifest. Restoring another organization’s dump, cloning into a new origin, or running source and copy simultaneously requires the separate fork/migration workflow.

Why recovery needs a key and a revision floor

The operation key is a random secret used only to identify one restore reconciliation. The raw value is validated from a protected regular file, domain-separated, and stored only as a digest. The same key may resume the same pending_floor operation; it must not authorize a different backup or deployment.

The published revision floor is the greatest instance revision known to have been served or observed outside the backup. It prevents an old database from returning with stale identity authority. Obtain it from an independent control-plane record or retained client-observed pin—never infer it from the restored database, hostname, wall clock, or build version.

Successful reconciliation calculates:

next instance revision = max(restored revision, published revision floor) + 1

N-1 rollback after an update

Rollback is available only for the exact version transition recorded by the update. Create one protected operation key:

sudo install -d -m 700 /root/nook-recovery
sudo sh -c 'umask 077; openssl rand -hex 32 > /root/nook-recovery/operation-key'
sudo chmod 600 /root/nook-recovery/operation-key

Run the exact transition with a trusted floor:

sudo nookctl rollback \
  --confirm <current-version>-to-<previous-version> \
  --operation-key-file /root/nook-recovery/operation-key \
  --published-revision-floor <trusted-positive-revision> \
  --minimum-trusted-revision-floor <trusted-positive-revision>

The command binds recovery to the application deployment ID inside the verified continuity manifest—not the separate local nookctl installation ID. It verifies the artifact, keeps the complete stack restart-fenced, then starts only the exact target release’s PostgreSQL container with restart policy no. API, admin, Redis, and Caddy remain stopped or masked while it restores the database, reconciles identity, and applies current migrations. The PostgreSQL container’s fixed Compose project, release file, working directory, identity, and health are proved before destructive work. It creates, retains, and prints the exact root-private receipt path under /var/lib/nook/support/rollback-recovery-<attempt>/; rollback has no external --receipt option and no public --reconcile-only option.

Resume an interrupted lifecycle recovery

Do not start rollback until both trusted floors are available; they are bound before the destructive boundary. If update or rollback is interrupted, read the privacy-safe phase and resume only the journal-bound transaction:

sudo nookctl diagnostics
sudo nookctl lifecycle-recover --confirm-deployment-id <deployment-uuid>
sudo nookctl status

For an update that reached a database boundary before revision floors were bound, obtain both values from independent operator evidence and add them once:

sudo nookctl lifecycle-recover \
  --confirm-deployment-id <deployment-uuid> \
  --published-revision-floor <trusted-positive-revision> \
  --minimum-trusted-revision-floor <trusted-positive-revision>

The recovery command accepts no replacement path, key, or receipt. Update recovery permits only a root-locked, fsynced null/null to exact positive floor-pair transition. A partial pair, changed pair, decrease, or value above the JavaScript safe-integer maximum is rejected. If power fails after the lifecycle WAL receives the pair but before a pending_floor restore WAL does, repeat the exact same command: it binds the second WAL and resumes reconciliation without restoring again. Rollback recovery must repeat the floors already bound by the original rollback command. Caddy remains masked and both releases remain restart-fenced until the exact nested receipt, migration, state commit, API readiness, local admin probe, and single serving release are proved. A successfully resumed rollback retains the same root-private receipt and prints its canonical path. If lifecycle and restore WALs both exist, every start or mutation first proves their exact deployment, release, artifact, manifest, operation-key, receipt, floor, and actor binding. Only lifecycle-recover may resume that matching nested restore; an unrelated or corrupt pair remains fail-closed.

Update and rollback also bind the exact managed Caddyfile digest and compare the terminal configuration’s public origins immediately before Caddy is unmasked. A changed file or origin keeps ingress masked. Rollback rejects a continuity manifest from another deployment before creating its WAL; update rechecks the same deployment binding immediately after the fresh fenced backup.

Full continuity restore

Before destructive work:

1. identify the exact dump and sibling manifest;
2. verify off-host checksums and retention provenance;
3. require a strict version-2 manifest and read deploymentId, instanceRevision, canonical origin, database.sourcePhysicalBytes, and database.restoreCapacityBytes;
4. compare the intended host/origin and independently retained revision floor;
5. stop user traffic and preserve a forensic copy if this is an incident;
6. create one operation key and a new protected receipt directory.

Then run:

sudo nookctl restore \
  --artifact /absolute/backups/nook-postgres-<timestamp>.dump \
  --manifest /absolute/backups/nook-postgres-<timestamp>.dump.nook-continuity.json \
  --operation-key-file /root/nook-recovery/operation-key \
  --expected-deployment-id <deployment-id-from-manifest> \
  --confirm-deployment-id <same-deployment-id> \
  --receipt /root/nook-recovery/restore-receipt-1.json \
  --published-revision-floor <trusted-positive-revision> \
  --minimum-trusted-revision-floor <trusted-positive-revision>

The two deployment-ID arguments are deliberately redundant: destructive restore requires an exact operator confirmation. Do not copy the ID from a different host or replace it to force acceptance.

A version-1 continuity pair can still be checked with --verify-only, but it cannot authorize a destructive restore or --reconcile-only. Version 2 rejects source databases above 40 GiB and dump artifacts above 42 GiB. Its capacity binding must equal 2 × database.sourcePhysicalBytes + 2 GiB, with an 82 GiB maximum.

Before stopping writers, the restore authority resolves the live PostgreSQL container’s actual PGDATA bind or named-volume mount without following links. It rejects a missing, ambiguous, read-only, group/world-writable, changed, or external-tablespace layout. The first capacity gate requires free bytes plus the conservatively reclaimable current database bytes to meet restoreCapacityBytes. It stops the API and admin writers, re-queries the database, and repeats the same gate. After drop/recreate and immediately before pg_restore, actual free bytes alone must meet the bound. Equality passes; one byte less fails. If the protected /var/tmp input snapshot shares the PGDATA filesystem, its space is already reflected in current free bytes and is not counted twice. A distinct snapshot filesystem is checked separately.

Interpret the receipt

State	Meaning	Action
completed	Identity reconciled to a revision above the trusted floor	Run diagnostics and client smoke, then destroy the operation key
pending_floor	Artifact restored but no external floor was supplied	Keep services stopped; resume same key with --reconcile-only
blocked	Trusted-floor conflict, overflow, identity/configuration mismatch, or other safe code	Do not retry blindly; preserve receipt and escalate
No receipt	Failure before reconciliation could write protected evidence	Inspect safe command output, state, files, and service status before any retry

Receipts are mode-0600 evidence and contain safe identifiers/digests, state, revisions, and a safe code—not the raw operation key or application content.

Restore uploaded files

File storage is a separate artifact. Keep API/workers stopped while restoring the selected archive, verify ownership/modes, and reconcile it with the database backup’s capture window. Never attach a file volume from another live deployment.

After database and file restore, test an old retained file and a new upload. Missing files must fail without exposing another tenant’s storage key or path.

Post-recovery proof

sudo nookctl diagnostics
sudo nookctl status
curl --disable --fail --silent --show-error http://127.0.0.1:3001/health/ready
curl --disable --fail --silent --show-error http://127.0.0.1:3000/login

Capture privacy-safe evidence:

• receipt is completed and the reconciled revision is above the floor;
• expected release/channel and canonical origin;
• all required containers healthy;
• safe row counts or business invariants, never message/file contents in general evidence;
• normal owner and member login;
• existing workspace/message/file accessibility;
• new message, notification, file, and voice smoke;
• tenant isolation and owner mutation still enforced;
• activation/entitlement state valid.

Only after all gates pass:

sudo shred --remove --zero /root/nook-recovery/operation-key

Retain receipts and the source backup according to the incident/retention policy. Never retain the raw key “just in case.”

Recovery anti-patterns

• Starting API/admin while the receipt is pending_floor or blocked.
• Editing deployment identity, revisions, migration tables, or entitlement rows with SQL.
• Restoring only a raw dump without its continuity manifest.
• Reusing an operation key for a different dump, deployment, or semantic request.
• Passing the operation key as a shell value, environment variable, URL, or command argument.
• Downgrading PostgreSQL or application images outside a signed N-1 release contract.
• Treating a VM snapshot as sufficient recovery evidence.
• Deleting the only pre-update backup while rollback validation is incomplete.