Operate and update Nook

Daily and weekly operating rhythm

Keep the routine small enough that it actually happens.

Frequency	Check
Continuous	External HTTPS/API health, certificate expiry, disk pressure, PostgreSQL availability, backup-job success
Daily	Failed mail/gateway delivery, critical release notice, unexpected container restart, storage growth
Weekly	nookctl diagnostics, off-host backup inventory, restore-floor evidence, security updates, operator access review
Before every Nook update	Fresh continuity backup, signed artifacts, supported source version, release notes, downtime notice, rollback point
Quarterly or after architecture change	Full isolated restore drill and client login/message/file/voice smoke

Health and diagnostics

Start with privacy-minimized commands:

sudo nookctl diagnostics
sudo nookctl status
sudo nookctl activation status
curl --disable --fail --silent --show-error http://127.0.0.1:3001/health/live

diagnostics reports installation identity, release, channel, platform, architecture, and whether required local files exist. It intentionally excludes configuration values, environment secrets, database content, messages, uploaded files, and log bodies.

Use bounded logs only when health is insufficient:

sudo nookctl logs --tail 200

Review output before sharing it. Generate a privacy-minimized support bundle only when necessary:

sudo nookctl support-bundle --output /var/lib/nook/support/support-bundle.tar.gz

The bundle is metadata-only by contract, but your support policy must still govern who can receive it and how long it is retained.

Back up the database

Choose a new leaf per backup job or retention unit. Do not pre-create it: nookctl creates the absent leaf with the invoking sudo operator as owner and rejects links or unsafe parents.

sudo nookctl backup \
  --output-dir /var/lib/nook/backups/manual-2026-08-04

A valid database backup is a pair:

• nook-postgres-<timestamp>.dump — PostgreSQL custom-format dump;
• the sibling .dump.nook-continuity.json — deployment identity, instance revision, configuration digest, API origin, artifact digest, source physical bytes, exact restore-capacity requirement, and manifest digest.

The backup briefly stops the API and admin writers, proves the database is read-only, measures pg_database_size immediately before and after pg_dump, and binds the larger measurement in a strict version-2 manifest. It rejects a source database above 40 GiB or a dump above 42 GiB. The bound requirement is restoreCapacityBytes = 2 × sourcePhysicalBytes + 2 GiB, up to 82 GiB. The tool restores the prior supported database setting and exactly the services that were running before it publishes either output file. A cleanup or restart failure leaves no published pair. Before stopping writers it fsyncs a root-private recovery journal, masks the exact nookctl-managed Caddy ingress, and sets API/admin restart policies to no. A crash or reboot therefore cannot publish or expose a half-quiesced backup; the next lifecycle command reconciles the bound state. Capture uses the fixed, marker-locked /var/tmp/nook-continuity-backup.stage slot. nookctl descriptor-reclaims only its exact dead inode through a fixed quarantine; it never glob-deletes temporary paths or follows TMPDIR.

Retain and transfer the pair together. A raw pg_dump, Docker volume copy, or VM snapshot is not a supported substitute for continuity recovery.

Uploaded files require their own backup. Configuration and permitted key material require a separate protected operator copy. Database and file backup capture is not atomic, so document the acceptable consistency window.

Verify backups before relying on them

Verification checks the exact artifact/manifest binding without serving restored data:

sudo nookctl restore \
  --artifact /absolute/path/to/nook-postgres-<timestamp>.dump \
  --manifest /absolute/path/to/nook-postgres-<timestamp>.dump.nook-continuity.json \
  --operation-key-file /root/nook-recovery/verification-operation-key \
  --expected-deployment-id <deployment-id-from-manifest> \
  --receipt /root/nook-recovery/verification-receipt.json \
  --confirm-deployment-id <deployment-id-from-manifest> \
  --verify-only

Create the operation-key file with mode 0600; never pass its value as a command argument. A verify result proves integrity, not that the organization can meet its RTO. Perform an isolated restore drill as described in Recover and roll back.

A destructive restore keeps Caddy masked and API/admin restart-fenced under a fsynced transaction journal through database restore, reconciliation, and migrations. It restores public ingress only after Docker health, local API /health/ready, local admin reachability, and the exact completed receipt pass within the bounded 120-second readiness window. status and diagnostics report a privacy-safe operation and phase after interruption; repeat only the exact bound restore command. Input bytes live in the fixed protected /var/tmp/nook-continuity-restore.stage/work snapshot. Before the WAL exists, a dead copy is safely reclaimed; after the WAL exists, only the same immutable operation intent may reopen those exact bytes. Changed inputs or intent remain fenced. Lifecycle rollback additionally restart-fences the complete fixed Compose project and starts only the exact target PostgreSQL container, with restart policy no, while API, admin, Redis, and Caddy remain unavailable. A durable data-ready phase and exact project/release/health proof precede the nested restore. If both lifecycle and restore journals exist, only an exact joined transaction may be resumed, and only with nookctl lifecycle-recover.

Understand release channels

Nook publishes independently signed stable, lts, and security channel manifests. Each binds:

• a monotonic sequence number;
• the exact release version and immutable image digests;
• supported source versions (normally N and N-1 only);
• minimum nookctl version;
• artifact hashes and sizes;
• publication and expiry timestamps;
• the signing-key fingerprint.

Stable/LTS manifests expire within 36 days; security manifests within 8 days. nookctl update rejects an expired or replayed manifest, wrong channel, unsupported source version, mutable image, candidate trust-anchor mismatch, or release/image mismatch.

Release notification email

Eligible Customer Portal primary owners and owners receive one concise email after a signed release is published. Deployment admins can opt in. The email states channel, source-version compatibility, expected downtime, affected Portal organizations, and notes. Its command block is limited to safe local preflight commands for diagnostics, status, and a continuity backup, followed by a call to continue in the authenticated Customer Portal. The email deliberately contains no update arguments, download command, grant, checksum, credential, or attached release.

The email cannot start an update. Always open the Customer Portal or official release source independently; do not install an attachment or follow an unexpected credential prompt.

Routine and important mail preferences live under Support & privacy. Critical security or compatibility mail cannot be disabled for eligible owner roles.

Pre-update checklist

• Current release is listed in supportedFromVersions.
• nookctl meets minimumNookctlVersion.
• Release/channel keys were verified through independent trust paths.
• Release notes, downtime, schema compatibility, and known issues were reviewed.
• A fresh continuity backup pair exists and has an off-host copy.
• Current configuration and permitted secret/key backups are protected.
• Current greatest publicly observed instance revision is retained outside the backup.
• Users received the maintenance window.
• An operator is available through post-update client smoke.

Run a signed update

First create an explicit operator backup and confirm the current deployment is healthy:

sudo nookctl diagnostics
sudo nookctl backup --output-dir /var/lib/nook/backups/manual-pre-update
sudo nookctl status

Then open authenticated Customer Portal, choose the target connected release, and compare the installed channel public-key fingerprint with the published trust page. Choose Request official artifact, copy the secure update command, then copy the one-use grant separately and paste it only at the command’s hidden prompt. The signed installer bundle already contains the channel manifest, detached signature, and candidate public key; do not download or place those inputs independently. Do not extract the bundle or run its package/helper from a user-writable directory.

The Portal command uses the same root-owned, disk-backed /var/tmp/nook-installer.* bootstrap as initial installation. It streams the exact Portal-authorized HTTPS response into a new exclusive-create, no-follow file, verifies its byte count and SHA-256 during that stream, safely extracts it, exact-pins the bundled release and channel keys to the installed trust anchors, verifies the signed release and package binding, and executes the package and image helper only from a second authenticated root snapshot. The protected script then calls nookctl update with the real --release-dir, --release-public-key, --channel-manifest, --channel-signature, --channel-public-key, --channel, and --image-env-file inputs. Update email contains only the safe diagnostics/backup/status preflight and a Portal call to action; email is never an update authority.

Connected update capacity does not reuse the clean-install 40 GiB formula. Before download, the protected command asks the installed host to compute exact live headroom on the state, continuity backup, release, /var/tmp, Docker, and live PostgreSQL PGDATA storage destinations. It measures their real filesystems, groups only destinations with the same device identity, reserves restoreCapacityBytes - conservative current database bytes on the PGDATA device for automatic rollback, and includes protected staging equal to four times the exact Portal artifact bytes. The bounded authority supports the declared 40 GiB database-size ceiling: its base plan is capped at 192 GiB, with at most 32 GiB of separately bound staging overhead, rather than silently applying a smaller database limit. After the archive inventory is verified, it repeats the host check with staging equal to twice the verified expanded size. The authenticated candidate then measures its exact release snapshot, expanded release archive, remaining unique image layers, and continuity headroom and must pass a final filesystem-aware capacity gate before package, image, or lifecycle mutation.

nookctl stages the verified override with that release and uses it after the base configuration for backup, restore, update recovery, rollback, and all other Compose paths. Image-creating commands use --pull never; a missing or changed binding fails closed.

The release and channel public keys serve separate trust purposes, but the connected lifecycle exact-pins both anchors at initial installation. Every update must present byte-for-byte identical anchors and fails closed on any mismatch; there is no in-place key-rotation workflow today. Future rotation requires an explicit authenticated transition design, recovery and rollback rules, and release evidence before customer use. The update stages the candidate, durably masks managed Caddy, restart-fences writers, creates a continuity backup without reopening them, stops the old release, switches immutable image/config references, starts the new release, and checks API readiness plus local admin reachability. If a health check fails, it restores the prior state and removes candidate staging; an ambiguous recovery remains fail closed. Each retry creates a distinct rollback directory. After an N-1 rollback, a later forward update re-verifies and atomically restages the retained non-active candidate instead of requiring an operator to delete release files manually.

Post-update verification

sudo nookctl diagnostics
sudo nookctl status
sudo nookctl ingress status
curl --disable --fail --silent --show-error http://127.0.0.1:3001/health/live

Then test with a non-operator account:

1. log in through the normal client origin;
2. load an existing workspace/channel and old messages;
3. send a new message and observe realtime delivery;
4. verify notification delivery;
5. upload and download a small non-sensitive test file;
6. join a voice room through the real firewall;
7. verify local administration and invitation delivery.

Do not delete the automatic pre-update backup until the release is accepted and retention policy permits it. If verification fails and the release supports N-1 rollback, follow Recover and roll back—do not downgrade database images manually.

Capacity and patching

• Alert before disk exhaustion; PostgreSQL, uploads, Docker images, and backups compete for space.
• Apply host security patches under your operating-system policy. Reboot only with a verified Nook backup and maintenance window.
• Use only release-pinned database/Redis images. An independent major-version upgrade is unsupported.
• Keep nookctl updated from the release-bound Debian package shown in Customer Portal; it may be newer than the active Nook release because lifecycle fixes must remain available during rollback. The connected installer does not add an APT repository. Do not assume packages.cloudylake.io is configured until a production repository is explicitly published and independently verified.
• Retain the active release and at most one explicitly selected rollback artifact after acceptance; remove superseded test packages and transfer bundles.

When to stop and escalate

Keep API/admin unavailable and inspect receipts when:

• artifact or signature verification fails;
• the channel sequence goes backwards or the manifest is expired;
• backup and manifest disagree;
• deployment identity or canonical origin differs;
• a restore is pending_floor or blocked;
• migrations fail;
• health is green locally but normal clients cannot authenticate or maintain tenant isolation.

Fail-closed is safer than serving a deployment whose identity or authorization generation is stale.