MetaMuseum
Menu
← Documentation home

Canonical Markdown source · Oct 20, 2018

Deployment Preflight Runbook

ops/deployment-preflight.md · 198 lines · SHA-256 97db694dd88b

This runbook turns the roadmap's deployment-foundation work into a repeatable launch-readiness check. It does not replace the Era C exit gate; it verifies that the deployed environment is ready to start collecting trustworthy staging or production evidence.

Commands

Controlled public beta / staging rehearsal:


pnpm launch:preflight

Production launch review:


pnpm launch:preflight:production

One-command launch evidence refresh:


pnpm launch:evidence -- --base-url=https://<deploy-host> --iiif-tile-url=https://<iiif-tile>
pnpm launch:evidence:production -- --base-url=https://<shareable-url> --public-read-base-url=https://<shareable-url> --iiif-tile-url=https://<iiif-tile> --validation-service-url=https://metamuseum-validation.onrender.com/validate --reconciliation-service-url=https://metamuseum-reconciliation.onrender.com

Artifacts:

  • `artifacts/launch/deployment-preflight-latest.json`
  • `artifacts/launch/runs/deployment-preflight-<timestamp>.json`

For production-target runs, localhost/private deployment targets are treated as

operator diagnostics. The command still writes the timestamped run artifact so

the failure can be inspected, but it does not replace

`deployment-preflight-latest.json` when production URL checks show `localhost`,

`127.0.0.1`, or `::1`.

The preflight, storage export, DR drill, launch review, and staging explore-smoke scripts load local `.env*` files for operator convenience, but explicit shell or CI environment variables always win.

For production evidence runs from a workstation that also has staging-only

values such as `METAMUSEUM_TEST_ROLE_OVERRIDE_TOKEN`, set

`METAMUSEUM_SKIP_ENV_FILES=1` and provide the required production variables

explicitly so local `.env*` files cannot pollute the preflight artifact.

What It Checks

the configured public base URL, so auth readiness is live-route evidence

rather than only secret presence.

`METAMUSEUM_SECRET_ROTATION_CONFIRMED_AT` must be current, and

`METAMUSEUM_SECRET_ROTATION_KEYS` must name the launch-critical keys rotated

after any exposure.

checked `.env*` paths, tracked sensitive filenames, and matching commit hashes;

production fails if sensitive env files are tracked, still appear in history,

or the git probe cannot run.

`METAMUSEUM_PUBLIC_READ_BASE_URL` set to the same public URL, so absolute

Open Graph image URLs do not point crawlers at a protected preview host.

Equal localhost or non-HTTPS values are still production failures because

crawler debug tools cannot fetch private origins.

preflight unless `CRON_SECRET` and `METAMUSEUM_OUTBOX_CRON_ENABLED=1` are set,

and live MediaWiki/Wikibase publication endpoints or bot tokens fail unless

`CRON_SECRET` and `METAMUSEUM_PUBLISH_QUEUE_CRON_ENABLED=1` are set.

`pnpm projection:readiness` records record count, discovery-workflow count,

graph-workflow count, and search/graph p95 metrics as `portable`, `watch`, or

`enable` / `enabled` evidence.

the local fallback/in-process behavior, while configured URLs must have a

`/health` probe with status and duration so cold starts and timeouts appear

in launch evidence. The long external-service probe timeout is only evidence

capture; production strict `/api/validate` is closed by default unless

`METAMUSEUM_STRICT_VALIDATION_PUBLIC=1` is set. A public Render-hosted strict

validator also requires paid/no-sleep capacity and a passing user-facing

health budget.

production probe-based uptime must target the final public HTTPS deployment

URL, not localhost or another private/non-HTTPS origin.

  • Storage resolves to Postgres-backed mode via `DATABASE_URL` and `METAMUSEUM_STORAGE_MODE`.
  • Database SSL mode is pinned to `sslmode=verify-full`; staging warns on `require`/missing values, and production fails until full verification is explicit.
  • Database authentication is live-probed with `SELECT 1`; production fails `runtime.databaseConnection` if the rotated `DATABASE_URL` cannot authenticate from the preflight environment, so a parseable URL cannot hide a stale Vercel or CI secret.
  • Active Vercel deployment metadata is captured without secret values; production fails `runtime.deploymentActivation` when the active deployment is missing, not `READY`, unparsable, or older than `METAMUSEUM_SECRET_ROTATION_CONFIRMED_AT`, so updating Vercel env vars without redeploying cannot satisfy strict launch evidence.
  • Auth.js launch secrets are present and not using the development fallback.
  • GitHub OAuth credentials are present for sign-in; failures name the exact missing variable without exposing secret values.
  • Auth.js deployed routes answer `/api/auth/signin` and `/api/auth/session` on
  • Secret rotation evidence is recorded without secret values:
  • Sensitive env-file history is checked from git: preflight records only the
  • Public base URL is configured for smoke, uptime, and SLO commands.
  • Production social-preview metadata has both `BASE_URL` and
  • Worker scheduler readiness is explicit: projection targets fail production
  • Projection scale readiness is explicit before target flags are enabled:
  • The configured public-read deployment answers `/`, `/api/health`, and `/api/records` with a non-empty records payload, so Vercel Deployment Protection and managed-storage seed failures are not blocking launch-critical public reads.
  • Optional Render validation/reconciliation mode is explicit: unset URLs record
  • Public-read uptime source is configured through Prometheus or scheduled probes;
  • k6 has a real deployed target and an explicit, fetchable `IIIF_TILE_URL` for launch SLO evidence.
  • Latest DR drill artifact is fresh and has a verified Postgres restore rehearsal.
  • Test-role override secrets are available for staging smoke, pass a basic secret-quality check, and are absent from production.

Required Environment Surface

Minimum public-beta/prod variables:

shell/CI variables instead of local `.env*` files.

latest exposed-secret rotation.

`AUTH_SECRET`, `AUTH_GITHUB_SECRET`, `DATABASE_URL`, `ANTHROPIC_API_KEY`,

`SMITHSONIAN_API_KEY`, and `METAMUSEUM_TEST_ROLE_OVERRIDE_TOKEN` after those

values have been rotated in local, CI, and deployment secret stores.

`OUTBOX_PROJECT_TO_GRAPHDB=1`.

`METAMUSEUM_MEDIAWIKI_` or `METAMUSEUM_WIKIBASE_` endpoints/tokens are set.

  • `DATABASE_URL` with `sslmode=verify-full`
  • `METAMUSEUM_STORAGE_MODE=postgres`
  • `METAMUSEUM_SKIP_ENV_FILES=1` when running production preflight from explicit
  • `AUTH_SECRET`
  • `AUTH_GITHUB_ID`
  • `AUTH_GITHUB_SECRET`
  • `METAMUSEUM_SECRET_ROTATION_CONFIRMED_AT` as an ISO timestamp or date for the
  • `METAMUSEUM_SECRET_ROTATION_KEYS` as a comma-separated list including
  • `BASE_URL`
  • `METAMUSEUM_PUBLIC_READ_BASE_URL` set to the same public URL as `BASE_URL`
  • `METAMUSEUM_UPTIME_PROMETHEUS_URL` or scheduled probe config via `METAMUSEUM_PUBLIC_READ_BASE_URL` set to the final public HTTPS deployment URL
  • `IIIF_TILE_URL`
  • `CRON_SECRET` when projection or live publication scale flags are enabled.
  • `METAMUSEUM_OUTBOX_CRON_ENABLED=1` before `OUTBOX_PROJECT_TO_SOLR=1` or
  • `METAMUSEUM_PUBLISH_QUEUE_CRON_ENABLED=1` before live

Optional external service evidence:

intentionally public. Leave unset/off for operator-only validation; production

route handling returns `404` by default.

cold starts need a longer readiness window.

validation service off sleeping/free-tier capacity before exposing strict

`/api/validate` to production users.

user-facing strict-validation health budget. This stays separate from

`METAMUSEUM_EXTERNAL_SERVICE_PROBE_TIMEOUT_MS`, which may remain high enough

to capture cold-start evidence.

calls; strict validation returns `502` instead of waiting indefinitely.

  • `VALIDATION_SERVICE_URL=https://metamuseum-validation.onrender.com/validate` for the Render validation service `/validate` URL.
  • `METAMUSEUM_STRICT_VALIDATION_PUBLIC=1` only when strict `/api/validate` is
  • `RECONCILIATION_SERVICE_URL=https://metamuseum-reconciliation.onrender.com` for the Render reconciliation service base URL.
  • `METAMUSEUM_EXTERNAL_SERVICE_PROBE_TIMEOUT_MS` (default `45000`) when Render
  • `METAMUSEUM_VALIDATION_SERVICE_CAPACITY=paid` after upgrading the Render
  • `METAMUSEUM_VALIDATION_SERVICE_MAX_HEALTH_MS` (default `5000`) for the
  • `VALIDATION_TIMEOUT_MS` (default `15000`) for runtime `/api/validate` proxy

When `VALIDATION_SERVICE_URL` is unset, non-strict import/inspect flows use the

local Linked Art profile fallback after the localhost validation default is

unavailable; strict `/api/validate` still requires a running validation service

and is closed in production unless `METAMUSEUM_STRICT_VALIDATION_PUBLIC=1`.

When `RECONCILIATION_SERVICE_URL` is unset, the app uses its in-process

deterministic reconciliation and optional LLM tiebreaker.

Staging-only smoke support:

  • `METAMUSEUM_TEST_ROLE_OVERRIDE_TOKEN` (generated by `pnpm launch:smoke-token`; same value on server and smoke runner)

Launch Review Sequence

  1. Configure the deployed environment and secrets.
  1. For staging, run `pnpm launch:smoke-token` once per token rotation; it updates local `.env` without printing the value, so copy `METAMUSEUM_TEST_ROLE_OVERRIDE_TOKEN` from the env file into both the staging server and smoke runner.
  1. Run `pnpm storage:export:postgres` after setting `DATABASE_URL`; the exporter uses the same managed-document contract as `src/utils/storage.ts`.
  1. Run `pnpm dr:drill` against the configured storage mode.
  1. Before enabling Solr/GraphDB projection flags, run `pnpm projection:readiness -- --active-discovery-workflows=<count> --active-graph-workflows=<count> --search-p95-ms=<ms> --graph-p95-ms=<ms>`.
  1. Before production preflight, finish the credential-rotation runbook and keep

the non-secret rotation stamp/key list in the deployment env. Then run

`pnpm launch:preflight` or `pnpm launch:preflight:production`; the command

probes the configured public URL to verify `/`, `/api/health`,

`/api/records`, `/api/auth/signin`, and `/api/auth/session` are reachable

without Vercel Authentication, fetches the configured `IIIF_TILE_URL`, probes configured

Render validation/reconciliation services at `/health`, checks sensitive

`.env*` tracking/history via git without printing secret values, captures

active Vercel deployment id/state/createdAt metadata, and, for production,

fails if `BASE_URL` and `METAMUSEUM_PUBLIC_READ_BASE_URL` are not the same

shareable public HTTPS URL or the active deployment predates the latest

launch-critical secret rotation. If `VALIDATION_SERVICE_URL` points at

Render for production, the same preflight also fails sleeping/free-tier

validation capacity or `/health` durations above the user-facing strict

validation budget even when the long cold-start probe eventually passes.

  1. Prefer the evidence runner when refreshing the full packet:
  1. Or run public smoke checks manually:
  1. Collect performance and exit-gate evidence if not using the runner:
  1. Generate the launch decision packet if not using the runner:
  1. After every domain change, redeploy and refresh the homepage in Facebook

Sharing Debugger plus the equivalent LinkedIn/X preview tools before

circulating public links.

  • `pnpm launch:evidence -- --base-url=https://<deploy-host> --iiif-tile-url=https://<iiif-tile>`
  • `pnpm launch:evidence:production -- --base-url=https://<shareable-url> --public-read-base-url=https://<shareable-url> --iiif-tile-url=https://<iiif-tile> --validation-service-url=https://metamuseum-validation.onrender.com/validate --reconciliation-service-url=https://metamuseum-reconciliation.onrender.com`
  • The runner writes `artifacts/launch/launch-evidence-latest.json`, keeps going after individual command failures, and records which evidence steps still block launch.
  • `BASE_URL=https://<deploy-host> pnpm smoke:public-trust`
  • `BASE_URL=https://<deploy-host> pnpm smoke:crawler-preview` writes `artifacts/launch/crawler-preview-latest.json` and verifies Facebook, Slack, Googlebot, LinkedIn, and X/Twitter-facing preview metadata plus image fetchability.
  • `BASE_URL=https://<deploy-host> pnpm a11y:check` writes `artifacts/launch/a11y-latest.json`.
  • `BASE_URL=https://<deploy-host> pnpm smoke:explore:matrix` writes `artifacts/launch/explore-smoke-latest.json` for staging researcher flow.
  • `BASE_URL=https://<deploy-host> IIIF_TILE_URL=https://iiif.micr.io/RFwqO/full/max/0/default.jpg pnpm k6:slo`
  • `pnpm era-c:exit-gate:evidence`
  • `pnpm launch:review` for controlled beta / staging.
  • `pnpm launch:beta:readiness` for the controlled beta go/no-go summary.
  • `pnpm launch:review:production` for fail-fast production review.

Pass Criteria

  • Staging can proceed with warnings only when the warnings are explicitly accepted for rehearsal.
  • Production must have `status: "pass"` in `deployment-preflight-latest.json`.
  • Passing deployment preflight means the environment is ready to collect evidence; it does not mean the 30-day SLO, uptime, adoption, or KPI gates have passed.
  • Passing launch review requires fresh launch evidence across hardening security/DR, preflight, exit-gate, crawler preview, public-trust, a11y, and explore import checks; `pnpm launch:evidence` refreshes that packet and records any remaining blockers. See `docs/ops/launch-review.md`.
  • Passing controlled beta readiness means staging can go live with only accepted evidence-collection warnings; production still requires `pnpm launch:review:production`.