← Documentation home

Canonical Markdown source · Oct 20, 2018

Go-Live & Evidence-Pipeline Checklist

ops/go-live-checklist.md · 168 lines · SHA-256 621cddad10fb

Operationalize the live Vercel deploy: finish auth, deploy the validation

service, and start the 30-day evidence clock. Companion to

deployment.md(../deployment.md).

Production URL: `https://www.metamuseum.org` (no trailing slash). Use this

canonical URL for Vercel, GitHub OAuth, GitHub Actions evidence variables, and

local launch checks.

Verified IIIF evidence tile: `https://iiif.micr.io/RFwqO/full/max/0/default.jpg`.

---

1. Vercel — finish the app config

Project → Settings → Environment Variables (Production), then redeploy:

| Var | Value | Why |

|---|---|---|

| `DATABASE_URL` | Neon pooled string with `sslmode=verify-full` | required for Postgres DR proof |

| `METAMUSEUM_STORAGE_MODE` | `postgres` | already set |

| `AUTH_SECRET` | (already set) | — |

| `BASE_URL` | `https://www.metamuseum.org` | canonical URL, SLO target |

| `METAMUSEUM_PUBLIC_READ_BASE_URL` | `https://www.metamuseum.org` | uptime/probe target and Open Graph metadata base; must exactly match `BASE_URL` |

| `AUTH_GITHUB_ID` | GitHub OAuth app client ID | enables sign-in |

| `AUTH_GITHUB_SECRET` | GitHub OAuth app secret | (pair for the above) |

| `VALIDATION_SERVICE_URL` | `https://metamuseum-validation.onrender.com/validate` | after step 2 |

| `RECONCILIATION_SERVICE_URL` | `https://metamuseum-reconciliation.onrender.com` | optional service probe/evidence |

| `METAMUSEUM_EXTERNAL_SERVICE_PROBE_TIMEOUT_MS` | `45000` | external service cold-start window |

Also: Settings → Deployment Protection → Vercel Authentication → Disabled

(so the public can reach the read-only demo). `pnpm launch:preflight:production`

now probes `/`, `/api/health`, `/api/auth/signin`, and `/api/auth/session`

on the configured public URL and fails if Vercel Authentication or another

protection wall blocks public reads or auth routes.

It also fetches `IIIF_TILE_URL`, so k6 tile-serving evidence cannot quietly

point at a broken, private, or non-HTTP image target.

It also fails production when `BASE_URL` and

`METAMUSEUM_PUBLIC_READ_BASE_URL` are missing or not the same shareable URL,

because crawlers resolve absolute Open Graph images from that base.

After every domain change: redeploy, run

`BASE_URL=https://www.metamuseum.org METAMUSEUM_PUBLIC_READ_BASE_URL=https://www.metamuseum.org IIIF_TILE_URL=https://iiif.micr.io/RFwqO/full/max/0/default.jpg VALIDATION_SERVICE_URL=https://metamuseum-validation.onrender.com/validate RECONCILIATION_SERVICE_URL=https://metamuseum-reconciliation.onrender.com pnpm launch:preflight:production`,

then run `BASE_URL=https://www.metamuseum.org pnpm smoke:crawler-preview`.

The smoke writes `artifacts/launch/crawler-preview-latest.json` and checks

Facebook, Slack, Googlebot, LinkedIn, and X/Twitter-facing

Open Graph/Twitter/canonical metadata plus preview image fetchability. Refresh

the homepage in Facebook Sharing Debugger and the equivalent LinkedIn/X preview

tools before sharing public links.

For a full evidence refresh, prefer

`pnpm launch:evidence:production -- --base-url=https://www.metamuseum.org --public-read-base-url=https://www.metamuseum.org --iiif-tile-url=https://iiif.micr.io/RFwqO/full/max/0/default.jpg --validation-service-url=https://metamuseum-validation.onrender.com/validate --reconciliation-service-url=https://metamuseum-reconciliation.onrender.com`;

it runs hardening security/DR evidence, the preflight, public trust, crawler

preview, a11y, explore, k6, Era C exit-gate, and launch-review steps while writing

`artifacts/launch/launch-evidence-latest.json`.

If `VALIDATION_SERVICE_URL` or `RECONCILIATION_SERVICE_URL` is configured,

`pnpm launch:preflight:production` also probes the matching Render `/health`

endpoint and records status plus duration. The Render probe timeout can be long

enough to observe cold starts, but user-facing strict validation is stricter:

production preflight fails a Render-hosted `VALIDATION_SERVICE_URL` unless

`METAMUSEUM_VALIDATION_SERVICE_CAPACITY=paid` proves no-sleep capacity and the

health duration is within `METAMUSEUM_VALIDATION_SERVICE_MAX_HEALTH_MS`.

Leave the URLs unset only when the local fallback mode is intentional:

non-strict validation uses the local Linked Art profile fallback, strict

`/api/validate` still needs a running validation service, and reconciliation

stays in-process.

Worker-dependent features are scheduled drains, not persistent Vercel

processes. After any `METAMUSEUM_OUTBOX_CRON_ENABLED`,

`OUTBOX_PROJECT_TO_SOLR`, `OUTBOX_PROJECT_TO_GRAPHDB`, or

`METAMUSEUM_PUBLISH_QUEUE_CRON_ENABLED` change, open `/workers` and verify

worker lag, next cron wakeup, effective next drain, and disabled modes before

claiming projection or publishing progress. Production preflight fails when

projection targets are enabled without `CRON_SECRET` plus

`METAMUSEUM_OUTBOX_CRON_ENABLED=1`, or when live MediaWiki/Wikibase endpoints

or bot tokens are configured without `CRON_SECRET` plus

`METAMUSEUM_PUBLISH_QUEUE_CRON_ENABLED=1`.

Before setting `OUTBOX_PROJECT_TO_SOLR=1` or `OUTBOX_PROJECT_TO_GRAPHDB=1`, run

`pnpm projection:readiness` with current discovery/search/graph metrics. Treat

`portable` as keep-disabled, `watch` as prepare credentials/backfill, `enable`

as the point where Solr/GraphDB target flags may be turned on after the

scheduled outbox drain is ready, and `enabled` as the monitoring state after

the scale path is active.

GitHub OAuth app: https://github.com/settings/developers → New OAuth App →

Homepage `https://www.metamuseum.org`, callback

`https://www.metamuseum.org/api/auth/callback/github`.

---

2. Render — validation and reconciliation services

The data-quality report (validation) is the literal pilot deliverable, and the

reconciliation service backs authority lookup evidence. Both Render services

are deployed from `render.yaml`; the AG2 worker is intentionally excluded.

  1. Validation: `https://metamuseum-validation.onrender.com/health`.
  1. Reconciliation: `https://metamuseum-reconciliation.onrender.com/health`.
  1. Set `VALIDATION_SERVICE_URL=https://metamuseum-validation.onrender.com/validate`

on Vercel.

  1. Upgrade the validation service off sleeping/free-tier capacity before public

strict validation, then set `METAMUSEUM_VALIDATION_SERVICE_CAPACITY=paid`.

  1. Set `RECONCILIATION_SERVICE_URL=https://metamuseum-reconciliation.onrender.com`

on Vercel so preflight records its `/health` probe too.

Free Render web services cold-start in ~30–60s. Fine for demos and rehearsal
evidence, but public strict `/api/validate` should stay off sleeping capacity.

---

3. GitHub Actions — start the evidence clock

The nightly `era-c-exit-gate-evidence.yml` workflow reads these. Set under

Settings → Secrets and variables → Actions.

Variables (Repository variables) — minimum to start k6 + AI evidence

| Variable | Value | Unlocks |

|---|---|---|

| `METAMUSEUM_EVIDENCE_BASE_URL` | `https://www.metamuseum.org` | k6 SLO + AI-query evidence |

| `METAMUSEUM_PUBLIC_READ_BASE_URL` | `https://www.metamuseum.org` | uptime probe target |

| `METAMUSEUM_EVIDENCE_IIIF_TILE_URL` | `https://iiif.micr.io/RFwqO/full/max/0/default.jpg` | required gate for the k6 step |

With just those three, the nightly run begins retaining k6 p95 samples and AI

cost telemetry. `SPARQL_URL` defaults to

`https://www.metamuseum.org/api/getty/sparql`.

CLI:


gh variable set METAMUSEUM_EVIDENCE_BASE_URL --body "https://www.metamuseum.org"
gh variable set METAMUSEUM_PUBLIC_READ_BASE_URL --body "https://www.metamuseum.org"
gh variable set METAMUSEUM_EVIDENCE_IIIF_TILE_URL --body "https://iiif.micr.io/RFwqO/full/max/0/default.jpg"

Still gated (separate workstreams, not just config)

| Evidence | Needs | Note |

|---|---|---|

| Public-read uptime (99.9% / 30d) | `METAMUSEUM_UPTIME_PROMETHEUS_URL` (secret) + `METAMUSEUM_UPTIME_PROMQL_*` vars | requires a Prometheus/monitoring backend scraping the app — infra decision |

| Activity-feed adoption (3/3) | `METAMUSEUM_ACTIVITY_CONSUMER_IDS` | requires 3 real external consumers sending `x-linked-art-consumer-id` — partnerships, not config |

| SOTA §26 KPIs | production export to `monitoring/kpi-evidence.json` | record-enrichment + reconciliation review counts |

---

Sequence

  1. Verify Vercel env vars + GitHub OAuth + Deployment Protection state + social

preview base URL parity plus Auth.js route reachability → run

`pnpm launch:evidence:production -- --base-url=https://www.metamuseum.org --public-read-base-url=https://www.metamuseum.org --iiif-tile-url=https://iiif.micr.io/RFwqO/full/max/0/default.jpg --validation-service-url=https://metamuseum-validation.onrender.com/validate --reconciliation-service-url=https://metamuseum-reconciliation.onrender.com`

to refresh the full blocker-preserving evidence packet.

  1. If running commands manually, verify Vercel env vars + GitHub OAuth +

Deployment Protection state + social

preview base URL parity + fetchable IIIF tile target → rerun

`BASE_URL=https://www.metamuseum.org METAMUSEUM_PUBLIC_READ_BASE_URL=https://www.metamuseum.org IIIF_TILE_URL=https://iiif.micr.io/RFwqO/full/max/0/default.jpg VALIDATION_SERVICE_URL=https://metamuseum-validation.onrender.com/validate RECONCILIATION_SERVICE_URL=https://metamuseum-reconciliation.onrender.com pnpm launch:preflight:production`

and `BASE_URL=https://www.metamuseum.org pnpm smoke:crawler-preview`

before launch review. Treat `pnpm review:goals:check` as the authority for

whether the current launch evidence is green; a stale preflight note cannot

override unresolved deployment-environment or real-world blockers.

Refresh crawler previews after every domain change.

  1. Keep the deployed validation/reconciliation service URLs set on Vercel and

rerun preflight after any Render redeploy so the external service probes are

in evidence.

  1. Set the 3 GitHub evidence variables → the nightly clock starts.
  1. For any enabled projection or publishing feature, verify `/workers` shows an

effective next drain time and no blocked/dead-letter status.

  1. Decide the uptime-monitoring approach (Prometheus stack) and begin partner

outreach for the 3 activity consumers — these are the long-pole 30-day gates.