MetaMuseum
Menu
← Documentation home

Canonical Markdown source · Oct 20, 2018

Deployment — Vercel + Render

deployment.md · 230 lines · SHA-256 95bec5b7f7a2

How the Meta Museum system maps onto hosted infrastructure.

| Piece | Host | Config |

|---|---|---|

| Next.js web app | Vercel | `vercel.json` |

| Postgres (system of record) | Neon or Vercel Postgres | `DATABASE_URL` |

| Validation service (pySHACL) | Render | `render.yaml` |

| Reconciliation service (+ Redis) | Render | `render.yaml` |

| AG2 worker (optional, disabled by default) | Render | `render.yaml` |

| Background workers (outbox / publish queue) | Vercel Cron, opt-in drains | `vercel.json`, `/api/cron/*`, `/workers` |

| Solr / GraphDB (Era C search + graph) | Off until projection readiness says `enable` | skipped unless `OUTBOX_PROJECT_TO_*` is explicitly enabled |

A read-only public demo needs only Vercel + Neon. The Render validation and reconciliation services are now deployed for readiness evidence; the workers add async projection/publishing.

Run `pnpm ops:profile` for one read-only summary of the current operating mode.

It separates the required Next.js + Postgres baseline from optional Python,

AG2, Solr, GraphDB, and publication-worker services, and warns when enabled

projection lacks its readiness check or scheduled drain. Use

`pnpm ops:profile:check` in a configured production shell before changing the

runtime topology.

Run `pnpm projection:readiness` before enabling Solr/GraphDB target flags. The

portable path remains correct until record volume, search/graph p95, or active

discovery workflow counts cross the documented thresholds.

<!-- BEGIN:PROJECT_STATS -->

<!-- Generated by `pnpm docs:stats`; do not edit by hand. -->

| Generated project stats | Current value |

|---|---|

| Next.js | `16.2.12` |

| React | `19.2.4` |

| App page files | root homepage + `76` non-root page files (`77` total) |

| API route handlers | `175` `app/api` route handlers |

<!-- END:PROJECT_STATS -->

---

Part A — Next.js on Vercel

One-time setup

  1. Import the repo in Vercel. Framework auto-detects as Next.js.
  1. `vercel.json` already overrides the build:
  1. Provision Neon Postgres (or Vercel Postgres) and copy the pooled connection string.
  1. Set the env vars below (Project → Settings → Environment Variables), then deploy.
  • `buildCommand: next build` — critical. The repo's `pnpm build` runs `session:closeout:check` first, which fails 24h after the last close-out and would break every deploy. `next build` skips that guard. Local `pnpm build` is unchanged and still enforces the guard.

Vercel environment variables

| Var | Required | Value |

|---|---|---|

| `DATABASE_URL` | ✅ | Neon pooled connection string |

| `METAMUSEUM_STORAGE_MODE` | ✅ | `postgres` |

| `AUTH_SECRET` | ✅ | `openssl rand -base64 32` |

| `BASE_URL` | ✅ | shareable production URL, for example `https://www.metamuseum.org` |

| `METAMUSEUM_PUBLIC_READ_BASE_URL` | ✅ | exactly the same URL as `BASE_URL` |

| `AUTH_GITHUB_ID` / `AUTH_GITHUB_SECRET` | ⛔ optional | enables GitHub sign-in; omit for public read-only |

| `VALIDATION_SERVICE_URL` | ✅ for launch readiness | `https://metamuseum-validation.onrender.com/validate` (Part B) |

| `RECONCILIATION_SERVICE_URL` | ✅ for launch readiness | `https://metamuseum-reconciliation.onrender.com` (Part B / ops probes) |

| `METAMUSEUM_EXTERNAL_SERVICE_PROBE_TIMEOUT_MS` | ⛔ optional | default `45000`; allows Render cold starts in preflight |

| `METAMUSEUM_VALIDATION_SERVICE_CAPACITY` | ✅ if strict validation is public on Render | `paid` after upgrading the validation service |

| `METAMUSEUM_VALIDATION_SERVICE_MAX_HEALTH_MS` | ⛔ optional | default `5000`; user-facing strict-validation health budget |

| `VALIDATION_TIMEOUT_MS` | ⛔ optional | default `15000`; runtime timeout for `/api/validate` proxy calls |

| `METAMUSEUM_AG2_BRIDGE_ENABLED` | ⛔ optional | leave unset (bridge stays off) |

| `CRON_SECRET` | ✅ for Vercel Cron | long random value; Vercel sends it as `Authorization: Bearer ...` |

| `METAMUSEUM_OUTBOX_CRON_ENABLED` | ⛔ optional | `1` only when Solr/GraphDB projection is enabled |

| `METAMUSEUM_PUBLISH_QUEUE_CRON_ENABLED` | ⛔ optional | `1` only when wiki/publication queue processing is enabled |

Notes:

secrets in `docs/ops/microsoft-outreach.md`, the exact canonical Web redirect

URI, and a connected `art@sunandrainworks.com` row. Keep

`METAMUSEUM_OUTREACH_CRON_ENABLED=0` until the Graph webhook is externally

validated. Production rejects client secrets and requires the registered

certificate/private-key JWT configuration; certificate authentication still

does not clear unattended production outreach or expand the one-address

allowlist.

storage registry in `src/utils/storage.ts`, which routes those documents to

Postgres in production. `tests/quality/route-storage-write-guard.test.ts`

walks `app/api` imports and fails when a production route can reach raw

filesystem JSON writes or a JSON target not listed in

`MANAGED_STORAGE_DOCUMENT_FILES`.

must stay identical in production. Next.js resolves Open Graph image URLs

against the public-read metadata base, and `pnpm launch:preflight:production`

now fails if the two variables diverge or if the public-read base is missing.

After every domain change, redeploy, run `pnpm smoke:crawler-preview`, and

refresh the homepage in Facebook Sharing Debugger plus the equivalent

LinkedIn/X crawler preview tools.

optional, but their mode is no longer implicit. When service URLs are unset,

deployment preflight records the local fallback path: non-strict validation

falls back to the local Linked Art profile checker, strict `/api/validate`

still needs a running validation service, and the app uses in-process

deterministic reconciliation. When either URL is set, preflight probes

`/health` with `METAMUSEUM_EXTERNAL_SERVICE_PROBE_TIMEOUT_MS` and records

status plus duration for launch evidence. Health probes intentionally tolerate

cold starts, but user-facing strict `/api/validate` has a separate runtime

timeout and production preflight fails an `onrender.com` validator unless

`METAMUSEUM_VALIDATION_SERVICE_CAPACITY` proves paid/no-sleep capacity and

the `/health` duration stays within `METAMUSEUM_VALIDATION_SERVICE_MAX_HEALTH_MS`.

  • Storage self-heals on a fresh DB. Core stores (`records.ts`) catch a missing Postgres document and seed an empty `[]`, so an empty Neon DB won't crash the app.
  • Runtime is Node, not Edge for API routes (required by `pg` + Auth.js v5) — this is the App Router default; no action needed.
  • Microsoft outreach is fail-closed. Production needs the identifiers and
  • Read-only FS guard: request-path JSON writes must go through the managed
  • Social preview base URL: `BASE_URL` and `METAMUSEUM_PUBLIC_READ_BASE_URL`
  • External service mode: validation and reconciliation Render services are

---

Part B — Python services on Render

`render.yaml` is a Blueprint defining the deployed FastAPI validation and

reconciliation services plus the Redis cache used by reconciliation.

Current deployed endpoints:

  • Validation health: `https://metamuseum-validation.onrender.com/health`
  • Validation API: `https://metamuseum-validation.onrender.com/validate`
  • Reconciliation health: `https://metamuseum-reconciliation.onrender.com/health`
  • Reconciliation API base: `https://metamuseum-reconciliation.onrender.com`

Setup

  1. In Render: New → Blueprint, point at this repo. It reads `render.yaml`.
  1. Each service: own `rootDir`, `pip install -r requirements.txt`, and

`uvicorn main:APP --host 0.0.0.0 --port $PORT` (bypasses the hardcoded

`127.0.0.1` in each `__main__`). Health check: `/health`.

  1. Redis (`metamuseum-reconcile-cache`) is wired into the reconciliation

service via `RECONCILIATION_REDIS_URL` automatically.

  1. AG2 worker is optional — delete that block from `render.yaml` unless you

plan to enable the bridge.

Wire services back to Vercel

Set these URLs on Vercel so deployment preflight captures live Render health

evidence:

is upgraded off sleeping/free-tier capacity for user-facing strict validation

when operator tooling or smoke evidence should use the deployed reconciliation service

  • `VALIDATION_SERVICE_URL = https://metamuseum-validation.onrender.com/validate`
  • `METAMUSEUM_VALIDATION_SERVICE_CAPACITY = paid` after the validation service
  • `RECONCILIATION_SERVICE_URL = https://metamuseum-reconciliation.onrender.com`

The reconciliation service is invoked by operator scripts/ops tooling rather

than the Next runtime; point those at `https://metamuseum-reconciliation.onrender.com`.

Free-tier caveat: Render free web services sleep after inactivity and
cold-start in ~30–60s. Fine for a demo; upgrade validation to paid/no-sleep
capacity before exposing strict `/api/validate` to production users, then set
`METAMUSEUM_VALIDATION_SERVICE_CAPACITY=paid` and rerun preflight.

---

Part C — Background workers

Vercel cannot run persistent Node processes, so the app uses secured Vercel

Cron routes for bounded drain cycles:

| Worker | Script | Needed when |

|---|---|---|

| Outbox projector | `pnpm outbox:projector` | Solr/GraphDB projection is enabled (off in this deploy) |

| Publish queue worker | `pnpm publish:queue:worker` | Wiki/publication publishing is used |

Configured cron routes:

  • `/api/cron/outbox` every 5 minutes.
  • `/api/cron/publish` every 10 minutes.

Both routes return `cache-control: no-store`. The schedules may be deployed

safely while the workers are disabled: disabled drains return `200` no-op JSON

instead of `503`, so Vercel Observability does not count intentional idle

schedules as production errors. Once a worker drain is enabled, that route

requires `Authorization: Bearer $CRON_SECRET` before any work can run:

Tune with `METAMUSEUM_OUTBOX_CRON_LIMIT` and optional

`METAMUSEUM_OUTBOX_CRON_WORKER_ID`. Projection still requires the usual

`OUTBOX_PROJECT_TO_SOLR` / `OUTBOX_PROJECT_TO_GRAPHDB` target env vars; the

cron route will not claim events until at least one target is explicitly `1`.

Direct projector scripts use the same target gates and skip Solr/GraphDB when

those vars are unset.

batch. Tune with `METAMUSEUM_PUBLISH_QUEUE_CRON_MAX_ITEMS`. The default

queue document is `storage/publish-queue.json`, which is a managed storage

document and therefore Postgres-backed in Vercel `postgres` mode.

  • `METAMUSEUM_OUTBOX_CRON_ENABLED=1` drains one bounded outbox projector batch.
  • `METAMUSEUM_PUBLISH_QUEUE_CRON_ENABLED=1` processes a bounded publish queue

Scale enablement guard:

`/api/cron/outbox` is deployed, `CRON_SECRET` is set, and

`METAMUSEUM_OUTBOX_CRON_ENABLED=1` is ready to drain at the configured

cadence. First run `pnpm projection:readiness`; production preflight fails if

projection targets are enabled without that scheduler.

`/api/cron/publish` is deployed, `CRON_SECRET` is set, and

`METAMUSEUM_PUBLISH_QUEUE_CRON_ENABLED=1` is ready. Production preflight

fails if live publication endpoints or bot tokens are configured without the

publish queue scheduler.

  • Keep `OUTBOX_PROJECT_TO_SOLR` and `OUTBOX_PROJECT_TO_GRAPHDB` off until
  • Keep live MediaWiki/Wikibase API URLs and bot tokens out of production until

Operator status:

next drain time, and disabled projection/publish mode.

review-goals, launch, long-term, ActivityStreams, pilot, DR, and

disabled-system drill artifacts.

agent checks. Use it after worker env changes so scheduled-but-disabled

features do not look as if they are progressing.

`pnpm staging:disabled-drill:check` weekly (`20 10 1`) while AG2,

Solr/GraphDB projection, and publication drains are intentionally off; run

the same command manually after staging env changes. It writes

`artifacts/staging-disabled-systems/latest.json`, uploads timestamped run

artifacts, confirms disabled paths do not make external network calls,

mock-exercises the first enabled AG2, Solr/GraphDB projection, and

publication webhook branches, and keeps latent worker/queue regressions

visible before real enablement.

  • `/workers` renders worker lag, backlog, next Vercel Cron wakeup, effective
  • `/readiness` renders a compact editor-gated status view over the latest
  • `/api/workers/status` exposes the same no-store JSON snapshot for probes or
  • `.github/workflows/staging-disabled-systems-drill.yml` runs

For heavier projection loads, keep these cron routes disabled and run the same

scripts as a Render background worker instead:

  • `pnpm outbox:projector:once`
  • `pnpm publish:queue:worker:once`

---

Quick path to a live demo

  1. Neon DB → copy pooled `DATABASE_URL`.
  1. Vercel import → set `DATABASE_URL`, `METAMUSEUM_STORAGE_MODE=postgres`,

`AUTH_SECRET`, `BASE_URL`, `METAMUSEUM_PUBLIC_READ_BASE_URL` → deploy.

  1. Render Blueprint services are live → set `VALIDATION_SERVICE_URL` and

`RECONCILIATION_SERVICE_URL` on Vercel → redeploy.

  1. Run `BASE_URL=https://<shareable-url> METAMUSEUM_PUBLIC_READ_BASE_URL=https://<shareable-url> VALIDATION_SERVICE_URL=https://metamuseum-validation.onrender.com/validate METAMUSEUM_VALIDATION_SERVICE_CAPACITY=paid RECONCILIATION_SERVICE_URL=https://metamuseum-reconciliation.onrender.com pnpm launch:preflight:production`; it will capture configured Render service `/health` status/duration and fail if strict validation is still on sleeping capacity or outside the user-facing health budget. Then run `BASE_URL=https://<shareable-url> pnpm smoke:crawler-preview` and re-scrape social previews in crawler debug tools.
  1. Set `CRON_SECRET`; leave worker enable flags off until projection/publishing is needed.