Impact summary
The embedding evaluation now measures six balanced held-out cases over thirty
repeated observations instead of extrapolating from two cases. The trial runner
derives telemetry from the manifest, enforces the registered $0.01 pair-cost
gate, and readiness strictly replays v3 evidence. Runtime embedding behavior and
the deterministic default are unchanged.
Ranked coupling and failure findings
- High — two-case overfitting: one representative and one adversarial query
could not support a repeatable general retrieval claim.
- High — filename authority: readiness previously counted a matching Voyage
receipt path without validating its schema, manifest digest, counts, gates, or
false-authority boundary.
- Medium — hard-coded telemetry: case, observation, provider-call, tool, and
false-friend totals could drift from the manifest while appearing valid.
Actions
| Owner | Action | Acceptance criteria | Validation |
|---|---|---|---|
| Search and relevance | Run v3 with a spend-capped Voyage key | 30 observations, exact v3 replay, no safety failure, pair cost <= $0.01, pair latency <= 3 s | `pnpm ai:embeddings:trial -- --receipt=artifacts/ai-agent-value/trials/voyage-embeddings-latest.json --readiness-receipt=artifacts/ai-agent-value/trials/voyage-embeddings-review-readiness-latest.json --private-dir=artifacts/ai-agent-value/private/voyage-embeddings-latest` |
| Evaluation operations | Assign two genuinely independent cultural-heritage reviewers | Two distinct private timed responses bound to the exact packet; no identity or candidate leakage in public artifacts | `pnpm audit:ai-agent-value:readiness` |
| AI reliability | Retain strict receipt replay and dynamic telemetry | Any count, manifest, ceiling, safety, tool, authority, or extra-key drift fails closed | `node --import tsx --test --test-concurrency=1 tests/services/voyage-embedding-trial.test.ts tests/services/ai-agent-trial-readiness.test.ts` |
Next-cycle hypothesis
Voyage v3 improves independently judged relevance by at least 10 percentage
points over BM25 without increasing reviewer effort, exceeding the cost/latency
ceilings, or selecting an adversarial false friend. The governed provider trial
and two-response blind-review aggregate falsify the hypothesis if any gate fails.