← Documentation home

Canonical Markdown source · Oct 20, 2018

Research evidence operations

research-commons/evidence-operations.md · 124 lines · SHA-256 61e959fca1aa

`pnpm research:evidence-operations` converts the latest A+ readiness report and

retained external-evidence artifacts into one tamper-evident operating report.

It does not collect external evidence or perform any external action.

Eight requirements, seven workstreams

The report covers open Research Commons production proof, production AI evaluation, external investigation review,

independent reviewers, measured improvement, external impact, attributable

acquisition, and positive net income. External review and reviewer qualification

share the independent-validation workstream, producing seven exact commands in

dependency-aware scorecard order.

The open-release lane begins with `pnpm research:open-release-handoff`. That safe

preparation step emits an exact-release, nineteen-probe packet with a blank human

decision and no network or deployment authority. The packet itself contains the

post-approval capture and import commands. A verified retained capture advances the

operations lane directly to `pnpm research:open-release-evidence`; missing or invalid

capture evidence returns to handoff preparation instead of suggesting an unapproved probe.

The production-AI command is also dependency-aware within its workstream. When

the explicitly consented collection lane is inactive, the queue starts with

`pnpm research:ai-question-collection-readiness`. That command generates the

exact digest-bound human-review packet but neither approves nor enables the

lane. Once collection is independently approved, activated, and has genuine

consented traffic, a blocked query-log candidate selects

`pnpm research:ai-question-attestation`; a verified 50-row candidate selects

`pnpm research:ai-question-export`. If no query-log readiness candidate exists

after that prerequisite, the queue starts with

`pnpm research:ai-manifest-prepare`, advances to

`pnpm research:ai-production-run-kit` only after a verified candidate worksheet

is ready for separate human labeling and approval, then advances to

`pnpm research:ai-production-evidence` only after a verified ready run kit, and

advances to `pnpm research:ai-production-eval` only after a verified ready

assembly. Missing, blocked, or digest-invalid intermediate artifacts fail back

to the earliest safe command. This progression performs no model call.

When a retained query-log attestation candidate exists, the production lane

starts even earlier. A blocked or digest-invalid candidate selects

`pnpm research:ai-question-attestation`; a verified candidate with at least 50

structurally eligible rows selects `pnpm research:ai-question-export`. Only

after a separately completed human attestation and private export does manifest

preparation apply. This stage resolution prevents the cockpit from suggesting a

50-question input that current aggregate evidence shows does not yet exist.

For each requirement it records:

genuine evidence.

  • readiness status and evidence state;
  • artifact path and capture time when present;
  • age against the requirement's 35-day or 365-day freshness window;
  • integrity-verification result;
  • retained blockers;
  • stable command ID, complete non-executing invocation template, and required

Invocation templates include every required direct input placeholder for the

current stage, such as `--input`, `--manifest`, `--bundle`, `--evidence`,

provider reports, qualified actions, settlements, and observed costs. They are

instructions only: the report never interpolates secrets, reads unapproved

external files, or executes the command.

The JSON report also gives every flag a structured contract: JSON file, HTTPS

URL, SHA-256, pseudonymous code, or ISO time; whether the value is private; and

whether it requires human attestation or may be machine-derived. Bounded agents

can therefore prepare local tasks without inferring authority or moving private

evidence into public artifacts.

An automated contract-drift test ties every operations command to its exact

`package.json` script and implementation file, confirms that the implementation

parses every declared flag, and requires that file to be covered by the

canonical research-release manifest. A renamed command, missing parser, or

unbound implementation therefore fails the release gate before an operator or

researcher receives stale instructions.

The editor-only organic-income cockpit is the human-facing consumer of this

artifact. It shows prioritized invocation templates and each input's privacy

and attestation status only after the report digest verifies. If the artifact is

absent, modified, or malformed, the cockpit reports that state and withholds all

commands. It never substitutes an invocation inferred from application state.

The report now retains the exact readiness requirements, lane snapshots, and

stage-specific command overrides used to build it. Verification reconstructs all

evidence states, freshness calculations, 30/60/90-day windows, grouped workstreams,

priorities, invocations, input contracts, alerts, and false authority flags before

requiring canonical equality. Rehashing a different command or priority therefore

cannot misroute the cockpit even when the outer digest is recomputed.

The acquisition workstream is prerequisite-aware as well. While the

question-led content contract lacks exact human editorial approval, it selects

`pnpm research:question-publication-readiness` and reports the six-guide review

blockers instead of prematurely asking for Search Console or campaign exports.

After approval, it returns to `pnpm research:acquisition-evidence`. Neither

command publishes, deploys, sends a campaign, or treats traffic as validation.

Evidence states are `satisfied`, `missing`, `blocked`, `invalid`, `stale`, or

`ready-but-unaccepted`. Missing is not rendered as zero, a blocked artifact is

not presented as absent, and a digest-invalid or future-dated artifact is never

trusted.

Blocked baseline, impact, acquisition, and economics writers use the same

canonical integrity finalizer as passing artifacts. Missing real inputs now

produce a tamper-evident `blocked` state rather than an unsigned fallback that

the operations report must classify as `invalid`.

Reporting windows

Every run deterministically emits exact 30-, 60-, and 90-day windows ending at

the evaluation timestamp. This lets provider exports and human evidence be

requested for one unambiguous interval rather than an informal “last month.”

The JSON and Markdown artifacts are retained under

`artifacts/research-a-plus/operations/`. Use `--now=<ISO timestamp>` for a

reproducible historical report.

Authority boundary

The operations report schedules evidence work only. Every workstream says that

external evidence is required and that side effects are unauthorized. It cannot

run a production model, recruit or contact a participant, approve a reviewer,

publish, send email, activate payments, create a settlement, or manufacture a

passing artifact. `--check` intentionally remains non-zero until all seven

external requirements are genuinely satisfied.