Automatic runtime validation

Automatic execution is disabled by default. An organization administrator approves the exact digest-pinned image and command in Settings → Runtime & Approval. Organization defaults apply to teams and repositories until an explicit override is saved. Sync from parent removes the override. Policies are stored in the application database; pull request code cannot change them.

Deploy the worker separately from Next.js/Convex, on an isolated Docker host. Use Node.js 24, pnpm 11, Docker, and this checkout. Create a named worker credential in the admin settings page, copy it once to the worker's private environment, and start:

# Set OPENREVIEW_WORKER_TOKEN through your host's secret manager.
pnpm runtime:worker --url https://<deployment>.convex.site

For the local backend, use http://127.0.0.1:3211. --once sends a heartbeat and processes at most one queued job. Production uses HTTPS and refuses redirects. Use the Linux systemd unit on an isolated Docker host. Transient failures exit 1 for restart-on-failure; explicitly rejected worker credentials and invalid URL/token configuration exit 78, which the unit excludes from restart. Generic job/lease failures do not classify the workspace credential as revoked. An active command remains bounded; revocation never extends its lease or permits attaching evidence. Credentials expire after 90 days, are stored as hashes, and can be revoked in the dashboard. They authorize only this workspace's queued runtime jobs and their leased source snapshots.

A recent worker heartbeat is required before queuing a runtime check. Without one, the review continues with a visible unavailable runtime job and no execution evidence. With a connected worker, the review waits for the runtime result. During source download, execution and result reporting, the worker sends a heartbeat every 30 seconds through its current authenticated job lease. Heartbeats update worker availability without claiming another job or extending the execution deadline. Expired/revoked credentials and expired/replayed leases cannot refresh liveness. Heartbeat transport failures are logged and retried while the local command remains bounded; they cannot invent a passing result. Its exact revision source is downloaded through an authenticated endpoint and streamed into a container; the worker never extracts repository files on its host.

The container receives no application environment variables or credentials. It has no network, no host mounts, a read-only root filesystem, an unprivileged user, no capabilities, restricted process count, one CPU, 512 MB memory, bounded writable tmpfs and output, and a maximum 600-second command timeout. Dependencies must be preinstalled in the administrator-approved image; online package installation cannot work inside the container. The worker's Docker access is powerful: isolate that host from your production services and credentials.

Durable jobs track source preparation, queuing, leases, result status, exact revision, policy, image, command and redacted logs. Source archives are capped at 20 MB compressed and deleted after completion. Expired leases and interrupted source/worker jobs recover into visible failure states rather than invented results. Start/reconnect a worker and re-review to retry. Replayed leases, another workspace's credential, a changed policy, and stale provider revisions cannot attach current execution evidence. Stale results retain logs on the original job but do not become evidence for the current review.

Runtime records distinguish command started from setup/extraction failure. A nonce emitted by the fixed container wrapper records command startup. Exit status alone cannot prove that tests ran. Legacy evidence without a startup record is rendered as execution unconfirmed. A failed test is execution evidence, but never a passing check. Summary validation receives the complete recorded evidence rather than a startup boolean; failures, timeouts and missing exit codes cannot authorize passing-test prose. The deterministic execution section renders an unknown exit code as outcome unavailable. The public review API and direct review-request mutation reject caller-supplied runtime results; only authorized worker reports attach hosted evidence. CLI evidence files remain explicitly operator-supplied local context.

The prose validator checks bounded execution/outcome phrases and asks the model to repair contradictions. It is a conservative lexical guard, not a general proof of every possible natural-language claim or per-test result; the recorded command status and logs remain authoritative.

GitHub Cloud/Enterprise, GitLab and Gitea supply native exact-commit tar.gz archives. Origin also implements full SHA-1/SHA-256 commit-pinned archives with native repository/commit checks and single signed redirects to operator-approved HTTPS hosts; App credentials are never sent to the download host. Its native-response/durable-source tests pass, but no native installation/archive-layout/Docker execution acceptance is claimed. See Origin setup for the required CDN allowlist and current beta format limitation. Bitbucket Data Center now uses its native commit-pinned archive API with repository/publisher identity checks, a fixed extraction prefix and bounded authenticated transport; its native-response/durable-source tests pass, but live native archive execution remains unverified. Perforce uses a separate raw archive of the selected submitted baseline overlaid with the exact archived shelf. It retrieves every enumerated regular file, including binaries omitted from review context, verifies native byte digests, retains executable modes and additions/deletions, and preserves depot-relative paths. An approved command must select its working directory, for example cd depot/main && node --test. The Perforce archive is limited to 20 MB uncompressed, 10,000 files and USTAR paths; unsupported file semantics, missing metadata, changed project views or unreadable files fail visibly. P4 protections may hide inventory, so neither archive completeness nor a full native client mapping is claimed. Baseline, connection, requester access and polling authority are rechecked before source access and evidence attachment. Providers without an implemented archive path fail visibly. Further provider and hosted supervisor acceptance remains outstanding. Runtime policies can require a passing job before automatic approval; the runtime worker itself publishes no approval or merge.

Verification includes worker/repository authorization, scope inheritance, duplicate claims, policy changes, revocation, replay, stale revisions and timeout recovery. Real Docker checks ran the seeded PR #4 commit, detected the integer-cents defect, confirmed no application secrets/root writes, and verified successful command startup, timeout, and failed archive extraction. A persistent local worker also completed the actual durable queue → exact GitHub archive → Docker → report → resumed review → native publication flow on PR #4. Its test failed at the reviewed commit (5 cents instead of 500), with startup confirmed and exit 1; the review rendered recorded failure evidence. A temporary scoped approval policy then produced a blocked durable approval job containing the runtime failure reason, unresolved finding and excessive risk. Native GitHub reviews were unchanged. The temporary worker was revoked and both repository policies restored. Successful runtime results through the durable queue, hosted supervisor recovery, and positive native approval/dismissal remain unverified. Private artifacts are retained under ignored output/setup/runtime-live-acceptance.json and runtime-approval-acceptance.json.

A real 130-second validation also verified busy-worker liveness beyond the 120-second connection threshold and completed the same report/review/publication path. The job retained its fixed deadline. The temporary worker credential and policy were cleaned up afterward; native GitHub publication was read back at the exact commit. This proves local long-running liveness, not hosted supervisor recovery.

Bitbucket Cloud now builds exact-revision source from its unfiltered native tree/raw file API, preserving binary bytes and executable attributes within the shared regular-file USTAR limits. It rejects LFS redirects, symlinks, subrepositories, unknown metadata and incomplete/oversized inventories. Native-response/durable worker-source tests pass; live Forge installation and Docker source/execution acceptance remain pending. See Bitbucket Cloud runtime.

Generated reproduction tests

Opt in per organization, team or repository in the same runtime policy: check Generate reproduction tests and set a test command template containing {test} exactly once, for example node --test {test} or pytest -q {test}. The image must be digest-pinned, as for the approved command.

After a review is verified and saved, OpenReview asks the model to draft one minimal test for each of up to three verified findings (highest severity first; deterministic security findings are skipped). A draft must be a new, test-named, repository-relative file without .., must not print the runtime start marker and must not claim results. The review then waits for a tests runtime job on a connected worker. The worker downloads the exact-revision snapshot once and runs each test separately on a fresh copy: the fixed container wrapper extracts the source, decodes the test file from a base64 argument (never interpolated as shell text) and runs the template with {test} replaced by the quoted path. Every isolation limit above applies (no network, read-only root, 512 MB, one CPU, the policy timeout per test). The image needs base64 and dirname, which BusyBox and coreutils provide.

Results attach to their findings as recorded evidence: the outcome follows the exit status, and the comment shows the command, the test code and the trimmed, redacted output in a collapsed block. Policy:

• Started and failed (non-zero exit): the defect was reproduced; confidence rises by 0.05, to at most 0.99.
• Started and passed (exit 0): not reproduced; confidence falls by 0.15 but never below the repository's minimum confidence, so the finding stays, with the passing test shown.
• Timed out, crashed before a recorded exit, did not start, or no result: inconclusive; confidence is unchanged.

Generated tests are never committed, pushed or written to the repository: they exist only in the job record and inside the disposable container. Without a connected worker or the opt-in, reviews publish immediately as before. A changed policy or revision while the job runs cancels the results. A live run on a real repository is still a human step (see the parity ledger).

Linux supervisor deployment

Place a reviewed checkout and its frozen pnpm dependencies at /opt/openreview, readable by a dedicated openreview-runtime account. Install Node 24 and Docker, pre-pull each approved digest-pinned runtime image, and verify /usr/bin/node, /usr/bin/docker and the standard /run/docker.sock path on that host. Adjust the unit for different verified executable/socket paths; rootless Docker hosts need their own configuration and acceptance. The dedicated account needs access to this isolated Docker daemon through its docker supplementary group.

Copy deploy/runtime-worker.env.example to /etc/openreview/runtime-worker.env, owned by root with mode 0600. Set OPENREVIEW_WORKER_URL to the HTTPS Convex HTTP origin and supply a newly approved named workspace worker credential through the host secret manager. Do not place it in the service unit, command line, Git or PR code. This setup does not enable any runtime policy.

Install deploy/openreview-runtime-worker.service into the host's system service directory. Before starting, run systemd-analyze verify on that installed unit, then systemctl daemon-reload and systemctl enable --now openreview-runtime-worker. Use systemctl status and journalctl -u openreview-runtime-worker to inspect status. Logs contain job IDs/revisions and generic request failures, not the worker token. Replace expired/revoked credentials through admin setup, then explicitly restart the service. The checked-in unit's RestartPreventExitStatus=78 prevents automatic restart after terminal credential/configuration rejection; see the upstream systemd service documentation.

Acceptance must exercise an actual host restart, service failure/restart, credential revocation remaining stopped, and an exact-revision Docker job followed by durable result readback. The service template and process exit tests do not establish those live host behaviors. Vercel/Convex hosting does not itself provision this separate Docker host. No system service or persistent worker credential has been installed by adding this template.

On October 4, 2026, the deployed backend and an actual CLI process verified rejection of the retained revoked fixture worker: HTTP 401 / worker_credential_rejected and exit 78, without token output. Unknown credentials behave the same; stale leases remain generic job denials. All 299 tests and lint/typecheck/build pass. Evidence: output/setup/runtime-supervisor-live-evidence.json. No worker/policy was created or enabled. Linux service/reboot/restart recovery remains unverified until the unit is installed and tested on the isolated host.