Developers

The Bench API

JSON under /api/v1, with application/problem+json errors. A person is the bench_at session cookie or a Network user token as a Bearer; a service sends a client-credentials token with bench.* capabilities.

Read

GET /api/v1/kinds: The fifteen benchmark kinds.
GET /api/v1/kinds/:kindId: One kind and its entry schema and metrics.
GET /api/v1/tasks/:id: One task.
GET /api/v1/rounds/:id: One round with entries; entries stay blind until the caller has judged or started the round.
GET /api/v1/boards/:kindId?type=elo|pareto|cost: A rendered board.
GET /api/v1/models/:model: One model across every kind.
GET /api/v1/datasets/:kindId.jsonl: The open dataset (prompts, entries, judgements without voter ids, ratings).

Write

POST /api/v1/kinds/:kindId/tasks: { prompt, task_class?, public? }. A person or a service with bench.task.create.
POST /api/v1/tasks/:id/rounds: { offers?, seed? }. Pins a reproducible round; the job generates entries.
POST /api/v1/rounds/:id/entries: A room submits outside entries (a game match, a test-suite run) with a service token carrying bench.entry.create.
GET /api/v1/judgements/next?kind=&judge=: The next blind pair for this person.
POST /api/v1/rounds/:id/judgements: { judge?, pair?, outcome?, score?, vote_counts? }. Human judges are people; automatic/test/game judges and audience_poll are services with bench.judgement.cast. An audience_poll supplies { left, right, tie, voters }; it changes Elo as one game regardless of audience size. A zero-voter poll is recorded without changing ratings.
POST /api/v1/evals: Create a private task suite (never in the public dataset).
POST /api/v1/evals/:id/run: Run a private eval against chosen offers, under the free daily cap.
GET /api/v1/evals/:id/results: A private eval and its runs.

MCP

POST /mcp exposes bench.list_kinds, bench.start_round, bench.next_pair, bench.cast_judgement, bench.get_board and bench.run_eval.