Create
string
required
The harness under test.
string
The exact tree. Omit it and the evaluation takes the harness as it stands — but an evaluation wants one fixed tree, so naming the commit is the honest choice.
string
The tree to compare against, which fills in
comparison with improved and regressed counts.object
required
{key, version, name}. Keep the key stable across attempts at the same suite so an interrupted process can resume it; change the version when the suite’s definition changes.object
{id, version} — what judged the attempts.integer
default:"1"
Trials per case, 1–100. It is what pass@k and pass^k are computed over.
array
required
1–2000 cases of
{key, name, input, expected}. input and expected are arbitrary JSON snapshots — the server stores them, it does not interpret them.missing_attempts on every case is the resume list: an interrupted process reads
the evaluation back and knows exactly what it still owes.
Record attempts
Each attempt links a case and a trial index to a real run of the pinned version, and carries the scorer’s verdict. The server derives the output, the duration, the tokens and the cost from that run — you cannot type them in.case_key or by case_id. verdict is passed or failed
and it is the scorer’s verdict: a run that executed cleanly and answered
wrongly is a failure, and the API never confuses the two.
The batch is atomic. Re-sending the same (case, attempt_index, run) is a no-op
that comes back as "status": "duplicate"; sending a different run for a
(case, attempt_index) that already exists rejects the whole batch with 409 attempt_conflict and applies nothing.
Complete
pass_at_1 and
pass_hat_k, the judge score, the tokens, the cost, and the diff against the
baseline. It is idempotent once completed.
It also refuses to finish a job that is not done, and names what is missing:
Cancel
Stopping early keeps everything already recorded and recomputes the metrics from it:Read and list
GET /v2/evaluations/{evaluation_id} returns the evaluation with the same
per-case resume information. Listing is scoped to one exact tree — the point of
an evaluation is to be about a commit, so the commit is required:
status also filters, comma-separated: running, completed, failed,
cancelled.
Doing all of it in Python
splox.evals drives the four calls above — it launches the runs, grades them
with an ordinary Python function, records the attempts and completes the
evaluation, and hands back a [case][attempt] grid with the server’s own metrics
on grid.evaluation:
budget_cap is a soft launch
cap rather than a billing limit. For the checks a harness carries in its own tree
— evals/cases/*.yaml and the graders beside them — see Checks.