Your test

TypeScript

const test = await pensieve.tests.retrieve("grade-audit");
for (const s of test.scores) console.log(s.kind, s.subject, s.score);
// version v8 0.95
// version v7 0.946
// …
// frontier frontier-2026-10-22 0.902
// general general-decision-2026-10-15 0.813

Python

test = client.tests.retrieve("grade-audit")
for s in test.scores:
    print(s.kind, s.subject, s.score)

HTTP

curl https://api.pensievelabs.co/v1/models/grade-audit/test -H "Authorization: Bearer $PENSIEVE_API_KEY"

Every model has one test: a set of your own cases, with the right answer for each, held out from training. Every version is scored on it, and so are the best frontier model and the best general decision model. That is how a claim like “4.4 points ahead of the frontier” is made, and how you can check it.

Every experiment is judged on it. A and B are scored on it after every seed.

What makes a good test

• It comes from your experts and your outcomes. Cases are set by the people who make the decision today, or come from what really happened. A sample model’s test is “5,000 held-out trials”, set by senior reviewers.
• No model trains on it. Test cases are kept out of every training set, and out of every experiment’s data.
• It has a metric you agree with. Usually the share of cases answered right; the model lists it in metric.
• It grows. When an expert answers a review case with add_to_test, or you add cases yourself, every version and frontier model is scored again.

Adding cases

TypeScript

await pensieve.tests.cases.add("grade-audit", {
  cases: [{ input: "Grader: PASS · patch deletes the failing assertion", answer: "false_pass", by: "maya@example.com" }],
});

Python

client.tests.cases.add("grade-audit", cases=[
    {"input": "Grader: PASS · patch deletes the failing assertion", "answer": "false_pass", "by": "maya@example.com"},
])

HTTP

curl https://api.pensievelabs.co/v1/models/grade-audit/test/cases \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"cases": [{"input": "Grader: PASS · patch deletes the failing assertion", "answer": "false_pass", "by": "maya@example.com"}]}'

What a score doesn’t prove

A score moves by chance

A score moves by chance alone. At about 90% right, a test of 5,000 cases moves by about 0.4 points either way; a test of 500 cases moves by about 1.3 points. Two models scored on the same cases are compared more tightly than one score is measured, so an experiment can keep a 0.4-point gain that held on every seed and the last check.

• A test score is a score on your cases as they were. If the cases your software sees change, the score can drift from live results. That is what outcomes catch.
• A lead over the frontier holds until the next frontier release. Every release is re-tested.