Frontier re-tests

TypeScript

for await (const retest of pensieve.frontier.list()) {
  console.log(retest.released_model, retest.results.map((r) => `${r.model} ${r.lead_before} → ${r.lead_after}`));
}

Python

for retest in client.frontier.list():
    print(retest.released_model, [(r.model, r.lead_before, r.lead_after) for r in retest.results])

HTTP

curl https://api.pensievelabs.co/v1/frontier -H "Authorization: Bearer $PENSIEVE_API_KEY"

A model’s lead over the frontier is only as good as the last frontier model it was tested against. So when a frontier lab or a general decision model ships, Pensieve scores the new model on every task’s test within days.

Each re-test lists, per model, the lead in points before and after, and what happened next. In the sandbox, the 22 October release narrowed grade-audit’s lead from 4.3 to 3.2 points. v7, already in its shadow run, went live on 27 October, 4.4 points ahead.

When a lead narrows

• The model keeps answering. A narrowed lead is still a lead.
• Agents start experiments to win it back, within the model’s budget.
• If the frontier model pulls ahead, the next version must beat it to ship. The live version stays until one does.

A model’s current lead is on the model itself, in lead: its score, the best frontier model’s score, the points between them, and when they were last tested.

Hearing about it

Subscribe a webhook to frontier.retested for every re-test, and model.lead_narrowed for the ones that matter.