Introduction

Pensieve finds the post-training that gets the best outcome from your data. Its agents run controlled experiments, one change at a time, and keep only what wins on your own held-out test.

curl "https://api.pensievelabs.co/v1/models/grade-audit/experiments?limit=3" \
  -H "Authorization: Bearer pk_test_sandbox"

That key is public, and the model is a sample. Two of its latest experiments are still running: RL with a lab’s later verdict as the reward, and practice cases written for a new set of Rust tasks. The third, weighting the grades labs overturned four times over, won on every seed and was kept.

What Pensieve does

You bring a decision your software makes, and the data behind it: the cases, what your experts decided, and what really happened next. Pensieve’s agents look for the post-training that turns that data into the best model for the decision.

There are many ways to post-train a model: which open model to start from, which data to show it and how to weight it, whether to fine-tune on examples, reward it for outcomes, tune it on pairs of better and worse answers, or distil a larger model’s reasoning first. Which works best depends on your data, and nobody can tell in advance. So Pensieve tests them. Each method is a plain markdown file the agents follow, so your team can add its own.

1. Connect your data. Streams, databases and files, read where they live. Part is held out as your test, and no model ever trains on it.
2. Agents run experiments. Each one changes one thing in the current recipe. The recipe with the change (B) and without it (A) are trained on the same seeds and scored on your test.
3. Only what wins is kept. A change counts when B beats A on every seed and on a last check against cases neither has seen. Everything else is dropped, with the reason on record.
4. The best recipe becomes a version. It ships once it beats the live version and the best frontier model on your test, runs in shadow, and is signed off by your expert.
5. What happens next feeds the next experiment. Outcomes and corrections become rewards and training data. A new frontier release, or drift in your data, starts a new round.

Why experiments

The best method depends on the data

On one sample model, weighting the grades a lab later overturned added 1.3 points; training on four times more context lost 0.1 and ran three times slower. Only a test on your data tells them apart.

Every result is controlled

A and B differ in one thing, train on the same seeds, and are scored on cases neither has seen. A gain on one seed that fails on the next is dropped.

Every result is on record

Each experiment keeps its hypothesis, its runs, its scores, what it cost, and why it was kept or dropped. You can read all of it, and suggest the next idea.

You own what it finds

The recipe each version is made of is yours to read. You keep the weights of every version you paid for, and models run where you choose.

Start here

QuickstartRead a sample model's experiments and the recipe they found, then call it.

ExperimentsHow an experiment is set up, run and judged.

Start from a data sourceConnect data, describe the decision, set a budget.

HTTP APIEvery operation, generated from the OpenAPI spec.