Skip to content
PensievePensievePensieve

Experiments

const e = await pensieve.experiments.retrieve("exp_0418");
console.log(e.hypothesis);
console.log(e.control, "→", e.variant);
console.log(e.gains, e.lead_points, e.status); // [1.1, 1.4, 1.3] 1.3 kept

An experiment is one controlled test of one change to a model’s post-training. It is how Pensieve learns what works on your data, rather than guessing.

How an experiment runs

Section titled “How an experiment runs”
  1. A hypothesis, from evidence. Agents start from something in your data: a kind of case the model gets wrong, an outcome it misses, a new task family with no history. They write what they expect a change to do, and why. Experiment exp_0418 started from this: 38% of the remaining errors were grades a lab later overturned.
  2. A and B. A is the current recipe. B is the same recipe with one change: here, overturned grades weighted four times in training. Only one thing differs, so a difference in score has one cause.
  3. Prep. If B needs new data or a new model first, it is built once: a reweighted training mix, practice cases written and checked, a reward built from outcomes, pairs of right and wrong answers.
  4. Three seeds. A and B are each trained three times, with different random seeds, and every run is scored on your held-out test. A gain that shows on one seed and not the next is luck, not a result.
  5. A last check. If B beat A on every seed, it is scored once more on cases neither has seen, held back for exactly this.
  6. Kept or dropped. A change that held everywhere joins the recipe, and the next experiments start from it. Anything else is dropped, and the reason is recorded.

For exp_0418, B beat A by 1.1, 1.4 and 1.3 points on the three seeds, and by 1.3 on the last check. It was kept, and is part of v7.

What is checked

Section titled “What is checked”

A result counts only when nothing could have inflated it. Each experiment lists its checks, such as:

  • no test case appears in B’s training data, or in its prep;
  • the gain held on new seeds, not only the first;
  • the change didn’t make some kinds of case worse, such as rare languages or tasks with strong test suites;
  • a reward never saw the outcome of a test case.

Cost and speed

Section titled “Cost and speed”

Each experiment records its compute cost and GPU hours. It also reports B’s speed, as a multiple of A’s: exp_0417, training on the whole patch, served at 0.3× and was dropped when its first-seed gain didn’t hold.

Experiments draw on the model’s monthly budget. When it runs out, agents hold their ideas for next month, and experiments.start is refused with conflict until you raise the cap. While experiments are paused, suggestions are refused the same way. See Follow and steer experiments.

Statuses

Section titled “Statuses”
Status Meaning
queued Waiting for the agents to design it, or for compute
running Prep or training under way; seeds_done says how far
kept Held on every seed and the last check; part of the recipe
dropped Didn’t hold; result says why
stopped Stopped by you, or by its cost cap; finished runs keep their results

Next

Section titled “Next”