Skip to content
PensievePensievePensieve

Start from a data source

const source = await pensieve.sources.connect({ name: "CI runs", kind: "webhook" });


const model = await pensieve.tasks.create({
  name: "Flaky test triage",
  decision: "Decide whether a failing CI test is a real bug or a flaky test, before it pages an engineer.",
  answers: [{ value: "real_bug" }, { value: "flaky_test" }],
  outcomes: ["fixed_in_code", "passed_on_retry"],
  sources: [source.id],
  test: { held_out_cases: 2000, metric: "Right call on each failing test" },
  monthly_budget_usd: 1500,
});

That is all Pensieve needs to start experimenting: data, the decision, the outcome you care about, and a budget. The answer is a new model in building, with its first experiment queued.

  1. Connect the data. Point Pensieve at the cases, your experts’ past decisions and what happened next. Each source lives where it is; the reader runs beside it. See Data sources.
  2. Describe the decision. One sentence, and the fixed set of answers your software acts on. Answers are values your code can switch on, such as real_bug and flaky_test, never free text.
  3. Name the outcomes. What happens later that says whether an answer was right: the bug was fixed in code, or the test passed when it ran again. Outcomes become rewards and checks.
  4. Agree the test. How many cases to hold out, and the metric. Your experts set the right answers; no model trains on them. See Your test.
  5. Set a monthly budget. It caps the compute experiments use. Serving the model isn’t counted.

What happens next

Section titled “What happens next”
  • The first recipe. Agents fine-tune an open base on your history: the baseline every later experiment has to beat.
  • Experiments. They read where the baseline is wrong and try changes, one at a time, as described in Experiments.
  • The frontier baseline. The best frontier models are scored on the same test, so you know the bar from day one.
  • A first version. When the best recipe beats the frontier on your test, it becomes a candidate and goes through the release gates.