<SYSTEM>This is the full developer documentation for Pensieve</SYSTEM>

# Introduction

> Pensieve runs experiments on your data to find the post-training that gets the best outcome, and turns the winner into a model your software calls.

Pensieve finds the post-training that gets the best outcome from your data. Its agents run controlled experiments, one change at a time, and keep only what wins on your own held-out test.

```bash
curl "https://api.pensievelabs.co/v1/models/grade-audit/experiments?limit=3" \
  -H "Authorization: Bearer pk_test_sandbox"
```

That key is public, and the model is a sample. Two of its latest experiments are still running: RL with a lab’s later verdict as the reward, and practice cases written for a new set of Rust tasks. The third, giving the grades labs overturned four times the weight, won on every seed and on the last check, and was kept.

## What Pensieve does

[Section titled “What Pensieve does”](#what-pensieve-does)

You bring a decision your software makes, and the data behind it: the cases, what your experts decided, and what really happened next. Pensieve’s agents look for the post-training that turns that data into the best model for the decision.

There are many ways to post-train a model: which open model to start from, which data to show it and how to weight it, whether to fine-tune on examples, reward it for outcomes, tune it on pairs of better and worse answers, or distil a larger model’s reasoning first. Which works best depends on your data, and nobody can tell in advance. So Pensieve tests them. Each method is a plain markdown file the agents follow, so your team can add its own.

1. **Connect your data.** Streams, databases and files, read where they live. Part is held out as your test, and no model ever trains on it.
2. **Agents run experiments.** Each one changes one thing in the current recipe. The recipe with the change (B) and without it (A) are trained on the same seeds and scored on your test.
3. **Only what wins is kept.** A change counts when B beats A on every seed and on a last check against cases neither has seen. Everything else is dropped, with the reason on record.
4. **The best recipe becomes a version.** It ships once it beats the live version and the best frontier model on your test, runs in shadow, and is signed off by your expert.
5. **What happens next feeds the next experiment.** Outcomes and corrections become rewards and training data. A new frontier release, or drift in your data, starts a new round.

## Why experiments

[Section titled “Why experiments”](#why-experiments)

The best method depends on the data

On one sample model, weighting the grades a lab later overturned added 1.3 points; training on four times more context lost 0.1 and ran three times slower. Only a test on your data tells them apart.

Every result is controlled

A and B differ in one thing, train on the same seeds, and are scored on cases neither has seen. A gain on one seed that fails on the next is dropped.

Every result is on record

Each experiment keeps its hypothesis, its runs, its scores, what it cost, and why it was kept or dropped. You can read all of it, and suggest the next idea.

You own what it finds

The recipe each version is made of is yours to read. You keep the weights of every version you paid for, and models run where you choose.

## Start here

[Section titled “Start here”](#start-here)

[Quickstart](/quickstart/)Read a sample model's experiments and the recipe they found, then call it.

[Experiments](/concepts/experiments/)How an experiment is set up, run and judged.

[Start from a data source](/guides/start-from-a-data-source/)Connect data, describe the decision, set a budget.

[HTTP API](/api/)Every operation, generated from the OpenAPI spec.

# Quickstart

> In the sandbox, read the experiments behind a sample model and the recipe they found, call the model, and send back what happened.

* By hand

  1. **Install an SDK, and set a sandbox key.** Any key that starts `pk_test_` reaches the sandbox. After `pk_test_`, use letters, digits and underscores, up to 64 of them. Make one up, and you get a workspace of your own with four sample models and their experiments.

     * TypeScript

       ```bash
       npm install @pensievelab/sdk
       export PENSIEVE_API_KEY=pk_test_yourname
       ```

     * Python

       ```bash
       pip install pensievelab
       export PENSIEVE_API_KEY=pk_test_yourname
       ```

     * HTTP

       ```bash
       export PENSIEVE_API_KEY=pk_test_yourname
       ```

     The packages aren’t on npm and PyPI yet. Until they are, the HTTP tab works as it is.

  2. **See what the agents kept.** `grade-audit` checks whether a coding task’s grade holds. List the experiments that won:

     * TypeScript

       ```ts
       import Pensieve from "@pensievelab/sdk";


       const pensieve = new Pensieve(); // reads PENSIEVE_API_KEY


       for await (const e of pensieve.experiments.list("grade-audit", { status: "kept" })) {
         console.log(e.id, e.method, e.lead_points, e.idea);
       }
       ```

     * Python

       ```python
       from pensievelab import Pensieve


       client = Pensieve()  # reads PENSIEVE_API_KEY


       for e in client.experiments.list("grade-audit", status="kept"):
           print(e.id, e.method, e.lead_points, e.idea)
       ```

     * HTTP

       ```bash
       curl "https://api.pensievelabs.co/v1/models/grade-audit/experiments?status=kept" \
         -H "Authorization: Bearer $PENSIEVE_API_KEY"
       ```

     ```text
     exp_0418 fine-tuning 1.3 Weight overturned grades 4×
     exp_0416 preference-tuning 0.9 Preference-tune on right and wrong grades
     exp_0414 distillation 0.4 Distil from a 70B open teacher first
     ```

  3. **Read how one was judged.** Each experiment trained the current recipe (A) and the change (B) on three seeds, then checked B on cases neither had seen:

     * TypeScript

       ```ts
       for await (const run of pensieve.experiments.runs.list("exp_0418")) {
         console.log(run.label, run.arm, run.seed, run.score ?? run.lead_points);
       }
       ```

     * Python

       ```python
       for run in client.experiments.runs.list("exp_0418"):
           print(run.label, run.arm, run.seed, run.score or run.lead_points)
       ```

     * HTTP

       ```bash
       curl https://api.pensievelabs.co/v1/experiments/exp_0418/runs \
         -H "Authorization: Bearer $PENSIEVE_API_KEY"
       ```

  4. **Read the recipe they add up to.** The live version, v7, is made of the three kept changes:

     * TypeScript

       ```ts
       const v7 = await pensieve.models.versions.retrieve("grade-audit", "v7");
       for (const step of v7.recipe?.steps ?? []) console.log(step.method, step.description);
       ```

     * Python

       ```python
       v7 = client.models.versions.retrieve("grade-audit", "v7")
       for step in v7.recipe.steps:
           print(step.method, step.description)
       ```

     * HTTP

       ```bash
       curl https://api.pensievelabs.co/v1/models/grade-audit/versions/v7 \
         -H "Authorization: Bearer $PENSIEVE_API_KEY"
       ```

  5. **Call the model.** The answer is always one of its fixed values, so your code can act on it:

     * TypeScript

       ```ts
       const decision = await pensieve.models.decide("grade-audit", {
         input: { task: "rust-2291", grade: "pass", log: "Test change: expected page size 20 → 21" },
         key: "rust-2291",
       });
       console.log(decision.answer, decision.confidence, decision.version); // false_pass 0.94 v7
       ```

     * Python

       ```python
       decision = client.models.decide(
           "grade-audit",
           input={"task": "rust-2291", "grade": "pass", "log": "Test change: expected page size 20 → 21"},
           key="rust-2291",
       )
       print(decision.answer, decision.confidence, decision.version)  # false_pass 0.94 v7
       ```

     * HTTP

       ```bash
       curl https://api.pensievelabs.co/v1/models/grade-audit/decide \
         -H "Authorization: Bearer $PENSIEVE_API_KEY" \
         -H "Content-Type: application/json" \
         -d '{"input": {"task": "rust-2291", "grade": "pass", "log": "Test change: expected page size 20 → 21"}, "key": "rust-2291"}'
       ```

  6. **Send back what happened.** When the lab rejects the task, say so. Outcomes check every version against reality, and become rewards and training data for the next experiments.

     * TypeScript

       ```ts
       await pensieve.outcomes.send(decision.id, { outcome: "lab_rejected" });
       ```

     * Python

       ```python
       client.outcomes.send(decision.id, outcome="lab_rejected")
       ```

     * HTTP

       ```bash
       # Call the model and keep the call's id
       CALL_ID=$(curl -s https://api.pensievelabs.co/v1/models/grade-audit/decide \
         -H "Authorization: Bearer $PENSIEVE_API_KEY" \
         -H "Content-Type: application/json" \
         -d '{"input": {"task": "rust-2291", "grade": "pass", "log": "Test change: expected page size 20 → 21"}, "key": "rust-2291"}' | jq -r .id)


       curl https://api.pensievelabs.co/v1/calls/$CALL_ID/outcome \
         -H "Authorization: Bearer $PENSIEVE_API_KEY" \
         -H "Content-Type: application/json" \
         -d '{"outcome": "lab_rejected"}'
       ```

* With your coding agent

  ```bash
  npm install @pensievelab/sdk
  export PENSIEVE_API_KEY=pk_test_yourname
  ```

* TypeScript

  ```bash
  pip install pensievelab
  export PENSIEVE_API_KEY=pk_test_yourname
  ```

* Python

  ```bash
  export PENSIEVE_API_KEY=pk_test_yourname
  ```

* HTTP

  ```ts
  import Pensieve from "@pensievelab/sdk";


  const pensieve = new Pensieve(); // reads PENSIEVE_API_KEY


  for await (const e of pensieve.experiments.list("grade-audit", { status: "kept" })) {
    console.log(e.id, e.method, e.lead_points, e.idea);
  }
  ```

* TypeScript

  ```python
  from pensievelab import Pensieve


  client = Pensieve()  # reads PENSIEVE_API_KEY


  for e in client.experiments.list("grade-audit", status="kept"):
      print(e.id, e.method, e.lead_points, e.idea)
  ```

* Python

  ```bash
  curl "https://api.pensievelabs.co/v1/models/grade-audit/experiments?status=kept" \
    -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

* HTTP

  ```ts
  for await (const run of pensieve.experiments.runs.list("exp_0418")) {
    console.log(run.label, run.arm, run.seed, run.score ?? run.lead_points);
  }
  ```

* TypeScript

  ```python
  for run in client.experiments.runs.list("exp_0418"):
      print(run.label, run.arm, run.seed, run.score or run.lead_points)
  ```

* Python

  ```bash
  curl https://api.pensievelabs.co/v1/experiments/exp_0418/runs \
    -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

* HTTP

  ```ts
  const v7 = await pensieve.models.versions.retrieve("grade-audit", "v7");
  for (const step of v7.recipe?.steps ?? []) console.log(step.method, step.description);
  ```

* TypeScript

  ```python
  v7 = client.models.versions.retrieve("grade-audit", "v7")
  for step in v7.recipe.steps:
      print(step.method, step.description)
  ```

* Python

  ```bash
  curl https://api.pensievelabs.co/v1/models/grade-audit/versions/v7 \
    -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

* HTTP

  ```ts
  const decision = await pensieve.models.decide("grade-audit", {
    input: { task: "rust-2291", grade: "pass", log: "Test change: expected page size 20 → 21" },
    key: "rust-2291",
  });
  console.log(decision.answer, decision.confidence, decision.version); // false_pass 0.94 v7
  ```

* TypeScript

  ```python
  decision = client.models.decide(
      "grade-audit",
      input={"task": "rust-2291", "grade": "pass", "log": "Test change: expected page size 20 → 21"},
      key="rust-2291",
  )
  print(decision.answer, decision.confidence, decision.version)  # false_pass 0.94 v7
  ```

* Python

  ```bash
  curl https://api.pensievelabs.co/v1/models/grade-audit/decide \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"input": {"task": "rust-2291", "grade": "pass", "log": "Test change: expected page size 20 → 21"}, "key": "rust-2291"}'
  ```

* HTTP

  ```ts
  await pensieve.outcomes.send(decision.id, { outcome: "lab_rejected" });
  ```

* TypeScript

  ```python
  client.outcomes.send(decision.id, outcome="lab_rejected")
  ```

* Python

  ```bash
  # Call the model and keep the call's id
  CALL_ID=$(curl -s https://api.pensievelabs.co/v1/models/grade-audit/decide \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"input": {"task": "rust-2291", "grade": "pass", "log": "Test change: expected page size 20 → 21"}, "key": "rust-2291"}' | jq -r .id)


  curl https://api.pensievelabs.co/v1/calls/$CALL_ID/outcome \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"outcome": "lab_rejected"}'
  ```

* HTTP

  Paste this into your coding agent, in the project that makes the decision today:

  ```text
  Read https://docs.pensievelabs.co/agents.md, then wire Pensieve into this project with a sandbox key.
  ```

  The agent finds the function that makes the decision, wraps it so every call is recorded as data for experiments, and tests it against the sandbox. It never needs a live key: you set `PENSIEVE_API_KEY` yourself. See [Use Pensieve from a coding agent](/agents/).

The sandbox

The four sample models, their experiments and their data are made up. `pk_test_sandbox` is public and shared by everyone; a key you make up is yours alone. Changes last about an hour. See [The sandbox](/reference/sandbox/).

## Next

[Section titled “Next”](#next)

* [Experiments](/concepts/experiments/): how an experiment is set up, run and judged.
* [Start from a data source](/guides/start-from-a-data-source/): what Pensieve needs to start experimenting on your data.

# Experiments

> Each experiment changes one thing in the current recipe, trains with and without it on the same seeds, and is kept only if it wins on your test.

* TypeScript

  ```ts
  const e = await pensieve.experiments.retrieve("exp_0418");
  console.log(e.hypothesis);
  console.log(e.control, "→", e.variant);
  console.log(e.gains, e.lead_points, e.status); // [1.1, 1.4, 1.3] 1.3 kept
  ```

* Python

  ```python
  e = client.experiments.retrieve("exp_0418")
  print(e.hypothesis)
  print(e.control, "→", e.variant)
  print(e.gains, e.lead_points, e.status)  # [1.1, 1.4, 1.3] 1.3 kept
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/experiments/exp_0418 -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

An experiment is one controlled test of one change to a model’s post-training. It is how Pensieve learns what works on your data, rather than guessing.

## How an experiment runs

[Section titled “How an experiment runs”](#how-an-experiment-runs)

1. **A hypothesis, from evidence.** Agents start from something in your data: a kind of case the model gets wrong, an outcome it misses, a new task family with no history. They write what they expect a change to do, and why. Experiment `exp_0418` started from this: 38% of the remaining errors were grades a lab later overturned.
2. **A and B.** A is the current recipe. B is the same recipe with one change: here, overturned grades weighted four times in training. Only one thing differs, so a difference in score has one cause.
3. **Prep.** If B needs new data or a new model first, it is built once: a reweighted training mix, practice cases written and checked, a reward built from outcomes, pairs of right and wrong answers.
4. **Three seeds.** A and B are each trained three times, with different random seeds, and every run is scored on your held-out test. A gain that shows on one seed and not the next is luck, not a result.
5. **A last check.** If B beat A on every seed, it is scored once more on cases neither has seen, held back for exactly this.
6. **Kept or dropped.** A change that held everywhere joins the recipe, and the next experiments start from it. Anything else is dropped, and the reason is recorded.

For `exp_0418`, B beat A by 1.1, 1.4 and 1.3 points on the three seeds, and by 1.3 on the last check. It was kept, and is part of v7.

## What is checked

[Section titled “What is checked”](#what-is-checked)

A result counts only when nothing could have inflated it. Each experiment lists its checks, such as:

* no test case appears in B’s training data, or in its prep;
* the gain held on new seeds, not only the first;
* the change didn’t make some kinds of case worse, such as rare languages or tasks with strong test suites;
* a reward never saw the outcome of a test case.

## Cost and speed

[Section titled “Cost and speed”](#cost-and-speed)

Each experiment records its compute cost and GPU hours. It also reports B’s speed, as a multiple of A’s: `exp_0417`, training on the whole patch, served at 0.3× and was dropped when its first-seed gain didn’t hold.

Experiments draw on the model’s monthly budget. When it runs out, agents hold their ideas for next month, and `experiments.start` is refused with `conflict` until you raise the cap. While experiments are paused, suggestions are refused the same way. See [Follow and steer experiments](/guides/steer-experiments/).

## Statuses

[Section titled “Statuses”](#statuses)

| Status    | Meaning                                                              |
| --------- | -------------------------------------------------------------------- |
| `queued`  | Waiting for the agents to design it, or for compute                  |
| `running` | Prep or training under way; `seeds_done` says how far                |
| `kept`    | Held on every seed and the last check; part of the recipe            |
| `dropped` | Didn’t hold; `result` says why                                       |
| `stopped` | Stopped by you, or by its cost cap; finished runs keep their results |

Ideas are welcome

Your team knows things the data doesn’t say. Suggest an idea, with a method and a cost cap if you like, and the agents design and run it like their own. See [Follow and steer experiments](/guides/steer-experiments/). A way of training that no method covers yet can be [written as a method of your own](/guides/write-a-method/).

## Next

[Section titled “Next”](#next)

* [Post-training methods](/concepts/post-training/): the markdown files experiments follow, and what they can change.
* [Recipes](/concepts/recipes/): what the kept changes add up to.

# Post-training methods

> Every method is a plain markdown file the agents follow. Pensieve ships five; write your own, or copy one and change it.

* TypeScript

  ```ts
  for await (const m of pensieve.methods.list()) {
    console.log(m.id, m.version, m.built_in, m.name);
  }
  ```

* Python

  ```python
  for m in client.methods.list():
      print(m.id, m.version, m.built_in, m.name)
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/methods -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

A method is how an experiment trains B. Each one is a plain markdown file: when to try it, what it needs, its steps, the settings an experiment can change, and what to check before a result counts. The agents read the file to design an experiment, and follow it as written.

Because a method is only text, it can be anything your team can describe: a training order, a new way to build pairs, a reward made from a signal only you have. [Write your own](/guides/write-a-method/), or copy one of Pensieve’s and change it.

## Built-in methods

[Section titled “Built-in methods”](#built-in-methods)

| Method              | How the model learns                                                                                | Try it when                                         |
| ------------------- | --------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| `fine-tuning`       | From examples of the right answer: your experts’ decisions, corrected calls, checked practice cases | There is any history; most recipes build on it      |
| `rl`                | From a reward for matching what really happened next                                                | Outcomes judge better than your experts’ first call |
| `preference-tuning` | From pairs of a better and a worse answer for the same case                                         | A right and a wrong answer look alike               |
| `distillation`      | From a larger open model’s step-by-step reasoning                                                   | The decision takes several steps to reason through  |
| `meta-training`     | From a checkpoint trained to adapt fast across many decision tasks                                  | New kinds of case arrive with little history        |

Methods chain. On one sample model, the recipe distils a teacher first, then fine-tunes with overturned grades weighted 4×, then preference-tunes on pairs of right and wrong grades. Each step was its own kept experiment.

## What a method file looks like

[Section titled “What a method file looks like”](#what-a-method-file-looks-like)

Only the title and the first paragraph are required: they become the method’s `name` and `description`. Pensieve’s own files then have four sections. This is the built-in `fine-tuning`, as the agents read it:

fine-tuning.md

```markdown
# Fine-tuning


Train the model on cases and the answers your experts gave them. Try it first on any history: most recipes build on it, and many changes worth testing are changes to what it learns from.


## Needs


- Cases with a known right answer: your experts' decisions, corrected calls, or outcomes that settle which answer was right.
- At least a few hundred per answer value. Fewer, and the experiment says so instead of training.


## Steps


1. Build the training mix from the model's data sources, leaving out every case held for the test.
2. Weight the mix as the settings say.
3. Train from the starting checkpoint, with the same steps for A and B.
4. Score every checkpoint on the held-out test.


## Settings


- `weighting`: how often each kind of case is shown. Default: every case once. Experiments often weight the cases the current version gets wrong, or ones whose outcome overturned the expert.
- `inputs`: which fields of a case the model sees. Default: every field in the model's input schema.
- `context`: how many tokens of each case it reads. Default 8k.
- `synthetic`: practice cases written by agents and checked by your experts, as a share of the mix. Default none.
- `epochs`: passes over the mix. Default 2.


## Checks


- No test case, or a near copy of one, is in the training mix or in any practice case.
- Practice cases are scored by your experts on a sample before training on them.
- A gain must not come from one kind of case getting worse: the test is also scored by kind.
```

* **Needs** is what must be true of your data. If it isn’t, the experiment says so instead of training.
* **Steps** is what the agents do, in order.
* **Settings** are what an experiment can change, with their defaults. An experiment’s `lever` names its change in words, such as `Data mix`, not by a setting’s name such as `weighting`.
* **Checks** are what must hold before a result counts, on top of the checks every experiment gets.

A method decides how B is trained, never how it is judged. Whatever a file says, A and B are trained on the same seeds and scored on your held-out test, and the test never enters training. A method can add checks; it can’t remove them.

## Versions

[Section titled “Versions”](#versions)

Every change to a method raises its `version`. Each experiment records the method and version it ran, so a result always leads back to the exact text the agents followed:

* TypeScript

  ```ts
  const e = await pensieve.experiments.retrieve("exp_0418");
  const m = await pensieve.methods.retrieve(e.method!, { version: e.method_version! });
  ```

* Python

  ```python
  e = client.experiments.retrieve("exp_0418")
  m = client.methods.retrieve(e.method, version=e.method_version)
  ```

* HTTP

  ```bash
  curl "https://api.pensievelabs.co/v1/methods/fine-tuning?version=1" -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

Experiments already designed keep the version they started with when the file changes. Versions outlive the method: a removed method leaves the list, but each of its versions stays readable with `version`, and its slug is never used again.

## What experiments change

[Section titled “What experiments change”](#what-experiments-change)

The same method can pull on different parts of the recipe:

| `lever`        | What changes                                                 | A sample experiment                                                                   |
| -------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| Data mix       | Which cases are shown, and how often                         | Weight grades the labs later overturned 4×: **kept**, +1.3                            |
| Synthetic data | Practice cases written by agents and checked by your experts | 3,400 practice trials for 340 new Rust tasks: running, +0.4 on two seeds              |
| Reward         | What RL rewards                                              | The labs’ accept-or-return verdict as the reward: running                             |
| Inputs         | What the model sees for each case                            | Add the coding agent’s stated goal to each command: **kept**, +1.1                    |
| Context        | How much of each case it reads                               | The whole patch, 32k tokens, not just the failing tests: dropped, −0.1, and 3× slower |
| Starting point | Which checkpoint training starts from                        | A meta-trained checkpoint: dropped, −0.4, and −2.1 on rare languages                  |
| Model size     | How large the model is                                       | A 1B model instead of 3B: dropped after two seeds, −0.9                               |

## How agents choose what to try

[Section titled “How agents choose what to try”](#how-agents-choose-what-to-try)

* **From the errors.** Agents read where the current version is wrong, on your test and in live outcomes, and look for what those cases share. Then they look for a method whose file says to try it there.
* **From what has paid before.** Every experiment’s result is kept with its method and lever. Ideas like the ones that paid on your model go first; ideas like the ones that keep failing go last.
* **Within the budget.** Cheap, likely ideas run first. A costly one, such as a larger context, runs when the evidence for it is strong.
* **From you.** Ideas you suggest, and methods you write, are designed and judged the same way. See [Follow and steer experiments](/guides/steer-experiments/).

Only open models are trained. Frontier models are not fine-tuned; they are the baseline every version must beat on your test.

# Write your own method

> Describe a post-training method in a markdown file, add it to your workspace, and the agents can test it like their own.

Your team may know a way to train for your decision that no built-in method covers. Write it down as a markdown file, and the agents can choose it for an experiment, or run it when you suggest an idea.

1. **Write the file.** Start with a `#` title and a paragraph saying what the method does and when to try it. The rest is yours; the sections Pensieve’s own methods use are a good start.

   methods/curriculum.md

   ```markdown
   # Curriculum fine-tuning


   Fine-tune on easy cases first, then hard ones. Try it when the model handles common cases
   well but misses the ones your experts argue about.


   ## Needs


   - Cases more than one expert answered, to tell how hard each one is.


   ## Steps


   1. Score each training case's difficulty by how often your experts disagreed on it.
   2. Train in stages, from the easiest cases to the hardest.
   3. Score every checkpoint on the held-out test.


   ## Settings


   - `stages`: how many stages. Default 3.
   - `order`: easy to hard, or hard to easy. Default easy to hard.


   ## Checks


   - The test's hard cases are scored on their own, and don't get worse.
   ```

2. **Add it to your workspace.** Its slug is how experiments name it.

   * TypeScript

     ```ts
     import { readFileSync } from "node:fs";


     const method = await pensieve.methods.create({
       slug: "curriculum",
       markdown: readFileSync("methods/curriculum.md", "utf8"),
     });
     console.log(method.name, method.version); // Curriculum fine-tuning 1
     ```

   * Python

     ```python
     from pathlib import Path


     method = client.methods.create(
         slug="curriculum",
         markdown=Path("methods/curriculum.md").read_text(),
     )
     print(method.name, method.version)  # Curriculum fine-tuning 1
     ```

   * HTTP

     ```bash
     jq -Rs '{slug: "curriculum", markdown: .}' methods/curriculum.md |
       curl https://api.pensievelabs.co/v1/methods \
         -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" -d @-
     ```

3. **Use it.** From now on the agents may choose it when their evidence fits the file. To test it now, suggest an idea and name it:

   * TypeScript

     ```ts
     await pensieve.experiments.start("grade-audit", {
       idea: "Train on the grades reviewers agreed on first, then the disputed ones",
       method: "curriculum",
       max_cost_usd: 60,
     });
     ```

   * Python

     ```python
     client.experiments.start(
         "grade-audit",
         idea="Train on the grades reviewers agreed on first, then the disputed ones",
         method="curriculum",
         max_cost_usd=60,
     )
     ```

   * HTTP

     ```bash
     curl https://api.pensievelabs.co/v1/models/grade-audit/experiments \
       -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
       -d '{"idea": "Train on the grades reviewers agreed on first, then the disputed ones", "method": "curriculum", "max_cost_usd": 60}'
     ```

   It is judged like any experiment: A and B on the same seeds, scored on your held-out test, and kept only if B wins on every seed and the last check.

## Change it

[Section titled “Change it”](#change-it)

Send the whole new file. The version goes up by one; experiments already designed keep the version they started with, and each records the version it ran.

* TypeScript

  ```ts
  await pensieve.methods.update("curriculum", { markdown: readFileSync("methods/curriculum.md", "utf8") });
  ```

* Python

  ```python
  client.methods.update("curriculum", markdown=Path("methods/curriculum.md").read_text())
  ```

* HTTP

  ```bash
  jq -Rs '{markdown: .}' methods/curriculum.md |
    curl -X PATCH https://api.pensievelabs.co/v1/methods/curriculum \
      -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" -d @-
  ```

Keep your method files in your own repository, next to the code that makes the decision, and add or update them from CI. Then a method’s history is in your version control as well as Pensieve’s.

## Start from a built-in one

[Section titled “Start from a built-in one”](#start-from-a-built-in-one)

Built-in methods can’t be changed, but they can be copied. Read one, change what you need, and add it under a new slug:

* TypeScript

  ```ts
  const { markdown } = await pensieve.methods.retrieve("rl");
  await pensieve.methods.create({
    slug: "rl-on-refunds",
    markdown: markdown.replace("# RL on outcomes", "# RL on refunds").replace("Default: matching the outcome.", "Default: no refund within 90 days."),
  });
  ```

* Python

  ```python
  markdown = client.methods.retrieve("rl").markdown
  client.methods.create(
      slug="rl-on-refunds",
      markdown=markdown.replace("# RL on outcomes", "# RL on refunds").replace("Default: matching the outcome.", "Default: no refund within 90 days."),
  )
  ```

* HTTP

  ```bash
  curl -s https://api.pensievelabs.co/v1/methods/rl -H "Authorization: Bearer $PENSIEVE_API_KEY" |
    jq '{slug: "rl-on-refunds", markdown: (.markdown | sub("# RL on outcomes"; "# RL on refunds") | sub("Default: matching the outcome\\."; "Default: no refund within 90 days."))}' |
    curl https://api.pensievelabs.co/v1/methods \
      -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" -d @-
  ```

What a method can't change

A method decides how B is trained, never how it is judged. Whatever a file says, A and B train on the same seeds, the held-out test never enters training, and a result counts only if it holds on every seed and the last check. A method can add checks; it can’t remove them.

## Remove it

[Section titled “Remove it”](#remove-it)

`methods.remove("curriculum")`. The method leaves the list, and the agents stop choosing it. Experiments that used it keep it on record, with the version they ran, and every version stays readable: `methods.retrieve("curriculum", { version: 1 })` still returns the text the agents followed, while a retrieve without `version` answers `not_found`. The slug can’t be used again, so a result never leads back to a different file. A method a queued or running experiment uses can’t be removed until that experiment ends.

# Use Pensieve from a coding agent

> Point your coding agent at one file, and it wires Pensieve into your project against the sandbox.

```text
Read https://docs.pensievelabs.co/agents.md, then wire Pensieve into this project with a sandbox key.
```

Paste that into Claude Code, Codex, Cursor or any agent that can read a web page, in the project that makes the decision today. The agent reads the instructions, finds the function that makes the decision, wraps it with `capture` so every call becomes data for experiments, and tests it against the sandbox.

## What the agent reads

[Section titled “What the agent reads”](#what-the-agent-reads)

| File                                 | What it holds                                                                                  |
| ------------------------------------ | ---------------------------------------------------------------------------------------------- |
| [`/agents.md`](/agents.md)           | Step-by-step instructions for wiring Pensieve into a project, and the rules the agent keeps to |
| [`/llms.txt`](/llms.txt)             | A short summary of Pensieve, with links to the two files below                                 |
| [`/llms-full.txt`](/llms-full.txt)   | Every page in one file                                                                         |
| [`/llms-small.txt`](/llms-small.txt) | Every page in one file, cut down for small context windows                                     |
| [`/openapi.json`](/openapi.json)     | The API contract: every operation, field and error                                             |

## What the agent keeps to

[Section titled “What the agent keeps to”](#what-the-agent-keeps-to)

* **It never handles a live key.** It works with a `pk_test_` sandbox key. You set `PENSIEVE_API_KEY` for production yourself, outside the chat.
* **It changes nothing your users see.** It wraps the existing call with `capture` and leaves the model observing. Your code keeps answering until a version passes its gates.
* **It fails open.** If Pensieve can’t be reached, your function answers, as it did before.
* **It asks before it steers.** Starting an experiment, or changing a model’s budget or routing, is your call. The agent asks you first.
* **It shows you the diff.** The change is a few lines around one function, plus a test.

Releases stay with people

An agent can read releases and the review queue, but sign-off is for the experts named in the app’s Settings. The instructions tell agents to ask you before signing off, splitting, promoting or rolling back.

# Record the calls you make today

> Wrap the function that makes the decision today with capture. Every call becomes data for experiments, and nothing changes for your users.

* TypeScript

  ```ts
  import Pensieve from "@pensievelab/sdk";


  const pensieve = new Pensieve();


  // Before: const result = await callFrontierModel(trial);
  const auditGrade = pensieve.capture("grade-audit", callFrontierModel, {
    answer: (result) => result.verdict,               // your result → one of the model's answers
    use: (decision) => ({ verdict: decision.answer }), // the model's answer → your result
    key: (trial) => trial.task,                        // a stable id for the case
    providerModel: "frontier-large",
  });


  const result = await auditGrade(trial); // same arguments, same result
  ```

* Python

  ```python
  from pensievelab import Pensieve


  client = Pensieve()


  @client.capture(
      "grade-audit",
      answer=lambda result: result["verdict"],            # your result → one of the model's answers
      use=lambda decision: {"verdict": decision.answer},  # the model's answer → your result
      key=lambda trial: trial["task"],                    # a stable id for the case
      provider_model="frontier-large",
  )
  def audit_grade(trial):
      return call_frontier_model(trial)


  result = audit_grade(trial)  # same arguments, same result
  ```

* HTTP

  Without an SDK, keep calling your function, and record each call yourself:

  ```bash
  curl https://api.pensievelabs.co/v1/captures \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "grade-audit", "calls": [{"input": {"task": "rust-2291", "grade": "pass"}, "answer": "grade_holds", "key": "rust-2291", "provider_model": "frontier-large", "latency_ms": 2140}]}'
  ```

  Send up to 1,000 calls in one request. Once the model is live, call `decide` first and fall back to your function when `routed` is `fallback`.

In the sandbox, `grade-audit` already routes calls to the model, so the wrapped function wouldn’t run and its answers wouldn’t be recorded. Set the sample model to `observe` in your sandbox workspace first:

* TypeScript

  ```ts
  await pensieve.models.routing.update("grade-audit", { mode: "observe" }); // sandbox only
  ```

* Python

  ```python
  client.models.routing.update("grade-audit", mode="observe")  # sandbox only
  ```

* HTTP

  ```bash
  curl -X PATCH https://api.pensievelabs.co/v1/models/grade-audit/routing \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"mode": "observe"}'
  ```

Most teams already make the decision, with a frontier model or a set of rules. That traffic is the best data source there is: real cases, in the shape the model will see them. `capture` wraps the function that makes the decision, records every call, and returns the same result, so the rest of your code doesn’t change.

The recorded calls, with the outcomes you send back, become your [test](/concepts/your-test/) and the data [experiments](/concepts/experiments/) train on. Once a recipe wins, the same wrapper lets the model answer.

## What happens on each call

[Section titled “What happens on each call”](#what-happens-on-each-call)

What `capture` does depends on the model’s [routing](/concepts/routing/), which you change in the app or through the API.

| Routing                 | Who answers                                                          | What is recorded                                                                        |
| ----------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `observe` (a new model) | Your function                                                        | The case and your function’s answer                                                     |
| `model`                 | The expert model, through `use`                                      | The decision. When the model is unsure, your function answers, and that is recorded too |
| `split`                 | The model for the share of calls you set, your function for the rest | Both                                                                                    |

While the model observes, the agents experiment on the recorded calls. Once a version passes its [gates](/concepts/releases/) and you switch routing to `model`, the model answers. Your function still answers whenever the model’s confidence is under the threshold.

If Pensieve is down, your code answers

`capture` never puts Pensieve in the way of your users. It gives Pensieve 2 seconds, with no retries in your request’s path, and falls back to your function on any failure. Nothing from Pensieve is thrown or raised into your code. Failures go to the client’s `onError` (`on_error` in Python), which logs a warning by default.

## The options

[Section titled “The options”](#the-options)

| Option          | Required | What it does                                                                                            |
| --------------- | -------- | ------------------------------------------------------------------------------------------------------- |
| `answer`        | Yes      | Turns your function’s result into one of the model’s answer values                                      |
| `use`           | No       | Turns a decision into your function’s result. Without it, the model only observes, whatever its routing |
| `key`           | No       | A stable id for the case, so you can send outcomes by it later                                          |
| `input`         | No       | Picks the case from your function’s arguments. Defaults to the first argument                           |
| `providerModel` | No       | The model your function calls, kept with each recorded call                                             |

Find a model’s answer values, outcomes and input schema with `models.retrieve`:

* TypeScript

  ```ts
  const model = await pensieve.models.retrieve("grade-audit");
  model.answers.map((a) => a.value); // ["grade_holds", "false_pass", "false_fail"]
  model.outcomes;                     // ["lab_accepted", "lab_rejected"]
  ```

* Python

  ```python
  model = client.models.retrieve("grade-audit")
  [a.value for a in model.answers]  # ['grade_holds', 'false_pass', 'false_fail']
  model.outcomes                    # ['lab_accepted', 'lab_rejected']
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/models/grade-audit -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

## Sending what was recorded

[Section titled “Sending what was recorded”](#sending-what-was-recorded)

Recorded calls go out in the background, in batches of 100 or every 2 seconds. A short-lived script or a test should send what is left before it exits:

* TypeScript

  ```ts
  await pensieve.flush();
  ```

* Python

  ```python
  client.flush()  # also runs when your program exits, and on client.close()
  ```

* HTTP

  Each request to `/v1/captures` is sent when you make it.

## Next

[Section titled “Next”](#next)

* [Send outcomes and corrections](/guides/outcomes-and-corrections/), so every version is checked against what really happened.
* [Routing and fallback](/concepts/routing/), to choose when the model takes over.

# Data sources

> What Pensieve learns from, what it holds out, and how outcomes become rewards.

* TypeScript

  ```ts
  for await (const s of pensieve.sources.list()) {
    console.log(s.name, s.via, s.volume, "·", s.held_out, "·", s.models);
  }
  ```

* Python

  ```python
  for s in client.sources.list():
      print(s.name, s.via, s.volume, "·", s.held_out, "·", s.models)
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/sources -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

Experiments are only as good as the data they run on. A data source is anything that holds your cases, your experts’ decisions, or what happened next.

## What makes good data

[Section titled “What makes good data”](#what-makes-good-data)

| Kind          | What it gives experiments                            | A sample source                                       |
| ------------- | ---------------------------------------------------- | ----------------------------------------------------- |
| **Cases**     | What the model will see: the input of every decision | Agent trials, a Kafka stream of 240k a day            |
| **Decisions** | What your experts decided, as examples to learn from | Reviewer verdicts, 3.1M in Postgres over two years    |
| **Outcomes**  | What really happened next, which can become a reward | `lab_accepted` and `lab_rejected`, sent back per call |
| **Context**   | What your experts looked at to decide                | Grader logs, 1.2 TB of files                          |

Outcomes matter most. Outcomes catch what experts miss. On one sample model, the labs sent back tasks whose wrong grades the reviewers had let through, so an experiment now uses whether each task was accepted as the reward.

If you have no history yet, start by [recording the calls you make today](/guides/wrap-your-frontier-call/). Each recorded call is a case, and the outcomes you send back complete it.

## Held out from training

[Section titled “Held out from training”](#held-out-from-training)

Every source says what is kept out of training, in `held_out`: the last 60 days of a stream, or the cases set aside as your [test](/concepts/your-test/). No experiment trains on held-out data, and every experiment checks that none leaked in.

## Where it is read

[Section titled “Where it is read”](#where-it-is-read)

Pensieve’s reader runs where your data lives: in your cloud, next to the database or bucket. Credentials are set in the app, never sent through the API. Models train and run where you choose: in your cloud or in Pensieve’s.

Kinds of source

`stream` (Kafka, Kinesis, Pub/Sub), `database` (Postgres, a warehouse), `files` (S3 and other buckets), `webhook` (events your systems post), and `sdk` (calls recorded with `capture`).

# Frontier re-tests

> Every frontier and general model release is scored on every task's test. A narrowed lead starts experiments.

* TypeScript

  ```ts
  for await (const retest of pensieve.frontier.list()) {
    console.log(retest.released_model, retest.results.map((r) => `${r.model} ${r.lead_before} → ${r.lead_after}`));
  }
  ```

* Python

  ```python
  for retest in client.frontier.list():
      print(retest.released_model, [(r.model, r.lead_before, r.lead_after) for r in retest.results])
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/frontier -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

A model’s lead over the frontier is only as good as the last frontier model it was tested against. So when a frontier lab or a general decision model ships, Pensieve scores the new model on every task’s [test](/concepts/your-test/) within days.

Each re-test lists, per model, the lead in points before and after, and what happened next. In the sandbox, the 22 October release narrowed `grade-audit`’s lead from 4.3 to 3.2 points. v7, already in its shadow run, went live on 27 October, 4.4 points ahead.

## When a lead narrows

[Section titled “When a lead narrows”](#when-a-lead-narrows)

* The model keeps answering. A narrowed lead is still a lead.
* Agents start experiments to win it back, within the model’s budget.
* If the frontier model pulls ahead, the next version must beat it to ship. The live version stays until one does.

A model’s current lead is on the model itself, in `lead`: its score, the best frontier model’s score, the points between them, and when they were last tested.

## Hearing about it

[Section titled “Hearing about it”](#hearing-about-it)

Subscribe a [webhook](/reference/webhooks/) to `frontier.retested` for every re-test, and `model.lead_narrowed` for the ones that matter.

# Recipes

> The post-training a version is made of, built from the experiments that were kept. Yours to read, and the starting point of every next experiment.

* TypeScript

  ```ts
  const { recipe } = await pensieve.models.versions.retrieve("grade-audit", "v7");
  console.log(recipe?.base_model);
  recipe?.steps.forEach((s, i) => console.log(i + 1, s.method, s.description));
  recipe?.data.forEach((d) => console.log(d.source, d.use));
  ```

* Python

  ```python
  recipe = client.models.versions.retrieve("grade-audit", "v7").recipe
  print(recipe.base_model)
  for i, step in enumerate(recipe.steps, 1):
      print(i, step.method, step.description)
  for d in recipe.data:
      print(d.source, d.use)
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/models/grade-audit/versions/v7 -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

A recipe is everything that made a version: the open model it starts from, the training steps in order, and the data each step learns from. It is the result of the experiments so far.

## How a recipe grows

[Section titled “How a recipe grows”](#how-a-recipe-grows)

Every experiment’s A is the current recipe. When B wins and is kept, the change joins the recipe, and the next experiments start from the new one. A version is a recipe that was trained, tested and given a number.

For the sample model `grade-audit`, v7’s recipe is three kept experiments on an open 8B base:

| Step | Method                                                             | From experiment  |
| ---- | ------------------------------------------------------------------ | ---------------- |
| 1    | Distil a 70B open teacher’s step-by-step checks                    | `exp_0414`, +0.4 |
| 2    | Fine-tune on reviewer verdicts, with overturned grades weighted 4× | `exp_0418`, +1.3 |
| 3    | Preference-tune on 9,000 pairs of a right and a wrong grade        | `exp_0416`, +0.9 |

Gains measured one change at a time overlap, so they don’t add up: together, the three put v7 1.2 points above v6 on the test. Two other ideas tried in the same weeks, a meta-trained starting point and a longer context, were dropped, so they aren’t in it.

## Why you can read it

[Section titled “Why you can read it”](#why-you-can-read-it)

* **It is the finding.** The recipe says what works on your data, and the experiments behind it say why. That knowledge stays yours.
* **It can be checked.** Each step points at the experiment that earned it, with its runs, scores and checks.
* **It is portable.** You keep the weights of every version you paid for. A recipe and its weights run on your own servers as well as Pensieve’s.

## Next

[Section titled “Next”](#next)

* [Data sources](/concepts/data-sources/): what recipes learn from.
* [Versions and release gates](/concepts/releases/): how a version reaches your users.

# Versions and release gates

> A version reaches your users only through its release gates. You hold the last one.

* TypeScript

  ```ts
  await pensieve.releases.stop("rel_grade_audit_v8"); // sandbox only: v8 is already in a shadow run there
  const release = await pensieve.releases.shadow("grade-audit", { version: "v8" });
  await pensieve.releases.signoff(release.id, { note: "Matches our reviewers on the Rust tasks." });
  await pensieve.releases.split(release.id, { percent: 10 });
  await pensieve.releases.promote(release.id, { expect: "v7" }); // only if v7 is still live


  // And back, in seconds:
  await pensieve.releases.rollback("grade-audit", { expect: "v8", reason: "Agreement dropped." });
  ```

* Python

  ```python
  client.releases.stop("rel_grade_audit_v8")  # sandbox only: v8 is already in a shadow run there
  release = client.releases.shadow("grade-audit", version="v8")
  client.releases.signoff(release.id, note="Matches our reviewers on the Rust tasks.")
  client.releases.split(release.id, percent=10)
  client.releases.promote(release.id, expect="v7")  # only if v7 is still live


  # And back, in seconds:
  client.releases.rollback("grade-audit", expect="v8", reason="Agreement dropped.")
  ```

* HTTP

  ```bash
  # Sandbox only: v8 is already in a shadow run there
  curl -X POST https://api.pensievelabs.co/v1/releases/rel_grade_audit_v8/stop -H "Authorization: Bearer $PENSIEVE_API_KEY"


  curl https://api.pensievelabs.co/v1/models/grade-audit/releases \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"version": "v8", "shadow_days": 7}'
  # then POST /v1/releases/{id}/signoff, /split, /promote, and POST /v1/models/grade-audit/rollback
  ```

In the sandbox, `grade-audit`’s v8 is already on day 3 of a 7-day shadow run, so the first line stops it before starting a new one. Leave that line out in a live workspace. The sandbox skips the wait, so the code above runs from top to bottom.

## The gates

[Section titled “The gates”](#the-gates)

Experiments keep improving the [recipe](/concepts/recipes/). When the best recipe so far is worth shipping, it is trained into a candidate version. A version goes live only when it has passed its gates, in order:

1. **It beats the live version on your test, if there is one.**
2. **It beats the best frontier model on your test.** A model that isn’t ahead of the frontier doesn’t ship, however long it has been building.
3. **It passes a shadow run.** It answers a copy of live traffic for a set number of days, 7 by default, and nothing it answers reaches your users. Its answers are checked against your experts and the outcomes.
4. **An expert signs it off.** The people named for sign-off in the app’s Settings see the shadow run’s results and approve, or send it back. In the sandbox, anyone can sign off.

Each release lists its gates, which have passed, and when.

## Rolling out

[Section titled “Rolling out”](#rolling-out)

After sign-off, choose how the version reaches your users:

* **`split`** sends a share of calls to the new version and the rest to the live one. Raise it as you gain confidence. Routing’s [`split` mode](/concepts/routing/) is a different thing: it shares calls between the model and your own code.
* **`promote`** makes it the live version. It takes `expect`, the version you believe is live now, and refuses with `conflict` if someone else changed it first.
* **`rollback`** moves traffic back to the previous version, which stays warm. It also takes `expect`.

Why expect

Two people, or a person and a script, can act on the same model at once. `expect` turns “promote v8” into “promote v8 over v7”. If v7 isn’t live any more, nothing happens and you get a `ConflictError` saying what is.

## Release states

[Section titled “Release states”](#release-states)

| State               | Meaning                                                   |
| ------------------- | --------------------------------------------------------- |
| `pending`           | A candidate that hasn’t passed the first two gates        |
| `shadow`            | In its shadow run                                         |
| `ready_for_signoff` | The shadow run is over; waiting for an expert             |
| `signed_off`        | Approved, not yet taking traffic                          |
| `split`             | Taking the share of traffic you set                       |
| `live`              | The live version                                          |
| `superseded`        | Went live, and a later release replaced it                |
| `rolled_back`       | Went live, then the model went back to the version before |
| `stopped`           | Stopped before going live                                 |

A version’s own status is `candidate`, `live`, `standby` (the previous version, kept warm for rollback) or `retired`.

A model’s status is `building` (no version beats the frontier yet), `shadow` (a version runs beside live traffic), `live` (a version takes traffic), `paused` (taken off traffic in the app, so your code answers every call until it is resumed) or `archived` (retired in the app: it answers no calls, and lists leave it out unless you ask for `status=archived`).

# Routing and fallback

> Who answers each call, the expert model or your own code, and when it falls back.

* TypeScript

  ```ts
  await pensieve.models.routing.update("grade-audit", { mode: "model", threshold: 0.8 });
  ```

* Python

  ```python
  client.models.routing.update("grade-audit", mode="model", threshold=0.8)
  ```

* HTTP

  ```bash
  curl -X PATCH https://api.pensievelabs.co/v1/models/grade-audit/routing \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"mode": "model", "threshold": 0.8}'
  ```

Every model has routing. It decides whether a call’s answer should come from the expert model or from your own code, and every decision says which in `routed`.

| Mode      | What happens                                                                                                       | `routed`                                    |
| --------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
| `observe` | Your code answers every call. Calls are recorded, through `capture`. A new model starts here                       | Always `fallback`, unless you pin `version` |
| `model`   | The expert model answers, unless its confidence is under `threshold`                                               | `model`, or `fallback` when unsure          |
| `split`   | The model answers `split_percent` of calls, unless its confidence is under `threshold`; your code answers the rest | `model` or `fallback`                       |

Routing’s `split` shares calls between the model and your own code. A release’s [`split`](/concepts/releases/#rolling-out) is a different thing: it shares calls between two versions of the model.

## The threshold

[Section titled “The threshold”](#the-threshold)

`threshold` is the confidence under which a call falls back to your code, in `model` and `split` modes alike. A higher threshold sends more calls to your code and fewer wrong answers to your users. Every call that falls back is recorded with your code’s answer, and those calls feed the next experiments: the cases the model is unsure about become training data.

Watch the share of calls that fall back in the app, or with `usage.retrieve()`. If it climbs, the cases your software sees may have changed: outcomes show whether the model is still right.

## Calls with no live version

[Section titled “Calls with no live version”](#calls-with-no-live-version)

A model that has no live version yet answers only when you pin one, which is useful for trying a candidate:

* TypeScript

  ```ts
  await pensieve.models.decide("agent-actions", { input: "Command: DROP TABLE users on prod", version: "v3" });
  ```

* Python

  ```python
  client.models.decide("agent-actions", input="Command: DROP TABLE users on prod", version="v3")
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/models/agent-actions/decide \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"input": "Command: DROP TABLE users on prod", "version": "v3"}'
  ```

Without `version`, the call is refused with `conflict`, and the message names a candidate to try.

# Your test

> The held-out cases every version and every frontier model is scored on. No model ever trains on them.

* TypeScript

  ```ts
  const test = await pensieve.tests.retrieve("grade-audit");
  for (const s of test.scores) console.log(s.kind, s.subject, s.score);
  // version v8 0.95
  // version v7 0.946
  // …
  // frontier frontier-2026-10-22 0.902
  // general general-decision-2026-10-15 0.813
  ```

* Python

  ```python
  test = client.tests.retrieve("grade-audit")
  for s in test.scores:
      print(s.kind, s.subject, s.score)
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/models/grade-audit/test -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

Every model has one test: a set of your own cases, with the right answer for each, held out from training. Every version is scored on it, and so are the best frontier model and the best general decision model. That is how a claim like “4.4 points ahead of the frontier” is made, and how you can check it.

**Every experiment is judged on it.** A and B are scored on it after every seed.

## What makes a good test

[Section titled “What makes a good test”](#what-makes-a-good-test)

* **It comes from your experts and your outcomes.** Cases are set by the people who make the decision today, or come from what really happened. A sample model’s test is “5,000 held-out trials”, set by senior reviewers.
* **No model trains on it.** Test cases are kept out of every training set, and out of every experiment’s data.
* **It has a metric you agree with.** Usually the share of cases answered right; the model lists it in `metric`.
* **It grows.** When an expert answers a review case with `add_to_test`, or you add cases yourself, every version and frontier model is scored again.

## Adding cases

[Section titled “Adding cases”](#adding-cases)

* TypeScript

  ```ts
  await pensieve.tests.cases.add("grade-audit", {
    cases: [{ input: "Grader: PASS · patch deletes the failing assertion", answer: "false_pass", by: "maya@example.com" }],
  });
  ```

* Python

  ```python
  client.tests.cases.add("grade-audit", cases=[
      {"input": "Grader: PASS · patch deletes the failing assertion", "answer": "false_pass", "by": "maya@example.com"},
  ])
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/models/grade-audit/test/cases \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"cases": [{"input": "Grader: PASS · patch deletes the failing assertion", "answer": "false_pass", "by": "maya@example.com"}]}'
  ```

## What a score doesn’t prove

[Section titled “What a score doesn’t prove”](#what-a-score-doesnt-prove)

A score moves by chance

A score moves by chance alone. At about 90% right, a test of 5,000 cases moves by about 0.4 points either way; a test of 500 cases moves by about 1.3 points. Two models scored on the same cases are compared more tightly than one score is measured, so an experiment can keep a 0.4-point gain that held on every seed and the last check.

* A test score is a score on your cases as they were. If the cases your software sees change, the score can drift from live results. That is what [outcomes](/guides/outcomes-and-corrections/) catch.
* A lead over the frontier holds until the next frontier release. Every release is [re-tested](/concepts/frontier-watch/).

# Send outcomes and corrections

> Tell Pensieve what really happened after each call, and what your experts would have answered.

* TypeScript

  ```ts
  // What really happened, by call id
  await pensieve.outcomes.send(decision.id, { outcome: "lab_rejected", note: "Lab found the edited test." });


  // An expert's answer for the case
  await pensieve.corrections.send(decision.id, { answer: "false_pass", by: "maya@example.com" });
  ```

* Python

  ```python
  # What really happened, by call id
  client.outcomes.send(decision.id, outcome="lab_rejected", note="Lab found the edited test.")


  # An expert's answer for the case
  client.corrections.send(decision.id, answer="false_pass", by="maya@example.com")
  ```

* HTTP

  ```bash
  # Call the model and keep the call's id
  CALL_ID=$(curl -s https://api.pensievelabs.co/v1/models/grade-audit/decide \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"input": {"task": "rust-2291", "grade": "pass"}, "key": "rust-2291"}' | jq -r .id)


  curl https://api.pensievelabs.co/v1/calls/$CALL_ID/outcome \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"outcome": "lab_rejected", "note": "Lab found the edited test."}'


  curl https://api.pensievelabs.co/v1/calls/$CALL_ID/correction \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"answer": "false_pass", "by": "maya@example.com"}'
  ```

Two kinds of feedback drive the experiments, and keep every version honest:

* **An outcome** is what really happened after the call: the lab accepted the task or sent it back, the engineer approved the command, the finding was fixed. Outcomes can become the reward an RL experiment trains on, and every version is checked against them, which is how drift is caught before your users notice.
* **A correction** is the answer an expert would have given. It becomes training data, and can become a test case.

Each model lists the outcomes it takes, in `outcomes`. A later outcome for the same call replaces the earlier one.

## By your own key, in bulk

[Section titled “By your own key, in bulk”](#by-your-own-key-in-bulk)

You may not keep Pensieve’s call id. Pass your own `key` when you decide or capture, then send outcomes by it, up to 1,000 at a time:

* TypeScript

  ```ts
  await pensieve.outcomes.sendMany({
    items: [
      { model: "grade-audit", key: "rust-2291", outcome: "lab_rejected" },
      { call_id: "cl_seed_grade_02", outcome: "lab_accepted" },
    ],
  });
  ```

* Python

  ```python
  client.outcomes.send_many(items=[
      {"model": "grade-audit", "key": "rust-2291", "outcome": "lab_rejected"},
      {"call_id": "cl_seed_grade_02", "outcome": "lab_accepted"},
  ])
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/outcomes \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"items": [{"model": "grade-audit", "key": "rust-2291", "outcome": "lab_rejected"}, {"call_id": "cl_seed_grade_02", "outcome": "lab_accepted"}]}'
  ```

An outcome may arrive before its call, for example when your code records calls in batches. Pensieve keeps it and matches it when the call arrives. The receipt shows which call each item matched, or `null` when it is still waiting.

## The review queue

[Section titled “The review queue”](#the-review-queue)

Pensieve sends some cases to your experts: calls where the model was unsure (`unsure`), calls where the outcome went against it (`disagreement`), and releases waiting for sign-off (`signoff`). Experts answer in the app, or your own review tool can answer through the API. A call’s case takes one of the model’s answers; a sign-off case takes only `approve` or `reject`. This loop answers the unsure cases with what your reviewer chose in your own tool:

* TypeScript

  ```ts
  for await (const c of pensieve.review.list({ state: "open", model: "grade-audit", kind: "unsure" })) {
    const answer = await askReviewer(c); // your review tool: "grade_holds", "false_pass" or "false_fail"
    await pensieve.review.answer(c.id, { answer, by: "maya@example.com", add_to_test: true });
  }
  ```

* Python

  ```python
  for case in client.review.list(state="open", model="grade-audit", kind="unsure"):
      answer = ask_reviewer(case)  # your review tool: "grade_holds", "false_pass" or "false_fail"
      client.review.answer(case.id, answer=answer, by="maya@example.com", add_to_test=True)
  ```

* HTTP

  ```bash
  curl "https://api.pensievelabs.co/v1/review?state=open&model=grade-audit&kind=unsure" -H "Authorization: Bearer $PENSIEVE_API_KEY"


  # Then answer each case with your reviewer's answer
  curl https://api.pensievelabs.co/v1/review/$CASE_ID/answer \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"answer": "false_pass", "by": "maya@example.com", "add_to_test": true}'
  ```

`add_to_test` also holds the case out of training, as a [test case](/concepts/your-test/).

# Start from a data source

> Connect your data, describe the decision and the outcome you want, and set a budget. The agents take it from there.

* TypeScript

  ```ts
  const source = await pensieve.sources.connect({ name: "CI runs", kind: "webhook" });


  const model = await pensieve.tasks.create({
    name: "Flaky test triage",
    decision: "Decide whether a failing CI test is a real bug or a flaky test, before it pages an engineer.",
    answers: [{ value: "real_bug" }, { value: "flaky_test" }],
    outcomes: ["fixed_in_code", "passed_on_retry"],
    sources: [source.id],
    test: { held_out_cases: 2000, metric: "Right call on each failing test" },
    monthly_budget_usd: 1500,
  });
  ```

* Python

  ```python
  source = client.sources.connect(name="CI runs", kind="webhook")


  model = client.tasks.create(
      name="Flaky test triage",
      decision="Decide whether a failing CI test is a real bug or a flaky test, before it pages an engineer.",
      answers=[{"value": "real_bug"}, {"value": "flaky_test"}],
      outcomes=["fixed_in_code", "passed_on_retry"],
      sources=[source.id],
      test={"held_out_cases": 2000, "metric": "Right call on each failing test"},
      monthly_budget_usd=1500,
  )
  ```

* HTTP

  ```bash
  SOURCE_ID=$(curl -s https://api.pensievelabs.co/v1/sources \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"name": "CI runs", "kind": "webhook"}' | jq -r .id)


  curl https://api.pensievelabs.co/v1/tasks \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"name": "Flaky test triage", "decision": "Decide whether a failing CI test is a real bug or a flaky test, before it pages an engineer.", "answers": [{"value": "real_bug"}, {"value": "flaky_test"}], "outcomes": ["fixed_in_code", "passed_on_retry"], "sources": ["'"$SOURCE_ID"'"], "test": {"held_out_cases": 2000, "metric": "Right call on each failing test"}, "monthly_budget_usd": 1500}'
  ```

That is all Pensieve needs to start experimenting: data, the decision, the outcome you care about, and a budget. The answer is a new model in `building`, with its first experiment queued.

1. **Connect the data.** Point Pensieve at the cases, your experts’ past decisions and what happened next. Each source lives where it is; the reader runs beside it. See [Data sources](/concepts/data-sources/).
2. **Describe the decision.** One sentence, and the fixed set of answers your software acts on. Answers are values your code can switch on, such as `real_bug` and `flaky_test`, never free text.
3. **Name the outcomes.** What happens later that says whether an answer was right: the bug was fixed in code, or the test passed when it ran again. Outcomes become rewards and checks.
4. **Agree the test.** How many cases to hold out, and the metric. Your experts set the right answers; no model trains on them. See [Your test](/concepts/your-test/).
5. **Set a monthly budget.** It caps the compute experiments use. Serving the model isn’t counted.

## What happens next

[Section titled “What happens next”](#what-happens-next)

* **The first recipe.** Agents fine-tune an open base on your history: the baseline every later experiment has to beat.
* **Experiments.** They read where the baseline is wrong and try changes, one at a time, as described in [Experiments](/concepts/experiments/).
* **The frontier baseline.** The best frontier models are scored on the same test, so you know the bar from day one.
* **A first version.** When the best recipe beats the frontier on your test, it becomes a candidate and goes through the [release gates](/concepts/releases/).

No history yet?

Record the calls you make today, with a frontier model or rules, and send back what happened. A few weeks of real traffic is often enough for a first test and a first recipe. See [Record the calls you make today](/guides/wrap-your-frontier-call/).

# Follow and steer experiments

> Watch experiments as they run, suggest ideas of your own, stop what you don't want, and set the budget.

* TypeScript

  ```ts
  for await (const e of pensieve.experiments.list("grade-audit", { status: "running" })) {
    console.log(e.id, e.idea, `${e.seeds_done}/${e.seeds_total} seeds`, e.gains);
  }
  ```

* Python

  ```python
  for e in client.experiments.list("grade-audit", status="running"):
      print(e.id, e.idea, f"{e.seeds_done}/{e.seeds_total} seeds", e.gains)
  ```

* HTTP

  ```bash
  curl "https://api.pensievelabs.co/v1/models/grade-audit/experiments?status=running" \
    -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

Agents run experiments on their own, within the model’s budget. You can follow every one, and steer them with what your team knows.

## Follow a run

[Section titled “Follow a run”](#follow-a-run)

Each experiment lists its runs: the prep step, A and B on every seed, and the last check. A training run shows its steps so far; a finished one shows its score on your test.

* TypeScript

  ```ts
  for await (const run of pensieve.experiments.runs.list("exp_0420")) {
    console.log(run.kind, run.arm, run.seed, run.status, run.steps_done, run.steps_total, run.score);
  }
  ```

* Python

  ```python
  for run in client.experiments.runs.list("exp_0420"):
      print(run.kind, run.arm, run.seed, run.status, run.steps_done, run.steps_total, run.score)
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/experiments/exp_0420/runs -H "Authorization: Bearer $PENSIEVE_API_KEY"
  ```

To hear when one ends, subscribe a [webhook](/reference/webhooks/) to `experiment.ended`.

## Suggest an idea

[Section titled “Suggest an idea”](#suggest-an-idea)

Your team knows things the data doesn’t say: a new kind of case coming next quarter, a signal nobody has used, a reason the experts disagree. Suggest it in a line. Name a method and a cost cap if you have a view: a built-in method, or [one you wrote](/guides/write-a-method/). The agents write the hypothesis, build B and run it like their own.

* TypeScript

  ```ts
  const e = await pensieve.experiments.start("grade-audit", {
    idea: "When two reviewers disagree, train on the senior reviewer's grade",
    method: "fine-tuning",
    max_cost_usd: 80,
  });
  ```

* Python

  ```python
  e = client.experiments.start(
      "grade-audit",
      idea="When two reviewers disagree, train on the senior reviewer's grade",
      method="fine-tuning",
      max_cost_usd=80,
  )
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/models/grade-audit/experiments \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"idea": "When two reviewers disagree, train on the senior reviewer'\''s grade", "method": "fine-tuning", "max_cost_usd": 80}'
  ```

It is judged like any other: kept only if it wins on every seed and the last check.

## Stop, pause and budget

[Section titled “Stop, pause and budget”](#stop-pause-and-budget)

* TypeScript

  ```ts
  await pensieve.experiments.stop("exp_0420");                              // finished runs keep their results
  await pensieve.models.update("grade-audit", { monthly_budget_usd: 2000 }); // raise or lower the cap
  await pensieve.models.update("grade-audit", { experimenting: false });     // pause the agents
  ```

* Python

  ```python
  client.experiments.stop("exp_0420")                               # finished runs keep their results
  client.models.update("grade-audit", monthly_budget_usd=2000)      # raise or lower the cap
  client.models.update("grade-audit", experimenting=False)          # pause the agents
  ```

* HTTP

  ```bash
  curl -X POST https://api.pensievelabs.co/v1/experiments/exp_0420/stop -H "Authorization: Bearer $PENSIEVE_API_KEY"
  curl -X PATCH https://api.pensievelabs.co/v1/models/grade-audit \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"monthly_budget_usd": 2000, "experimenting": false}'
  ```

A model’s `budget` shows this month’s cap and what experiments have spent. When it runs out, agents hold their ideas for next month, and `experiments.start` is refused with `conflict` until you raise the cap. While experiments are paused, suggestions are refused the same way. Pausing stops new experiments; running ones finish their current runs. The live version keeps answering either way.

What experiments cost

Each experiment records its compute cost and GPU hours. On the sample model `grade-audit`, its seven experiments range from $9, for an RL test still running, to $66 for a longer context, which was dropped.

# Keys, versions and lists

> How every request is authenticated and versioned, how lists page, and what ids look like.

```bash
curl "https://api.pensievelabs.co/v1/models?limit=2" \
  -H "Authorization: Bearer pk_test_sandbox" \
  -H "Pensieve-Version: 2026-11-08"
```

## Keys

[Section titled “Keys”](#keys)

Send your key as `Authorization: Bearer <key>`.

* `pk_test_` keys reach a sandbox workspace, where models answer from sample data. See [The sandbox](/reference/sandbox/).
* `pk_live_` keys reach your live workspace. Make them in the app’s Settings, or with `keys.create()`; the secret is shown once.

Keep live keys out of code and out of chats with coding agents. Set `PENSIEVE_API_KEY` in your environment; both SDKs read it.

## API versions

[Section titled “API versions”](#api-versions)

Pin the version with `Pensieve-Version: 2026-11-08`. Without the header, you get the version your workspace was created on. Every response says which version answered, in `Pensieve-Version`. The SDKs send the version they were built for.

Within a version, the API only adds: new operations, new optional fields, new fields in answers, and new values in enums such as `status`. Write code that ignores fields it doesn’t know and treats an unknown enum value gracefully.

## Lists

[Section titled “Lists”](#lists)

Every list has the same shape:

```json
{ "object": "list", "data": [ ... ], "has_more": true, "next_cursor": "cur_20" }
```

Pass `limit` (1 to 100, 20 by default) and, for the next page, `cursor` set to the `next_cursor` you got. The SDKs page for you.

## Ids

[Section titled “Ids”](#ids)

| Prefix  | Object                                                    |
| ------- | --------------------------------------------------------- |
| `mdl_`  | Model (calls also take its slug, such as `grade-audit`)   |
| `ver_`  | Version (paths take its label, such as `v7`, not this id) |
| `cl_`   | Call: a decision, or a call your code answered            |
| `bat_`  | Batch                                                     |
| `case_` | Test case                                                 |
| `exp_`  | Experiment                                                |
| `run_`  | Experiment run                                            |
| `rel_`  | Release                                                   |
| `rev_`  | Review case                                               |
| `rt_`   | Frontier re-test                                          |
| `src_`  | Data source                                               |
| `key_`  | API key                                                   |
| `mem_`  | Person in the workspace                                   |
| `whe_`  | Webhook endpoint                                          |
| `evt_`  | Event                                                     |
| `req_`  | Request, in `Pensieve-Request-Id`                         |

Post-training methods have no prefix: a method’s id is its slug, such as `fine-tuning` or one you chose.

Times are ISO 8601 in UTC, such as `2026-10-27T10:00:00.000Z`. Scores are between 0 and 1; leads are in points out of 100.

# Errors and retries

> One error body for every failure, which errors to retry, and how Idempotency-Key makes a retry safe.

```json
{
  "error": {
    "type": "conflict",
    "message": "v7 is live, not v6. Check what changed, then send `expect: \"v7\"`.",
    "param": "expect",
    "request_id": "req_8a4f2fedbd4af9da"
  }
}
```

Every error has this body. `message` is written for a person and says what to do. `param` names the request field the error is about, such as `cases[0].input`, or is `null`. `request_id` is also in the `Pensieve-Request-Id` header of every response; quote it when you ask for help.

## Types

[Section titled “Types”](#types)

| Status | `type`            | Means                                                                                 | Retry?                          |
| ------ | ----------------- | ------------------------------------------------------------------------------------- | ------------------------------- |
| 400    | `invalid_request` | A field is missing, unknown or wrong; `param` names it                                | No: fix the request             |
| 401    | `authentication`  | No key, an unknown key, or a revoked one                                              | No                              |
| 403    | `permission`      | The key can’t do this                                                                 | No                              |
| 404    | `not_found`       | No such model, call, release or route                                                 | No                              |
| 409    | `conflict`        | The object’s state refuses it, such as promoting before sign-off, or a stale `expect` | No: read the state, then decide |
| 429    | `rate_limit`      | Too many requests; `Retry-After` says when to try again                               | Yes                             |
| 501    | `not_available`   | Part of the API, but not served here yet                                              | No                              |
| 500    | `server`          | Something failed on Pensieve’s side                                                   | Yes                             |

## Retries

[Section titled “Retries”](#retries)

The SDKs try again on their own, twice by default: after a dropped connection, a timeout, or a 408, 429, 500, 502, 503 or 504. They wait half a second, then a second, doubling up to 8 seconds, with some jitter. A `Retry-After` header wins, up to a minute. They never retry a 501 or any other 4xx.

## Idempotency

[Section titled “Idempotency”](#idempotency)

Every POST takes an `Idempotency-Key` header. Send the same key again within 24 hours and you get the first answer back, with `Idempotent-Replayed: true`, instead of the work being done twice. Send the same key with a different request and you get a `conflict`.

The SDKs make up a key for every POST and keep it across their retries. Pass your own to make a retry safe across processes, such as a job that may run twice:

```ts
await pensieve.models.decide("grade-audit", { input }, { idempotencyKey: `grade-${trial.task}` });
```

```python
client.models.decide("grade-audit", input=case, idempotency_key=f"grade-{case['task']}")
```

## Rate limits

[Section titled “Rate limits”](#rate-limits)

The sandbox allows 300 requests per key every 10 seconds. Live limits are set per workspace.

# Python SDK

> pensievelab: every API operation as a typed method, sync and async, with retries, paging, capture and webhook checks.

```bash
pip install pensievelab
```

```python
from pensievelab import Pensieve


client = Pensieve()  # reads PENSIEVE_API_KEY
decision = client.models.decide("grade-audit", input=trial, key=trial["task"])
```

It needs Python 3.10 or later, and depends on `httpx` and `pydantic`. Every answer is a typed model generated from the [OpenAPI spec](/openapi.json); fields a newer API adds are kept, so an older SDK keeps working.

## The client

[Section titled “The client”](#the-client)

```python
client = Pensieve(
    api_key="pk_test_yourname",            # or PENSIEVE_API_KEY
    base_url="https://api.pensievelabs.co", # or PENSIEVE_BASE_URL
    timeout=60.0,                           # seconds per try
    max_retries=2,
    http_client=None,                       # your own httpx.Client
    default_headers=None,
    webhook_secret=None,                    # or PENSIEVE_WEBHOOK_SECRET
    on_error=None,                          # errors from capture's background work
)
```

Use it as a context manager, or call `client.close()`. Any call takes `idempotency_key`, `timeout`, `max_retries` and `extra_headers` keyword arguments.

## Async

[Section titled “Async”](#async)

`AsyncPensieve` has the same methods, as coroutines:

```python
import asyncio


from pensievelab import AsyncPensieve


async def main():
    async with AsyncPensieve() as client:
        decision = await client.models.decide("grade-audit", input=trial)
        async for call in client.calls.list(model="grade-audit"):
            ...


asyncio.run(main())
```

## Methods

[Section titled “Methods”](#methods)

Path ids are positional; every other field is a keyword argument named as in the API.

| Resource           | Methods                                                                         |
| ------------------ | ------------------------------------------------------------------------------- |
| `models`           | `list`, `retrieve`, `update`, `decide`                                          |
| `models.routing`   | `update`                                                                        |
| `models.versions`  | `list`, `retrieve`, `weights`                                                   |
| `batches`          | `create`, `retrieve`                                                            |
| `calls`            | `list`, `retrieve`                                                              |
| `outcomes`         | `send`, `send_many`                                                             |
| `corrections`      | `send`                                                                          |
| `captures`         | `create`                                                                        |
| `tests`            | `retrieve`                                                                      |
| `tests.cases`      | `list`, `add`                                                                   |
| `experiments`      | `list`, `start`, `retrieve`, `stop`                                             |
| `experiments.runs` | `list`                                                                          |
| `methods`          | `list`, `create`, `retrieve`, `update`, `remove`                                |
| `releases`         | `list`, `shadow`, `retrieve`, `signoff`, `split`, `promote`, `stop`, `rollback` |
| `review`           | `list`, `retrieve`, `answer`                                                    |
| `tasks`            | `create`                                                                        |
| `frontier`         | `list`                                                                          |
| `sources`          | `list`, `connect`, `retrieve`, `update`, `remove`                               |
| `keys`             | `list`, `create`, `revoke`                                                      |
| `members`          | `list`, `invite`, `update`, `remove`                                            |
| `usage`            | `retrieve`                                                                      |
| `webhooks`         | `list`, `create`, `remove`, `unwrap`                                            |
| `events`           | `list`                                                                          |
| `workspace`        | `retrieve`                                                                      |

Plus the `@client.capture(...)` decorator and `client.flush()`: see [Record the calls you make today](/guides/wrap-your-frontier-call/).

## Lists

[Section titled “Lists”](#lists)

Iterating a list gives every item across pages. The page itself has `data`, `has_more`, `next_cursor` and `get_next_page()`:

```python
for call in client.calls.list(model="grade-audit"):
    ...


page = client.calls.list(model="grade-audit", limit=50)
page.data
if page.has_next_page():
    page = page.get_next_page()
```

## Request ids

[Section titled “Request ids”](#request-ids)

Every answer carries the id of the request that made it, to quote when asking for help:

```python
model = client.models.retrieve("grade-audit")
model._request_id  # 'req_...'
```

## Errors

[Section titled “Errors”](#errors)

```python
import pensievelab


try:
    client.releases.promote(release.id, expect="v7")
except pensievelab.ConflictError as e:
    print(e.param, e.message, e.request_id)
```

`BadRequestError` 400, `AuthenticationError` 401, `PermissionDeniedError` 403, `NotFoundError` 404, `ConflictError` 409, `RateLimitError` 429, `NotAvailableError` 501, `InternalServerError` 5xx, and `APIConnectionError` and `APITimeoutError` when no answer came back. See [Errors and retries](/reference/errors/).

## Webhooks

[Section titled “Webhooks”](#webhooks)

```python
event = client.webhooks.unwrap(request.body, request.headers)  # checks the signature and age
```

See [Webhooks and events](/reference/webhooks/).

# The sandbox

> A workspace of sample models you can call with any pk_test_ key, before any contract.

```bash
curl https://api.pensievelabs.co/v1/models -H "Authorization: Bearer pk_test_sandbox"
```

The sandbox answers every request with sample data: four made-up models, with the experiments behind them, their runs and recipes, versions, releases, data sources and frontier re-tests. No real client, result or score is in it.

## Keys

[Section titled “Keys”](#keys)

* **`pk_test_sandbox`** is public. Everyone who uses it shares one workspace, so others may change what you see.
* **Any other `pk_test_` key you make up**, such as `pk_test_yourname`, gets a workspace of its own, seeded with the same sample data. After `pk_test_`, a key takes letters, digits and underscores, up to 64 of them; anything else gets `401`.
* **`pk_live_` keys** get `401`: “Live keys aren’t served yet”.

## The four models

[Section titled “The four models”](#the-four-models)

**Grade audit** · `grade-audit`

Live on v7, v8 in shadow; seven experiments, three kept. Checks whether a coding task’s grade holds: `grade_holds`, `false_pass`, `false_fail`.

**Agent action approval** · `agent-actions`

v3 in shadow, nothing live yet; four experiments, two kept. Allows, asks about or blocks an agent’s command: `allow`, `ask`, `block`.

**Review comment filter** · `review-comments`

Building: 0.7 points behind the frontier, with three experiments running. Posts or drops a review comment: `post`, `drop`.

**Finding triage** · `finding-triage`

Live on v4, v5 in shadow after drift. Sorts scanner findings: `real`, `false_positive`.

## How it differs from a live workspace

[Section titled “How it differs from a live workspace”](#how-it-differs-from-a-live-workspace)

* **It forgets.** A workspace resets to the sample data after an hour unused, and may reset sooner.
* **It doesn’t wait, and doesn’t train.** Shadow runs and batches finish as soon as they start, so a whole release takes a minute to try. Experiments you start stay queued: no GPU runs behind the sandbox.
* **Anyone can sign off.** Any key can sign off a release; a live workspace takes sign-off only from the people named in the app’s Settings.
* **Added test cases aren’t scored.** Cases you add count toward the test’s size, but no version or frontier model is scored on them again.
* **Early outcomes are kept.** An outcome sent before its call is matched when a call with that id, or that model and key, is recorded, as in a live workspace.
* **Some things aren’t served yet.** Weight downloads answer `501 not_available`. Events are recorded, but no webhooks are sent.
* **Models answer by rule.** The sample models follow simple rules over the case’s text, so a case unlike the samples gets a plausible answer, not a trained one.

# TypeScript SDK

> @pensievelab/sdk: every API operation as a typed method, with retries, paging, capture and webhook checks. No dependencies.

```bash
npm install @pensievelab/sdk
```

```ts
import Pensieve from "@pensievelab/sdk";


const pensieve = new Pensieve(); // reads PENSIEVE_API_KEY
const decision = await pensieve.models.decide("grade-audit", { input: trial, key: trial.task });
```

It has no dependencies, and runs on Node 18 and later, in browsers and on edge runtimes. Types for every request and answer are generated from the [OpenAPI spec](/openapi.json).

## The client

[Section titled “The client”](#the-client)

```ts
new Pensieve({
  apiKey: "pk_test_yourname",             // or PENSIEVE_API_KEY
  baseURL: "https://api.pensievelabs.co", // or PENSIEVE_BASE_URL
  timeout: 60_000,                        // milliseconds per try
  maxRetries: 2,
  fetch: customFetch,                     // any fetch-compatible function
  defaultHeaders: {},
  webhookSecret: "whsec_...",             // or PENSIEVE_WEBHOOK_SECRET
  onError: (error) => {},                 // errors from capture's background work
});
```

Any call takes these options as its last argument: `{ idempotencyKey, timeout, maxRetries, signal, headers }`.

## Methods

[Section titled “Methods”](#methods)

Path ids come first, then an object of fields named as in the API.

| Resource           | Methods                                                                         |
| ------------------ | ------------------------------------------------------------------------------- |
| `models`           | `list`, `retrieve`, `update`, `decide`                                          |
| `models.routing`   | `update`                                                                        |
| `models.versions`  | `list`, `retrieve`, `weights`                                                   |
| `batches`          | `create`, `retrieve`, `createAndWait`                                           |
| `calls`            | `list`, `retrieve`                                                              |
| `outcomes`         | `send`, `sendMany`                                                              |
| `corrections`      | `send`                                                                          |
| `captures`         | `create`                                                                        |
| `tests`            | `retrieve`                                                                      |
| `tests.cases`      | `list`, `add`                                                                   |
| `experiments`      | `list`, `start`, `retrieve`, `stop`                                             |
| `experiments.runs` | `list`                                                                          |
| `methods`          | `list`, `create`, `retrieve`, `update`, `remove`                                |
| `releases`         | `list`, `shadow`, `retrieve`, `signoff`, `split`, `promote`, `stop`, `rollback` |
| `review`           | `list`, `retrieve`, `answer`                                                    |
| `tasks`            | `create`                                                                        |
| `frontier`         | `list`                                                                          |
| `sources`          | `list`, `connect`, `retrieve`, `update`, `remove`                               |
| `keys`             | `list`, `create`, `revoke`                                                      |
| `members`          | `list`, `invite`, `update`, `remove`                                            |
| `usage`            | `retrieve`                                                                      |
| `webhooks`         | `list`, `create`, `remove`, `unwrap`, `sign`                                    |
| `events`           | `list`                                                                          |
| `workspace`        | `retrieve`                                                                      |

Plus `pensieve.capture(model, fn, options)` and `pensieve.flush()`: see [Record the calls you make today](/guides/wrap-your-frontier-call/).

## Lists

[Section titled “Lists”](#lists)

Await a list for one page, or loop over it for every item across pages:

```ts
const page = await pensieve.calls.list({ model: "grade-audit", limit: 50 });
page.data;          // this page
if (page.hasNextPage()) {
  const next = await page.getNextPage();
}


for await (const call of pensieve.calls.list({ model: "grade-audit" })) {
  // every call, fetched a page at a time
}
```

## The raw response

[Section titled “The raw response”](#the-raw-response)

```ts
const { data, response, requestId } = await pensieve.models.retrieve("grade-audit").withResponse();
```

## Errors

[Section titled “Errors”](#errors)

Every error is a class you can catch. API errors carry `status`, `type`, `param` and `requestId`:

```ts
import { ConflictError } from "@pensievelab/sdk";


try {
  await pensieve.releases.promote(release.id, { expect: "v7" });
} catch (error) {
  if (error instanceof ConflictError) console.log(error.param, error.message);
  else throw error;
}
```

`BadRequestError` 400, `AuthenticationError` 401, `PermissionDeniedError` 403, `NotFoundError` 404, `ConflictError` 409, `RateLimitError` 429, `NotAvailableError` 501, `InternalServerError` 5xx, and `APIConnectionError`, `APITimeoutError` and `APIUserAbortError` when no answer came back. See [Errors and retries](/reference/errors/).

## Webhooks

[Section titled “Webhooks”](#webhooks)

```ts
const event = await pensieve.webhooks.unwrap(rawBody, request.headers); // checks the signature and age
```

See [Webhooks and events](/reference/webhooks/).

# Webhooks and events

> Hear about releases, frontier re-tests, experiments and review cases as they happen.

* TypeScript

  ```ts
  const endpoint = await pensieve.webhooks.create({
    url: "https://example.com/hooks/pensieve",
    events: ["release.ready_for_signoff", "frontier.retested"],
  });
  endpoint.secret; // whsec_..., shown once
  ```

* Python

  ```python
  endpoint = client.webhooks.create(
      url="https://example.com/hooks/pensieve",
      events=["release.ready_for_signoff", "frontier.retested"],
  )
  endpoint.secret  # whsec_..., shown once
  ```

* HTTP

  ```bash
  curl https://api.pensievelabs.co/v1/webhooks \
    -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
    -d '{"url": "https://example.com/hooks/pensieve", "events": ["release.ready_for_signoff", "frontier.retested"]}'
  ```

In the sandbox

The sandbox records events but doesn’t send webhooks yet. Read them with `events.list()`, or `GET /v1/events`.

## Events

[Section titled “Events”](#events)

| Type                        | When                                                                |
| --------------------------- | ------------------------------------------------------------------- |
| `release.ready_for_signoff` | A release finished its shadow run and waits for an expert           |
| `release.promoted`          | A version went live                                                 |
| `release.rolled_back`       | A model went back to its previous version                           |
| `frontier.retested`         | A frontier or general model release was scored on every task’s test |
| `model.lead_narrowed`       | A re-test narrowed a model’s lead; agents have started experiments  |
| `experiment.ended`          | An experiment was kept, dropped or stopped                          |
| `review.case_added`         | A case or a release joined the review queue                         |
| `batch.completed`           | A batch finished                                                    |

Each delivery is an event: `id`, `type`, `created_at`, and `data`, the object it is about. The same events are kept for 30 days at `/v1/events`, for catching up after downtime.

## Checking a delivery

[Section titled “Checking a delivery”](#checking-a-delivery)

Each delivery carries a `Pensieve-Signature` header:

```text
Pensieve-Signature: t=1794209235,v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd
```

`v1` is the HMAC-SHA256 of `<t>.<raw body>`, keyed with the endpoint’s secret. Reject a delivery whose signature doesn’t match, or whose `t` is more than five minutes old. The SDKs do both. Pass the raw body exactly as it arrived:

* TypeScript

  ```ts
  app.post("/hooks/pensieve", express.raw({ type: "application/json" }), async (req, res) => {
    const event = await pensieve.webhooks.unwrap(req.body, req.headers); // uses webhookSecret
    if (event.type === "release.ready_for_signoff") notifyReviewers(event.data);
    res.sendStatus(200);
  });
  ```

* Python

  ```python
  @app.post("/hooks/pensieve")
  async def hook(request: Request):
      event = client.webhooks.unwrap(await request.body(), request.headers)  # uses webhook_secret
      if event.type == "release.ready_for_signoff":
          notify_reviewers(event.data)
      return {"ok": True}
  ```

* HTTP

  ```bash
  # T and V1 are the two parts of the Pensieve-Signature header; body.json is the raw body as it arrived.
  EXPECTED=$({ printf '%s.' "$T"; cat body.json; } | openssl dgst -sha256 -hmac "$PENSIEVE_WEBHOOK_SECRET" | awk '{print $NF}')
  [ "$EXPECTED" = "$V1" ] && [ $(( $(date +%s) - T )) -le 300 ] && echo "valid"
  ```

Answer with any 2xx to acknowledge a delivery.