Follow and steer experiments

for await (const e of pensieve.experiments.list("grade-audit", { status: "running" })) {
  console.log(e.id, e.idea, `${e.seeds_done}/${e.seeds_total} seeds`, e.gains);
}

for e in client.experiments.list("grade-audit", status="running"):
    print(e.id, e.idea, f"{e.seeds_done}/{e.seeds_total} seeds", e.gains)

curl "https://api.pensievelabs.co/v1/models/grade-audit/experiments?status=running" \
  -H "Authorization: Bearer $PENSIEVE_API_KEY"

Agents run experiments on their own, within the model’s budget. You can follow every one, and steer them with what your team knows.

Follow a run

Each experiment lists its runs: the prep step, A and B on every seed, and the last check. A training run shows its steps so far; a finished one shows its score on your test.

for await (const run of pensieve.experiments.runs.list("exp_0420")) {
  console.log(run.kind, run.arm, run.seed, run.status, run.steps_done, run.steps_total, run.score);
}

for run in client.experiments.runs.list("exp_0420"):
    print(run.kind, run.arm, run.seed, run.status, run.steps_done, run.steps_total, run.score)

curl https://api.pensievelabs.co/v1/experiments/exp_0420/runs -H "Authorization: Bearer $PENSIEVE_API_KEY"

To hear when one ends, subscribe a webhook to experiment.ended.

Suggest an idea

Your team knows things the data doesn’t say: a new kind of case coming next quarter, a signal nobody has used, a reason the experts disagree. Suggest it in a line. Name a method and a cost cap if you have a view: a built-in method, or one you wrote. The agents write the hypothesis, build B and run it like their own.

const e = await pensieve.experiments.start("grade-audit", {
  idea: "When two reviewers disagree, train on the senior reviewer's grade",
  method: "fine-tuning",
  max_cost_usd: 80,
});

e = client.experiments.start(
    "grade-audit",
    idea="When two reviewers disagree, train on the senior reviewer's grade",
    method="fine-tuning",
    max_cost_usd=80,
)

curl https://api.pensievelabs.co/v1/models/grade-audit/experiments \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"idea": "When two reviewers disagree, train on the senior reviewer'\''s grade", "method": "fine-tuning", "max_cost_usd": 80}'

It is judged like any other: kept only if it wins on every seed and the last check.

Stop, pause and budget

await pensieve.experiments.stop("exp_0420");                              // finished runs keep their results
await pensieve.models.update("grade-audit", { monthly_budget_usd: 2000 }); // raise or lower the cap
await pensieve.models.update("grade-audit", { experimenting: false });     // pause the agents

client.experiments.stop("exp_0420")                               # finished runs keep their results
client.models.update("grade-audit", monthly_budget_usd=2000)      # raise or lower the cap
client.models.update("grade-audit", experimenting=False)          # pause the agents

curl -X POST https://api.pensievelabs.co/v1/experiments/exp_0420/stop -H "Authorization: Bearer $PENSIEVE_API_KEY"
curl -X PATCH https://api.pensievelabs.co/v1/models/grade-audit \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"monthly_budget_usd": 2000, "experimenting": false}'

A model’s budget shows this month’s cap and what experiments have spent. When it runs out, agents hold their ideas for next month, and experiments.start is refused with conflict until you raise the cap. While experiments are paused, suggestions are refused the same way. Pausing stops new experiments; running ones finish their current runs. The live version keeps answering either way.