Record the calls you make today

TypeScript

import Pensieve from "@pensievelab/sdk";

const pensieve = new Pensieve();

// Before: const result = await callFrontierModel(trial);
const auditGrade = pensieve.capture("grade-audit", callFrontierModel, {
  answer: (result) => result.verdict,               // your result → one of the model's answers
  use: (decision) => ({ verdict: decision.answer }), // the model's answer → your result
  key: (trial) => trial.task,                        // a stable id for the case
  providerModel: "frontier-large",
});

const result = await auditGrade(trial); // same arguments, same result

Python

from pensievelab import Pensieve

client = Pensieve()

@client.capture(
    "grade-audit",
    answer=lambda result: result["verdict"],            # your result → one of the model's answers
    use=lambda decision: {"verdict": decision.answer},  # the model's answer → your result
    key=lambda trial: trial["task"],                    # a stable id for the case
    provider_model="frontier-large",
)
def audit_grade(trial):
    return call_frontier_model(trial)

result = audit_grade(trial)  # same arguments, same result

HTTP

Without an SDK, keep calling your function, and record each call yourself:

curl https://api.pensievelabs.co/v1/captures \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "grade-audit", "calls": [{"input": {"task": "rust-2291", "grade": "pass"}, "answer": "grade_holds", "key": "rust-2291", "provider_model": "frontier-large", "latency_ms": 2140}]}'

Send up to 1,000 calls in one request. Once the model is live, call decide first and fall back to your function when routed is fallback.

In the sandbox, grade-audit already routes calls to the model, so the wrapped function wouldn’t run and its answers wouldn’t be recorded. Set the sample model to observe in your sandbox workspace first:

TypeScript

await pensieve.models.routing.update("grade-audit", { mode: "observe" }); // sandbox only

Python

client.models.routing.update("grade-audit", mode="observe")  # sandbox only

HTTP

curl -X PATCH https://api.pensievelabs.co/v1/models/grade-audit/routing \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"mode": "observe"}'

Most teams already make the decision, with a frontier model or a set of rules. That traffic is the best data source there is: real cases, in the shape the model will see them. capture wraps the function that makes the decision, records every call, and returns the same result, so the rest of your code doesn’t change.

The recorded calls, with the outcomes you send back, become your test and the data experiments train on. Once a recipe wins, the same wrapper lets the model answer.

What happens on each call

What capture does depends on the model’s routing, which you change in the app or through the API.

Routing	Who answers	What is recorded
observe (a new model)	Your function	The case and your function’s answer
model	The expert model, through use	The decision. When the model is unsure, your function answers, and that is recorded too
split	The model for the share of calls you set, your function for the rest	Both

While the model observes, the agents experiment on the recorded calls. Once a version passes its gates and you switch routing to model, the model answers. Your function still answers whenever the model’s confidence is under the threshold.

If Pensieve is down, your code answers

capture never puts Pensieve in the way of your users. It gives Pensieve 2 seconds, with no retries in your request’s path, and falls back to your function on any failure. Nothing from Pensieve is thrown or raised into your code. Failures go to the client’s onError (on_error in Python), which logs a warning by default.

The options

Option	Required	What it does
answer	Yes	Turns your function’s result into one of the model’s answer values
use	No	Turns a decision into your function’s result. Without it, the model only observes, whatever its routing
key	No	A stable id for the case, so you can send outcomes by it later
input	No	Picks the case from your function’s arguments. Defaults to the first argument
providerModel	No	The model your function calls, kept with each recorded call

Find a model’s answer values, outcomes and input schema with models.retrieve:

TypeScript

const model = await pensieve.models.retrieve("grade-audit");
model.answers.map((a) => a.value); // ["grade_holds", "false_pass", "false_fail"]
model.outcomes;                     // ["lab_accepted", "lab_rejected"]

Python

model = client.models.retrieve("grade-audit")
[a.value for a in model.answers]  # ['grade_holds', 'false_pass', 'false_fail']
model.outcomes                    # ['lab_accepted', 'lab_rejected']

HTTP

curl https://api.pensievelabs.co/v1/models/grade-audit -H "Authorization: Bearer $PENSIEVE_API_KEY"

Sending what was recorded

Recorded calls go out in the background, in batches of 100 or every 2 seconds. A short-lived script or a test should send what is left before it exits:

TypeScript

await pensieve.flush();

Python

client.flush()  # also runs when your program exits, and on client.close()

HTTP

Each request to /v1/captures is sent when you make it.

Next

• Send outcomes and corrections, so every version is checked against what really happened.
• Routing and fallback, to choose when the model takes over.