Send outcomes and corrections

TypeScript

// What really happened, by call id
await pensieve.outcomes.send(decision.id, { outcome: "lab_rejected", note: "Lab found the edited test." });

// An expert's answer for the case
await pensieve.corrections.send(decision.id, { answer: "false_pass", by: "maya@example.com" });

Python

# What really happened, by call id
client.outcomes.send(decision.id, outcome="lab_rejected", note="Lab found the edited test.")

# An expert's answer for the case
client.corrections.send(decision.id, answer="false_pass", by="maya@example.com")

HTTP

# Call the model and keep the call's id
CALL_ID=$(curl -s https://api.pensievelabs.co/v1/models/grade-audit/decide \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"input": {"task": "rust-2291", "grade": "pass"}, "key": "rust-2291"}' | jq -r .id)

curl https://api.pensievelabs.co/v1/calls/$CALL_ID/outcome \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"outcome": "lab_rejected", "note": "Lab found the edited test."}'

curl https://api.pensievelabs.co/v1/calls/$CALL_ID/correction \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"answer": "false_pass", "by": "maya@example.com"}'

Two kinds of feedback drive the experiments, and keep every version honest:

• An outcome is what really happened after the call: the lab accepted the task or sent it back, the engineer approved the command, the finding was fixed. Outcomes can become the reward an RL experiment trains on, and every version is checked against them, which is how drift is caught before your users notice.
• A correction is the answer an expert would have given. It becomes training data, and can become a test case.

Each model lists the outcomes it takes, in outcomes. A later outcome for the same call replaces the earlier one.

By your own key, in bulk

You may not keep Pensieve’s call id. Pass your own key when you decide or capture, then send outcomes by it, up to 1,000 at a time:

TypeScript

await pensieve.outcomes.sendMany({
  items: [
    { model: "grade-audit", key: "rust-2291", outcome: "lab_rejected" },
    { call_id: "cl_seed_grade_02", outcome: "lab_accepted" },
  ],
});

Python

client.outcomes.send_many(items=[
    {"model": "grade-audit", "key": "rust-2291", "outcome": "lab_rejected"},
    {"call_id": "cl_seed_grade_02", "outcome": "lab_accepted"},
])

HTTP

curl https://api.pensievelabs.co/v1/outcomes \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"items": [{"model": "grade-audit", "key": "rust-2291", "outcome": "lab_rejected"}, {"call_id": "cl_seed_grade_02", "outcome": "lab_accepted"}]}'

An outcome may arrive before its call, for example when your code records calls in batches. Pensieve keeps it and matches it when the call arrives. The receipt shows which call each item matched, or null when it is still waiting.

The review queue

Pensieve sends some cases to your experts: calls where the model was unsure (unsure), calls where the outcome went against it (disagreement), and releases waiting for sign-off (signoff). Experts answer in the app, or your own review tool can answer through the API. A call’s case takes one of the model’s answers; a sign-off case takes only approve or reject. This loop answers the unsure cases with what your reviewer chose in your own tool:

TypeScript

for await (const c of pensieve.review.list({ state: "open", model: "grade-audit", kind: "unsure" })) {
  const answer = await askReviewer(c); // your review tool: "grade_holds", "false_pass" or "false_fail"
  await pensieve.review.answer(c.id, { answer, by: "maya@example.com", add_to_test: true });
}

Python

for case in client.review.list(state="open", model="grade-audit", kind="unsure"):
    answer = ask_reviewer(case)  # your review tool: "grade_holds", "false_pass" or "false_fail"
    client.review.answer(case.id, answer=answer, by="maya@example.com", add_to_test=True)

HTTP

curl "https://api.pensievelabs.co/v1/review?state=open&model=grade-audit&kind=unsure" -H "Authorization: Bearer $PENSIEVE_API_KEY"

# Then answer each case with your reviewer's answer
curl https://api.pensievelabs.co/v1/review/$CASE_ID/answer \
  -H "Authorization: Bearer $PENSIEVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"answer": "false_pass", "by": "maya@example.com", "add_to_test": true}'

add_to_test also holds the case out of training, as a test case.