Skip to content
PensievePensievePensieve

Assign a task

POST
/v1/tasks
curl --request POST \
  --url https://api.pensievelabs.co/v1/tasks \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "name": "Flaky test triage", "decision": "Decide whether a failing CI test is a real bug or a flaky test, before it pages an engineer.", "answers": [ { "value": "real_bug" }, { "value": "flaky_test" } ], "test": { "held_out_cases": 2000, "metric": "Right call on each failing test" }, "runs_in": "your_vpc", "monthly_budget_usd": 1500 }'

Describe a decision your software makes. Pensieve’s agents run experiments on your data to find the post-training for it, starting with a first recipe that is queued at once. Kept changes add up to a recipe; a trained recipe is a version. A version ships once it beats the live version (if there is one) and the best frontier model on your test, runs in shadow, and is signed off by your expert.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Header Parameters

Section titled “Header Parameters”
Idempotency-Key
string
<= 255 characters

Any unique string. Sending it again within 24 hours returns the first response.

Request Bodyrequired

Section titled “Request Bodyrequired”
Media typeapplication/json
object
name
required
string
>= 2 characters <= 80 characters
decision
required

The judgement call your software makes.

string
>= 10 characters <= 1000 characters
answers
required
Array<object>
>= 2 items <= 20 items
object
value
required
string
/^[a-z][a-z0-9_]*$/
label
string
description
string
outcomes
Array<string>
sources

Ids of connected sources it learns from.

Array<string>
test
object
held_out_cases
integer
>= 100
metric
string
runs_in
string
default: your_vpc
Allowed values: your_vpc pensieve_cloud
monthly_budget_usd
number
example

One example case, as text.

string

Example 

{
  "name": "Flaky test triage",
  "decision": "Decide whether a failing CI test is a real bug or a flaky test, before it pages an engineer.",
  "answers": [
    {
      "value": "real_bug"
    },
    {
      "value": "flaky_test"
    }
  ],
  "test": {
    "held_out_cases": 2000,
    "metric": "Right call on each failing test"
  },
  "runs_in": "your_vpc",
  "monthly_budget_usd": 1500
}

Responses

Section titled “Responses”

200

Section titled “200”

The new model, building.

Media typeapplication/json
object
id
required
string
object
required
Allowed value: model
slug
required
string
name
required
string
decision
required

The decision the model makes, in one sentence.

string
status
required

building: no version beats the frontier yet. shadow: a version runs beside live traffic. live: a version takes traffic. paused: taken off traffic in the app, so your code answers every call until it is resumed. archived: retired in the app; it answers no calls, and lists leave it out unless you ask for status=archived.

string
Allowed values: building shadow live paused archived
live_version
required
string | null
answers
required
Array<object>
object
value
required

What your code receives.

string
label
required
string
description
string | null
outcomes
required

The outcomes you send back for its calls.

Array<string>
input_schema

JSON Schema for input, when the model takes structured cases.

object | null
metric
required
string
lead
required
One of:

The live version (or best candidate) against the best frontier model, on your test.

object
model_score
required
number
frontier_score
required
number
points
required

The lead in points out of 100; negative when behind.

number
frontier_model
required

The best frontier model at the last re-test.

string
tested_at
required
string format: date-time
routing
required
object
object
required
Allowed value: routing
model
required
string
mode
required
string
Allowed values: observe model split
threshold
required

Under this confidence a call falls back.

number
<= 1
split_percent
required
integer | null
<= 100
updated_at
required
string format: date-time
experimenting
required

Whether Pensieve’s agents are running experiments on it. Pause and resume them with models.update.

boolean
budget
required

Compute for experiments this month. Serving is not counted here.

object
monthly_usd
required

The cap; null for none.

number | null
spent_usd
required
number
owner
required
string
calls_per_day
required
integer
created_at
required
string format: date-time

Example 

{
  "id": "mdl_7f3a91c2",
  "object": "model",
  "slug": "grade-audit",
  "name": "Grade audit",
  "status": "building",
  "live_version": "v7",
  "answers": [
    {
      "value": "false_pass",
      "label": "False pass"
    }
  ],
  "metric": "Right verdict on the grade",
  "routing": {
    "object": "routing",
    "mode": "observe"
  },
  "owner": "Environments team"
}

default

Section titled “default”

Something went wrong; error.type says what.

Media typeapplication/json
object
error
required
object
type
required
string
Allowed values: invalid_request authentication permission not_found conflict rate_limit not_available server
message
required
string
param

The request field the error is about.

string | null
request_id
required
string

Example 

{
  "error": {
    "type": "invalid_request"
  }
}