Skip to content
PensievePensievePensieve

Suggest an experiment

POST
/v1/models/{model}/experiments
curl --request POST \
  --url https://api.pensievelabs.co/v1/models/grade-audit/experiments \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "idea": "When two reviewers disagree, train on the senior reviewer'\''s grade", "method": "fine-tuning", "max_cost_usd": 80 }'

An idea for the agents to test, such as a data source to weight differently or a reward to try. They write the hypothesis, pick the method if you don’t, build B’s data, and run A and B on three seeds. Refused with conflict while the model’s experiments are paused, or once this month’s budget is spent.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
model
required
string

The model’s slug, such as grade-audit, or its mdl_ id.

Example 

grade-audit

Header Parameters

Section titled “Header Parameters”
Idempotency-Key
string
<= 255 characters

Any unique string. Sending it again within 24 hours returns the first response.

Request Bodyrequired

Section titled “Request Bodyrequired”
Media typeapplication/json
object
idea
required

The idea, in a line. The agents write the hypothesis and design the runs.

string
>= 3 characters <= 500 characters
method

A post-training method’s slug, such as fine-tuning. See List methods for what your workspace has.

string
<= 64 characters /^[a-z0-9]+(-[a-z0-9]+)*$/
max_cost_usd

Stop the experiment if its compute would cost more than this. It also counts against the model’s monthly budget.

number
>= 1

Example 

{
  "idea": "When two reviewers disagree, train on the senior reviewer's grade",
  "method": "fine-tuning",
  "max_cost_usd": 80
}

Responses

Section titled “Responses”

200

Section titled “200”

The experiment, queued or running.

Media typeapplication/json

One controlled test of a change to the model’s post-training. A is the current recipe; B is the current recipe with one change. Both are trained on the same data and seeds and scored on your held-out test. A change is kept only if B beats A on every seed and on a last check against cases neither has seen.

object
id
required
string
object
required
Allowed value: experiment
model
required
string
status
required
string
Allowed values: queued running kept dropped stopped
idea
required

The idea, in a line.

string
hypothesis
required

What the agents expect, and why it would beat the current recipe. Null until they design it.

string | null
method
required
One of:

A post-training method’s slug, such as fine-tuning. See List methods for what your workspace has.

string
<= 64 characters /^[a-z0-9]+(-[a-z0-9]+)*$/
method_version
required

The version of the method’s file the experiment ran. Retrieve that version to read exactly what the agents followed.

integer | null
>= 1
lever
required

The part of the recipe the change pulls on, such as Data mix, Reward, Synthetic data or Model size.

string | null
control
required

A: the current recipe.

string | null
variant
required

B: the current recipe with the change.

string | null
prep
required

The step that builds B’s data or model before training, such as writing practice cases or building a reward.

string | null
seeds_done
required
integer
seeds_total
required
integer
gains
required

B minus A on each finished seed, in points on your test.

Array<number>
lead_points
required

Kept: B’s lead on the last check. Dropped: the mean of gains. Null while queued or running, and when stopped.

number | null
throughput
required

B’s cases per second as a multiple of A’s. Under 1 is slower to serve.

number | null
checks
required

What was checked before the result counted, such as no test case leaking into training.

Array<string>
cost_usd
required

Compute spent so far.

number
max_cost_usd
required

Its cost cap, if one was set. It stops before its compute would cost more.

number | null
gpu_hours
required
number
why
required

The evidence that led the agents to try it.

string | null
result
required

What happened: kept and part of a version, or dropped and why.

string | null
started_at
required
string | null format: date-time
ended_at
required
string | null format: date-time

Example 

{
  "id": "exp_0420",
  "object": "experiment",
  "status": "queued",
  "method": "fine-tuning",
  "lever": "Reward"
}

default

Section titled “default”

Something went wrong; error.type says what.

Media typeapplication/json
object
error
required
object
type
required
string
Allowed values: invalid_request authentication permission not_found conflict rate_limit not_available server
message
required
string
param

The request field the error is about.

string | null
request_id
required
string

Example 

{
  "error": {
    "type": "invalid_request"
  }
}