Skip to content
PensievePensievePensieve

Add test cases

POST
/v1/models/{model}/test/cases
curl --request POST \
  --url https://api.pensievelabs.co/v1/models/grade-audit/test/cases \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "cases": [ { "input": "Grader: PASS · patch deletes the failing assertion", "answer": "false_pass", "by": "maya@example.com" } ] }'

New cases are held out from training. Every version and frontier model is scored on them again.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
model
required
string

The model’s slug, such as grade-audit, or its mdl_ id.

Example 

grade-audit

Header Parameters

Section titled “Header Parameters”
Idempotency-Key
string
<= 255 characters

Any unique string. Sending it again within 24 hours returns the first response.

Request Bodyrequired

Section titled “Request Bodyrequired”
Media typeapplication/json
object
cases
required
Array<object>
>= 1 items <= 1000 items
object
input
required
One of:
object
key
additional properties
any
answer
required
string
by
string

Example 

{
  "cases": [
    {
      "input": "Grader: PASS · patch deletes the failing assertion",
      "answer": "false_pass",
      "by": "maya@example.com"
    }
  ]
}

Responses

Section titled “Responses”

200

Section titled “200”

The ids of the added cases.

Media typeapplication/json
object
object
required
Allowed value: receipt
received
required

How many items were recorded.

integer
ids
required

For each item in order, the call or test case it was recorded as (for a capture with decision_id, that decision’s call). For an outcome, the call it was matched to, or null while it waits for its call: it is matched when a call with that id, or that model and key, is recorded.

Array<string | null>

Example 

{
  "object": "receipt"
}

default

Section titled “default”

Something went wrong; error.type says what.

Media typeapplication/json
object
error
required
object
type
required
string
Allowed values: invalid_request authentication permission not_found conflict rate_limit not_available server
message
required
string
param

The request field the error is about.

string | null
request_id
required
string

Example 

{
  "error": {
    "type": "invalid_request"
  }
}