Skip to content
PensievePensievePensieve

The sandbox

Terminal window
curl https://api.pensievelabs.co/v1/models -H "Authorization: Bearer pk_test_sandbox"

The sandbox answers every request with sample data: four made-up models, with the experiments behind them, their runs and recipes, versions, releases, data sources and frontier re-tests. No real client, result or score is in it.

Keys

Section titled “Keys”
  • pk_test_sandbox is public. Everyone who uses it shares one workspace, so others may change what you see.
  • Any other pk_test_ key you make up, such as pk_test_yourname, gets a workspace of its own, seeded with the same sample data. After pk_test_, a key takes letters, digits and underscores, up to 64 of them; anything else gets 401.
  • pk_live_ keys get 401: “Live keys aren’t served yet”.

The four models

Section titled “The four models”

Grade audit · grade-audit

Live on v7, v8 in shadow; seven experiments, three kept. Checks whether a coding task’s grade holds: grade_holds, false_pass, false_fail.

Agent action approval · agent-actions

v3 in shadow, nothing live yet; four experiments, two kept. Allows, asks about or blocks an agent’s command: allow, ask, block.

Review comment filter · review-comments

Building: 0.7 points behind the frontier, with three experiments running. Posts or drops a review comment: post, drop.

Finding triage · finding-triage

Live on v4, v5 in shadow after drift. Sorts scanner findings: real, false_positive.

How it differs from a live workspace

Section titled “How it differs from a live workspace”
  • It forgets. A workspace resets to the sample data after an hour unused, and may reset sooner.
  • It doesn’t wait, and doesn’t train. Shadow runs and batches finish as soon as they start, so a whole release takes a minute to try. Experiments you start stay queued: no GPU runs behind the sandbox.
  • Anyone can sign off. Any key can sign off a release; a live workspace takes sign-off only from the people named in the app’s Settings.
  • Added test cases aren’t scored. Cases you add count toward the test’s size, but no version or frontier model is scored on them again.
  • Early outcomes are kept. An outcome sent before its call is matched when a call with that id, or that model and key, is recorded, as in a live workspace.
  • Some things aren’t served yet. Weight downloads answer 501 not_available. Events are recorded, but no webhooks are sent.
  • Models answer by rule. The sample models follow simple rules over the case’s text, so a case unlike the samples gets a plausible answer, not a trained one.