Skip to content
PensievePensievePensieve

Data sources

for await (const s of pensieve.sources.list()) {
  console.log(s.name, s.via, s.volume, "·", s.held_out, "·", s.models);
}

Experiments are only as good as the data they run on. A data source is anything that holds your cases, your experts’ decisions, or what happened next.

What makes good data

Section titled “What makes good data”
Kind What it gives experiments A sample source
Cases What the model will see: the input of every decision Agent trials, a Kafka stream of 240k a day
Decisions What your experts decided, as examples to learn from Reviewer verdicts, 3.1M in Postgres over two years
Outcomes What really happened next, which can become a reward lab_accepted and lab_rejected, sent back per call
Context What your experts looked at to decide Grader logs, 1.2 TB of files

Outcomes matter most. Outcomes catch what experts miss. On one sample model, the labs sent back tasks whose wrong grades the reviewers had let through, so an experiment now uses whether each task was accepted as the reward.

If you have no history yet, start by recording the calls you make today. Each recorded call is a case, and the outcomes you send back complete it.

Held out from training

Section titled “Held out from training”

Every source says what is kept out of training, in held_out: the last 60 days of a stream, or the cases set aside as your test. No experiment trains on held-out data, and every experiment checks that none leaked in.

Where it is read

Section titled “Where it is read”

Pensieve’s reader runs where your data lives: in your cloud, next to the database or bucket. Credentials are set in the app, never sent through the API. Models train and run where you choose: in your cloud or in Pensieve’s.