Skip to content
PensievePensievePensieve

Quickstart

  1. Install an SDK, and set a sandbox key. Any key that starts pk_test_ reaches the sandbox. After pk_test_, use letters, digits and underscores, up to 64 of them. Make one up, and you get a workspace of your own with four sample models and their experiments.

    Terminal window
    npm install @pensievelab/sdk
    export PENSIEVE_API_KEY=pk_test_yourname

    The packages aren’t on npm and PyPI yet. Until they are, the HTTP tab works as it is.

  2. See what the agents kept. grade-audit checks whether a coding task’s grade holds. List the experiments that won:

    import Pensieve from "@pensievelab/sdk";
    

    const pensieve = new Pensieve(); // reads PENSIEVE_API_KEY
    

    for await (const e of pensieve.experiments.list("grade-audit", { status: "kept" })) {
      console.log(e.id, e.method, e.lead_points, e.idea);
    }
     
    exp_0418 fine-tuning 1.3 Weight overturned grades 4×
    exp_0416 preference-tuning 0.9 Preference-tune on right and wrong grades
    exp_0414 distillation 0.4 Distil from a 70B open teacher first

  3. Read how one was judged. Each experiment trained the current recipe (A) and the change (B) on three seeds, then checked B on cases neither had seen:

    for await (const run of pensieve.experiments.runs.list("exp_0418")) {
      console.log(run.label, run.arm, run.seed, run.score ?? run.lead_points);
    }

  4. Read the recipe they add up to. The live version, v7, is made of the three kept changes:

    const v7 = await pensieve.models.versions.retrieve("grade-audit", "v7");
    for (const step of v7.recipe?.steps ?? []) console.log(step.method, step.description);

  5. Call the model. The answer is always one of its fixed values, so your code can act on it:

    const decision = await pensieve.models.decide("grade-audit", {
      input: { task: "rust-2291", grade: "pass", log: "Test change: expected page size 20 → 21" },
      key: "rust-2291",
    });
    console.log(decision.answer, decision.confidence, decision.version); // false_pass 0.94 v7

  6. Send back what happened. When the lab rejects the task, say so. Outcomes check every version against reality, and become rewards and training data for the next experiments.

    await pensieve.outcomes.send(decision.id, { outcome: "lab_rejected" });

Next

Section titled “Next”