Skip to content
PensievePensievePensieve

Write your own method

Your team may know a way to train for your decision that no built-in method covers. Write it down as a markdown file, and the agents can choose it for an experiment, or run it when you suggest an idea.

  1. Write the file. Start with a # title and a paragraph saying what the method does and when to try it. The rest is yours; the sections Pensieve’s own methods use are a good start.

    methods/curriculum.md
    # Curriculum fine-tuning
    

    Fine-tune on easy cases first, then hard ones. Try it when the model handles common cases
    well but misses the ones your experts argue about.
    

    ## Needs
    

    - Cases more than one expert answered, to tell how hard each one is.
    

    ## Steps
    

    1. Score each training case's difficulty by how often your experts disagreed on it.
    2. Train in stages, from the easiest cases to the hardest.
    3. Score every checkpoint on the held-out test.
    

    ## Settings
    

    - `stages`: how many stages. Default 3.
    - `order`: easy to hard, or hard to easy. Default easy to hard.
    

    ## Checks
    

    - The test's hard cases are scored on their own, and don't get worse.

  2. Add it to your workspace. Its slug is how experiments name it.

    import { readFileSync } from "node:fs";
    

    const method = await pensieve.methods.create({
      slug: "curriculum",
      markdown: readFileSync("methods/curriculum.md", "utf8"),
    });
    console.log(method.name, method.version); // Curriculum fine-tuning 1

  3. Use it. From now on the agents may choose it when their evidence fits the file. To test it now, suggest an idea and name it:

    await pensieve.experiments.start("grade-audit", {
      idea: "Train on the grades reviewers agreed on first, then the disputed ones",
      method: "curriculum",
      max_cost_usd: 60,
    });

    It is judged like any experiment: A and B on the same seeds, scored on your held-out test, and kept only if B wins on every seed and the last check.

Change it

Section titled “Change it”

Send the whole new file. The version goes up by one; experiments already designed keep the version they started with, and each records the version it ran.

await pensieve.methods.update("curriculum", { markdown: readFileSync("methods/curriculum.md", "utf8") });

Keep your method files in your own repository, next to the code that makes the decision, and add or update them from CI. Then a method’s history is in your version control as well as Pensieve’s.

Start from a built-in one

Section titled “Start from a built-in one”

Built-in methods can’t be changed, but they can be copied. Read one, change what you need, and add it under a new slug:

const { markdown } = await pensieve.methods.retrieve("rl");
await pensieve.methods.create({
  slug: "rl-on-refunds",
  markdown: markdown.replace("# RL on outcomes", "# RL on refunds").replace("Default: matching the outcome.", "Default: no refund within 90 days."),
});

Remove it

Section titled “Remove it”

methods.remove("curriculum"). The method leaves the list, and the agents stop choosing it. Experiments that used it keep it on record, with the version they ran, and every version stays readable: methods.retrieve("curriculum", { version: 1 }) still returns the text the agents followed, while a retrieve without version answers not_found. The slug can’t be used again, so a result never leads back to a different file. A method a queued or running experiment uses can’t be removed until that experiment ends.