Skip to content
PensievePensievePensieve

Routing and fallback

await pensieve.models.routing.update("grade-audit", { mode: "model", threshold: 0.8 });

Every model has routing. It decides whether a call’s answer should come from the expert model or from your own code, and every decision says which in routed.

Mode What happens routed
observe Your code answers every call. Calls are recorded, through capture. A new model starts here Always fallback, unless you pin version
model The expert model answers, unless its confidence is under threshold model, or fallback when unsure
split The model answers split_percent of calls, unless its confidence is under threshold; your code answers the rest model or fallback

Routing’s split shares calls between the model and your own code. A release’s split is a different thing: it shares calls between two versions of the model.

The threshold

Section titled “The threshold”

threshold is the confidence under which a call falls back to your code, in model and split modes alike. A higher threshold sends more calls to your code and fewer wrong answers to your users. Every call that falls back is recorded with your code’s answer, and those calls feed the next experiments: the cases the model is unsure about become training data.

Watch the share of calls that fall back in the app, or with usage.retrieve(). If it climbs, the cases your software sees may have changed: outcomes show whether the model is still right.

Calls with no live version

Section titled “Calls with no live version”

A model that has no live version yet answers only when you pin one, which is useful for trying a candidate:

await pensieve.models.decide("agent-actions", { input: "Command: DROP TABLE users on prod", version: "v3" });

Without version, the call is refused with conflict, and the message names a candidate to try.