Private evals, on your device

What this release includes

• Eval suites — one suite per surface or workflow (support triage, contract extract, coding agent, and so on).
• Cases — input + expected reference, with tags for slicing later.
• Graders — exact match, contains, regex, JSON field equality, and JSON keyword lists.
• Runs — paste outputs (or load demo answers), score instantly, and review per-case grader detail.
• Import / export — move suites as JSON so eval definitions can live next to your product code.

How to use it

Start from the seeded support-triage suite, replace the cases with your own traffic, tune graders until false positives and false negatives match your risk tolerance, then run every prompt or model change through the same suite before you ship.

Privacy model

This release runs entirely in the browser. Suite data is stored in localStorage on the device you use. Nothing is uploaded to getEvals servers. That is the same local-first instinct as Puma Browser and PumaAI: AI on your terms. Export JSON when you want a backup or to share a suite with a teammate.

What comes next

This is an experiment. Shared workspaces, API runners, and LLM-as-judge graders with audit trails may follow. If you want to try it with us, write the founders.