About getEvals

Why this exists

Most public benches measure exams. Most private work never gets a bench at all. People still have to decide whether a model, a prompt, or an agent harness is good enough for the task in front of them.

getEvals is a small attempt to close that gap: private suites you can run on your device, and public benches on work people actually do. We prefer small reproducible experiments, visible limitations, and a clear line between measurement and hypothesis — the same habits as the PumaAI lab.

What we build

The app is a local loop: cases from your domain, graders that encode what “good” means, and runs that catch regressions before they ship. Data stays in the browser. Nothing is uploaded.

Publicly, we score models on real-world tasks — legal research, finance, coding, healthcare admin, and more. Published scores use privately held test sets so the industry is not training to the answer key.

Some benches are for the public good. Public Benefits Bench asks whether AI can be trusted to answer SNAP questions for the families who rely on the program.

How we work

PumaAI is a small research lab led by Yuriy Dybskiy, founder of Puma. The lab studies private local inference, observable agent harnesses, and memory store. getEvals is the evals experiment next to that work.

Puma’s values are optimism, kindness, curiosity, speed, and grit. We are not trying to become the measurement institution of a trillion-dollar market. We are trying to make useful evals that stay in your hands.

The Puma family

• Puma.tech — AI, on your terms.
• PumaAI — research lab. Private local inference, observable agents, memory store.
• Puma Browser — local and cloud AI, right in your browser.
• Puma OS — a private AI operating system for your company.