Methodology
What we actually score
PumaAI prefers small reproducible experiments, visible limitations, and a clear line between what we measured and what we inferred. getEvals follows that. The pages here describe the private app and the browser-agent harness. They do not describe a model index we have not run.
Private suites
A suite is cases and graders. A case is an input and an expected reference. The graders in this release are exact match, contains, regex, a JSON field, and a keyword list. You paste outputs, or load the demo answers, and the page scores them in the browser. The suite stays in localStorage on that device. Nothing is uploaded.
Browser-agent harness
The public bench is eleven tasks on fixture sites served from localhost. A task is a start path, an instruction, and a checker. A pass is the checker returning true: a number, exact text, JSON fields, or the form that was submitted. The environment is Playwright, with a step budget of 20 and the actions click, type, select, check, goto, and back.
Published runs, read from the result files: heuristic-scripted (baseline) passes 7 of 11; naive-first-match (baseline) passes 1 of 11. Tokens and cost on a run are whatever that file records. The language-model adapter is in the repo. It has a row on the leaderboard only after a results file is added and the site is rebuilt.
The task list, the per-task grid, and the commands are on the browser agent page. Source is harness/browser-agent-eval/.
What a number is
A score on this site is a count of checker passes on that task list, or a grader result on a suite you wrote. It describes that run. It does not travel to a different task, a different site, or a different harness. It is not a substitute for a private suite on your own work.