Measurement, not hypothesis

Task design

Real-world work is messy. Our benches try to reflect that, which means looking at more than a single multiple-choice score:

• Tool use: Can the model call the right tools to solve the problem?
• Modalities: Images, tables, files — not only text.
• Reasoning: Models now spend tokens thinking. Does that help on the actual task?
• Long context: Legal documents, large codebases, long threads.
• Long horizon: Work that takes minutes or hours, not one shot.

Public and private sets

Open test sets leak. Once a bench is in a training corpus, the score stops meaning what you think it means. For most benches we keep three layers:

• Public validation set: Open, so you can see the kind of samples we use.
• Private validation set: Held back for internal checks. We show that it correlates with the published suite.
• Test set: Private at all times. This is the only set we publish scores from.

Metrics

• Accuracy: Strict checks, and rubric-based LLM-as-a-judge where a strict match is the wrong tool.
• Latency: Time to a complete response.
• Cost: What it costs to run the model from an API.
• More: Tool-use stats, qualitative error notes, and pairwise comparisons.

Error bars

We report standard errors with scores. For a single run, that is the standard error of the mean over instance scores. For multiple runs, we estimate uncertainty over runs. For composite benches, we pool weighted variance from each part.

Evaluating agent harnesses

Models rarely work alone. They sit in a harness — tools, memory, a loop. PumaAI researches how to evaluate those systems. Our benches test tool-calling, multi-turn flows, coding, and computer-use, because that is closer to how people actually run agents.

A score is a measurement on a task set. It is not a forecast, and it is not a substitute for running your own private suite on your own work.