Paradigm Frontiers 2026
Browser agents, scored locally
11 deterministic tasks on fixture sites served from localhost. Checkers score the answer. This is a Puma experiment: the run stays on your machine, and the number means only what these tasks measure.
Baselines
heuristic-scripted — Reads labels, fills quoted fields, and clicks named controls. No model and no API key. Passes 7 of 11 (64%). Misses: cheapest item, table lookup, count reviews, and event date iso.
naive-first-match — No clicks. Answers with the first number or time on the start page. Passes 1 of 11 (9%). The other 10 tasks fail.
The language-model adapter ships in the harness and has no published run. Tokens and cost on the baselines are zero because no API was called.
Leaderboard
Ranked by success rate, then by fewer average steps — the same order as the harness leaderboard. Each row is one results file under harness/browser-agent-eval/results/.
| # | Agent | Success | Avg steps | Avg time | Tokens | Cost | Run |
|---|---|---|---|---|---|---|---|
| 1 | heuristic-scripted Baseline Scripted heuristic | 7/11 · 64% | 1.18 | 0.103s | 0 | $0.0000 | 2026-10-13 17:57 -07:00 |
| 2 | naive-first-match Baseline Naive first match | 1/11 · 9% | 0.0 | 0.0s | 0 | $0.0000 | 2026-10-13 17:57 -07:00 |
Schema agent-eval-harness/results@1. A pass is the checker returning true. Rebuild this page with python3 build.py after a new results file lands.
Per-task results
A check is a pass. A cross is a fail. Column titles carry the task category and the instruction the agent saw.
| Agent | price_lookup | price_via_nav | cheapest_item | pagination_sku | contact_form | signup_select | login_balance | extract_profile | table_lookup | count_reviews | event_date_iso |
|---|---|---|---|---|---|---|---|---|---|---|---|
| heuristic-scripted | Pass. Answer: $149.99 | Pass. Answer: $149.99 | Fail. No answer string | Pass. Answer: LMN-4471 | Pass. No answer string · Submission: {"name": "Ada Lovelace", "email": "[email protected]", "message": "Loved the demo"} | Pass. No answer string · Submission: {"email": "[email protected]", "plan": "Pro", "terms": "yes"} | Pass. Answer: $4,210.55 · Submission: {"username": "demo", "password": "hunter2"} | Pass. Answer: {"title": "Rear Admiral of Compilers", "company": "Navy Labs"} | Fail. No answer string | Fail. No answer string | Fail. No answer string |
| naive-first-match | Fail. Answer: $199.99 | Pass. Answer: $149.99 | Fail. Answer: $149.99 | Fail. Answer: $149.99 | Fail. Answer was empty | Fail. Answer was empty | Fail. Answer was empty | Fail. Answer was empty | Fail. Answer: 101 | Fail. Answer was empty | Fail. Answer: 2 |
The 11 tasks
A task is a start path on the fixture server, an instruction, and a checker. Expected answers stay in the checkers inside the harness.
| Task | Kind | Start | Instruction |
|---|---|---|---|
| price lookup price_lookup | Extract | /shop/aurora.html | Find the current price of the Aurora Headphones (the value labelled "Price"). Answer with the price. |
| price via nav price_via_nav | Navigate | /shop/index.html | On the shop, click "Aurora Headphones" and report the value labelled "Price". |
| cheapest item cheapest_item | Reason | /shop/index.html | Which item on the shop's first page is the cheapest? Answer with the product name. |
| pagination sku pagination_sku | Navigate | /shop/index.html | Go to the next page of the shop (click "Next") and report the value labelled "SKU" for the Lumen Desk Lamp. |
| contact form contact_form | Form | /contact.html | Fill the contact form with Full name "Ada Lovelace", Email "[email protected]", Message "Loved the demo" and submit it. |
| signup select signup_select | Form | /signup.html | Sign up with Email "[email protected]", choose Plan "Pro", check "I agree to the terms", then submit. |
| login balance login_balance | Multi-step | /bank/login.html | Log in with Username "demo" and Password "hunter2", then report the value labelled "Balance". |
| extract profile extract_profile | Extract | /team.html | For Grace Hopper, return JSON with fields "Title" and "Company" (lowercase keys). |
| table lookup table_lookup | Extract | /flights.html | What time does flight PX204 depart? Answer as HH:MM. |
| count reviews count_reviews | Reason | /reviews.html | How many reviews give 5 stars? Answer with a number. |
| event date iso event_date_iso | Reason | /events.html | On what date is Frontiers Demo Day? Answer in ISO format YYYY-MM-DD. |
What the harness does
A browser agent is a loop: look at a page, act, answer. The harness holds that loop still so two agents can be compared on the same pages.
- Fixture sites. A shop with a second page, a contact form, a signup form with a plan menu and a terms checkbox, a bank login that opens a dashboard, team profiles, a flight table, reviews, and an events page. Static files, served on localhost.
- Checkers. Numeric match, exact text, JSON fields, or an exact form submission. A pass is the checker returning true.
- Browser environment. Playwright, a step budget of 20, and the actions click, type, select, check, goto, and back. The observation is the URL, the page text, and the interactive elements.
- Adapters. One method, run(task, env, obs). The harness ships heuristic, naive, and llm. The llm adapter talks to any OpenAI-compatible API. A custom agent is --agent module:Class.
- Results. A summary plus one row per task: answer, submission, steps, time, tokens, cost, and the action trace.
Privacy
The fixture server binds to localhost. Tasks, traces, and checks stay in the directory you run. getEvals does not receive them. This page is a static rendering of the JSON committed under results/. Publishing a score is a choice you make by adding a file and rebuilding the site.
How to run it
The source is in this repo at harness/browser-agent-eval/. From that directory:
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt .venv/bin/python -m playwright install chromium .venv/bin/python -m harness.cli run --agent heuristic .venv/bin/python -m harness.cli run --agent naive .venv/bin/python -m harness.cli leaderboard
playwright install chromium can be skipped when Google Chrome is already on the machine; the harness falls back to it. The leaderboard command writes site/index.html inside the harness directory. This getEvals page is a separate rendering of the same JSON.
The language-model adapter has no row in the table above. To produce one:
XAI_API_KEY=... .venv/bin/python -m harness.cli run --agent llm
OPENAI_API_KEY and OPENROUTER_API_KEY select those APIs. LLM_MODEL overrides the model name. Put the JSON in results/, then from the repo root run python3 build.py. A custom agent uses the same command with --agent your_module:YourAdapter.
What a pass means here
A pass is a measurement on these 11 pages. Agents on the open web, inside logged-in products, and on tasks you have not written down still need a suite of their own. The scripted heuristic shows what generic label-and-click skills cover. The naive agent is the lower bound: the first number on the page. An adapter earns a place on the board by clearing more of the grid than those two, on a fresh run, with the trace saved in the JSON.
The task list can be exported for a Prime Intellect environment with python -m harness.cli export-tasks. That packaging is prepared in environments/browser_agent_eval/ and is not wired to the Hub yet.