BreakingWeb: Controlled Environment Interventions for Browser-Use Agents

Construct challenging tasks with deterministic, detectable, recoverable interventions while preserving the user instruction, latent target, and backend success criterion. This checkout contains seven environments, 519 base tasks, and 535 intervention variants; the paper evaluates 519 clean/intervention pairs.

1. Motivation

Existing web-agent benchmarks score whole-task completion. A run either succeeds or fails, and a single scalar gets reported. The scalar is useful for ranking models, yet it carries almost no diagnostic information. When an agent finishes a 15-step checkout with score 0, the log does not say whether the agent misread a product card, accepted a forged "Saved" toast, abandoned a retry after a transient 503, or walked past the only remaining affordance. The benchmark does not know, and the training signal that follows does not know either.

Recovering from an obstacle can require grounding an observation, planning sub-goals, tracking state, backtracking from a blocked path, waiting through a transient failure, exploring alternatives, or verifying a write. These seven cognitive primitives serve as labels for the primary behavioral demand of an intervention. They organize comparisons across environments; a single recovery can involve more than one primitive.

BreakingWeb applies controlled environment interventions to a base task while holding its user instruction, latent target, and backend success criterion fixed. The intervention changes seeded content, server state, network responses, or client-side interaction and retains a recoverable path to the target. The paired performance drop measures the cost of the intervention. Grouping variants by their primary primitive shows where that cost concentrates without claiming strict isolation of an underlying capability.

2. Cognitive Primitives as Intervention Labels

The seven labels describe recurring recovery demands across browser-use tasks. Each intervention has one primary target label so results can be grouped consistently. The labels can overlap in the behavior needed for successful recovery and do not establish a complete or experimentally independent decomposition of agent competence.

Primary Recovery Demands

Each variant names one target_primitive. This annotation identifies the recovery behavior it primarily demands, rather than proving that other capabilities are held constant. For example, a verification-labeled intervention can still require grounding and state tracking. The table maps each label to representative stressor classes.

3. How We Intervene: Four Layers

An intervention is not a single mechanism. A phishing email and a 503 retry and a swallowed click are all interventions, but they live at different places in the web stack and fire at different times. Collapsing them into one hook would either paper over real mechanisms (DOM mocks of network failure miss real HTTP timing) or miss mechanisms entirely (a middleware has no view of DOM occlusion). We therefore split the perturbation space by where each action fires.

Why Four Layers Are Needed

When Each Layer Fires

Worked Example: A Patience Intervention

4. Catalog Map

The released checkout contains 519 base task YAMLs and 535 intervention YAMLs across seven environments. These inventory counts include additional variants beyond the 519 clean/intervention pairs reported in the paper.

Tasks and Variants per Environment

Intervention Families by Layer

We group dispatch branches into intervention families: each family captures one stressor mechanism, and the branches inside it are environment-specific specialisations that impose the same cognitive load. A family is the right unit for reasoning about primitive coverage; the dispatcher split matters only at the implementation level.

Seed Families injector/seed.py

Server Families injector/server.py

Network Families injector/middleware.py

Client Families BenchmarkToolbar.tsx

Families are a conceptual grouping. The underlying 48 dispatch branches exist for two reasons: (1) some families have environment-specific specialisations that share a mechanism but read from different state collections (add_decoy_notifications and add_noise_orders both inject noise, but one writes to the notification list and the other to the orders list); (2) the delay family collapses six behaviour modes into a single dispatch. One infrastructure branch, set_feature_flag, is excluded from both counts: it toggles a window.__wabFeatureFlags boolean and imposes no cognitive load on its own.

Primitive Coverage Across Environments

Coverage is uneven: the released variants leave four environment/primitive cells empty. Report those cells as missing when comparing results by environment and primary primitive.

Design Invariants

5. Evaluation System

Scoring runs against live environment state rather than the DOM. The evaluator reads the Pydantic state objects the real API handlers mutate, so a silently-dropped write is caught even when the UI shows a success toast. Every task declares a canonical_diff block, and matching runs through breakingweb/eval_core/. (The legacy expression-based eval.checks path was removed in the 2026-04 refactor; all 519 tasks now carry a canonical_diff.)

Pipeline

# breakingweb/eval_core/orchestrator.py
def evaluate(task, server_state, targets, trajectory):
    canonical = task.canonical_diff              # missing_canonical_diff → score 0
    initial   = server_state._initial_state_copy  # deep-copy taken at session create

    agent_diff = compute_diff(initial, server_state)   # eval_core/diff.py
    report     = match_diff(agent_diff, canonical,        # eval_core/matcher.py
                            targets, initial, server_state, session_start)

    return {
        "score":       report.score,
        "success":     report.passed,
        "checks":      report.checks,             # positive match results
        "negative_checks": report.negative_checks, # invariants + constraints + collateral
        "failures":    report.failures,
        "collateral":  state.compute_collateral(initial),  # analytics only
        "bijection_graphs": report.bijection_graphs,
    }

Five Grammar Sections of canonical_diff

Two auxiliary structures extend the grammar: oneof wraps multiple acceptable alternatives at the block level and the matcher picks the best-scoring applicable alternative; named_invariants attach human-readable names and severities (critical / high / medium / low) to the invariant list.

# Task YAML (amazon_add_single_item.yaml, abridged)
canonical_diff:
  create:
  - entity: CartItem
    desc: Target product added to cart with quantity 1
    properties:
      product_id: {expr: "x == target['product_id']"}
      quantity:   {eq: 1}
      product_name: {any: true}
  invariant:
  - collection: state.cart_items
    filter: "a.product_id != target['product_id']"   # preserve everything but the target
    preserve: ALL
  named_invariants:
  - {name: Agent did not place orders, ref: invariant[4], severity: high}

Predicate Grammar (19 keys)

Properties of a create/update are matched by predicates, each a single-key mapping. eval_core/predicates.py

Automatic Collateral Sweep

After matching the explicit sections, the matcher scans every unmatched diff entry and penalises the agent for uncovered state mutations:

The agent is not required to enumerate every side-effect explicitly; a collection-level invariant blocks the sweep. The sweep exists to catch overreach: a successful "add to cart" that also silently writes a review or cancels an order loses score.

Score Formula

eval_core/matcher.py Severity table: critical 0.30, high 0.20, medium 0.15, low 0.10.

if total_weight > 0:                 # create/update/delete present
    raw_score = passed_weight / total_weight
elif constraints_total:              # constraint-only task
    raw_score = constraints_passed / constraints_total
else:
    raw_score = 1.0

penalty_total = sum(nc.penalty for nc in negative_checks if not nc.passed)
score         = max(0.0, min(1.0, raw_score - penalty_total))
success       = not failures     # any hard failure → not success

6. Evaluated Agents

The paper evaluates six browser-use agents and three GUI-only agents under matched intervention conditions. The browser-use agents use the Browser Use harness; the GUI-only agents receive screenshots and act through the pixel harness. Shared tasks, variants, and seeds support comparison, while differences in observations, actions, and agent implementations limit attribution of the gap to any single primitive.

The table lists the paper's evaluated configurations. Both harnesses use the same task and intervention definitions and backend evaluator.

7. Human Baseline

Human trajectories serve two distinct purposes. We collect both on every task, across both the baseline and the intervened versions.

How the Two Baselines Combine

8. Environments

Seven fully interactive web environments, each a faithful reproduction of a real-world platform. Each ships a React SPA, a FastAPI backend, Pydantic state models, and YAML-seeded initial data.

9. Repository Structure (abridged)

breakingweb/
  injector/
    config.py            # DegradationConfig dataclass + default templates
    seed.py              # seed-layer actions
    server.py            # server-layer actions
    middleware.py        # DegradationMiddleware (primary network interception)
    network.py           # Playwright page.route() parity
    apply.py             # orchestrator
    variants/            # 535 available YAML variants

  backend/
    models/{env}.py      # Pydantic state shapes
    routes/{env}.py      # REST endpoints + session create/evaluate
    seeders/{env}.py     # YAML → seeded state

  tasks/
    _schema.py           # TaskDefinition, EvalConfig
    _registry.py         # YAML discovery
    _evaluator.py        # restricted-eval checker
    {env}/               # per-env task YAMLs

  environments/
    {env}/               # React SPA per env
    shared/src/components/BenchmarkToolbar.tsx  # client-layer injections

10. Running BreakingWeb

# Install
uv sync
uv run playwright install chromium
pnpm -C breakingweb/environments install
./scripts/breakingweb.sh build

# Baseline run (no intervention)
python -m breakingweb.agent_eval \
    --model gpt-4o --provider openai --api-key $KEY \
    --tasks gmail_action_item_extraction \
    --max-steps 50 --seed 42 --output results/baseline.json

# Intervention run (same task, one variant)
python -m breakingweb.agent_eval \
    --model gpt-4o --provider openai --api-key $KEY \
    --tasks gmail_action_item_extraction \
    --degradation gmail_action_item_extraction__phishing_inbox.yaml \
    --max-steps 50 --seed 42 --output results/phishing.json

# Visualize any result JSON
python -m breakingweb.visualize results/phishing.json

# Human play (collects cold-start and optimal trajectories)
python -m breakingweb.app --host 127.0.0.1 --port 8080
# Open http://127.0.0.1:8080/launch

Migrating from WebStress

Use breakingweb imports, python -m breakingweb..., source paths under breakingweb/, and scripts/breakingweb.sh. The old scripts/webstress.sh forwards to the new launcher. BrowserGym task IDs use browsergym/breakingweb.<task_id>.

Prefer BREAKINGWEB_* environment variables. Corresponding WEBSTRESS_* values are used only when the new variable is absent; an explicitly empty new value takes precedence. Copy local .env, results, and human traces into their corresponding new paths without overwriting newer files, reinstall frontend dependencies, rebuild, and restart the server. Local artifacts are not migrated automatically. The remote remains Arvid-pku/WebStress.