TL;DR

Every simple break the small model handles is one agent session you don't have to pay for, in tokens or in waiting.

The problem

You know this one. A developer renames a button. Its data-test goes from proceed-1 to something else, the text changes from "Proceed to checkout" to "Continue to checkout", and the next morning your checkout test is red:

waiting for getByTestId('proceed-1')

Nothing is broken for the user. The test just lost track of a button. Fixing it by hand takes about five minutes, but it keeps happening. Handing it to an AI agent saves your time, but you pay in tokens and a couple of minutes of agent work for what is often a one-word change.

I looked at three ways of using AI here, and they cost very different amounts:

What I wanted to know was how many breaks the cheap, fast third option can handle on its own, and how to pass the rest to the second option without it going wrong.

What I used

Playwright runs the tests and does the clicking. I tried everything on a Playwright suite for the public Practice Software Testing (Toolshop) demo shop. It has page objects, fixtures and a full checkout flow, so it's a reasonable stand-in for a real project.

A System One decision model, Jev or Laya. These are a different kind of model. They don't write text. You give them a state and a typed question ("which of these options?"), and in one quick pass they return a choice, a probability for every option, and a calibrated confidence.

• Jev (TypeSafe AI) is hosted and paid, about $0.084 per million input tokens. Output is free.

• Laya is open source (Apache-2.0) and runs on your own machine.

Both use the same API (POST /v1/systemone), so moving between them is a URL change. I used Laya for everything below, on a laptop's integrated Intel Arc GPU. Once warm, it took about 150 ms per decision. Nothing in the design depends on which one you pick.

The Playwright Test agent (Claude) is Playwright's official healer agent. It's my fallback for breaks the small model can't handle.

Where this fits: maintenance, not creation

A test suite has two phases, and AI helps differently in each.

Creating tests. This is where large models shine. The Playwright Test agents (Claude) can explore the app, plan the scenarios and write the test scripts for you: the planner, the generator and the healer working together. It's a one-time cost per test, and well worth it. Nothing in this post replaces that.

Maintaining tests. This is the part that never ends. Once the tests exist, the app keeps changing: buttons get renamed, ids change, text gets reworded. Every change breaks a locator, and every break needs healing. Most of these fixes are small, but they happen again and again, and if every one goes to a large agent, you pay tokens and minutes every time.

This project focuses on that second phase: the healing. The idea is simple. Let a small, fast model handle the routine fixes, and keep the Playwright Test agent (Claude) for the breaks that really need its reasoning.

Two choices follow from that:

• The healer runs only when a test breaks. Your CI keeps running plain Playwright specs, so while tests pass, healing costs nothing.

• Every fix ends as a code change a human reviews. The healer doesn't quietly adapt the test at run time. It edits the page object, proves the fix by re-running the test, and leaves the diff for you, the same way the Playwright Test agent (Claude) works.

How a System One model works

A large language model like Claude reads a prompt and writes an answer, one token at a time. A System One model like Laya or Jev does something much smaller. You give it a situation and a multiple-choice question, and in one quick pass it gives you back the option it picked, a probability for every option, and a confidence score for the answer. It doesn't write sentences or explain itself. Think of it as a fast, gut-feel decision rather than slow, step-by-step reasoning.

In this project, the situation is the intent of the broken locator, and the options are the clickable elements Playwright found on the page. Simplified, one question looks like this:

Intent: button "Proceed to checkout" — cart (step 1) to sign-in (step 2)
Which of these is it now?
  button "Continue to checkout"
  button "Continue Shopping"
  link   "Practice Software Testing - Toolshop"
  ...

The model answers with the number of the option it picked, and the confidence score tells the healer whether to trust it. You'll see real answers in the scenarios below.

Small models like these are touchy about how a question is asked. Rewording it or shuffling the options can swing the confidence a lot (I'll share the numbers in the lessons at the end). So code builds every question, never by hand, always in the same tested format: at most 10 options, always in page order. The exact template is in my repo if you want to see it.

How it works

Here is the whole thing on one page: the healer, the guardrails around it, and the point where the Playwright Test agent (Claude) takes over. The numbers on the flow cards match the numbered rules further down.

And here is the same flow as it runs in the terminal:

npx playwright test ── ✅ pass ──► done (no AI involved)
        │
        ❌ waiting for getByTestId('proceed-1')
        ▼
npm run heal
   1. REPORT   which locator broke?           test-results/results.json
   2. MEMORY   what was it meant to be?       @intent note in the page object
   3. EYES     what was on the page?          snapshot saved when the test failed
   4. LAYA     which element is it now?       choice + confidence, ~0.2 s, free
   5. FIX + VERIFY: edit page object → re-run the failing test
        ├── ✅ passes → keep it; a human reviews the diff
        └── ❌ fails, or confidence < threshold → UNDO → hand-off note → Playwright Test agent (Claude)

A few of these steps need a bit more explanation.

Memory. A test id tells you where an element was, not what it was for. So I added a short note above each important locator:

/** @intent button "Proceed to checkout" — cart (step 1) to sign-in (step 2) */
readonly proceed1Button: Locator = this.page.getByTestId('proceed-1');

The purpose matters here, because the shop has three different buttons that are all called "Proceed to checkout", one for each checkout step.

Eyes: When a test fails, a Playwright fixture lists every visible clickable element on the live page, with its real attributes and in page order, and attaches that list to the test report. The healer runs later and doesn't need a browser to know what the page looked like.

Decision: Laya gets the intent plus at most 10 candidate elements, and it answers with just a number: which candidate it is. The healer then builds the new locator from that element's real attributes (its data-test, or its role and name). The model can't invent a selector, because it only picks from things Playwright actually found on the page. It can still pick the wrong one, though, and that's why every fix gets verified.

Safety rail: The healer edits the page object and re-runs the failing test. If the test still fails, it puts the file back exactly as it was.

Hand-off: If Laya isn't confident, or its fix fails, the healer writes a ready-made request for the Playwright Test agent (Claude). In my setup I paste it in by hand, on purpose: I want a person to decide when it's worth spending tokens. This is a choice, not a limitation. If your team prefers a fully automatic flow, the healer can start the Playwright Test agent (Claude) on its own through the Claude Code command line, passing the same request. Teams that automate it usually have the agent open a pull request for review, so the fix still gets a human check before it's merged.

The rules I ended up with

Some of these were design decisions from day one. Others I added after getting burned. They kick in at different points of a heal, so here they are in order.

Before the model is asked

1. Heal only after a failure. CI runs plain Playwright specs, so while your tests pass, nothing costs a token.

2. Know which locator broke, and what it was for. When a test fails, Playwright writes the error into test-results/results.json. The healer reads the broken locator from it (getByTestId('proceed-1')) and finds every page object that defines it. Those are the "listed locators" you'll see later, and the only ones a fix is allowed to touch. Each one carries an @intent note, so the model knows what the element was for.

When the model is asked

1. The model chooses, it never invents. Laya (or Jev) picks from elements that actually exist on the page and never writes a selector. The healer builds the locator from the chosen element's real attributes.

After the model answers

1. No confidence, no edit. Below the threshold, the code stays untouched and the healer goes straight to the hand-off.

2. Verify every fix. The healer re-runs the failing test, and if it still fails, the change is undone automatically. This is the rule that saved me in scenario 2.

When it goes to the Playwright Test agent (Claude)

1. Give the agent strict rules. The hand-off note lists the locators from rule 2, with file, line and property name, and the agent may change only those. If the app itself looks broken, it explains instead of editing the test. The agent does write the new selector itself, so its fix gets the same re-run check.

2. Keep a human in the loop. Someone reviews every fix, and spending paid tokens is a human call too. I paste the hand-off note in by hand, but you can automate it and have the agent open a pull request instead.

Scenario 1: the button is renamed, and Laya fixes it alone

I simulated the developers renaming the cart's checkout button.

The test breaks, and a plain text search for "Proceed to checkout" would have broken too.

This is what Laya saw and answered (10 candidates, top three shown):

0.878  button "Continue to checkout"   <- picked
0.092  button "Continue Shopping"
0.006  link   "Practice Software Testing - Toolshop"
confidence 0.78, 243 ms

It matched meaning, not text. "Proceed" and "Continue to checkout" do the same job. It also stepped around a nasty trap: "Continue Shopping" shares a word with the new button but does the opposite thing.

From there the healer did its job. It built the new locator from the picked element's real id, getByTestId('checkout-continue'), and updated it in both page objects that used proceed-1. It found the duplicate through the @intent notes. Then it re-ran the test, which passed in 6.8 s, and printed "Fixed and verified". I reviewed the diff and that was it.

The cost was 0 tokens. It was about 0.25 s of Laya time plus one test re-run, all on my laptop.

Scenario 2: the button becomes an icon, Laya fails safely, and the agent steps in

This time I replaced the button's text with an arrow.

Nothing on the page says "checkout" anymore. Here's what Laya answered:

0.653  button "Continue Shopping"   <- picked   ❌ wrong
0.218  button "→"
confidence 0.51, 241 ms

Laya was wrong, and 0.51 was just above my 0.5 threshold, so the healer applied the fix. Then the safety rail earned its place. The re-run took the test back to shopping, where it failed at the next step. The healer undid its change, so the page objects were back exactly as they had been, and it wrote the hand-off note:

Use the playwright-test-healer agent to fix the failing test
tests/spec/e2e/checkout/checkout.spec.ts:7 (project toolshop-e2e).
The locator getByTestId('proceed-1') no longer finds anything. It is defined in:
- tests/pages/pom/cart.page.ts:6 (proceed1Button)
- tests/pages/pom/checkout.page.ts:15 (proceed1Button)
It was meant to be: button "Proceed to checkout" — cart (step 1) to sign-in (step 2).
The fast local healer (Laya) could not fix it: it tried getByTestId('continue-shopping'),
but the test still failed, so the change was undone.
Rules: change only the locator(s) listed above, keep the @intent comments, …
If the app itself looks broken, do not change the test; explain why instead.

The Playwright Test agent (Claude) took it from there. It ran the test and read the snapshot of the failing page. It worked out that the "→" sitting next to "Continue Shopping" was the checkout button. Then it changed both locators to getByRole('button', { name: '→' }) and re-ran the test.

That cost about 26,000 tokens, 12 tool calls and roughly 2 minutes.

On its run, the test also hit an unrelated, intermittent failure in the billing-address form. The agent reported it and left it alone, exactly as the rules said. I re-ran the test twice afterwards and it passed both times, so it's a flaky step in the suite that's now on my list.

Where the tokens and time actually go

The savings come from four places.

Simple breaks are free and take seconds. Renames and rewordings are, in my experience, the most common kind of locator break, and Laya fixes and verifies them locally. The one agent run I measured cost about 26k tokens and 2 minutes, so every break Laya handles avoids a session like that. I didn't send scenario 1 to the agent, so I can't give you its exact cost. It would have been the same kind of session.

Passing tests cost nothing. No AI runs in CI. A runtime agent, by comparison, has an LLM read the page at every step of every run.

Hard breaks get a head start. The hand-off note already holds the diagnosis (broken locator, files, intent, what Laya tried), so the agent doesn't start from zero.

You decide when to spend. The paid fallback only starts when a person pastes the hand-off note.

Time matters as much as tokens here. A Laya decision takes about 0.2 s, so the total is dominated by re-running the test to verify the fix, and you'd want to do that anyway.

If I had to draw it, it's an escalation ladder, and each step costs more and can handle more:

Laya (free, ~0.2 s) → verify → Playwright Test agent (Claude) (tokens, ~minutes) → human

Limits, and what I'd do about them

This is a prototype built to answer one question, so it has limits. Most of them come from my setup, not from the approach.

• I only handle getByTestId breaks for now. The idea isn't tied to test ids, but my code is. Other locator types (getByRole, getByText, getByLabel, CSS) fail with their own error messages, so they need small changes in my repo: extend the failure pattern in heal.ts to recognise them, and match them to their @intent note in intents.ts. The decision part stays the same, because the model only ever chooses between elements on the page.

• I only collect clickable elements. Inputs, dropdowns and checkboxes can be added to the page snapshot in the same way.

• Elements without text or a label are invisible to the model. That's partly why scenario 2 failed. Adding aria-labels to icon-only buttons fixes that and helps accessibility too.

• Small context. Laya's English model reads about 512 tokens, so busy pages get trimmed to 10 candidates, and the right one can get cut. If you need more room, switch to Jev, which handles about 32k to 64k tokens and up to 255 options per question. Jev does have a cost, though. It's hosted and paid (about \(0.084 per million input tokens, output is free), so the "0 tokens" in the title applies to the Laya setup I ran. Even so, Jev is far cheaper than the Playwright Test agent (Claude): a 512-token question costs roughly \)0.00004, while the one agent run I measured used about 26,000 tokens. It's the same API, so only the config changes (plus raising the 10-option cap I set). Laya's multilingual model, with up to about 8k tokens, is a free middle ground.

• My 0.5 threshold isn't calibrated. A wrong answer scored 0.51. The proper fix is to run the healer over many simulated breaks, record confidence against correctness, and pick the threshold from that data. Verification stays as the safety net either way.

• The hand-off is manual, by choice. A team happy to spend the tokens could start the Playwright Test agent (Claude) automatically through the Claude Code command line, ideally opening a pull request for review instead of changing tests silently.

Try it

The code is in my GitHub repo: playwright-laya-healer. Both scenarios are included as a switch, so you can watch Laya fix the first one and hand the second to the Playwright Test agent (Claude).

You'll need Node.js 24+, Python for Laya (CPU works, a GPU is faster), and Claude Code to run the Playwright Test agent (Claude) fallback.

About me

I'm Sahil, an SDET and AI-powered QA engineer. I help teams build test automation that is faster to maintain and cheaper to run, using AI where it actually helps. More of my work is at sahilsharmaqa.com.