AI in automation testing is five primitives, not one smart model

Why "AI in automation testing" is the right phrase

The common framing is that AI replacesautomation testing. That framing is wrong in a way that produces bad tools. Automation testing still happens. Playwright is still the runtime. Chromium still launches. A selector still has to resolve to a DOM node. What AI adds is a small set of primitives that handle situations classical automation could not: streaming AI-rendered content that doesn't finish when the network goes quiet, multi-field OTP inputs that drop characters when you type them one at a time, disposable email verification inside a test run, webhook assertions that require an HTTP client. These are primitives, not intelligence. The AI picks which one to call. The primitive does the work.

The rest of this guide walks each of the five primitives, with the specific lines in the Assrt agent source where each is implemented, so you can replicate the pattern in any stack.

The five primitives at a glance

Primitive 1: the OTP paste one-liner (the anchor fact)

OTP-gated signup flows are where every previous generation of AI browser tester broke. The pattern is a row of six single-character inputs, usually input[maxlength="1"], controlled by React. An agent that types one character per field loses characters to focus transitions and autoadvance logic. An agent that tries a paste on a single input fills only that input. The version that works is a synthetic ClipboardEvent fired on the parent of the first input, carrying a DataTransfer with the full code. The browser fans the paste across the sibling inputs, React processes it as one event, every digit lands.

Assrt hardcodes the working expression into the system prompt the agent sees on every run, at line 235 of agent.ts. The next line forbids the agent from modifying it except to substitute the real code. This is a deliberate choice: letting the model improvise produces a different broken variant every run.

What one signup flow looks like end to end

The primitives don't matter on their own. They matter when an agent strings them together. Here is a real six-step signup flow as the agent traverses it, with the specific tool at each step.

The same flow as a sequence diagram, showing the actors the agent talks to: the browser (via Playwright MCP), the disposable-inbox service, and eventually the app's own UI.

Primitive 2: adaptive DOM stability, not setTimeout

AI apps stream. Chat UIs paint tokens a character at a time for several seconds after the network goes quiet. Generative UIs rewrite whole sections after a user clicks. A hardcoded sleep either flakes (too short for a slow stream) or wastes time (too long for a fast one). Playwright's built-in waits key off network events, which miss this entirely because the network went quiet long before the DOM did.

wait_for_stable measures what actually matters for a downstream click: DOM activity. It injects a MutationObserver into the live page, counts mutations, and returns as soon as the counter has been flat for the configured quiet window (default 2 seconds, clamped to 10). The timeout defaults to 30 seconds, clamped to 60.

Primitive 3: a real inbox, not a mock

Three tools make up the inbox layer, all defined between lines 115 and 131 of agent.ts: create_temp_email, wait_for_verification_code, and check_email_inbox. They talk to temp-mail.io through a thin client in email.ts. When a test hits a signup form, the system prompt tells the agent to reach for this tool sequence: create an address, use it in the form, submit, poll for the OTP, then paste with the expression from Primitive 1.

This is the difference between "test the signup flow" and "seed a fixture user and skip verification." The fixture route hides every bug that lives in the verify-email step, which is exactly the set of bugs that makes it to production. A real inbox primitive makes the full path testable.

Primitive 4: http_request for downstream assertions

Half of a modern app's tests are about what the page did to a downstream system, not what the page shows. When you click "Send notification," the interesting assertion is that Telegram got a message, that your Slack webhook fired, or that a Stripe event landed. Classical automation suites either skipped these (flaky, partial) or mocked them (lies).

http_request is a first-class tool in the same agent. The agent decides when to call it based on the English step. A #Case like the one below mixes UI actions and webhook verification in a single scenario, no second framework.

Primitive 5: unbounded recovery, bounded exits

Line 7 of agent.ts sets MAX_STEPS_PER_SCENARIO = Infinity. That looks dangerous; it is the most conservative choice. A flaky middle step should not kill a long integration run. Instead, when an action fails, the recovery rules in the system prompt say: snapshot the page fresh, look at what changed, pick a different accessibility ref, try again, and if still stuck after three tries, call complete_scenario with passed=false and evidence.

The agent always exits. The exit is explicit, records evidence, and is reviewable in results/latest.json. The infinity cap buys the agent the room to recover across a transient flake instead of surfacing it as a failure.

The system, one beam at a time

Inputs flow into a single model call per turn. The model picks a primitive, the primitive talks to its specific external surface, and structured events stream back to the caller. There is no orchestration layer hidden behind it.

What the numbers actually are

No invented benchmarks. Each number is a constant you can find in the source: agent.ts lines 7 (MAX_STEPS), 9 (DEFAULT_ANTHROPIC_MODEL), 16-196 (TOOLS array), 957-958 (wait_for_stable clamps), 124-125 (OTP timeout default), and 1087 for the file length.

Watching it run on a real signup flow

This is the log from a single assrt run against a staging signup. Notice how the tool calls alternate: snapshot, act, wait_for_stable, snapshot again. The OTP paste is a single evaluate call. The webhook check is a single http_request.

Against what "AI testing" usually means

Where this leaves a team considering AI in automation testing

The moment you start using AI to drive a browser, the question stops being "how smart is the model" and starts being "what primitives did the framework give it." Ask a vendor how they stabilize for streaming content. Ask them how they clear a real OTP. Ask them whether their agent can call an external API in the same #Case. If those answers are not on-disk TypeScript, you are buying a black box whose failure modes nobody can fix for you.

Assrt's five primitives are on disk, named, and within two hundred lines of each other. The LLM is swappable. Your test plan is Markdown. The point of this guide is not to pitch the product; it is to show you the specific shape of the primitives so you can evaluate any tool in the category, not just this one. If you want the easiest path to actually running them on your stack, talk to us.

Frequently asked questions about AI in automation testing