How AI Agents Browse the Web: Playwright, MCP, and Accessibility Snapshots

Have you ever wondered how AI agents can navigate websites, click buttons and fill out forms without actually “seeing” the screen?

While vision models — AI that can process images — are popular, they are often slow, expensive and prone to hallucinations. Examples include OpenAI’s GPT-4, Anthropic’s Claude 3 Opus and Microsoft’s Copilot with Vision. There is a smarter, faster approach: combining Playwright, the Model Context Protocol (MCP) and accessibility snapshots.

In this post we break down exactly how large language models (LLMs) steer web browsers using purely text-based semantic data.

1. The core components

Before we look at how they work together, let’s define the three main players.

Playwright is a powerful open-source automation framework, usually used for testing web apps. It can control browsers (Chrome, Firefox, Safari) programmatically. More at playwright.dev.

Model Context Protocol (MCP) is an open standard created to help AI models connect securely to external tools and data sources. Think of it as a universal API that lets an LLM say: “Hey, run this specific function on my behalf.”

Accessibility snapshots are the secret sauce. Browsers generate an Accessibility Object Model (AOM) to help screen readers navigate pages for visually impaired users. Instead of raw HTML or pixels, this is a clean, structured tree of interactive elements — “Button: Submit”, “Textbox: Username”. More on the format and how to access the elements at playwright.dev/mcp/snapshots.

2. Why skip the vision models?

If an AI wants to click a “Log In” button, taking a screenshot, sending it to a vision model, calculating the bounding box coordinates and then issuing a click command is incredibly inefficient.

The text-based approach wins on three counts.

• Speed. Parsing a JSON accessibility tree takes milliseconds. Processing images takes seconds.
• Cost. Text tokens are vastly cheaper than image tokens.
• Accuracy. Vision models can misinterpret modern UI designs. An accessibility tree tells the AI exactly what an element is and what state it is in — disabled, checked, and so on.

3. How the magic happens, step by step

Step 1: The user gives a command

You tell your AI agent: “Log into my dashboard and download the latest report.”

Step 2: The agent calls an MCP tool

The agent has no built-in browser. Instead it sends a request via the Model Context Protocol to a connected Playwright server, asking it to navigate to the login page.

page.goto(url)

Step 3: Playwright navigates and takes a snapshot

Playwright drives the headless browser to the URL. Instead of taking a visual screenshot, it runs page.accessibility.snapshot(), which generates a clean JSON representation of the page’s UI.

{  
  "role": "WebArea", 
  "children": [    
    { "role": "textbox", "name": "Email Address" },    
    { "role": "textbox", "name": "Password" },    
    { "role": "button", "name": "Log In" }  
]
}

Step 4: The LLM reads the snapshot

The Playwright MCP server sends the JSON snapshot back to the LLM. Because LLMs are very good at reading JSON and understanding context, the agent immediately works out that there is a textbox for an email address and a button named “Log In”.

Step 5: The agent takes action

The agent formulates its next move and sends another MCP command back to Playwright.

await page.getByRole('textbox', { name: 'Email Address' }).fill('ai@example.com');
await page.getByRole('button', { name: 'Log In' }).click();

‍Playwright executes these commands in the real browser, takes a new accessibility snapshot of the next page, and the loop continues until the task is complete.

Summary

By bridging Playwright and AI agents through the Model Context Protocol, we can build remarkably robust web automation.

Relying on accessibility snapshots instead of vision models gives the LLM a perfect, text-based map of the user interface. It proves that sometimes, to make an AI better at navigating the web, you don’t need to give it eyes — you just need to give it a better map.