Build a Pydantic AI agent on Steel
Pydantic AI is a Python framework for agents from the team behind Pydantic. A tool gets what it needs from the agent's dependencies, and the agent's answer is checked against a Pydantic model. You pass a Steel session as the dependency, and the tools send Commands to it: the model reads each page as a snapshot, and clicks and types on its elements by ref. Every tool acts in the same cloud browser, with the session's proxies, CAPTCHA solving and live view.
Before you begin
- Python 3.11 or later.
- A Steel API key in the
STEEL_API_KEYenvironment variable. See API keys. - An OpenAI API key in the
OPENAI_API_KEYenvironment variable. Pydantic AI reads it for a model whose name starts withopenai:.
Install Pydantic AI and the Steel SDK. The tools send Commands with the Steel SDK, so you do not install Playwright.
python3 -m venv .venv
source .venv/bin/activate
pip install pydantic-ai steel-sdkSet up the integration
Give the agent a browser
This program gives a Pydantic AI agent five browser tools over a Steel session.
import asyncio
from dataclasses import dataclass
from typing import Literal, Optional
from pydantic import BaseModel
from pydantic_ai import Agent, RunContext
from steel import AsyncSteel, omit
from steel.types import CommandParam
@dataclass
class BrowserDeps:
steel: AsyncSteel
session_id: str
after: Optional[str] = None
class Todo(BaseModel):
title: str
completed: bool
class TodoList(BaseModel):
todos: list[Todo]
agent = Agent(
"openai:gpt-6.1-terra",
deps_type=BrowserDeps,
output_type=TodoList,
instructions=(
"You operate a Steel cloud browser via tools. "
"Call navigate to open a URL. To act on an element, pass its ref from the latest snapshot. "
"Do not invent data."
),
)
async def run(deps: BrowserDeps, commands: list[CommandParam], mode: Literal["snapshot", "diff"]) -> str:
response = await deps.steel.sessions.commands(
deps.session_id, commands=commands, after=deps.after or omit, observe={"mode": mode}
)
deps.after = response.next or deps.after
lines = []
if response.page:
lines.append(f"{response.page.title} ({response.page.url})")
for result in response.results:
if not result.ok and result.error:
lines.append(f"{result.command} failed: {result.error.message}")
if response.observe:
lines.append(response.observe.snapshot or response.observe.diff or "")
return "\n".join(lines)
@agent.tool
async def navigate(ctx: RunContext[BrowserDeps], url: str) -> str:
"""Open a URL. Returns the page as a snapshot: its text, and a ref such as e5
for each element that you can act on."""
return await run(ctx.deps, [{"command": "navigate", "url": url}], "snapshot")
@agent.tool
async def snapshot(ctx: RunContext[BrowserDeps]) -> str:
"""Read the current page again, as a snapshot with refs."""
return await run(ctx.deps, [{"command": "snapshot"}], "snapshot")
@agent.tool
async def click(ctx: RunContext[BrowserDeps], ref: str) -> str:
"""Click the element with this ref. Returns the lines of the snapshot that changed."""
return await run(ctx.deps, [{"command": "click", "target": ref}], "diff")
@agent.tool
async def fill(ctx: RunContext[BrowserDeps], ref: str, value: str) -> str:
"""Replace the text in the field with this ref. Returns the lines of the snapshot that changed."""
return await run(ctx.deps, [{"command": "fill", "target": ref, "value": value}], "diff")
@agent.tool
async def press(ctx: RunContext[BrowserDeps], key: str) -> str:
"""Press a key, such as Enter, in the element that has focus. Returns the lines
of the snapshot that changed."""
return await run(ctx.deps, [{"command": "press", "key": key}], "diff")
async def main():
steel = AsyncSteel()
session = await steel.sessions.create()
print("Session viewer:", session.session_viewer_url)
try:
result = await agent.run(
"Open https://demo.playwright.dev/todomvc, add the todos Buy milk and Walk the dog, "
"mark Buy milk as completed, and return the todo list.",
deps=BrowserDeps(steel=steel, session_id=session.id),
)
print(result.output.model_dump_json(indent=2))
finally:
await steel.sessions.release(session.id)
asyncio.run(main())It creates a session, and passes the client and the session's ID to the agent in BrowserDeps. Each tool that @agent.tool registers gets them from ctx.deps, and sends one request with sessions.commands through run. navigate and snapshot return the page as a snapshot, where each element that the model can act on has a ref. click, fill and press act, and return only the lines of the page that changed. A command that fails returns its error as text, and the model can read the page again.
BrowserDeps also keeps the next of the last response, and run sends it as after, so that a retried request runs its commands once. Build your own agent harness explains it. The program releases the session in finally.
output_type=TodoList makes result.output a TodoList object. The model reads each tool's docstring and parameters to decide when and how to call it.
Run the program
python agent.pyThe program prints the session viewer URL first. Open it to watch the agent add the todos and check one.
Session viewer: https://app.steel.dev/sessions/4c52c28c-eb22-413c-bfcd-cf368d7ee782Then it prints the agent's answer as JSON, checked against TodoList. It has two todos: Buy milk, with completed set to true, and Walk the dog, with completed set to false.
Add tools and Steel features
- To give the agent more actions, add tools with
@agent.toolthat send other commands, such asselect,scrollorwait. Commands reference lists each command and its fields. - To get other data back, change the Pydantic models, for example add
items_left: inttoTodoList. Pydantic AI checks the model's answer against them. - To add Steel features, pass them to
steel.sessions.create(), for exampleuse_proxyfor a Steel proxy andsolve_captchafor CAPTCHA solving. Pydantic AI does not see them. Session configuration lists every option.
Next steps
- Pydantic AI documentation
- Cookbook starter, with an
extracttool that reads rows by CSS selector from GitHub's trending page.