Skip to content

Build a Pydantic AI agent on Steel

Set up your agent

Give your coding agent a Steel browser and the Steel skills, so that it browses and writes Steel code the right way. Learn more

Install the Steel CLI. It signs you in and installs the skills for your coding agents.

curl -fsS https://setup.steel.dev | sh

Or send this prompt to your coding agent, and it does the same:

Read https://setup.steel.dev/install.md and follow it to set up Steel for this coding client.
Then continue my requested task using the installed Steel skill.

Set up your agent →

Pydantic AI is a Python framework for agents from the team behind Pydantic. A tool gets what it needs from the agent's dependencies, and the agent's answer is checked against a Pydantic model. You pass a Steel session as the dependency, and the tools send Commands to it: the model reads each page as a snapshot, and clicks and types on its elements by ref. Every tool acts in the same cloud browser, with the session's proxies, CAPTCHA solving and live view.

Before you begin

  • Python 3.11 or later.
  • A Steel API key in the STEEL_API_KEY environment variable. See API keys.
  • An OpenAI API key in the OPENAI_API_KEY environment variable. Pydantic AI reads it for a model whose name starts with openai:.

Install Pydantic AI and the Steel SDK. The tools send Commands with the Steel SDK, so you do not install Playwright.

python3 -m venv .venv
source .venv/bin/activate
pip install pydantic-ai steel-sdk

Set up the integration

Give the agent a browser

This program gives a Pydantic AI agent five browser tools over a Steel session.

agent.py
import asyncio
from dataclasses import dataclass
from typing import Literal, Optional

from pydantic import BaseModel
from pydantic_ai import Agent, RunContext
from steel import AsyncSteel, omit
from steel.types import CommandParam


@dataclass
class BrowserDeps:
    steel: AsyncSteel
    session_id: str
    after: Optional[str] = None


class Todo(BaseModel):
    title: str
    completed: bool


class TodoList(BaseModel):
    todos: list[Todo]


agent = Agent(
    "openai:gpt-6.1-terra",
    deps_type=BrowserDeps,
    output_type=TodoList,
    instructions=(
        "You operate a Steel cloud browser via tools. "
        "Call navigate to open a URL. To act on an element, pass its ref from the latest snapshot. "
        "Do not invent data."
    ),
)


async def run(deps: BrowserDeps, commands: list[CommandParam], mode: Literal["snapshot", "diff"]) -> str:
    response = await deps.steel.sessions.commands(
        deps.session_id, commands=commands, after=deps.after or omit, observe={"mode": mode}
    )
    deps.after = response.next or deps.after
    lines = []
    if response.page:
        lines.append(f"{response.page.title} ({response.page.url})")
    for result in response.results:
        if not result.ok and result.error:
            lines.append(f"{result.command} failed: {result.error.message}")
    if response.observe:
        lines.append(response.observe.snapshot or response.observe.diff or "")
    return "\n".join(lines)


@agent.tool
async def navigate(ctx: RunContext[BrowserDeps], url: str) -> str:
    """Open a URL. Returns the page as a snapshot: its text, and a ref such as e5
    for each element that you can act on."""
    return await run(ctx.deps, [{"command": "navigate", "url": url}], "snapshot")


@agent.tool
async def snapshot(ctx: RunContext[BrowserDeps]) -> str:
    """Read the current page again, as a snapshot with refs."""
    return await run(ctx.deps, [{"command": "snapshot"}], "snapshot")


@agent.tool
async def click(ctx: RunContext[BrowserDeps], ref: str) -> str:
    """Click the element with this ref. Returns the lines of the snapshot that changed."""
    return await run(ctx.deps, [{"command": "click", "target": ref}], "diff")


@agent.tool
async def fill(ctx: RunContext[BrowserDeps], ref: str, value: str) -> str:
    """Replace the text in the field with this ref. Returns the lines of the snapshot that changed."""
    return await run(ctx.deps, [{"command": "fill", "target": ref, "value": value}], "diff")


@agent.tool
async def press(ctx: RunContext[BrowserDeps], key: str) -> str:
    """Press a key, such as Enter, in the element that has focus. Returns the lines
    of the snapshot that changed."""
    return await run(ctx.deps, [{"command": "press", "key": key}], "diff")


async def main():
    steel = AsyncSteel()
    session = await steel.sessions.create()
    print("Session viewer:", session.session_viewer_url)

    try:
        result = await agent.run(
            "Open https://demo.playwright.dev/todomvc, add the todos Buy milk and Walk the dog, "
            "mark Buy milk as completed, and return the todo list.",
            deps=BrowserDeps(steel=steel, session_id=session.id),
        )
        print(result.output.model_dump_json(indent=2))
    finally:
        await steel.sessions.release(session.id)


asyncio.run(main())

It creates a session, and passes the client and the session's ID to the agent in BrowserDeps. Each tool that @agent.tool registers gets them from ctx.deps, and sends one request with sessions.commands through run. navigate and snapshot return the page as a snapshot, where each element that the model can act on has a ref. click, fill and press act, and return only the lines of the page that changed. A command that fails returns its error as text, and the model can read the page again.

BrowserDeps also keeps the next of the last response, and run sends it as after, so that a retried request runs its commands once. Build your own agent harness explains it. The program releases the session in finally.

output_type=TodoList makes result.output a TodoList object. The model reads each tool's docstring and parameters to decide when and how to call it.

Run the program

python agent.py

The program prints the session viewer URL first. Open it to watch the agent add the todos and check one.

Session viewer: https://app.steel.dev/sessions/4c52c28c-eb22-413c-bfcd-cf368d7ee782

Then it prints the agent's answer as JSON, checked against TodoList. It has two todos: Buy milk, with completed set to true, and Walk the dog, with completed set to false.

Add tools and Steel features

  • To give the agent more actions, add tools with @agent.tool that send other commands, such as select, scroll or wait. Commands reference lists each command and its fields.
  • To get other data back, change the Pydantic models, for example add items_left: int to TodoList. Pydantic AI checks the model's answer against them.
  • To add Steel features, pass them to steel.sessions.create(), for example use_proxy for a Steel proxy and solve_captcha for CAPTCHA solving. Pydantic AI does not see them. Session configuration lists every option.

Next steps