Build an OpenAI Agents SDK agent on Steel
The OpenAI Agents SDK is OpenAI's toolkit for agents, with typed tools, handoffs, guardrails and tracing. You give an agent tools that send Commands to a Steel session. The model reads each page as a snapshot, and clicks and types on its elements by ref. The page loads in a cloud browser, with the session's proxies, CAPTCHA solving and live view. The SDK has TypeScript and Python versions. In TypeScript a tool is a tool(), and in Python it is a function with the @function_tool decorator.
Before you begin
- Node.js 22 or later, or Python 3.11 or later.
- A Steel API key in the
STEEL_API_KEYenvironment variable. See API keys. - An OpenAI API key in the
OPENAI_API_KEYenvironment variable. The Agents SDK reads it from there.
Install the Agents SDK and the Steel SDK. The tools send Commands with the Steel SDK, so you do not install Playwright.
npm install @openai/agents steel-sdk zodpython3 -m venv .venv
source .venv/bin/activate
pip install openai-agents steel-sdkSet up the integration
Give the agent browser tools
This program gives an OpenAI agent five browser tools over a Steel session.
import { Agent, run as runAgent, tool } from "@openai/agents";
import Steel from "steel-sdk";
import { z } from "zod";
const steel = new Steel();
const session = await steel.sessions.create();
console.log("Session viewer:", session.sessionViewerUrl);
type Commands = Parameters<typeof steel.sessions.commands>[1]["commands"];
let after: string | undefined;
async function run(commands: Commands, mode: "snapshot" | "diff") {
const response = await steel.sessions.commands(session.id, { commands, after, observe: { mode } });
after = response.next ?? after;
const failed = response.results.find((result) => !result.ok);
return [
response.page ? `${response.page.title} (${response.page.url})` : "",
failed?.error ? `${failed.command} failed: ${failed.error.message}` : "",
response.observe?.snapshot ?? response.observe?.diff ?? "",
]
.filter(Boolean)
.join("\n");
}
try {
const navigate = tool({
name: "navigate",
description:
"Open a URL. Returns the page as a snapshot: its text, and a ref such as e5 for each element that you can act on.",
parameters: z.object({ url: z.string() }),
execute: ({ url }) => run([{ command: "navigate", url }], "snapshot"),
});
const snapshot = tool({
name: "snapshot",
description: "Read the current page again, as a snapshot with refs.",
parameters: z.object({}),
execute: () => run([{ command: "snapshot" }], "snapshot"),
});
const click = tool({
name: "click",
description: "Click the element with this ref. Returns the lines of the snapshot that changed.",
parameters: z.object({ ref: z.string() }),
execute: ({ ref }) => run([{ command: "click", target: ref }], "diff"),
});
const fill = tool({
name: "fill",
description: "Replace the text in the field with this ref. Returns the lines of the snapshot that changed.",
parameters: z.object({ ref: z.string(), value: z.string() }),
execute: ({ ref, value }) => run([{ command: "fill", target: ref, value }], "diff"),
});
const press = tool({
name: "press",
description: "Press a key, such as Enter, in the element that has focus. Returns the lines of the snapshot that changed.",
parameters: z.object({ key: z.string() }),
execute: ({ key }) => run([{ command: "press", key }], "diff"),
});
const agent = new Agent({
name: "SteelBrowser",
instructions: [
"You operate a Steel cloud browser via tools.",
"Call navigate to open a URL. To act on an element, pass its ref from the latest snapshot.",
"Do not invent data.",
].join(" "),
model: "gpt-6.1-terra",
tools: [navigate, snapshot, click, fill, press],
});
const result = await runAgent(
agent,
"Open https://demo.playwright.dev/todomvc, add the todos Buy milk and Walk the dog, mark Buy milk as completed, and report how many items are left.",
{ maxTurns: 20 },
);
console.log(result.finalOutput);
} finally {
await steel.sessions.release(session.id);
}import asyncio
from typing import Literal, Optional
from agents import Agent, Runner, function_tool
from steel import AsyncSteel, omit
from steel.types import CommandParam
steel = AsyncSteel()
session_id = ""
after: Optional[str] = None
async def run(commands: list[CommandParam], mode: Literal["snapshot", "diff"]) -> str:
global after
response = await steel.sessions.commands(
session_id, commands=commands, after=after or omit, observe={"mode": mode}
)
after = response.next or after
lines = []
if response.page:
lines.append(f"{response.page.title} ({response.page.url})")
for result in response.results:
if not result.ok and result.error:
lines.append(f"{result.command} failed: {result.error.message}")
if response.observe:
lines.append(response.observe.snapshot or response.observe.diff or "")
return "\n".join(lines)
@function_tool
async def navigate(url: str) -> str:
"""Open a URL. Returns the page as a snapshot: its text, and a ref such as e5
for each element that you can act on."""
return await run([{"command": "navigate", "url": url}], "snapshot")
@function_tool
async def snapshot() -> str:
"""Read the current page again, as a snapshot with refs."""
return await run([{"command": "snapshot"}], "snapshot")
@function_tool
async def click(ref: str) -> str:
"""Click the element with this ref. Returns the lines of the snapshot that changed."""
return await run([{"command": "click", "target": ref}], "diff")
@function_tool
async def fill(ref: str, value: str) -> str:
"""Replace the text in the field with this ref. Returns the lines of the snapshot that changed."""
return await run([{"command": "fill", "target": ref, "value": value}], "diff")
@function_tool
async def press(key: str) -> str:
"""Press a key, such as Enter, in the element that has focus. Returns the lines
of the snapshot that changed."""
return await run([{"command": "press", "key": key}], "diff")
agent = Agent(
name="SteelBrowser",
instructions=(
"You operate a Steel cloud browser via tools. "
"Call navigate to open a URL. To act on an element, pass its ref from the latest snapshot. "
"Do not invent data."
),
model="gpt-6.1-terra",
tools=[navigate, snapshot, click, fill, press],
)
async def main():
global session_id
session = await steel.sessions.create()
session_id = session.id
print("Session viewer:", session.session_viewer_url)
try:
result = await Runner.run(
agent,
input=(
"Open https://demo.playwright.dev/todomvc, add the todos Buy milk and Walk the dog, "
"mark Buy milk as completed, and report how many items are left."
),
max_turns=20,
)
print(result.final_output)
finally:
await steel.sessions.release(session.id)
asyncio.run(main())Each tool calls run, which sends one request with sessions.commands and turns the response into text for the model. navigate and snapshot return the page as a snapshot, where each element that the agent can act on has a ref. click, fill and press take a ref or a key, and return only the lines of the page that changed. A failed command comes back as text too, such as a ref that is no longer on the page, so the model can take a new snapshot and try again.
The agent's own run is imported as runAgent in TypeScript, so that the name run is free for the function that sends commands. Each request sends the next of the last response as after. A retry of that request runs its commands once, as Build your own agent harness explains. The program releases the session in finally.
In TypeScript, the url parameter is a plain z.string(). OpenAI's strict JSON schema does not accept the uri format that z.string().url() adds.
Run the program
npx tsx agent.mtspython agent.pyThe program prints the session viewer URL first. Open it to watch the agent work in the browser.
Session viewer: https://app.steel.dev/sessions/ef897af0-ba75-4157-8f1d-36efa65b4c53The agent types each todo into the field and presses Enter, then checks the box next to Buy milk. At the end, the program prints the agent's answer: one item is left, Walk the dog. The wording changes from run to run.
Add tools, typed output and Steel features
- To give the agent more actions, add tools that send other commands, such as
select,scrollorwait. Commands reference lists each command and its fields. - To get typed output, set
outputType(output_typein Python) on the agent to a Zod schema or a Pydantic model.finalOutput(final_outputin Python) then has that type. - To add Steel features, pass them to
steel.sessions.create(), for exampleuseProxy(use_proxy) for a Steel proxy andsolveCaptcha(solve_captcha) for CAPTCHA solving. The Agents SDK does not see them. Session configuration lists every option.
Next steps
- OpenAI Agents SDK documentation: TypeScript and Python.
- Cookbook starters: TypeScript and Python.