Skip to content

Build an OpenAI Agents SDK agent on Steel

Set up your agent

Give your coding agent a Steel browser and the Steel skills, so that it browses and writes Steel code the right way. Learn more

Install the Steel CLI. It signs you in and installs the skills for your coding agents.

curl -fsS https://setup.steel.dev | sh

Or send this prompt to your coding agent, and it does the same:

Read https://setup.steel.dev/install.md and follow it to set up Steel for this coding client.
Then continue my requested task using the installed Steel skill.

Set up your agent →

The OpenAI Agents SDK is OpenAI's toolkit for agents, with typed tools, handoffs, guardrails and tracing. You give an agent tools that send Commands to a Steel session. The model reads each page as a snapshot, and clicks and types on its elements by ref. The page loads in a cloud browser, with the session's proxies, CAPTCHA solving and live view. The SDK has TypeScript and Python versions. In TypeScript a tool is a tool(), and in Python it is a function with the @function_tool decorator.

Before you begin

  • Node.js 22 or later, or Python 3.11 or later.
  • A Steel API key in the STEEL_API_KEY environment variable. See API keys.
  • An OpenAI API key in the OPENAI_API_KEY environment variable. The Agents SDK reads it from there.

Install the Agents SDK and the Steel SDK. The tools send Commands with the Steel SDK, so you do not install Playwright.

npm install @openai/agents steel-sdk zod

Set up the integration

Give the agent browser tools

This program gives an OpenAI agent five browser tools over a Steel session.

agent.mts
import { Agent, run as runAgent, tool } from "@openai/agents";
import Steel from "steel-sdk";
import { z } from "zod";

const steel = new Steel();
const session = await steel.sessions.create();

console.log("Session viewer:", session.sessionViewerUrl);

type Commands = Parameters<typeof steel.sessions.commands>[1]["commands"];
let after: string | undefined;

async function run(commands: Commands, mode: "snapshot" | "diff") {
  const response = await steel.sessions.commands(session.id, { commands, after, observe: { mode } });
  after = response.next ?? after;
  const failed = response.results.find((result) => !result.ok);
  return [
    response.page ? `${response.page.title} (${response.page.url})` : "",
    failed?.error ? `${failed.command} failed: ${failed.error.message}` : "",
    response.observe?.snapshot ?? response.observe?.diff ?? "",
  ]
    .filter(Boolean)
    .join("\n");
}

try {
  const navigate = tool({
    name: "navigate",
    description:
      "Open a URL. Returns the page as a snapshot: its text, and a ref such as e5 for each element that you can act on.",
    parameters: z.object({ url: z.string() }),
    execute: ({ url }) => run([{ command: "navigate", url }], "snapshot"),
  });

  const snapshot = tool({
    name: "snapshot",
    description: "Read the current page again, as a snapshot with refs.",
    parameters: z.object({}),
    execute: () => run([{ command: "snapshot" }], "snapshot"),
  });

  const click = tool({
    name: "click",
    description: "Click the element with this ref. Returns the lines of the snapshot that changed.",
    parameters: z.object({ ref: z.string() }),
    execute: ({ ref }) => run([{ command: "click", target: ref }], "diff"),
  });

  const fill = tool({
    name: "fill",
    description: "Replace the text in the field with this ref. Returns the lines of the snapshot that changed.",
    parameters: z.object({ ref: z.string(), value: z.string() }),
    execute: ({ ref, value }) => run([{ command: "fill", target: ref, value }], "diff"),
  });

  const press = tool({
    name: "press",
    description: "Press a key, such as Enter, in the element that has focus. Returns the lines of the snapshot that changed.",
    parameters: z.object({ key: z.string() }),
    execute: ({ key }) => run([{ command: "press", key }], "diff"),
  });

  const agent = new Agent({
    name: "SteelBrowser",
    instructions: [
      "You operate a Steel cloud browser via tools.",
      "Call navigate to open a URL. To act on an element, pass its ref from the latest snapshot.",
      "Do not invent data.",
    ].join(" "),
    model: "gpt-6.1-terra",
    tools: [navigate, snapshot, click, fill, press],
  });

  const result = await runAgent(
    agent,
    "Open https://demo.playwright.dev/todomvc, add the todos Buy milk and Walk the dog, mark Buy milk as completed, and report how many items are left.",
    { maxTurns: 20 },
  );
  console.log(result.finalOutput);
} finally {
  await steel.sessions.release(session.id);
}

Each tool calls run, which sends one request with sessions.commands and turns the response into text for the model. navigate and snapshot return the page as a snapshot, where each element that the agent can act on has a ref. click, fill and press take a ref or a key, and return only the lines of the page that changed. A failed command comes back as text too, such as a ref that is no longer on the page, so the model can take a new snapshot and try again.

The agent's own run is imported as runAgent in TypeScript, so that the name run is free for the function that sends commands. Each request sends the next of the last response as after. A retry of that request runs its commands once, as Build your own agent harness explains. The program releases the session in finally.

In TypeScript, the url parameter is a plain z.string(). OpenAI's strict JSON schema does not accept the uri format that z.string().url() adds.

Run the program

npx tsx agent.mts

The program prints the session viewer URL first. Open it to watch the agent work in the browser.

Session viewer: https://app.steel.dev/sessions/ef897af0-ba75-4157-8f1d-36efa65b4c53

The agent types each todo into the field and presses Enter, then checks the box next to Buy milk. At the end, the program prints the agent's answer: one item is left, Walk the dog. The wording changes from run to run.

Add tools, typed output and Steel features

  • To give the agent more actions, add tools that send other commands, such as select, scroll or wait. Commands reference lists each command and its fields.
  • To get typed output, set outputType (output_type in Python) on the agent to a Zod schema or a Pydantic model. finalOutput (final_output in Python) then has that type.
  • To add Steel features, pass them to steel.sessions.create(), for example useProxy (use_proxy) for a Steel proxy and solveCaptcha (solve_captcha) for CAPTCHA solving. The Agents SDK does not see them. Session configuration lists every option.

Next steps