Add Steel browser tools to the AI SDK
The Vercel AI SDK is a TypeScript toolkit for AI applications, with typed tools and one interface for many model providers. You give an AI SDK agent tools whose execute functions send Commands to a Steel session. The agent reads the page as a snapshot, in which each element that it can act on has a ref, and it clicks and types by ref. The page loads in a cloud browser, with the session's proxies, CAPTCHA solving and live view. The AI SDK is for TypeScript, so this page has TypeScript only.
Before you begin
- Node.js 22 or later.
- A Steel API key in the
STEEL_API_KEYenvironment variable. See API keys. - A key for your model provider. The program on this page uses Anthropic, and its provider package reads the key from
ANTHROPIC_API_KEY.
Install the AI SDK, a provider package, the Steel SDK and Zod. The program uses AI SDK 7 and its ToolLoopAgent. The tools send Commands with the Steel SDK, so you do not install Playwright.
npm install ai @ai-sdk/anthropic steel-sdk zodSet up the integration
Give the agent browser tools
This program gives an AI SDK agent five browser tools over a Steel session.
import { anthropic } from "@ai-sdk/anthropic";
import { ToolLoopAgent, isStepCount, tool } from "ai";
import Steel from "steel-sdk";
import { z } from "zod";
const steel = new Steel();
const session = await steel.sessions.create();
console.log("Session viewer:", session.sessionViewerUrl);
type Commands = Parameters<typeof steel.sessions.commands>[1]["commands"];
let after: string | undefined;
async function run(commands: Commands, mode: "snapshot" | "diff") {
const response = await steel.sessions.commands(session.id, { commands, after, observe: { mode } });
after = response.next ?? after;
const failed = response.results.find((result) => !result.ok);
return [
response.page ? `${response.page.title} (${response.page.url})` : "",
failed?.error ? `${failed.command} failed: ${failed.error.message}` : "",
response.observe?.snapshot ?? response.observe?.diff ?? "",
]
.filter(Boolean)
.join("\n");
}
try {
const navigate = tool({
description:
"Open a URL. Returns the page as a snapshot: its text, and a ref such as e5 for each element that you can act on.",
inputSchema: z.object({ url: z.string().url() }),
execute: ({ url }) => run([{ command: "navigate", url }], "snapshot"),
});
const snapshot = tool({
description: "Read the current page again, as a snapshot with refs.",
inputSchema: z.object({}),
execute: () => run([{ command: "snapshot" }], "snapshot"),
});
const click = tool({
description: "Click the element with this ref. Returns the lines of the snapshot that changed.",
inputSchema: z.object({ ref: z.string() }),
execute: ({ ref }) => run([{ command: "click", target: ref }], "diff"),
});
const fill = tool({
description: "Replace the text in the field with this ref. Returns the lines of the snapshot that changed.",
inputSchema: z.object({ ref: z.string(), value: z.string() }),
execute: ({ ref, value }) => run([{ command: "fill", target: ref, value }], "diff"),
});
const press = tool({
description: "Press a key, such as Enter, in the element that has focus. Returns the lines of the snapshot that changed.",
inputSchema: z.object({ key: z.string() }),
execute: ({ key }) => run([{ command: "press", key }], "diff"),
});
const agent = new ToolLoopAgent({
model: anthropic("claude-haiku-5-5"),
instructions: [
"You operate a Steel cloud browser via tools.",
"Call navigate to open a URL. To act on an element, pass its ref from the latest snapshot.",
"Do not invent data.",
].join(" "),
tools: { navigate, snapshot, click, fill, press },
stopWhen: [isStepCount(20)],
});
const result = await agent.generate({
prompt:
"Open https://demo.playwright.dev/todomvc, add the todos Buy milk and Walk the dog, mark Buy milk as completed, and report how many items are left.",
});
console.log(result.text);
} finally {
await steel.sessions.release(session.id);
}Each tool's execute sends one request to the session with steel.sessions.commands, through the run function. navigate and snapshot return the page as a snapshot: its text and structure, with a ref for each element that the agent can act on. click, fill and press act on a ref, and return a diff: only the lines that changed. When a command fails, for example because its ref is no longer on the page, the tool returns the error, and the model can read the page again.
Each request sends the next of the last response as after. A retry of the same request then runs its commands only once, as Build your own agent harness explains. The program releases the session in finally.
Run the program
npx tsx agent.mtsThe program prints the session viewer URL first. Open it to watch the agent work in the browser.
Session viewer: https://app.steel.dev/sessions/ed041f2c-a712-47dc-a1a0-8e3ea29d9a1aIn the viewer, the agent types the two todos into the field, presses Enter after each, and checks the box next to Buy milk. Then the program prints the agent's answer: one item is left, Walk the dog. The wording changes from run to run.
Add tools and Steel features
- To give the agent more actions, add tools that send other commands, such as
select,scrollorwait. Commands reference lists each command and its fields. To act on several elements in one request, pass several commands in one array. - To use another provider, install its package and change
model. - To add Steel features, pass them to
steel.sessions.create(). For example,{ useProxy: true, solveCaptcha: true }adds a Steel proxy and CAPTCHA solving. The AI SDK does not see them. Session configuration lists every option.
Next steps
- AI SDK agents:
ToolLoopAgent, tools and loop control. - Embed a live browser in your app: Show the session next to a chat.
- Cookbook starters: a typed browser agent and a Next.js chat app with the live view.