Run Claude computer use on Steel
On the current Claude models, computer use is the computer_toolset_20260801 toolset. With it, Claude asks for actions, such as a click at pixel coordinates or a key press, often several in one response. It asks for a screenshot only when it needs to see the screen. Your program runs each action in a Steel session with Computer actions and returns each result to Claude.
Earlier models, such as Claude Opus 4.7, use the computer_20251124 tool, which has different requests and results. Anthropic's computer use documentation lists the tool for each model.
To route the session through a Steel proxy or to solve CAPTCHAs, set those options when you create the session.
Before you begin
- A Steel API key in the
STEEL_API_KEYenvironment variable. See API keys. - An Anthropic API key in the
ANTHROPIC_API_KEYenvironment variable. - Node.js 20 or later, or Python 3.11 or later.
Install the Steel and Anthropic SDKs:
npm install steel-sdk @anthropic-ai/sdkpython3 -m venv .venv
source .venv/bin/activate
pip install steel-sdk anthropicSet up the integration
Connect Claude to a session
This program creates a session, gives Claude a task, and runs each action that Claude asks for in the session, in order. It stops when Claude returns text and no action, and it releases the session in finally.
import Anthropic from "@anthropic-ai/sdk";
import Steel from "steel-sdk";
const TASK = "Open https://example.com and tell me the title of the page.";
const HALTED = "Not executed: an earlier computer action in this turn failed.";
const BUTTONS = {
left_click: "left",
right_click: "right",
middle_click: "middle",
double_click: "left",
} as const;
const WHEEL: Record<string, [number, number]> = {
down: [0, 1],
up: [0, -1],
right: [1, 0],
left: [-1, 0],
};
const steel = new Steel();
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
function toSteelActions(
name: string,
input: any,
at: number[],
): Steel.SessionComputerParams[] {
if (input.text && name !== "type" && name !== "key") {
throw new Error("Steel holds no modifier key during a click or a scroll.");
}
switch (name) {
case "screenshot":
return [{ action: "take_screenshot" }];
case "cursor_position":
return [{ action: "get_cursor_position" }];
case "left_click":
case "right_click":
case "middle_click":
case "double_click":
return [
{
action: "click_mouse",
button: BUTTONS[name],
num_clicks: name === "double_click" ? 2 : 1,
coordinates: at,
},
];
case "left_click_drag":
return [{ action: "drag_mouse", path: [input.start_coordinate, at] }];
case "mouse_move":
return [{ action: "move_mouse", coordinates: at }];
case "scroll": {
const [delta_x, delta_y] = WHEEL[input.scroll_direction];
return Array(input.scroll_amount).fill({
action: "scroll",
coordinates: at,
delta_x,
delta_y,
});
}
case "type":
return [{ action: "type_text", text: input.text }];
case "key":
return Array(input.repeat ?? 1).fill({
action: "press_key",
keys: input.text.split("+"),
});
case "wait":
return [{ action: "wait", duration: input.duration }];
default:
throw new Error(`Steel has no action for ${name}.`);
}
}
const session = await steel.sessions.create({
dimensions: { width: 1280, height: 768 },
timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);
const messages: Anthropic.MessageParam[] = [{ role: "user", content: TASK }];
let cursor = [0, 0];
try {
for (let turn = 0; turn < 50; turn++) {
const response = await anthropic.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
messages,
tools: [
{
type: "computer_toolset_20260801",
configs: {
zoom: { enabled: false },
hold_key: { enabled: false },
triple_click: { enabled: false },
left_mouse_down: { enabled: false },
left_mouse_up: { enabled: false },
},
},
],
});
messages.push({ role: "assistant", content: response.content });
if (response.stop_reason === "refusal") console.log("Claude declined the task.");
const results: Anthropic.ToolResultBlockParam[] = [];
let failed = false;
for (const block of response.content) {
if (block.type === "text") console.log(block.text);
if (block.type !== "tool_use") continue;
console.log(block.name, JSON.stringify(block.input));
const input = block.input as any;
let content: Anthropic.ToolResultBlockParam["content"] = HALTED;
if (!failed) {
try {
const at = input.coordinate ?? cursor;
let result: Steel.SessionComputerResponse = {};
for (const action of toSteelActions(block.name, input, at)) {
result = await steel.sessions.computer(session.id, action);
if (result.error) throw new Error(result.error);
}
cursor = at;
content = result.base64_image
? [
{
type: "image",
source: {
type: "base64",
media_type: "image/png",
data: result.base64_image,
},
},
]
: block.name === "cursor_position"
? (result.output ?? "")
: "OK";
} catch (error) {
content = String(error);
failed = true;
}
}
results.push({
type: "tool_result",
tool_use_id: block.id,
toolset_name: "computer",
content,
is_error: failed,
});
}
if (results.length === 0) break;
messages.push({ role: "user", content: results });
}
} finally {
await steel.sessions.release(session.id);
}import json
import os
from anthropic import Anthropic
from steel import Steel
TASK = "Open https://example.com and tell me the title of the page."
HALTED = "Not executed: an earlier computer action in this turn failed."
BUTTONS = {
"left_click": "left",
"right_click": "right",
"middle_click": "middle",
"double_click": "left",
}
WHEEL = {"down": (0, 1), "up": (0, -1), "right": (1, 0), "left": (-1, 0)}
steel = Steel()
anthropic = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
def to_steel_actions(name, tool_input, at):
if tool_input.get("text") and name not in ("type", "key"):
raise ValueError("Steel holds no modifier key during a click or a scroll.")
if name == "screenshot":
return [{"action": "take_screenshot"}]
if name == "cursor_position":
return [{"action": "get_cursor_position"}]
if name in BUTTONS:
return [
{
"action": "click_mouse",
"button": BUTTONS[name],
"num_clicks": 2 if name == "double_click" else 1,
"coordinates": at,
}
]
if name == "left_click_drag":
return [{"action": "drag_mouse", "path": [tool_input["start_coordinate"], at]}]
if name == "mouse_move":
return [{"action": "move_mouse", "coordinates": at}]
if name == "scroll":
delta_x, delta_y = WHEEL[tool_input["scroll_direction"]]
step = {"action": "scroll", "coordinates": at, "delta_x": delta_x, "delta_y": delta_y}
return [step] * tool_input["scroll_amount"]
if name == "type":
return [{"action": "type_text", "text": tool_input["text"]}]
if name == "key":
press = {"action": "press_key", "keys": tool_input["text"].split("+")}
return [press] * tool_input.get("repeat", 1)
if name == "wait":
return [{"action": "wait", "duration": tool_input["duration"]}]
raise ValueError(f"Steel has no action for {name}.")
session = steel.sessions.create(
dimensions={"width": 1280, "height": 768},
api_timeout=900000,
)
print("Session viewer:", session.session_viewer_url)
messages = [{"role": "user", "content": TASK}]
cursor = [0, 0]
try:
for turn in range(50):
response = anthropic.messages.create(
model="claude-opus-5-5",
max_tokens=16000,
messages=messages,
tools=[
{
"type": "computer_toolset_20260801",
"configs": {
"zoom": {"enabled": False},
"hold_key": {"enabled": False},
"triple_click": {"enabled": False},
"left_mouse_down": {"enabled": False},
"left_mouse_up": {"enabled": False},
},
}
],
)
messages.append({"role": "assistant", "content": response.content})
if response.stop_reason == "refusal":
print("Claude declined the task.")
results = []
failed = False
for block in response.content:
if block.type == "text":
print(block.text)
if block.type != "tool_use":
continue
print(block.name, json.dumps(block.input))
content = HALTED
if not failed:
try:
at = block.input.get("coordinate") or cursor
result = None
for action in to_steel_actions(block.name, block.input, at):
result = steel.sessions.computer(session.id, **action)
if result.error:
raise RuntimeError(result.error)
cursor = at
if result.base64_image:
content = [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": result.base64_image,
},
}
]
elif block.name == "cursor_position":
content = result.output or ""
else:
content = "OK"
except Exception as error:
content = str(error)
failed = True
results.append(
{
"type": "tool_result",
"tool_use_id": block.id,
"toolset_name": "computer",
"content": content,
"is_error": failed,
}
)
if not results:
break
messages.append({"role": "user", "content": results})
finally:
steel.sessions.release(session.id)Run the program
npx tsx claude-computer-use.mtspython claude_computer_use.pyThe program prints the session viewer URL, each action that Claude takes, and Claude's answer. Open the session viewer to watch Claude work. Your output varies from run to run.
Session viewer: https://app.steel.dev/sessions/384d94f4-cdeb-4eae-990d-f1dd33fac6db
I'll start by taking a screenshot to see what's on the screen.
screenshot {}
left_click {"coordinate":[656,63]}
key {"text":"ctrl+a"}
type {"text":"https://example.com\n"}
wait {"duration":2}
screenshot {}
The page title of https://example.com is **"Example Domain"**, as shown on the browser tab.
The page says the domain is for use in documentation examples and isn't meant for testing or monitoring. That notice appears in several languages, including Arabic, Chinese, French, Russian and Spanish.Session viewer: https://app.steel.dev/sessions/b9c91108-0d8a-45c7-8f6b-7e933af8544f
screenshot {}
left_click {"coordinate": [656, 63]}
key {"text": "ctrl+a"}
type {"text": "https://example.com\n"}
wait {"duration": 2}
screenshot {}
The page title is **"Example Domain"**, as shown on the browser tab after loading https://example.com.
The page itself explains that the domain is for use in documentation examples and that it isn't a service.How the program maps the actions
Claude and Steel name the same actions differently. toSteelActions (to_steel_actions in Python) turns one Claude action into one or more Steel actions:
| Claude action | Steel action | Translation |
|---|---|---|
screenshot | take_screenshot | None. |
left_click, right_click, middle_click | click_mouse | The button comes from the action name. |
double_click | click_mouse | num_clicks is 2. |
left_click_drag | drag_mouse | The path goes from start_coordinate to coordinate. |
mouse_move | move_mouse | None. |
scroll | scroll | One Steel action for each step of scroll_amount, because Steel turns the wheel one step for each action. |
type | type_text | None. |
key | press_key | ctrl+l becomes ["ctrl", "l"]. The program sends one Steel action for each repeat. |
wait | wait | duration passes through. Both count it in seconds. |
cursor_position | get_cursor_position | Claude gets the output of the Steel action as text. |
Notes on the program
- Claude gives coordinates in pixels of the screenshots that it receives. A Steel screenshot has the size of the session's
dimensions, so the program passes coordinates through without change. - Only
screenshotreturns an image to Claude.cursor_positionreturns the position as text, and every other result is the textOK. - Each result repeats
toolset_name: "computer". Claude's API rejects a result without it. - If an action fails, the program does not run the later actions of the same response. Each of them gets the error text that Anthropic's documentation gives for this case, and Claude plans again.
- The toolset's
configsturn off the actions that have no Steel equivalent:zoom,hold_key,triple_click,left_mouse_downandleft_mouse_up. Claude then does not ask for them. - Steel holds no modifier key during a click or a scroll. If Claude asks for one, the program returns an error, and Claude tries another way.
- Without
coordinate, a Claude action acts at the cursor position. Steel clicks at[0, 0]when an action has nocoordinates, so the program tracks the cursor position and sends it. press_keyalso accepts the key names that Claude uses, such asReturnandPage_Down, so the program passes Claude's keys through. Computer actions describespress_key.- An agent loop can run longer than the default
timeout, so the session has atimeoutof 15 minutes. Limits lists the longesttimeoutthat each plan allows.
Next steps
- Anthropic computer use documentation: The toolset, its actions and the models that support it.
- Cookbook starters: TypeScript, Python, Go, Rust, and a mobile viewport variant. They use the earlier
computer_20251124tool with Claude Opus 4.7.