Skip to content

Run Claude computer use on Steel

Set up your agent

Give your coding agent a Steel browser and the Steel skills, so that it browses and writes Steel code the right way. Learn more

Install the Steel CLI. It signs you in and installs the skills for your coding agents.

curl -fsS https://setup.steel.dev | sh

Or send this prompt to your coding agent, and it does the same:

Read https://setup.steel.dev/install.md and follow it to set up Steel for this coding client.
Then continue my requested task using the installed Steel skill.

Set up your agent →

On the current Claude models, computer use is the computer_toolset_20260801 toolset. With it, Claude asks for actions, such as a click at pixel coordinates or a key press, often several in one response. It asks for a screenshot only when it needs to see the screen. Your program runs each action in a Steel session with Computer actions and returns each result to Claude.

Earlier models, such as Claude Opus 4.7, use the computer_20251124 tool, which has different requests and results. Anthropic's computer use documentation lists the tool for each model.

To route the session through a Steel proxy or to solve CAPTCHAs, set those options when you create the session.

Before you begin

  • A Steel API key in the STEEL_API_KEY environment variable. See API keys.
  • An Anthropic API key in the ANTHROPIC_API_KEY environment variable.
  • Node.js 20 or later, or Python 3.11 or later.

Install the Steel and Anthropic SDKs:

npm install steel-sdk @anthropic-ai/sdk

Set up the integration

Connect Claude to a session

This program creates a session, gives Claude a task, and runs each action that Claude asks for in the session, in order. It stops when Claude returns text and no action, and it releases the session in finally.

claude-computer-use.mts
import Anthropic from "@anthropic-ai/sdk";
import Steel from "steel-sdk";

const TASK = "Open https://example.com and tell me the title of the page.";
const HALTED = "Not executed: an earlier computer action in this turn failed.";
const BUTTONS = {
  left_click: "left",
  right_click: "right",
  middle_click: "middle",
  double_click: "left",
} as const;
const WHEEL: Record<string, [number, number]> = {
  down: [0, 1],
  up: [0, -1],
  right: [1, 0],
  left: [-1, 0],
};

const steel = new Steel();
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

function toSteelActions(
  name: string,
  input: any,
  at: number[],
): Steel.SessionComputerParams[] {
  if (input.text && name !== "type" && name !== "key") {
    throw new Error("Steel holds no modifier key during a click or a scroll.");
  }
  switch (name) {
    case "screenshot":
      return [{ action: "take_screenshot" }];
    case "cursor_position":
      return [{ action: "get_cursor_position" }];
    case "left_click":
    case "right_click":
    case "middle_click":
    case "double_click":
      return [
        {
          action: "click_mouse",
          button: BUTTONS[name],
          num_clicks: name === "double_click" ? 2 : 1,
          coordinates: at,
        },
      ];
    case "left_click_drag":
      return [{ action: "drag_mouse", path: [input.start_coordinate, at] }];
    case "mouse_move":
      return [{ action: "move_mouse", coordinates: at }];
    case "scroll": {
      const [delta_x, delta_y] = WHEEL[input.scroll_direction];
      return Array(input.scroll_amount).fill({
        action: "scroll",
        coordinates: at,
        delta_x,
        delta_y,
      });
    }
    case "type":
      return [{ action: "type_text", text: input.text }];
    case "key":
      return Array(input.repeat ?? 1).fill({
        action: "press_key",
        keys: input.text.split("+"),
      });
    case "wait":
      return [{ action: "wait", duration: input.duration }];
    default:
      throw new Error(`Steel has no action for ${name}.`);
  }
}

const session = await steel.sessions.create({
  dimensions: { width: 1280, height: 768 },
  timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);

const messages: Anthropic.MessageParam[] = [{ role: "user", content: TASK }];
let cursor = [0, 0];

try {
  for (let turn = 0; turn < 50; turn++) {
    const response = await anthropic.messages.create({
      model: "claude-opus-5-5",
      max_tokens: 16000,
      messages,
      tools: [
        {
          type: "computer_toolset_20260801",
          configs: {
            zoom: { enabled: false },
            hold_key: { enabled: false },
            triple_click: { enabled: false },
            left_mouse_down: { enabled: false },
            left_mouse_up: { enabled: false },
          },
        },
      ],
    });
    messages.push({ role: "assistant", content: response.content });
    if (response.stop_reason === "refusal") console.log("Claude declined the task.");

    const results: Anthropic.ToolResultBlockParam[] = [];
    let failed = false;
    for (const block of response.content) {
      if (block.type === "text") console.log(block.text);
      if (block.type !== "tool_use") continue;
      console.log(block.name, JSON.stringify(block.input));

      const input = block.input as any;
      let content: Anthropic.ToolResultBlockParam["content"] = HALTED;
      if (!failed) {
        try {
          const at = input.coordinate ?? cursor;
          let result: Steel.SessionComputerResponse = {};
          for (const action of toSteelActions(block.name, input, at)) {
            result = await steel.sessions.computer(session.id, action);
            if (result.error) throw new Error(result.error);
          }
          cursor = at;
          content = result.base64_image
            ? [
                {
                  type: "image",
                  source: {
                    type: "base64",
                    media_type: "image/png",
                    data: result.base64_image,
                  },
                },
              ]
            : block.name === "cursor_position"
              ? (result.output ?? "")
              : "OK";
        } catch (error) {
          content = String(error);
          failed = true;
        }
      }
      results.push({
        type: "tool_result",
        tool_use_id: block.id,
        toolset_name: "computer",
        content,
        is_error: failed,
      });
    }

    if (results.length === 0) break;
    messages.push({ role: "user", content: results });
  }
} finally {
  await steel.sessions.release(session.id);
}

Run the program

npx tsx claude-computer-use.mts

The program prints the session viewer URL, each action that Claude takes, and Claude's answer. Open the session viewer to watch Claude work. Your output varies from run to run.

Session viewer: https://app.steel.dev/sessions/384d94f4-cdeb-4eae-990d-f1dd33fac6db
I'll start by taking a screenshot to see what's on the screen.
screenshot {}
left_click {"coordinate":[656,63]}
key {"text":"ctrl+a"}
type {"text":"https://example.com\n"}
wait {"duration":2}
screenshot {}
The page title of https://example.com is **"Example Domain"**, as shown on the browser tab.

The page says the domain is for use in documentation examples and isn't meant for testing or monitoring. That notice appears in several languages, including Arabic, Chinese, French, Russian and Spanish.

How the program maps the actions

Claude and Steel name the same actions differently. toSteelActions (to_steel_actions in Python) turns one Claude action into one or more Steel actions:

Claude actionSteel actionTranslation
screenshottake_screenshotNone.
left_click, right_click, middle_clickclick_mouseThe button comes from the action name.
double_clickclick_mousenum_clicks is 2.
left_click_dragdrag_mouseThe path goes from start_coordinate to coordinate.
mouse_movemove_mouseNone.
scrollscrollOne Steel action for each step of scroll_amount, because Steel turns the wheel one step for each action.
typetype_textNone.
keypress_keyctrl+l becomes ["ctrl", "l"]. The program sends one Steel action for each repeat.
waitwaitduration passes through. Both count it in seconds.
cursor_positionget_cursor_positionClaude gets the output of the Steel action as text.

Notes on the program

  • Claude gives coordinates in pixels of the screenshots that it receives. A Steel screenshot has the size of the session's dimensions, so the program passes coordinates through without change.
  • Only screenshot returns an image to Claude. cursor_position returns the position as text, and every other result is the text OK.
  • Each result repeats toolset_name: "computer". Claude's API rejects a result without it.
  • If an action fails, the program does not run the later actions of the same response. Each of them gets the error text that Anthropic's documentation gives for this case, and Claude plans again.
  • The toolset's configs turn off the actions that have no Steel equivalent: zoom, hold_key, triple_click, left_mouse_down and left_mouse_up. Claude then does not ask for them.
  • Steel holds no modifier key during a click or a scroll. If Claude asks for one, the program returns an error, and Claude tries another way.
  • Without coordinate, a Claude action acts at the cursor position. Steel clicks at [0, 0] when an action has no coordinates, so the program tracks the cursor position and sends it.
  • press_key also accepts the key names that Claude uses, such as Return and Page_Down, so the program passes Claude's keys through. Computer actions describes press_key.
  • An agent loop can run longer than the default timeout, so the session has a timeout of 15 minutes. Limits lists the longest timeout that each plan allows.

Next steps