Skip to content

Computer actions

Set up your agent

Give your coding agent a Steel browser and the Steel skills, so that it browses and writes Steel code the right way. Learn more

Install the Steel CLI. It signs you in and installs the skills for your coding agents.

curl -fsS https://setup.steel.dev | sh

Or send this prompt to your coding agent, and it does the same:

Read https://setup.steel.dev/install.md and follow it to set up Steel for this coding client.
Then continue my requested task using the installed Steel skill.

Set up your agent →

Computer actions send mouse and keyboard input to a session at pixel coordinates, and return a screenshot of the screen. A computer-use model, such as Claude, OpenAI or Gemini computer use, looks at a screenshot and answers with an action such as "click at 640, 412". Computer actions run that action in the session and return the next screenshot in the same call. Use them when your agent works from screenshots. When your agent reads the page as text and acts on elements, use Commands instead.

Before you begin

Put your Steel API key in the STEEL_API_KEY environment variable, and install the Steel SDK for your language: TypeScript, Python, Go or Rust.

Send an action

Call client.sessions.computer with the session ID and one action. Set screenshot to true on an action to get a screenshot after it, in base64_image.

This program opens a page the way a person does. It focuses the address bar with Control+L, types the URL and presses Enter. Then it waits two seconds, saves a screenshot, and releases the session.

computer.mts
import { writeFile } from "node:fs/promises";
import Steel from "steel-sdk";

const client = new Steel();
const session = await client.sessions.create({
  dimensions: { width: 1280, height: 768 },
});

try {
  await client.sessions.computer(session.id, { action: "press_key", keys: ["Control", "l"] });
  await client.sessions.computer(session.id, { action: "type_text", text: "https://example.com" });
  await client.sessions.computer(session.id, { action: "press_key", keys: ["Enter"] });
  const result = await client.sessions.computer(session.id, {
    action: "wait",
    duration: 2,
    screenshot: true,
  });
  if (result.base64_image) {
    await writeFile("screen.png", Buffer.from(result.base64_image, "base64"));
    console.log("Saved screen.png");
  }
} finally {
  await client.sessions.release(session.id);
}

In Rust, the program also needs cargo add base64. Run the program:

npx tsx computer.mts

The program saves a PNG screenshot of the page in screen.png:

Saved screen.png

Each action has its own fields. coordinates and the points of path are [x, y] in screen pixels.

ActionRequired fieldsOptional fields
move_mousecoordinatesscreenshot
click_mousenonecoordinates, button (left, right or middle; left by default), num_clicks (2 for a double click; 1 by default), screenshot
drag_mousepath, an array of [x, y] pointsscreenshot
scrollnonecoordinates, delta_x, delta_y (0 by default), screenshot
press_keykeys, an array of keys pressed togetherscreenshot
type_texttextscreenshot
waitduration in secondsscreenshot
take_screenshotnonenone
get_cursor_positionnonenone

click_mouse clicks once for any num_clicks other than 2. drag_mouse holds the left button along the path.

Without coordinates, click_mouse and scroll act at [0, 0], the top-left corner of the screen, and not at the cursor position. Send coordinates with each of these actions.

scroll turns the mouse wheel one step at coordinates, in the direction of the larger of delta_x and delta_y. A positive delta_y scrolls down and a negative one scrolls up. The size of the delta does not change the distance, so send more scroll actions to scroll further.

press_key presses one key, with any of Control, Shift, Alt and Meta held down. Steel ignores the case of modifier names, and Ctrl also works for Control. Name the key as the browser does, such as Enter, Escape, Tab, Backspace, ArrowLeft, PageDown or F5, or give one character, such as l. The key names that Claude uses, such as Return and Page_Down, also work. If keys has more than one key that is not a modifier, Steel presses only the last one.

Steel does not support hold_keys, click_type, the duration of press_key, or the buttons back and forward, so do not send them. The API does not reject them, but an action with hold_keys holds no key, and back and forward click the left button.

The response has a message about the action in output, and error when the action failed. The Sessions API reference has every field.

Take a screenshot

take_screenshot returns the screen without changing it. A computer-use loop starts with it.

const shot = await client.sessions.computer(session.id, { action: "take_screenshot" });
await writeFile("screen.png", Buffer.from(shot.base64_image ?? "", "base64"));

The screenshot has the size of the session's screen. Set that size with dimensions when you create the session. See Set the device and viewport.

Run a computer-use loop

A computer-use agent repeats these steps until the model has nothing left to do:

  1. Take a screenshot with take_screenshot.
  2. Send the screenshot and task to the model.
  3. Turn the action that the model returns into a computer action, and set screenshot to true on it.
  4. Send the new screenshot back to the model as the result of its action. Stop when the model answers with text and no action.

Tell the model the same screen size as the session's dimensions. If the two differ, the model's clicks land in the wrong place.

Each model names its actions in its own way, so your code maps them to computer actions. For Claude, the mapping looks like this:

Claude actionComputer action
left_click, right_click, double_clickclick_mouse, with button and num_clicks
typetype_text
keypress_key, with the keys split at +: ctrl+l becomes ["ctrl", "l"]
scrollscroll, with the direction turned into the sign of delta_x or delta_y
screenshottake_screenshot

Release the session in finally, so that a failed run does not leave a browser running. In Go, release it in a defer. In Rust, release it after the async block. The integration guides have complete loops for each model:

Check that it worked

Open screen.png, the file that the program saves. It shows the Example Domain page, and it is 1280 by 768 pixels, the dimensions of the session. In a computer-use loop, watch the session in the live view to see each click land where the model meant it to.

Next steps

  • Live view: Watch the agent work in the session.