Computer actions
Computer actions send mouse and keyboard input to a session at pixel coordinates, and return a screenshot of the screen. A computer-use model, such as Claude, OpenAI or Gemini computer use, looks at a screenshot and answers with an action such as "click at 640, 412". Computer actions run that action in the session and return the next screenshot in the same call. Use them when your agent works from screenshots. When your agent reads the page as text and acts on elements, use Commands instead.
Before you begin
Put your Steel API key in the STEEL_API_KEY environment variable, and install the Steel SDK for your language: TypeScript, Python, Go or Rust.
Send an action
Call client.sessions.computer with the session ID and one action. Set screenshot to true on an action to get a screenshot after it, in base64_image.
This program opens a page the way a person does. It focuses the address bar with Control+L, types the URL and presses Enter. Then it waits two seconds, saves a screenshot, and releases the session.
import { writeFile } from "node:fs/promises";
import Steel from "steel-sdk";
const client = new Steel();
const session = await client.sessions.create({
dimensions: { width: 1280, height: 768 },
});
try {
await client.sessions.computer(session.id, { action: "press_key", keys: ["Control", "l"] });
await client.sessions.computer(session.id, { action: "type_text", text: "https://example.com" });
await client.sessions.computer(session.id, { action: "press_key", keys: ["Enter"] });
const result = await client.sessions.computer(session.id, {
action: "wait",
duration: 2,
screenshot: true,
});
if (result.base64_image) {
await writeFile("screen.png", Buffer.from(result.base64_image, "base64"));
console.log("Saved screen.png");
}
} finally {
await client.sessions.release(session.id);
}import base64
from steel import Steel
client = Steel()
session = client.sessions.create(dimensions={"width": 1280, "height": 768})
try:
client.sessions.computer(session.id, action="press_key", keys=["Control", "l"])
client.sessions.computer(session.id, action="type_text", text="https://example.com")
client.sessions.computer(session.id, action="press_key", keys=["Enter"])
result = client.sessions.computer(
session.id,
action="wait",
duration=2,
screenshot=True,
)
if result.base64_image:
with open("screen.png", "wb") as file:
file.write(base64.b64decode(result.base64_image))
print("Saved screen.png")
finally:
client.sessions.release(session.id)package main
import (
"context"
"encoding/base64"
"fmt"
"log"
"os"
"github.com/steel-dev/sdk/go"
)
func main() {
if err := run(context.Background()); err != nil {
log.Fatal(err)
}
}
func run(ctx context.Context) error {
client := steel.NewClient()
session, err := client.Sessions.New(ctx, steel.SessionNewParams{
Dimensions: steel.F(steel.SessionNewParamsDimensions{
Width: steel.F(int64(1280)),
Height: steel.F(int64(768)),
}),
})
if err != nil {
return err
}
defer func() {
if _, err := client.Sessions.Release(ctx, session.ID); err != nil {
log.Print(err)
}
}()
_, err = client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
Body: steel.SessionComputerParamsBodyPressKey{
Action: steel.F(steel.SessionComputerParamsBodyPressKeyActionPressKey),
Keys: steel.F([]string{"Control", "l"}),
},
})
if err != nil {
return err
}
_, err = client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
Body: steel.SessionComputerParamsBodyTypeText{
Action: steel.F(steel.SessionComputerParamsBodyTypeTextActionTypeText),
Text: steel.F("https://example.com"),
},
})
if err != nil {
return err
}
_, err = client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
Body: steel.SessionComputerParamsBodyPressKey{
Action: steel.F(steel.SessionComputerParamsBodyPressKeyActionPressKey),
Keys: steel.F([]string{"Enter"}),
},
})
if err != nil {
return err
}
result, err := client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
Body: steel.SessionComputerParamsBodyWait{
Action: steel.F(steel.SessionComputerParamsBodyWaitActionWait),
Duration: steel.F(2.0),
Screenshot: steel.F(true),
},
})
if err != nil {
return err
}
if result.Base64Image != "" {
image, err := base64.StdEncoding.DecodeString(result.Base64Image)
if err != nil {
return err
}
if err := os.WriteFile("screen.png", image, 0o644); err != nil {
return err
}
fmt.Println("Saved screen.png")
}
return nil
}use base64::Engine;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = steel_rs::Client::new();
let session = client
.sessions()
.create(steel_rs::SessionCreateParams {
dimensions: Some(steel_rs::SessionCreateParamsDimensions {
width: 1280,
height: 768,
}),
..Default::default()
})
.await?;
let result: Result<(), Box<dyn std::error::Error>> = async {
client
.sessions()
.computer(
&session.id,
steel_rs::SessionComputerParams::PressKey(
steel_rs::SessionComputerParamsPressKey::new(vec![
"Control".to_owned(),
"l".to_owned(),
]),
),
)
.await?;
client
.sessions()
.computer(
&session.id,
steel_rs::SessionComputerParams::TypeText(
steel_rs::SessionComputerParamsTypeText::new("https://example.com".to_owned()),
),
)
.await?;
client
.sessions()
.computer(
&session.id,
steel_rs::SessionComputerParams::PressKey(
steel_rs::SessionComputerParamsPressKey::new(vec!["Enter".to_owned()]),
),
)
.await?;
let result = client
.sessions()
.computer(
&session.id,
steel_rs::SessionComputerParams::Wait(steel_rs::SessionComputerParamsWait {
screenshot: Some(true),
..steel_rs::SessionComputerParamsWait::new(2.0)
}),
)
.await?;
if let Some(image) = result.base64_image {
std::fs::write(
"screen.png",
base64::engine::general_purpose::STANDARD.decode(image)?,
)?;
println!("Saved screen.png");
}
Ok(())
}
.await;
client.sessions().release(&session.id).await?;
result
}In Rust, the program also needs cargo add base64. Run the program:
npx tsx computer.mtspython computer.pygo run .cargo runThe program saves a PNG screenshot of the page in screen.png:
Saved screen.pngEach action has its own fields. coordinates and the points of path are [x, y] in screen pixels.
| Action | Required fields | Optional fields |
|---|---|---|
move_mouse | coordinates | screenshot |
click_mouse | none | coordinates, button (left, right or middle; left by default), num_clicks (2 for a double click; 1 by default), screenshot |
drag_mouse | path, an array of [x, y] points | screenshot |
scroll | none | coordinates, delta_x, delta_y (0 by default), screenshot |
press_key | keys, an array of keys pressed together | screenshot |
type_text | text | screenshot |
wait | duration in seconds | screenshot |
take_screenshot | none | none |
get_cursor_position | none | none |
click_mouse clicks once for any num_clicks other than 2. drag_mouse holds the left button along the path.
Without coordinates, click_mouse and scroll act at [0, 0], the top-left corner of the screen, and not at the cursor position. Send coordinates with each of these actions.
scroll turns the mouse wheel one step at coordinates, in the direction of the larger of delta_x and delta_y. A positive delta_y scrolls down and a negative one scrolls up. The size of the delta does not change the distance, so send more scroll actions to scroll further.
press_key presses one key, with any of Control, Shift, Alt and Meta held down. Steel ignores the case of modifier names, and Ctrl also works for Control. Name the key as the browser does, such as Enter, Escape, Tab, Backspace, ArrowLeft, PageDown or F5, or give one character, such as l. The key names that Claude uses, such as Return and Page_Down, also work. If keys has more than one key that is not a modifier, Steel presses only the last one.
Steel does not support hold_keys, click_type, the duration of press_key, or the buttons back and forward, so do not send them. The API does not reject them, but an action with hold_keys holds no key, and back and forward click the left button.
The response has a message about the action in output, and error when the action failed. The Sessions API reference has every field.
Take a screenshot
take_screenshot returns the screen without changing it. A computer-use loop starts with it.
const shot = await client.sessions.computer(session.id, { action: "take_screenshot" });
await writeFile("screen.png", Buffer.from(shot.base64_image ?? "", "base64"));shot = client.sessions.computer(session.id, action="take_screenshot")
with open("screen.png", "wb") as file:
file.write(base64.b64decode(shot.base64_image))shot, err := client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
Body: steel.SessionComputerParamsBodyTakeScreenshot{
Action: steel.F(steel.SessionComputerParamsBodyTakeScreenshotActionTakeScreenshot),
},
})
if err != nil {
return err
}
image, err := base64.StdEncoding.DecodeString(shot.Base64Image)
if err != nil {
return err
}
if err := os.WriteFile("screen.png", image, 0o644); err != nil {
return err
}let shot = client
.sessions()
.computer(
&session.id,
steel_rs::SessionComputerParams::TakeScreenshot(
steel_rs::SessionComputerParamsTakeScreenshot::default(),
),
)
.await?;
std::fs::write(
"screen.png",
base64::engine::general_purpose::STANDARD
.decode(shot.base64_image.unwrap_or_default())?,
)?;The screenshot has the size of the session's screen. Set that size with dimensions when you create the session. See Set the device and viewport.
Run a computer-use loop
A computer-use agent repeats these steps until the model has nothing left to do:
- Take a screenshot with
take_screenshot. - Send the screenshot and task to the model.
- Turn the action that the model returns into a computer action, and set
screenshottotrueon it. - Send the new screenshot back to the model as the result of its action. Stop when the model answers with text and no action.
Tell the model the same screen size as the session's dimensions. If the two differ, the model's clicks land in the wrong place.
Each model names its actions in its own way, so your code maps them to computer actions. For Claude, the mapping looks like this:
| Claude action | Computer action |
|---|---|
left_click, right_click, double_click | click_mouse, with button and num_clicks |
type | type_text |
key | press_key, with the keys split at +: ctrl+l becomes ["ctrl", "l"] |
scroll | scroll, with the direction turned into the sign of delta_x or delta_y |
screenshot | take_screenshot |
Release the session in finally, so that a failed run does not leave a browser running. In Go, release it in a defer. In Rust, release it after the async block. The integration guides have complete loops for each model:
Check that it worked
Open screen.png, the file that the program saves. It shows the Example Domain page, and it is 1280 by 768 pixels, the dimensions of the session. In a computer-use loop, watch the session in the live view to see each click land where the model meant it to.
Next steps
- Live view: Watch the agent work in the session.