---
site: steel.dev
collection: docs
canonical_url: https://steel.dev/docs/control/computer-actions
markdown_url: https://steel.dev/docs/control/computer-actions.md
content_signal: ai-input=yes, ai-train=yes, search=yes
title: Computer actions
description: Move the mouse, press keys, scroll, and take screenshots in a session with one endpoint, built for computer-use models.
summary: Move the mouse, press keys, scroll, and take screenshots in a session with one endpoint, built for computer-use models.
type: how-to
reviewed: 2026-11-12
prerequisites:
  - https://steel.dev/docs/control.md
  - https://steel.dev/docs/control/snapshots-and-refs.md
related:
  - https://steel.dev/docs/control/skills.md
  - https://steel.dev/docs/control/mcp-server.md
  - https://steel.dev/docs/control/cdp.md
  - https://steel.dev/docs/control/commands.md
  - https://steel.dev/docs/control/scrape.md
agent_setup: https://setup.steel.dev/install.md
docs_index: https://steel.dev/docs/sitemap.md
---

# Computer actions

Computer actions send mouse and keyboard input to a session at pixel coordinates, and return a screenshot of the screen. A computer-use model, such as Claude, OpenAI or Gemini computer use, looks at a screenshot and answers with an action such as "click at 640, 412". Computer actions run that action in the session and return the next screenshot in the same call. Use them when your agent works from screenshots. When your agent reads the page as text and acts on elements, use [Commands](https://steel.dev/docs/control/commands) instead.

## Before you begin

Put your Steel API key in the `STEEL_API_KEY` environment variable, and install the Steel SDK for your language: [TypeScript](https://steel.dev/docs/reference/sdks/typescript#install), [Python](https://steel.dev/docs/reference/sdks/python#install), [Go](https://steel.dev/docs/reference/sdks/go#install) or [Rust](https://steel.dev/docs/reference/sdks/rust#install).

## Send an action

Call `client.sessions.computer` with the session ID and one action. Set `screenshot` to `true` on an action to get a screenshot after it, in `base64_image`.

This program opens a page the way a person does. It focuses the address bar with Control+L, types the URL and presses Enter. Then it waits two seconds, saves a screenshot, and releases the session.

**TypeScript**

```ts file=computer.mts
import { writeFile } from "node:fs/promises";
import Steel from "steel-sdk";

const client = new Steel();
const session = await client.sessions.create({
  dimensions: { width: 1280, height: 768 },
});

try {
  await client.sessions.computer(session.id, { action: "press_key", keys: ["Control", "l"] });
  await client.sessions.computer(session.id, { action: "type_text", text: "https://example.com" });
  await client.sessions.computer(session.id, { action: "press_key", keys: ["Enter"] });
  const result = await client.sessions.computer(session.id, {
    action: "wait",
    duration: 2,
    screenshot: true,
  });
  if (result.base64_image) {
    await writeFile("screen.png", Buffer.from(result.base64_image, "base64"));
    console.log("Saved screen.png");
  }
} finally {
  await client.sessions.release(session.id);
}
```

**Python**

```python file=computer.py
import base64

from steel import Steel

client = Steel()
session = client.sessions.create(dimensions={"width": 1280, "height": 768})

try:
    client.sessions.computer(session.id, action="press_key", keys=["Control", "l"])
    client.sessions.computer(session.id, action="type_text", text="https://example.com")
    client.sessions.computer(session.id, action="press_key", keys=["Enter"])
    result = client.sessions.computer(
        session.id,
        action="wait",
        duration=2,
        screenshot=True,
    )
    if result.base64_image:
        with open("screen.png", "wb") as file:
            file.write(base64.b64decode(result.base64_image))
        print("Saved screen.png")
finally:
    client.sessions.release(session.id)
```

**Go**

```go file=main.go
package main

import (
	"context"
	"encoding/base64"
	"fmt"
	"log"
	"os"

	"github.com/steel-dev/sdk/go"
)

func main() {
	if err := run(context.Background()); err != nil {
		log.Fatal(err)
	}
}

func run(ctx context.Context) error {
	client := steel.NewClient()
	session, err := client.Sessions.New(ctx, steel.SessionNewParams{
		Dimensions: steel.F(steel.SessionNewParamsDimensions{
			Width:  steel.F(int64(1280)),
			Height: steel.F(int64(768)),
		}),
	})
	if err != nil {
		return err
	}
	defer func() {
		if _, err := client.Sessions.Release(ctx, session.ID); err != nil {
			log.Print(err)
		}
	}()

	_, err = client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
		Body: steel.SessionComputerParamsBodyPressKey{
			Action: steel.F(steel.SessionComputerParamsBodyPressKeyActionPressKey),
			Keys:   steel.F([]string{"Control", "l"}),
		},
	})
	if err != nil {
		return err
	}
	_, err = client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
		Body: steel.SessionComputerParamsBodyTypeText{
			Action: steel.F(steel.SessionComputerParamsBodyTypeTextActionTypeText),
			Text:   steel.F("https://example.com"),
		},
	})
	if err != nil {
		return err
	}
	_, err = client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
		Body: steel.SessionComputerParamsBodyPressKey{
			Action: steel.F(steel.SessionComputerParamsBodyPressKeyActionPressKey),
			Keys:   steel.F([]string{"Enter"}),
		},
	})
	if err != nil {
		return err
	}
	result, err := client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
		Body: steel.SessionComputerParamsBodyWait{
			Action:     steel.F(steel.SessionComputerParamsBodyWaitActionWait),
			Duration:   steel.F(2.0),
			Screenshot: steel.F(true),
		},
	})
	if err != nil {
		return err
	}
	if result.Base64Image != "" {
		image, err := base64.StdEncoding.DecodeString(result.Base64Image)
		if err != nil {
			return err
		}
		if err := os.WriteFile("screen.png", image, 0o644); err != nil {
			return err
		}
		fmt.Println("Saved screen.png")
	}
	return nil
}
```

**Rust**

```rust file=src/main.rs
use base64::Engine;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = steel_rs::Client::new();
    let session = client
        .sessions()
        .create(steel_rs::SessionCreateParams {
            dimensions: Some(steel_rs::SessionCreateParamsDimensions {
                width: 1280,
                height: 768,
            }),
            ..Default::default()
        })
        .await?;

    let result: Result<(), Box<dyn std::error::Error>> = async {
        client
            .sessions()
            .computer(
                &session.id,
                steel_rs::SessionComputerParams::PressKey(
                    steel_rs::SessionComputerParamsPressKey::new(vec![
                        "Control".to_owned(),
                        "l".to_owned(),
                    ]),
                ),
            )
            .await?;
        client
            .sessions()
            .computer(
                &session.id,
                steel_rs::SessionComputerParams::TypeText(
                    steel_rs::SessionComputerParamsTypeText::new("https://example.com".to_owned()),
                ),
            )
            .await?;
        client
            .sessions()
            .computer(
                &session.id,
                steel_rs::SessionComputerParams::PressKey(
                    steel_rs::SessionComputerParamsPressKey::new(vec!["Enter".to_owned()]),
                ),
            )
            .await?;
        let result = client
            .sessions()
            .computer(
                &session.id,
                steel_rs::SessionComputerParams::Wait(steel_rs::SessionComputerParamsWait {
                    screenshot: Some(true),
                    ..steel_rs::SessionComputerParamsWait::new(2.0)
                }),
            )
            .await?;
        if let Some(image) = result.base64_image {
            std::fs::write(
                "screen.png",
                base64::engine::general_purpose::STANDARD.decode(image)?,
            )?;
            println!("Saved screen.png");
        }
        Ok(())
    }
    .await;

    client.sessions().release(&session.id).await?;
    result
}
```

In Rust, the program also needs `cargo add base64`. Run the program:

**TypeScript**

```sh
npx tsx computer.mts
```

**Python**

```sh
python computer.py
```

**Go**

```sh
go run .
```

**Rust**

```sh
cargo run
```

The program saves a PNG screenshot of the page in `screen.png`:

```text nocopy
Saved screen.png
```

Each action has its own fields. `coordinates` and the points of `path` are `[x, y]` in screen pixels.

| Action | Required fields | Optional fields |
| - | - | - |
| `move_mouse` | `coordinates` | `screenshot` |
| `click_mouse` | none | `coordinates`, `button` (`left`, `right` or `middle`; `left` by default), `num_clicks` (2 for a double click; 1 by default), `screenshot` |
| `drag_mouse` | `path`, an array of `[x, y]` points | `screenshot` |
| `scroll` | none | `coordinates`, `delta_x`, `delta_y` (0 by default), `screenshot` |
| `press_key` | `keys`, an array of keys pressed together | `screenshot` |
| `type_text` | `text` | `screenshot` |
| `wait` | `duration` in seconds | `screenshot` |
| `take_screenshot` | none | none |
| `get_cursor_position` | none | none |

`click_mouse` clicks once for any `num_clicks` other than 2. `drag_mouse` holds the left button along the path.

Without `coordinates`, `click_mouse` and `scroll` act at `[0, 0]`, the top-left corner of the screen, and not at the cursor position. Send `coordinates` with each of these actions.

`scroll` turns the mouse wheel one step at `coordinates`, in the direction of the larger of `delta_x` and `delta_y`. A positive `delta_y` scrolls down and a negative one scrolls up. The size of the delta does not change the distance, so send more `scroll` actions to scroll further.

`press_key` presses one key, with any of `Control`, `Shift`, `Alt` and `Meta` held down. Steel ignores the case of modifier names, and `Ctrl` also works for `Control`. Name the key as the browser does, such as `Enter`, `Escape`, `Tab`, `Backspace`, `ArrowLeft`, `PageDown` or `F5`, or give one character, such as `l`. The key names that Claude uses, such as `Return` and `Page_Down`, also work. If `keys` has more than one key that is not a modifier, Steel presses only the last one.

Steel does not support `hold_keys`, `click_type`, the `duration` of `press_key`, or the buttons `back` and `forward`, so do not send them. The API does not reject them, but an action with `hold_keys` holds no key, and `back` and `forward` click the left button.

The response has a message about the action in `output`, and `error` when the action failed. The [Sessions API](https://steel.dev/docs/reference/api/sessions) reference has every field.

## Take a screenshot

`take_screenshot` returns the screen without changing it. A computer-use loop starts with it.

**TypeScript**

```ts
const shot = await client.sessions.computer(session.id, { action: "take_screenshot" });
await writeFile("screen.png", Buffer.from(shot.base64_image ?? "", "base64"));
```

**Python**

```python
shot = client.sessions.computer(session.id, action="take_screenshot")
with open("screen.png", "wb") as file:
    file.write(base64.b64decode(shot.base64_image))
```

**Go**

```go
shot, err := client.Sessions.Computer(ctx, session.ID, steel.SessionComputerParams{
	Body: steel.SessionComputerParamsBodyTakeScreenshot{
		Action: steel.F(steel.SessionComputerParamsBodyTakeScreenshotActionTakeScreenshot),
	},
})
if err != nil {
	return err
}
image, err := base64.StdEncoding.DecodeString(shot.Base64Image)
if err != nil {
	return err
}
if err := os.WriteFile("screen.png", image, 0o644); err != nil {
	return err
}
```

**Rust**

```rust
let shot = client
    .sessions()
    .computer(
        &session.id,
        steel_rs::SessionComputerParams::TakeScreenshot(
            steel_rs::SessionComputerParamsTakeScreenshot::default(),
        ),
    )
    .await?;
std::fs::write(
    "screen.png",
    base64::engine::general_purpose::STANDARD
        .decode(shot.base64_image.unwrap_or_default())?,
)?;
```

The screenshot has the size of the session's screen. Set that size with `dimensions` when you create the session. See [Set the device and viewport](https://steel.dev/docs/sessions/devices).

## Run a computer-use loop

A computer-use agent repeats these steps until the model has nothing left to do:

1. Take a screenshot with `take_screenshot`.
2. Send the screenshot and task to the model.
3. Turn the action that the model returns into a computer action, and set `screenshot` to `true` on it.
4. Send the new screenshot back to the model as the result of its action. Stop when the model answers with text and no action.

Tell the model the same screen size as the session's `dimensions`. If the two differ, the model's clicks land in the wrong place.

Each model names its actions in its own way, so your code maps them to computer actions. For Claude, the mapping looks like this:

| Claude action | Computer action |
| - | - |
| `left_click`, `right_click`, `double_click` | `click_mouse`, with `button` and `num_clicks` |
| `type` | `type_text` |
| `key` | `press_key`, with the keys split at `+`: `ctrl+l` becomes `["ctrl", "l"]` |
| `scroll` | `scroll`, with the direction turned into the sign of `delta_x` or `delta_y` |
| `screenshot` | `take_screenshot` |

Release the session in `finally`, so that a failed run does not leave a browser running. In Go, release it in a `defer`. In Rust, release it after the `async` block. The integration guides have complete loops for each model:

- [Claude computer use](https://steel.dev/docs/integrations/claude-computer-use)
- [OpenAI computer use](https://steel.dev/docs/integrations/openai-computer-use)
- [Gemini computer use](https://steel.dev/docs/integrations/gemini-computer-use)

## Check that it worked

Open `screen.png`, the file that the program saves. It shows the Example Domain page, and it is 1280 by 768 pixels, the `dimensions` of the session. In a computer-use loop, watch the session in the [live view](https://steel.dev/docs/observability/live-view) to see each click land where the model meant it to.

## Next steps

- [Live view](https://steel.dev/docs/observability/live-view): Watch the agent work in the session.
