---
site: steel.dev
collection: docs
canonical_url: https://steel.dev/docs/integrations/openai-computer-use
markdown_url: https://steel.dev/docs/integrations/openai-computer-use.md
content_signal: ai-input=yes, ai-train=yes, search=yes
title: Run OpenAI computer use on Steel
description: Connect OpenAI's computer-use tool to a Steel session, so the model sees screenshots and controls the browser.
summary: Connect OpenAI's computer-use tool to a Steel session, so the model sees screenshots and controls the browser.
type: integration-guide
reviewed: 2026-11-12
prerequisites: []
related:
  - https://steel.dev/docs/integrations/claude-computer-use.md
  - https://steel.dev/docs/integrations/gemini-computer-use.md
agent_setup: https://setup.steel.dev/install.md
docs_index: https://steel.dev/docs/sitemap.md
---

# Run OpenAI computer use on Steel

OpenAI's computer-use tool runs on the Responses API. At each step, the model looks at a screenshot and returns a `computer_call` with a batch of actions, such as a click at pixel coordinates and a key press. Your program runs the batch in a Steel session with [Computer actions](https://steel.dev/docs/control/computer-actions), then sends one screenshot back for the whole call.

The program calls the Responses API with `fetch` (`requests` in Python), so it needs no OpenAI SDK. To route the session through a [Steel proxy](https://steel.dev/docs/stealth/proxies) or to [solve CAPTCHAs](https://steel.dev/docs/stealth/captcha-solving), set those options when you create the session.

## Before you begin

- A Steel API key in the `STEEL_API_KEY` environment variable. See [API keys](https://steel.dev/docs/account/api-keys).
- An OpenAI API key in the `OPENAI_API_KEY` environment variable, for a model that supports computer use.
- Node.js 20 or later, or Python 3.10 or later.

Install the Steel SDK. In Python, also install `requests` for the HTTP calls:

**TypeScript**

```sh
npm install steel-sdk
```

**Python**

```sh
python3 -m venv .venv
source .venv/bin/activate
pip install steel-sdk requests
```

## Set up the integration

### Connect the model to a session

This program creates a session, gives the model a task, and runs each action that the model asks for in the session. It stops when the model returns no action or asks for a safety check, and it releases the session in `finally`.

**TypeScript**

```ts file=openai-computer-use.mts
import Steel from "steel-sdk";

const WIDTH = 1440;
const HEIGHT = 900;
const TASK = "Open https://example.com and tell me the title of the page.";

const KEY_NAMES: Record<string, string> = {
  RETURN: "Enter", ENTER: "Enter", ESC: "Escape", TAB: "Tab",
  BACKSPACE: "Backspace", DELETE: "Delete", SPACE: "Space",
  CTRL: "Control", ALT: "Alt", SHIFT: "Shift", CMD: "Meta", SUPER: "Meta",
  UP: "ArrowUp", DOWN: "ArrowDown", LEFT: "ArrowLeft", RIGHT: "ArrowRight",
  PAGEUP: "PageUp", PAGEDOWN: "PageDown", HOME: "Home", END: "End",
};
const BUTTONS: Record<string, "left" | "right" | "middle"> = {
  left: "left",
  right: "right",
  wheel: "middle",
};

const steel = new Steel();

async function createResponse(params: Record<string, unknown>): Promise<any> {
  const response = await fetch("https://api.openai.com/v1/responses", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(params),
  });
  if (!response.ok) {
    throw new Error(`OpenAI API error: ${response.status} ${await response.text()}`);
  }
  return response.json();
}

function toKey(key: string): string {
  const name = KEY_NAMES[key.toUpperCase()];
  if (name) return name;
  return key.length === 1 ? key.toLowerCase() : key;
}

function toSteelAction(action: any): Steel.SessionComputerParams {
  const coordinates = [action.x ?? WIDTH / 2, action.y ?? HEIGHT / 2];
  switch (action.type) {
    case "click":
      return {
        action: "click_mouse",
        button: BUTTONS[action.button] ?? "left",
        coordinates,
      };
    case "double_click":
      return { action: "click_mouse", button: "left", num_clicks: 2, coordinates };
    case "move":
      return { action: "move_mouse", coordinates };
    case "drag":
      return {
        action: "drag_mouse",
        path: action.path.map((point: any) => [point.x, point.y]),
      };
    case "scroll":
      return {
        action: "scroll",
        coordinates,
        delta_x: action.scroll_x ?? 0,
        delta_y: action.scroll_y ?? 0,
      };
    case "type":
      return { action: "type_text", text: action.text };
    case "keypress":
      return { action: "press_key", keys: action.keys.map(toKey) };
    case "wait":
      return { action: "wait", duration: 1 };
    default:
      return { action: "take_screenshot" };
  }
}

const session = await steel.sessions.create({
  dimensions: { width: WIDTH, height: HEIGHT },
  timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);

let input: unknown[] = [{ role: "user", content: TASK }];
let previousResponseId: string | undefined;

try {
  for (let turn = 0; turn < 50; turn++) {
    const response = await createResponse({
      model: "gpt-6.1-terra",
      input,
      tools: [{ type: "computer" }],
      previous_response_id: previousResponseId,
      reasoning: { effort: "medium" },
      truncation: "auto",
    });
    previousResponseId = response.id;

    const checks = response.output.flatMap(
      (item: any) => item.pending_safety_checks ?? [],
    );
    if (checks.length > 0) {
      for (const check of checks) console.log("Safety check:", check.message);
      console.log("Stopped. A person must approve these checks before the actions run.");
      break;
    }

    const outputs: unknown[] = [];
    let failed = false;
    for (const item of response.output) {
      if (item.type === "message") console.log(item.content?.[0]?.text);
      if (item.type !== "computer_call") continue;

      const actions = item.actions ?? (item.action ? [item.action] : []);
      for (const action of actions) {
        if (failed) break;
        console.log(JSON.stringify(action));
        const result = await steel.sessions.computer(session.id, toSteelAction(action));
        if (result.error) {
          console.log("Steel could not run the action:", result.error);
          failed = true;
        }
      }

      const screenshot = await steel.sessions.computer(session.id, {
        action: "take_screenshot",
      });
      if (screenshot.error) throw new Error(screenshot.error);
      outputs.push({
        type: "computer_call_output",
        call_id: item.call_id,
        output: {
          type: "computer_screenshot",
          image_url: `data:image/png;base64,${screenshot.base64_image}`,
          detail: "original",
        },
      });
    }

    if (outputs.length === 0) break;
    input = outputs;
  }
} finally {
  await steel.sessions.release(session.id);
}
```

**Python**

```python file=openai_computer_use.py
import json
import os

import requests
from steel import Steel

WIDTH = 1440
HEIGHT = 900
TASK = "Open https://example.com and tell me the title of the page."

KEY_NAMES = {
    "RETURN": "Enter", "ENTER": "Enter", "ESC": "Escape", "TAB": "Tab",
    "BACKSPACE": "Backspace", "DELETE": "Delete", "SPACE": "Space",
    "CTRL": "Control", "ALT": "Alt", "SHIFT": "Shift", "CMD": "Meta", "SUPER": "Meta",
    "UP": "ArrowUp", "DOWN": "ArrowDown", "LEFT": "ArrowLeft", "RIGHT": "ArrowRight",
    "PAGEUP": "PageUp", "PAGEDOWN": "PageDown", "HOME": "Home", "END": "End",
}
BUTTONS = {"left": "left", "right": "right", "wheel": "middle"}

steel = Steel()


def create_response(**params):
    response = requests.post(
        "https://api.openai.com/v1/responses",
        headers={
            "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
            "Content-Type": "application/json",
        },
        json=params,
    )
    if response.status_code != 200:
        raise RuntimeError(f"OpenAI API error: {response.status_code} {response.text}")
    return response.json()


def to_key(key):
    name = KEY_NAMES.get(key.upper())
    if name:
        return name
    return key.lower() if len(key) == 1 else key


def to_steel_action(action):
    kind = action["type"]
    coordinates = [action.get("x", WIDTH // 2), action.get("y", HEIGHT // 2)]
    if kind == "click":
        return {
            "action": "click_mouse",
            "button": BUTTONS.get(action.get("button"), "left"),
            "coordinates": coordinates,
        }
    if kind == "double_click":
        return {"action": "click_mouse", "button": "left", "num_clicks": 2, "coordinates": coordinates}
    if kind == "move":
        return {"action": "move_mouse", "coordinates": coordinates}
    if kind == "drag":
        return {"action": "drag_mouse", "path": [[point["x"], point["y"]] for point in action["path"]]}
    if kind == "scroll":
        return {
            "action": "scroll",
            "coordinates": coordinates,
            "delta_x": action.get("scroll_x", 0),
            "delta_y": action.get("scroll_y", 0),
        }
    if kind == "type":
        return {"action": "type_text", "text": action["text"]}
    if kind == "keypress":
        return {"action": "press_key", "keys": [to_key(key) for key in action["keys"]]}
    if kind == "wait":
        return {"action": "wait", "duration": 1}
    return {"action": "take_screenshot"}


session = steel.sessions.create(
    dimensions={"width": WIDTH, "height": HEIGHT},
    api_timeout=900000,
)
print("Session viewer:", session.session_viewer_url)

next_input = [{"role": "user", "content": TASK}]
previous_response_id = None

try:
    for turn in range(50):
        params = {
            "model": "gpt-6.1-terra",
            "input": next_input,
            "tools": [{"type": "computer"}],
            "reasoning": {"effort": "medium"},
            "truncation": "auto",
        }
        if previous_response_id:
            params["previous_response_id"] = previous_response_id
        response = create_response(**params)
        previous_response_id = response["id"]

        checks = [
            check
            for item in response["output"]
            for check in item.get("pending_safety_checks") or []
        ]
        if checks:
            for check in checks:
                print("Safety check:", check.get("message"))
            print("Stopped. A person must approve these checks before the actions run.")
            break

        outputs = []
        failed = False
        for item in response["output"]:
            if item["type"] == "message" and item.get("content"):
                print(item["content"][0].get("text", ""))
            if item["type"] != "computer_call":
                continue

            actions = item.get("actions") or ([item["action"]] if item.get("action") else [])
            for action in actions:
                if failed:
                    break
                print(json.dumps(action))
                result = steel.sessions.computer(session.id, **to_steel_action(action))
                if result.error:
                    print("Steel could not run the action:", result.error)
                    failed = True

            screenshot = steel.sessions.computer(session.id, action="take_screenshot")
            if screenshot.error:
                raise RuntimeError(screenshot.error)
            outputs.append(
                {
                    "type": "computer_call_output",
                    "call_id": item["call_id"],
                    "output": {
                        "type": "computer_screenshot",
                        "image_url": f"data:image/png;base64,{screenshot.base64_image}",
                        "detail": "original",
                    },
                }
            )

        if not outputs:
            break
        next_input = outputs
finally:
    steel.sessions.release(session.id)
```

### Run the program

**TypeScript**

```sh
npx tsx openai-computer-use.mts
```

**Python**

```sh
python openai_computer_use.py
```

The program prints the [session viewer](https://steel.dev/docs/observability/live-view#open-the-live-view) URL, then each action in JSON, then the model's answer. Open the session viewer to watch the model work.

### How the program maps the actions

OpenAI and Steel name the same actions differently. `toSteelAction` (`to_steel_action` in Python) translates them:

| OpenAI action | Steel action | Translation |
| - | - | - |
| `click` | `click_mouse` | `wheel`, OpenAI's name for the middle button, becomes `middle`. Any other button, such as `back` or `forward`, becomes `left`. |
| `double_click` | `click_mouse` | `num_clicks` is 2. |
| `move` | `move_mouse` | None. |
| `drag` | `drag_mouse` | Each point of `path` becomes an `[x, y]` pair. |
| `scroll` | `scroll` | `scroll_x` and `scroll_y` become `delta_x` and `delta_y`. Steel turns the wheel one step in the direction of the larger one, whatever its size. |
| `type` | `type_text` | None. |
| `keypress` | `press_key` | `CTRL` becomes `Control`, `ENTER` becomes `Enter`, and a single letter becomes lower case. |
| `wait` | `wait` | OpenAI's `wait` has no duration. The program waits 1 second. |
| `screenshot`, and any other | `take_screenshot` | None. |

### Notes on the program

- The Responses API keeps the conversation on OpenAI's side. After the first turn, each request sends only the new outputs and the `previous_response_id`.
- One `computer_call` can hold several actions. The program runs them in order, then takes one screenshot and returns it with the call's `call_id`.
- When Steel returns an `error` for an action, the program prints it and runs no more actions in that turn. It still returns the screenshot, so the model sees the page as it is and can try another way. If the screenshot itself fails, the program stops.
- Each screenshot goes back with `detail: "original"`, so OpenAI keeps its full resolution and the model's coordinates match the session's screen.
- When a `computer_call` has `pending_safety_checks`, do not run its actions until a person approves them. Then send the checks back in `acknowledged_safety_checks` with the screenshot. This program stops when a check appears.
- An agent loop can run longer than the default `timeout`, so the session has a `timeout` of 15 minutes. [Limits](https://steel.dev/docs/platform/limits#session-length) lists the longest `timeout` that each plan allows.

## Next steps

- [OpenAI computer use documentation](https://developers.openai.com/api/docs/guides/tools-computer-use): The tool, its actions and its safety checks.
- [Set the device and viewport](https://steel.dev/docs/sessions/devices): Change the 1440 by 900 screen that the program gives the model.
- Cookbook starters: [TypeScript](https://github.com/steel-dev/steel-cookbook/tree/main/examples/openai-computer-use-ts), [Python](https://github.com/steel-dev/steel-cookbook/tree/main/examples/openai-computer-use-py), [Go](https://github.com/steel-dev/steel-cookbook/tree/main/examples/openai-computer-use-go) and [Rust](https://github.com/steel-dev/steel-cookbook/tree/main/examples/openai-computer-use-rs).
