Run OpenAI computer use on Steel
OpenAI's computer-use tool runs on the Responses API. At each step, the model looks at a screenshot and returns a computer_call with a batch of actions, such as a click at pixel coordinates and a key press. Your program runs the batch in a Steel session with Computer actions, then sends one screenshot back for the whole call.
The program calls the Responses API with fetch (requests in Python), so it needs no OpenAI SDK. To route the session through a Steel proxy or to solve CAPTCHAs, set those options when you create the session.
Before you begin
- A Steel API key in the
STEEL_API_KEYenvironment variable. See API keys. - An OpenAI API key in the
OPENAI_API_KEYenvironment variable, for a model that supports computer use. - Node.js 20 or later, or Python 3.10 or later.
Install the Steel SDK. In Python, also install requests for the HTTP calls:
npm install steel-sdkpython3 -m venv .venv
source .venv/bin/activate
pip install steel-sdk requestsSet up the integration
Connect the model to a session
This program creates a session, gives the model a task, and runs each action that the model asks for in the session. It stops when the model returns no action or asks for a safety check, and it releases the session in finally.
import Steel from "steel-sdk";
const WIDTH = 1440;
const HEIGHT = 900;
const TASK = "Open https://example.com and tell me the title of the page.";
const KEY_NAMES: Record<string, string> = {
RETURN: "Enter", ENTER: "Enter", ESC: "Escape", TAB: "Tab",
BACKSPACE: "Backspace", DELETE: "Delete", SPACE: "Space",
CTRL: "Control", ALT: "Alt", SHIFT: "Shift", CMD: "Meta", SUPER: "Meta",
UP: "ArrowUp", DOWN: "ArrowDown", LEFT: "ArrowLeft", RIGHT: "ArrowRight",
PAGEUP: "PageUp", PAGEDOWN: "PageDown", HOME: "Home", END: "End",
};
const BUTTONS: Record<string, "left" | "right" | "middle"> = {
left: "left",
right: "right",
wheel: "middle",
};
const steel = new Steel();
async function createResponse(params: Record<string, unknown>): Promise<any> {
const response = await fetch("https://api.openai.com/v1/responses", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify(params),
});
if (!response.ok) {
throw new Error(`OpenAI API error: ${response.status} ${await response.text()}`);
}
return response.json();
}
function toKey(key: string): string {
const name = KEY_NAMES[key.toUpperCase()];
if (name) return name;
return key.length === 1 ? key.toLowerCase() : key;
}
function toSteelAction(action: any): Steel.SessionComputerParams {
const coordinates = [action.x ?? WIDTH / 2, action.y ?? HEIGHT / 2];
switch (action.type) {
case "click":
return {
action: "click_mouse",
button: BUTTONS[action.button] ?? "left",
coordinates,
};
case "double_click":
return { action: "click_mouse", button: "left", num_clicks: 2, coordinates };
case "move":
return { action: "move_mouse", coordinates };
case "drag":
return {
action: "drag_mouse",
path: action.path.map((point: any) => [point.x, point.y]),
};
case "scroll":
return {
action: "scroll",
coordinates,
delta_x: action.scroll_x ?? 0,
delta_y: action.scroll_y ?? 0,
};
case "type":
return { action: "type_text", text: action.text };
case "keypress":
return { action: "press_key", keys: action.keys.map(toKey) };
case "wait":
return { action: "wait", duration: 1 };
default:
return { action: "take_screenshot" };
}
}
const session = await steel.sessions.create({
dimensions: { width: WIDTH, height: HEIGHT },
timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);
let input: unknown[] = [{ role: "user", content: TASK }];
let previousResponseId: string | undefined;
try {
for (let turn = 0; turn < 50; turn++) {
const response = await createResponse({
model: "gpt-6.1-terra",
input,
tools: [{ type: "computer" }],
previous_response_id: previousResponseId,
reasoning: { effort: "medium" },
truncation: "auto",
});
previousResponseId = response.id;
const checks = response.output.flatMap(
(item: any) => item.pending_safety_checks ?? [],
);
if (checks.length > 0) {
for (const check of checks) console.log("Safety check:", check.message);
console.log("Stopped. A person must approve these checks before the actions run.");
break;
}
const outputs: unknown[] = [];
let failed = false;
for (const item of response.output) {
if (item.type === "message") console.log(item.content?.[0]?.text);
if (item.type !== "computer_call") continue;
const actions = item.actions ?? (item.action ? [item.action] : []);
for (const action of actions) {
if (failed) break;
console.log(JSON.stringify(action));
const result = await steel.sessions.computer(session.id, toSteelAction(action));
if (result.error) {
console.log("Steel could not run the action:", result.error);
failed = true;
}
}
const screenshot = await steel.sessions.computer(session.id, {
action: "take_screenshot",
});
if (screenshot.error) throw new Error(screenshot.error);
outputs.push({
type: "computer_call_output",
call_id: item.call_id,
output: {
type: "computer_screenshot",
image_url: `data:image/png;base64,${screenshot.base64_image}`,
detail: "original",
},
});
}
if (outputs.length === 0) break;
input = outputs;
}
} finally {
await steel.sessions.release(session.id);
}import json
import os
import requests
from steel import Steel
WIDTH = 1440
HEIGHT = 900
TASK = "Open https://example.com and tell me the title of the page."
KEY_NAMES = {
"RETURN": "Enter", "ENTER": "Enter", "ESC": "Escape", "TAB": "Tab",
"BACKSPACE": "Backspace", "DELETE": "Delete", "SPACE": "Space",
"CTRL": "Control", "ALT": "Alt", "SHIFT": "Shift", "CMD": "Meta", "SUPER": "Meta",
"UP": "ArrowUp", "DOWN": "ArrowDown", "LEFT": "ArrowLeft", "RIGHT": "ArrowRight",
"PAGEUP": "PageUp", "PAGEDOWN": "PageDown", "HOME": "Home", "END": "End",
}
BUTTONS = {"left": "left", "right": "right", "wheel": "middle"}
steel = Steel()
def create_response(**params):
response = requests.post(
"https://api.openai.com/v1/responses",
headers={
"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
"Content-Type": "application/json",
},
json=params,
)
if response.status_code != 200:
raise RuntimeError(f"OpenAI API error: {response.status_code} {response.text}")
return response.json()
def to_key(key):
name = KEY_NAMES.get(key.upper())
if name:
return name
return key.lower() if len(key) == 1 else key
def to_steel_action(action):
kind = action["type"]
coordinates = [action.get("x", WIDTH // 2), action.get("y", HEIGHT // 2)]
if kind == "click":
return {
"action": "click_mouse",
"button": BUTTONS.get(action.get("button"), "left"),
"coordinates": coordinates,
}
if kind == "double_click":
return {"action": "click_mouse", "button": "left", "num_clicks": 2, "coordinates": coordinates}
if kind == "move":
return {"action": "move_mouse", "coordinates": coordinates}
if kind == "drag":
return {"action": "drag_mouse", "path": [[point["x"], point["y"]] for point in action["path"]]}
if kind == "scroll":
return {
"action": "scroll",
"coordinates": coordinates,
"delta_x": action.get("scroll_x", 0),
"delta_y": action.get("scroll_y", 0),
}
if kind == "type":
return {"action": "type_text", "text": action["text"]}
if kind == "keypress":
return {"action": "press_key", "keys": [to_key(key) for key in action["keys"]]}
if kind == "wait":
return {"action": "wait", "duration": 1}
return {"action": "take_screenshot"}
session = steel.sessions.create(
dimensions={"width": WIDTH, "height": HEIGHT},
api_timeout=900000,
)
print("Session viewer:", session.session_viewer_url)
next_input = [{"role": "user", "content": TASK}]
previous_response_id = None
try:
for turn in range(50):
params = {
"model": "gpt-6.1-terra",
"input": next_input,
"tools": [{"type": "computer"}],
"reasoning": {"effort": "medium"},
"truncation": "auto",
}
if previous_response_id:
params["previous_response_id"] = previous_response_id
response = create_response(**params)
previous_response_id = response["id"]
checks = [
check
for item in response["output"]
for check in item.get("pending_safety_checks") or []
]
if checks:
for check in checks:
print("Safety check:", check.get("message"))
print("Stopped. A person must approve these checks before the actions run.")
break
outputs = []
failed = False
for item in response["output"]:
if item["type"] == "message" and item.get("content"):
print(item["content"][0].get("text", ""))
if item["type"] != "computer_call":
continue
actions = item.get("actions") or ([item["action"]] if item.get("action") else [])
for action in actions:
if failed:
break
print(json.dumps(action))
result = steel.sessions.computer(session.id, **to_steel_action(action))
if result.error:
print("Steel could not run the action:", result.error)
failed = True
screenshot = steel.sessions.computer(session.id, action="take_screenshot")
if screenshot.error:
raise RuntimeError(screenshot.error)
outputs.append(
{
"type": "computer_call_output",
"call_id": item["call_id"],
"output": {
"type": "computer_screenshot",
"image_url": f"data:image/png;base64,{screenshot.base64_image}",
"detail": "original",
},
}
)
if not outputs:
break
next_input = outputs
finally:
steel.sessions.release(session.id)Run the program
npx tsx openai-computer-use.mtspython openai_computer_use.pyThe program prints the session viewer URL, then each action in JSON, then the model's answer. Open the session viewer to watch the model work.
How the program maps the actions
OpenAI and Steel name the same actions differently. toSteelAction (to_steel_action in Python) translates them:
| OpenAI action | Steel action | Translation |
|---|---|---|
click | click_mouse | wheel, OpenAI's name for the middle button, becomes middle. Any other button, such as back or forward, becomes left. |
double_click | click_mouse | num_clicks is 2. |
move | move_mouse | None. |
drag | drag_mouse | Each point of path becomes an [x, y] pair. |
scroll | scroll | scroll_x and scroll_y become delta_x and delta_y. Steel turns the wheel one step in the direction of the larger one, whatever its size. |
type | type_text | None. |
keypress | press_key | CTRL becomes Control, ENTER becomes Enter, and a single letter becomes lower case. |
wait | wait | OpenAI's wait has no duration. The program waits 1 second. |
screenshot, and any other | take_screenshot | None. |
Notes on the program
- The Responses API keeps the conversation on OpenAI's side. After the first turn, each request sends only the new outputs and the
previous_response_id. - One
computer_callcan hold several actions. The program runs them in order, then takes one screenshot and returns it with the call'scall_id. - When Steel returns an
errorfor an action, the program prints it and runs no more actions in that turn. It still returns the screenshot, so the model sees the page as it is and can try another way. If the screenshot itself fails, the program stops. - Each screenshot goes back with
detail: "original", so OpenAI keeps its full resolution and the model's coordinates match the session's screen. - When a
computer_callhaspending_safety_checks, do not run its actions until a person approves them. Then send the checks back inacknowledged_safety_checkswith the screenshot. This program stops when a check appears. - An agent loop can run longer than the default
timeout, so the session has atimeoutof 15 minutes. Limits lists the longesttimeoutthat each plan allows.
Next steps
- OpenAI computer use documentation: The tool, its actions and its safety checks.
- Set the device and viewport: Change the 1440 by 900 screen that the program gives the model.
- Cookbook starters: TypeScript, Python, Go and Rust.