Skip to content

Run OpenAI computer use on Steel

Set up your agent

Give your coding agent a Steel browser and the Steel skills, so that it browses and writes Steel code the right way. Learn more

Install the Steel CLI. It signs you in and installs the skills for your coding agents.

curl -fsS https://setup.steel.dev | sh

Or send this prompt to your coding agent, and it does the same:

Read https://setup.steel.dev/install.md and follow it to set up Steel for this coding client.
Then continue my requested task using the installed Steel skill.

Set up your agent →

OpenAI's computer-use tool runs on the Responses API. At each step, the model looks at a screenshot and returns a computer_call with a batch of actions, such as a click at pixel coordinates and a key press. Your program runs the batch in a Steel session with Computer actions, then sends one screenshot back for the whole call.

The program calls the Responses API with fetch (requests in Python), so it needs no OpenAI SDK. To route the session through a Steel proxy or to solve CAPTCHAs, set those options when you create the session.

Before you begin

  • A Steel API key in the STEEL_API_KEY environment variable. See API keys.
  • An OpenAI API key in the OPENAI_API_KEY environment variable, for a model that supports computer use.
  • Node.js 20 or later, or Python 3.10 or later.

Install the Steel SDK. In Python, also install requests for the HTTP calls:

npm install steel-sdk

Set up the integration

Connect the model to a session

This program creates a session, gives the model a task, and runs each action that the model asks for in the session. It stops when the model returns no action or asks for a safety check, and it releases the session in finally.

openai-computer-use.mts
import Steel from "steel-sdk";

const WIDTH = 1440;
const HEIGHT = 900;
const TASK = "Open https://example.com and tell me the title of the page.";

const KEY_NAMES: Record<string, string> = {
  RETURN: "Enter", ENTER: "Enter", ESC: "Escape", TAB: "Tab",
  BACKSPACE: "Backspace", DELETE: "Delete", SPACE: "Space",
  CTRL: "Control", ALT: "Alt", SHIFT: "Shift", CMD: "Meta", SUPER: "Meta",
  UP: "ArrowUp", DOWN: "ArrowDown", LEFT: "ArrowLeft", RIGHT: "ArrowRight",
  PAGEUP: "PageUp", PAGEDOWN: "PageDown", HOME: "Home", END: "End",
};
const BUTTONS: Record<string, "left" | "right" | "middle"> = {
  left: "left",
  right: "right",
  wheel: "middle",
};

const steel = new Steel();

async function createResponse(params: Record<string, unknown>): Promise<any> {
  const response = await fetch("https://api.openai.com/v1/responses", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(params),
  });
  if (!response.ok) {
    throw new Error(`OpenAI API error: ${response.status} ${await response.text()}`);
  }
  return response.json();
}

function toKey(key: string): string {
  const name = KEY_NAMES[key.toUpperCase()];
  if (name) return name;
  return key.length === 1 ? key.toLowerCase() : key;
}

function toSteelAction(action: any): Steel.SessionComputerParams {
  const coordinates = [action.x ?? WIDTH / 2, action.y ?? HEIGHT / 2];
  switch (action.type) {
    case "click":
      return {
        action: "click_mouse",
        button: BUTTONS[action.button] ?? "left",
        coordinates,
      };
    case "double_click":
      return { action: "click_mouse", button: "left", num_clicks: 2, coordinates };
    case "move":
      return { action: "move_mouse", coordinates };
    case "drag":
      return {
        action: "drag_mouse",
        path: action.path.map((point: any) => [point.x, point.y]),
      };
    case "scroll":
      return {
        action: "scroll",
        coordinates,
        delta_x: action.scroll_x ?? 0,
        delta_y: action.scroll_y ?? 0,
      };
    case "type":
      return { action: "type_text", text: action.text };
    case "keypress":
      return { action: "press_key", keys: action.keys.map(toKey) };
    case "wait":
      return { action: "wait", duration: 1 };
    default:
      return { action: "take_screenshot" };
  }
}

const session = await steel.sessions.create({
  dimensions: { width: WIDTH, height: HEIGHT },
  timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);

let input: unknown[] = [{ role: "user", content: TASK }];
let previousResponseId: string | undefined;

try {
  for (let turn = 0; turn < 50; turn++) {
    const response = await createResponse({
      model: "gpt-6.1-terra",
      input,
      tools: [{ type: "computer" }],
      previous_response_id: previousResponseId,
      reasoning: { effort: "medium" },
      truncation: "auto",
    });
    previousResponseId = response.id;

    const checks = response.output.flatMap(
      (item: any) => item.pending_safety_checks ?? [],
    );
    if (checks.length > 0) {
      for (const check of checks) console.log("Safety check:", check.message);
      console.log("Stopped. A person must approve these checks before the actions run.");
      break;
    }

    const outputs: unknown[] = [];
    let failed = false;
    for (const item of response.output) {
      if (item.type === "message") console.log(item.content?.[0]?.text);
      if (item.type !== "computer_call") continue;

      const actions = item.actions ?? (item.action ? [item.action] : []);
      for (const action of actions) {
        if (failed) break;
        console.log(JSON.stringify(action));
        const result = await steel.sessions.computer(session.id, toSteelAction(action));
        if (result.error) {
          console.log("Steel could not run the action:", result.error);
          failed = true;
        }
      }

      const screenshot = await steel.sessions.computer(session.id, {
        action: "take_screenshot",
      });
      if (screenshot.error) throw new Error(screenshot.error);
      outputs.push({
        type: "computer_call_output",
        call_id: item.call_id,
        output: {
          type: "computer_screenshot",
          image_url: `data:image/png;base64,${screenshot.base64_image}`,
          detail: "original",
        },
      });
    }

    if (outputs.length === 0) break;
    input = outputs;
  }
} finally {
  await steel.sessions.release(session.id);
}

Run the program

npx tsx openai-computer-use.mts

The program prints the session viewer URL, then each action in JSON, then the model's answer. Open the session viewer to watch the model work.

How the program maps the actions

OpenAI and Steel name the same actions differently. toSteelAction (to_steel_action in Python) translates them:

OpenAI actionSteel actionTranslation
clickclick_mousewheel, OpenAI's name for the middle button, becomes middle. Any other button, such as back or forward, becomes left.
double_clickclick_mousenum_clicks is 2.
movemove_mouseNone.
dragdrag_mouseEach point of path becomes an [x, y] pair.
scrollscrollscroll_x and scroll_y become delta_x and delta_y. Steel turns the wheel one step in the direction of the larger one, whatever its size.
typetype_textNone.
keypresspress_keyCTRL becomes Control, ENTER becomes Enter, and a single letter becomes lower case.
waitwaitOpenAI's wait has no duration. The program waits 1 second.
screenshot, and any othertake_screenshotNone.

Notes on the program

  • The Responses API keeps the conversation on OpenAI's side. After the first turn, each request sends only the new outputs and the previous_response_id.
  • One computer_call can hold several actions. The program runs them in order, then takes one screenshot and returns it with the call's call_id.
  • When Steel returns an error for an action, the program prints it and runs no more actions in that turn. It still returns the screenshot, so the model sees the page as it is and can try another way. If the screenshot itself fails, the program stops.
  • Each screenshot goes back with detail: "original", so OpenAI keeps its full resolution and the model's coordinates match the session's screen.
  • When a computer_call has pending_safety_checks, do not run its actions until a person approves them. Then send the checks back in acknowledged_safety_checks with the screenshot. This program stops when a check appears.
  • An agent loop can run longer than the default timeout, so the session has a timeout of 15 minutes. Limits lists the longest timeout that each plan allows.

Next steps