Skip to content

Run Gemini computer use on Steel

Set up your agent

Give your coding agent a Steel browser and the Steel skills, so that it browses and writes Steel code the right way. Learn more

Install the Steel CLI. It signs you in and installs the skills for your coding agents.

curl -fsS https://setup.steel.dev | sh

Or send this prompt to your coding agent, and it does the same:

Read https://setup.steel.dev/install.md and follow it to set up Steel for this coding client.
Then continue my requested task using the installed Steel skill.

Set up your agent →

Gemini's computer-use tool calls browser functions, such as click, type or navigate. It places every point on a 1000 by 1000 grid, whatever the screen size. Your program scales each point to the session's screen, runs each function as one or more Computer actions in a Steel session, and sends a screenshot back with the function's response.

The program uses gemini-4-hydrogen and the names of its functions. Earlier models name some functions differently: gemini-3-flash-preview calls click_at and type_text_at, for example. The program stops with an error at the first name that it does not know. Computer use in the Gemini API lists the functions of the current models.

To route the session through a Steel proxy or to solve CAPTCHAs, set those options when you create the session.

Before you begin

  • A Steel API key in the STEEL_API_KEY environment variable. See API keys.
  • A Gemini API key in the GEMINI_API_KEY environment variable, with access to gemini-4-hydrogen.
  • Node.js 22 or later, or Python 3.11 or later.

Install the Steel and Google Gen AI SDKs:

npm install steel-sdk @google/genai

Set up the integration

Connect Gemini to a session

This program creates a session, gives Gemini a task, and runs each function that Gemini calls in the session. Before the program runs a function that Gemini marks for confirmation, it asks you in the terminal. It stops when Gemini returns text and no function call, and it releases the session in finally.

gemini-computer-use.mts
import { createInterface } from "node:readline/promises";
import {
  Environment,
  GoogleGenAI,
  ThinkingLevel,
  type Content,
  type FunctionCall,
  type Part,
} from "@google/genai";
import Steel from "steel-sdk";

const WIDTH = 1440;
const HEIGHT = 900;
const TASK = "Open https://example.com and tell me the title of the page.";

const KEY_NAMES: Record<string, string> = {
  RETURN: "Enter", ENTER: "Enter", ESC: "Escape", TAB: "Tab",
  BACKSPACE: "Backspace", DELETE: "Delete", SPACE: "Space",
  CTRL: "Control", ALT: "Alt", SHIFT: "Shift", CMD: "Meta", SUPER: "Meta",
  UP: "ArrowUp", DOWN: "ArrowDown", LEFT: "ArrowLeft", RIGHT: "ArrowRight",
  PAGEUP: "PageUp", PAGEDOWN: "PageDown", HOME: "Home", END: "End",
};

const HALTED = "Not executed: an earlier action in this turn failed.";

const steel = new Steel();
const gemini = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
let currentUrl = "about:blank";

const toX = (x: number) => Math.round((x / 1000) * WIDTH);
const toY = (y: number) => Math.round((y / 1000) * HEIGHT);

function toKeys(keys: string | string[]): string[] {
  const list = Array.isArray(keys) ? keys : keys.split("+");
  return list.map((key) => KEY_NAMES[key.trim().toUpperCase()] ?? key.trim());
}

function openUrl(url: string): Steel.SessionComputerParams[] {
  currentUrl = /^https?:\/\//.test(url) ? url : `https://${url}`;
  return [
    { action: "press_key", keys: ["Control", "l"] },
    { action: "type_text", text: currentUrl },
    { action: "press_key", keys: ["Enter"] },
    { action: "wait", duration: 2 },
  ];
}

function toSteelActions(call: FunctionCall): Steel.SessionComputerParams[] {
  const args: any = call.args ?? {};
  const coordinates = [toX(args.x ?? 500), toY(args.y ?? 500)];
  switch (call.name) {
    case "click":
      return [{ action: "click_mouse", button: "left", coordinates }];
    case "double_click":
      return [{ action: "click_mouse", button: "left", coordinates, num_clicks: 2 }];
    case "right_click":
      return [{ action: "click_mouse", button: "right", coordinates }];
    case "middle_click":
      return [{ action: "click_mouse", button: "middle", coordinates }];
    case "move":
      return [{ action: "move_mouse", coordinates }];
    case "drag_and_drop":
      return [
        {
          action: "drag_mouse",
          path: [
            [toX(args.start_x), toY(args.start_y)],
            [toX(args.end_x), toY(args.end_y)],
          ],
        },
      ];
    case "type": {
      const steps: Steel.SessionComputerParams[] = [
        { action: "type_text", text: args.text },
      ];
      if (args.press_enter) steps.push({ action: "press_key", keys: ["Enter"] });
      steps.push({ action: "wait", duration: 1 });
      return steps;
    }
    case "scroll": {
      const size = args.magnitude_in_pixels ?? 300;
      const deltas: Record<string, [number, number]> = {
        down: [0, size],
        up: [0, -size],
        right: [size, 0],
        left: [-size, 0],
      };
      const [delta_x, delta_y] = deltas[args.direction] ?? [0, size];
      return [{ action: "scroll", coordinates, delta_x, delta_y }];
    }
    case "press_key":
      return [{ action: "press_key", keys: toKeys(args.key) }];
    case "hotkey":
      return [{ action: "press_key", keys: toKeys(args.keys) }];
    case "wait":
      return [{ action: "wait", duration: args.seconds ?? 1 }];
    case "navigate":
      return openUrl(args.url);
    case "go_back":
      return [{ action: "press_key", keys: ["Alt", "ArrowLeft"] }];
    case "go_forward":
      return [{ action: "press_key", keys: ["Alt", "ArrowRight"] }];
    case "take_screenshot":
      return [];
    default:
      throw new Error(`The program has no Steel actions for ${call.name}.`);
  }
}

async function confirm(call: FunctionCall): Promise<Record<string, string>> {
  const decision: any = call.args?.safety_decision;
  if (decision?.decision !== "require_confirmation") return {};
  const terminal = createInterface({ input: process.stdin, output: process.stdout });
  const answer = await terminal.question(`${decision.explanation}\nRun this action? (y/n) `);
  terminal.close();
  if (answer.trim().toLowerCase() !== "y") {
    throw new Error("You did not approve the action.");
  }
  return { safety_acknowledgement: "true" };
}

const session = await steel.sessions.create({
  dimensions: { width: WIDTH, height: HEIGHT },
  timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);

const contents: Content[] = [{ role: "user", parts: [{ text: TASK }] }];

try {
  for (let turn = 0; turn < 50; turn++) {
    const response = await gemini.models.generateContent({
      model: "gemini-4-hydrogen",
      contents,
      config: {
        tools: [
          {
            computerUse: {
              environment: Environment.ENVIRONMENT_BROWSER,
              excludedPredefinedFunctions: [
                "triple_click", "mouse_down", "mouse_up", "key_down", "key_up",
              ],
            },
          },
        ],
        thinkingConfig: { thinkingLevel: ThinkingLevel.LOW },
      },
    });
    const content = response.candidates?.[0]?.content;
    if (!content) break;
    contents.push(content);

    const calls = (content.parts ?? []).flatMap((part) =>
      part.functionCall ? [part.functionCall] : [],
    );
    if (calls.length === 0) {
      console.log((content.parts ?? []).map((part) => part.text ?? "").join(" "));
      break;
    }

    const parts: Part[] = [];
    let failed = false;
    for (const call of calls) {
      console.log(call.name, JSON.stringify(call.args));
      const result: Record<string, string> = {};
      if (failed) {
        result.error = HALTED;
      } else {
        Object.assign(result, await confirm(call));
        for (const action of toSteelActions(call)) {
          const { error } = await steel.sessions.computer(session.id, action);
          if (error) {
            result.error = error;
            failed = true;
            break;
          }
        }
      }
      const screenshot = await steel.sessions.computer(session.id, {
        action: "take_screenshot",
      });
      if (screenshot.error) throw new Error(screenshot.error);
      parts.push(
        { functionResponse: { name: call.name, response: { url: currentUrl, ...result } } },
        { inlineData: { mimeType: "image/png", data: screenshot.base64_image } },
      );
    }
    contents.push({ role: "user", parts });
  }
} finally {
  await steel.sessions.release(session.id);
}

Run the program

npx tsx gemini-computer-use.mts

The program prints the session viewer URL, then each function that Gemini calls with its arguments, then Gemini's answer. Open the session viewer to watch Gemini work.

How the program maps the functions

toSteelActions (to_steel_actions in Python) turns each function into one or more Steel actions, then the program takes one screenshot:

Gemini functionSteel actions
click, double_click, right_click, middle_clickclick_mouse with that button. double_click sets num_clicks to 2.
movemove_mouse.
drag_and_dropdrag_mouse from the start point to the end point.
typetype_text where the focus is, then Enter if Gemini asks for it, and a wait of 1 second.
scrollscroll at the point, by magnitude_in_pixels, or 300 when Gemini leaves it out. Steel turns the wheel one step, whatever the size.
press_key, hotkeypress_key. ctrl+a becomes ["Control", "a"].
waitwait for the number of seconds.
navigateControl+L, type_text with the URL, Enter, and a wait of 2 seconds.
go_back, go_forwardpress_key with Alt+ArrowLeft or Alt+ArrowRight.
take_screenshotNone. The program only takes a screenshot.
Any other functionNone. The program stops with an error.

Notes on the program

  • Scale every coordinate from the 1000 by 1000 grid to the session's dimensions. The program does this with toX and toY. When a call has no point, the program uses the center of the screen.
  • Each function response carries the current URL, and the program sends the screenshot after it as an image. The program tracks the URL from navigate.
  • When Steel returns an error for an action, the program skips the call's other actions and puts the error in the call's response, so Gemini can try another way. The calls after it in the same turn do not run, and their responses say so. If the screenshot itself fails, the program stops.
  • Each function call carries an intent, Gemini's reason for the call. The program prints it with the other arguments.
  • The program sets the thinking level to LOW, so a turn takes about 10 seconds. At the model's default, HIGH, a turn takes 40 seconds to 2 minutes.
  • excludedPredefinedFunctions (excluded_predefined_functions) turns off five functions that the program does not map: triple_click, mouse_down, mouse_up, key_down and key_up.
  • Gemini keeps no conversation on its side. Each request sends the full contents, with every earlier screenshot, so long tasks send large requests.
  • A function call can carry a safety_decision whose decision is require_confirmation. Google's terms do not let you skip this confirmation. The program prints Gemini's explanation and asks you before it runs the call. If you approve, it adds safety_acknowledgement: "true" to that call's response. If you do not, it stops.
  • An agent loop can run longer than the default timeout, so the session has a timeout of 15 minutes. Limits lists the longest timeout that each plan allows.

Next steps