---
site: steel.dev
collection: docs
canonical_url: https://steel.dev/docs/integrations/gemini-computer-use
markdown_url: https://steel.dev/docs/integrations/gemini-computer-use.md
content_signal: ai-input=yes, ai-train=yes, search=yes
title: Run Gemini computer use on Steel
description: Connect Gemini's computer-use tool to a Steel session, so the model sees screenshots and controls the browser.
summary: Connect Gemini's computer-use tool to a Steel session, so the model sees screenshots and controls the browser.
type: integration-guide
reviewed: 2026-11-12
prerequisites: []
related:
  - https://steel.dev/docs/integrations/claude-computer-use.md
  - https://steel.dev/docs/integrations/openai-computer-use.md
agent_setup: https://setup.steel.dev/install.md
docs_index: https://steel.dev/docs/sitemap.md
---

# Run Gemini computer use on Steel

Gemini's computer-use tool calls browser functions, such as `click`, `type` or `navigate`. It places every point on a 1000 by 1000 grid, whatever the screen size. Your program scales each point to the session's screen, runs each function as one or more [Computer actions](https://steel.dev/docs/control/computer-actions) in a Steel session, and sends a screenshot back with the function's response.

The program uses `gemini-4-hydrogen` and the names of its functions. Earlier models name some functions differently: `gemini-3-flash-preview` calls `click_at` and `type_text_at`, for example. The program stops with an error at the first name that it does not know. [Computer use in the Gemini API](https://aistudio.google.com/docs/computer-use) lists the functions of the current models.

To route the session through a [Steel proxy](https://steel.dev/docs/stealth/proxies) or to [solve CAPTCHAs](https://steel.dev/docs/stealth/captcha-solving), set those options when you create the session.

## Before you begin

- A Steel API key in the `STEEL_API_KEY` environment variable. See [API keys](https://steel.dev/docs/account/api-keys).
- A Gemini API key in the `GEMINI_API_KEY` environment variable, with access to `gemini-4-hydrogen`.
- Node.js 22 or later, or Python 3.11 or later.

Install the Steel and Google Gen AI SDKs:

**TypeScript**

```sh
npm install steel-sdk @google/genai
```

**Python**

```sh
python3 -m venv .venv
source .venv/bin/activate
pip install steel-sdk google-genai
```

## Set up the integration

### Connect Gemini to a session

This program creates a session, gives Gemini a task, and runs each function that Gemini calls in the session. Before the program runs a function that Gemini marks for confirmation, it asks you in the terminal. It stops when Gemini returns text and no function call, and it releases the session in `finally`.

**TypeScript**

```ts file=gemini-computer-use.mts
import { createInterface } from "node:readline/promises";
import {
  Environment,
  GoogleGenAI,
  ThinkingLevel,
  type Content,
  type FunctionCall,
  type Part,
} from "@google/genai";
import Steel from "steel-sdk";

const WIDTH = 1440;
const HEIGHT = 900;
const TASK = "Open https://example.com and tell me the title of the page.";

const KEY_NAMES: Record<string, string> = {
  RETURN: "Enter", ENTER: "Enter", ESC: "Escape", TAB: "Tab",
  BACKSPACE: "Backspace", DELETE: "Delete", SPACE: "Space",
  CTRL: "Control", ALT: "Alt", SHIFT: "Shift", CMD: "Meta", SUPER: "Meta",
  UP: "ArrowUp", DOWN: "ArrowDown", LEFT: "ArrowLeft", RIGHT: "ArrowRight",
  PAGEUP: "PageUp", PAGEDOWN: "PageDown", HOME: "Home", END: "End",
};

const HALTED = "Not executed: an earlier action in this turn failed.";

const steel = new Steel();
const gemini = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
let currentUrl = "about:blank";

const toX = (x: number) => Math.round((x / 1000) * WIDTH);
const toY = (y: number) => Math.round((y / 1000) * HEIGHT);

function toKeys(keys: string | string[]): string[] {
  const list = Array.isArray(keys) ? keys : keys.split("+");
  return list.map((key) => KEY_NAMES[key.trim().toUpperCase()] ?? key.trim());
}

function openUrl(url: string): Steel.SessionComputerParams[] {
  currentUrl = /^https?:\/\//.test(url) ? url : `https://${url}`;
  return [
    { action: "press_key", keys: ["Control", "l"] },
    { action: "type_text", text: currentUrl },
    { action: "press_key", keys: ["Enter"] },
    { action: "wait", duration: 2 },
  ];
}

function toSteelActions(call: FunctionCall): Steel.SessionComputerParams[] {
  const args: any = call.args ?? {};
  const coordinates = [toX(args.x ?? 500), toY(args.y ?? 500)];
  switch (call.name) {
    case "click":
      return [{ action: "click_mouse", button: "left", coordinates }];
    case "double_click":
      return [{ action: "click_mouse", button: "left", coordinates, num_clicks: 2 }];
    case "right_click":
      return [{ action: "click_mouse", button: "right", coordinates }];
    case "middle_click":
      return [{ action: "click_mouse", button: "middle", coordinates }];
    case "move":
      return [{ action: "move_mouse", coordinates }];
    case "drag_and_drop":
      return [
        {
          action: "drag_mouse",
          path: [
            [toX(args.start_x), toY(args.start_y)],
            [toX(args.end_x), toY(args.end_y)],
          ],
        },
      ];
    case "type": {
      const steps: Steel.SessionComputerParams[] = [
        { action: "type_text", text: args.text },
      ];
      if (args.press_enter) steps.push({ action: "press_key", keys: ["Enter"] });
      steps.push({ action: "wait", duration: 1 });
      return steps;
    }
    case "scroll": {
      const size = args.magnitude_in_pixels ?? 300;
      const deltas: Record<string, [number, number]> = {
        down: [0, size],
        up: [0, -size],
        right: [size, 0],
        left: [-size, 0],
      };
      const [delta_x, delta_y] = deltas[args.direction] ?? [0, size];
      return [{ action: "scroll", coordinates, delta_x, delta_y }];
    }
    case "press_key":
      return [{ action: "press_key", keys: toKeys(args.key) }];
    case "hotkey":
      return [{ action: "press_key", keys: toKeys(args.keys) }];
    case "wait":
      return [{ action: "wait", duration: args.seconds ?? 1 }];
    case "navigate":
      return openUrl(args.url);
    case "go_back":
      return [{ action: "press_key", keys: ["Alt", "ArrowLeft"] }];
    case "go_forward":
      return [{ action: "press_key", keys: ["Alt", "ArrowRight"] }];
    case "take_screenshot":
      return [];
    default:
      throw new Error(`The program has no Steel actions for ${call.name}.`);
  }
}

async function confirm(call: FunctionCall): Promise<Record<string, string>> {
  const decision: any = call.args?.safety_decision;
  if (decision?.decision !== "require_confirmation") return {};
  const terminal = createInterface({ input: process.stdin, output: process.stdout });
  const answer = await terminal.question(`${decision.explanation}\nRun this action? (y/n) `);
  terminal.close();
  if (answer.trim().toLowerCase() !== "y") {
    throw new Error("You did not approve the action.");
  }
  return { safety_acknowledgement: "true" };
}

const session = await steel.sessions.create({
  dimensions: { width: WIDTH, height: HEIGHT },
  timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);

const contents: Content[] = [{ role: "user", parts: [{ text: TASK }] }];

try {
  for (let turn = 0; turn < 50; turn++) {
    const response = await gemini.models.generateContent({
      model: "gemini-4-hydrogen",
      contents,
      config: {
        tools: [
          {
            computerUse: {
              environment: Environment.ENVIRONMENT_BROWSER,
              excludedPredefinedFunctions: [
                "triple_click", "mouse_down", "mouse_up", "key_down", "key_up",
              ],
            },
          },
        ],
        thinkingConfig: { thinkingLevel: ThinkingLevel.LOW },
      },
    });
    const content = response.candidates?.[0]?.content;
    if (!content) break;
    contents.push(content);

    const calls = (content.parts ?? []).flatMap((part) =>
      part.functionCall ? [part.functionCall] : [],
    );
    if (calls.length === 0) {
      console.log((content.parts ?? []).map((part) => part.text ?? "").join(" "));
      break;
    }

    const parts: Part[] = [];
    let failed = false;
    for (const call of calls) {
      console.log(call.name, JSON.stringify(call.args));
      const result: Record<string, string> = {};
      if (failed) {
        result.error = HALTED;
      } else {
        Object.assign(result, await confirm(call));
        for (const action of toSteelActions(call)) {
          const { error } = await steel.sessions.computer(session.id, action);
          if (error) {
            result.error = error;
            failed = true;
            break;
          }
        }
      }
      const screenshot = await steel.sessions.computer(session.id, {
        action: "take_screenshot",
      });
      if (screenshot.error) throw new Error(screenshot.error);
      parts.push(
        { functionResponse: { name: call.name, response: { url: currentUrl, ...result } } },
        { inlineData: { mimeType: "image/png", data: screenshot.base64_image } },
      );
    }
    contents.push({ role: "user", parts });
  }
} finally {
  await steel.sessions.release(session.id);
}
```

**Python**

```python file=gemini_computer_use.py
import json
import os
import re

from google import genai
from google.genai import types
from steel import Steel

WIDTH = 1440
HEIGHT = 900
TASK = "Open https://example.com and tell me the title of the page."

KEY_NAMES = {
    "RETURN": "Enter", "ENTER": "Enter", "ESC": "Escape", "TAB": "Tab",
    "BACKSPACE": "Backspace", "DELETE": "Delete", "SPACE": "Space",
    "CTRL": "Control", "ALT": "Alt", "SHIFT": "Shift", "CMD": "Meta", "SUPER": "Meta",
    "UP": "ArrowUp", "DOWN": "ArrowDown", "LEFT": "ArrowLeft", "RIGHT": "ArrowRight",
    "PAGEUP": "PageUp", "PAGEDOWN": "PageDown", "HOME": "Home", "END": "End",
}

HALTED = "Not executed: an earlier action in this turn failed."

steel = Steel()
gemini = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
state = {"url": "about:blank"}


def to_x(x):
    return int(x / 1000 * WIDTH)


def to_y(y):
    return int(y / 1000 * HEIGHT)


def to_keys(keys):
    keys = keys if isinstance(keys, list) else keys.split("+")
    return [KEY_NAMES.get(key.strip().upper(), key.strip()) for key in keys]


def open_url(url):
    state["url"] = url if re.match(r"https?://", url) else "https://" + url
    return [
        {"action": "press_key", "keys": ["Control", "l"]},
        {"action": "type_text", "text": state["url"]},
        {"action": "press_key", "keys": ["Enter"]},
        {"action": "wait", "duration": 2},
    ]


def to_steel_actions(name, args):
    coordinates = [to_x(args.get("x", 500)), to_y(args.get("y", 500))]
    if name in ("click", "double_click", "right_click", "middle_click"):
        button = {"right_click": "right", "middle_click": "middle"}.get(name, "left")
        action = {"action": "click_mouse", "button": button, "coordinates": coordinates}
        if name == "double_click":
            action["num_clicks"] = 2
        return [action]
    if name == "move":
        return [{"action": "move_mouse", "coordinates": coordinates}]
    if name == "drag_and_drop":
        start = [to_x(args["start_x"]), to_y(args["start_y"])]
        end = [to_x(args["end_x"]), to_y(args["end_y"])]
        return [{"action": "drag_mouse", "path": [start, end]}]
    if name == "type":
        steps = [{"action": "type_text", "text": args["text"]}]
        if args.get("press_enter"):
            steps.append({"action": "press_key", "keys": ["Enter"]})
        steps.append({"action": "wait", "duration": 1})
        return steps
    if name == "scroll":
        size = args.get("magnitude_in_pixels", 300)
        delta_x, delta_y = {
            "down": (0, size),
            "up": (0, -size),
            "right": (size, 0),
            "left": (-size, 0),
        }.get(args.get("direction"), (0, size))
        return [{"action": "scroll", "coordinates": coordinates, "delta_x": delta_x, "delta_y": delta_y}]
    if name == "press_key":
        return [{"action": "press_key", "keys": to_keys(args["key"])}]
    if name == "hotkey":
        return [{"action": "press_key", "keys": to_keys(args["keys"])}]
    if name == "wait":
        return [{"action": "wait", "duration": args.get("seconds", 1)}]
    if name == "navigate":
        return open_url(args["url"])
    if name == "go_back":
        return [{"action": "press_key", "keys": ["Alt", "ArrowLeft"]}]
    if name == "go_forward":
        return [{"action": "press_key", "keys": ["Alt", "ArrowRight"]}]
    if name == "take_screenshot":
        return []
    raise ValueError(f"The program has no Steel actions for {name}.")


def confirm(args):
    decision = args.get("safety_decision") or {}
    if decision.get("decision") != "require_confirmation":
        return {}
    print(decision.get("explanation"))
    if input("Run this action? (y/n) ").strip().lower() != "y":
        raise RuntimeError("You did not approve the action.")
    return {"safety_acknowledgement": "true"}


session = steel.sessions.create(
    dimensions={"width": WIDTH, "height": HEIGHT},
    api_timeout=900000,
)
print("Session viewer:", session.session_viewer_url)

config = types.GenerateContentConfig(
    tools=[
        types.Tool(
            computer_use=types.ComputerUse(
                environment=types.Environment.ENVIRONMENT_BROWSER,
                excluded_predefined_functions=[
                    "triple_click", "mouse_down", "mouse_up", "key_down", "key_up",
                ],
            )
        )
    ],
    thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW),
)
contents = [types.Content(role="user", parts=[types.Part(text=TASK)])]

try:
    for turn in range(50):
        response = gemini.models.generate_content(
            model="gemini-4-hydrogen",
            contents=contents,
            config=config,
        )
        content = response.candidates[0].content if response.candidates else None
        if not content:
            break
        contents.append(content)

        calls = [part.function_call for part in content.parts or [] if part.function_call]
        if not calls:
            print(" ".join(part.text for part in content.parts or [] if part.text))
            break

        parts = []
        failed = False
        for call in calls:
            args = call.args or {}
            print(call.name, json.dumps(args))
            result = {}
            if failed:
                result["error"] = HALTED
            else:
                result.update(confirm(args))
                for action in to_steel_actions(call.name, args):
                    error = steel.sessions.computer(session.id, **action).error
                    if error:
                        result["error"] = error
                        failed = True
                        break
            screenshot = steel.sessions.computer(session.id, action="take_screenshot")
            if screenshot.error:
                raise RuntimeError(screenshot.error)
            parts.append(
                types.Part(
                    function_response=types.FunctionResponse(
                        name=call.name,
                        response={"url": state["url"], **result},
                    )
                )
            )
            parts.append(
                types.Part(
                    inline_data=types.Blob(mime_type="image/png", data=screenshot.base64_image)
                )
            )
        contents.append(types.Content(role="user", parts=parts))
finally:
    steel.sessions.release(session.id)
```

### Run the program

**TypeScript**

```sh
npx tsx gemini-computer-use.mts
```

**Python**

```sh
python gemini_computer_use.py
```

The program prints the [session viewer](https://steel.dev/docs/observability/live-view#open-the-live-view) URL, then each function that Gemini calls with its arguments, then Gemini's answer. Open the session viewer to watch Gemini work.

### How the program maps the functions

`toSteelActions` (`to_steel_actions` in Python) turns each function into one or more Steel actions, then the program takes one screenshot:

| Gemini function | Steel actions |
| - | - |
| `click`, `double_click`, `right_click`, `middle_click` | `click_mouse` with that button. `double_click` sets `num_clicks` to 2. |
| `move` | `move_mouse`. |
| `drag_and_drop` | `drag_mouse` from the start point to the end point. |
| `type` | `type_text` where the focus is, then `Enter` if Gemini asks for it, and a `wait` of 1 second. |
| `scroll` | `scroll` at the point, by `magnitude_in_pixels`, or 300 when Gemini leaves it out. Steel turns the wheel one step, whatever the size. |
| `press_key`, `hotkey` | `press_key`. `ctrl+a` becomes `["Control", "a"]`. |
| `wait` | `wait` for the number of `seconds`. |
| `navigate` | `Control+L`, `type_text` with the URL, `Enter`, and a `wait` of 2 seconds. |
| `go_back`, `go_forward` | `press_key` with `Alt+ArrowLeft` or `Alt+ArrowRight`. |
| `take_screenshot` | None. The program only takes a screenshot. |
| Any other function | None. The program stops with an error. |

### Notes on the program

- Scale every coordinate from the 1000 by 1000 grid to the session's [`dimensions`](https://steel.dev/docs/sessions/devices). The program does this with `toX` and `toY`. When a call has no point, the program uses the center of the screen.
- Each function response carries the current URL, and the program sends the screenshot after it as an image. The program tracks the URL from `navigate`.
- When Steel returns an `error` for an action, the program skips the call's other actions and puts the error in the call's response, so Gemini can try another way. The calls after it in the same turn do not run, and their responses say so. If the screenshot itself fails, the program stops.
- Each function call carries an `intent`, Gemini's reason for the call. The program prints it with the other arguments.
- The program sets the thinking level to `LOW`, so a turn takes about 10 seconds. At the model's default, `HIGH`, a turn takes 40 seconds to 2 minutes.
- `excludedPredefinedFunctions` (`excluded_predefined_functions`) turns off five functions that the program does not map: `triple_click`, `mouse_down`, `mouse_up`, `key_down` and `key_up`.
- Gemini keeps no conversation on its side. Each request sends the full `contents`, with every earlier screenshot, so long tasks send large requests.
- A function call can carry a `safety_decision` whose `decision` is `require_confirmation`. Google's terms do not let you skip this confirmation. The program prints Gemini's explanation and asks you before it runs the call. If you approve, it adds `safety_acknowledgement: "true"` to that call's response. If you do not, it stops.
- An agent loop can run longer than the default `timeout`, so the session has a `timeout` of 15 minutes. [Limits](https://steel.dev/docs/platform/limits#session-length) lists the longest `timeout` that each plan allows.

## Next steps

- [Computer use on Gemini Enterprise Agent Platform](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/computer-use): The functions of each model, including Gemini 3 Flash, and the safety decisions.
- [Computer use in the Gemini API](https://aistudio.google.com/docs/computer-use): The current models and their actions, through the Interactions API.
- Cookbook starters: [TypeScript](https://github.com/steel-dev/steel-cookbook/tree/main/examples/gemini-computer-use-ts), [Python](https://github.com/steel-dev/steel-cookbook/tree/main/examples/gemini-computer-use-py), [Go](https://github.com/steel-dev/steel-cookbook/tree/main/examples/gemini-computer-use-go) and [Rust](https://github.com/steel-dev/steel-cookbook/tree/main/examples/gemini-computer-use-rs).
