Run Gemini computer use on Steel
Gemini's computer-use tool calls browser functions, such as click, type or navigate. It places every point on a 1000 by 1000 grid, whatever the screen size. Your program scales each point to the session's screen, runs each function as one or more Computer actions in a Steel session, and sends a screenshot back with the function's response.
The program uses gemini-4-hydrogen and the names of its functions. Earlier models name some functions differently: gemini-3-flash-preview calls click_at and type_text_at, for example. The program stops with an error at the first name that it does not know. Computer use in the Gemini API lists the functions of the current models.
To route the session through a Steel proxy or to solve CAPTCHAs, set those options when you create the session.
Before you begin
- A Steel API key in the
STEEL_API_KEYenvironment variable. See API keys. - A Gemini API key in the
GEMINI_API_KEYenvironment variable, with access togemini-4-hydrogen. - Node.js 22 or later, or Python 3.11 or later.
Install the Steel and Google Gen AI SDKs:
npm install steel-sdk @google/genaipython3 -m venv .venv
source .venv/bin/activate
pip install steel-sdk google-genaiSet up the integration
Connect Gemini to a session
This program creates a session, gives Gemini a task, and runs each function that Gemini calls in the session. Before the program runs a function that Gemini marks for confirmation, it asks you in the terminal. It stops when Gemini returns text and no function call, and it releases the session in finally.
import { createInterface } from "node:readline/promises";
import {
Environment,
GoogleGenAI,
ThinkingLevel,
type Content,
type FunctionCall,
type Part,
} from "@google/genai";
import Steel from "steel-sdk";
const WIDTH = 1440;
const HEIGHT = 900;
const TASK = "Open https://example.com and tell me the title of the page.";
const KEY_NAMES: Record<string, string> = {
RETURN: "Enter", ENTER: "Enter", ESC: "Escape", TAB: "Tab",
BACKSPACE: "Backspace", DELETE: "Delete", SPACE: "Space",
CTRL: "Control", ALT: "Alt", SHIFT: "Shift", CMD: "Meta", SUPER: "Meta",
UP: "ArrowUp", DOWN: "ArrowDown", LEFT: "ArrowLeft", RIGHT: "ArrowRight",
PAGEUP: "PageUp", PAGEDOWN: "PageDown", HOME: "Home", END: "End",
};
const HALTED = "Not executed: an earlier action in this turn failed.";
const steel = new Steel();
const gemini = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
let currentUrl = "about:blank";
const toX = (x: number) => Math.round((x / 1000) * WIDTH);
const toY = (y: number) => Math.round((y / 1000) * HEIGHT);
function toKeys(keys: string | string[]): string[] {
const list = Array.isArray(keys) ? keys : keys.split("+");
return list.map((key) => KEY_NAMES[key.trim().toUpperCase()] ?? key.trim());
}
function openUrl(url: string): Steel.SessionComputerParams[] {
currentUrl = /^https?:\/\//.test(url) ? url : `https://${url}`;
return [
{ action: "press_key", keys: ["Control", "l"] },
{ action: "type_text", text: currentUrl },
{ action: "press_key", keys: ["Enter"] },
{ action: "wait", duration: 2 },
];
}
function toSteelActions(call: FunctionCall): Steel.SessionComputerParams[] {
const args: any = call.args ?? {};
const coordinates = [toX(args.x ?? 500), toY(args.y ?? 500)];
switch (call.name) {
case "click":
return [{ action: "click_mouse", button: "left", coordinates }];
case "double_click":
return [{ action: "click_mouse", button: "left", coordinates, num_clicks: 2 }];
case "right_click":
return [{ action: "click_mouse", button: "right", coordinates }];
case "middle_click":
return [{ action: "click_mouse", button: "middle", coordinates }];
case "move":
return [{ action: "move_mouse", coordinates }];
case "drag_and_drop":
return [
{
action: "drag_mouse",
path: [
[toX(args.start_x), toY(args.start_y)],
[toX(args.end_x), toY(args.end_y)],
],
},
];
case "type": {
const steps: Steel.SessionComputerParams[] = [
{ action: "type_text", text: args.text },
];
if (args.press_enter) steps.push({ action: "press_key", keys: ["Enter"] });
steps.push({ action: "wait", duration: 1 });
return steps;
}
case "scroll": {
const size = args.magnitude_in_pixels ?? 300;
const deltas: Record<string, [number, number]> = {
down: [0, size],
up: [0, -size],
right: [size, 0],
left: [-size, 0],
};
const [delta_x, delta_y] = deltas[args.direction] ?? [0, size];
return [{ action: "scroll", coordinates, delta_x, delta_y }];
}
case "press_key":
return [{ action: "press_key", keys: toKeys(args.key) }];
case "hotkey":
return [{ action: "press_key", keys: toKeys(args.keys) }];
case "wait":
return [{ action: "wait", duration: args.seconds ?? 1 }];
case "navigate":
return openUrl(args.url);
case "go_back":
return [{ action: "press_key", keys: ["Alt", "ArrowLeft"] }];
case "go_forward":
return [{ action: "press_key", keys: ["Alt", "ArrowRight"] }];
case "take_screenshot":
return [];
default:
throw new Error(`The program has no Steel actions for ${call.name}.`);
}
}
async function confirm(call: FunctionCall): Promise<Record<string, string>> {
const decision: any = call.args?.safety_decision;
if (decision?.decision !== "require_confirmation") return {};
const terminal = createInterface({ input: process.stdin, output: process.stdout });
const answer = await terminal.question(`${decision.explanation}\nRun this action? (y/n) `);
terminal.close();
if (answer.trim().toLowerCase() !== "y") {
throw new Error("You did not approve the action.");
}
return { safety_acknowledgement: "true" };
}
const session = await steel.sessions.create({
dimensions: { width: WIDTH, height: HEIGHT },
timeout: 900000,
});
console.log("Session viewer:", session.sessionViewerUrl);
const contents: Content[] = [{ role: "user", parts: [{ text: TASK }] }];
try {
for (let turn = 0; turn < 50; turn++) {
const response = await gemini.models.generateContent({
model: "gemini-4-hydrogen",
contents,
config: {
tools: [
{
computerUse: {
environment: Environment.ENVIRONMENT_BROWSER,
excludedPredefinedFunctions: [
"triple_click", "mouse_down", "mouse_up", "key_down", "key_up",
],
},
},
],
thinkingConfig: { thinkingLevel: ThinkingLevel.LOW },
},
});
const content = response.candidates?.[0]?.content;
if (!content) break;
contents.push(content);
const calls = (content.parts ?? []).flatMap((part) =>
part.functionCall ? [part.functionCall] : [],
);
if (calls.length === 0) {
console.log((content.parts ?? []).map((part) => part.text ?? "").join(" "));
break;
}
const parts: Part[] = [];
let failed = false;
for (const call of calls) {
console.log(call.name, JSON.stringify(call.args));
const result: Record<string, string> = {};
if (failed) {
result.error = HALTED;
} else {
Object.assign(result, await confirm(call));
for (const action of toSteelActions(call)) {
const { error } = await steel.sessions.computer(session.id, action);
if (error) {
result.error = error;
failed = true;
break;
}
}
}
const screenshot = await steel.sessions.computer(session.id, {
action: "take_screenshot",
});
if (screenshot.error) throw new Error(screenshot.error);
parts.push(
{ functionResponse: { name: call.name, response: { url: currentUrl, ...result } } },
{ inlineData: { mimeType: "image/png", data: screenshot.base64_image } },
);
}
contents.push({ role: "user", parts });
}
} finally {
await steel.sessions.release(session.id);
}import json
import os
import re
from google import genai
from google.genai import types
from steel import Steel
WIDTH = 1440
HEIGHT = 900
TASK = "Open https://example.com and tell me the title of the page."
KEY_NAMES = {
"RETURN": "Enter", "ENTER": "Enter", "ESC": "Escape", "TAB": "Tab",
"BACKSPACE": "Backspace", "DELETE": "Delete", "SPACE": "Space",
"CTRL": "Control", "ALT": "Alt", "SHIFT": "Shift", "CMD": "Meta", "SUPER": "Meta",
"UP": "ArrowUp", "DOWN": "ArrowDown", "LEFT": "ArrowLeft", "RIGHT": "ArrowRight",
"PAGEUP": "PageUp", "PAGEDOWN": "PageDown", "HOME": "Home", "END": "End",
}
HALTED = "Not executed: an earlier action in this turn failed."
steel = Steel()
gemini = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
state = {"url": "about:blank"}
def to_x(x):
return int(x / 1000 * WIDTH)
def to_y(y):
return int(y / 1000 * HEIGHT)
def to_keys(keys):
keys = keys if isinstance(keys, list) else keys.split("+")
return [KEY_NAMES.get(key.strip().upper(), key.strip()) for key in keys]
def open_url(url):
state["url"] = url if re.match(r"https?://", url) else "https://" + url
return [
{"action": "press_key", "keys": ["Control", "l"]},
{"action": "type_text", "text": state["url"]},
{"action": "press_key", "keys": ["Enter"]},
{"action": "wait", "duration": 2},
]
def to_steel_actions(name, args):
coordinates = [to_x(args.get("x", 500)), to_y(args.get("y", 500))]
if name in ("click", "double_click", "right_click", "middle_click"):
button = {"right_click": "right", "middle_click": "middle"}.get(name, "left")
action = {"action": "click_mouse", "button": button, "coordinates": coordinates}
if name == "double_click":
action["num_clicks"] = 2
return [action]
if name == "move":
return [{"action": "move_mouse", "coordinates": coordinates}]
if name == "drag_and_drop":
start = [to_x(args["start_x"]), to_y(args["start_y"])]
end = [to_x(args["end_x"]), to_y(args["end_y"])]
return [{"action": "drag_mouse", "path": [start, end]}]
if name == "type":
steps = [{"action": "type_text", "text": args["text"]}]
if args.get("press_enter"):
steps.append({"action": "press_key", "keys": ["Enter"]})
steps.append({"action": "wait", "duration": 1})
return steps
if name == "scroll":
size = args.get("magnitude_in_pixels", 300)
delta_x, delta_y = {
"down": (0, size),
"up": (0, -size),
"right": (size, 0),
"left": (-size, 0),
}.get(args.get("direction"), (0, size))
return [{"action": "scroll", "coordinates": coordinates, "delta_x": delta_x, "delta_y": delta_y}]
if name == "press_key":
return [{"action": "press_key", "keys": to_keys(args["key"])}]
if name == "hotkey":
return [{"action": "press_key", "keys": to_keys(args["keys"])}]
if name == "wait":
return [{"action": "wait", "duration": args.get("seconds", 1)}]
if name == "navigate":
return open_url(args["url"])
if name == "go_back":
return [{"action": "press_key", "keys": ["Alt", "ArrowLeft"]}]
if name == "go_forward":
return [{"action": "press_key", "keys": ["Alt", "ArrowRight"]}]
if name == "take_screenshot":
return []
raise ValueError(f"The program has no Steel actions for {name}.")
def confirm(args):
decision = args.get("safety_decision") or {}
if decision.get("decision") != "require_confirmation":
return {}
print(decision.get("explanation"))
if input("Run this action? (y/n) ").strip().lower() != "y":
raise RuntimeError("You did not approve the action.")
return {"safety_acknowledgement": "true"}
session = steel.sessions.create(
dimensions={"width": WIDTH, "height": HEIGHT},
api_timeout=900000,
)
print("Session viewer:", session.session_viewer_url)
config = types.GenerateContentConfig(
tools=[
types.Tool(
computer_use=types.ComputerUse(
environment=types.Environment.ENVIRONMENT_BROWSER,
excluded_predefined_functions=[
"triple_click", "mouse_down", "mouse_up", "key_down", "key_up",
],
)
)
],
thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW),
)
contents = [types.Content(role="user", parts=[types.Part(text=TASK)])]
try:
for turn in range(50):
response = gemini.models.generate_content(
model="gemini-4-hydrogen",
contents=contents,
config=config,
)
content = response.candidates[0].content if response.candidates else None
if not content:
break
contents.append(content)
calls = [part.function_call for part in content.parts or [] if part.function_call]
if not calls:
print(" ".join(part.text for part in content.parts or [] if part.text))
break
parts = []
failed = False
for call in calls:
args = call.args or {}
print(call.name, json.dumps(args))
result = {}
if failed:
result["error"] = HALTED
else:
result.update(confirm(args))
for action in to_steel_actions(call.name, args):
error = steel.sessions.computer(session.id, **action).error
if error:
result["error"] = error
failed = True
break
screenshot = steel.sessions.computer(session.id, action="take_screenshot")
if screenshot.error:
raise RuntimeError(screenshot.error)
parts.append(
types.Part(
function_response=types.FunctionResponse(
name=call.name,
response={"url": state["url"], **result},
)
)
)
parts.append(
types.Part(
inline_data=types.Blob(mime_type="image/png", data=screenshot.base64_image)
)
)
contents.append(types.Content(role="user", parts=parts))
finally:
steel.sessions.release(session.id)Run the program
npx tsx gemini-computer-use.mtspython gemini_computer_use.pyThe program prints the session viewer URL, then each function that Gemini calls with its arguments, then Gemini's answer. Open the session viewer to watch Gemini work.
How the program maps the functions
toSteelActions (to_steel_actions in Python) turns each function into one or more Steel actions, then the program takes one screenshot:
| Gemini function | Steel actions |
|---|---|
click, double_click, right_click, middle_click | click_mouse with that button. double_click sets num_clicks to 2. |
move | move_mouse. |
drag_and_drop | drag_mouse from the start point to the end point. |
type | type_text where the focus is, then Enter if Gemini asks for it, and a wait of 1 second. |
scroll | scroll at the point, by magnitude_in_pixels, or 300 when Gemini leaves it out. Steel turns the wheel one step, whatever the size. |
press_key, hotkey | press_key. ctrl+a becomes ["Control", "a"]. |
wait | wait for the number of seconds. |
navigate | Control+L, type_text with the URL, Enter, and a wait of 2 seconds. |
go_back, go_forward | press_key with Alt+ArrowLeft or Alt+ArrowRight. |
take_screenshot | None. The program only takes a screenshot. |
| Any other function | None. The program stops with an error. |
Notes on the program
- Scale every coordinate from the 1000 by 1000 grid to the session's
dimensions. The program does this withtoXandtoY. When a call has no point, the program uses the center of the screen. - Each function response carries the current URL, and the program sends the screenshot after it as an image. The program tracks the URL from
navigate. - When Steel returns an
errorfor an action, the program skips the call's other actions and puts the error in the call's response, so Gemini can try another way. The calls after it in the same turn do not run, and their responses say so. If the screenshot itself fails, the program stops. - Each function call carries an
intent, Gemini's reason for the call. The program prints it with the other arguments. - The program sets the thinking level to
LOW, so a turn takes about 10 seconds. At the model's default,HIGH, a turn takes 40 seconds to 2 minutes. excludedPredefinedFunctions(excluded_predefined_functions) turns off five functions that the program does not map:triple_click,mouse_down,mouse_up,key_downandkey_up.- Gemini keeps no conversation on its side. Each request sends the full
contents, with every earlier screenshot, so long tasks send large requests. - A function call can carry a
safety_decisionwhosedecisionisrequire_confirmation. Google's terms do not let you skip this confirmation. The program prints Gemini's explanation and asks you before it runs the call. If you approve, it addssafety_acknowledgement: "true"to that call's response. If you do not, it stops. - An agent loop can run longer than the default
timeout, so the session has atimeoutof 15 minutes. Limits lists the longesttimeoutthat each plan allows.
Next steps
- Computer use on Gemini Enterprise Agent Platform: The functions of each model, including Gemini 3 Flash, and the safety decisions.
- Computer use in the Gemini API: The current models and their actions, through the Interactions API.
- Cookbook starters: TypeScript, Python, Go and Rust.