Run sessions at scale
A pool of workers runs many tasks at once. Each worker drives one session, and you keep the number of workers below your plan's limit of concurrent sessions. To keep the pool running, queue the work instead of probing the limit, retry only the errors that Steel marks as retryable, release every session, and check each run's signals.
Before you begin
Put your Steel API key in the STEEL_API_KEY environment variable, and install the Steel SDK and Playwright with npm install steel-sdk playwright or with pip install steel-sdk playwright in a virtual environment.
Know your limits
Your plan sets these limits for a pool of sessions:
- How many sessions can be live at the same time in your organization.
- How many sessions you can create in a minute. Other API requests do not count toward this rate.
- The longest
timeoutthat a session can have.
Limits has the values for each plan. Only live sessions count toward the concurrency limit. A session stops counting when it is released or ends on its timeout.
Each response of sessions.create carries three headers about the concurrency limit, and so does the 429 that you get at the limit:
| Header | What it holds |
|---|---|
x-steel-session-limit | How many sessions can be live at once. |
x-steel-session-active | How many sessions are live, this one included. |
x-steel-session-remaining | How many more sessions you can create now. |
This program creates a session, prints the three headers, and releases the session.
import Steel from "steel-sdk";
const client = new Steel();
const { data: session, response } = await client.sessions.create().withResponse();
try {
const limit = response.headers.get("x-steel-session-limit");
const active = response.headers.get("x-steel-session-active");
const remaining = response.headers.get("x-steel-session-remaining");
console.log(`limit ${limit}, active ${active}, remaining ${remaining}`);
} finally {
await client.sessions.release(session.id);
}from steel import Steel
client = Steel()
response = client.sessions.with_raw_response.create()
session = response.parse()
try:
limit = response.headers.get("x-steel-session-limit")
active = response.headers.get("x-steel-session-active")
remaining = response.headers.get("x-steel-session-remaining")
print(f"limit {limit}, active {active}, remaining {remaining}")
finally:
client.sessions.release(session.id)Run the program:
npx tsx limits.mtspython limits.pylimit 10, active 1, remaining 9The limit counts every live session in your organization, from all your projects, workers and team members. Size each pool for its share of that limit. In Go, pass the request option option.WithResponseInto to read the raw response. In Rust, only a failed call returns the headers, in steel_rs::Error::Api.
When you reach the limit
When your organization already has as many live sessions as the plan allows, sessions.create fails with status 429 and the code concurrency_limit_reached. No session is created.
{
"error": "Too Many Requests",
"code": "concurrency_limit_reached",
"message": "Concurrent session limit reached for your launch plan (10 of 10 sessions running). Release a running session (POST /v1/sessions/{id}/release) before creating another, or upgrade your plan for a higher limit.",
"retryable": true,
"linkToDocs": "https://docs.steel.dev/overview/pricinglimits"
}The SDKs retry a 429 twice by default, with a short wait between tries. If every try fails, the TypeScript and Python SDKs raise RateLimitError. In Go and Rust, the call returns an error with the status 429: a *steel.Error in Go and a steel_rs::Error::Api in Rust. Release a session that you no longer need, or wait for one to end, then try again.
A 429 can also mean that you created too many sessions in one minute. That response has the code rate_limited, and retryAfterSeconds says how long to wait. To raise either limit, change your plan in Billing and usage.
Queue work behind the limit
Put the tasks in a queue in your own code, and start a fixed number of workers that take tasks from it. Each worker creates a session, runs one task, and releases the session before it takes the next task. The pool never asks for more sessions than it has workers.
Do not probe the limit by creating sessions until Steel refuses one. When you reach the limit, creating a session fails with 429 and the code concurrency_limit_reached.
This program visits 20 pages with 5 workers. Each session has a job ID in its metadata, a timeout, and an inactivity timeout.
import Steel from "steel-sdk";
import { chromium } from "playwright";
const client = new Steel();
const CONCURRENCY = 5;
const JOB_ID = "job-1042";
const queue = Array.from({ length: 20 }, (_, i) => `https://example.com/?page=${i}`);
async function visit(url: string): Promise<string> {
const session = await client.sessions.create({
timeout: 120000,
inactivityTimeout: 30000,
metadata: { job_id: JOB_ID },
});
try {
const browser = await chromium.connectOverCDP(session.websocketUrl);
const context = browser.contexts()[0];
const page = context.pages()[0] ?? (await context.newPage());
await page.goto(url);
const title = await page.title();
await browser.close();
return title;
} finally {
await client.sessions.release(session.id);
}
}
async function worker() {
for (let url = queue.shift(); url; url = queue.shift()) {
try {
console.log(url, await visit(url));
} catch (error) {
console.error(url, String(error));
}
}
}
await Promise.all(Array.from({ length: CONCURRENCY }, worker));import asyncio
from playwright.async_api import async_playwright
from steel import AsyncSteel
client = AsyncSteel()
CONCURRENCY = 5
JOB_ID = "job-1042"
async def visit(playwright, url: str) -> str:
session = await client.sessions.create(
api_timeout=120000,
inactivity_timeout=30000,
metadata={"job_id": JOB_ID},
)
try:
browser = await playwright.chromium.connect_over_cdp(session.websocket_url)
context = browser.contexts[0]
page = context.pages[0] if context.pages else await context.new_page()
await page.goto(url)
title = await page.title()
await browser.close()
return title
finally:
await client.sessions.release(session.id)
async def worker(playwright, queue: asyncio.Queue) -> None:
while not queue.empty():
url = queue.get_nowait()
try:
print(url, await visit(playwright, url))
except Exception as error:
print(url, error)
async def main() -> None:
queue = asyncio.Queue()
for i in range(20):
queue.put_nowait(f"https://example.com/?page={i}")
async with async_playwright() as playwright:
await asyncio.gather(*(worker(playwright, queue) for _ in range(CONCURRENCY)))
asyncio.run(main())Run the program:
npx tsx pool.mtspython pool.pyThe program prints one line for each of the 20 pages, in the order that the workers finish. Each line shows a page and its title. The output starts like this:
https://example.com/?page=0 Example Domain
https://example.com/?page=3 Example Domain
https://example.com/?page=1 Example DomainSet CONCURRENCY below your limit, and leave room for other work in the organization. In production, use a job queue or a durable workflow engine for the queue. The examples include pools on Temporal, Restate and Trigger.dev.
Retry safely
Each error from the API says whether a retry can help. The body has retryable, and the x-should-retry header has the same value. When Steel knows how long to wait, the body has retryAfterSeconds, and the Retry-After header has the same value.
{
"error": "Conflict",
"code": "profile_not_ready",
"message": "Profile '91209599-cf67-4fbc-a393-0a92fa6e72b3' is not ready yet. It is saved after the session that created it is released. If that session is still running, release it, then retry in 5 seconds.",
"param": "profileId",
"retryable": true,
"retryAfterSeconds": 5,
"linkToDocs": "https://docs.steel.dev/overview/sessions-api/reusing-auth-context"
}Retry only when retryable is true. An error with retryable set to false fails again until you change the request or the account. Examples are an invalid option, a feature that your plan does not include, and the billing errors insufficient_funds and budget_exceeded, which stop new sessions until you add credits or the spending limit resets. Stop the pool when you get one of these. Errors lists each code, and Billing and usage explains the billing errors.
The Steel SDK already retries for you. It retries a request up to twice when the response has x-should-retry: true, waits for the time in Retry-After when the response has that header, and never retries a response with x-should-retry: false. When a response has no such header, the SDK retries 408, 409, 429 and 5xx responses. Set maxRetries (max_retries in Python and Rust, option.WithMaxRetries in Go) on the client to change the number.
From version 0.23.0, the SDK also sends an Idempotency-Key header on each request that is not a GET, and keeps the same key across its retries. Only Commands requests use the key, so a retried session create can still start a second session. The sweep in Always release sessions releases it. See Idempotency.
When the SDK gives up, the error reaches your worker. The worker can retry too, with a function that reads the error body:
async function withRetry<T>(run: () => Promise<T>, attempts = 3): Promise<T> {
for (let attempt = 1; ; attempt++) {
try {
return await run();
} catch (error) {
const body =
error instanceof Steel.APIError
? (error.error as { retryable?: boolean; retryAfterSeconds?: number } | undefined)
: undefined;
if (!body?.retryable || attempt >= attempts) throw error;
await new Promise((resolve) => setTimeout(resolve, (body.retryAfterSeconds ?? 5) * 1000));
}
}
}
const session = await withRetry(() =>
client.sessions.create({ metadata: { job_id: JOB_ID } }),
);import steel
async def with_retry(run, attempts: int = 3):
for attempt in range(1, attempts + 1):
try:
return await run()
except steel.APIStatusError as error:
body = error.body if isinstance(error.body, dict) else {}
if not body.get("retryable") or attempt == attempts:
raise
await asyncio.sleep(body.get("retryAfterSeconds", 5))
session = await with_retry(
lambda: client.sessions.create(metadata={"job_id": JOB_ID})
)import (
"encoding/json"
"errors"
"time"
"github.com/steel-dev/sdk/go/shared"
)
withRetry := func(run func() error, attempts int) error {
for attempt := 1; ; attempt++ {
err := run()
var apiErr *steel.Error
if err == nil || !errors.As(err, &apiErr) {
return err
}
body := struct {
Retryable bool `json:"retryable"`
RetryAfterSeconds float64 `json:"retryAfterSeconds"`
}{RetryAfterSeconds: 5}
raw := []byte(apiErr.JSON.RawJSON())
if json.Unmarshal(raw, &body) != nil || !body.Retryable || attempt >= attempts {
return err
}
time.Sleep(time.Duration(body.RetryAfterSeconds * float64(time.Second)))
}
}
var session *steel.Session
err := withRetry(func() error {
var err error
session, err = client.Sessions.New(ctx, steel.SessionNewParams{
Metadata: steel.F(map[string]steel.SessionNewParamsMetadataUnion{
"job_id": shared.UnionString(jobID),
}),
})
return err
}, 3)
if err != nil {
return err
}async fn with_retry<T, F, Fut>(
mut run: F,
attempts: u32,
) -> Result<T, Box<dyn std::error::Error>>
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<T, steel_rs::Error>>,
{
let mut attempt = 1;
loop {
let error = match run().await {
Ok(value) => return Ok(value),
Err(error) => error,
};
let body: serde_json::Value = error
.body()
.and_then(|body| serde_json::from_str(body).ok())
.unwrap_or_default();
if body["retryable"] != true || attempt >= attempts {
return Err(error.into());
}
let seconds = body["retryAfterSeconds"].as_f64().unwrap_or(5.0);
tokio::time::sleep(std::time::Duration::from_secs_f64(seconds)).await;
attempt += 1;
}
}
let session = with_retry(
|| async {
client
.sessions()
.create(steel_rs::SessionCreateParams {
metadata: Some(std::collections::HashMap::from([(
"job_id".to_owned(),
steel_rs::SessionCreateParamsMetadata::String(JOB_ID.to_owned()),
)])),
..Default::default()
})
.await
},
3,
)
.await?;In Rust, this code needs cargo add serde_json.
A failure inside the task, such as a page that does not load, is not an API error. Retry the task in a new session, and set a limit on the attempts. Before you retry a step that must not happen twice, such as a form that sends a payment, check whether the first attempt worked.
Always release sessions
A session that you do not release keeps a place in your concurrency limit, and it costs browser time until its timeout. To release every session on time:
- Release the session in
finally, as the pool program does, so that each worker releases its session even when the task fails. In Go, release it in adefer. In Rust, release it after theasyncblock. - Set
timeoutto the longest time that a task needs, with a margin. SetinactivityTimeoutso that Steel releases a session when the worker that drives it stops, for example after a crash. Set timeouts explains both. - Sweep for sessions that stay live. A worker can stop before
finallyruns, and a create request that times out on your side can still have created a session. When a job ends, and when a worker starts, release the live sessions that carry your job's metadata:
for await (const session of client.sessions.list({
status: "live",
metadata: { job_id: JOB_ID },
})) {
await client.sessions.release(session.id);
}async for session in client.sessions.list(
status="live",
metadata={"job_id": JOB_ID},
):
await client.sessions.release(session.id)iter := client.Sessions.ListAutoPaging(ctx, steel.SessionListParams{
Status: steel.F(steel.SessionListParamsStatusLive),
Metadata: steel.F(map[string]string{"job_id": jobID}),
})
for iter.Next() {
session := iter.Current()
if _, err := client.Sessions.Release(ctx, session.ID); err != nil {
return err
}
}
if err := iter.Err(); err != nil {
return err
}let sessions = client
.sessions()
.list_auto_paging(steel_rs::SessionListParams {
status: Some(steel_rs::SessionListParamsStatus::Live),
metadata: Some(std::collections::HashMap::from([(
"job_id".to_owned(),
JOB_ID.to_owned(),
)])),
..Default::default()
});
let mut sessions = std::pin::pin!(sessions);
while let Some(session) = futures_util::StreamExt::next(&mut sessions).await {
let session = session?;
client.sessions().release(&session.id).await?;
}In Rust, this code needs cargo add futures-util.
client.sessions.releaseAll() (release_all() in Python and Rust, ReleaseAll() in Go) releases every live session that your API key can reach, including the sessions of other jobs. Keep it for emergencies. See Release all sessions.
Watch signals and cost
When a job ends, check how its sessions went. Signals show the sessions that hit a block or a CAPTCHA, and the release reason shows the sessions that ran out of time. Steel adds signals within a few seconds after a session ends.
const blocked: string[] = [];
for await (const session of client.sessions.list({
metadata: { job_id: JOB_ID },
signal: ["bot_detection_triggered"],
})) {
blocked.push(session.id);
}
console.log(`${blocked.length} sessions were blocked`);blocked = []
async for session in client.sessions.list(
metadata={"job_id": JOB_ID},
signal=["bot_detection_triggered"],
):
blocked.append(session.id)
print(f"{len(blocked)} sessions were blocked")var blocked []string
iter := client.Sessions.ListAutoPaging(ctx, steel.SessionListParams{
Metadata: steel.F(map[string]string{"job_id": jobID}),
Signal: steel.F([]steel.SessionListParamsSignal{steel.SessionListParamsSignalBotDetectionTriggered}),
})
for iter.Next() {
blocked = append(blocked, iter.Current().ID)
}
if err := iter.Err(); err != nil {
return err
}
fmt.Printf("%d sessions were blocked\n", len(blocked))let mut blocked: Vec<String> = Vec::new();
let sessions = client
.sessions()
.list_auto_paging(steel_rs::SessionListParams {
metadata: Some(std::collections::HashMap::from([(
"job_id".to_owned(),
JOB_ID.to_owned(),
)])),
signal: Some(vec![
steel_rs::SessionListParamsSignal::BotDetectionTriggered,
]),
..Default::default()
});
let mut sessions = std::pin::pin!(sessions);
while let Some(session) = futures_util::StreamExt::next(&mut sessions).await {
blocked.push(session?.id);
}
println!("{} sessions were blocked", blocked.len());2 sessions were blockedA rise in blocked sessions can mean that a site changed its checks. See Get past bot detection. To find out why a session failed, see Debug a failed session.
To know what a job costs, read the cost of each of its sessions after release. Track cost per customer adds the costs up by a metadata key.
Check that it worked
When the pool program ends, it has printed a line for each of the 20 pages. A line with an error instead of a title is a task that failed. Then list the job's live sessions with client.sessions.list, with status set to live and the job's job_id in metadata, as the sweep in Always release sessions does. The list is empty when every worker released its session.