Skip to content

Web scraping

Set up your agent

Give your coding agent a Steel browser and the Steel skills, so that it browses and writes Steel code the right way. Learn more

Install the Steel CLI. It signs you in and installs the skills for your coding agents.

curl -fsS https://setup.steel.dev | sh

Or send this prompt to your coding agent, and it does the same:

Read https://setup.steel.dev/install.md and follow it to set up Steel for this coding client.
Then continue my requested task using the installed Steel skill.

Set up your agent →

Scrape, screenshot and PDF are Quick actions. Each one loads a URL in a browser and returns the result in one request. Steel starts the browser, waits for the page, and closes the browser, so there is no session to create or release. Scrape returns the page's content, screenshot returns a link to a PNG image of the page, and PDF returns a link to a PDF of it. Use them when you have a URL and need its content or a copy of it, for example to give a page to a language model. When the task needs a login, clicks, or several pages, create a session instead, and use the screenshot and pdf commands of Commands or your browser library there.

A scrape's response always has the page's content, its metadata and its links. The metadata includes the title, description, Open Graph tags, JSON-LD data and HTTP status code.

Before you begin

Put your Steel API key in the STEEL_API_KEY environment variable, and install the Steel SDK for your language: TypeScript, Python, Go or Rust.

Scrape a page

Call client.scrape with the URL and the formats that you want. A URL without a scheme is fetched over HTTPS.

This program scrapes a page as Markdown and prints its title, status code and content.

scrape.mts
import Steel from "steel-sdk";

const client = new Steel();

const result = await client.scrape({
  url: "https://example.com",
  format: ["markdown"],
});

console.log(result.metadata.title, result.metadata.statusCode);
console.log(result.content.markdown);

Run the program:

npx tsx scrape.mts

The first line of the output is the title and status code. The page's Markdown follows it. The output starts like this:

Example Domain 200
This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.

Choose the output format

Pass one or more formats in format. Each format fills one field of content. Without format, the response has html only.

FormatFieldUse it for
htmlcontent.htmlThe page's HTML after its JavaScript ran.
cleaned_htmlcontent.cleaned_htmlHTML without scripts and styles.
markdowncontent.markdownText for a language model.
readabilitycontent.readabilityThe article that Mozilla Readability extracts from the page, as an object.

To get a screenshot or a PDF of the same page in the same request, set screenshot or pdf to true. The response then has screenshot.url or pdf.url, a link to the file.

const result = await client.scrape({
  url: "https://example.com",
  format: ["markdown", "readability"],
  screenshot: true,
  pdf: true,
});

console.log(result.screenshot?.url);
console.log(result.pdf?.url);

The links stay valid for 7 days, like the ones that Screenshot and PDF return.

Control how the page loads

Every Quick action takes these options:

OptionPythonGoRustWhat it does
delaydelayDelaydelayMilliseconds to wait after the page loads, before the capture. Use it for pages that render their content late.
useProxyuse_proxyUseProxyuse_proxyRoutes the request through a Steel residential proxy. See Proxies.
const result = await client.scrape({
  url: "https://example.com/pricing",
  format: ["markdown"],
  delay: 5000,
  useProxy: true,
});

A Quick action cannot target a proxy country, use your own proxy, or load a profile. Use a session for those.

Take a screenshot

Call client.screenshot with the URL. By default, the image shows the visible part of the page.

This program captures a page and prints the link to the image.

screenshot.mts
import Steel from "steel-sdk";

const client = new Steel();

const result = await client.screenshot({
  url: "https://example.com",
});

console.log(result.url);

Run the program:

npx tsx screenshot.mts

The program prints the link to the PNG image:

https://images.steel.dev/v1/static/artifacts/TOKEN/FILE_NAME.png

Screenshot takes these options:

OptionPythonGoRustWhat it does
fullPagefull_pageFullPagefull_pageCaptures the whole page instead of the visible part. The default is false.
delaydelayDelaydelayMilliseconds to wait after the page loads, before the capture. Use it for pages that render their content late.
useProxyuse_proxyUseProxyuse_proxyRoutes the request through a Steel residential proxy. See Proxies.

Capture the full page

Set fullPage to true to capture the whole page, from top to bottom, and not only the visible part.

const result = await client.screenshot({
  url: "https://example.com",
  fullPage: true,
});

Create a PDF

Call client.pdf with the URL.

This program turns a page into a PDF and prints the link to the file.

pdf.mts
import Steel from "steel-sdk";

const client = new Steel();

const result = await client.pdf({
  url: "https://example.com",
});

console.log(result.url);

Run the program:

npx tsx pdf.mts

The program prints the link to the PDF:

https://files.steel.dev/v1/static/artifacts/TOKEN/FILE_NAME.pdf

PDF takes these options:

OptionPythonGoRustWhat it does
delaydelayDelaydelayMilliseconds to wait after the page loads, before Steel creates the PDF. Use it for pages that render their content late.
useProxyuse_proxyUseProxyuse_proxyRoutes the request through a Steel residential proxy. See Proxies.

A screenshot and a PDF each return a link. Anyone who has the link can open the file for 7 days, without an API key. Do not post the link in public logs when the page shows private data. To keep the image longer, download it:

import { writeFile } from "node:fs/promises";

const response = await fetch(result.url);
await writeFile("page.png", Buffer.from(await response.arrayBuffer()));

The same code saves a PDF. Name the file page.pdf instead.

Check that it worked

In the output of the scrape program, the status code from metadata.statusCode (metadata.status_code in Python) is 200, and the page's text follows as Markdown. Open a link that the screenshot or PDF program prints. The image or the PDF shows the page, and the link opens without an API key.

Limits and pricing

Each Quick action starts a short-lived browser, so it counts toward your plan's limit of concurrent sessions while it runs. The scrape, screenshot and PDF endpoints each accept 20 requests a minute from each client IP address. Limits lists each plan's limits.

A scrape is billed for each request, and a screenshot or a PDF is not. Steel also bills browser time while a Quick action's short-lived browser runs, and a Steel proxy adds a charge for its bandwidth. Pricing has the rates.

Troubleshooting

'url' must point to a public destination

'url' must point to a public destination. Localhost, private, link-local, and other reserved IP addresses are not allowed.

Cause: The URL names localhost, or its host resolves to a private or reserved IP address. Quick actions load only public pages. The response has the status 400. A host that does not resolve gets 422 with the message The hostname in 'url' could not be resolved. Verify it is publicly reachable and retry.

Fix: Use a public URL.

Verify: The request returns the page's content.

The scrape action timed out

The scrape action timed out while waiting for the page. Retry, or try a simpler target page; increase 'delay' only if the page genuinely needs more time to settle.

Cause: The page did not finish loading in time. Screenshots and PDFs return the same message with screenshot or PDF instead of scrape.

Fix: Retry the request. Raise delay only when the page needs more time to show its content.

Verify: The retry returns the page's content.

Steel ran the action but could not store its result file

Steel ran the action but could not store its result file, so there is no URL to return. Retry later, and contact support if it persists.

Cause: Steel took the screenshot or made the PDF, but it could not save the file, so it has no URL to return. The response has the status 502 and retryable: false. A scrape gets this error only when it asks for a screenshot or a PDF.

Fix: Send the request again after a few minutes. If it keeps failing, contact Support.

Verify: The response has the file's URL.

deposit_required

On Launch, useProxy needs a verified account. You verify it once, with $10 of paid credits, as Verify a Launch account explains. To continue without it, set useProxy to false. Screenshots and PDFs return the same error. Errors lists the messages.

Steel proxies are not available for scraping on the hobby plan

Steel proxies are not available for scraping on the hobby plan. Set 'useProxy' to false, or upgrade your plan to use Steel proxies.

Cause: The request sets useProxy, but your organization is on the legacy Hobby plan, which does not include Steel proxies. The response has the status 402. Screenshots and PDFs return the same message with screenshots or PDFs instead of scraping.

Fix: Set useProxy to false, or change your plan in Billing and usage. To use your own proxy, create a session with it.

Verify: The request returns the page's content.

Next steps