---
site: steel.dev
collection: docs
canonical_url: https://steel.dev/docs/control/scrape
markdown_url: https://steel.dev/docs/control/scrape.md
content_signal: ai-input=yes, ai-train=yes, search=yes
title: Web scraping
description: Get a page as Markdown or HTML, a screenshot or a PDF with one request. Steel runs the browser, so there is no session to create or release.
summary: Get a page as Markdown or HTML, a screenshot or a PDF with one request.
type: how-to
reviewed: 2026-11-12
prerequisites:
  - https://steel.dev/docs/control.md
  - https://steel.dev/docs/control/snapshots-and-refs.md
related:
  - https://steel.dev/docs/control/skills.md
  - https://steel.dev/docs/control/mcp-server.md
  - https://steel.dev/docs/control/cdp.md
  - https://steel.dev/docs/control/commands.md
  - https://steel.dev/docs/control/computer-actions.md
agent_setup: https://setup.steel.dev/install.md
docs_index: https://steel.dev/docs/sitemap.md
---

# Web scraping

Scrape, screenshot and PDF are Quick actions. Each one loads a URL in a browser and returns the result in one request. Steel starts the browser, waits for the page, and closes the browser, so there is no session to create or release. Scrape returns the page's content, screenshot returns a link to a PNG image of the page, and PDF returns a link to a PDF of it. Use them when you have a URL and need its content or a copy of it, for example to give a page to a language model. When the task needs a login, clicks, or several pages, create a [session](https://steel.dev/docs/sessions) instead, and use the `screenshot` and `pdf` commands of [Commands](https://steel.dev/docs/control/commands) or your browser library there.

A scrape's response always has the page's content, its metadata and its links. The metadata includes the title, description, Open Graph tags, JSON-LD data and HTTP status code.

## Before you begin

Put your Steel API key in the `STEEL_API_KEY` environment variable, and install the Steel SDK for your language: [TypeScript](https://steel.dev/docs/reference/sdks/typescript#install), [Python](https://steel.dev/docs/reference/sdks/python#install), [Go](https://steel.dev/docs/reference/sdks/go#install) or [Rust](https://steel.dev/docs/reference/sdks/rust#install).

## Scrape a page

Call `client.scrape` with the URL and the formats that you want. A URL without a scheme is fetched over HTTPS.

This program scrapes a page as Markdown and prints its title, status code and content.

**TypeScript**

```ts file=scrape.mts
import Steel from "steel-sdk";

const client = new Steel();

const result = await client.scrape({
  url: "https://example.com",
  format: ["markdown"],
});

console.log(result.metadata.title, result.metadata.statusCode);
console.log(result.content.markdown);
```

**Python**

```python file=scrape.py
from steel import Steel

client = Steel()

result = client.scrape(
    url="https://example.com",
    format=["markdown"],
)

print(result.metadata.title, result.metadata.status_code)
print(result.content.markdown)
```

**Go**

```go file=main.go
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/steel-dev/sdk/go"
)

func main() {
	if err := run(context.Background()); err != nil {
		log.Fatal(err)
	}
}

func run(ctx context.Context) error {
	client := steel.NewClient()

	result, err := client.Scrape(ctx, steel.ScrapeParams{
		URL:    steel.F("https://example.com"),
		Format: steel.F([]steel.ScrapeParamsFormat{steel.ScrapeParamsFormatMarkdown}),
	})
	if err != nil {
		return err
	}

	fmt.Println(result.Metadata.Title, result.Metadata.StatusCode)
	fmt.Println(result.Content.Markdown)
	return nil
}
```

**Rust**

```rust file=src/main.rs
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = steel_rs::Client::new();

    let result = client
        .scrape(steel_rs::ScrapeParams {
            format: Some(vec![steel_rs::ScrapeParamsFormat::Markdown]),
            ..steel_rs::ScrapeParams::new("https://example.com".to_owned())
        })
        .await?;

    let title = result.metadata.title.unwrap_or_default();
    let markdown = result.content.markdown.unwrap_or_default();
    println!("{title} {}", result.metadata.status_code);
    println!("{markdown}");
    Ok(())
}
```

Run the program:

**TypeScript**

```sh
npx tsx scrape.mts
```

**Python**

```sh
python scrape.py
```

**Go**

```sh
go run .
```

**Rust**

```sh
cargo run
```

The first line of the output is the title and status code. The page's Markdown follows it. The output starts like this:

```text nocopy
Example Domain 200
This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.
```

## Choose the output format

Pass one or more formats in `format`. Each format fills one field of `content`. Without `format`, the response has `html` only.

| Format | Field | Use it for |
| - | - | - |
| `html` | `content.html` | The page's HTML after its JavaScript ran. |
| `cleaned_html` | `content.cleaned_html` | HTML without scripts and styles. |
| `markdown` | `content.markdown` | Text for a language model. |
| `readability` | `content.readability` | The article that Mozilla Readability extracts from the page, as an object. |

To get a screenshot or a PDF of the same page in the same request, set `screenshot` or `pdf` to `true`. The response then has `screenshot.url` or `pdf.url`, a link to the file.

**TypeScript**

```ts
const result = await client.scrape({
  url: "https://example.com",
  format: ["markdown", "readability"],
  screenshot: true,
  pdf: true,
});

console.log(result.screenshot?.url);
console.log(result.pdf?.url);
```

**Python**

```python
result = client.scrape(
    url="https://example.com",
    format=["markdown", "readability"],
    screenshot=True,
    pdf=True,
)

print(result.screenshot.url if result.screenshot else None)
print(result.pdf.url if result.pdf else None)
```

**Go**

```go
result, err := client.Scrape(ctx, steel.ScrapeParams{
	URL: steel.F("https://example.com"),
	Format: steel.F([]steel.ScrapeParamsFormat{
		steel.ScrapeParamsFormatMarkdown,
		steel.ScrapeParamsFormatReadability,
	}),
	Screenshot: steel.F(true),
	Pdf:        steel.F(true),
})
if err != nil {
	return err
}

fmt.Println(result.Screenshot.URL)
fmt.Println(result.Pdf.URL)
```

**Rust**

```rust
let result = client
    .scrape(steel_rs::ScrapeParams {
        format: Some(vec![
            steel_rs::ScrapeParamsFormat::Markdown,
            steel_rs::ScrapeParamsFormat::Readability,
        ]),
        screenshot: Some(true),
        pdf: Some(true),
        ..steel_rs::ScrapeParams::new("https://example.com".to_owned())
    })
    .await?;

let screenshot = result.screenshot.map(|s| s.url).unwrap_or_default();
let pdf = result.pdf.map(|p| p.url).unwrap_or_default();
println!("{screenshot}");
println!("{pdf}");
```

The links stay valid for 7 days, like the ones that [Screenshot](https://steel.dev/docs/control/scrape#take-a-screenshot) and [PDF](https://steel.dev/docs/control/scrape#create-a-pdf) return.

## Control how the page loads

Every Quick action takes these options:

| Option | Python | Go | Rust | What it does |
| - | - | - | - | - |
| `delay` | `delay` | `Delay` | `delay` | Milliseconds to wait after the page loads, before the capture. Use it for pages that render their content late. |
| `useProxy` | `use_proxy` | `UseProxy` | `use_proxy` | Routes the request through a Steel residential proxy. See [Proxies](https://steel.dev/docs/stealth/proxies). |

**TypeScript**

```ts
const result = await client.scrape({
  url: "https://example.com/pricing",
  format: ["markdown"],
  delay: 5000,
  useProxy: true,
});
```

**Python**

```python
result = client.scrape(
    url="https://example.com/pricing",
    format=["markdown"],
    delay=5000,
    use_proxy=True,
)
```

**Go**

```go
result, err := client.Scrape(ctx, steel.ScrapeParams{
	URL:      steel.F("https://example.com/pricing"),
	Format:   steel.F([]steel.ScrapeParamsFormat{steel.ScrapeParamsFormatMarkdown}),
	Delay:    steel.F(5000.0),
	UseProxy: steel.F(true),
})
if err != nil {
	return err
}
```

**Rust**

```rust
let result = client
    .scrape(steel_rs::ScrapeParams {
        format: Some(vec![steel_rs::ScrapeParamsFormat::Markdown]),
        delay: Some(5000.0),
        use_proxy: Some(true),
        ..steel_rs::ScrapeParams::new("https://example.com/pricing".to_owned())
    })
    .await?;
```

A Quick action cannot target a proxy country, use your own proxy, or load a profile. Use a [session](https://steel.dev/docs/sessions) for those.

## Take a screenshot

Call `client.screenshot` with the URL. By default, the image shows the visible part of the page.

This program captures a page and prints the link to the image.

**TypeScript**

```ts file=screenshot.mts
import Steel from "steel-sdk";

const client = new Steel();

const result = await client.screenshot({
  url: "https://example.com",
});

console.log(result.url);
```

**Python**

```python file=screenshot.py
from steel import Steel

client = Steel()

result = client.screenshot(
    url="https://example.com",
)

print(result.url)
```

**Go**

```go file=main.go
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/steel-dev/sdk/go"
)

func main() {
	if err := run(context.Background()); err != nil {
		log.Fatal(err)
	}
}

func run(ctx context.Context) error {
	client := steel.NewClient()

	result, err := client.Screenshot(ctx, steel.ScreenshotParams{
		URL: steel.F("https://example.com"),
	})
	if err != nil {
		return err
	}

	fmt.Println(result.URL)
	return nil
}
```

**Rust**

```rust file=src/main.rs
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = steel_rs::Client::new();

    let result = client
        .screenshot(steel_rs::ScreenshotParams::new(
            "https://example.com".to_owned(),
        ))
        .await?;

    println!("{}", result.url);
    Ok(())
}
```

Run the program:

**TypeScript**

```sh
npx tsx screenshot.mts
```

**Python**

```sh
python screenshot.py
```

**Go**

```sh
go run .
```

**Rust**

```sh
cargo run
```

The program prints the link to the PNG image:

```text nocopy
https://images.steel.dev/v1/static/artifacts/TOKEN/FILE_NAME.png
```

Screenshot takes these options:

| Option | Python | Go | Rust | What it does |
| - | - | - | - | - |
| `fullPage` | `full_page` | `FullPage` | `full_page` | Captures the whole page instead of the visible part. The default is `false`. |
| `delay` | `delay` | `Delay` | `delay` | Milliseconds to wait after the page loads, before the capture. Use it for pages that render their content late. |
| `useProxy` | `use_proxy` | `UseProxy` | `use_proxy` | Routes the request through a Steel residential proxy. See [Proxies](https://steel.dev/docs/stealth/proxies). |

### Capture the full page

Set `fullPage` to `true` to capture the whole page, from top to bottom, and not only the visible part.

**TypeScript**

```ts
const result = await client.screenshot({
  url: "https://example.com",
  fullPage: true,
});
```

**Python**

```python
result = client.screenshot(
    url="https://example.com",
    full_page=True,
)
```

**Go**

```go
result, err := client.Screenshot(ctx, steel.ScreenshotParams{
	URL:      steel.F("https://example.com"),
	FullPage: steel.F(true),
})
if err != nil {
	return err
}
```

**Rust**

```rust
let result = client
    .screenshot(steel_rs::ScreenshotParams {
        full_page: Some(true),
        ..steel_rs::ScreenshotParams::new("https://example.com".to_owned())
    })
    .await?;
```

## Create a PDF

Call `client.pdf` with the URL.

This program turns a page into a PDF and prints the link to the file.

**TypeScript**

```ts file=pdf.mts
import Steel from "steel-sdk";

const client = new Steel();

const result = await client.pdf({
  url: "https://example.com",
});

console.log(result.url);
```

**Python**

```python file=pdf.py
from steel import Steel

client = Steel()

result = client.pdf(
    url="https://example.com",
)

print(result.url)
```

**Go**

```go file=main.go
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/steel-dev/sdk/go"
)

func main() {
	if err := run(context.Background()); err != nil {
		log.Fatal(err)
	}
}

func run(ctx context.Context) error {
	client := steel.NewClient()

	result, err := client.Pdf(ctx, steel.PdfParams{
		URL: steel.F("https://example.com"),
	})
	if err != nil {
		return err
	}

	fmt.Println(result.URL)
	return nil
}
```

**Rust**

```rust file=src/main.rs
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = steel_rs::Client::new();

    let result = client
        .pdf(steel_rs::PdfParams::new("https://example.com".to_owned()))
        .await?;

    println!("{}", result.url);
    Ok(())
}
```

Run the program:

**TypeScript**

```sh
npx tsx pdf.mts
```

**Python**

```sh
python pdf.py
```

**Go**

```sh
go run .
```

**Rust**

```sh
cargo run
```

The program prints the link to the PDF:

```text nocopy
https://files.steel.dev/v1/static/artifacts/TOKEN/FILE_NAME.pdf
```

PDF takes these options:

| Option | Python | Go | Rust | What it does |
| - | - | - | - | - |
| `delay` | `delay` | `Delay` | `delay` | Milliseconds to wait after the page loads, before Steel creates the PDF. Use it for pages that render their content late. |
| `useProxy` | `use_proxy` | `UseProxy` | `use_proxy` | Routes the request through a Steel residential proxy. See [Proxies](https://steel.dev/docs/stealth/proxies). |

## Use the link

A screenshot and a PDF each return a link. Anyone who has the link can open the file for 7 days, without an API key. Do not post the link in public logs when the page shows private data. To keep the image longer, download it:

**TypeScript**

```ts
import { writeFile } from "node:fs/promises";

const response = await fetch(result.url);
await writeFile("page.png", Buffer.from(await response.arrayBuffer()));
```

**Python**

```python
import httpx

response = httpx.get(result.url)
response.raise_for_status()
with open("page.png", "wb") as file:
    file.write(response.content)
```

The same code saves a PDF. Name the file `page.pdf` instead.

## Check that it worked

In the output of the scrape program, the status code from `metadata.statusCode` (`metadata.status_code` in Python) is `200`, and the page's text follows as Markdown. Open a link that the screenshot or PDF program prints. The image or the PDF shows the page, and the link opens without an API key.

## Limits and pricing

Each Quick action starts a short-lived browser, so it counts toward your plan's limit of concurrent sessions while it runs. The scrape, screenshot and PDF endpoints each accept 20 requests a minute from each client IP address. [Limits](https://steel.dev/docs/platform/limits) lists each plan's limits.

A scrape is billed for each request, and a screenshot or a PDF is not. Steel also bills browser time while a Quick action's short-lived browser runs, and a Steel proxy adds a charge for its bandwidth. [Pricing](https://steel.dev/docs/platform/pricing) has the rates.

## Troubleshooting

### 'url' must point to a public destination

```text nocopy
'url' must point to a public destination. Localhost, private, link-local, and other reserved IP addresses are not allowed.
```

**Cause:** The URL names `localhost`, or its host resolves to a private or reserved IP address. Quick actions load only public pages. The response has the status `400`. A host that does not resolve gets `422` with the message `The hostname in 'url' could not be resolved. Verify it is publicly reachable and retry.`

**Fix:** Use a public URL.

**Verify:** The request returns the page's content.

### The scrape action timed out

```text nocopy
The scrape action timed out while waiting for the page. Retry, or try a simpler target page; increase 'delay' only if the page genuinely needs more time to settle.
```

**Cause:** The page did not finish loading in time. Screenshots and PDFs return the same message with `screenshot` or `PDF` instead of `scrape`.

**Fix:** Retry the request. Raise `delay` only when the page needs more time to show its content.

**Verify:** The retry returns the page's content.

### Steel ran the action but could not store its result file

```text nocopy
Steel ran the action but could not store its result file, so there is no URL to return. Retry later, and contact support if it persists.
```

**Cause:** Steel took the screenshot or made the PDF, but it could not save the file, so it has no URL to return. The response has the status `502` and `retryable: false`. A scrape gets this error only when it asks for a screenshot or a PDF.

**Fix:** Send the request again after a few minutes. If it keeps failing, contact [Support](https://steel.dev/docs/support).

**Verify:** The response has the file's URL.

### `deposit_required`

On Launch, `useProxy` needs a verified account. You verify it once, with $10 of paid credits, as [Verify a Launch account](https://steel.dev/docs/platform/pricing#verify-a-launch-account) explains. To continue without it, set `useProxy` to `false`. Screenshots and PDFs return the same error. [Errors](https://steel.dev/docs/reference/errors#deposit-required) lists the messages.

### Steel proxies are not available for scraping on the hobby plan

```text nocopy
Steel proxies are not available for scraping on the hobby plan. Set 'useProxy' to false, or upgrade your plan to use Steel proxies.
```

**Cause:** The request sets `useProxy`, but your organization is on the legacy Hobby plan, which does not include Steel proxies. The response has the status `402`. Screenshots and PDFs return the same message with `screenshots` or `PDFs` instead of `scraping`.

**Fix:** Set `useProxy` to `false`, or change your plan in [Billing and usage](https://steel.dev/docs/platform/billing-and-usage). To use your own proxy, create a session with it.

**Verify:** The request returns the page's content.

## Next steps

- [Commands](https://steel.dev/docs/control/commands): Sign in, click and read a page step by step in a session, when one request is not enough.
- [Quick actions API](https://steel.dev/docs/reference/api/quick-actions): Look up every field of the scrape, screenshot and PDF responses, such as `links`.
