Build a web scraping agent with n8n and Bright Data

A scraping agent is a small loop: something decides which URL to read, something fetches that page without getting blocked, and something turns the raw HTML into fields another system can store. This tutorial builds that loop in n8n and Bright Data. n8n is the workflow runner. Bright Data's Web Unlocker is the fetch. You will end with a workflow you can run by hand, then the same workflow behind a webhook so another agent can hand it a URL.

You do not need a custom n8n node. The HTTP Request node that ships with n8n is enough. Bright Data also documents a community node aimed at the Web Unlocker API. Use that later if you prefer a form over a JSON body. The request below is the one their REST reference shows, so it keeps working if you move the workflow from n8n Cloud to a self-hosted instance.

What you are building

The workflow has four steps.

1. A trigger. Start with the manual trigger so you can debug. Swap in a webhook when the steps succeed.
2. One item that carries the target URL and the Bright Data zone name.
3. An HTTP Request node that POSTs those values to https://api.brightdata.com/request.
4. A Code node that strips scripts, styles, and tags, and returns a short text field.

An optional fifth step asks a chat model to emit JSON. That step is what makes the workflow an agent rather than a downloader. Keep it optional until the fetch itself is reliable. A model cannot recover a page you never received.

The catalogue pages for both tools are on IQ.net: n8n and Bright Data. Open those if you want the index record before you create accounts. The buttons further down are the affiliate links.

Create the Bright Data zone and key

Sign in to Bright Data and create a Web Unlocker zone if you do not already have one. The zone name is not a secret, but it is account-specific. The REST examples use web_unlocker1 as a placeholder. Replace it with the name shown on your zone. A request with a zone name that does not exist on the account is rejected before any fetch runs.

Create an API key in the account user settings. Bright Data's request API expects it as a bearer token:

Authorization: Bearer YOUR_API_KEY

Put the key in an n8n credential, not in the workflow JSON and not in a chat transcript. Anyone who can read the key can spend the zone's traffic.

Confirm the key with curl before you touch n8n. A failure here is easier to read in a terminal than inside a node.

curl --request POST \
  --url https://api.brightdata.com/request \
  --header "Authorization: Bearer $BRIGHT_DATA_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "zone": "YOUR_ZONE_NAME",
    "url": "https://example.com",
    "format": "raw"
  }'

format is either raw or json. Use raw for this tutorial. You get the page body as text, which is what the Code node expects. url must include https://. A URL without a scheme fails validation.

Bright Data documents two status layers on this direct API. Authentication problems and an unknown zone come back as a real HTTP error, such as 401 or 400, with a short message. Once the request is accepted by the unlocker, the outer HTTP status is often 200, and the target site's status is in the x-brd-status-code response header. If a run returns HTML that is actually an error page, read x-brd-error and x-brd-error-code before you blame the parser. The same codes are described in Bright Data's Web Unlocker error reference.

Add a country only when you need one. The field is a two-letter code, for example "country": "us". Leaving it out lets the zone pick a location. Do not invent a country code the zone is not allowed to use.

Open Bright Data

Build the n8n workflow

Create a blank workflow in n8n.

Add a Manual Trigger. Name it Manual. You will click Execute while testing.

Add a Set node, the one n8n labels Edit Fields. Name it Target. Create two string fields:

• url set to https://example.com
• zone set to your zone name

Using your own fields, instead of typing the URL into the HTTP node, means the webhook version later in this tutorial can pass the same names through.

Add an HTTP Request node named Bright Data request.

• Method: POST
• URL: https://api.brightdata.com/request
• Authentication: Generic credential type, Header Auth
• Send body: on
• Body content type: JSON
• Response format: Text
• Timeout: 90000 milliseconds

Create the Header Auth credential with the header name Authorization and the value Bearer YOUR_API_KEY, including the word Bearer and a space. n8n stores that credential outside the workflow export, which is what you want.

Set the JSON body to an expression so it reads the Set node:

{
  "zone": "{{ $json.zone }}",
  "url": "{{ $json.url }}",
  "format": "raw"
}

In the node UI, the body field should be an expression. If you paste this as plain text, n8n sends the braces literally and Bright Data never sees the URL. After the first successful run, open the output. The HTML string is usually on data. If your n8n version names that field differently, use the name you see when you edit the Code node below. Do not assume a JSON property called html.

Add a Code node named HTML to text. Language: JavaScript. Paste this:

const item = $input.first().json;
const html = typeof item.data === "string"
  ? item.data
  : typeof item.body === "string"
    ? item.body
    : "";
if (!html) {
  throw new Error("Bright Data returned no text body. Open the HTTP node output and map the field that holds the page.");
}
const text = html
  .replace(/<script[\s\S]*?<\/script>/gi, " ")
  .replace(/<style[\s\S]*?<\/style>/gi, " ")
  .replace(/<[^>]+>/g, " ")
  .replace(/&nbsp;/gi, " ")
  .replace(/&amp;/gi, "&")
  .replace(/&lt;/gi, "<")
  .replace(/&gt;/gi, ">")
  .replace(/\s+/g, " ")
  .trim()
  .slice(0, 12000);
const target = $("Target").first().json;
return [{ json: { url: target.url, text, chars: text.length } }];

The slice at 12,000 characters is deliberate. A model context window is not a place to dump a full retail homepage, and n8n executions get hard to open when every item carries a megabyte of markup. If you need the whole page, write text to object storage or a table and pass a pointer to the next step.

Run the workflow. example.com should come back as a short sentence. Then point url at a page you are allowed to read. If the Code node throws, the HTTP node did not return text. Check the key, the zone name, and the response headers before you change the regular expressions.

Open n8n

Import the same workflow

If you would rather import than click, save this as bright-data-scrape.json and import it from the n8n workflow menu. After import, attach your Header Auth credential on the HTTP node. The file does not contain a key.

{
  "name": "Bright Data page to text",
  "nodes": [
    {
      "parameters": {},
      "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "name": "Manual",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [240, 300]
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            { "id": "url", "name": "url", "value": "https://example.com", "type": "string" },
            { "id": "zone", "name": "zone", "value": "YOUR_ZONE_NAME", "type": "string" }
          ]
        },
        "options": {}
      },
      "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
      "name": "Target",
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [480, 300]
    },
    {
      "parameters": {
        "method": "POST",
        "url": "https://api.brightdata.com/request",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={\n  \"zone\": \"{{ $json.zone }}\",\n  \"url\": \"{{ $json.url }}\",\n  \"format\": \"raw\"\n}",
        "options": {
          "response": { "response": { "responseFormat": "text" } },
          "timeout": 90000
        }
      },
      "id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
      "name": "Bright Data request",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [740, 300]
    },
    {
      "parameters": {
        "language": "javaScript",
        "jsCode": "const item = $input.first().json;\nconst html = typeof item.data === 'string' ? item.data : '';\nconst text = html.replace(/<script[\\s\\S]*?<\\/script>/gi, ' ').replace(/<style[\\s\\S]*?<\\/style>/gi, ' ').replace(/<[^>]+>/g, ' ').replace(/\\s+/g, ' ').trim().slice(0, 12000);\nreturn [{ json: { url: $('Target').first().json.url, text } }];"
      },
      "id": "d4e5f6a7-b8c9-0123-def0-234567890123",
      "name": "HTML to text",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [1000, 300]
    }
  ],
  "connections": {
    "Manual": { "main": [[{ "node": "Target", "type": "main", "index": 0 }]] },
    "Target": { "main": [[{ "node": "Bright Data request", "type": "main", "index": 0 }]] },
    "Bright Data request": { "main": [[{ "node": "HTML to text", "type": "main", "index": 0 }]] }
  },
  "pinData": {},
  "active": false,
  "settings": { "executionOrder": "v1" }
}

n8n changes node parameter names between versions. If the import flags the HTTP node, do not fight the file. Recreate that one node from the field list above and keep the Set and Code nodes. The behavior you want is a POST with a bearer token, a JSON body, a text response, and a 90 second timeout.

Turn the fetch into an extraction

The text field is still a page, not a decision. Add one more step only after example.com works. The simplest agent-shaped step is an HTTP Request to whatever chat API you already use, with a system instruction that forbids guessing:

You extract facts from page text. Return one JSON object and nothing else.
Keys: title (string or null), summary (string, at most 400 characters), facts (array of short strings).
Use only the page text. If a value is not on the page, use null or an empty array.

Send text from the Code node as the user message. Parse the model output with a second Code node that runs JSON.parse inside try/catch and throws a readable error when the model adds a preamble. Store the object. Do not store the bearer token in the same table as the extracted facts.

If you do not want to operate a model key inside n8n yet, stop at the text field and hand text to the agent that called the webhook. That caller already has a model. Your workflow's job is the fetch.

Accept work from another agent

Replace the manual trigger with a Webhook node. Method POST. Path scrape. Respond when the last node finishes.

Expect a body like this:

{
  "url": "https://example.com",
  "zone": "YOUR_ZONE_NAME"
}

Point the Set node at the webhook body ({{ $json.body.url }} on many n8n versions, or {{ $json.url }} if the node already flattened the body). Run it once from the webhook test URL and read which shape you actually received. Then lock the expression to that shape.

The caller, which might be another n8n workflow or a small script, posts the URL and waits. Treat the webhook as an internal endpoint. n8n can require header auth on the webhook. Turn that on before you expose the URL beyond your own machine. The Bright Data key stays in the credential. The caller never sees it.

A schedule trigger is the other useful front door. At a set time, read a list of URLs from a table, and loop the HTTP node with Split In Batches. Keep the batch size at 1 while you learn the zone's error headers. Parallel bursts make rate and block errors harder to attribute.

When a run fails

Work through the failures in this order.

• 401, or a message about the API key. The Header Auth value is missing Bearer , or the key was rotated.
• 400 and zone ... not found. The zone field does not match a zone on that account. Copy it from the zone list rather than from a blog post.
• 400 and a validation message about the URL. The URL has no scheme, or the JSON is still the literal {{ }} because the body was not an expression.
• Outer 200, but x-brd-error is set. The unlocker refused or could not complete the fetch. Read the error code. Do not feed that body to the model.
• Outer 200, empty Code node. Response format is still JSON, so n8n tried to parse HTML and produced nothing useful. Switch the HTTP node to text.
• A timeout at 90 seconds. Some pages are slow. Raise the timeout once, not to several minutes, and record the URL. A page that never returns is a product decision, not a parser bug.

Also decide what you will not fetch. The workflow will request whatever URL you give it. Point it at pages you are allowed to read. Do not use it to collect login-gated personal data, and do not hide the traffic source from a site whose terms you have agreed to respect. Bright Data applies its own destination rules. A policy block is a stop, not a prompt to try a different header.

Where to go next

This workflow is a single URL in and text out. When the agent needs a cleaned markdown document, or a crawl of many pages on one site, read Give your AI agent live web access with Firecrawl. When you are choosing between URL-in APIs and a recorded robot, read Scraping APIs for AI agents. Both of those guides use the same rule as this one: the key lives in a credential, and the model sees the page text, not the secret.

Open Bright Data

Open n8n