Building a Web Crawler on Cloudflare Workers with Puppeteer: Lessons from the Edge

I was building a hobby RAG (Retrieval Augmented Generation) project that needed to ingest website content as a knowledge source. The first attempt was a plain fetch(). Grab the HTML, strip the tags, chunk the text, embed it.

That works for some sites. But a lot of modern marketing pages are client-side rendered: the raw HTML response is an almost-empty shell, and the actual content only appears after JavaScript runs in a browser. For a knowledge base tool, that means indexing empty pages.

So I tried a full browser inside a Cloudflare Worker.

The Setup: Cloudflare Browser Rendering

Cloudflare’s Browser Rendering service gives you a managed Chromium instance accessible via a Puppeteer-compatible API, bound to your Worker through a CRAWLER environment binding. One entry in wrangler.jsonc, and you have a real browser you can script from the edge.

import puppeteer from "@cloudflare/puppeteer";

let browser: puppeteer.Browser | null = null;
try {
  browser = await puppeteer.launch(context.CRAWLER);
} catch {
  throw new Error('Failed to launch browser');
}

No Selenium grid, no server to manage.

Respecting robots.txt First

Before navigating anywhere, the crawler checks whether it’s allowed to. The robots-parser library makes this straightforward in Workers.

import robotsParser from "robots-parser";

const robotsTextPath = new URL(url).origin + "/robots.txt";
const response = await fetch(robotsTextPath);
const robots = robotsParser(robotsTextPath, await response.text());

const isAllowed = robots.isAllowed(url) ?? true;
if (!isAllowed) {
  return; // exit cleanly
}

If robots.txt doesn’t exist, we assume allowed. If the URL is blocked, we exit. Skipping this is how you cause unintended load on someone else’s server.

Three Iterations

v1 was minimal: open a page, collect every <a href>. Running it against a real site returned 167 links, most of them duplicates from header, footer, and mobile nav, and many pointing to external domains.

v2 introduced a BFS queue with a visited set to deduplicate. Better, but digitory.com and digitory.com/ both got enqueued as separate URLs (same page, different strings), and the crawler was following links to social media profiles and CDN hostnames.

v3 added path scoping: only enqueue URLs that are sub-paths of the seed URL. If you give it https://example.com/docs, it only follows links that start with that prefix. Everything else is skipped.

Crawling page 1: https://digitory.com
Raw links: 202 | Filtered links: 167 | Added 30 path-scoped links
Skipped 10 external links

Crawling page 2: https://digitory.com/
Queue size: 29 | Visited: 1
Added 0 path-scoped links (already queued)

The Performance Reality

Locally with wrangler dev, each page completed in 150 to 200 milliseconds.

On Cloudflare’s actual infrastructure with a remote browser instance, it was a different story:

Page	Time
digitory.com (first load, cold browser)	7,552ms
digitory.com/ (second load, warm browser)	1,090ms
About Us	933ms
Cloud Kitchens	24,930ms
Average (pages 3+)	~3,500ms

The first page cold-boots the browser instance. Pages 2+ reuse the warm browser, which is why they drop significantly. But some pages with heavy third-party scripts or web fonts spiked to 25 seconds regardless.

The Key Insight

After staring at these numbers long enough:

If a page takes more than 1 second to render, browser rendering is useless and expensive.

Cloudflare bills Browser Rendering by the minute. A 25-second page load for a page that would return full content in 50ms via plain fetch() is a massive cost multiplier for zero benefit.

Browser rendering makes sense when the page content genuinely requires JavaScript execution to exist: SPAs, client-rendered dashboards, pages behind frontend frameworks. If the raw HTML from fetch() already contains meaningful text, use fetch().

The Service Limits Worth Knowing

On the free plan: 1 concurrent browser, 10 browser-minutes per day. On paid plans: up to 120 concurrent browsers, 1 new browser per second.

Browsers auto-terminate after 60 seconds of inactivity by default (up to 10 minutes). For a crawl session, keep the browser alive across pages rather than launching a new one per URL. One session, one browser, many pages.

The @cloudflare/puppeteer package is a fork of upstream Puppeteer, so not every Puppeteer feature is available. Check the API coverage before building anything that depends on advanced automation.