Deep Dive Topics

Page 6 of 6

Handling Rate Limits and Retries When Scraping

Handling Rate Limits and Retries When Scraping A scraper that ignores rate limits fails twice: it gets blocked, and it burdens the server it depends on. When a site returns HTTP 4...

Respecting robots.txt and Crawl Delays

Respecting robots.txt and Crawl Delays Most scrapers that get blocked were never malicious — they simply never asked permission. A site publishes its rules for automated agents at...

Scraping Cursor-Based Pagination APIs

Scraping Cursor-Based Pagination APIs Modern listing endpoints have largely abandoned ?page= and ?offset= in favour of an opaque cursor: the server hands back a token such as eyJp...

Scraping Infinite Scroll Pages with Playwright

Scraping Infinite Scroll Pages with Playwright An infinite-scroll feed never exposes its full dataset in the DOM at once. New items appear only when you scroll near the bottom, an...

Blocking Images and Fonts to Speed Up Scraping

Blocking Images and Fonts to Speed Up Scraping A scraper that renders a product catalogue downloads hero photography, sprite sheets, tracking pixels, webfont subsets and video pos...

Rotating Proxies in Playwright

Rotating Proxies in Playwright A scraper that sends every request from one egress address gets rate limited, then challenged, then blocked — and once the address is burned, restar...

Running Parallel Scrapers with Worker Pools

Running Parallel Scrapers with Worker Pools A scraper that walks a list of URLs one at a time is limited by network latency, not by your machine: the CPU idles for hundreds of mil...

Extracting JSON-LD and Microdata

Extracting JSON-LD and Microdata Most commercial pages already publish the record you are trying to scrape. Product pages, recipes, job listings, events and articles carry a schem...

Extracting Tables and Lists to JSON with Playwright

Extracting Tables and Lists to JSON with Playwright HTML tables and lists are the most common shape of structured data on the web — pricing grids, sortable reports, leaderboards, ...

Scraping Data Behind Login Sessions

Scraping Data Behind Login Sessions Much of the most valuable data lives behind a login — account dashboards, order histories, members-only reports. Logging in on every extraction...