Detecting Brightdata's (formerly Luminati Networks) Data Collector as a Bot

Posted on June 05, 2021 in Security • Tagged with proxy-provider, bot-detection, anti-scraping • 10 min read

In this blog article I demonstrate several bullet-proof ways how to detect Brightdata Data Collector as a bot without any doubt.


Continue reading

Avoid Puppeteer or Playwright for Web Scraping

Posted on May 20, 2021 in Scraping • Tagged with web scraping, crawling, puppeteer, playwright, CDP • 10 min read

In this blog post I explain why it is best to avoid puppeteer and playwright for web scraping.


Continue reading

Detecting Datacenter and Residential Proxies

Posted on April 24, 2021 in Security • Tagged with Proxy Detection • 7 min read

Detecting proxys can't be that hard? Can it?


Continue reading

Behavioral Analysis for Bot Detection

Posted on April 11, 2021 in Security • Tagged with Behavioral Analysis, Bot Detection • 4 min read

Behavioral analysis is an interesting approach to detect bots. It surely is not the panacea for bot detection, but it certainly is an useful extension in your bot hunting tool belt.


Continue reading

TCP/IP Fingerprinting for VPN and Proxy Detection

Posted on March 13, 2021 in Security • Tagged with TCP, IP, fingerprinting, Proxy, VPN • 9 min read

TCP/IP fingerprinting is as old as the Internet itself. But this technique seems to have lost it's relevancy in our modern times. However, with the rise of Proxy and VPN Providers, TCP/IP fingerprinting becomes interesting again from a security perspective.


Continue reading

Detecting scraping services

Posted on March 11, 2021 in Scraping • Tagged with detecting, scraping, security, fingerprint • 13 min read

In this blog post I will demonstrate how it is possible to detect several scraping services: luminati.io, ScrapingBee, scraperapi.com, scrapingrobot.com, scrapfly.io.


Continue reading