Posted on July 18, 2021 in Security • Tagged with Internet Bots, CAPTCHA, Bot Detection • 15 min read
There are unique challenges when developing a passive bot detecting system. In this blog article, I explain some of the obstacles that need to be overcome in order to detect advanced bots without presenting a CAPTCHA. I also explain how bot programmers can benefit from the architectural challenges that bot detection systems inherently suffer from.
Posted on June 20, 2021 in Security • Tagged with API, IP-Address-Check, Datacenter, Cloud-Provider • 3 min read
For security reasons, it's often helpful to check if an IP Address belongs to a datacenter or cloud computing provider such as Amazon AWS or Microsoft Azure. Therefore, I have developed a simple public API that helps you to check if an IP address belongs to a datacenter / cloud provider.
Posted on June 07, 2021 in Security • Tagged with proxy-detection, anti-scraping • 10 min read
VPN's and Proxy Servers can be detected by comparing latencies measured with JavaScript in the browser with the corresponding latency of the TCP/IP handshake on the server.
Posted on June 05, 2021 in Security • Tagged with proxy-provider, bot-detection, anti-scraping • 10 min read
In this blog article I demonstrate several bullet-proof ways how to detect Brightdata Data Collector as a bot without any doubt.
Posted on May 20, 2021 in Scraping • Tagged with web scraping, crawling, puppeteer, playwright, CDP • 10 min read
In this blog post I explain why it is best to avoid puppeteer and playwright for web scraping.
Posted on April 24, 2021 in Security • Tagged with Proxy Detection • 7 min read