What it really takes to scrape without getting detected.
Continue reading
So you want to Scrape like the Big Boys? 🚀
Posted on November 03, 2021 in Security • Tagged with scraping, industrial level scraping, big boys • 7 min read
7 different ways to detect Proxies
Posted on October 16, 2021 in Security • Tagged with proxy, proxy-detection, bot-detection, proxy-provider, anti-scraping • 14 min read
In this blog post, I demonstrate 7 different efficient ways how to detect a proxy server when the client is visiting a web server with a browser that has a proxy / VPN configured.
Continue reading
Where is the World headed to?
Posted on October 03, 2021 in Society • Tagged with Collapse, Society, Economy, Second-Machine-Age, Work • 18 min read
My personal opinion on economical and sociological topics in the near (2030) and mid-term (2050) future. This essay is divided into the following categories: Climate Crisis, Globalisation & Economy, Geopolitics & War, Advance of Medicine & Technology.
Continue reading
On the Architecture of Bot Detection Services
Posted on July 18, 2021 in Security • Tagged with Internet Bots, CAPTCHA, Bot Detection • 15 min read
There are unique challenges when developing a passive bot detecting system. In this blog article, I explain some of the obstacles that need to be overcome in order to detect advanced bots without presenting a CAPTCHA. I also explain how bot programmers can benefit from the architectural challenges that bot detection systems inherently suffer from.
Continue reading
API to Check if an IP Address belongs to a Datacenter / Cloud Provider
Posted on June 20, 2021 in Security • Tagged with API, IP-Address-Check, Datacenter, Cloud-Provider • 3 min read
For security reasons, it's often helpful to check if an IP Address belongs to a datacenter or cloud computing provider such as Amazon AWS or Microsoft Azure. Therefore, I have developed a simple public API that helps you to check if an IP address belongs to a datacenter / cloud provider.
Continue reading
Detecting Proxies and VPN's with Latency Measurements
Posted on June 07, 2021 in Security • Tagged with proxy-detection, anti-scraping • 10 min read
VPN's and Proxy Servers can be detected by comparing latencies measured with JavaScript in the browser with the corresponding latency of the TCP/IP handshake on the server.
Continue reading