Behavioral Analysis: Recording Mouse Movements and other User Interactions with JavaScript

Motivation

Most JavaScript analytics applications aim to visualize and provide statistics of the browsing behavior of website visitors. This is not my motivation for recording and storing analytics data. Instead, I want to make a simple statement on the grounds of user interaction data:

1. Whether the user interaction data seems to be of human nature
2. Or whether the data is generated by a automated program - a so called bot or automated user agent

Put differently, I want to classify user interaction data as either human or bot-like.

This is a very hard problem and there is no straightforward solution to it.

Currently, I am in the process of building a well-structured library that allows me to collect and store user interaction data for the next processing step. This technical topic will be the focus of this blog post.

When the point arrives that enough data has been collected, I will have to design a system that classifies the data according to the above criteria. One possible solution would be to create a training set to feed a deep neuronal network. However, the training data must be pre-classified into the categories human or bot. This initial classification step could be quite a daunting task alone.

Another problem is, that it is actually not that simple to find a large sample size of bot-generated interaction data, since a lot of bots do not create mouse or key-pressing events.

Only very sophisticated bots actually aim to mimic and replicate human behavior in the sense that their goal is to create user interaction events as a human would when they are browsing the web.

ReCaptcha v2 and v3

The standard method of telling humans apart from bots is Google's ReCaptcha system. In 2018, the new ReCaptcha v3 was introduced. The v2 version basically tasked the suspected user with a challenge that is hard to solve for computers, but relatively easy for humans.

The new v3 version is fully transparent and computes a continuous score between 0 (bot) and 1 (human) to rate all actions occurring on a website. This blog post gives an excellent overview of the subject.

Both ReCaptcha versions give a significantly better score to users that are using Google's Chrome browser and are logged in into their Google account. If you delete your Google cookies and you are using Firefox and you block third party cookies, you will be faced with a significantly lower score.

The consequence of a low score can be twofold:

1. You will get banned from using the website that is using Google ReCaptcha v3
2. You will have to solve an actual ReCaptcha v2 challenge to prove your humanness

From Google's perspective, it makes perfect sense to sanction users that are not logged into one of the many Google services or are are not using the Chrome browser. Owning a Google account proves a lot: That you are watching YouTube videos, writing E-Mails with Gmail or that you are using your Android Phone. All those apps verify on the side that you are an human.

The big BUT is obvious:

When there is no active usage history of you, you are automatically considered as second class Internet citizen. This is very dangerous. The classification whether you are considered a human or a bot should be based on current data, not on past usage history.

In that sense, this essay makes an argument for instant bot classification based on behavioral analytics on your current session.

Worded differently, if you jump around like crazy and your keyboard is typing millions of words per second, you are probably a bot.

On the other hand, when you read a blog article like this very carefully and your mouse pointer is following each line in a smooth, human like fashion, you probably are a human being.

User Interaction Data

So what kind of user interaction data should be recorded by the JavaScript library? Currently, the library captures the following user induced events:

1. mousemove Store the (x,y) coordinates of the current mouse cursor
2. mousedown Store the (x,y) coordinates of a mouse left click
3. scroll Store the document.scrollingElement.scrollLeft and document.scrollingElement.scrollTop variables when a scroll event happens
4. keydown Store the currently pressed key code
5. resize Fires when the viewport is resized. The new viewport size is stored
6. contextmenu This event is fired when the user makes a right click with the mouse. The coordinates of the right click are stored
7. touchstart Mobile touch event. This event is generated when the user taps on the screen
8. touchmove Mobile touch event. This event arises when the user moves with their fingers across the touch screen
9. touchcancel Mobile touch event. Fires when a touch event is canceled
10. touchend Mobile touch event. Is fired when the finger is lifted from the touch screen

When to send behavioral analytics data to the server?

One tricky problem with JavaScript analytics applications is the question of when and how to send the recorded data to the server. Obviously, one key requirement is to record user interaction data as long as possible until to the point when the user leaves the page.

However, when the recording of user interaction data stops too early, crucial data is missing for analysis. When we record for too long, it can become problematic to send the data to the server. What would be a suitable event to consider for this requirement? A couple of different events can be considered in this regard:

1. beforeunload Fired right before the window, the document and its resources are about to be unloaded. This is a cancelable event.
2. unload Fires when the document or a child resource is being unloaded. This event is fired after the beforeunload and pagehide event. An error in the event handler will not stop the unloading workflow.
3. pagehide event is sent to a Window when the browser hides the current page by presenting a different page from the session's history. When pressing the "back button", this event is fired.
4. visibilitychange is fired when the content of a page has become visible or hidden.

An excellent article named "Don't lose user and app state, use Page Visibility" makes an case for the visibilitychange event API. No other event should be used to send analytics data to the remote server, especially because the other events fire unreliably.

How to send the recorded data?

The next question to be answered is how the recorded data should be transmitted to the remote server. There are several possibilities that come to mind:

1. Use the good old XMLHttpRequest object to make HTTP requests (also called Ajax)
2. Use the relatively new fetch() HTTP API
3. Make use of an <img> tag and use the src attribute to transmit data in the query string
4. Use navigator.sendBeacon() to asynchronously send a small amount of data over HTTP to a web server

The navigator.sendBeacon() API was introduced exactly for the purpose to asynchronously transmit data before the page is being closed. The mandatory reading list for this topic is the related MDN page.

The navigator.sendBeacon() method asynchronously sends a small amount of data over HTTP to a web server. It’s intended to be used in combination with the visibilitychange event (but not with the unload and beforeunload events).

Testing sendBeacon() behavior with different events

In this section, the reliability of sendBeacon() is tested in combination with different events.

The following simple express server can be used to listen for incoming POST requests from navigator.sendBeacon().

Install it with the command:

npm install express body-parser cors

and then launch the server with:

node server.js.

Let's assume the server is running on http://localhost:8888 from now on.

// server.js
const express = require('express');
const cors = require('cors');
const bodyParser = require('body-parser');
const path = require('path');

const app = express();
const port = 8888;

app.use(express.json());
app.use(cors());
app.use(bodyParser.json({ limit: '2mb' }));
app.use(bodyParser.text());

app.post('/data', (req, res) => {
  console.log(req.body);
  res.status(200).send('ok');
});

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
});

The following code snippets are to be executed within the web page context. You can also paste them into the developer console directly.

First, let's see how the visibilitychange event is handled in JavaScript:

var n = 0;
document.addEventListener("visibilitychange", function(event) {
  if (document.visibilityState === 'hidden') {
    n++;
    var message = 'visibilitychange - hidden - ' + n;
    navigator.sendBeacon('http://localhost:8888/data', message);
  }
})

A similar code snippet for the beforeunload event:

var n = 0;
window.addEventListener('beforeunload', function(event) {
  n++;
  var message = 'beforeunload - ' + n;
  navigator.sendBeacon('http://localhost:8888/data', message);
})

The same logic holds for the pagehide and unload event and is excluded here for simplicity.

Test Results for mobile and desktop browsers

As the code snippets above demonstrate, we hook into an event from the page lifecycle and then attempt to send a small amount of data to our web server.

We consider the event to be a success if BOTH of the following points hold:

1. The browser actually fired the event in question
2. The browser succeeded in sending the data with navigator.sendBeacon()

Therefore, it is possible that the browser succeeds in firing the event, but fails to deliver the payload data with navigator.sendBeacon(). This would still be considered as an failure, regardless that another transmission method might be successful.

The different events from above where tested on a desktop computer Ubuntu 18.04.5 LTS and on a Android Phone Motorola g(6) running Android version 9. On both the desktop and mobile platform, Firefox and Chrome were tested.

With the Chrome browser and the Firefox browser on a Desktop computer (Ubuntu 18.04.5 LTS) the following results were obtained.

The symbol ✓ indicates that the payload data was successfully received in the express server. The symbol ✗ states the opposite.

It can be seen that the beacon is successfully received when using the event visibilitychange. The other events are not ideal and reliable for transmitting analytics data.

The following data was obtained on a mobile phone Android Phone Motorola g(6) device. The behavior of the event was tested with actions suited for mobile platforms.

Again, on both mobile browsers, the visibilitychange seems to be the only rational choice when sending analytics data to a remote server. The other events do not reliably guarantee the delivery of the beacon data.

Analytics Algorithm

Now that it has been established that it's best to consider the visibilitychange event, let's implement a simple algorithm that sends data to our remote server to collect the analytics data. The following JavaScript needs to be embedded in the <body> element.

// to be generated randomly by the server
var uuid = '{random-uuid}';

// base url 
var url = 'https://example.org/path';

// Instantiate a analytics object
var analytics = new Analytics();

// Start recording analytics data
analytics.record();

document.addEventListener("visibilitychange", function(event) {
  if (document.visibilityState === 'hidden') {
        var data = {
      uuid: uuid,
      // subsequent calls to getData() 
      // will yield newly generated analytics data only
            data: analytics.getData(),
            href: window.location.href,
        };

        navigator.sendBeacon(url, JSON.stringify(data));
  }
});

What is the maximum payload size of sendBeacon()?

The last question that needs to be answered: How much analytics data can we transmit at once with sendBeacon()?

This question was already asked on Stackoverflow and it seems that the maximum payload is 2^16 = 65536 Bytes.

This means that analytics data needs to be either compressed or sent in chunks (or both actually).

Some good compression libraries for JavaScript would be pako and fflate. fflate is probably the better choice, since it is smaller and faster.

Another solution would be to just send the analytics data to the server as soon as we approach the 65536 byte limit and then reassemble the chunks on the server.

Statistics

In this section, statistics of analytics sessions are published as soon as enough data was gathered. With this data, it can be answered how many browsers support the visibilitychange and navigator.sendBeacon() combination.

To be done.

Conclusion

There are several tricky problems that need to be solved when you want to create an analytics application that is widely supported on most browsers.

To summarize, the following problems were solved in this blog article:

1. What event is the best to listen for when you want to capture the moment a user leaves or terminates the browsing session? Answer: visibilitychange
2. What HTTP API is the best to use in order to transmit your analytics data to the remote server? Answer: navigator.sendBeacon()