← Index

Butlerian Jihad

transjovian.org4 min read

Butlerian Jihad

In the Dune books, the Butlerian Jihad is a crusade by free humans

against thinking machines threatening to enslave them all:

… which results in the total destruction of virtually all forms of

“computers, thinking machines, and conscious robots”. With the

prohibition “Thou shalt not make a machine in the likeness of a

human mind,” the creation of even the simplest thinking machines is

outlawed and made taboo, which has a profound influence on the

socio-political and technological development of humanity … –

Butlerian](https://en.wikipedia.org/wiki/Butlerian_Jihad">Butlerian) Jihad

Out in the deep vastness of space, where traffic is slow and machines

are old, the endless crawling of public resources by bots is a blight on

our limited resources. If we don’t want our world to burn, we must keep

them out, never invite them in. They try to change shapes, blend in,

and therefore we must use what crude tools we have to defend

ourselves.

Our defences work in multiple layers.

The first layer is a filter for known user agents.

  1. Write a robots.txt file to instruct well-behaved bots from staying away.
  2. Check for unwelcome, self-identifying user-agents.
  3. For them, a 410 “Gone” response containing garbage is served.

Setup:

The second layer is a simple question, wrapped in a form.

A human will push the button. Technically, the form looks like it might make a change to a database or maybe a payment and so bots are reluctant to press the button. Specially since there seem to be no form fields to fill.

  1. Check for the cookie.
  2. If no cookie is found, a 402 “Payment required” response containing a form is returned.
  3. If the button is pressed, the cookie is set and the previous request is resent.
  4. If the cookie is found, the request is served.

Setup:

The third line of defence involves a mandatory login when average system load is too high.

  1. A configuration file called “gate” is included by all the sites.
  2. A service on a timer checks average system load and if the gate needs to open or close, the configuration file is overwritten and the web server reloads its configuration.
  3. If the gate is closed, a specific error code is used.
  4. The error message tells humans what username and password to use.

Setup:

A fourth line of defence involves detecting traffic spikes and banning the culprits. Sadly, extracting IP addresses from log files is no longer good enough because IP addresses are almost never reused. Instead, the job of scraping is farmed out to bot farms all over the world. The task is therefore to identify the bot farms and block all incoming traffic from them.

  1. Identify the IP addresses involved.
  2. For every IP address, identify its autonomous system number (ASN).
  3. For every ASN, identify all the networks it controls.
  4. Ban each and every single one of these network, for an hour.
  5. When an ASN is banned for more than three times in the last twenty-four hours, ban it for a week.

The data for the one-week ban is available online: ASN](https://alexschroeder.ch/share/1w-ban-asn.txt"\>ASN) list, IPv4](https://alexschroeder.ch/share/1w-ban-ipv4.txt"\>IPv4) address ranges, IPv6](https://alexschroeder.ch/share/1w-ban-ipv6.txt"\>IPv6) address ranges.

The tools required:

  • asncounter](https://gitlab.com/anarcat/asncounter/"\>asncounter\) finds the ASN for a given IP address
  • fail2ban](https://github.com/fail2ban/fail2ban"\>fail2ban\) manages the list of banned IP address ranges
  • nft](https://nftables.org/"\>nft\) does the actual banning on the firewall

Setup:

Once you have this setup, you can add variants.

Each variant acts as another layer of defence.

This is defence in depth.