[View original](https://transjovian.org/view/fight-bots/index)

<h1>Butlerian Jihad</h1>

<p>In the <em>Dune</em> books, the Butlerian Jihad is a crusade by free humans

against thinking machines threatening to enslave them all:</p>

<blockquote>

<p>… which results in the total destruction of virtually all forms of

“computers, thinking machines, and conscious robots”. With the

prohibition “Thou shalt not make a machine in the likeness of a

human mind,” the creation of even the simplest thinking machines is

outlawed and made taboo, which has a profound influence on the

socio-political and technological development of humanity … –

<a href="[https://en.wikipedia.org/wiki/Butlerian\_Jihad">Butlerian](https://en.wikipedia.org/wiki/Butlerian_Jihad"\>Butlerian) Jihad</a></p>

</blockquote>

<p>Out in the deep vastness of space, where traffic is slow and machines

are old, the endless crawling of public resources by bots is a blight on

our limited resources. If we don’t want our world to burn, we must keep

them out, never invite them in. They try to change shapes, blend in,

and therefore we must use what crude tools we have to defend

ourselves.</p>

<p>Our defences work in multiple layers.</p>

<p>The first layer is a filter for known user agents.</p>

<ol>

<li>Write a <code>robots.txt</code> file to instruct well-behaved bots from staying away.</li>

<li>Check for unwelcome, self-identifying user-agents.</li>

<li>For them, a 410 “Gone” response containing garbage is served.</li>

</ol>

<p>Setup:</p>

<ul>

<li>The site’s <a href="[https://transjovian.org/robots.txt">robots.txt</a>](https://transjovian.org/robots.txt"\>robots.txt\</a\>) file denies almost all bots.</li>

<li><a href="blocklist">Configure Apache</a> to check for unwelcome user-agents.</li>

<li>Optionally, <a href="garbage">install a garbage server</a> to feed them.</li>

</ul>

<p>The second layer is a simple question, wrapped in a form.

A human will push the button. Technically, the form looks like it might make a change to a database or maybe a payment and so bots are reluctant to press the button. Specially since there seem to be no form fields to fill.</p>

<ol>

<li>Check for the cookie.</li>

<li>If no cookie is found, a 402 “Payment required” response containing a form is returned.</li>

<li>If the button is pressed, the cookie is set and the previous request is resent.</li>

<li>If the cookie is found, the request is served.</li>

</ol>

<p>Setup:</p>

<ul>

<li><a href="payment-form">Create the form</a> to be used as the “error page”</li>

<li><a href="botcheck">Configure Apache</a> to check for the cookie</li>

</ul>

<p>The third line of defence involves a mandatory login when average system load is too high.</p>

<ol>

<li>A configuration file called “gate” is included by all the sites.</li>

<li>A service on a timer checks average system load and if the gate needs to open or close, the configuration file is overwritten and the web server reloads its configuration.</li>

<li>If the gate is closed, a specific error code is used.</li>

<li>The error message tells humans what username and password to use.</li>

</ol>

<p>Setup:</p>

<ul>

<li><a href="gate">Create the gate</a> (web server config file and password file).</li>

<li><a href="gate-service">Create a service and timer</a> to open or close the gate depending on average system load.</li>

</ul>

<p>A fourth line of defence involves detecting traffic spikes and banning the culprits. Sadly, extracting IP addresses from log files is no longer good enough because IP addresses are almost never reused. Instead, the job of scraping is farmed out to bot farms all over the world. The task is therefore to identify the bot farms and block all incoming traffic from them.</p>

<ol>

<li>Identify the IP addresses involved.</li>

<li>For every IP address, identify its autonomous system number (ASN).</li>

<li>For every ASN, identify all the networks it controls.</li>

<li>Ban each and every single one of these network, for an hour.</li>

<li>When an ASN is banned for more than three times in the last twenty-four hours, ban it for a week.</li>

</ol>

<p>The data for the one-week ban is available online: <a href="[https://alexschroeder.ch/share/1w-ban-asn.txt">ASN](https://alexschroeder.ch/share/1w-ban-asn.txt"\>ASN) list</a>, <a href="[https://alexschroeder.ch/share/1w-ban-ipv4.txt">IPv4](https://alexschroeder.ch/share/1w-ban-ipv4.txt"\>IPv4) address ranges</a>, <a href="[https://alexschroeder.ch/share/1w-ban-ipv6.txt">IPv6](https://alexschroeder.ch/share/1w-ban-ipv6.txt"\>IPv6) address ranges</a>.</p>

<p>The tools required:</p>

<ul>

<li><a href="[https://gitlab.com/anarcat/asncounter/">asncounter</a>](https://gitlab.com/anarcat/asncounter/"\>asncounter\</a\>) finds the ASN for a given IP address</li>

<li><a href="[https://github.com/fail2ban/fail2ban">fail2ban</a>](https://github.com/fail2ban/fail2ban"\>fail2ban\</a\>) manages the list of banned IP address ranges</li>

<li><a href="[https://nftables.org/">nft</a>](https://nftables.org/"\>nft\</a\>) does the actual banning on the firewall</li>

</ul>

<p>Setup:</p>

<ul>

<li><a href="nft">Switch to nft</a> instead of using IP tables</li>

<li><a href="jails">Create two jails</a> for the 1h bans and the 1w bans</li>

<li><a href="timer">Create a timer</a> to call the service every 10min</li>

<li><a href="service">Create a service</a> to call the script</li>

<li><a href="script">Create a script</a> to populate the 1h jail</li>

<li><a href="allow-lists">Create allow-lists</a> to make exceptions</li>

</ul>

<p>Once you have this setup, you can add <a href="variants">variants</a>.

Each variant acts as another layer of defence.</p>

<p>This is defence in depth.</p>