How to Block 100,000+ Individual IP addresses

Question

Introduction

How do you Block large number of IP address from your web application/server. Obviously that can easily be done in PHP or any programming language

$ipList = []; // array list or from database
if (in_array(getIP(), $ipList)) {
    // Log IP & Access information
    header("https://www.google.com.ng/search?q=fool"); // redirect
    exit(); // exit
} 

Or Using htaccess

order allow,deny
deny from 123.45.6.7
deny from 012.34.5.
# .... the list continues
allow from all

The issues

• Am trying to block a whole 100k plus individual IPs not subnets
• Am trying to avoid user getting to PHP before blocking such IP
• 100000+ is over 1.5MB and that is a lot if information to be loading in htaccess all the time
• Database of IP still growing ... and they would be nee to dynamically add more values
• To set bans in iptables for 100000+ is just ridiculous (Might Be wrong)

Stupid Idea

order allow,deny
deny from database    <-------- Not sure if this is possible
allow from all

Question

• Is it possible for htaccess to get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any
• Is there a visible solution to manage such kind of issue in production
• I know the best solution is Block the IPs at the firewall level is there any way to pragmatically add/remove IP to the firewall

Finally

My approach might be totally wrong ... all I want is a visible solution since spammers and botnets are on the rise ...

Please this has nothing to do with DOS attack its a simple ... get lost response

Update

• Firewall : Cisco PIX 515UR

Solution 1

Something that you can try is keeping a list of the IP addresses you want to block in a text file or convert it to a dbm hash file, then use mod_rewrite's RewriteMap. You'd have to set this up in your server/vhost config. You cannot initialize a map in an htaccess file.

RewriteEngine On
RewriteMap deny_ips txt:/path/to/deny_ips.txt

RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0
RewriteRule ^ - [L,F]

The /path/to/deny_ips.txt file would look something like this:

12.34.56.78 1
11.22.33.44 1
etc.

Essentially, an IP that you want to deny and a space then a "1". Any IP in this text file will cause the server to return a 403 Forbidden. To speed things up a bit you can use the httxt2dbm to generate a dbm hash and then you'd define the mapping as so:

RewriteMap deny_ips dbm:/path/to/deny_ips.dbm

I'm not sure what the performance hit is for using mod_rewrite like this with a lot of IPs, but a quick benchmark test on apache 2.2 running on a 3Ghz i686 under linux, the difference between 5 IPs in the list versus 102418 is negligible. According to ab's output, they're nearly identical.

---

Addressing specific questions:

Is it possible for htaccess to get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any

Using a rewrite map, you can use the "prg" map type to run an external program for a mapping type. You can then write a perl, php, etc. script to talk to a database in order to look up an IP address. Also note that caveats listed under "Caution". You'd then use this map like you would any other map (RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0). This would essentially create a bottleneck for all requests. Not the best solution for talking to a database.

In apache 2.4 though, there is a dbd/fastdbd map type, which allows you to create queries through mod_dbd. This is a much better option and the mod_dbd module manages connections to the database, pools connections, etc. So the map definition would look something like:

RewriteMap deny_ips "fastdbd:SELECT active FROM deny_ips WHERE source = %s"

Assuming you have a table "deny_ips" with 2 columns "source" (the IP address) and "active" (1 for active, 0 for inactive).

Is there a visible solution to manage such kind of issue in production

If you are storing all of the blocked IPs in the database, it's a matter of managing the contents of your database table. If you are using the dbm map type, I know at least perl has a DBI for managing dbm files, so you can use that to add/remove IP entries from the deny list. I've never used it before so I can't really say much about it. Managing a flat text file is going to be a lot trickier, especially if you plan on removing entries, and not just append to it. Outside of using a database and apache 2.4's mod_dbd, I don't think any of these solutions are out of the box or production ready. It's going to require custom work.

I know the best solution is Block the IPs at the firewall level is there any way to pragmatically add/remove IP to the firewall

For IPtables, there is a perl interface that's marked as Beta, but I've never used it before. There's libiptc but according to netfilter's faq:

Is there an C/C++ API for adding/removing rules?

The answer unfortunately is: No.

Now you might think 'but what about libiptc?'. As has been pointed out numerous times on the mailinglist(s), libiptc was NEVER meant to be used as a public interface. We don't guarantee a stable interface, and it is planned to remove it in the next incarnation of linux packet filtering. libiptc is way too low-layer to be used reasonably anyway.

We are well aware that there is a fundamental lack for such an API, and we are working on improving that situation. Until then, it is recommended to either use system() or open a pipe into stdin of iptables-restore. The latter will give you a way better performance.

So I don't know how viable a libiptc solution is if there's no API stability.

Solution 2

ANOTHER PERSPECTIVE

Hello. You can check if an address is blocked or not, via accessing two bytes in two data chunks each 8KB long. Yes, I am serious... Please be patient because it takes a little bit long to explain it.

THE THEORY

An IP address is an address, actually a 4 byte number.

The question is, what if we make it to address bit positions?.

The answer: Well ok, we will have

  2^32 = 4 Giga Bits 

of addressing space and that will take

 4Gb/8 = 512 Mega Bytes

of allocation. Ouch! But do not worry, we are not going to block everything in the ipverse and 512MB is an exaggeration.

This can open us a path to the solution.

The Lilliputian Case

Think of a Lilliputian world which there exists only ip addresses from 0 to 65535. So addresses are like 0.1 or 42.42 up to 255.255.

Now King of this world wants to block several L-IP (lilliput ip) addresses.

First he builds a virtual 2D bit map which is 256 * 256 bits long that takes up :

 64 K Bits = 8 K Bytes.

He decides to block that nasty "revolution" site which he hates because he is the king, the address is 56.28 for instance.

Address     = (56 * 256) + 28  = 14364.(bit position in whole map)
Byte in map = floor(14364 / 8) =  1795.
Bit position= 14364 % 8        =     4.(modulus)

He opens the map file, accesses 1795th byte and sets the bit 4 (by an | 16), then writes it back to mark the site as blocked.

When his script sees the 56.28, it does the same calculation and looks at the bit, and if it is set, blocks the address.

Now what is the moral of the story? Well we can use this lilliputian structure.

THE PRACTICE

The Real World Case

We can apply the Lilliputian case to real world with a "use it when you need" approach since allocating a 512MB file is not a good choice.

Think of a database table named BLOCKS with entries like that:

IpHead(key): unsigned 16 bit integer,
Map        : 8KB BLOB(fixed size),
EntryCount : unsigned 16 bit integer.

And another table with just one entry with the structure below named BASE

Map        : 8KB BLOB(fixed size).

Now lets say you have an incoming address 56.28.10.2

Script accesses BASE table and gets the Map.

It looks up the higher order IP numbers 56.28:

Address     = (56 * 256) + 28  = 14364.(bit position in whole map)
Byte in map = floor(14364 / 8) =  1795.
Bit position= 14364 % 8        =     4.(modulus)

Looks at byte 1795 bit 4 in the Map.

If bit is not set no further operation is needed meaning there is no blocked ip address in range 56.28.0.0 - 56.28.255.255 .

If bit is set then the script accesses the BLOCKS table.

The higher order IP numbers were 56.28 which gives 14364 so the script queries the BLOCKS table with index IpHead = 14364. Fetches the record. The record should exist since it is marked at BASE.

Script does the calculation for lower order IP address

Address     = (10 * 256) + 2   = 2562.(bit position in whole map)
Byte in map = floor(2562 / 8) =   320.
Bit position= 2562 % 8        =     2.(modulus)

Then it checks if address is blocked by looking at bit 2 of byte 320 of field Map.

Job done!

Q1: Why do we use BASE at all, we could directly query BLOCKS with 14364.

A1: Yes we could but BASE map lookup will be faster then BTREE search of any database server.

Q2: What is the EntryCount field in BLOCKS table for?

A2: It is the count of ip addresses blocked in the map field at the same record. So if we unblock ip's and EntryCount reaches 0 that BLOCKS record becomes unnecessary. It can be erased and the corresponding bit on BASE map will be unset.

IMHO this approach will be lightning fast. Also for the blob allocation is 8K per record. Since db servers keep blobs in seperate files, file systems with 4K, 8K or multiples of 4K paging will react fast.

In case blocked addresses are too dispersed

Well that is a concern, which will make the database BLOCKS table to grow unnecessarily.

But for such cases the alternative is to use a 256*256*256 bit cube which is 16777216 bits long, equaling to 2097152 bytes = 2MB.

For our previous example Higher Ip resolving is :

(56 * 65536)+(28 * 256)+10      

So BASE will become a 2MB file instead of a db table record, which will be opened (fopen etc.) and bit will be addressed via seeking (like fseek, never read whole file contents, unnecessary) then access the BLOCKS table with structure below:

IpHead(key): unsigned 32 bit integer, (only 24 bit is used)
Map        : 32 unsigned 8 bit integers(char maybe),(256 bit fixed)
EntryCount : unsigned 8 bit integer. 

Here is the php example code for block checking of bitplane-bitplane (8K 8K) version:

Side Note: This script can be optimized further via elimination of several calls etc.. But written like this for keeping it easy to understand.

<?
define('BLOCK_ON_ERROR', true); // WARNING if true errors block everyone

$shost = 'hosturl';
$suser = 'username';
$spass = 'password';
$sdbip = 'database';
$slink = null;

$slink = mysqli_connect($shost, $suser, $spass, $sdbip);
if (! $slink) {
    $blocked = BLOCK_ON_ERROR;
} else {
    $blocked = isBlocked();
    mysqli_close($slink); // clean, tidy...
}

if ($blocked) {
    // do what ever you want when blocked
} else {
    // do what ever you want when not blocked
}
exit(0);

function getUserIp() {
    $st = array(
            'HTTP_CLIENT_IP',
            'REMOTE_ADDR',
            'HTTP_X_FORWARDED_FOR'
    );
    foreach ( $st as $v )
        if (! empty($_SERVER[$v]))
            return ($_SERVER[$v]);
    return ("");
}

function ipToArray($ip) {
    $ip = explode('.', $ip);
    foreach ( $ip as $k => $v )
        $ip[$k] = intval($v);
    return ($ip);
}

function calculateBitPos($IpH, $IpL) {
    $BitAdr = ($IpH * 256) + $IpL;
    $BytAdr = floor($BitAdr / 8);
    $BitOfs = $BitAdr % 8;
    $BitMask = 1;
    $BitMask = $BitMask << $BitOfs;
    return (array(
            'bytePos' => $BytAdr,
            'bitMask' => $BitMask
    ));
}

function getBaseMap($link) {
    $q = 'SELECT * FROM BASE WHERE id = 0';
    $r = mysqli_query($link, $q);
    if (! $r)
        return (null);
    $m = mysqli_fetch_assoc($r);
    mysqli_free_result($r);
    return ($m['map']);
}

function getBlocksMap($link, $IpHead) {
    $q = "SELECT * FROM BLOCKS WHERE IpHead = $IpHead";
    $r = mysqli_query($link, $q);
    if (! $r)
        return (null);
    $m = mysqli_fetch_assoc($r);
    mysqli_free_result($r);
    return ($m['map']);
}

function isBlocked() {
    global $slink;
    $ip = getUserIp();
    if($ip == "")
        return (BLOCK_ON_ERROR);
    $ip = ipToArray($ip);

    // here you can embed preliminary checks like ip[0] = 10 exit(0)
    // for unblocking or blocking address range 10 or 192 or 127 etc....

    // Look at base table base record.
    // map is a php string, which in fact is a good byte array
    $map = getBaseMap($slink); 
    if (! $map)
        return (BLOCK_ON_ERROR);
    $p = calculateBitPos($ip[0], $ip[1]);
    $c = ord($map[$p['bytePos']]);
    if (($c & $p['bitMask']) == 0)
        return (false); // No address blocked

    // Look at blocks table related record
    $map = getBlocksMap($slink, $p[0]);
    if (! $map)
        return (BLOCK_ON_ERROR);
    $p = calculateBitPos($ip[2], $ip[3]);
    $c = ord($map[$p['bytePos']]);
    return (($c & $p['bitMask']) != 0);
}

?> 

I hope this helps.

If you have questions on the details, I will be happy to answer.

Solution 3

Block the traffic before it reaches the www server using iptables and ipset.

Catch the blacklisted IP traffic in the filter table of the INPUT chain assuming your web server is on the same machine. If you are blocking IPs on a router you will want the FORWARD chain.

First create the ipset:

ipset create ip_blacklist hash:ip

IPs can be added via:

ipset add ip_blacklist xxx.xxx.xxx.xxx

Add the ipset match rule to your iptables (DROP all packets match to ipset):

iptables --table filter --insert INPUT --match set --match-set ip_blacklist src -j DROP

This will stop the blacklisted traffic before the www server.

Edit: I had a chance to look up the default maximum size and it is 65536 so you will need to adjust this to support 100000+ entries:

ipset create ip_blacklist hash:ip maxelem 120000

You can also tweak the hash size:

ipset create ip_blacklist hash:ip maxelem 120000 hashsize 16384 (Must be a power of 2)

My experience is ipset lookup has negligible effect on my system (~45000 entries). There are a number of test cases on the net. Memory for the set is a limiting factor.

Solution 4

You need to do this with an external firewall, not in PHP. I recommend pfSense or PF. I have used it before and it is very easy to use, very intuitive, and extremely powerful. It is the choice of the best sys-admins. I run it on FreeBSD, but it works great on OpenBSD as well. I am a Linux guy so it pains me to say this, but don't try to run it on Linux. BSD is easy, and you can figure it out quickly.

An awesome feature for pfSense is the ability to configure using scripts and restricting the configuration access to a single network interface (so that only things on the LAN can configure it). It also has a couple of ID10T level features to keep you from cutting off your own access accidentally.

You should also be aware that many spammers can switch IPs quickly using things like Tor. To fix this you should include in your block list the addresses that are known tor exit nodes (this list is available from various places).

Solution 5

If you want a way to add / remove via code take a look at denyhosts. You could either maintain the IP list via code or patch the source to read from whatever location you want.

View more solutions

Comments

[Baba]

I already said I know the best solution is Block the IPs at the firewall level is there any way to pragmatically add/remove IP to the firewall .....

[2to1mux]

Well, I just gave you a suggestion for doing so. Write a simple bash script that queries your mysql database and adds IP addresses to the firewall blacklist. You could also, as @Populus suggested, utilize system calls via PHP to do this if you would rather handle the logic of adding IP addresses in PHP.

[Baba]

+nice one ... never used htaccess with file .. .. any performance issues ?

[Jon Lin]

@Baba The map needs to be defined in the vhost/server config, will cause error in htaccess. The rule itself can be placed in an htaccess file though, and obviously, it's a bit slower than in vhost/server config, but the file is short (doesn't contain the 100K IPs), so there shouldn't be any additional performance issues.

[TerryE]

@JonLin, yup the rule could be moved to a .htaccess file, but as you point out the map needs to be defined in the vhost/server config, so what's the point of the performance hit of moving the rule into an .htaccess file?

[Jon Lin]

@TerryE There's no point taking the hit, except maybe for lazy convenience. I only brought it up because Baba did in his comment.

[Alexander]

Unfortunately, this targets specifically SSH

[Alexander]

@JonLin, and how to handle updates? I am failing to see a convenient method to update the file without leading to locking/concurrency/etcetera issues

[Jon Lin]

@Alexander A thread safe solution to append to a file will depend on the OS. On linux, since the IP + space + "1" is smaller than {PIPE_BUF}, a single echo "$IP 1" >> /path/to/deny_ips.txt should be thread safe (See: linux.die.net/man/3/write). For windows, I don't know.

[Baba]

Are you sure you really want to block all Kung Fu fans ?

[Jason]

I took a different approach personally that yielded success for (looks at watch) about 4 months, but it's breaking down and they're getting past it now. I'm tempted to nuke access from orbit; it's the only way to be sure.

[Chris Wesseling]

-1, the Chinese have a hard time accessing a neutral web, as it is. Don't make it harder on them by blocking based on where their requests come from instead on their merits.

[Baba]

the perl interface is just too old ... any idea on a PHP interface ?

[Jon Lin]

@Baba You mean for iptables? There isn't one. The best you can do with that without writing something custom is to directly call the iptables command to add/remove rules, and maybe setup the sudoers so that the user running php can sudo iptables without a password.

[Volkan]

For Management: Adding/Removing IP's =>A backend PHP script will suffice. To check incoming ip's a:php script, b: .htaccess invoking some magic thing (again maybe php) that I am not familiar. Generally speaking, implementing this is a childs play, Integrating it to a firewall or htaccess is a mystery for me. I can add php examples to my answer if you want.

[Volkan]

I added a blocking script for 8K 8K bitplane mode.

[Jason]

When the site in question is a store that isn't likely to ship to them anyway, their access concerns are secondary to our ability to functionally use our own site's communication features.

[rkosegi]

@ethrbunny : to be precise, any protocol wrapper by tcp_wrappers.

[Moak]

Or you could automatically prepend the file php.net/manual/en/ini.core.php#ini.auto-prepend-file

[Baba]

include $_GET['r']; without is a bad idea ? So many security issues with that ...

[ncm]

thanks @Moak. yes you can but with this solution you can more than appending or prepending.I mean for example you can filter the file before run or you can include another file or redirect user to another page and you can do what you can do with php.(-:

[ncm]

thanks @Baba sure tou are right.but if you have mentioned to what I had said after that."its so simple.and its real one is so complicated.but the main idea is the same. what do you think?". and As I said in previous comment you can do so many things As maybe all frameworks do that (especially MVCframeworks) like routing IP filtering and ... .(-:

[Baba]

Would they be any locking issue writing to /path/to/deny_ips.txt from php ? I hope no restart would be required for each added IP address at real time

[Jon Lin]

@Baba According to the RewriteMap docs: The looked-up keys are cached by httpd until the mtime (modified time) of the mapfile changes, or the httpd server is restarted. This ensures better performance on maps that are called by many requests. Youwon't need to restart.

[Mr. Alien]

@Ihsan Where the heck you are? No more PHP room? ping me so I will delete this comment

[hek2mgl]

Using Apache to block IP's is totally bollocks. Why you want to unwrap the packet to HTTP level when you only need IP information for blocking? Now blocked users can stress your CPU and network stack. Good work! Note: Blocking IPs is done at IP level, not on HTTP level

[Camilo Martin]

This is a cute solution, but we're reaching the point where somewhere out there some people may use IPv6.

[Admin]

This is very pro way of doing. I would prefer to use this ipset concept