Analyzing Apache access logs to quickly identify abuse

Question

So someone in China just loves abusing the crap out of my networking tools site up at yougetsignal.com.

I've been trying to analyze my access.log files to see exactly what they are so excited about. Most of the services on my site are rate limited using MySQL calls to record a history of usage, and for the most part that has been a great way to curb excessive use. Every once and awhile they will use a ton of IP addresses to hit the service up to avoid the IP rate limit.

I've been playing around with these two commands to analyze my Apache access.log file:

sudo cat access.log | awk '{print $1}' | sort | uniq -c |sort -n

// Output sample:
// 3453 83.251.250.130
// 3888 79.136.28.38
// 4268 94.75.220.77
// 7116 61.147.122.183

and

sudo tail -50000 access.log | awk '{print $1}' | sort | uniq -c |sort -n

// Output sample:
// 276 66.249.68.86
// 365 155.212.251.138
// 1093 61.147.122.183

and lastly:

awk '{print $7}' access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10

// Output sample:
// 15405 heavy-use-file.php
// 22339 heavy-use-file.php
// 218121 very-heavy-use-file.php

The last thing I need is something that will tie the access counts, IP addresses, and files all together. Is it possible to write a shell command that will show me the IP addresses that have made the most requests, and specifically which file they have been requesting descending by count? I'm not a Linux master, so I'm reaching out for help here.

Solution 1

How about :

awk '{print $1,$7}' access.log | cut -d? -f1 | sort | uniq -c |sort -nr

Also have you had a look at AWStats ? It's quite awesome to analyse all data from your log files

Solution 2

apachetop is your friend. It provides you a view similar to top command, but for Apache access log. It is possible to sort the view by hits, kbytes and so on.

You can use it in real time or feed an older log to it.