- General Notes
- Install
- Post Install Checks
- cronjob
- Example Outputs
- Testing
- Rate Limits
- White Listing
- Black Listing
- Bad Referrers
- blockbots.conf
- Customize Configurations
- globalblacklist.conf
- ngxtop
Installation commands for Mitchell Krog developed Ultimate Bad Bot Blocker are for Centmin Mod 123.09beta01 or higher LEMP stack on CentOS 6/7 specifically due to the differences in Centmin Mod's Nginx structure. If you have existing bad bot blocking & rate limiting setup, you will need to remove or comment out those include files with hash # in front of them first for
include /usr/local/nginx/conf/botlimit.conf;in/usr/local/nginx/conf/nginx.confinclude /usr/local/nginx/conf/blockbots.conf;within each of your Centmin Mod Nginx vhost config files within directory at/usr/local/nginx/conf/conf.d
Also update-ngxblocker script and cronjob doesn't seem to update itself from mitchellkrogza/nginx-ultimate-bad-bot-blocker#157 (comment) so for now need to manually update it using command and may need to setup a cronjob for that.
wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/update-ngxblocker -O /usr/local/sbin/update-ngxblocker && chmod 700 /usr/local/sbin/update-ngxblocker
Instructions below are provided as is with no support provided by me. For issues with false postives blocks etc, you will need to contact the official developer on their Ultimate Bad Bot Blocker issue tracker.
There's a mailing list and Slack Channel too:
Actual install commands for Nginx Ultimate Bad Bot Blocker installed at /usr/local/nginx/conf/ultimate-badbot-blocker and where the global bad bot blacklisting is contained in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf.
# download and install
wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -O /usr/local/sbin/install-ngxblocker
chmod +x /usr/local/sbin/install-ngxblocker
mkdir -p /usr/local/nginx/conf/ultimate-badbot-blocker
# backup nginx.conf and conf.d directory before install
cp -a /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf-backup-b4-badbot
cp -a /usr/local/nginx/conf/conf.d/ /usr/local/nginx/conf/conf.d-backup-b4-badbot
# dry run
install-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
# live run
install-ngxblocker -x -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
# fix duplicate directives
sed -i 's|^server_names_hash_|#server_names_hash_|g' /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
# dry run
setup-ngxblocker -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
# live run
setup-ngxblocker -x -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
I'll probably have to write up a script to check and notify me when the Ultimate Bad Bot Blocker installed scripts in /usr/local/sbin/ are modified from their initial installed versions just to keep track when cron updates change these files
/usr/local/sbin/install-ngxblocker/usr/local/sbin/setup-ngxblocker/usr/local/sbin/update-ngxblocker
Maybe via checking md5sum hashes
md5sum /usr/local/sbin/install-ngxblocker | tee install-ngxblocker.md5
938734ed5ae1001f90930f4b01fa51f8 /usr/local/sbin/install-ngxblocker
md5sum -c install-ngxblocker.md5
/usr/local/sbin/install-ngxblocker: OK
md5sum /usr/local/sbin/setup-ngxblocker | tee setup-ngxblocker.md5
c5cb4d154caeb956413dadf59072c7aa /usr/local/sbin/setup-ngxblocker
md5sum -c setup-ngxblocker.md5
/usr/local/sbin/setup-ngxblocker: OK
md5sum /usr/local/sbin/update-ngxblocker | tee update-ngxblocker.md5
47375490d11e99505931bc3b753ba338 /usr/local/sbin/update-ngxblocker
md5sum -c update-ngxblocker.md5
/usr/local/sbin/update-ngxblocker: OK
You can check nginx.conf config file at /usr/local/nginx/conf/nginx.conf against the backed up copy at /usr/local/nginx/conf/nginx.conf-backup-b4-badbot which you would of backed up in initial install commands above.
setup-ngxblocker doesn't setup /usr/local/nginx/conf/nginx.conf includes properly as it inserted include files in wrong place in centmin mod nginx.conf outside of http{] context
nginx -t
nginx: [emerg] "server_names_hash_bucket_size" directive is not allowed here in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf:15
nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed
sdiff side by side compare of backed up copy versus live copy of nginx.conf
sdiff -w 200 -s /usr/local/nginx/conf/nginx.conf-backup-b4-badbot /usr/local/nginx/conf/nginx.conf
> # Bad Bot Blocker
> include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;
> include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;
>
universal diff compare with 10 line context output of backed up copy versus live copy of nginx.conf
diff -U10 /usr/local/nginx/conf/nginx.conf-backup-b4-badbot /usr/local/nginx/conf/nginx.conf
--- /usr/local/nginx/conf/nginx.conf-backup-b4-badbot 2018-03-07 01:39:33.843255502 +0000
+++ /usr/local/nginx/conf/nginx.conf 2018-04-02 16:52:06.659257479 +0000
@@ -1,11 +1,15 @@
user nginx nginx;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;
+
worker_processes 2;
worker_priority -10;
worker_rlimit_nofile 260000;
timer_resolution 100ms;
pcre_jit on;
include /usr/local/nginx/conf/dynamic-modules.conf;
fix is to move the 3 lines within http{} context after existing variables_hash_max_size 2048; line
http {
map_hash_bucket_size 128;
map_hash_max_size 4096;
server_names_hash_bucket_size 128;
server_names_hash_max_size 2048;
variables_hash_max_size 2048;
# Bad Bot Blocker
include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;
include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;
then recheck and test nginx conf and you will get a new error for duplicate server_names_hash_bucket_size as /usr/local/nginx/conf/nginx.conf also lists this directive and it's being duplicated in thie bad bot installer's include file on line 15 of /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
nginx -t
nginx: [emerg] "server_names_hash_bucket_size" directive is duplicate in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf:15
nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed
commented instructions in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf already mention to comment out line 15 if it conflicts with your existing nginx.conf settings
cat -n /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
1 ##############################################################################
2 # _ __ _ #
3 # / |/ /__ _(_)__ __ __ #
4 # / / _ `/ / _ \\ \ / #
5 # /_/|_/\_, /_/_//_/_\_\ #
6 # __/___/ __ ___ __ ___ __ __ #
7 # / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #
8 # / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #
9 # /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #
10 # #
11 ##############################################################################
12
13 # Version 1.1
14
15 server_names_hash_bucket_size 128;
16 server_names_hash_max_size 4096;
17 limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
18 limit_conn_zone $binary_remote_addr zone=addr:50m;
19
20 # ****************************************************************************
21 # NOTE: IF you are using a system like Nginx-Proxy from @JWilder
22 # ****************************************************************************
23 # Repo URL: https://github.com/jwilder/nginx-proxy
24 # You will need to comment out the first line here as follows.
25 # #server_names_hash_bucket_size 128;
26 # You will also need to modify the nginx.tmpl file to add the default include
27 # include /etc/nginx/conf.d/*
28 # ****************************************************************************
So edit /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf line 15 to comment it out with a hash in front. Also comment out server_names_hash_max_size on line 16 as nginx.conf also already has that. Above instructions have been updated for sed edit and commenting out so you wouldn't need to manually do this anyway.
sed -i 's|^server_names_hash_|#server_names_hash_|g' /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
cat -n /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
1 ##############################################################################
2 # _ __ _ #
3 # / |/ /__ _(_)__ __ __ #
4 # / / _ `/ / _ \\ \ / #
5 # /_/|_/\_, /_/_//_/_\_\ #
6 # __/___/ __ ___ __ ___ __ __ #
7 # / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #
8 # / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #
9 # /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #
10 # #
11 ##############################################################################
12
13 # Version 1.1
14
15 #server_names_hash_bucket_size 128;
16 #server_names_hash_max_size 4096;
17 limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
18 limit_conn_zone $binary_remote_addr zone=addr:50m;
19
20 # ****************************************************************************
21 # NOTE: IF you are using a system like Nginx-Proxy from @JWilder
22 # ****************************************************************************
23 # Repo URL: https://github.com/jwilder/nginx-proxy
24 # You will need to comment out the first line here as follows.
25 # #server_names_hash_bucket_size 128;
26 # You will also need to modify the nginx.tmpl file to add the default include
27 # include /etc/nginx/conf.d/*
28 # ****************************************************************************
then edit /usr/local/nginx/conf/nginx.conf to raise the value of server_names_hash_max_size from 2048 to 4096
http {
map_hash_bucket_size 128;
map_hash_max_size 4096;
server_names_hash_bucket_size 128;
server_names_hash_max_size 4096;
variables_hash_max_size 2048;
recheck nginx config
nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
You can check vhosts' config files in /usr/local/nginx/conf/conf.d against the backed up copy of them at /usr/local/nginx/conf/conf.d-backup-b4-badbot which you would of backed up in initial install commands above.
diff -qr /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/demodomain.com.conf and /usr/local/nginx/conf/conf.d/demodomain.com.conf differ
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain1.com.conf and /usr/local/nginx/conf/conf.d/domain1.com.conf differ
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain2.com.conf and /usr/local/nginx/conf/conf.d/domain2.com.conf differ
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.conf and /usr/local/nginx/conf/conf.d/http2.domain1.com.conf differ
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.ssl.conf and /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf differ
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub1.domain1.com.conf and /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf differ
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub2.domain2.com.conf and /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf differ
Files /usr/local/nginx/conf/conf.d-backup-b4-badbot/virtual.conf and /usr/local/nginx/conf/conf.d/virtual.conf differ
for the most part centmin mod nginx generated vhosts had the include files added in correct place above the first location / context with exception of 2 vhost files where added include files weren't added directly above location / but were added further up in vhost config files below. Though technically it is still correct so nothing needed to correct.
/usr/local/nginx/conf/conf.d/demodomain.com.conf/usr/local/nginx/conf/conf.d/virtual.conf
diff compare
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d
diff compare output
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/demodomain.com.conf /usr/local/nginx/conf/conf.d/demodomain.com.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/demodomain.com.conf 2018-03-07 01:39:33.105393742 +0000
+++ /usr/local/nginx/conf/conf.d/demodomain.com.conf 2018-04-02 16:52:06.858264720 +0000
@@ -10,8 +10,12 @@
server {
listen 80;
server_name www.demodomain.com;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
# limit_conn limit_per_ip 16;
# ssi on;
access_log /home/nginx/domains/demodomain.com/log/access.log ;
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain1.com.conf /usr/local/nginx/conf/conf.d/domain1.com.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain1.com.conf 2018-04-02 16:20:00.996652290 +0000
+++ /usr/local/nginx/conf/conf.d/domain1.com.conf 2018-04-02 16:52:07.268279631 +0000
@@ -36,8 +36,12 @@
# server and/or vhost site
#include /usr/local/nginx/conf/cloudflare.conf;
include /usr/local/nginx/conf/503include-main.conf;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
location / {
include /usr/local/nginx/conf/503include-only.conf;
# block common exploits, sql injections etc
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain2.com.conf /usr/local/nginx/conf/conf.d/domain2.com.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain2.com.conf 2018-04-02 16:20:11.412014562 +0000
+++ /usr/local/nginx/conf/conf.d/domain2.com.conf 2018-04-02 16:52:07.466286830 +0000
@@ -36,8 +36,12 @@
# server and/or vhost site
#include /usr/local/nginx/conf/cloudflare.conf;
include /usr/local/nginx/conf/503include-main.conf;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
location / {
include /usr/local/nginx/conf/503include-only.conf;
# block common exploits, sql injections etc
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.conf /usr/local/nginx/conf/conf.d/http2.domain1.com.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.conf 2018-04-02 16:24:20.198637890 +0000
+++ /usr/local/nginx/conf/conf.d/http2.domain1.com.conf 2018-04-02 16:52:08.071308824 +0000
@@ -36,8 +36,12 @@
# server and/or vhost site
#include /usr/local/nginx/conf/cloudflare.conf;
include /usr/local/nginx/conf/503include-main.conf;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
location / {
include /usr/local/nginx/conf/503include-only.conf;
# block common exploits, sql injections etc
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.ssl.conf /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.ssl.conf 2018-04-02 16:24:20.205638109 +0000
+++ /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf 2018-04-02 16:52:08.285316598 +0000
@@ -65,8 +65,12 @@
# server and/or vhost site
#include /usr/local/nginx/conf/cloudflare.conf;
include /usr/local/nginx/conf/503include-main.conf;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
location / {
include /usr/local/nginx/conf/503include-only.conf;
# block common exploits, sql injections etc
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub1.domain1.com.conf /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub1.domain1.com.conf 2018-04-02 16:20:26.498539308 +0000
+++ /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf 2018-04-02 16:52:07.664294028 +0000
@@ -36,8 +36,12 @@
# server and/or vhost site
#include /usr/local/nginx/conf/cloudflare.conf;
include /usr/local/nginx/conf/503include-main.conf;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
location / {
include /usr/local/nginx/conf/503include-only.conf;
# block common exploits, sql injections etc
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub2.domain2.com.conf /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub2.domain2.com.conf 2018-04-02 16:20:39.443989582 +0000
+++ /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf 2018-04-02 16:52:07.870301518 +0000
@@ -36,8 +36,12 @@
# server and/or vhost site
#include /usr/local/nginx/conf/cloudflare.conf;
include /usr/local/nginx/conf/503include-main.conf;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
location / {
include /usr/local/nginx/conf/503include-only.conf;
# block common exploits, sql injections etc
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/virtual.conf /usr/local/nginx/conf/conf.d/virtual.conf
--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/virtual.conf 2018-03-07 01:39:33.106393555 +0000
+++ /usr/local/nginx/conf/conf.d/virtual.conf 2018-04-02 16:52:07.060272069 +0000
@@ -1,8 +1,12 @@
server {
listen 80 default_server backlog=2048 reuseport fastopen=256;
server_name centos7.localdomain;
root html;
+ # Bad Bot Blocker
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
+ include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
+
access_log /var/log/nginx/localhost.access.log combined buffer=8k flush=1m;
error_log /var/log/nginx/localhost.error.log error;
Once all fixed you can restart nginx server
service nginx restart
or centmin mod cmd shortcut
ngxrestart
setup cronjob via cronjob -e command to invoke nano text editor replacing [email protected] with your email address for update notifications. Note there's currently a bug with using emails with Gmail type +alias usernames I logged to their issue tracker mitchellkrogza/nginx-ultimate-bad-bot-blocker#157
00 */8 * * * /usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e [email protected]
you can do a manual update check too
without email notification add -n
/usr/local/sbin/update-ngxblocker -n -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
example without email notifictaion
/usr/local/sbin/update-ngxblocker -n -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
LOCAL Version: 3.2018.04.1080
Updated: Mon Apr 2 15:35:14 SAST 2018
REMOTE Version: 3.2018.04.1080
Updated: Mon Apr 2 15:35:14 SAST 2018
Latest Blacklist Already Installed: 3.2018.04.1080
with email notification
/usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e [email protected]
example with email notification
/usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e [email protected]
LOCAL Version: 3.2018.04.1080
Updated: Mon Apr 2 15:35:14 SAST 2018
REMOTE Version: 3.2018.04.1080
Updated: Mon Apr 2 15:35:14 SAST 2018
Latest Blacklist Already Installed: 3.2018.04.1080
Emailing report to: [email protected]
when updates are available
/usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e [email protected]
LOCAL Version: 3.2018.04.1080
Updated: Mon Apr 2 15:35:14 SAST 2018
REMOTE Version: 3.2018.04.1082
Updated: Tue Apr 3 09:40:52 SAST 2018
Update Available => 3.2018.04.1082
Downloading: globalblacklist.conf ...[OK]
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
Updating bots.d path: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d => /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
Reloading NGINX configuration...[OK]
Emailing report to: [email protected]
install-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
** Dry Run ** | not updating files | run as 'install-ngxblocker -x' to install files.
Creating directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/conf.d/globalblacklist.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
Downloading [FROM]=> [REPO]/conf.d/botblocker-nginx-settings.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/bots.d/blockbots.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf
Downloading [FROM]=> [REPO]/bots.d/ddos.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf
Downloading [FROM]=> [REPO]/bots.d/custom-bad-referrers.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf
Downloading [FROM]=> [REPO]/bots.d/bad-referrer-words.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf
Downloading [FROM]=> [REPO]/bots.d/blacklist-domains.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-domains.conf
Downloading [FROM]=> [REPO]/bots.d/blacklist-ips.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf
Downloading [FROM]=> [REPO]/bots.d/blacklist-user-agents.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-user-agents.conf
Downloading [FROM]=> [REPO]/bots.d/whitelist-domains.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf
Downloading [FROM]=> [REPO]/bots.d/whitelist-ips.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/setup-ngxblocker [TO]=> /usr/local/sbin/setup-ngxblocker
Downloading [FROM]=> [REPO]/update-ngxblocker [TO]=> /usr/local/sbin/update-ngxblocker
setup-ngxblocker -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
** Dry Run ** | not updating files | run as 'setup-ngxblocker -x' to setup files.
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf; => /usr/local/nginx/conf/nginx.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf; => /usr/local/nginx/conf/nginx.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/demodomain.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/demodomain.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/virtual.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/virtual.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf
Whitelisting ip: xxx.xxx.xxx.xxx => /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf
Web directory not found ('/var/www'): not whitelisting domains.
Checking for missing includes:
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
install-ngxblocker -x -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Creating directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/conf.d/globalblacklist.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf...OK
Downloading [FROM]=> [REPO]/conf.d/botblocker-nginx-settings.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf...OK
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/bots.d/blockbots.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf...OK
Downloading [FROM]=> [REPO]/bots.d/ddos.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf...OK
Downloading [FROM]=> [REPO]/bots.d/custom-bad-referrers.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf...OK
Downloading [FROM]=> [REPO]/bots.d/bad-referrer-words.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf...OK
Downloading [FROM]=> [REPO]/bots.d/blacklist-domains.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-domains.conf...OK
Downloading [FROM]=> [REPO]/bots.d/blacklist-ips.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf...OK
Downloading [FROM]=> [REPO]/bots.d/blacklist-user-agents.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-user-agents.conf...OK
Downloading [FROM]=> [REPO]/bots.d/whitelist-domains.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf...OK
Downloading [FROM]=> [REPO]/bots.d/whitelist-ips.conf [TO]=> /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf...OK
REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
Downloading [FROM]=> [REPO]/setup-ngxblocker [TO]=> /usr/local/sbin/setup-ngxblocker...OK
Downloading [FROM]=> [REPO]/update-ngxblocker [TO]=> /usr/local/sbin/update-ngxblocker...OK
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
setup-ngxblocker -x -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf; => /usr/local/nginx/conf/nginx.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf; => /usr/local/nginx/conf/nginx.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/demodomain.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/demodomain.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/virtual.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/virtual.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf
inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; => /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf
Whitelisting ip: xxx.xxx.xxx.xxx => /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf
Web directory not found ('/var/www'): not whitelisting domains.
Checking for missing includes:
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
ls -lah /usr/local/sbin/ | grep ngxblocker
-rwx------ 1 root root 9.5K Apr 2 16:17 install-ngxblocker
-rwx------ 1 root root 13K Apr 2 16:27 setup-ngxblocker
-rwx------ 1 root root 12K Apr 2 16:27 update-ngxblocker
ls -lah /usr/local/nginx/conf/ultimate-badbot-blocker
total 240K
drwxr-xr-x 3 root root 83 Apr 2 16:27 .
drwxr-xr-x. 8 root root 4.0K Apr 2 16:25 ..
-rw------- 1 root root 1.8K Apr 2 16:27 botblocker-nginx-settings.conf
drwxr-xr-x 2 root root 4.0K Apr 2 16:27 bots.d
-rw------- 1 root root 228K Apr 2 16:27 globalblacklist.conf
ls -lah /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/
total 44K
drwxr-xr-x 2 root root 4.0K Apr 2 16:27 .
drwxr-xr-x 3 root root 83 Apr 2 16:27 ..
-rw------- 1 root root 3.5K Apr 2 16:27 bad-referrer-words.conf
-rw------- 1 root root 2.4K Apr 2 16:27 blacklist-domains.conf
-rw------- 1 root root 7.0K Apr 2 16:27 blacklist-ips.conf
-rw------- 1 root root 3.3K Apr 2 16:27 blacklist-user-agents.conf
-rw------- 1 root root 2.1K Apr 2 16:27 blockbots.conf
-rw------- 1 root root 2.6K Apr 2 16:27 custom-bad-referrers.conf
-rw------- 1 root root 1.8K Apr 2 16:27 ddos.conf
-rw------- 1 root root 2.5K Apr 2 16:27 whitelist-domains.conf
-rw------- 1 root root 1.7K Apr 2 16:27 whitelist-ips.conf
Run the following commands one by one from a terminal on another linux machine against your own domain name. substitute http://domain1.com in the examples below with your REAL domain name. Should respond with HTTP 200 status OK code
curl -I -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://domain1.com
curl -I -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://domain1.com
HTTP/1.1 200 OK
Date: Mon, 02 Apr 2018 17:54:54 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 6040
Last-Modified: Mon, 02 Apr 2018 16:20:00 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5ac25830-1798"
Server: nginx centminmod
X-Powered-By: centminmod
Accept-Ranges: bytes
Should respond with: curl: (52) Empty reply from server for bad referrer domain 100dollars-seo.com which is listed in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
curl -I http://domain1.com -e http://100dollars-seo.com
curl -I http://domain1.com -e http://100dollars-seo.com
curl: (52) Empty reply from server
grep 100dollars /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
"~*\b100dollars\-seo\.com\b" 1;
Test the rate limiting of a bot with user agent = Baidu which is on the aggressive rate limiting profile of 2 requests/second specified in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
# limits for Zone $bad_bot = 2
# this rate limiting will only take effect if you change any of the bots and change
# their block value from 1 to 2.
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
Siege load test with user agent Baidu rate limited with a HTTP 503 status code served up to rated limited requests
siege -b -c10 -r100 -A "Baidu" http://domain1.com/
Lifting the server siege...
Transactions: 277 hits
Availability: 51.87 %
Elapsed time: 138.37 secs
Data transferred: 1.10 MB
Response time: 4.90 secs
Transaction rate: 2.00 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 9.80
Successful transactions: 277
Failed transactions: 257
Longest transaction: 5.01
Shortest transaction: 0.00
Rate limiting configuration settings are located in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf and /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf
limit_req_zoneis setup with zone namedfloodwith 90 requests/s rate limit with nodelay burst of 200 requests for normal rate limits set to1while there's a more aggressive rate limit of 2 requests per second for items set to2contained in/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conflimit_conn_zoneis setup with connection limit set in/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.confat 200 simultaneous IP connections
The global bad bot blacklisting is contained in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf. You can see the source master list at https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/conf.d/globalblacklist.conf.
Next to each entry is a number 0, 1, 2 or 3 which denote the following:
### Note that:
### 0 = allowed - no limits
### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
at end of /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
cat /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
# BAD BOT RATE LIMITING ZONE
# limits for Zone $bad_bot = 1
# Nothing Set - you can set a different zone limiter here if you like
# We issue a 444 response instead to all bad bots.
# limits for Zone $bad_bot = 2
# this rate limiting will only take effect if you change any of the bots and change
# their block value from 1 to 2.
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
cat /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
# / / _ `/ / _ \\ \ / #
# /_/|_/\_, /_/_//_/_\_\ #
# __/___/ __ ___ __ ___ __ __ #
# / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #
# / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #
# /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #
# #
##############################################################################
# Version 1.1
#server_names_hash_bucket_size 128;
#server_names_hash_max_size 4096;
limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
limit_conn_zone $binary_remote_addr zone=addr:50m;
# ****************************************************************************
# NOTE: IF you are using a system like Nginx-Proxy from @JWilder
# ****************************************************************************
# Repo URL: https://github.com/jwilder/nginx-proxy
# You will need to comment out the first line here as follows.
# #server_names_hash_bucket_size 128;
# You will also need to modify the nginx.tmpl file to add the default include
# include /etc/nginx/conf.d/*
# ****************************************************************************
cat /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf
#######################################################################
### VERSION INFORMATION #
###################################################
### Version: V3.2017.01
### Updated: Sun Jan 29 11:35:32 SAST 2017
###################################################
### VERSION INFORMATION ##
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
# / / _ `/ / _ \\ \ / #
# /_/|_/\_, /_/_//_/_\_\ #
# __/___/ __ ___ __ ___ __ __ #
# / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #
# / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #
# /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #
# #
##############################################################################
# Author: Mitchell Krog <[email protected]> - https://github.com/mitchellkrogza/
# Include this in a vhost file within a server {} block using and include statement like below
# server {
# #Config stuff here
# include /etc/nginx/bots.d/blockbots.conf
# include /etc/nginx/bots.d/ddos.conf
# #Other config stuff here
# }
#######################################################################
limit_conn addr 200;
limit_req zone=flood burst=200 nodelay;
Domain and IP whitelisting are set in respective include files at:
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf
Domain and IP blacking are set in respective include files at:
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-domains.conf/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf
custom bad referrers and bad referrer words are set in respective include files at:
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf
The bad bot blocking logic is located in /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf
# Author: Mitchell Krog <[email protected]> - https://github.com/mitchellkrogza/
### VERSION INFORMATION #
###################################################
### Version: V3.2017.02
### Updated: Mon Aug 21 11:29:32 SAST 2017
###################################################
### VERSION INFORMATION ##
##############################################################################
# _ __ _ #
# / |/ /__ _(_)__ __ __ #
# / / _ `/ / _ \\ \ / #
# /_/|_/\_, /_/_//_/_\_\ #
# __/___/ __ ___ __ ___ __ __ #
# / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #
# / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #
# /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #
# #
##############################################################################
# Include this in a vhost file within a server {} block using and include statement like below
# server {
# #Config stuff here
# include /etc/nginx/bots.d/blockbots.conf
# include /etc/nginx/bots.d/ddos.conf
# #Other config stuff here
# }
#######################################################################
# BOTS
# ****
#limit_conn bot1_connlimit 100;
limit_conn bot2_connlimit 10;
#limit_req zone=bot1_reqlimitip burst=50;
limit_req zone=bot2_reqlimitip burst=10;
if ($bad_bot = '3') {
return 444;
}
# BAD REFER WORDS
# ***************
if ($bad_words) {
return 444;
}
# REFERERS
# ********
if ($bad_referer) {
return 444;
}
# IP BLOCKS
# *********
if ($validate_client) {
return 444;
}
#######################################################################
You can now customize any of the following files below to suit your environment or requirements. These include files never get modified during an update using the auto update script /usr/local/sbin/update-ngxblocker outlined in above cronjob section so whatever customizations you do here will never be overwritten during an update.
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-user-agents.conf
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf
/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf
Nginx Ultimate Bad Bot Blocker is installed at /usr/local/nginx/conf/ultimate-badbot-blocker and where the global bad bot blacklisting is contained in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf. You can see the source master list at https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/conf.d/globalblacklist.conf.
Example of Good bots allowed
# ***********************************************
# Allow Good User-Agent Strings We Know and Trust
# ***********************************************
# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*\badidxbot\b" 0;
"~*\bAdsBot-Google\b" 0;
"~*\baolbuild\b" 0;
"~*\bbingbot\b" 0;
"~*\bbingpreview\b" 0;
"~*\bDoCoMo\b" 0;
"~*\bduckduckgo\b" 0;
"~*\bfacebookexternalhit\b" 0;
"~*\bFeedfetcher-Google\b" 0;
"~*\bGooglebot\b" 0;
"~*\bGooglebot-Image\b" 0;
"~*\bGooglebot-Mobile\b" 0;
"~*\bGooglebot-News\b" 0;
"~*\bGooglebot/Test\b" 0;
"~*\bGooglebot-Video\b" 0;
"~*\bGoogle-HTTP-Java-Client\b" 0;
"~*\bGravityscan\b" 0;
"~*\bgsa-crawler\b" 0;
"~*\bJakarta\ Commons\b" 0;
"~*\bKraken/0.1\b" 0;
"~*\bLinkedInBot\b" 0;
"~*\bMediapartners-Google\b" 0;
"~*\bmsnbot\b" 0;
"~*\bmsnbot-media\b" 0;
"~*\bSAMSUNG\b" 0;
"~*\bSlackbot\b" 0;
"~*\bSlackbot-LinkExpanding\b" 0;
"~*\bslurp\b" 0;
"~*\bteoma\b" 0;
"~*\bTwitterBot\b" 0;
"~*\bWordpress\b" 0;
"~*\byahoo\b" 0;
# END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###
User Agent strings allowed but rated limited
# ***************************************************
# User-Agent Strings Allowed Through but Rate Limited
# ***************************************************
# Some people block libwww-perl, it used widely in many valid (non rogue) agents
# I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited
# START ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*\bjetmon\b" 1;
"~*\blibwww-perl\b" 1;
"~*\bLynx\b" 1;
"~*\bmunin\b" 1;
"~*\bPresto\b" 1;
"~*\bWget/1.15\b" 1;
# END ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
User Agent strings allowed but more aggressive/restrictive rate limiting
# **************************************************************
# Rate Limited User-Agents who get a bit aggressive on bandwidth
# **************************************************************
# START LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
"~*\bAlexa\b" 2;
"~*\barchive.org\b" 2;
"~*\bBaidu\b" 2;
"~*\bBUbiNG\b" 2;
"~*\bFlipboardProxy\b" 2;
"~*\bia_archiver\b" 2;
"~*\bMSIE\ 7.0\b" 2;
"~*\bProximic\b" 2;
"~*\bR6_CommentReader\b" 2;
"~*\bR6_FeedFetcher\b" 2;
"~*\bRED/1\b" 2;
"~*\bRPT-HTTPClient\b" 2;
"~*\bsfFeedReader/0.9\b" 2;
"~*\bSpaidu\b" 2;
"~*\bUptimeRobot/2.0\b" 2;
"~*\bYandexBot\b" 2;
"~*\bYandexImages\b" 2;
# END LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###
bad bots listed in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf section under title:
# *********************************************
# Bad User-Agent Strings That We Block Outright
# *********************************************
You can use ngxtop to analyse your Centmin Mod Nginx access logs and keep an eye on Nginx HTTP 444 status codes and also other user agent strings, HTTP status codes etc as outlined here.
CentOS 7 ngxtop install
yum -y install python-pip
pip install --upgrade pip
pip install ngxtop
Check for HTTP 444 status codes /home/nginx/domains/domain1.com/log/access.log access log
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
Only one entry for 444 exists due to test domain1.com test command curl -I http://domain1.com -e http://100dollars-seo.com only ran once
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
running for 0 seconds, 1 records processed: 2434.30 req/sec
Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 1 | 0.000 | 0 | 0 | 1 | 0 |
Detailed:
| request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| / | 1 | 0.000 | 0 | 0 | 1 | 0 |
alternatively you can use native -i 'status == 444' flag
cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'
cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'
running for 0 seconds, 1 records processed: 369.97 req/sec
Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 1 | 0.000 | 0 | 0 | 1 | 0 |
Detailed:
| request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| / | 1 | 0.000 | 0 | 0 | 1 | 0 |
print request url, HTTP Status code and http user agent
cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent
cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent
running for 0 seconds, 1 records processed: 335.92 req/sec
request, status, http_user_agent:
| request | status | http_user_agent |
|-----------------+----------+-------------------|
| HEAD / HTTP/1.1 | 444 | curl/7.29.0 |
group by remote IP address via --group-by remote_addr
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr
output where 192.168.0.1 is the visitor ip address
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr
running for 0 seconds, 1 records processed: 1597.22 req/sec
Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 1 | 0.000 | 0 | 0 | 1 | 0 |
Detailed:
| remote_addr | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------------+---------+------------------+-------+-------+-------+-------|
| 192.168.0.1 | 1 | 0.000 | 0 | 0 | 1 | 0 |
group by user agent string via --group-by http_user_agent
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent
the test command was run via curl hence curl user agent string
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent
running for 0 seconds, 1 records processed: 1402.31 req/sec
Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 1 | 0.000 | 0 | 0 | 1 | 0 |
Detailed:
| http_user_agent | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|-------------------+---------+------------------+-------+-------+-------+-------|
| curl/7.29.0 | 1 | 0.000 | 0 | 0 | 1 | 0 |
filter by specific date using grep i.e. April 2nd would filter by '02/Apr'
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent
grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent
running for 0 seconds, 1 records processed: 1742.54 req/sec
Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 1 | 0.000 | 0 | 0 | 1 | 0 |
Detailed:
| http_user_agent | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|-------------------+---------+------------------+-------+-------+-------+-------|
| curl/7.29.0 | 1 | 0.000 | 0 | 0 | 1 | 0 |
With Baidu user agent string rate limited results for HTTP 503 status codes in acces log
cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
running for 1 seconds, 6081 records processed: 7868.07 req/sec
Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 6081 | 2060.460 | 373 | 0 | 81 | 5627 |
Detailed:
| request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| / | 6081 | 2060.460 | 373 | 0 | 81 | 5627 |
cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent
running for 1 seconds, 6153 records processed: 7625.33 req/sec
Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 6153 | 2056.997 | 432 | 0 | 87 | 5634 |
Detailed:
| http_user_agent | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|-------------------+---------+------------------+-------+-------+-------+-------|
| Baidu | 6151 | 2057.666 | 431 | 0 | 86 | 5634 |
| curl/7.29.0 | 2 | 0.000 | 1 | 0 | 1 | 0 |