Client denied by server configuration on Apache 2.4

Question

We have migrated multiple Drupal 7 / 8 sites to a new stack, where the main changes were replacing Apache 2.2 and FastCGI with Apache 2.4 and PHP-FPM.

We have the following error on multiple sites:

[Fri Oct 19 09:06:26.333135 2018] [:error] [pid 6415:tid 140550690748160] [client 93.xxx.xxx.xxx:0] client denied by server configuration: /var/www/html/example.com/js, referer: https://www.example.com/some-page

The /js path is the coming from the JS Drupal module, but it occurs on other paths defined by our own custom Drupal routes (hook_menu on D7).

This is the vhost file:

<VirtualHost *:80>
 ServerName example.com
 ServerAlias www.example.com
 ServerAdmin [email protected]
 UseCanonicalName Off

 DocumentRoot /var/www/html/example.com

 ErrorLog /var/www/logs/example.com.error.log
 LogLevel warn
 CustomLog /var/www/logs/example.com.log combined
 <Directory /var/www/html/example.com>
    Options -Indexes +FollowSymLinks +ExecCGI
    AllowOverride All
    Require all granted
  </Directory>

</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost *:443>
 Protocols h2 http/1.1
 ServerName example.com
 ServerAlias www.example.com
 ServerAdmin [email protected]
 UseCanonicalName Off

 DocumentRoot /var/www/html/example.com

 ErrorLog /var/www/logs/example.com.error.log
 LogLevel warn
 CustomLog /var/www/logs/example.com.log combined
 <Directory /var/www/html/example.com>
    Options -Indexes +FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>

SSLCertificateFile /etc/letsencrypt/live/www.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

I tried to grep /etc/ and /var/www/html for Order and Allow / Deny (Apache 2.2 old syntax), but I couldn't find anything important, only one thing which comes from Apache's default configuration files and it is in an if statement that won't run on our case)

We also added Options -MultiViews to our Drupal .htaccess to fix another issue with Apache 2.4, not sure if it is relevant.

Note that the error only appears from time to time and not always, which makes it much harder to debug.

Any help will be appreciated.

Update

We are using mod_mpm_event if it is somehow relevant.

Apache's php.conf file:

AddType text/html .php

DirectoryIndex index.php

<IfModule  mod_php5.c>
    <Proxy "unix:/var/run/php-fpm/default.sock|fcgi://php-fpm">
        ProxySet disablereuse=off
    </Proxy>
    <FilesMatch \.php$>
        SetHandler proxy:fcgi://php-fpm
    </FilesMatch>
</IfModule>

.htaccess file - we are using the regular Drupal 7 .htaccess file with the following modifications: JS module rewrite rules, above any other rewrite rules (line 62)

RewriteCond %{REQUEST_URI} ^\/([a-z]{2}\/)?js\/.*
RewriteRule ^(.*)$ js.php?q=$1 [L,QSA]
RewriteCond %{QUERY_STRING} (^|&)q=((\/)?[a-z]{2})?(\/)?js\/.*
RewriteRule .* js.php [L]

In addition to that, we have added Options -MultiViews as already mentioned on the original question.

I don't think that the JS module and its redirects is the issue, since we also have problems with other custom Drupal menu paths which are handled by the core and the default .htaccess file.

Maybe the issue is something with the php-fpm Apache's handler?

Solution

Finally, I managed to solve the problem.

mod_evasive was blocking some of the requests for some reason, although we are using the same configuration as we were using with Apache 2.2, it seems that it is handling X-Forwarded-For differently than Apache 2.2 with mod_extract_forwarder does, and therefore detects some of our async requests as DDOS.

To verify that it was the issue, I used ab -n 100 -c 5 -p payload.txt -T 'application/x-www-form-urlencoded' https://www.example.com/js/mycallback and immediately saw errors on our error log, and once disabling mod_evasive those stopped.

We ended up disabling mod_evasive entirely (we have WAF with DDOS protection in front of our applications so this isn't that important to us anyway).

Comments

[bgtvfr]

Are there some .htaccess files in /var/www/html/example.com/js ?

[rosa.tarraga]

/js isn't a directory it is a js.php file.

[bgtvfr]

So there must exists some kind of rewrite rule to redirect example.com/js to /var/www/html/example.com/js.php content?

[Oldskool]

Please share your .htaccess file(s) as well.

[rosa.tarraga]

@Oldskool I have added the .htaccess and more data to the original question. Thanks.

[rosa.tarraga]

The above is defined on a global Apache configuration file (php.conf if I am not wrong), I'll attach those lines as well

[Fanfurlio]

It should be ok if it is defined in a global conf file, so that may not be the cause. I found helpful to check both access_log and error_log to debug those "client denied" errors, you may have an error in your .htaccess or a global mod_rewrite configuration.

[rosa.tarraga]

I have added more data to the original question including Apache's php-fpm handler. Can you please have a look?