strange 401 error appears for some urls when using .htaccess to redirect http to https

Question

OK, here is the 7th day of unsuccessfull attempt to find an answer why 401 error appears...

Now, .htaccess in the root folder contains the only 3 strings (was simplified) and there are NO more .htaccess files in the project:

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

So, it redirects all requests to be https. It works fine for any urls, even for /administration directory.

So,

http://mydomain.com

becomes

https://mydomain.com

If https://mydomain.com was entered, there are no redirections.

http://mydomain.com/administration/index.php

becomes

https://mydomain.com/administration/index.php

If https://mydomain.com/administration/index.php was entered, there are no redirections.

That's clear, and the problem is below.

I want /administration directory to be password protected. My Shared Hosting Control Panel allows to protect directories without manual creating of .htaccess and .htpasswd (you choose a directory to protect, create username and password, and .htaccess and .htpasswd are created automatically). So, .htaccess appears in the /administration folder. .htpasswd appears somewhere else, the path to .htpasswd is correct, and everything looks correct (it works the same way as to create it manually). So, there are 2 .htaccess files in the project, one in the root directory and one in the /administration directory (with .htpasswd at the directory .htaccess knows where it is).

Once the password is created, the results are:

You enter:

https://mydomain.com/administration/index.php

Then it asks to enter a password. If you enter it correctly, https://mydomain.com/administration/index.php is displayed. The result: works perfect.

But, if you enter http://mydomain.com/administration/index.php (yes, http, without S) then instead of redirecting to the same,but https page, it redirects to

https://mydomain.com/401.shtml (starts with httpS)

by unknown reason and even does NOT ask a password. Why?

I've contacted a customer support regarding this question and they are sure the problem is in .htaccess file, and they do not fix .htaccess files (that's clear, they do not, I don't mind).

Why does this happen? Did I forget to put some flags, or some options to change default settings in the .htaccess file?

P.S.Creating .htaccess and .htpasswd manually (not from hosting Control Panel) for the folder /administration causes the same 401 error in case if not https, but http was entered.

And the problem appears with URLs to /administration directory only.

Thank you.

Solution 1

Try using this instead. Not the L and R flag.

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Also clear your browsers cache first, to remove the old incorrect redirect.

If that doesn't work try using this.

RewriteCond %{HTTPS} !on
RewriteCond %{THE_REQUEST} ^(GET|HEAD)\ ([^\ ]+)
RewriteRule ^ https://%{HTTP_HOST}%2 [L,R=301]

I feel a bit bad about writing it, as it seems kind of hackish in my view.

EDIT Seems the 2nd option fixed the problem. So here is the explanation as to why it works.

The authentication module is executed before the rewrite module. Because the username and password is not send when first requesting the page, the authentication module internally 'rewrites' the request url to the 401 page's url. After this mod_rewrite comes and %{THE_REQUEST} now contains 401.shtml instead of the original url. So the resulting redirect contains the 401.shtml, and not the url you want.

The get to the original (not 'rewritten') url, you need to extract it from %{THE_REQUEST}. THE_REQUEST is in the form [requestmethod] [url] HTTP[versionnumber]. The RewriteCond extracts just the middle part ([url]).

For completeness I added the [L,R=301] flags to the second solution.

Solution 2

I think I found an even better solution to this!

Just add this to your .htaccess

ErrorDocument 401 "Unauthorized"

Solution found at:

http://forum.kohanaframework.org/discussion/8934/solved-for-reall-this-time-p-htaccess-folder-password-protection/

-- EDIT

I eventually found the root cause of the issue was ModSecurity flagging my POST data (script and iframe tags cause issues). It would try to return a 401/403 but couldn't find the default error document because ModSecurity had made my htaccess go haywire.

Using ErrorDocument 401 "Unauthorized" bypassed the missing error document problem but did nothing to address the root cause.

For this I ended up using javascript to add 'salt' to anything which was neither whitespace nor a word character...

  $("form").submit(function(event) {
    $("textarea,[type=text]").each(function() {
      $(this).val($(this).val().replace(/([^\s\w])/g, "foobar$1salt"));
    });
  });

then PHP to strip the salt again...

function stripSalt($value) {
  if (is_array($value)) $value = array_map('stripSalt', $value);
  else $value = preg_replace("/(?:foobar)+(.)(?:salt)+/", "$1", $value);

  return $value;
}
$_POST = stripSalt($_POST);

Very, Very, Very Important Note:
Do not use "foobar$1salt" otherwise this post has just shown hackers how to bypass your ModSecurity!

Regex Notes:
I thought it may be worth mentioning what's going on here...

(?:foobar)+ = match first half of salt one or more times but don't store this as a matched group;

(.) = match any character and store this as the first and only group (accessible via $1);

(?:salt)+ = match second half of salt one or more times but don't store this as a matched group.

It's important to match the salt multiple times per character because if you've hit submit and then you use the back button you will go back to the form with all the salt still in there. Hit submit again and more salt gets added. This can happen again and again until you end up with something like: foobarfoobarfoobarfoobar>saltsaltsaltsalt

Comments

[Haradzieniec]

The first one has the same behavior - causes 401 error when type http (for https this is OK). The second one causes "Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request." for any URL.

[Gerben]

I forgot the escape the 2nd space-character. 500 error is gone now. See above for edited code.

[Haradzieniec]

It works!!!!!! Thank you very much! Works for all browsers. What was the reason and how do you know this should work? (Could you please explain me the second string / the difference between #2 and #1? Of course if you can...) Thank you very much for your help, you are my life-saver. BTW, do I need the last_line [L] flag at the end? I will probably add www. and modify the view of URLs a bit. Thanks a lot!!! Thank you, thank you, thank you!!!

[Gerben]

You are more than welcome. Glad to be of help. See edit above for the explanation. Remember to accept the answer; thanks.

[ggkmath]

I had the same problem. Somehow the accepted "answer" above didn't work for me. But this one did.

[Damien Black]

While this solution did work for me, I was also redirecting from a naked domain to www. And this solution would require me to enter my credentials twice for some reason. Gerben's answer, for whatever reason, does not require the double credentials entry.

[Hoogs]

Perhaps in this situation it would be better to ensure you are submitting data to a url which already includes the www?