Mercurial client error 255 and HTTP error 404 when attempting to push large files to server

Question

Problem:

19/06/10 Update: More evidence problem is server-side. Receiving this error on Windows 7 command line (see below for full traceback):

URLError: <urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host>
    abort: error: An existing connection was forcibly closed by the remote host

When attempting to push a changeset that contains 6 large files (.exe, .dmg, etc) to my remote server my client (MacHG) is reporting the error:

"Error During Push. Mercurial reported error number 255: abort: HTTP Error 404: Not Found"

What does the error even mean?! The only thing unique (that I can tell) about this commit is the size, type, and filenames of the files. How can I determine which exact file within the changeset is failing? How can I delete the corrupt changeset from the repository? In a different post, someone reported using "mq" extensions to effectively delete an erroneous changeset from the history within a repository, but mq looks overly complicated for what I'm trying to solve.

---

Background:

I can push and pull the following: source files, directories, .class files and a .jar file to and from the server, using both MacHG and toirtoise HG.

I successfully committed to my local repository the addition for the first time the 6 large .exe, .dmg etc installer files (about 130Mb total).

In the following commit to my local repository, I removed ("untracked" / forget) the 6 files causing the problem, however the previous (failing) changeset is still queued to be pushed to the server (i.e. my local host is trying to push the "add" and then the "remove" to the remote server - and keep aligned with the "keep everything in history" philosophy of the source control system).

I can commit .txt .java files etc using TortoiseHG from Windows PCs. I haven't actually testing committing or pushing the same large files using TortoiseHG.

Please help!

---

Setup:

Client applications = MacHG v0.9.7 (SCM 1.5.4), and TortoiseHG v1.0.4 (SCM 1.5.4)

Server = HTTPS, IIS7.5, Mercurial 1.5.4, Python 2.6.5, setup using these instructions:

http://www.jeremyskinner.co.uk/mercurial-on-iis7/

In IIS7.5 the CGI handler is configured to handle ALL verbs (not just GET, POST and HEAD).

My hgweb.cgi file on the server is as follows:

#!/usr/bin/env python
#
# An example hgweb CGI script, edit as necessary

# Path to repo or hgweb config to serve (see 'hg help hgweb')
#config = "/path/to/repo/or/config"

# Uncomment and adjust if Mercurial is not installed system-wide:
#import sys; sys.path.insert(0, "/path/to/python/lib")

# Uncomment to send python tracebacks to the browser if an error occurs:
#import cgitb; cgitb.enable()

from mercurial import demandimport; demandimport.enable()
from mercurial.hgweb import hgweb, wsgicgi
application = hgweb('C:\inetpub\wwwroot\hg\hgweb.config')
wsgicgi.launch(application)

My hgweb.config file on the server is as follows:

[collections]
C:\Mercurial Repositories = C:\Mercurial Repositories

[web]
baseurl = /hg
allow_push = usernamea
allow_push = usernameb

Output from the command line from my macbook (both Mercurial and MacHG installed) using -v and --trackback flags:

macbook15:hgrepos coderunner$ hg -v --traceback push
pushing to https://coderunner:***@hg.mydomain.com.au/hg/hgrepos
searching for changes
3 changesets found
Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/mercurial/dispatch.py", line 50, in _runcatch
    return _dispatch(ui, args)
  File "/Library/Python/2.6/site-packages/mercurial/dispatch.py", line 471, in _dispatch
    return runcommand(lui, repo, cmd, fullargs, ui, options, d)
  File "/Library/Python/2.6/site-packages/mercurial/dispatch.py", line 341, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/Library/Python/2.6/site-packages/mercurial/dispatch.py", line 522, in _runcommand
    return checkargs()
  File "/Library/Python/2.6/site-packages/mercurial/dispatch.py", line 476, in checkargs
    return cmdfunc()
  File "/Library/Python/2.6/site-packages/mercurial/dispatch.py", line 470, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/Library/Python/2.6/site-packages/mercurial/util.py", line 401, in check
    return func(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/mercurial/commands.py", line 2462, in push
    r = repo.push(other, opts.get('force'), revs=revs)
  File "/Library/Python/2.6/site-packages/mercurial/localrepo.py", line 1491, in push
    return self.push_unbundle(remote, force, revs)
  File "/Library/Python/2.6/site-packages/mercurial/localrepo.py", line 1636, in push_unbundle
    return remote.unbundle(cg, remote_heads, 'push')
  File "/Library/Python/2.6/site-packages/mercurial/httprepo.py", line 235, in unbundle
    heads=' '.join(map(hex, heads)))
  File "/Library/Python/2.6/site-packages/mercurial/httprepo.py", line 134, in do_read
    fp = self.do_cmd(cmd, **args)
  File "/Library/Python/2.6/site-packages/mercurial/httprepo.py", line 85, in do_cmd
    resp = self.urlopener.open(req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 389, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 502, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 427, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 361, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 510, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
abort: HTTP Error 404: Not Found
macbook15:hgrepos coderunner$ 

Output from Windows 7 host (has only TortoiseHG installed) attempting to push the same files to the server (different changset, but contains the same 6 file additions as the changeset being pushed from the macbook)

c:\repositories\hgrepos>hg -v --traceback push
pushing to https://coderunner:***@hg.mydomain.com.au/hg/hgrepos
searching for changes
1 changesets found
Traceback (most recent call last):
  File "mercurial\dispatch.pyo", line 50, in _runcatch
  File "mercurial\dispatch.pyo", line 471, in _dispatch
  File "mercurial\dispatch.pyo", line 341, in runcommand
  File "mercurial\dispatch.pyo", line 522, in _runcommand
  File "mercurial\dispatch.pyo", line 476, in checkargs
  File "mercurial\dispatch.pyo", line 470, in <lambda>
  File "mercurial\util.pyo", line 401, in check
  File "mercurial\commands.pyo", line 2462, in push
  File "mercurial\localrepo.pyo", line 1491, in push
  File "mercurial\localrepo.pyo", line 1636, in push_unbundle
  File "mercurial\httprepo.pyo", line 235, in unbundle
  File "mercurial\httprepo.pyo", line 134, in do_read
  File "mercurial\httprepo.pyo", line 85, in do_cmd
  File "urllib2.pyo", line 389, in open
  File "urllib2.pyo", line 407, in _open
  File "urllib2.pyo", line 367, in _call_chain
  File "mercurial\url.pyo", line 523, in https_open
  File "mercurial\keepalive.pyo", line 259, in do_open
URLError: <urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host>
abort: error: An existing connection was forcibly closed by the remote host

c:\repositories\hgrepos>

It is a keep-alive issue? Is IIS7.5 at fault? Python 2.6.5 at fault?

Solution 1

Went through the same pain points...

With the default settings on the IIS server, you will not be able to push large repositories to the server, as IIS has a default maximum request length of only 4 MB, and a timeout for CGI scripts of 15 min, making it impossible to upload large files. To enable the uploading of large files (and this is not easy to find on the web…), do the following:
1. In IIS Manager, click on the web site node, and click the Limits… link.
2. Then specify a connection time-out sufficiently large (I chose 1 hour here, or 3600 seconds)
3. Next, click the node containing hg (as per the installation procedure), then double-click CGI
4. Specify a sufficiently-long time out for CGI scripts (e.g., 10 hours)

Now, edit C:\inetpub\wwwroot\hg\web.config, so that it has a new <security> section under <system.webserver>, and a <httpRuntime> specification under <system.web>:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
[…]
    <security>
         <requestFiltering>
           <requestLimits maxAllowedContentLength ="2147482624" />
      </requestFiltering>
    </security>
  </system.webServer>
  <system.web>
    <httpRuntime
      executionTimeout="540000" maxRequestLength="2097151"/>
  </system.web>
</configuration>

This specifies an http timeout of a bit more than 6 days, and a maximum upload limit of about 2 GB.

Solution 2

Had the same issue using IIS 7 as server. Tried the solution above which resolved the error 255 issue, but still got Errorno 10054 with larger files. I then increased the Connection Time-out in IIS which worked.

To change: Web Site -> Manage Web Site -> Advanced Settings -> Connection Limits -> Connection Time-out. The default is 2 minutes. Changed mine to 20 minutes and it worked.

Not sure why this works but seems that Mercurial makes a connection to the server, takes a while to process larger files, then only sends a request. By that time IIS has disconnected the client.

Comments

[Admin]

Thank you for posting your solution. I've only partially-tested it so far. So far so good, in that the problem doesn't occur quite as much - but it is still occurring. I will test over the next few days and post my results soon.

[Admin]

Your solution did help somewhat, but not completely. I can now push approx 300Mb successfully, however only via the command line on both Mac OS 10.6.4 and Win7. The 300Mb push fails under both MacHG and TortoiseHG. I tested a 1.3GB push of 5000 files, ranging in size from 1kB to 600MB. It failed, with IIS7.5 returning "Error 502 - bad gateway". According to IIS documentation, this means that the CGI script returned an improper HTTP response.

[Admin]

It may seem crazy uploading large files, but I was hoping to use Mercurial to keep historic changes to database dumps, downloaded installer packages etc, so that the project doesn't have multiple document stores. I can't afford any more time troubleshooting and debugging, so I've decided to use an FTP server to store larger files. A disappointing but necessary compromise, as otherwise Mercurial seemed to be the best choice for source control.

[jocull]

This seems like the right path, but things are different on IIS 6 and I can't seem to get it working.

[André Santaló]

For some reason this didn't helped me, but dividing the push into 4 smaller pieces worked, so I think it was a problem with size after all.

[bor]

This was the problem in my case. Or at least I too have a problem related to SSL. Over https the connection was forcefully / abruptly closed always after the same amount of bytes sent. Over http push completed successfully (110MB in total). python 2.7.12 x64, mercurial 3.9 x64, IIS 7.5 - Win Server 2008 R2

[bor]

I could fix the problem using the IISCrypto but had to apply the 'best practices' as disabling only the SSLv2 wasn't enough. Now push over https is working fine.