Rsync hangs: expand file_list pointer array to N bytes, did move

Question

Rsync is going into "interruptible sleep" mode after transferring some files from a local folder to a NFS folder. The folder I am trying to backup contains more than 180gb of data.

This is what rsync outputs before it hangs:

[sender] expand file_list pointer array to 524288 bytes, did move

I am running Ubuntu Server 14.04 LTS with rsync version 3.1.0 protocol version 31 and I am running rsync with these options:

/usr/bin/rsync -rHAXxvvut --numeric-ids --progress {SRC_FOLDER} {NFS_FOLDER}

Thanks for any hints

Solution

Considering that the rsync you're using is an open-source software, it's quite easy to get accesso to related source code.

After downloading the main .tar.gz and applied the Ubuntu patch (rsync_3.1.0-2ubuntu0.4.diff.gz), you end up with exactly the code underlying the rsync you're using. Something like this:

$ mkdir rsync
$ cd rsync/
$ wget http://archive.ubuntu.com/ubuntu/pool/main/r/rsync/rsync_3.1.0.orig.tar.gz
$ wget http://archive.ubuntu.com/ubuntu/pool/main/r/rsync/rsync_3.1.0-2ubuntu0.4.diff.gz
$ gzip -d rsync_3.1.0-2ubuntu0.4.diff.gz
$ tar zxvf rsync_3.1.0.orig.tar.gz 
$ cd rsync-3.1.0/
$ patch -p1 < ../rsync_3.1.0-2ubuntu0.4.diff

Now a simple grep can quickly tell us the context of your error message:

$ grep -r 'expand file_list pointer array to' 
flist.c:        rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, did%s move\n",

So you're lucky, as your error message is used in a single fragment of a single file. Nameli: flist.c.

Let's give a look:

It's relatively easy to guess that the routine containing the error message (lines 325, 326, 327, 328) is named flist_expand and sounds like something needed to ensure that the whole file list (to rsync) can be held in a properly sized in-memory structure (aka: the more files you need to rsync, the more memory is required to handle rsync-computations, and as such a list is not known "in advance", it need to be dinamically computed, by allocating proper chunks of memory to a "list" [more or less]).

So, I would bet that your problem rely NOT on the size of data you're rsync-ing, but on the number of files. I'd try splitting your rsync in multiple sub-rsync, by focusing on internal subfolders.

Actually, it would be nice to better investigate the:

1. line 328: (new_ptr == flist->files) ? " not" : "");
2. line 334: out_of_memory("flist_expand");

but this goes much beyond my initial goal :-)

Anyway, I would bet that checking your logs you would find some "out of memory" message.... :-)

HTH!

Comments

[Diamond]

Adding the -v (verbose) option more than once can output more info. Try to add something similar to: -vv and see the output.

[jithujose]

@ringø yes it's 64 bit

[jithujose]

Unfortunately I've only NFS or FTP access to the remote directory.