rsync to/from tar archive

Question

Is it possible to rsync a real filesystem (remote) with a tar archive (local)? If so, how?

Problem is I need to correctly backup user/group/permission settings and, while I have root access on remote I would like to avoid running as root on the local machine.

My first (and foremost) usage case would be to update a remote embedded target (ARM9) from a .tar produced by Buildroot. I do not have the "real thing" on disk (I can produce a copy while being root) and I would like to avoid transferring the whole rootfs to update a few files.

Solution 1

The right answer seems to be to unpack the tar archive using fakeroot (to avoid becoming root) and then use rsync. Possibly repack the archive, if needed.

Unfortunately things are not this easy because bad interaction between ssh and fakeroot. I will detail what I did to help whoever will search.

Theory is straightforward:

1. create a temp directory
2. unpack tar archive into it
3. rysnc as needed
4. if something changed locally repack int a new tar archive
5. cleanup

In order to preserve all user/group/permissions steps 2..4 must be done under fakeroot.

Catch is rsync uses ssh for communication (and I want it to!) and thus, being "fakerooted" it tries to open root credentials (in /root/.ssh/), failing badly. The following set of options work for me.

#!/bin/bash

target=myHost
here=$(pwd)

# 1. create a temp directory
cd /tmp
mkdir TMPfs
cd TMPfs

fakeroot bash <<- EOF
    # 2. unpack tar archive into it
    tar xf $here/archive.tar
    # 3. rysnc as needed (ssh options are *the* relevant thing)
    rsync -av -e "ssh -i $HOME/.ssh/id_rsa -oUserKnownHostsFile=$HOME/.ssh/known_hosts" . root@$target:/
    # 4. if something changed locally repack int a new tar archive (not needed here)
EOF

# 5. cleanup
rm -rf *
cd ..
rmdir TMPfs

I still get the error "Could not create directory '/root/.ssh'." but it appears to be benign (no files are created/used there).

Solution 2

I haven't actually tried this, but it should work.

Using 'archivemount' (source from:)

http://www.cybernoia.de/software/archivemount/

and a 'libarchive' included in many distros (suse, redhat, etc)...

Or a pre-built one from:

https://rpmfind.net/linux/rpm2html/search.php?query=archivemount

You can mount a tar-archive using the fusermount facility in linux.

From there, you should be able to use rsync directly to the final system.

I wrote a simple passthrough batchfile to test rsync's passthrough:

#!/bin/bash
# ussh -- use root@ssh to target system
exec ssh  root@"$@"

then, as a test, used rsync to pass dir 'test1' to 'ishtar', calling it /tmp/test2 on the target:

RSYNC_RSH=$PWD/Ussh rsync -uva /tmp/test1/ ishtar:/tmp/test2

It will ask you for the password of the target sys's root logon, or you could setup the target system to accept a root login via a certificate so no password would be needed.

This would seem to be the most efficient way to do what you want (you might need to modify the rsync options to not copy dir times and things like that), but is this the type of thing you were looking for?

-Astara

Comments

[Admin]

No, you can not sync files directly into/from a tar archive with rsync. How about just extracting the tar file into a directory hierarchy, then running rsync against that, and finally putting everything back into a tar archive again? That would save you from transferring files, but does require some fiddling with TAR files, of course.

[Admin]

@SamiLaine: as said my problem with that approach is I need to preserve user/group/permission settings and that is only possible for root. I am currently looking into using fakeroot, which should be exactly what I need. I will post my solution as "Answer", if it actually works.

[Astara]

What do you mean "unpack tar using fake root"? Are permissions a problem? I.e. I guess I assumed that you were transferring a tar archive of your files via rsync to your directory on another system. If you are talking about preserving permissions, does the tar you use even copy / preserve meta data other than UID/GID's? rsync can transfer ACL's and other extended attributes (metadata), but most current/standard tar programs won't store that info, So presumption was you didn't need it. Why are you packing into a tar archive? --- if something changes locally, why not rsync directly?

[Astara]

Otherwise, you could transfer the tar-archive and unpack it over the target and use the --keep-newer-files switch, which says don't replace existing files that are newer than the version in the tar archive. But if you just have a few files, not sure why you won't just rsync directly. If you own files on both ends, don't see why you would need fakeroot. Conversely, if can become root on both systems, just use 'rsync' as root. If you use ssh, just create a key on source system(as root), and store that in remote-root's .ssh dir. Then you can login w/no password (if ssh config'ed that way).

[ZioByte]

@Astara: The tar is the complete root-filesystem as packed by Buildroot; it is complete with GID/UID/permissions as needed and is packed into a GNUtar archive by the build system itself. I don't really want to meddle with it. Since it may be quite sizeable and target may be very remote I am using rsync to update target with minimal transfer time. I will need to become root on target in order to overwrite system files, but that may not be the case on the development system.

[ZioByte]

I will check as soon as I find a bit of time, as it looks much more efficient than current method; just one perplexity: If I actually "mount" the tar archive and "just use" it as normal user won't I be denied access to all files "readable from root only"? There are many of those in a rootfs, even excluding log files (empty by definition).

[Astara]

The access issue you mention shouldn't be an issue. If there are root-read-only files in the image, then how would 'Buildroot' (created by a user, right?) read the files to put in the tar-image in the first place?

[ZioByte]

I'm afraid You don't understand the complexity ;) Buildroot produces a full directory mimicking the root filesystem but, obviously, with the ownerships of the current user. When everything is ready it does a pass under fakeroot (see: wiki.debian.org/FakeRoot) to prepare the tar archive with proper ownerships and permissions. This is exactly (in reverse) what I am currently doing to ship everything to target, where I can recreate the uid/gid/permissions because there I am root. Tar mounting via loopback is a nice idea, but I would have to become root to run rsync :(

[Astara]

Which is true: Buildroot creates a tar-image from a real-root, or Buildroot creates a tar image from a tree that is entirely owned by the user? Does Buildroot use 'fakeroot' to simulate ownership of the files in the user-tree, so Buildroot "sees" files owned by 'root' (or other users) when it is creating the tar image? IF that's the case, then why not rsync from the original user tree that was used to create the 'root' (where all the files are owned by the user), but use fakeroot under 'rsync', so rsync will see the files with the "correct" file perms+ownerships?

[Astara]

@ZioByte: If you use fakeroot to create a tar with root-perms, you've created a type of "protected-image" that can't be extracted (keeping perms) w/o root privs and, in the same way, won't be fully accessible if you mount it as an archive. You need to access the files before you've "protected" the archive, then run fakeroot under rsync, so rsync thinks it is transferring root-owned files -- that the the remote rsync (which is running under root) will extract w/the proper permissions.

[Marcus]

Archivemount unfortunately doesn't support symbolic links. So using rsync to backup a whole system to this is not feasible, really.

[Astara]

Didn't you say you had root permission on the target, but not on the source. However, to make a full copy from the source, you need to have all files readable, at the very least. In that case, you could rsync the files from the source->target having it run as root only on the target, or, I'd think faster, would be to make a tar-archive on the source (it will copy the files owned by root just fine), then mount that archive on the target where you are root and unpack that. At this point, not sure why we were using fakeroot to begin with.

[Astara]

BTW, rsync uses 'rsh' by default which has lower security but also has lower security requirements.