GUI program to create partition backup image no larger than its files?

Question

I like 'Disks' (gnome-disk-utility) and its ability to create backups of system partitions.

But at some point I read on the internet about a program that would create such images that are not as large as the whole partition, but only as large as the files on it. Now I cannot find it again.

Is there such an utility?

Solution 1

As there doesn't seem to be any program that all by itself would satisfy the two conditions:

1) Have a 'Disks'-like GUI;

2) Create disk-image as large as the space used,

I was thinking about a rather simple workaround, but one involving two GUI programs, very common in Ubuntu:

One would be the same 'Disks' (gnome-disk-utility), used to to create a backup image of a given partition, and the second would be Gparted, to shrink the partition as much as possible before creating its backup image, and to resize it back to original after that.

I see no inconvenient in a such operation: shrinking a partition down to minimum would only take a few minutes (about a minute for 10GB) and would create an empty space to the right. After making a backup image of the disk with 'Disks', using Gparted the partition can be resized back to the right into the empty space up to its initial state, and that would take only a few seconds.

Tested on a partition with an Ubuntu installation:

• shrink partition with Gparted as much as possible

• backup image of that partition with Disks

• restore image with Disks onto a partition larger than the image; that partition should be un-mounted first: How to correctly restore system partition (iso image) with 'Disks'.

Solution 2

The following answer uses a TUI, not a GUI solution:

Depending on your use-case (I use it to make system back-ups) you can use CloneZilla Live. It can create:

• sparse images,
• compressed images of:
  • entire systems,
  • entire drives
  • simply individual partitions.

The main advantages are:

• that it boots from:
  • external media
  • internal media
• comes with its own OS (based on Ubuntu)
• it can make exact cold system back-ups that are easily restored.

On top of all these goodies, it can copy MBRs as well.

If a cold system back-up is not what you're looking for, you can still use the technology that CloneZilla is based upon and that is partclone.

Solution 3

Achieving smaller disk images is usually done by ways of not copying zero outed blocks from the initial file system in the image. The concept is called sparse space in files or puching holes in files, and your host file system has to support it. a bit command line but try this:

sudo apt-get install virt-manager qemu-utils 

And then once it's install you can use :

qemu-img convert -f raw -O raw  /dev/sdX /home/<user>/Ubuntu.img

This will create a sparse ( zeros in the file aren't written) image of your disk. where -f = format, -O output format. sdX the disk ( can use partitions as well). The program has advanced features such as snapshots and different output formats (vdi, vmdk, etc). Your place where you save the image has to support sparse files, or else the size will be just as big, since zeros in the partition will be written to disk. And to make the most of it, do run a program such as sfill or zerofree found in apt on the partition, because free space does not always mean zero space, and you want to maximize the amount of zero space on the file system.

Bonus ninja points. Since linux thinks of disks as "files", then you could also use plain cp, with the condition that your output partition where you park the file knows sparse files.

sudo cp --sparse=always /dev/sdX /home/<user>/Ubuntu.img

Which will create a raw copy by means of cp..

Examples of filesystems that support sparse : ext4, btrfs, xfs, ntfs ( ntfs-3g however might not allow sparse writes).

Examples of filesystems that do not : fat32.

Solution 4

Edit: This will not work well if files deleted from the filesystem are still present on the partition, so first it's necessary to wipe the free space on the target partition:

1. Mount the target partition
2. Run dd if=/dev/zero of=<path_to_partition_volume>/file.dd && rm <path_to_partition_volume>/file.dd
3. Unmount the target partition

Then:

---

You can read the whole partition block by block with dd and output everything to a compression utility like gzip. Most likely any compression tool, even using the fastest/lowest compression settings, will compress all the free space during the process.

With gzip:

sudo dd if=/dev/sdX | GZIP=-1 gzip >/path/to/backup.gz

(GZIP=-1 temporarily sets the environment variable GZIP to -1 to achieve the fastest/lowest compression)

To restore a backup:

gzip -d /path/to/backup.gz >/dev/sdX

Solution 5

How about using ddrescue with -S option?

ddrescue -S /dev/sdXY /<path_to_backup>/backup.img

This will clone each block but zero blocks found in block device /dev/sdXY to /<path_to_backup>/backup.img

View more solutions

Comments

[muru]

CloneZilla, perhaps?

[Admin]

As I see clonezilla can be run only as a live session. I have edited to specify that I am looking for a program that can be installed and run from a OS, I am using multiple ubuntus and I prefer this solution

[Admin]

I have edited to specify that I am looking for a GUI program that can be installed and run from a OS, I am using multiple ubuntus and I prefer this solution

[Admin]

I have edited to specify that I am looking for a GUI program that can be installed and run from a OS, I am using multiple ubuntus and I prefer this solution

[Admin]

what would be the command to restore the image?

[Admin]

Sounds great. But what would be the command to restore the image?

[kos]

@cipricus I found out that this is actually highly system-dependant, and won't work on every distribution, so i added a simpler yet maybe just a little slower method to accomplish the same without downloading additional software, which is the method i actually use

[user283885]

Usually on server farms we don't have that much in GUI options and build scripts around said commands. This is why command line is usually more complete/powerful on Unix. Converting back is just as easy as inverting the parameters.. sudo cp /home/<user>/Ubuntu.img /dev/sdX, or if you fancy dd, sudo dd if=/home/<user>/Ubuntu.img of=/dev/sdX. Same thing with qemu-img convert.. If you say, want to plant a virtual machine on real hardware. When it writes back, it will write zeros to disk, thus you unsparse, as it needs to properly address the partition's geometry.

[user283885]

As a matter of fact, copying a sparse file to a location that doesn't support/prefer off the bat sparse files will make a regular full size file out of your image.

[Admin]

I know how powerful command-line is, but I am interested in a GUI, like gnome-disks-utility.

[user283885]

I understand your point and I apologize that I am not aware of any GUI tools to suit your needs. I just thought that at least something second to that might be of use to you.

[Admin]

how to install pyclonezilla? not one google entry for this question

[Admin]

It seems there is no such gui yet, so your solution might be the best.

[user283885]

Welcome to this side of the fence. The reason I pointed to server farms is because of how most development is done lately. For example, the qemu-img is an utility from a much larger project ( openstack, started by redhat, rackspace & nasa). You have these big companies that build software to suit their data centers and then you have guys like us who piggyback on that effort. Sure it works, but it's spartan at best. For now, the best I can do is share the know how with others.

[Fabby]

I was too quick to point the GUI out: It doesn't work. CLI only! I'll delete my answer after you've left a note that you've read this.

[Admin]

As there is no program, it seems, to fulfill both conditions: 1) Have a 'Disks'-like GUI; 2) Create disk-image as large as the space used, there is no point in deleting your answer. CLI answers to GUI questions are usually presented here and often accepted, with good reason. I was just trying to push for what I wanted.

[Admin]

What do you think about shrinking the partition with gparted before using Disks to backup it?

[Admin]

What do you think about shrinking the partition with gparted before using Disks to backup it?

[Admin]

What do you think about shrinking the partition with gparted before using Disks to backup it?

[kos]

@cipricus Better wait for someone else to say something about this, because i'm not sure, even tough i think it should be fine to do it that way either

[user283885]

It depend. For example xfs doesn't support shrink. ext4,ntfs and btrfs can shrink for example. I did use gparted to shrink ntfs disks before, and it worked. I usually go from old disks to virtual images to run them in kvm, when the hardware to run it becomes obsolete (How does one maintain a very important databse rooted and licensed on a win95 in 2015?.. think of visual foxpro server..) and it happens so often that the data is business required, and thus growing is more common.

[Fabby]

:D I would take a backup and then shrink... ;-)

[Admin]

@that's the proper way of course...

[Fabby]

As I've just received an upvote to my answer, reading if you ever found a perfect answer, and this seems to be the one! Why don't you accept this one?

[PhilJ]

I found the virt-sparsify method does certainly reduce the size of the .img file but it requires a very large /tmp directory, roughly equal to the size of the partition being imaged. It gave out a warning message but when I tried the --tmp option shown above (varied for my setup) it gave me >unknown option `--tmp' I'm not sure now how to go about diverting the /tmp output elsewhere and any advice on that would be helpful to me and to future readers of this post.

[PhilJ]

@kos I found the ddrescue command you give produced an image quite effortlessly but it did not reduce the size of the file at all. I was imaging a NTFS partition to ext4, both of which seem to work with sparse files but the .img file size shown in Nautilus exactly matched the originating partition size. So it looks as though fiddling about with gParted might still be needed?

[cipricus]

I'm the author of the question and of the above answer. Shrinking a partition is a risky action before which a back-up is recommended. Using a shrinking-partition action to make the back-up gives a logical loop. A much safer solution for a ext4 partition is to use Disks to make an image of the entire partition and then use e2fsck -f -y xxx.img and resize2fs -M xxx.img to resize as said here.