Resume interrupted copying process

Question

How to resume securely and reliably process of copying file $A into backup location $B done with pv "$A" > "$B" or cat "$A" > "$B" ?

(let's assume file $A is very big, e.g. LVM2 snapshot file)

Is it achievable with dd ?

Preffered: bash or python (preferably python3) solutions.

Example scenario: pv "$A" > "$B" interrupted after copying 90%. How to resume it, in order to finish copying process but not repeating whole work again ?

Solution 1

Yes you can use dd to skip the blocks.

A="file1"
B="file2"

BLOCKSIZE=512  # default bs for dd

size_b=$(stat -c "%s" "$B")
skip_blocks=$((size_b / BLOCKSIZE))

dd if="$A" of="$B" skip=$skip_blocks seek=$skip_blocks bs=$BLOCKSIZE

The important parameters here are skip as well as seek:

• skip: skip BLOCKS ibs-sized blocks at start of input
• seek: skip BLOCKS obs-sized blocks at start of output

Solution 2

You want rsync:

rsync -a --append "$A" "$B"

Solution 3

Did you try dd skip with an offset of B's real file size (independent of the partition block size)?

That would get you the missing part. At that point you could directly cat them together into a new file with cat "$B" "$A2" >> "$C"; #mv "$C" "$B" (where $C is of course the missing part on a path with enough space).

cat works fine for concatenating binaries too and in this case you do not have multiple file headers that would precent simple scripted merges. There's a chance the end of $B is corrupted, but in that case you could cut it short and reread more of $A in the initial dd step.

Comments

[Grzegorz Wierzowiecki]

You might like to utilize: unix.stackexchange.com/a/12538/9689 and unix.stackexchange.com/questions/32941/…

[Grzegorz Wierzowiecki]

I do not want to use rsync. Let's assume A is 1TB and I've copied already 900GB of data and there are 100G remaining. rsync would read whole 1TB, while I need only the last 100G ! -> Read first about algorithm which is used: en.wikipedia.org/wiki/Rsync#Algorithm

[killermist]

@GrzegorzWierzowiecki I think you're wrong. Unless my experience with rsync is entirely wrong, rsync will read what it needs to read to verify the output is right, until it hits a point where there start to appear disparities, and then it will continue from that marked point. This looks to be pretty-much exactly what is needed.

[Thor]

For this to work you also need to add --append.

[Grzegorz Wierzowiecki]

Thanks @Thor for stressing this option. As I've double checked it seems that behaviour of --append has changed since version 3.0.0. Could you ensure me if currently --append ignores already copied part, while --append-verify reads whole A for checksum check ? (before version 3.0.0. --append behaved like --append-verify -> That's the reason of misunderstanding)