is 'dd' command taking too long?
Solution 1
In the future, you should use pv
to get a running progress bar.
sudo apt-get install pv
With pv
installed, let's assume you want to clone a 20GB drive, /dev/foo
, to another drive (20GB or larger!), /dev/baz
:
sudo dd if=/dev/foo bs=4M | pv -s 20G | sudo dd of=/dev/baz bs=4M
Important bits to notice: the bs=4M
argument sets the blocksize for dd operations to 4MB, which drastically improves the speed of the whole thing. And the -s 20G
argument tells pv
how big this operation is expected to be, so it can give you an ETA as well as a current speed.
I love pv
so hard it should probably be illegal.
Note that while doing it this way is intuitive and nice and neat from left to right ordering, piping to and from STDOUT can incur a performance penalty if you're talking about really fast data streams. The following syntax is faster, if you're looking at moving several hundred MB/sec:
pv -s 20G < /dev/foo > /dev/baz
The -s 20G
is optional, if you actually know how big (or about how big) the stream will be, it allows pv
to give you a time estimate for completion. Without that, pv
will try to figure out how large the dataset is if possible (for instance, it knows how big a file is) but if it can't (eg with a block device, not a file), it just tells you the rate of transfer without guessing how long things will take.
Solution 2
You can see how far it has got by sending it a SIGUSR1 signal in order to see how much data it has copied and the transfer rate:
kill -SIGUSR1 $(pidof dd)
For copying activity you are limited by I/O speed of the device, so the CPU should not be fully loaded, so don't worry about that.
Solution 3
I've used pv as well as (ps and kill) in the past as suggested in the other answers, but more recently I've just been using dc3dd
instead which produces the same results while providing a progress report throughout the process.
You can check to see if it's already installed with: which dc3dd
If not you can install it with sudo apt-get install dc3dd
The command switches are similar to dd (for cloning, although wiping is a bit more straightforward).
In your case I would use the command dc3dd if=/dev/sda of=/dev/sdb
Edit:
Recent versions of dd
from the coreutils package version 8.24+ included in Ubuntu 16.04 and later include a status parameter. You can accomplish the same result with dd
by adding the status=progress
switch to your dd
command line.
Example: dd if=/dev/zero of=/dev/null count=1000 status=progress
Related videos on Youtube
![Admin](/assets/logo_square_200-5d0d61d6853298bd2a4fe063103715b4daf2819fc21225efa21dfb93e61952ea.png)
Admin
Updated on September 18, 2022Comments
-
Admin almost 2 years
I set up
dd
to clone a smaller system 40.00GB hard drive (/dev/sda) to a new bigger 111.00GB one connected via a USB reader (dev/sdb) and Its been going for two hours now. The activity meter on the new hard drive shows it's doing something. But the CPU is only about 20%. When is this thing going to complete? Should I re-start the process? -
Admin over 11 yearsNoted. Well, I killed the process to figure out it was going quite slowly.. had to redo the partition table but it's back to all empty now. You know of any better ways to clone a hard drive the won't take all day?
-
Sergey over 11 years@SeanWebber: running that command should not actually terminate the
dd
process. Fromman dd
: "Sending a USR1 signal to a runningdd
process makes it print I/O statistics to standard error and then resume copying." To speed up the process try specifying a larger block size as Jim Salter suggests in the other answer. -
Colin Ian King over 11 yearsIf course, pv is very useful, but if you are copying a lot of data it adds more context switches and buffer copies so it will consume more CPU and perhaps may slow things down a small amount. But since copying data is rate limited on the drives it won't add much overhead.
-
Jim Salter over 11 yearsIn actual practice, you are VANISHINGLY unlikely to see pv add so much as 0.1% to the time-to-copy. If you do, it's almost certainly an operation so blindingly fast that there was no point in trying to add a progress bar in the first place.
-
Colin Ian King over 11 yearsVery true. I'm just and old style engineer who is just used to saving every cycle I can.
-
Jim Salter almost 11 yearsrevisiting this later yet: after some conversation with the author of pv, I discovered that you can avoid the speed penalty by taking dd out of the equation entirely:
pv < /dev/sda > /dev/sdb
works just fine, and will go every bit as fast as the underlying hardware is capable of going. -
intrepidus over 10 yearsOS X's dd must be broken since sending the SIGUSR1 killed it for me as well.
-
isaaclw about 10 yearsThanks @JimSalter I was just noticing that my piping:
dd if=/dev/sda | pv | dd of=/file
was using up 20%, 60%, and 20% CPU respectively. Which to me seems to be a problem. -
Parto almost 10 yearsPlease add more info to this answer like how to mount it using
mount
and notudisks
. -
ssnobody over 8 yearsFrom elsewhere: kill -INFO $(pgrep ^dd$) on BSD systems (like OSX).
-
Jacksonkr over 8 years
dd: bs: illegal numeric value
so I went withbs=32m
-
Elder Geek over 7 yearsThere's an easier way to get progress output from
dd
now. See may answer! -
tjespe about 7 yearsWill this kill the process?
-
Colin Ian King about 7 yearsNo, the SIGUSR1 is handled by dd and it won't kill it at all. Only SIGKILL, SIGHUP etc will kill it.
-
Shayan over 4 years@JimSalter you should definitely add your comment to your answer, because comments tend to get deleted a lot, thanks
-
pauljohn32 over 4 yearsI just did this and it does not kill the process. Here is example output:
10712217+0 records in [newline] 10712217+0 records out [newline] 5484655104 bytes (5.5 GB, 5.1 GiB) copied, 2233.64 s, 2.5 MB/s
-
dawid over 3 yearsI love this because it doesn't required you to start copying from scratch to see the progress.
-
Martin over 2 yearsMac users:
pv
can be installed using Homebrew,brew install pv
. As mentioned in prev commentsM
is an illegal numeric value, so usem
instead, e.g.4m
.