hexdump vs xxd format difference

Question

I was searching for how to do a reverse hexdump and found xxd mentioned. However, it does not seem to work with simply:

xxd -r hexdumpfile > binaryfile

I then compared the difference between outputs of xxd infile and hexdump infile, and found three differences:

1. xxd output has a colon after the address
2. xxd output has the positions in the data reversed (for example, 5a42 in hexdump output becomes 425a in xxd output)
3. There are some extra characters after each line

I only have the hexdumped version of certain files on a server. How can I correctly get back the binary data using xxd?

Solution 1

There's no one command that I know of that will do the conversion, but it can easily be broken up into a few steps:

1. Strip addresses from hexdump output using sed
2. Convert into binary using xxd
3. Endian conversion (for example, 5a42 becomes 425a) using dd

Here's the full command:

sed 's/^[0-9]*//' hexdump | xxd -r -p | dd conv=swab of=binaryfile

Solution 2

Try to add -p.

xxd -r -p hexdumpfile > binaryfile

Solution 3

This answer is a cross-post from https://stackoverflow.com/a/52834021/6770384

You can do the conversion in one sed command. It's sufficient to add the : after the address and change the endianness (switching ab12 to 12ab).

sed -E 's/ /: /;s/ (..)(..)/ \2\1/g;$d' dump | xxd -r

Known Bugs (see comment section)

• A trailing null byte is added if the original file was of odd length (e.g. 1, 3, 5, 7, ..., byte long).
• Repeating sections of the original file are not restored correctly if they were hexdumped using a *.

Comments

[barlop]

I haven't really tested precisely what you ask, but try xxd -p and -b, -p is plain -b is binary, as well as your -r. As to byte order, that'd to do with big endian and little endian, and maybe xxd can't reverse that.. but it's related to how the bytes are stored in the file.Like, if you have a text file in notepad, you can save it as unicode 16-bit big endian, or little endian or UTF-8 or whatever, and you see the difference from xxd. The od command might display it differently.

[jpaugh]

For future reference, hexdump allows you to choose the endianness used in its output, which could make this simpler. (Why now? Found your question helpful for something unrelated.)

[Smeterlink]

does not work if the hexdump ends with 0a

[Smeterlink]

does not work if the hexdump ends with 0a

[Socowi]

@Smeterlink Thank you for this observation! You are right: This command produces a trailing null byte which wasn't in the original file if the length in bytes was an odd number. I also found another bug. hexdump may hide repeated parts using a *. This command also cannot restore these parts correctly. Example: yes | head -n100 | hexdump | sed -E 's/ /: /;s/ (..)(..)/ \2\1/g' | xxd -r prints only 8 y instead of 100. Fixing these things would require more than this one-liner. Maybe I add another script in the future.

[Dagelf]

Amazing. And not documented in the help... THANKS!