"grep: memory exhausted" error on large partition

Question

I was editing a text file with my notes on linux commands when I noticed a big chunk of it was missing (copy without paste, probably). The problem is that I already saved the document. (And this is a simple editor, so no hidden copies)

Now I found a number of blog posts (this one inparticular for instance), which show how you can easily search through a partion for text strings using grep:

$ sudo grep -a -C100 'sudo lshw -c' /dev/sdb2 > file.txt

But I get this after a while:

grep: memory exhausted

I understand from this answer that it is about grep reading lines bigger than memory, so I guess I need similar code but without find.

It is 2 TB NTFS partition on a 3 TB harddisc.

Solution

The grep program reads a line at a time into memory. A line is defined as everything after one newline character and up to the next one. With binary data, there could be a very large space without any newlines.

You could try using grep -z. This tells grep to treat null bytes as the input record separator instead of newlines. Extremely large chunks of binary data are less likely to contain no null byte than no newline. In practice, the most likely chunk of data with no newline is a long sequence of null bytes in an area of the disk that's never been written to yet. Large amounts of text data containing no null bytes are likely to not be so large as to exhaust memory. Another benefit of grep -z is that the output will contain whole blocks (typically 1–4kB) of text, not just one line.

Instead of grep, you could try a dedicated utility such as PhotoRec (part of TestDisk). Despite the name, it isn't limited to photos. These utilities know the filesystem structure, so they can sometimes recover a deleted file that spanned multiple non-consecutive blocks.

Of course there's never any guarantee that you will be able to recover old data. It may have been overwritten.

Comments

[mikeserv]

Try strings | grep, maybe. And there's always fold or cut or, - as I usually prefer, dd cbs="$length" conv=unblock <blkdev | grep