How to programatically remove empty new lines from a file?

Question

I need to remove empty new lines from text file in terminal.

Example file

123
123
123
123

123



123

I've searched in google a bit but only solution found was sed '/^$/d' myFile > tt and it does not help. I still got these empty new lines.

Solution 1

You can use grep for this purpose.

Try: grep -v ^$ filename.

---

Your command also seems to work:

$ cat t
123
123
123
123

123



123
$ sed '/^$/d' t
123
123
123
123
123
123
$ sed '/^$/d' t >tt
$ cat tt
123
123
123
123
123
123
$

Solution 2

Using AWK: NF is AWK's in-built variable that stores the number of fields in each record. When used by it self, it will print only those lines where NF!=0.

[jaypal~/Temp]$ cat t
123
123
123
123

123



123

[jaypal~/Temp]$ awk 'NF' t
123
123
123
123
123
123

Solution 3

There's probably a 100 ways to do this, this is one:

grep -vE "$^" yourfile

Solution 4

The solution you posted will delete empty lines:

sed '/^$/d' myFile > output

This won't work for lines that contain spaces, or hard tabs, or any other kind of whitespace, including carriage returns. This means, if your text file originated on Windows, that solution probably won't work.

On Unix, lines are simply separated with a line feed. On Windows, they are separated with a carriage return and a line feed.

Here's one way to delete empty lines, compatible with Windows' text format:

sed '/^\x0D\?$/d' myFile > output

If you want to delete lines that contain only whitespace, this should work (on Ubuntu at least):

sed -r '/^\s*\x0D?$/d' myFile > output

Solution 5

Use grep for empty lines:

grep . file

Or try the ex-way (cross-platform compatible) for blank lines (e.g. containing whitespaces):

ex -s +'v/\S/d' -cwq test.txt

For multiple files (edit in-place):

ex -s +'bufdo!v/\S/d' -cxa *.txt

^{Note: The :bufdo command is not POSIX.}

Without modifying the file (just print on the standard output):

cat test.txt | ex -s +'v/\S/d' +%p +q! /dev/stdin

View more solutions

Comments

[Flimm]

The solution you suggested works for me. Are you sure those are empty lines? If they have any spaces or hard tabs in, they won't be deleted.

[ladaghini]

Just confirming @Flimm's answer. Either something is wrong with sed or those aren't really truly empty lines, i.e. they have spaces or tabs in them. If you want to take whitespace into account, try sed '/^[ \t]*$/d' original.txt > emptyRemoved.txt. Also, I hope you realize that tt is the file with the empty lines removed. Your suggested solution doesn't affect myFile.

[Kevin]

As ladaghini pointed out, this won't affect the original file. If you do want it to change the original, use the -i flag.

[Volker Siegel]

+1 for obscuring the obscurity. Mathematicians call that obscurity of the second order, and write it o'' or so. But with a Greek o. And '' is not " of course.