Convert line endings

Question

I have been using d2u to convert line endings. After installing Puppy Linux I noticed that it does not come with d2u, but dos2unix. Then I noticed that Ubuntu is missing both by default.

What is another way to convert line endings?

Solution 1

Some options:

Using tr

tr -d '\15\32' < windows.txt > unix.txt

OR

tr -d '\r' < windows.txt > unix.txt 

Using perl

perl -p -e 's/\r$//' < windows.txt > unix.txt

Using sed

sed 's/^M$//' windows.txt > unix.txt

OR

sed 's/\r$//' windows.txt > unix.txt

To obtain ^M, you have to type CTRL-V and then CTRL-M.

Solution 2

Doing this with POSIX is tricky:

• POSIX Sed does not support \r or \15. Even if it did, the in place option -i is not POSIX

• POSIX Awk does support \r and \15, however the -i inplace option is not POSIX

• d2u and dos2unix are not POSIX utilities, but ex is

• POSIX ex does not support \r, \15, \n or \12

To remove carriage returns:

awk 'BEGIN{RS="^$";ORS="";getline;gsub("\r","");print>ARGV[1]}' file

To add carriage returns:

awk 'BEGIN{RS="^$";ORS="";getline;gsub("\n","\r&");print>ARGV[1]}' file

Comments

[Davoud Taghawi-Nejad]

on my mac only this works: tr -d '\r' < windows.txt > unix.txt

[Loren]

I learned on mac you cannot use tr to open and write to the same file. That results in a blank file, but writing to a different name works great!

[nhed]

@Loren i think that should be your assumption with any redirection. The destination file is opened before the reading of the source. Some commands let you do "in-place" like sed's -i but use intermediate/backup files anyway

[Ed Morton]

Those awk scripts are GNU awk only due to multi-character RS (more than 1 char in a RS invokes undefined behavior in POSIX so some POSIX awks will silently drop the $ and retain just the ^, others can do whatever else they like), they would produce unexpected results when the getline fails, they will only operate on the first line of the input, and they will corrupt the input file in some situations and if they were fixed to operate on all lines would cause an infinite loop in others by writing to the input file as it's being read. Do not execute those scripts.

[Ed Morton]

These answers are generally correct and you could add awk '{sub(/\r$/,"")}1' windows.txt > unix.tx but be aware that the tr is deleting all \rs from the input, not just those that occur at the end of each line as the perl, sed, and now awk scripts would do.

[PeterM]

This work on the same file, ie. it replace line endings in-place. While the tr solutions require different file as an output.

[jocassid]

Redirecting stdout and stdin in the same line kinda messes with my head so I used: cat windows_newlines.txt | tr -d '\r' > unix_newlines.txt

[Samuel]

sed or tr is not working for me.

[Samuel]

tr -d '\15\32' working in Ubuntu18