Why does the column command not align columns properly?

Question

In another question I asked on how to hide all .pyc files when using ls, Ignacio suggested the following:

This, as I mentioned above, doesn't work precisely, since the output is occasionally misaligned:

ceasarbautista@hse140:~/Desktop/Statistics/statistics/markov$ ls
README          __init__.pyc        markov.py       matrix2graph.pyc    pathfinder.pyc      priority_dict.pyc   spanning.py
__init__.py     graph.py        matrix2graph.py     pathfinder.py       priority_dict.py    space.py        vector.py
ceasarbautista@hse140:~/Desktop/Statistics/statistics/markov$ ls | grep -v '\.pyc$' | column
README      graph.py        matrix2graph.py priority_dict.py    spanning.py
__init__.py markov.py   pathfinder.py   space.py        vector.py

While I got a satisfactory answer, I'm curious: why does column do this (and can it be fixed to print correctly)?

Solution 1

When called without any options, column makes each delimited strings align to the nearest TAB-STOP COLUMN. In a terminal, that is typically on every 8th character column.. Have a look at this example:

Create a file (ztxt) containing some tab-character \t delimited strings, spread over 3 lines which end with newline-character \n:

aaa1\taaaaaaaaaaaaaa2\taaaaaaaaaaaaaaaaaaa3\taaa4
bbbbbbb1\tbbb2\tbbb3
ccc1\tccc2

---

Output of: column ztxt - perfectly aligned to the nearest TAB-STOP

aaa1    aaaaaaaaaaaaaa2 aaaaaaaaaaaaaaaaaaa3    aaa4
bbbbbb1 bbb2    bbb3
ccc1    ccc2
|       |       |       |       |       |       |
|-------|-------|-------|-------|-------|-------|

---

To align the LSH of each successive field to the one above it, you need to use the -t option, eg. column -t ztxt

aaa1     aaaaaaaaaaaaaa2  aaaaaaaaaaaaaaaaaaa3  aaa4
bbbbbb1  bbb2             bbb3
ccc1     ccc2

---

If all your data is in one long stream without any newline breaks, you can uss a filter introduce them; eg one every 4 fields. sed can do it with this command.

sed -re 's/(([^\t]*\t){3}[^\t]*)\t/\1\n/g' 

By default, the column command will merge multiple adjacent delimiters into a single delimiter.. To cater for this in the sed filter, it also needs:

sed -re 's/\t+/\t/g;' 

So the command to split up a continuous stream of tab-delimited strings, as every 4th string is:

<ztxt sed -re 's/\t+/\t/g;s/(([^\t]*\t){3}[^\t]*)\t/\1\n/g' | column -t  

The output of such a contionuous input stream is (using the original sample input, but modified by replacing the original newlines with tab-characters--it must still keep its trailing \n):

aaa1      aaaaaaaaaaaaaa2  aaaaaaaaaaaaaaaaaaa3  aaa4
bbbbbbb1  bbb2             bbb3                  ccc1
ccc2

Solution 2

For the multiple delimiters bit. Annoyingly the -n option is only available in Debian, which handled multiple delimiters.

column -t -n <file>

Solution 3

To show non-printable characters (as C escape codes whenever possible) in your ls output you also can use the -b option to the ls command:

ls -Cb | grep -v '\.pyc$' | column -t

Comments

[Admin]

What does shoving it through od -c say?

[Admin]

gist.github.com/1308298

[Admin]

In the output shown above (and in your other post), even the ls output is misaligned. Perhaps it's something with your terminal. Trying adding | cat -A to the end of the pipeline, which will expose any hidden control or escape sequences that might be corrupting the output.

[Admin]

So I don't have a -A (I'm using OS Lion if that matters), but presume -evt ought to do the same thing: gist.github.com/1308445

[Admin]

have you tried something like this? ls -1 |column -c `tput cols` |column -t basically I used column 2 times, looks non-sense but actually works for my case here at least

[Aquarius Power]

any idea why this ls * -1 |column -t fails horribly? in my case, using the column or not, give the same (one column) output. I've been trying to use column for a long long time and it never worked properly :(. I wonder if there is some column alternative that we can fix the column width even with loss of displayed data to finally have an actual pretty looking and easy to read output. My last attempt will be to code some script to do all that :/

[underscore_d]

and this isn't the accepted answer because..? great detailed post.

[underscore_d]

"I don't see how to make it split on a single tab." What, does -s'[tab]' not work for you? Where, as usual, [tab] means either type a literal tab in your script or press Ctrl+V, Tab at the shell prompt.

[underscore_d]

Non-printable characters are not relevant here; the issue is the alignment, and it's off for the reasons the top-voted answer explained, not because of control or oddly spaced characters in filenames.

[mchid]

@underscore_d Detailed but have you tried this? -t option prints out one single column so this doesn't work unless the file is already in columns to begin with.

[mchid]

You can post the script here so that people can copy and paste the script. We have no idea what is in the compressed file so this is shady. Most of the time people post scripts in answers.