Delete empty column from csv file with bash script

Question

Looking for some way to look at a csv file and delete columns including the header that have no values in the subsequent lines that follow.

Perhaps if I wanted to delete column Test03 from below including Test03 in the first line.

Test01,Test02,Test03,Test04  
11,22,,44  
11,22,,44  
11,22,,44  
11,22,,44  
11,22,,44  
11,22,,44  

Solution

Here's an awk solution that performs agnostic to whichever columns are empty (ignoring the header).

awk -F, '{
    a[NR]=$0
}NR>1{
    for (i=1;i<=NF;i++) 
        if(length($i)!=0) b[i]++
}END{
    for (k=1;k<=NR;k++) { 
        LINE="" ; 
        split(a[k],c,",") ; 
        for (j=1;j<=NF;j++) 
            if(b[j]>0) 
                LINE=LINE","c[j] ; 
        print substr(LINE,2,length(LINE)-1)
    } 
}' test.csv

Comments

[John1024]

The third column has a value in the first row but not the rest. Do you want to delete the third column in all but the first row?

[bgStack15]

I can help you with an awk command for this operation, but tell me this: Will the second line always be representative of the rest of the following lines?

[user1988900]

Yes the second line and subsequent lines will have no values in column 3.

[user1988900]

Yes that will remove the empty column from lines after the first but I want to remove the column entirely if there are no values but in the first automatically.

[nobody]

This is fine for plain numeric data, but beware that CSV values can contain commas (when properly quoted) and this will not handle that - you would need a full CSV parser.

[user1988900]

Yes aware of cut and it's great if you know which columns are empty but for this I don't always know. Andrew - which CSV parser would you refer to?

[Scott - Слава Україні]

I'm not a big fan of the style of starting the next stanza on the same line as the } of the previous one, but, otherwise, this looks like a good answer.  Welcome to Super User, and good job!  I hope you continue to make contributions as good as this.