How to remove a column or multiple columns from file using shell command?

Question

Input:

ARCHIVE  B1_NAME  B2_NAME  B3_NAME  ELEMENT  INFO_NAM WERT PROCID
-------- -------- -------- -------- -------- -------- ---- ------
15MinAvg AIRSS    33-GIS   DMDMGIS1 I        MvAvr15m 1123  CP
15MinAvg AIRSS    33-GIS   DMDMGIS1 P        MvAvr15m 2344  CP
15MinAvg AIRSS    33-GIS   DMDMGIS1 Q        MvAvr15m 4545  CP
15MinAvg AIRSS    33-GIS   DMDMGIS2 I        MvAvr15m 6576  CP
15MinAvg AIRSS    33-GIS   DMDMGIS2 P        MvAvr15m 4355  CP
15MinAvg AIRSS    33-GIS   DMDMGIS2 Q        MvAvr15m 6664  CP

Output:

ARCHIVE  B1_NAME  B2_NAME  B3_NAME  ELEMENT WERT
-------- -------- -------- -------- ------- ----
15MinAvg AIRSS    33-GIS   DMDMGIS1 I       1123
15MinAvg AIRSS    33-GIS   DMDMGIS1 P       2344
15MinAvg AIRSS    33-GIS   DMDMGIS1 Q       4545
15MinAvg AIRSS    33-GIS   DMDMGIS2 I       6576
15MinAvg AIRSS    33-GIS   DMDMGIS2 P       4355
15MinAvg AIRSS    33-GIS   DMDMGIS2 Q       6664

I want to delete the two columns INFO_NAM and PROCID from my input file.

Solution 1

Don't print 6^th and 8^th column

awk '{$6=$8=""; print $0}' file

Solution 2

This has been answered before elsewhere on Stack Overflow:

• delete a column with awk or sed

• Deleting columns from a file with awk or from command line on linux

  etc..

I believe awk is the best for that.

awk '{print $1,$2,$3,$4,$5,$7}' file

It is possible to use cut as well.

cut -f1,2,3,4,5,7 file

Solution 3

The best way to delete columns is with cut:

cut --complement -d' ' -f6,8 file

Explanation:

• -d specifies the delimiter; in this case, a space
• --complement keeps the columns other than the ones specified in -f
• -f specifies the columns to cut (rather than the columns to keep, since --complement is being used); in this case, columns 6 and 8

From the cut man page:

  -d, --delimiter=DELIM
        use DELIM instead of TAB for field delimiter

  -f, --fields=LIST
         select only these fields;  also print any line that contains
         no delimiter character, unless the -s option is specified

  --complement
         complement the set of selected bytes, characters or fields

Some others have proposed using a construction like awk '{$2=$4=""; print $0}', but this doesn't work in cases where the delimiter is something besides space; it makes the contents of the field the empty string, but does not remove the field altogether. For example, echo "a|b|c|d|e" | awk -v FS='|' -v OFS='|' '{$2=$4=""; print}' yields a||c||e.

Solution 4

Use printf to preserve the format of each field. Each field is x characters long and the minus sign justifies the string left.

awk '{ printf("%-8s %-8s %-8s %-8s %-8s %-4s\n", $1, $2, $3, $4, $5, $7)}' file

Solution 5

If you're open to a Perl solution...

perl -ane 'printf("%-8s %-8s %-8s %-8s %-8s %-4s\n", $F[0], $F[1], $F[2], $F[3], $F[4], $F[6])' file

These command-line options are used:

• -n loop around every line of the input file, do not automatically print every line

• -a autosplit mode – split input lines into the @F array. Defaults to splitting on whitespace

• -e execute the following perl code

View more solutions

Comments

[Drav Sloan]

Give an example for cut, or do not mention it?

[Nainita]

awk is working...... but whats happening... the format is changed.... its just taking the field but not the exact structure... and cut is not working... @AndreiR

[Nainita]

I have tried this, cut –c1-45,55-58. But before executing this I have to count the total characters then only I can us this. But I wanted to print the columns with the same as my input format

[AndreiR]

@Nainita maybe you have different separator that is not space. In awk you should specify it (i.e. using -F "\t" if tabular). Using cut, it is necessary to use -d (cut -d, if using comma).

[alhelal]

@AndreiR cut -f1 file doesn't work. see this image

[AndreiR]

@alhelal can you try cut -f 1 -d ' ' ?

[enharmonic]

This doesn't work in cases where the delimiter is something besides space; it makes the contents of the field the empty string, but does not remove the field altogether. See echo "a|b|c|d|e" | awk -v FS='|' -v OFS='|' '{$2=$4=""; print}' yields a||c||e

[Yan Foto]

Best answer: easy to understand, well explained!

[gented]

I don't understand why these answers with $X="" are all over the place on SO: this does NOT omit the column, rather it sets them to the empty string whilst keeping the delimiters around.

[Rick]

MacOS aka BSD doesn't seem to have --compliement

[enharmonic]

@Rick see stackoverflow.com/questions/45851402/… for workaround using brew install coreutils and gcut in terminal.

[Rick]

Ok thanks I will take a look

[Johnny Utahh]

fyi. the cut-based answere below (and here: unix.stackexchange.com/a/615758/36362) works for me, on macOS 10.14.6 anyway.

[Johnny Utahh]

Works for me on macOS 10.14.6 with homebrew coreutils.