Pipe input into a script

Question

I have written a shell script in ksh to convert a CSV file into Spreadsheet XML file. It takes an existing CSV file (the path to which is a variable in the script), and then creates a new output file .xls. The script has no positional parameters. The file name of the CSV is currently hardcoded into the script.

I would like to amend the script so it can take the input CSV data from a pipe, and so that the .xls output data can also be piped or redirected (>) to a file on the command line.

How is this achieved?

I am struggling to find documentation on how to write a shell script to take input from a pipe. It appears that 'read' is only used for std input from kb.

Thanks.

Edit : script below for info (now amended to take input from a pipe via the cat, as per the answer to the question.

#!/bin/ksh
#Script to convert a .csv data to "Spreadsheet ML" XML format - the XML scheme for Excel 2003
#
#   Take CSV data as standard input
#   Out XLS data as standard output
#

DATE=`date +%Y%m%d`

#define tmp files
INPUT=tmp.csv
IN_FILE=in_file.csv

#take standard input and save as $INPUT (tmp.csv)
cat > $INPUT

#clean input data and save as $IN_FILE (in_file.csv)
grep '.' $INPUT | sed 's/ *,/,/g' | sed 's/, */,/g' > $IN_FILE

#delete original $INPUT file (tmp.csv)
rm $INPUT

#detect the number of columns and rows in the input file
ROWS=`wc -l < $IN_FILE | sed 's/ //g' `
COLS=`awk -F',' '{print NF; exit}' $IN_FILE`
#echo "Total columns is $COLS"
#echo "Total rows  is $ROWS"

#create start of Excel File
echo "<?xml version=\"1.0\"?>
<?mso-application progid=\"Excel.Sheet\"?> 
<Workbook xmlns=\"urn:schemas-microsoft-com:office:spreadsheet\"
        xmlns:o=\"urn:schemas-microsoft-com:office:office\"
        xmlns:x=\"urn:schemas-microsoft-com:office:excel\"
        xmlns:ss=\"urn:schemas-microsoft-com:office:spreadsheet\"
        xmlns:html=\"http://www.w3.org/TR/REC-html40\">
<DocumentProperties xmlns=\"urn:schemas-microsoft-com:office:office\">
      <Author>Ben Hamilton</Author>
      <LastAuthor>Ben Hamilton</LastAuthor>
      <Created>${DATE}</Created>
      <Company>MCC</Company>
      <Version>10.2625</Version>
</DocumentProperties>
<ExcelWorkbook xmlns=\"urn:schemas-microsoft-com:office:excel\">
        <WindowHeight>6135</WindowHeight>
        <WindowWidth>8445</WindowWidth>
        <WindowTopX>240</WindowTopX>
        <WindowTopY>120</WindowTopY>
        <ProtectStructure>False</ProtectStructure>
        <ProtectWindows>False</ProtectWindows>
</ExcelWorkbook>

<Styles>
      <Style ss:ID=\"Default\" ss:Name=\"Normal\">
            <Alignment ss:Vertical=\"Bottom\" />
            <Borders />
            <Font />
            <Interior />
            <NumberFormat />
            <Protection />
      </Style>
      <Style ss:ID=\"AcadDate\">
      <NumberFormat ss:Format=\"Short Date\"/>    
      </Style> 
</Styles>
<Worksheet ss:Name=\"Sheet 1\">
<Table>
<Column ss:AutoFitWidth=\"1\" />"

#for each row in turn, create the XML elements for row/column
r=1
while (( r <= $ROWS ))
do
   echo "<Row>\n" 
    c=1
    while (( c <= $COLS ))
    do
        DATA=`sed -n "${r}p" $IN_FILE | cut -d "," -f $c `

        if [[ "${DATA}" == [0-9][0-9]\.[0-9][0-9]\.[0-9][0-9][0-9][0-9] ]]; then

            DD=`echo $DATA | cut -d "." -f 1`
            MM=`echo $DATA | cut -d "." -f 2`
            YYYY=`echo $DATA | cut -d "." -f 3`     
            echo "<Cell ss:StyleID=\"AcadDate\"><Data ss:Type=\"DateTime\">${YYYY}-${MM}-${DD}T00:00:00.000</Data></Cell>"
        else        
            echo "<Cell><Data ss:Type=\"String\">${DATA}</Data></Cell>" 
        fi
        (( c+=1 ))
    done
    echo "</Row>"
   (( r+=1 ))
done

echo "</Table>\n</Worksheet>\n</Workbook>"


rm $IN_FILE > /dev/null

exit 0

Solution 1

Commands inherit their standard input from the process that starts them. In your case, your script provides its standard input for each command that it runs. A simple example script:

#!/bin/bash
cat > foo.txt

Piping data into your shell script causes cat to read that data, since cat inherits its standard input from your script.

$ echo "Hello world" | myscript.sh
$ cat foo.txt
Hello world

The read command is provided by the shell for reading text from standard input into a shell variable if you don't have another command to read or process your script's standard input.

#!/bin/bash

read foo
echo "You entered '$foo'"

$ echo bob | myscript.sh
You entered 'bob'

Solution 2

There is one problem here. If you run the script without first checking to ensure there is input on stdin, then it will hang till something is typed.

So, to get around this, you can check to ensure there is stdin first, and if not, then use a command line argument instead if given.

Create a script called "testPipe.sh"

#!/bin/bash
# Check to see if a pipe exists on stdin.
if [ -p /dev/stdin ]; then
        echo "Data was piped to this script!"
        # If we want to read the input line by line
        while IFS= read line; do
                echo "Line: ${line}"
        done
        # Or if we want to simply grab all the data, we can simply use cat instead
        # cat
else
        echo "No input was found on stdin, skipping!"
        # Checking to ensure a filename was specified and that it exists
        if [ -f "$1" ]; then
                echo "Filename specified: ${1}"
                echo "Doing things now.."
        else
                echo "No input given!"
        fi
fi

Then to test:

Let's add some stuff to a test.txt file and then pipe the output to our script.

printf "stuff\nmore stuff\n" > test.txt
cat test.txt | ./testPipe.sh

Output: Data was piped to this script! Line: stuff Line: more stuff

Now let's test if not providing any input:

./testPipe.sh

Output: No input was found on stdin, skipping! No input given!

Now let's test if providing a valid filename:

./testPipe.sh test.txt

Output: No input was found on stdin, skipping! Filename specified: test.txt Doing things now..

And finally, let's test using an invalid filename:

./testPipe.sh invalidFile.txt

Output: No input was found on stdin, skipping! No input given!

Explanation: Programs like read and cat will use the stdin if it is available within the shell, otherwise they will wait for input.

Credit goes to Mike from this page in his answer showing how to check for stdin input: https://unix.stackexchange.com/questions/33049/check-if-pipe-is-empty-and-run-a-command-on-the-data-if-it-isnt?newreg=fb5b291531dd4100837b12bc1836456f

Solution 3

If the external program (that you are scripting) already takes input from stdin, your script does not need to do anything. For example, awk reads from stdin, so a short script to count words per line:

#!/bin/sh
awk '{print NF}'

Then

./myscript.sh <<END
one
one two
one two three
END

outputs

1
2
3

Comments

[Aurovrata]

this is much better answer, especially in the context of the question relating to a csv file, one assumes the pipeed input to contain more than one line.

[Matthias]

Doesn't seem to support multiline input

[chepner]

No, read reads exactly one line (i.e., up to the first linefeed).