Bash script does not continue to read the next line of file
Solution 1
I think that should do the same and seems to be correct:
#!/bin/bash
CSVFILE=/tmp/file.csv
cat "$@" | while read line; do
echo "Executing '$line'"
START=$(date +%s)
eval $line &> /dev/null
END=$(date +%s)
let DIFF=$END-$START
echo "$line, $START, $END, $DIFF" >> "$CSVFILE"
echo "It took ${DIFF}s"
done
no?
Solution 2
ffmpeg reads STDIN and exhausts it. The solution is to call ffmpeg with:
ffmpeg </dev/null ...
See the detailed explanation here: http://mywiki.wooledge.org/BashFAQ/089
Update:
Since ffmpeg version 1.0, there is also the -nostdin
option, so this can be used instead:
ffmpeg -nostdin ...
Solution 3
I just had the same problem.
I believe ffmpeg is responsible for this behaviour.
My solution for this problem:
1) Call ffmpeg with an "&" at the end of your ffmpeg command line
2) Since now the skript will not wait till completion of the ffmpeg process, we have to prevent our script from starting several ffmpeg processes. We achieve this goal by delaying the loop pass while there is at least one running ffmpeg process.
#!/bin/bash
cat FileList.txt |
while read VideoFile; do
<place your ffmpeg command line here> &
FFMPEGStillRunning="true"
while [ "$FFMPEGStillRunning" = "true" ]; do
Process=$(ps -C ffmpeg | grep -o -e "ffmpeg" )
if [ -n "$Process" ]; then
FFMPEGStillRunning="true"
else
FFMPEGStillRunning="false"
fi
sleep 2s
done
done
Solution 4
I would add echos before and after the eval to see what it's about to eval (in case it's treating the whole file as one big long line) and after (in case one of the ffmpeg commands is taking forever).
Solution 5
Unless you are planning to read something from standard input after the loop, you don't need to preserve and restore the original standard input (though it is good to see you know how).
Similarly, I don't see a reason for dinking with IFS at all. There is certainly no need to restore the value of IFS before exit - this is a real shell you are using, not a DOS BAT file.
When you do:
read var1 var2 var3
the shell assigns the first field to $var1
, the second to $var2
, and the rest of the line to $var3
. In the case where there's just one variable - your script, for example - the whole line goes into the variable, just as you want it to.
Inside the process line function, you probably don't want to throw away error output from the executed command. You probably do want to think about checking the exit status of the command. The echo
with error redirection is ... unusual, and overkill. If you're sufficiently sure that the commands can't fail, then go ahead with ignoring the error. Is the command 'chatty'; if so, throw away the chat by all means. If not, maybe you don't need to throw away standard output, either.
The script as a whole should probably diagnose when it is given multiple files to process since it ignores the extraneous ones.
You could simplify your file handling by using just:
cat "$@" |
while read line
do
processline "$line"
done
The cat
command automatically reports errors (and continues after them) and processes all the input files, or reads standard input if there are no arguments left. The use of double quotes around the variable means that it is passed as a single unit (and therefore unparsed into separate words).
The use of date
and bc
is interesting - I'd not seen that before.
All in all, I'd be looking at something like:
#!/bin/bash
# Time execution of commands read from a file, line by line.
# Log commands and times to CSV logfile "file.csv"
processLine(){
START=$(date +%s.%N)
eval "$@" > /dev/null
STATUS=$?
END=$(date +%s.%N)
DIFF=$(echo "$END - $START" | bc)
echo "$line, $START, $END, $DIFF, $STATUS" >> file.csv
echo "${DIFF}s: $STATUS: $line"
}
cat "$@" |
while read line
do
processLine "$line"
done
Abs
Updated on June 29, 2022Comments
-
Abs almost 2 years
I have a shell script that saves the output of a command that is executed to a CSV file. It reads the command it has to execute from a shell script which is in this format:
ffmpeg -i /home/test/videos/avi/418kb.avi /home/test/videos/done/418kb.flv ffmpeg -i /home/test/videos/avi/1253kb.avi /home/test/videos/done/1253kb.flv ffmpeg -i /home/test/videos/avi/2093kb.avi /home/test/videos/done/2093kb.flv
You can see each line is an ffmpeg command. However, the script just executes the first line. Just a minute ago it was doing nearly all of the commands. It was missing half for some reason. I edited the text file that contained the commands and now it will only do the first line. Here is my bash script:
#!/bin/bash # Shell script utility to read a file line line. # Once line is read it will run processLine() function #Function processLine processLine(){ line="$@" START=$(date +%s.%N) eval $line > /dev/null 2>&1 END=$(date +%s.%N) DIFF=$(echo "$END - $START" | bc) echo "$line, $START, $END, $DIFF" >> file.csv 2>&1 echo "It took $DIFF seconds" echo $line } # Store file name FILE="" # get file name as command line argument # Else read it from standard input device if [ "$1" == "" ]; then FILE="/dev/stdin" else FILE="$1" # make sure file exist and readable if [ ! -f $FILE ]; then echo "$FILE : does not exists" exit 1 elif [ ! -r $FILE ]; then echo "$FILE: can not read" exit 2 fi fi # read $FILE using the file descriptors # Set loop separator to end of line BAKIFS=$IFS IFS=$(echo -en "\n\b") exec 3<&0 exec 0<$FILE while read line do # use $line variable to process line in processLine() function processLine $line done exec 0<&3 # restore $IFS which was used to determine what the field separators are BAKIFS=$ORIGIFS exit 0
Thank you for any help.
UPDATE 2
Its the ffmpeg commands rather than the shell script that isn't working. But I should of been using just "\b" as Paul pointed out. I am also making use of Johannes's shorter script.
-
Abs about 15 yearsThanks that much clearer! :) Its still only processing parts of the file. I think its due to ffmpeg!
-
Johannes Weiss about 15 yearsI tested it with various commands and it works. Perhaps ffmpeg only runs VERY long?
-
Paul Tomblin about 15 yearsDoes it echo the "It took ..." after the last ffmpeg? Maybe it's not finishing as fast as you think it should?
-
mouviciel about 15 yearsDepending on the conversion, ffmpeg can be very long. You can get a progress status by not filtering out standard error.
-
Abs about 15 yearsYes, its reading some commands wrong. For example, some of the commands are broken "os/mp4/4457kb.mp4 /home/test/videos/done/4457kb.flv " - I have checked there is a line break in the file!
-
Abs about 15 yearsYes, FFmpeg does take some time in some cases but shouldn't the shell script wait until the command is complete??
-
Johannes Weiss about 15 yearsyes it should and I think it will ;-) What exactly does happen?
-
Abs about 15 yearsI just found out quite a few of my commands can not execute due to reasons such as "Unknown format is not supported as input pixel format". I will have to find good sample videos that can actually be executed!
-
Jonathan Leffler about 15 yearsDoesn't '&>' get treated as '&' (run in background) and '>' (redirect - to /dev/null)? Or is it a bash extension of some sort?
-
Johannes Weiss about 15 yearsno &> means redirect stdout AND stderr. The equivalent of > /dev/null 2>&1 or > /dev/null 2> /dev/null
-
Johannes Weiss about 15 yearsAbs, ok but that's a ffmpeg problem not a script problem
-
Magne almost 3 yearsor call ffmpeg with the
-nostdin
option -
mivk almost 3 years@Magne : indeed, this is now a better option. Thanks for resurrecting that old answer so that I could update it.