Shell Scripting unwanted '?' character at the end of file name

Question

I get an unwanted '?' at the end of my file name while doing this:

emplid=$(grep -a "Student ID" "$i".txt  | sed 's/(Student ID:  //g' | sed 's/)Tj//g' ) 
 #gets emplid by doing a grep from some text file
echo "$emplid"   #prints employee id correctly 
cp "$i" "$emplid".pdf  #getting an extra '?' character after emplid and before .pdf

i.e instead of getting the file name like 123456.pdf , I get 123456?.pdf . Why is this happening if the echo prints correctly? How can I remove trailing question mark characters ?

Solution

It sounds like your script file has DOS-style line endings (\r\n) instead of unix-style (just \n) -- when a script in this format, the \r gets treated as part of the commands. In this instance, it's getting included in $emplid and therefore in the filename.

Many platforms support the dos2unix command to convert the file to unix-style line endings. And once it's converted, stick to text editors that support unix-style text files.

EDIT: I had assumed the problem line endings were in the shell script, but it looks like they're in the input file ("$i".txt) instead. You can use dos2unix on the input file to clean it and/or add a cleaning step to the sed command in your script. BTW, you can have a single instance of sed apply several edits with the -e option:

emplid=$(grep -a "Student ID" "$i".txt  | sed '-e s/(Student ID:  //g' -e 's/)Tj//g' -e $'s/\r$//' )

I'd recommend against using sed 's/.$//' -- if the file is in unix format, that'll cut off the last character of the filename.

Comments

[sampson-chen]

You used cp "$i" "$emplid".pdf but no .txt on "$i", is the file being copied correctly? If so, try removing the quotes around $i and $emplid on your cp line

[pbhd]

Do echo "$emplid>xxx.txt and then check the contents of the file with e.g. od -c xxx.txt and see whats at the end.

[premprakash]

@sampson-chen : I tried cp $i.txt $emplid.pdf , cp "$i".txt "$emplid".pdf . No luck .

[premprakash]

@pbhd : I get this 0000000 0 0 0 0 1 5 8 5 7 \r \n 0000013

[pbhd]

Hmm. Then answer of Gordon applies. Either you need to reformat your files with dos2unix to get rid of the cr lf sequence (here seen as \r (\n comes from the echo-command)), or as a quick hack, do a sed 's/.$//' to get rid of the last chars on the line. Or even use sed 's/\r$//', but not sure wether thats the right syntax inside sed to specify cr

[premprakash]

@pbhd : sed 's/.$//' worked . Thanks .

[premprakash]

Yes , I guess you are write . I did what pbhd mentioned and I see this 0000000 0 0 0 0 1 5 8 5 7 \r \n 0000013 . How can I remove \r ? to make my cp command work ?

[premprakash]

I Used dos2unix -k "$emplid".pdf after the cp command . It din't work . Any suggestions ?

[pbhd]

Nah, you have to clean your input file, thatone you are greping for the Student-id, because this one gives you the rubbish-char at end.

[premprakash]

@pbhd : What do you mean by clean my input file ? Should I delete unwanted character ? Please advice..

[pbhd]

in that line: ###emplid=$(grep -a "Student ID" "$i".txt | sed 's/(Student ID: //g' | sed 's/)Tj//g' ) ### you are reading a file $i.txt. That file needs ro be cleaned/converted with dos2unix.

[premprakash]

@Gordon : Can you explain what this part does ? sed '-e s/(Student ID: //g' -e 's/)Tj//g' -e $'s/\r$//' )

[Gordon Davisson]

@premprakash: It's a single sed command, applying three edits: the first removes all occurrences of "(Student ID: ", the second removes all occurrences of ")Tj" (those two are directly from your original), and the third removes a carriage return at the end of the line (if it exists). Note that in $'...' bash substitutes escape sequences, so \r gets turned into a carriage return (and the $ after it makes it only match at the end of the line).

[Ungeheuer]

I was testing how fgets() works when it leaves text in stdin stream. It created a file using what was left in the stream, and appended a question mark after the end of the file extension.