How to expand shell variables in a text file?

bash shell unix solaris ksh

27,025

Solution 1

This solution is not elegant, but it works. Create a script call shell_expansion.sh:

echo 'cat <<END_OF_TEXT' >  temp.sh
cat "$1"                 >> temp.sh
echo 'END_OF_TEXT'       >> temp.sh
bash temp.sh >> "$2"
rm temp.sh

You can then invoke this script as followed:

bash shell_expansion.sh Text_File.msh Text_File_expanded.msh

Solution 2

This question has been asked in another thread, and this is the best answer IMO:

export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh

if on Mac, install gettext first: brew install gettext

see: Forcing bash to expand variables in a string loaded from a file

Solution 3

If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):

when test.txt contains

${line1}
${line2}

then:

>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork

Obviously if you just want to print it you can take out the extra value=$() and echo "$value".

Solution 4

If a Perl solution is ok for you:

Sample file:

$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';

Solution:

$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';

Solution 5

One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:

#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15;   # Clean up the temporary file 

(
  echo 'cat <<END_OF_TEXT'
  cat "$@"
  echo 'END_OF_TEXT'
) > $FILE
. $FILE

The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.

Thanks to both previous answers for their ideas!

View more solutions

27,025

Kent Pawar

www.linkedin.com/in/KentPawar https://g.dev/kent

Updated on August 15, 2020

Comments

Kent Pawar almost 4 years
Consider a ASCII text file (lets say it contains code of a non-shell scripting language):

Text_File.msh:
```
spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....
```
Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables. What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:

Text_File_expanded.msh:
```
spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....
```
Consider:
```
$ a=123
$ echo "$a"
123
```
So technically this should do the trick:
```
$ echo "`cat Text_File.msh`" > Text_File_expanded.msh
```
...but it doesn't work as expected and the output-file while is identical to the source.

So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.
- Guru over 11 years
  
  Will your variables be within single quotes as shown above?
- Kent Pawar over 11 years
  
  @Guru - Currently, it is in single quotes or no surrounding quotes at all in the text file.
- abyss.7 about 10 years
  
  related question: superuser.com/q/235738/126693
Kent Pawar over 11 years

Thanks @Hai Vu. I first set the variable $ LOG_FILE_PATH=/some/path then ran $ ./shell_expansion.sh Text_File.msh Text_File_expanded.msh (and also $ bash shell_expansion.sh Text_File.msh Text_File_expanded.msh) - but in the Text_File_expanded.msh the $LOG_FILE_PATH entries are replaced with blanks spaces. I may be doing this incorrectly..
Kent Pawar over 11 years

I had not exported the variables.. After I tried $ export LOG_FILE_PATH=/sone/path/ it worked fine. Thanks! :)
Kent Pawar over 11 years

Thanks @Guru - I tried it and it works fine.. As I need to avoid a dependency on Perl I cannot apply this solution. I am not a Perl expert but are we sure that \w will always capture the shell variable name correctly..?
Kent Pawar over 11 years

Ok I goggled and got my answer - 1. PERL reference: \w will Match "word" character (alphanumeric plus ""). 2. UNIX shell reference: The name of a variable can contain only alphanumeric characters and "".
Kent Pawar over 11 years

Hi All - Typos in my above comment: the "_" was taken as not marked as code and so was used to make the text italics.. Sorry.
Kent Pawar almost 11 years

Thanks @PaulGear! Welcome to StackOverflow. +1 For using trap to clean up temporary files. I have suggested some edits to make it easier for newbies like myself to understand the code; kindly review the edits and revert them if required. thanks
Kent Pawar almost 11 years

I didn't get which signal the 0 refers to.
Paul Gear almost 11 years

0 signifies normal exit rather than a signal.
Paul Gear almost 11 years

I'm not sure why you think semis are necessary on variable assignments and traps but not echos and cats, but the comments are useful so i don't plan to revert.
Kent Pawar almost 11 years

Thanks Paul. About the semis- just added them while adding the comments; not that its necessary, just a force of habit :)
FooF almost 9 years

There is no need to use temporary files, just use eval - see @ThirteenThirtySeven's answer. Also the method of using temporary files is somewhat problematic (multiple instances of shell_expansion.sh will write all to the same file, permission issues, error checking etc - temporary file will create many potential complications you need to be prepared to handle or know to avoid with certainty).
aprodan about 7 years

I tried your script . it works almost fine. It will replace indeed all env variables regardless they are defined or not. I think it would be useful to avoid replacement of variables which are not defined.
aprodan about 7 years

this solution will replace also variables which are not defined. Some times you would like to avoid that and keep variables placeholders.
Wildcard over 6 years

If test.txt contains, for example, "; touch /etc/your_system_is_pwned; echo " you will see Bad Things happen.
mjd2 over 5 years

eval is more dangerous than using temp files.
JacobTDC about 5 years

I find that echo -e "cat <<END_OF_TEXT\n$(cat [filename])\nEND_OF_TEXT" | bash works well without a temp file.
Hai Vu about 5 years

Thanks, @JacobTDC
Jon Gretar almost 5 years

Nice one @JacobTDC. Works perfectly. Watch out for usage of $$ with procedural code, and also don't include the text END_OF_TEXT ;-)
William Pursell over 4 years

@mjd2 How is eval more dangerous that using a temp file? I disagree with the assertion. If the tempfile is kept it makes it easier to do a post-mortem to determine what went wrong, but it provides no additional safety over eval.
Dediche Anonime about 3 years

@JacobTDC is not the same thing. This short syntax interpret all escape characters inside the document and not only the two linefeeds, producing potentially unwanted results.
George Herolyants over 2 years

It won't work assuming $LOG_FILE_PATH contains slashes. It will be simply expanded inside the pattern causing an error like "bad flag in substitute command". Just tried it with $PATH.
ocarlsen over 2 years

Thank you this is awesome
Neil C. Obremski about 2 years

2 things to remember with envsubst is that environment variables must be exported for it to work and it isn't preinstalled on a lot of systems. If on Debian/Ubuntu: sudo apt-get install gettext