How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?

text-processing sed awk
17,564

Solution 1

This is difficult in sed because in sed s/TMPHEADERS/"$HEADERS"/, the value of the variable is substituted by the shell before the command reaches sed, so characters like &\/ that appear in $HEADERS have a special meaning in the replacement text. Sed does not have a variable lookup facility.

This task is fairly easy with awk. Pass the HEADERS variable in the environment:

export HEADERS
</test/emailtemplate awk '
    {gsub(/^TMPHEADERS$/, environ["HEADERS"]); print}
' > /test/output

or as an awk variable:

</test/emailtemplate awk -v HEADERS="$HEADERS" '
    {gsub(/^TMPHEADERS$/, HEADERS); print}
' > /test/output

Solution 2

There are several possibilities:

sed You would have to escape every char in "$HEADERS" to make the sed call safe.

HEADERS_ESCAPED="$(echo "$HEADERS" | sed -n 's/./\\&/gp')"
# double-useless use of cat BTW
sed "s/TMPHEADERS/$HEADERS_ESCAPED/" /test/emailtemplate > /test/output

BTW: There is absolutely no reason to call sed once for every change:

sed -n -e s/X/"$X"/ -e s/Y/"$Y" -e s/Z/"$Z"/ -e p inputfile > outputfile

bash itself

while IFS= read -r line; do
  line="${line/TMPDATE/"$TMPDATE"}"
  # ...
  line="${line/TMPHEADERS/"$HEADERS"}"
  printf '%s\n' "$line"
done <inputfile >outputfile

Solution 3

I'm not sure I understand what you want to do. If you want to, for example, replace the string TMPHEADERS with the long list of header info you posted, this Perl solution can do it:

perl -e 'open(A,"header"); ## open the header file
         $h=join("",<A>); ## save its contents in a string
         while(<>){ ## go through the input file
            s/TMPHEADERS/$h/; ## replace
            print        ## print each line
         }' template

The file template is the template from your question and the file header has the header lines you posted. The result is

Alert Report

Alert has triggered at TMPDATE

HEADERS
-------
From [email protected] Thu Apr 25 20:18:19 2013
Return-Path: <[email protected]>
Received: from nm30-vm0.bullet.mail.ne1.yahoo.com (nm30-vm0.bullet.mail.ne1.yahoo.com [98.138.11.36])
by serv.example.com (8.14.4/8.14.4) with ESMTP id r3Q3IJVV009411
for <[email protected]>; Thu, 25 Apr 2013 20:18:19 -0700
Received: from [108.108.108.108] by nm30.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:19 -0000
Received: from [98.138.87.11] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
Received: from [127.0.0.1] by omp1001.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: [email protected]
Received: (qmail 29701 invoked by uid 65501); 26 Apr 2013 03:18:17 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1366946297; bh=yjMzVONHAyJxQob7tLNdIN2RpfGHWVw1Kb5Qr+enYF8=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=CQGilBx8NertE4j75dsfsdfs+IKRzIx5vlH5YdzqFLd4ThfEyMs11sdfsdfsdXsAH1yteACnwSER+QXJQ80BfLGbJnIWm+29I8A4geOPIHGKoOUCnPaD+/0bHfAps0JIcwEju8Tcvg4VDVWw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
s=s1024; d=yahoo.com;
h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
b=5TIRL55VM2J2lPLsX9iCE4sdlkfjlEM2245M6qzg1oGrnZd61ykL4xQSsc3cYNz95fwNy67aRRC89n6xcti28ee5rjmlK0MDIskSB5sKlv165mNjmzF1LNx6uFXalI8QGSwiQt2uWLYvI7RrTVeZFELDfFVZyqygEl5k=;
X-YMail-OSG: EDT4ym4VM1lVQMdtAQ5zqfE59jR1Mtip4vVL1fBzNxFdGvA
YjAJ3MXC1EusloknrsPx3drxzR1b4PFErK.UhdgWePhK7TTCHhhju4XP4i7x
76WASceqp77T5itvZmilv5UuICJw3BCEd0fdADctfBYhLNwoALxjp6cnJMmE
Z4dYVtlp5vUFqg1pHxqGOXqrtjeZffM4dMftnn.Q8LlVEkj3pZ6ZJV_kKFtj
vGEGS5PAW0tIHWPEqVERYzmDOfF5sVSQLayPi6EM_i1OE038434laijEWbH0
nZt1Vkg3syO0t1BaTLN4B1bXeS8cv3GlbLO4ot7zVwA3sH4UhsC5M6xiWNFU
3iroObJ5BObL99VO3ktvC4KzekAWJ_fE85TQJhQKj6Iolgb4xlWa2x414xuA
awO4pJI9grDjycUcmhmKwLZEt_.0OBLfSi5MSviaiCNMuU5qIdHm7VCGdORP
Mc68rDkpmJE9I9Z.QZfhH5cFxqqmpyIOMTs0iIBGYz5d9QHMWCuo-
Received: from [102.102.102.102] by web12341105.mail.ne1.yahoo.com via HTTP; Thu, 25 Apr 2013 20:18:17 PDT
X-Rocket-MIMEInfo: 002.001,VGVzdGluZyAxMjMgTGEgbGEgbGEBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.141.536
Message-ID: <[email protected]>
Date: Thu, 25 Apr 2013 20:18:17 -0700 (PDT)
From: Test Account <[email protected]>
Reply-To: Test Account <[email protected]>
Subject: Test
To: "[email protected]" <[email protected]>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="102743210-1541163991-1366946297=:49149"


SOURCE IP
---------
TMPSOURCEIP

You can extend this for two sets of headers as follows:

perl -e 'open(A,"header1");
         $h1=join("",<A>);
         open(B,"header2");
         $h2=join("",<B>);
         while(<>){
            s/TMPHEADERS/$h1/;
            s/TMPSOURCEIP/$h2/;
            print        
         }' template

Solution 4

Using the m4 macro processor to replace TMPSOURCEIP and TMPDATE with things defined on the command line and TMPHEADERS with whatever is in the file headers.txt. The template is in template.txt.

$ m4 -DTMPDATE="$(date)" -DTMPSOURCEIP="1.1.1.1" -DTMPHEADERS='include(headers.txt)' template.txt
Alert Report

Alert has triggered at Mon Sep 25 18:46:34 CEST 2017

HEADERS
-------
From [email protected] Thu Apr 25 20:18:19 2013
Return-Path: <[email protected]>
Received: from nm30-vm0.bullet.mail.ne1.yahoo.com (nm30-vm0.bullet.mail.ne1.yahoo.com [98.138.11.36])
by serv.example.com (8.14.4/8.14.4) with ESMTP id r3Q3IJVV009411
(etc.)


SOURCE IP
---------
1.1.1.1

If you already have the headers in a variable, you may obviously replace -DTMPHEADERS='include(headers.txt)' with -DTMPHEADERS="$headers".

Share:
17,564

Related videos on Youtube

Mike B
Author by

Mike B

Updated on September 18, 2022

Comments

  • Mike B
    Mike B over 1 year

    CentOS 6.3

    I'm trying to get a small script to send an email containing a copy of email headers in the body (for the purpose of internal reporting).

    The template file contains the following: 

    Alert Report

Alert has triggered at TMPDATE

HEADERS
-------
TMPHEADERS

SOURCE IP
---------
TMPSOURCEIP

    I want my script to find and replace the "TMP" place holders with actual information relevant to the alert.

    The date and source IP variables seem to work fine but if/when I try to find/replace the TMPHEADERS, it causes strange symptoms including the entire output to be blank. I'm reasonably certain that it has to do with the headers containing lots of special characters (@ % & etc).

    Can someone please advise on how best to accomplish this via bash?

    My current code looks like this: 

    cat /test/emailtemplate | sed s/TMPHEADERS/"$HEADERS"/ > /test/output

    UPDATE

    As per request, here's an example set of headers I'm trying to insert into the template file (IP addresses and email address have been changed to protect the innocent :-) ): 

    From [email protected] Thu Apr 25 20:18:19 2013
Return-Path: <[email protected]>
Received: from nm30-vm0.bullet.mail.ne1.yahoo.com (nm30-vm0.bullet.mail.ne1.yahoo.com [98.138.11.36])
by serv.example.com (8.14.4/8.14.4) with ESMTP id r3Q3IJVV009411
for <[email protected]>; Thu, 25 Apr 2013 20:18:19 -0700
Received: from [108.108.108.108] by nm30.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:19 -0000
Received: from [98.138.87.11] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
Received: from [127.0.0.1] by omp1001.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: [email protected]
Received: (qmail 29701 invoked by uid 65501); 26 Apr 2013 03:18:17 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1366946297; bh=yjMzVONHAyJxQob7tLNdIN2RpfGHWVw1Kb5Qr+enYF8=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=CQGilBx8NertE4j75dsfsdfs+IKRzIx5vlH5YdzqFLd4ThfEyMs11sdfsdfsdXsAH1yteACnwSER+QXJQ80BfLGbJnIWm+29I8A4geOPIHGKoOUCnPaD+/0bHfAps0JIcwEju8Tcvg4VDVWw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
s=s1024; d=yahoo.com;
h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
b=5TIRL55VM2J2lPLsX9iCE4sdlkfjlEM2245M6qzg1oGrnZd61ykL4xQSsc3cYNz95fwNy67aRRC89n6xcti28ee5rjmlK0MDIskSB5sKlv165mNjmzF1LNx6uFXalI8QGSwiQt2uWLYvI7RrTVeZFELDfFVZyqygEl5k=;
X-YMail-OSG: EDT4ym4VM1lVQMdtAQ5zqfE59jR1Mtip4vVL1fBzNxFdGvA
YjAJ3MXC1EusloknrsPx3drxzR1b4PFErK.UhdgWePhK7TTCHhhju4XP4i7x
76WASceqp77T5itvZmilv5UuICJw3BCEd0fdADctfBYhLNwoALxjp6cnJMmE
Z4dYVtlp5vUFqg1pHxqGOXqrtjeZffM4dMftnn.Q8LlVEkj3pZ6ZJV_kKFtj
vGEGS5PAW0tIHWPEqVERYzmDOfF5sVSQLayPi6EM_i1OE038434laijEWbH0
nZt1Vkg3syO0t1BaTLN4B1bXeS8cv3GlbLO4ot7zVwA3sH4UhsC5M6xiWNFU
3iroObJ5BObL99VO3ktvC4KzekAWJ_fE85TQJhQKj6Iolgb4xlWa2x414xuA
awO4pJI9grDjycUcmhmKwLZEt_.0OBLfSi5MSviaiCNMuU5qIdHm7VCGdORP
Mc68rDkpmJE9I9Z.QZfhH5cFxqqmpyIOMTs0iIBGYz5d9QHMWCuo-
Received: from [102.102.102.102] by web12341105.mail.ne1.yahoo.com via HTTP; Thu, 25 Apr 2013 20:18:17 PDT
X-Rocket-MIMEInfo: 002.001,VGVzdGluZyAxMjMgTGEgbGEgbGEBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.141.536
Message-ID: <[email protected]>
Date: Thu, 25 Apr 2013 20:18:17 -0700 (PDT)
From: Test Account <[email protected]>
Reply-To: Test Account <[email protected]>
Subject: Test
To: "[email protected]" <[email protected]>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="102743210-1541163991-1366946297=:49149"
    • manatwork
      manatwork almost 11 years
      Simple text manipulation tools have limitations, they are suitable only for basic template handling. Better see Tool to create text files from a template. Has links to similar Stack Exchange and Server Fault questions, see those too.
    • terdon
      terdon almost 11 years
      Could you give us some examples of the headers that fail?
    • Mike B
      Mike B almost 11 years
      @terdon Absolutely. I updated the question to include some example headers (with different ips and email addresses).
  • Hauke Laging
    Hauke Laging almost 11 years
    @terdon Indeed, that was the wrong variable. sed does not expand anything (due to the escaping of every char in the variable). What do you mean? Have you tried this, doesn't it work?
  • terdon
    terdon almost 11 years
    My bad, I always quote my sed commands (I thought that was necessary) and bash variables are not expanded (I mean that 's/foo/$bar/' actually printed $bar and not the variable's contents). Your command worked fine without the quotes (as you had posted it) it failed when I put it in single quotes. Sorry, force of habit, feell free to roll back my edit.
  • Mike B
    Mike B almost 11 years
    Thanks for the prompt help but I'm getting an error: sed: -e expression #1, char 121: unterminated s' command` I'm guessing I must have configured the command wrong: sed "s/TMPHEADERS/$HEADERS_ESCAPED/" /home/foo/scripts/fooalert/emailtemplate > /home/foo/scripts/fooalert/testing1234 I've confirmed that the headers are now all escaped properly.
  • Mike B
    Mike B almost 11 years
    I respect this answer but unfortunately I don't know perl well enough to maintain it on an on-going basis. I realize that's not a good answer (or reason not to use more efficient code). Thanks.
  • Hauke Laging
    Hauke Laging almost 11 years
    @MikeB Can you find out what char causes the problem? I am not sure whether it is possible to have a newline in the replacement text. Is it possible that $HEADERS_ESCAPED contains a newline?
  • dimo414
    dimo414 about 3 years
    +1 to this approach, configuring awk's environment is much less error-prone than trying to wrestle with sed's syntax.

Recents

Why Is PNG file with Drop Shadow in Flutter Web App Grainy?
How to troubleshoot crashes detected by Google Play Store for Flutter app
Cupertino DateTime picker interfering with scroll behaviour
Why does awk -F work for most letters, but not for the letter "t"?
Flutter change focus color and icon color but not works
How to print and connect to printer using flutter desktop via usb?
Critical issues have been reported with the following SDK versions: com.google.android.gms:play-services-safetynet:17.0.0
Flutter Dart - get localized country name from country code
navigatorState is null when using pushNamed Navigation onGenerateRoutes of GetMaterialPage
Android Sdk manager not found- Flutter doctor error
Flutter Laravel Push Notification without using any third party like(firebase,onesignal..etc)
How to change the color of ElevatedButton when entering text in TextField

Related

Print lines between (and including) two patterns
Extracting part of lines with specific pattern using awk or sed
Delete multiple columns using awk or sed
How to append Line to previous Line?
Remove lines based on pattern but keeping first n lines that match
How do I replace multiple lines with single word in file(inplace replace)?
How can I merge files on a line by line basis?
Easily get a particular column from output without sed or awk
How to remove symbols from a column using awk
Delete every Nth line in shell