Environment variables are not set when my function is called in a pipeline

Question

I have the following recursive function to set environment variables:

function par_set {
  PAR=$1
  VAL=$2
  if [ "" != "$1" ]
  then
    export ${PAR}=${VAL}
    echo ${PAR}=${VAL}
    shift
    shift
    par_set $*
  fi
}

If I call it by itself, it both sets the variable and echoes to stdout:

$ par_set FN WORKS
FN=WORKS
$ echo "FN = "$FN
FN = WORKS

Redirecting stdout to a file also works:

$ par_set REDIR WORKS > out
cat out
REDIR=WORKS
$ echo "REDIR = "$REDIR
REDIR = WORKS

But, if I pipe stdout to another command, the variable doesn't get set:

$ par_set PIPE FAILS |sed -e's/FAILS/BARFS/'
PIPE=BARFS
$ echo "PIPE = "$PIPE
PIPE =

Why does the pipe prevent the function from exporting the variable? Is there a way to fix this without resorting to temp files or named pipes?

Solved:

Working code thanks to Gilles:

par_set $(echo $*|tr '=' ' ') > >(sed -e's/^/  /' >> ${LOG})

This allows the script to be called thusly:

$ . ./script.sh PROCESS_SUB ROCKS PIPELINES=NOGOOD
$ echo $PROCESS_SUB
ROCKS
$ echo $PIPELINES
NOGOOD
$ cat log
7:20140606155622162731431:script.sh:29581:Parse Command Line parameters.  Params must be in matched pairs separated by one or more '=' or ' '.
  PROCESS_SUB=ROCKS
  PIPELINES=NOGOOD

Project hosted on bitbucket https://bitbucket.org/adalby/monitor-bash if interested in full code.

Solution 1

Each part of a pipeline (i.e. each side of the pipe) runs in a separate process (called a subshell, when a shell forks a subprocess to run part of the script). In par_set PIPE FAILS |sed -e's/FAILS/BARFS/', the PIPE variable is set in the subprocess that executes the left-hand side of the pipe. This change is not reflected in the parent process (environment variables do not transfer between processes, they are only inherited by subprocesses.

The left-hand side of a pipe always runs in a subshell. Some shells (ATT ksh, zsh) run the right-hand side in the parent shells; most also run the right-hand side in a subshell.

If you want to both redirect the output of a part of the script and run that part in the parent shell, in ksh/bash/zsh, you can use process substitution.

par_set PROCESS SUBSTITUTION > >(sed s/ION/ED/)

With any POSIX shell, you can redirect the output to a named pipe.

mkfifo f
<f grep NAMED= &
par_set NAMED PIPE >f

---

Oh, and you're missing quotes around variable substitutions, your code breaks on things like par_set name 'value with spaces' star '*'.

export "${PAR}=${VAL}"
…
par_set "$@"

Solution 2

This doesn't work because each side of the pipe runs in a subshell in bash, and variables set in a subshell are local to that subshell.

Update:

It looks like it's easy to pass variables from the parent to the child shell, but really hard to do it the other way. Some workarounds are named pipes, temp files, writing to stdout and reading in the parent, etc.

Some references:

http://mywiki.wooledge.org/BashFAQ/024
https://stackoverflow.com/q/15541321/3565972
https://stackoverflow.com/a/15383353/3565972
http://forums.opensuse.org/showthread.php/458979-How-export-variable-in-subshell-back-out-to-parent

Comments

[savanto]

@Andrew Still trying to find a workaround, but failing! =)

[Andrew]

@Savanto-Thanks. I did not know that about $$ vs $BASHPID or the pipe forcing subshells.

[Gilles 'SO- stop being evil']

That's true, but irrelevant: Andrew is trying to use the variables after the pipeline ends, not on the other side of the pipe.

[mikeserv]

@Gilles - well he can do that. |pipeline | sh

[Andrew]

Overall goal is system monitoring scripts. Would like to be able to call them interactively, from other scripts, or from cron. Want to be able to pass parameters by setting env vars, passing on command line, or config file. Function exists to take input from one or more sources and set the environment correctly. However, I also want to be able to optionally log the output (after running through sed to format), hence the need to pipe.

[mikeserv]

@Andrew - If you want to log a list of current shell env vars do set | sed 's/earch/replace/w /path/to/log'. That will write to a log and /dev/stdout But if you want to pipe it out you can do that too: echo var=val | { . /dev/stdin ; echo $var ; } | sed 's/val/midpipe' outputs midpipe

[Andrew]

@mikeserv - I don't want to log all the env vars. My log emulates syslog to an extent--SEV:TIMESTAMP:SCRIPT:PID:MSG. One of the vars set is the threshold for sev to log. For normal operation, that's fine-one line per event. However, if a script is breaking, I can bump up to debugging. In debug mode I sometimes need additional information. If it is in debug mode, I want to log the variables as they are set, so I can trace through an error. I want the additional info to be indented (via sed) to distinguish them both visually, and for easy parsing, from normal log lines.

[mikeserv]

@Andrew - that makes perfect sense. In that case you can let the shell do more of the work there too, though. Consider shell options -v and -x and -a. Try set -a ; var=val ; export -p for instance - it provides a very easy marker. Just advice though. Your function definitely makes more sense to me now. You can redirect a file-descriptor for a whole function, too so fn() { stuff >&3 ; } ; { fn | cat ; } 3>&1 will display stuff's output. I would use set -v and or -x and do 2>&1 | sed ...parse... >log

[Andrew]

Process substitution for the win! I knew I could use a named pipe or temp file, but those are ugly, have poor concurrency, and leave a mess behind if the script dies (trap helps with the last one). The space thing is intentional. By convention, variables passed on the command line are in name/value pairs and separated by '=' and/or ' '.

[Andrew]

@mikeserv- Thanks for all the comments. I went with Gilles suggestion of process substitution, but having to argue for my code makes me a better programmer.