.bashrc doesn't export the path?

Question

Every time I login to our company cluster, I first get a Kerberos ticket and then try to submit a job, but it fails to find spark-ubmit:

-bash-4.1$ ./pipeline.runtime/scripts/vpspark-submit.sh crow-env/ /user/gsamaras/crow_spm-1.0.2.3.tar.gz --properties-file $SPARK_CONF_PROPERTIES --num-executors 1 foo.py 
./pipeline.runtime/scripts/vpspark-submit.sh: line 51: spark-submit: command not found

If however, I source the .bashrc, it works fine.

>> cat .bashrc
[ -f /etc/bashrc ] && . /etc/bashrc
export PATH=${SPARK_HOME}/bin:$PATH
export SPARK_CONF_DIR=~/tripodrt-spark/conf/
export SPARK_CONF_PROPERTIES=~/tripodrt-spark/conf/spark-overrides-tt.conf

Why? I would expect that .bashrc runs on login. I am on Red Hat Enterprise Linux.

Solution

From the bash manual:

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists.

So if your shell is a login shell, ~/.bashrc is not read.

I would put additions to PATH in ~/.bash_profile instead, and/or possibly source ~/.bashrc from there.

Some shells makes a distinction between two or three types of shell initialization files:

1. A profile file that is read by login shells.
2. A file that is read by interactive shells.
3. And possibly, a file that is read by non-interactive shells, i.e. scripts.

For bash:

1. ~/.bash_profile
2. ~/.bashrc
3. Whatever file $BASH_ENV points to.

The point of separating the profile startup script from the script read by interactive shells is to let you do (potentially heavy) set-up tasks for all your future shell sessions once and for all. This usually involves setting the PATH at the minimum, but also any other exported variables (i.e. environment variables).

The interactive shell initialization script (~/.bashrc in the case of bash) then only needs to do thing specifically needed for interactive shells. This includes setting shell variables like PS1 that shouldn't be exported, and defining aliases and functions etc. You should also set up environment variables that need to be set for this session and any started process from this session, such as GPG_TTY (GPG_TTY="$(tty)") if you're using GnuPG.

I tend to source $HOME/.bashrc from my ~/.bash_profile after setting my PATH and adjusting my umask the way I want them, and let ~/.bashrc set up history file variables, the PS1 prompt, PAGER and EDITOR and a few aliases and functions and whatnot. (That's half a lie; I actually exec the ksh93 shell from my profile if I can find it, but if I can't, I continue with bash). I also renice my shell in the interactive shell initialization file (with renice -n 4 $$), because I often work on a shared machine with loads of other people that needs the CPU more than I do (but that's just me).

The ksh93 shell uses

1. ~/.profile
2. ~.kshrc (or whatever $ENV points to)

ksh93 is a bit saner than bash in that it knows there is such a thing as an interactive login shell, so it reads $ENV (if it is set) for those shell sessions too, after reading ~/.profile.

The pdksh shell also reads ~/.kshrc for non-interactive shells, which annoys me (and breaks a few important scripts that I use), so I have

case $- in
    *i*)    ;;
    *)      return      ;;
esac

at the top of my ~/.kshrc file.