Running a bash script from systemd as if I logged in

Question

I have a service implemented as a Bash script. Inside it's actually a node.js application, but it could be anything for the purposes of this question.

The basic use case is that we develop/debug the script in an interactive shell, then enable it as a service.

If I ssh to my device (logging in) my entire environment is there, and I can launch my script at the command line, no problem. The shell I'm in has picked up the right environment from .bashrc, .profile, etc. and everything works.

But if I launch it from a systemd service, it source the environment as it would for an interactive shell. There are provisions to add environment variables to the unit, but for what I'm doing here that just duplicates what my interactive environment does and becomes an easy way to forget something.

For a real production service I'm sure there's a Right Way of doing this, but this is a lab application and I need to make it easy so no one working on the script needs to know how systemd works.

So the question: What do I need to do to get a script written to run under an interactive Bash shell to run from systemd?

I have tried launching bash with --login, but that doesn't work. Nor does explicitly doing a source .bashrc in an outer script.

Here's an example I'm working with to highlight the problem:

core script (job.sh):

#!/bin/bash
cd
while [ 1 ]
do
    pwd
    echo USER is $USER
    echo PATH is $PATH
    which node
    node --version
    sleep 1
done

I call it from another script (wrapper.sh):

#!/bin/bash
echo starting wrapper...
bash --login /home/droid/job.sh

and here is my job.service unit in /etc/systemd/system/job.service:

[Unit]
Description=Job Daemon

[Service]
User=droid
ExecStart=/home/droid/wrapper.sh

[Install]
WantedBy=multi-user.target

What do I not understand about systemd that prevents me from getting this to work?

Solution 1

The difference between running the script via systemd and running it directly is the environment. You can test it like this. In your Unit file, add this to the [Service] section, for testing:

StandardOutput=console

Then in your bash script, at the top add this line to dump the environment:

env

Now run the script inside and outside of systemd and compare the environment variables that are dumped.

It's a feature of systemd that it tightly controls the environment. This both improves security and provides consistency.

You can read more about how systemd manages the environment in the systemd.exec.

Getting something to run the same via the CLI and via system is easy once once you have it running as you'd like from systemd. Run it via the CLI like this:

systemctl start your-unit-name

Then systemd will run with the exactly the same environment.

You asked for the opposite: To have systemd run the service using your "bash" environment. But that is a messy, changing environment which can lead to inconsistent results. By contrast, the systemd runtime environment is strictly controlled, producing consistent, reproducible results. That's why should turn the question inside out and ask how you can your services on the CLI using the same, consistent, controlled execution environment that systemd uses.

Solution 2

While .profile, is read by any login instance of the shell, .bashrc is explicitly intended only to be sourced by interactive non-login shells. Indeed, depending on your distro, the default .bashrc may have something like this at the top to make extra sure the file is only processed by interactive shells:

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

This is likely why manually sourcing .bashrc didn't work.

You could probably get it to work by specifying --login and ensuring that the needed environment variables are set in .profile rather than .bashrc. Probably better, though, would be to have a separate file of environment variables specifically for this service, and ensure that the file is sourced both from the systemd service and when running it manually.

Comments

[Jim B.]

I understand it's messier, and if this was a real product I was going to ship to customers, I would package it as you describe, but the truth is this is a lab environment, and I can't teach everyone involved in the project how to understand the nuances of the systemd environment. Are you telling me that systemd intentionally prevents me from creating my own shell downstream from it and managing environment variables?

[Jim B.]

If you can point me to something that describes how systemd pulls off this feat of preventing me from setting environment variables in the bash shell I launched, I'll call that my answer. :-)

[Jim B.]

Better yet, I'm looking for a workaround here. Barring something like this I'll just use /etc/rc.local, which I know is a hack but it meets my requirement.

[Mark Stosberg]

Updated my answer to fix the link systemd.exec docs, particularly the section on Environment= and EnvironmentFile=. Use those to set the environment in your systemd execution environment.

[Jim B.]

Pardon my ignorance of the details of how processes work, but the docs say this about "Environment=" in systemd: "Sets environment variables for executed processes". Does this mean the execution environment can intercede in a child's bash shell to prevent it from modifying the environment? I assumed the environment was something created by the shell, but this tells me that bash depends on a higher power to manage the environment.

[Mark Stosberg]

For safety and consistency, when systemd launches a process, it starts with nearly-blank slate and adds in the environment variables you define you in the systemd unit file. It inherits almost nothing from the environment it's run in. If you start a bash process from systemd, it can set whatever environment variables it likes, but it won't inherit any from an outer bash environment that may have run a systemd command to start a service.

[Jim B.]

OK, this brings me back to this: Can I get bash to pick up my .bashrc or .profile when a bash script is launched by systemd?

[Mark Stosberg]

I think so, just explicitly source the files from the bash process run via systemd: try source ~/.bashrc

[Jim B.]

Yah, that doesn't work. It's like systemd has some magic mojo that prevents bash from sourcing .bashrc. I can replicate the environment with explicit export commands, but that violates the premise of what I'm trying to do.