Ansible module_stdout: command not found when using sudo on Centos 7

Question

I am connecting to centos 7 virtual and physical machines using ansible with a user called ansible that has passwordless sudo permissions on all the boxes. On most machines it works, but on one box I get this error:

FAILED! => { "changed": false, "module_stderr": "Shared connection to ... closed.\r\n", "module_stdout": "/var/tmp/sclpbsoCZ: line 8: -H: command not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127 }

I set up passwordless sudo using advice from this link: https://code-maven.com/enable-ansible-passwordless-sudo (basically, edit /etc/sudoers to let user ansible sudo without a password)

I thought these machines were identical, but obviously something is amiss.

Adding -vvv to the ansible-playbook command lets me see that the command it is trying to run is:

'/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '

How is it that this one machine isn't working with ansible?

Solution

I found this, but at first blush it seemed unrelated to ansible:

https://unix.stackexchange.com/questions/192809/sudo-i-returns-an-error

It turns out that centos has scl (software collections) that can be enabled.

I had devtoolset-7 enabled at boot:

https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/

This happens to wrap sudo with a script that doesn't seem to actually handle all of the options of the original sudo. It seems like this could be considered a bug (or at least bad behavior) in devtoolset-7.

For now the fix was to remove the entry in /etc/profile.d/ that was enabling the devtoolset-7.

It was running this code:

source /opt/rh/devtoolset-7/enable

If I ssh'd into the machine and ran the offending command:

sudo -H -S -n -u root /bin/sh

It would give me the same error:

# sudo -H -S -n -u root /bin/sh /var/tmp/sclKpdWFR: line 8: -H: command not found

And pointed me to the problem when I queried which sudo was being used...

# which sudo /opt/rh/devtoolset-7/root/usr/bin/sudo

What a rabbit hole!

Comments

[Karthikeyan D]

This is also true with devtoolset-8! Of course if your .bash_profile or .bashrc are using source scl_source enable devtoolset-8 or similar you can't disable it once you're in a shell. You have to comment it out of the file that's running it at shell creation (as you mentioned). I was unable to use become_exe: /usr/bin/sudo to override the sudo Ansible used but someone else may have more luck.

[poleguy]

I was also able to patch the devtolset-7 sudo to work correctly. It's insane to me that RedHat took the risk to replace sudo with a hacky script. I won't even post the hacky fix, but it is possible. This would allow you, @Dan, to get ansible to work without restarting the shell, but really... ick.