Incrond running but not executing commands CentOS 6.4

Question

I have copied this question over here from StackOverflow...

I have installed incron from the EPEL repository (0.5.9) (before you ask; YES, I also tried downloading the source and compiling locally (0.5.10); same results) and am attempting to set up a process on my CentOS 6.4 (final) virtual box that I have successfully prototyped on my local Ubuntu 12.04 machine (process including incron works perfectly under Ubuntu):

Some information up front:

• SELinux is set PERMISSIVE (I have also tried DISABLED, it seems to make no difference!)
• I have added my script(s) and called executables to all with NOPASSWD using visudo to get around some permissions issues...

• I have tried to apply the policy for incron as desribed here http://blog.siphos.be/2013/05/a-selinux-policy-for-incron-finishing-up/ but I keep getting

  libsepol.policydb_read: policydb magic number 0x696c6f70 does not match expected magic number 0xf97cff8c or 0xf97cff8d semodule_package: Error while reading policy module from incron.te

  when I try to use semodule_package

• I have tried to define environment both in the script itself and in a pre-script caller.sh by doing this I am able to env > /tmp/envfile.txt which indicates the script is running in a clone of the sudo env (as I intended) however I still get no output from my script, either the work it is supposed to do on it's input file or any of it's logs...

• it's gotta be permissions right? (I'm about one more day of this away from just doing chmod -r 777 on the root file system ;) )

• Under Ubuntu 12.04 the simple act of doing sudo incrotab -e and entering /tmp/ IN_CREATE,IN_NO_LOOP env > /home/username/envfile.txt shows that incron is already running in the root/sudo environment, as the incron documentation suggests it should be.

• Is this all SELinux-related stuff, or is there something else going on in CentOS thats making this so difficult?

If I invoke my script from the command line with sudo /path/to/my/script.sh arguments it works like a charm.

or for that matter if I invoke caller.sh it all works fine, but when caller.sh is invoked by incrond its the same, can't even env > /dev/pts/0 from it. (though I CAN env > /tmp/envfile.txt

sudo service incrond status verifies that incrond is running. root and myusername are added to /etc/incron.allow, /etc/incron.deny is empty.

the below is deprecated, see bullet points above, but read it for backstory and expansion...

My incrotab for root is:

/path/to/dropfolder/ IN_CLOSE_WRITE sudo /path/to/my/script.sh $@/$#

Events in /path/to/dropfolder/ result in NOTHING useful happening. logs are generated in /var/log/cron, no messages, no actions occur on the files in the folder.

So I have researched: it was suggested that cron runs in a minimal environment and that to execute complex commands/scripts you may have to execute your .bashrc and/or export your PATH at the beginning of the cron command.

Edit: Documentation indicates that incron run from system tables or root takes env from its host enviroment, therefore only incron executed by non-root users should require any sort of env or PATH fiddling

So… incrontab for root:

/path/to/dropfolder IN_CLOSE_WRITE . /home/myusername/.bashrc; export PATH=$PATH:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin; sudo /path/to/my/script.sh $@/$#"

No dice… tried && instead of ; = no dice. If you can think of a variation on the above I've probably tried it…

So, lets try a little restorecond -R /usr/sbin/incrond /etc/incron* action! Huh, no change there either. service incrond stop followed by service incrond start and then service incrond restart … nope, nope, and nope.

Drastic measures: yum remove incron and yum install incron, chkconfig incrond on and then for good measure a sudo reboot!

how about the ol' touch ./autorelabel and reboot? nope!

Nothing.

I don't even get anything from /tmp/ IN_ALL_EVENTS echo boo> > /home/myusername/boofile.txt, hence my inability to even do an env > envfile.txt to check whether incron is running in a sparse environment… (see above note)

And yet: service incrond status yields incrond (pid xxxx) is running...

further examination of /var/log/cron yields results like this: Aug 14 15:05:30 hostname incrond[1584]: (root) CMD (sudo /path/to/DropFolder/script/subfolder/script-Beta-1.sh /home/username/DropFolder/testfile.file) -yes I made sure my script was executable..

if I set root's incrontab to contain /tmp/ IN_ALL_EVENTS,IN_NO_LOOP env > /tmp/envfile.txt I get nothing. tail /var/log/cron contains: Aug 15 10:06:32 hostname incrond[1584]: (root) CMD (env > /tmp/envfile.txt) but the file doesn't exist in /tmp/

so incrond is indeed attempting to do SOMETHING, but I get no output anywhere... even a echo > /dev/pts/0 produces nada results.

if I make root's incrontab /tmp/ IN_ALL_EVENTS,IN_NO_LOOP . ./home/print/.bashrc; env > /tmp/envfile.txt as was suggested in a number of threads I found dealing with cron environment issues I get Aug 15 12:30:25 hostname incrond[1726]: cannot exec process: Permission denied the pid here is different from the one that claims it issued the command, so there's obviously some child process spawning happening here... that was mentioned in the SELinux policy link above, and I feel they're related, but shouldn't PREMISSIVE and DISABLED SELinux settings not care about that??

before I tried setting SELinux to DISABLED I was getting entries in /var/log/audit/audit.log that indicated incron was trying to do things and they were taged res=success at the end... which seems to indicate SELinux was letting stuff happen, but nothing happens! Since setting SELinux to DISABLED and back to PERMISSIVE and rebooting (several times) I'm not getting any entries in /var/log/audit/audit.log related to incrond aside from service start etc related things. WTH?

---

I have tried all of the above described incron tactics as root (sudo incrontab), as a plain user, and though system tables (located in /etc/incron.d) with the same results :p

I have thrown the kitchen sink at this (as far as I understand the contents of the sink) and I can't find a solution...What am I missing? I hope someone can make me feel like an idiot in short order!

Solution 1

Ok, here's the answer:

Despite all documentation I can find declaring it to be the contrary, incrond under CentOS 6.4 runs in a sparse environment and behaves as cron. this is NOT the case under Ubuntu, where incron inherits its environment from root for system tables and root tables and only user tables run in a sparse environment. This of course means that if you're calling a script (I am) the script needs to build it's environment and every thing must have a full path. EVERYTHING. (well, except for shell builtins :p)

numerous Google and and Bing and Stack Overflow and Server Fault searches all told me that cron operates this way, but they all also seemed to indicate that incron operates as described in the documentation, which it DOES under Ubuntu...

Bottom line, it works now, yay!

• I guess this just goes to show: when in doubt define the environment and use full paths... er, actually, just do that all time and save yourself a headache.

(this does not address my problem with applying a SELinux security policy for incron, but I'll worry about that later, in another post...)

Solution 2

I had the same issue. After a lot of trial and error I found that my original line of

/path/to/watch IN_CLOSE_WRITE /usr/local/bin/mycommand $@/$#

does not work but the following does work:

/path/to/watch IN_CLOSE_WRITE /bin/sh /usr/local/bin/mycommand $@/$#

I guess incron doesn't honour scripts with a shebang (yet?) and needs an interpreter within the command.

edit: after a little more testing I found that if the command is a (bash or shell) script it will need the .sh extension or it needs to be prepended with an interpreter like /bin/sh. So the following examples will both work (under CentOS 6.4 at least)

/path/to/watch IN_CLOSE_WRITE /bin/sh /usr/local/bin/mycommand $@/$#
/path/to/watch IN_CLOSE_WRITE /usr/local/bin/mycommand.sh $@/$#

Solution 3

Due to insufficient reputation, my contribution comes as a separate answer rather than a comment on user199085's.

Yes, the OP says centOS but for the ubuntu folks, I also hadd to stick sudo -u my_user_name before the bin/sh bit. So, the line becomes

/path/to/watch IN_CLOSE_WRITE sudo -u my_user_name /bin/sh /usr/local/bin/mycommand $@/$#

Comments

[Janne Pikkarainen]

What if you place your script under /usr/local/bin or some other more "generic" path?

[JawzX]

As of this morning I have managed to get the script to execute when called from incrond (I have it export it's USER PATH SHELL HOSTNAME LOGNAME etc... and then execute /etc/bashrc) , it writes it's logs, and performs the cp and mv functions it's supposed to, but none of the helper applications it calls are running, or if they are they are running unprivileged and are not being allowed to read and write to the filesystem. This despite the script calling them with sudo and the relevant apps being marked NOPASSWD in the sudoers file -they work when called at cli :P

[JawzX]

also of note: /var/log/audit/audit.log shows no evidence of the helper applications running ARGH.

[rams]

thanks for posting this. Can you elaborate on "Script needs to build it's environment". I am a linux noob and not sure how to build the env. Is adding #!bin/bash considered building?