permission denied on scripts in ~/bin

Question

I set some additional locations to the PATH environment variable in my ~/.bashrc so that these are included/sourced in logins and non-interactive scripts that are scheduled with cron.

I've noticed though that on one system the PATH is modified correctly, but none of the scripts within will run despite ownership and permissions being set correctly (as far as I can tell).

$ ls -l
total 756
-rw-r-xr-x 1 slackline slackline    300 Sep  6 07:35 backup
-rwxr-xr-x 1 slackline slackline    978 Dec 30 10:28 bbc_mpd
-rwxr-xr-x 1 slackline slackline 355483 Nov 29 07:31 get_iplayer
-rwxr-xr-x 1 slackline slackline    110 Sep  6 07:35 rsync.albums
-rwxr-xr-x 1 slackline slackline    114 Sep  6 07:35 rsync.climbing
-rwxr-xr-x 1 slackline slackline    108 Sep  6 07:35 rsync.films
-rwxr-xr-x 1 slackline slackline    125 Sep  6 07:35 rsync.mixes
-rwxr-xr-x 1 slackline slackline    117 Sep  6 07:35 rsync.pics
-rwxr-xr-x 1 slackline slackline    117 Sep  6 07:35 rsync.torrents
-rwxr-xr-x 1 slackline slackline     95 Sep  6 07:35 rsync.work

The contents of one of my scripts, which synchronizes a directory to my NAS to back it up:

$ cat ~/bin/rsync.work 
#!/bin/bash
source ~/.keychain/$HOSTNAME-sh
/usr/bin/rsync -avz /mnt/work/*  readynas:~/work/.

which fails to run when called:

$ rsync.work
bash: /home/slackline/bin/rsync.work: Permission denied

but works when preceeded with bash -x :

$ bash -x /home/slackline/bin/rsync.work
+ source /home/slackline/.keychain/kimura-sh
++ SSH_AUTH_SOCK=/tmp/ssh-P3GL1A3Juwhe/agent.4209
++ export SSH_AUTH_SOCK
++ SSH_AGENT_PID=4210
++ export SSH_AGENT_PID
+ /usr/bin/rsync -avz /mnt/work/android /mnt/work/arch /mnt/work/classes /mnt/work/doc /mnt/work/linux /mnt/work/lost+found /mnt/work/nc151.tar /mnt/work/nc152now-11.rar /mnt/work/personal /mnt/work/ref /mnt/work/scharr 'readynas:~/work/.'
sending incremental file list

sent 1,176,907 bytes  received 19,786 bytes  30,296.03 bytes/sec
total size is 27,852,538,230  speedup is 23,274.59
$ set -x ; ~/bin/rsync.work ; set +x
+ /home/slackline/bin/rsync.work
bash: /home/slackline/bin/rsync.work: Permission denied
+ set +x
$ set -x ; bash -x ~/bin/rsync.work ; set +x
+ bash -x /home/slackline/bin/rsync.work
+ source /home/slackline/.keychain/kimura-sh
++ SSH_AUTH_SOCK=/tmp/ssh-P3GL1A3Juwhe/agent.4209
++ export SSH_AUTH_SOCK
++ SSH_AGENT_PID=4210
++ export SSH_AGENT_PID
+ /usr/bin/rsync -avz /mnt/work/android /mnt/work/arch /mnt/work/classes /mnt/work/doc /mnt/work/linux /mnt/work/lost+found /mnt/work/nc151.tar /mnt/work/nc152now-11.rar /mnt/work/personal /mnt/work/ref /mnt/work/scharr 'readynas:~/work/.'
sending incremental file list

sent 1,174,755 bytes  received 19,786 bytes  39,165.28 bytes/sec
total size is 27,852,538,230  speedup is 23,316.52
+ set +x

My ~/.bashrc has the following line in it.

$ grep PATH ~/.bashrc
# Additions to system PATH
PATH="/home/slackline/bin:$PATH:/usr/local/stata/:/usr/local/stattransfer/"
export PATH

And I can run the rsync command at the command line myself (so it's not a case of permission being denied on the SSH connection).

$ /usr/bin/rsync -avz /mnt/work/*  readynas:~/work/.
sending incremental file list

sent 1,176,723 bytes  received 19,786 bytes  32,781.07 bytes/sec
total size is 27,852,538,230  speedup is 23,278.17

(Backup is obviously up to date). The version of Bash installed is:

$ eix -Ic bash
[I] app-admin/eselect-bashcomp (1.3.6@08/29/13): Manage contributed bash-completion scripts
[I] app-shells/bash (4.2_p45@08/16/13): The standard GNU Bourne again shell
[I] app-shells/bash-completion (2.1@08/28/13): Programmable Completion for bash
[I] app-shells/gentoo-bashcomp (20121024@08/28/13): Gentoo-specific bash command-line completions (emerge, ebuild, equery, repoman, layman, etc)
Found 4 matches.

The permissions on the directory (and its structure) are:

$ ls -l ~/ | grep bin
drwxr-xr-x  2 slackline slackline 4096 Dec 30 10:29 bin
$ stat -c"%n (%U) %a" / /home /home/slackline /home/slackline/bin
/ (root) 755
/home (root) 755
/home/slackline (slackline) 755
/home/slackline/bin (slackline) 755

And an strace shows

$ strace rsync.work
strace: Can't stat 'rsync.work': No such file or directory
$ echo $PATH
/home/slackline/bin:~/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.8.2:/usr/games/bin:/usr/local/stata/:/usr/local/stattransfer/:/usr/local/stata/:/usr/local/stattransfer/
$ ls -l ~/bin/ | grep work
-rwxr-xr-x 1 slackline slackline     95 Sep  6 07:35 rsync.work
$ rsync.work 
bash: /home/slackline/bin/rsync.work: Permission denied

I can't work out what's going wrong here and would be grateful of any thoughts/ideas on how to trouble shoot this.

EDIT : Tidied up the various edits made in response to questions to hopefully read a bit more coherently and make it easier to follow what I'd tried and how it fits in with Mark Plotnick's solution.

Solution

You mentioned in the comments that your home directory's filesystem is mounted with the users mount option.

$ grep home /etc/fstab
LABEL=home /home ext4 noatime,users 0 4 –

The users mount option implies noexec. From mount(8):

users Allow every user to mount and unmount the filesystem.
      This option implies the options noexec, nosuid, and nodev (unless 
      overridden by subsequent options, as in the option line 
      users,exec,dev,suid).

Comments

[Angel Todorov]

What are the permissions of the directory? ls -l .

[TryingToLearn]

Directory permissions now included, they're owned by slackline:slackline and read/write/execute for owner and read/execute for group and others.

[yorkshiredev]

Can you give us a complete permissions scheme, using stat -c"%n (%U) %a" / /home /home/slackline /home/slackline/bin maybe ?

[TryingToLearn]

complete scheme added, can't see anything wrong myself, but I may not have a proper grasp of permissions.

[X Tian]

what does bash -x /home/slackline/bin/rsync.work give you ? also try bash -x /home/slackline/tmp/android/helium/linux/run.sh

[TryingToLearn]

Ahha, thats looking promising for ~/bin but not for the other script, output added above.

[phemmer]

This can be solved by 2 things. 1) what is the shebang line of the script (the very first line which starts with #!. 2) the output of strace rsync.work.

[TryingToLearn]

1) all of my scripts in ~/bin/ start with #!/bin/bash (was in the example near the top); 2) I've added the output above.

[Mark Plotnick]

Is the filesystem containing your home directory mounted noexec?

[Mark Plotnick]

OK, that explains it. users implies noexec. You need to override that by including exec in the options.

[TryingToLearn]

Thats solved it, thank you very much. Was a little too hasty when migrating my install to its new HD and didn't copy over /etc/fstab faithfully to the new install, thank you again.