Jenkins - Publish Over SSH - UNSTABLE - Permission denied

Question

Our jenkins deploy process checks out our git repo into the workspace and then copies the files from the workspace into our staging environment, using the publish over SSH plugin.

2 Days ago we had a commit and automatic deploy at 10:34 AM and at 02:13 PM. The AM commit deployed fine, since the PM commit, the publishing job reports this:

BUILD SUCCESSFUL
Total time: 15 seconds
SSH: Connecting from host [...]
SSH: Connecting with configuration [...] ...
SSH: Disconnecting configuration [...] ...
ERROR: Exception when publishing, exception message [Permission denied]
Build step 'Send files or execute commands over SSH' changed build result to UNSTABLE

The commit itself was a minor commit, containing css and javascript files for the frontend. Nothing spectacular.

The entire staging environment has been recursively set to chmod 777 and chown'd to the user and group that is used by the deploy over ssh configuration (temporarily, to make sure we are not running into file permission issues)

Still the task reports permission denied.

The configuration for the deploy over ssh task has no changed in the last 2 months. Same keys, same user, same passwords, etc.

Sadly the error message does not contain any helpful details.

Another job that deploys a different project into the same webserver only in a different webroot works just fine.

Server is running FreeBSD and is stationed at a hoster.

Before we contact them, we want to make sure we have checked everything on our side.

Solution

The problem was that our Jenkins job does not clear its workspace. Ever.
There was a folder in the workspace that was originally on git, but was taken out, and resides outside the web root. But since it doesn’t get cleaned, artifacts of that folder were still remaining in the workspace.

Now someone made a change to that folder on staging (directly via SSH) and forgot to set the permissions correctly. So when the "publish over ssh" job jumped in, it tried to copy the old artifacts, but didn’t have permissions.

Since the folder resides outside of the webroot, which we have chown’d and set to 777 for testing, we failed to see it.

The Jenkins job has been changed to clear the workspace after each job and the permissions have been set correctly. Deployment is now working again.

@Others: Please mind the comment to the original question by Iony. It's very helpful to get to the bottom of things.

Also, in the advanced tab of the publish over ssh job, you can select "verbose output". This will let the console output of the running job be verbose. See Publish over - Plugins Wiki at jenkins-ci.org