puppet master REST API returns 403 when running under passenger works when master runs from command line

Question

I am using the standard auth.conf provided in puppet install for the puppet master which is running through passenger under Nginx. However for most of the catalog, files and certitifcate request I get a 403 response.

### Authenticated paths - these apply only when the client
### has a valid certificate and is thus authenticated

# allow nodes to retrieve their own catalog
path ~ ^/catalog/([^/]+)$
method find
allow $1

# allow nodes to retrieve their own node definition
path ~ ^/node/([^/]+)$
method find
allow $1

# allow all nodes to access the certificates services
path ~ ^/certificate_revocation_list/ca
method find 
allow *

# allow all nodes to store their reports
path /report
method save
allow *

# unconditionally allow access to all file services
# which means in practice that fileserver.conf will
# still be used
path /file
allow *

### Unauthenticated ACL, for clients for which the current master doesn't
### have a valid certificate; we allow authenticated users, too, because
### there isn't a great harm in letting that request through.

# allow access to the master CA
path /certificate/ca
auth any
method find
allow *

path /certificate/
auth any
method find
allow *

path /certificate_request
auth any
method find, save
allow *

path /facts
auth any
method find, search
allow *

# this one is not stricly necessary, but it has the merit
# of showing the default policy, which is deny everything else
path /
auth any

Puppet master however does not seems to be following this as I get this error on client

[amisr1@blramisr195602 ~]$ sudo puppet agent --no-daemonize --verbose --server bangvmpllda02.XXXXX.com
[sudo] password for amisr1: 
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /certificate_revocation_list/ca [find] at :110
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [search] at :110
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110 Could not retrieve file metadata for puppet://devops.XXXXX.com/plugins: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /catalog/blramisr195602.XXXXX.com [find] at :110
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /report/blramisr195602.XXXXX.com [save] at :110

and the server logs show

XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/certificate_revocation_list/ca? HTTP/1.1" 403 102 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadatas/plugins?links=manage&recurse=true&&ignore=---+%0A++-+%22.svn%22%0A++-+CVS%0A++-+%22.git%22&checksum_type=md5 HTTP/1.1" 403 95 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "POST /production/catalog/blramisr195602.XXXXX.com HTTP/1.1" 403 106 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "PUT /production/report/blramisr195602.XXXXX.com HTTP/1.1" 403 105 "-" "Ruby"

thefile server conf file is as follows (and goin by what they say on puppet site, It is better to regulate access in auth.conf for reaching file server and then allow file server to server all)

[files]
  path /apps/puppet/files
  allow *
[private]
  path /apps/puppet/private/%H
  allow *
[modules]
  allow *

I am using server and client version 3

Nginx has been compiled using the following options

nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) 
TLS SNI support enabled
configure arguments: --prefix=/apps/nginx --conf-path=/apps/nginx/nginx.conf --pid-path=/apps/nginx/run/nginx.pid --error-log-path=/apps/nginx/logs/error.log --http-log-path=/apps/nginx/logs/access.log --with-http_ssl_module --with-http_gzip_static_module --add-module=/usr/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/nginx --add-module=/apps/Downloads/nginx/nginx-auth-ldap-master/

and the standard nginx puppet master conf

server {
ssl                on;
listen                     8140 ssl;
server_name        _;

passenger_enabled          on;
passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn; 
passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify; 
passenger_min_instances    5;

access_log                 logs/puppet_access.log;
error_log                  logs/puppet_error.log;

root                       /apps/nginx/html/rack/public;

ssl_certificate            /var/lib/puppet/ssl/certs/bangvmpllda02.XXXXXX.com.pem;
ssl_certificate_key        /var/lib/puppet/ssl/private_keys/bangvmpllda02.XXXXXX.com.pem;
ssl_crl                    /var/lib/puppet/ssl/ca/ca_crl.pem;
ssl_client_certificate     /var/lib/puppet/ssl/certs/ca.pem;
ssl_ciphers                SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_prefer_server_ciphers  on;
ssl_verify_client          optional;
ssl_verify_depth           1;
ssl_session_cache          shared:SSL:128m;
ssl_session_timeout        5m;
}

Puppet is picking up the correct settings from the files mentioned because config print command points to /etc/puppet

[amisr1@bangvmpllDA02 puppet]$ sudo puppet config print | grep conf
async_storeconfigs = false
authconfig = /etc/puppet/namespaceauth.conf
autosign = /etc/puppet/autosign.conf
catalog_cache_terminus = store_configs
confdir = /etc/puppet
config = /etc/puppet/puppet.conf
config_file_name = puppet.conf
config_version = ""
configprint = all
configtimeout = 120
dblocation = /var/lib/puppet/state/clientconfigs.sqlite3
deviceconfig = /etc/puppet/device.conf
fileserverconfig = /etc/puppet/fileserver.conf
genconfig = false
hiera_config = /etc/puppet/hiera.yaml
localconfig = /var/lib/puppet/state/localconfig
name = config
rest_authconfig = /etc/puppet/auth.conf
storeconfigs = true
storeconfigs_backend = puppetdb
tagmap = /etc/puppet/tagmail.conf
thin_storeconfigs = false

I checked the firewall rules on this VM; 80, 443, 8140, 3000 are allowed. Do I still have to tweak any specifics to auth.conf for getting this to work?

Update

I added verbose logging to the puppet master and restarted nginx; here's the additional info I see in logs

Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Could not resolve 10.209.47.31: no name for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 access[/] (info): defaulting to no access for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 Puppet (warning): Denying access: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
10.209.47.31 - - [10/Dec/2012:18:19:15 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"

On the agent machine facter fqdn and hostname both return a fully qualified host name

[amisr1@blramisr195602 ~]$ sudo facter fqdn
blramisr195602.XXXXXXX.com

I then updated the agent configuration to add

dns_alt_names = 10.209.47.31

cleaned all certificates on master and agent and regenerated the certificates and signed them on master using the option --allow-dns-alt-names

[amisr1@bangvmpllDA02 ~]$ sudo puppet cert sign blramisr195602.XXXXXX.com

Error: CSR 'blramisr195602.XXXXXX.com' contains subject alternative names (DNS:10.209.47.31, DNS:blramisr195602.XXXXXX.com), which are 
disallowed. Use `puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com` to sign this request.

[amisr1@bangvmpllDA02 ~]$ sudo puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com

Signed certificate request for blramisr195602.XXXXXX.com
Removing file Puppet::SSL::CertificateRequest blramisr195602.XXXXXX.com at '/var/lib/puppet/ssl/ca/requests/blramisr195602.XXXXXX.com.pem'

however, that doesn't help either; I get same errors as before. Not sure why in the logs it shows comparing access rules by IP and not hostname. Is there any Nginx configuration to change this behavior?

Solution 1

I compared it with another setup we had on nginx; it seems the problem was due to the properties

ssl_client_header = SSL_CLIENT_S_D
ssl_client_verify_header = SSL_CLIENT_VERIFY

present in master's puppet.conf. Commenting them out from there and leaving the configuration for them as it is in Nginx fixed the problem.

Solution 2

This symptom will crop up if you are used to using Puppet under Nginx/Passenger and you upgrade to passenger 5.0+. I found the solution here.

Passenger 5.0 has replaced "passenger_set_cgi_param" with "passenger_set_header" which, if you've made it this far, you've figured out. What you might've missed however is that "passenger_set_header" automatically prepends the HTTP_ to the value, so you'll need to remove it from your HTTP_X_CLIENT_S_DN and HTTP_X_CLIENT_VERIFY lines.

Comments

[Anadi Misra]

** Update **: everything works fine if I run puppet master through command line; and then restart the agent. But gives errors again when I run master as a passenger app in nginx.

[Anadi Misra]

yes that's how it is. they are all owned by user puppet, group puppet, nginx also runs under the same user. The problem only occurs when I run master under nginx, if run it as system service it all works fine, no 403s for these URLs.

[Sander Visser]

phusionpassenger.com/library/config/nginx/reference/… You should pass X-Client-DN X-Client-Verify like this to get it working