multiple puppet masters

Question

I would like to set up an additional puppet master but have the CA server handled by only 1 puppet master. I have set this up as per the documentation here:

http://docs.puppetlabs.com/guides/scaling_multiple_masters.html

I have configured my second puppet master as follows:

[main]
...
ca = false
ca_server = puppet-master1.test.net

I am using passenger so I am a bit confused how the virtual-host.conf file should look for my second puppet-master2.test.net. Here is mine (updated as per Shane Maddens answer):

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.18
PassengerRuby /usr/bin/ruby

Listen 8140

<VirtualHost *:8140>

    ProxyPassMatch ^/([^/]+/certificate.*)$ https://puppet-master1.test.net:8140/$1

    SSLEngine on
    SSLProtocol -ALL +SSLv3 +TLSv1
    SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

    SSLCertificateFile      /var/lib/puppet/ssl/certs/puppet-master2.test.net.pem
    SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppet-master2.test.net.pem
    #SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
    #SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
    # If Apache complains about invalid signatures on the CRL, you can try disabling
    # CRL checking by commenting the next line, but this is not recommended.
    #SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth  1
    # The `ExportCertData` option is needed for agent certificate expiration warnings
    SSLOptions +StdEnvVars +ExportCertData

    # This header needs to be set if using a loadbalancer or proxy
    RequestHeader unset X-Forwarded-For

    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    DocumentRoot /etc/puppet/rack/public/
    RackBaseURI /
    <Directory /etc/puppet/rack/>
            Options None
            AllowOverride None
            Order allow,deny
            allow from all
    </Directory>
</VirtualHost>

I have commented out the #SSLCertificateChainFile, #SSLCACertificateFile & #SSLCARevocationFile - this is not a CA server so not sure I need this. How would I get passenger to work with these?

I would like to use ProxyPassMatch which I have configured as per the documentation. I don't want to specify a ca server in every puppet.conf file.

I am getting this error when trying to get create a cert from a puppet client pointing to the second puppet master server (puppet-master2.test.net):

[root@puppet-client2 ~]# puppet agent --test
Error: Could not request certificate: Could not intern from s: nested asn1 error
Exiting; failed to retrieve certificate and waitforcert is disabled

On the puppet client I have this

[main]

server = puppet-master2.test.net

What have I missed?

Cheers, Oli

Solution 1

This part of the documentation..

ProxyPassMatch ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppet_ca/
ProxyPassReverse ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppet_ca/

..is actually wrong in several ways. ProxyPassReverse can't take a regex (and isn't needed anyway), it's not actually using the requested URL in the request that's sent to the CA, and it can trigger unintentional proxying for non-certificate-related API calls for a node that has certificate in its name.

Instead, use this:

ProxyPassMatch ^/([^/]+/certificate.*)$ https://puppet-master1.test.net:8140/$1

Put it inside your <VirtualHost> block, and you can get rid of the <Proxy balancer://puppet_ca>.

The error you're getting means that you're getting something other than a certificate back from the attempt to retrieve your certificate -- this could be caused by the configuration problem above, but might also be indicative of a different error. Get that config changed out, blow away your /var/lib/puppet/ssl on the client (since the certificate request probably failed too) and see if it's working - if not, add --verbose to a run and we'll see what's going on.

Solution 2

Nope.

Don't do this. If you're looking to scale puppet by having multiple masters, you're going the wrong way about it. I'm well aware that puppetlabs have produced a document that you linked saying how they recommend doing MM puppet, but it's actually far easier to go masterless.

So the best way to scale puppet is to go masterless, where you have a central git (or other DVCS) repository, and clone down a copy of your manifests, and run them locally with puppet apply.

Comments

[Tom O'Connor]

Have you seen this? groups.google.com/forum/?fromgroups=#!topic/puppet-users/… Alternatively.. Is this you from a year ago?!

[Oli]

For those of you who have hit the same issue, I missed this @Oli Aha, yup - add SSLProxyEngine On to your <VirtualHost> block. – Shane Madden 6 hours ago. This is NOT in the additional puppet master documentation. You need to use this method of ProxyPassMatch ProxyPassMatch ^/([^/]+/certificate.*)$ https://puppet-master1.test.net:8140/$1 changing you server url and you need to add this SSLProxyEngine On to your <VirtualHost> block.. Hope this helps.. I am emailing puppet labs to get them to add this to their documentation

[Oli]

Yeah I have considered this. I have multiple geographic sites. It's not necessarily about scaling here, but about running a single puppet master in a geographic site, with all of the PM reporting back to a centralised puppetdb. So the PM is acting like a proxy for the facts that the inventory service uses, so I do not have to worry about opening up guest firewall rules on the gateway. I have gitlab running up in my environment, so your method is doable. If I go masterless, can I still take advantage of puppetdb and dashboard? Or if I wanted to use multiple PMs, then what am I doing wrong?

[ravi yarlagadda]

@Tom It's certainly easier, but it's not appropriate in all environments - particularly if you have sensitive data (passwords, etc.) that is provided to a node via its catalog, or if you want to use anything centralized like the inventory service, dashboard, stored configs, etc. Multi-master setups actually got more pleasant with the SRV record feature in 3.0, I'll concede that they're still a pain in the ass, but they work pretty well once set up.

[Oli]

Thanks for the reply. I have set up my virtual host file as you explained, but its unfortunately not working. I get this on the client... Error: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol. I am unsure my virtual-host.conf file is correct. I have updated my question with a different VH file. Also, openssl s_client -connect puppet:8140 -showcerts returns a valid puppet-master1.test.net CA cert when run on the agent and the second puppet-master, puppet-master2.test.net.

[Oli]

I have noticed that running tcpdump -s 1024 -l -A port 8140 -i eth0 on my 2nd puppet-master shows connections but when I run the same command on my CA puppet-master server, nothing shows up. So it looks like the request from the agent is not hitting my CA. I just need help to confirm my VH file is correct..

[ravi yarlagadda]

@Oli The client's expecting an SSL endpoint, so it'll fail to make its request (and there will be nothing sent to the CA master). Can you turn SSL back on and see what it's doing then?

[Oli]

here is what I have done. I have reverted back to the updated first conf in my question. I have then blown away the ssl directory on PM2 rm -rf /var/lib/puppet/ssl). I have then created a new cert by running a puppet agent --test. Notice in the vh conf file I have commented out the ca SSL paths. Puppet agent -t is running fine on PM2. When I connect a client to PM2 with the ProxyPassMatch for /certificate configured (as above), I get this The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

[ravi yarlagadda]

@Oli Sounds like a 503 error - is there anything in the Apache error log on master2?

[Oli]

Ok - after a light wobble, here is the output from the error_log from PM2 and a cert request from the PC. [Wed Dec 19 03:04:51 2012] [error] [client 10.33.22.160] SSL Proxy requested for puppet-master2.test.net:8140 but not enabled [Hint: SSLProxyEngine] [Wed Dec 19 03:04:51 2012] [error] proxy: HTTPS: failed to enable ssl support for 10.33.22.160:8140 (puppet-master1.test.net)

[ravi yarlagadda]

@Oli Aha, yup - add SSLProxyEngine On to your <VirtualHost> block.

[Oli]

Yep that has fixed my ProxyPassMatch issue.. Thanks! So the PC can now request a cert and the request arrives on the PM1. Great stuff! However, I have signed the cert on the PM1 and run a puppet agent --test on the Puppet-Client (it has server = puppet-master2.test.net in puppet.conf [main]) and am getting this error.. Error: Failed to apply catalog: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca

[Oli]

In the PM2 http error logs, I see this... [Wed Dec 19 03:14:09 2012] [warn] Proxy client certificate callback: (puppet-master2.test.net:8140) downstream server wanted client certificate but none are configured [Wed Dec 19 03:14:09 2012] [error] [client 10.33.22.167] Certificate Verification: Error (19): self signed certificate in certificate chain

[Oli]

I am going to mark this question correct as it has fixed up my issue. I will then try to figure out what is going wrong with the puppet client and the cert error. Thanks so much for your help so far.. really appreciate it.. :)

[ravi yarlagadda]

@Oli Is the CA server running Apache too? It shouldn't require a client certificate - though you might need to tweak its auth.conf to allow the download of the certificate revocation list. On the verification error, try uncommenting SSLCACertificateFile and setting it to SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem. But yeah, our comment stream is getting a little lengthy - it'll probably be cleaner to open up a new question for the current state of things.