PuPHPet, Vagrant and SSH - SSH Connection Timeout

Question

I built a new CentOS 6.5 vagrant box and after provisioning, the SSH connection keeps timing out. I've attempted the steps in Vagrant ssh fails with VirtualBox but I receive errors (see below). I initially thought that it may be a problem with my /etc/hosts, but after commenting out any potential conflicts I still have the same problem; so, I created an Ubuntu 14.04 vagrant box and that machine booted up flawlessly. Both machine's Vagrantfiles / config.yaml were built using PuPHPet.com.

The host's details:

Host OS: Kubuntu 14.04 (3.13.0-29-generic)
Virtualbox: 4.3.12 r93733
Vagrant: 1.6.0
Basebox: puphpet/centos65-x64   (virtualbox, 1.0.1)

After editing my Vagrantfile to include directives for enabling the GUI and setting the max retries to 150 (per the above thread), I received the following error messages:

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
There are errors in the configuration of this machine. Please fix
the following errors and try again:

SSH:
* The following settings shouldn't exist: max_tries

vm:
* The following settings shouldn't exist: boot_mode

So, I have since removed them. I booted the machine up with VirtualBox, and it came up smoothly.

IP Assigned: 192.168.56.101

The /etc/sysconfig/network-scripts/ifcfg-eth1 has the following config set:

DEVICE="eth1"
BOOTPROTO="none"
IPV6INIT="yes"
MTU="1500"
NM_CONTROLLER="yes"
ONBOOT="yes"
TYPE="Ethernet"
#VAGRANT-BEGIN
NM_CONTROLLER=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.56.101
NETMASK=255.255.255.0
DEVICE=eth1
PEERDNS=no
#VAGRANT-END

[Aside] As I was typing that out (I couldn't copy/paste from the VM CLI unfortunately), I noticed that there was a second declaration for NM_CONTROLLED, so I am going to comment out the NM_CONTROLLED="yes" now just to clean it up. [/Aside]

Moving right along... I'm able to ping the guest from the host; and, I've attempted to ssh using the standard way (ssh [email protected]) but received an error stating No route to host.

$ ping 192.168.56.101
PING 192.168.56.101 (192.168.56.101) 56(84) bytes of data.
64 bytes from 192.168.56.101: icmp_seq=1 ttl=64 time=1.94 ms
64 bytes from 192.168.56.101: icmp_seq=2 ttl=64 time=0.430 ms
^C
--- 192.168.56.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.430/1.187/1.945/0.758 ms

[03:03 PM]-[josej@josej-desktop]-[~/Projects/CentOS-6.5]
$ ssh [email protected]
ssh: connect to host 192.168.56.101 port 22: No route to host

Clearly, there is a route to the host as I can ping it ;). So, I thought that the sshd wasn't running but as we can see below, it is:

[vagrant@localhost ~]$ ps aux | grep sshd
root    1321    0.0    0.1    66608    1200 ?        Ss    20:02    0:00 /usr/sbin/sshd

I was expecting it to refuse the connection since I didn't pass in the SSH key, but since that's not the case and sshd is running.

So, next I tested whether or not the guest could ping Google, which failed. So, checking route we see:

[vagrant@localhost ~]$ route
Kernel IP routing table
Destination    Gateway    Genmask         ... Iface
10.0.2.0       *          255.255.255.0   ... eth0
192.168.56.0   *          255.255.255.0   ... eth1

So, my best guess is there is a networking problem somewhere. Since the host can ping the guest, and the guest can ping the host (ping 192.168.56.1 works), the route file looks good imo, and the networking config is right, but the guest can't reach any destinations beyond the gateway... I am pretty stumped at this point.

I'm going to go poke through the iptables and disable SELinux for the time being. Hopefully that will help flesh out the problem. Anyway, I'm open to any advice as I've been struggling with this for two days now and have only gone in circles.

And, finally, here is the Vagrantfile and puphpet/config.yaml:

require 'yaml'

dir = File.dirname(File.expand_path(__FILE__))

configValues = YAML.load_file("#{dir}/puphpet/config.yaml")
data = configValues['vagrantfile-local']

Vagrant.configure("2") do |config|
  config.vm.box = "#{data['vm']['box']}"
  config.vm.box_url = "#{data['vm']['box_url']}"

  if data['vm']['hostname'].to_s.strip.length != 0
    config.vm.hostname = "#{data['vm']['hostname']}"
  end

  if data['vm']['network']['private_network'].to_s != ''
    config.vm.network "private_network", ip: "#{data['vm']['network']['private_network']}"
  end

  data['vm']['network']['forwarded_port'].each do |i, port|
    if port['guest'] != '' && port['host'] != ''
      config.vm.network :forwarded_port, guest: port['guest'].to_i, host: port['host'].to_i
    end
  end

  data['vm']['synced_folder'].each do |i, folder|
    if folder['source'] != '' && folder['target'] != ''
      nfs = (folder['nfs'] == "true") ? "nfs" : nil
      if nfs == "nfs"
        config.vm.synced_folder "#{folder['source']}", "#{folder['target']}", id: "#{i}", type: nfs
      else
        config.vm.synced_folder "#{folder['source']}", "#{folder['target']}", id: "#{i}", type: nfs,
          group: 'www-data', owner: 'www-data', mount_options: ["dmode=775", "fmode=764"]
      end
    end
  end

  config.vm.usable_port_range = (10200..10500)

  if data['vm']['chosen_provider'].empty? || data['vm']['chosen_provider'] == "virtualbox"
    ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox'

    config.vm.provider :virtualbox do |virtualbox|
      data['vm']['provider']['virtualbox']['modifyvm'].each do |key, value|
        if key == "memory"
          next
        end

        if key == "natdnshostresolver1"
          value = value ? "on" : "off"
        end

        virtualbox.customize ["modifyvm", :id, "--#{key}", "#{value}"]
      end

      virtualbox.customize ["modifyvm", :id, "--memory", "#{data['vm']['memory']}"]

      if data['vm']['hostname'].to_s.strip.length != 0
        virtualbox.customize ["modifyvm", :id, "--name", config.vm.hostname]
      end
    end
  end

  if data['vm']['chosen_provider'] == "vmware_fusion" || data['vm']['chosen_provider'] == "vmware_workstation"
    ENV['VAGRANT_DEFAULT_PROVIDER'] = (data['vm']['chosen_provider'] == "vmware_fusion") ? "vmware_fusion" : "vmware_workstation"

    config.vm.provider "vmware_fusion" do |v|
      data['vm']['provider']['vmware'].each do |key, value|
        if key == "memsize"
          next
        end

        v.vmx["#{key}"] = "#{value}"
      end

      v.vmx["memsize"] = "#{data['vm']['memory']}"

      if data['vm']['hostname'].to_s.strip.length != 0
        v.vmx["displayName"] = config.vm.hostname
      end
    end
  end

  if data['vm']['chosen_provider'] == "parallels"
    ENV['VAGRANT_DEFAULT_PROVIDER'] = "parallels"

    config.vm.provider "parallels" do |v|
      data['vm']['provider']['parallels'].each do |key, value|
        if key == "memsize"
          next
        end

        v.customize ["set", :id, "--#{key}", "#{value}"]
      end

      v.memory = "#{data['vm']['memory']}"

      if data['vm']['hostname'].to_s.strip.length != 0
        v.name = config.vm.hostname
      end
    end
  end

  ssh_username = !data['ssh']['username'].nil? ? data['ssh']['username'] : "vagrant"

  config.vm.provision "shell" do |s|
    s.path = "puphpet/shell/initial-setup.sh"
    s.args = "/vagrant/puphpet"
  end
  config.vm.provision "shell" do |kg|
    kg.path = "puphpet/shell/ssh-keygen.sh"
    kg.args = "#{ssh_username}"
  end
  config.vm.provision :shell, :path => "puphpet/shell/update-puppet.sh"

  config.vm.provision :puppet do |puppet|
    puppet.facter = {
      "ssh_username"     => "#{ssh_username}",
      "provisioner_type" => ENV['VAGRANT_DEFAULT_PROVIDER'],
      "vm_target_key"    => 'vagrantfile-local',
    }
    puppet.manifests_path = "#{data['vm']['provision']['puppet']['manifests_path']}"
    puppet.manifest_file = "#{data['vm']['provision']['puppet']['manifest_file']}"
    puppet.module_path = "#{data['vm']['provision']['puppet']['module_path']}"

    if !data['vm']['provision']['puppet']['options'].empty?
      puppet.options = data['vm']['provision']['puppet']['options']
    end
  end

  config.vm.provision :shell do |s|
    s.path = "puphpet/shell/execute-files.sh"
    s.args = ["exec-once", "exec-always"]
  end
  config.vm.provision :shell, run: "always" do |s|
    s.path = "puphpet/shell/execute-files.sh"
    s.args = ["startup-once", "startup-always"]
  end
  config.vm.provision :shell, :path => "puphpet/shell/important-notices.sh"

  if File.file?("#{dir}/puphpet/files/dot/ssh/id_rsa")
    config.ssh.private_key_path = [
      "#{dir}/puphpet/files/dot/ssh/id_rsa",
      "#{dir}/puphpet/files/dot/ssh/insecure_private_key"
    ]
  end

  if !data['ssh']['host'].nil?
    config.ssh.host = "#{data['ssh']['host']}"
  end
  if !data['ssh']['port'].nil?
    config.ssh.port = "#{data['ssh']['port']}"
  end
  if !data['ssh']['username'].nil?
    config.ssh.username = "#{data['ssh']['username']}"
  end
  if !data['ssh']['guest_port'].nil?
    config.ssh.guest_port = data['ssh']['guest_port']
  end
  if !data['ssh']['shell'].nil?
    config.ssh.shell = "#{data['ssh']['shell']}"
  end
  if !data['ssh']['keep_alive'].nil?
    config.ssh.keep_alive = data['ssh']['keep_alive']
  end
  if !data['ssh']['forward_agent'].nil?
    config.ssh.forward_agent = data['ssh']['forward_agent']
  end
  if !data['ssh']['forward_x11'].nil?
    config.ssh.forward_x11 = data['ssh']['forward_x11']
  end
  if !data['vagrant']['host'].nil?
    config.vagrant.host = data['vagrant']['host'].gsub(":", "").intern
  end
end

---

Config.yaml

---
vagrantfile-local:
    vm:
        box: puphpet/centos65-x64
        box_url: puphpet/centos65-x64
        hostname: null
        memory: '1024'
        chosen_provider: virtualbox
        network:
            private_network: 192.168.56.101
            forwarded_port:
                RVPy8vdfEyb4:
                    host: '5329'
                    guest: '22'
        provider:
            virtualbox:
                modifyvm:
                    natdnshostresolver1: on
                setextradata:
                    VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root: 1
            vmware:
                numvcpus: 1
            parallels:
                cpus: 1
        provision:
            puppet:
                manifests_path: puphpet/puppet
                manifest_file: manifest.pp
                module_path: puphpet/puppet/modules
                options:
                    - '--verbose'
                    - '--hiera_config /vagrant/puphpet/puppet/hiera.yaml'
                    - '--parser future'
        synced_folder:
            Fmuml1NNvztV:
                source: ./
                target: /var/www
                nfs: 'true'
        usable_port_range: 2200..2250
    ssh:
        host: null
        port: null
        private_key_path: null
        username: vagrant
        guest_port: null
        keep_alive: true
        forward_agent: false
        forward_x11: false
        shell: 'bash -l'
    vagrant:
        host: detect
server:
    packages:
        - vim-common
    dot_files:
        -
            bash_aliases: null
    _prevent_empty: ''
mailcatcher:
    install: '1'
    settings:
        smtp_ip: 0.0.0.0
        smtp_port: 1025
        http_ip: 0.0.0.0
        http_port: '1080'
        mailcatcher_path: /usr/local/bin
        log_path: /var/log/mailcatcher/mailcatcher.log
firewall:
    install: '1'
    rules:
        IzA1MvHnvQ4g:
            port: '10000'
            priority: '100'
            proto: tcp
            action: accept
apache:
    install: '1'
    settings:
        user: www-data
        group: www-data
        default_vhost: true
        manage_user: false
        manage_group: false
        sendfile: 0
    modules:
        - php
        - rewrite
    vhosts:
        jeFgLNotkCuO:
            servername: awesome.dev
            serveraliases:
                - www.awesome.dev
            docroot: /var/www/awesome
            port: '80'
            setenv:
                - 'APP_ENV dev'
            override:
                - All
            options:
                - Indexes
                - FollowSymLinks
                - MultiViews
            custom_fragment: ''
            ssl_cert: ''
            ssl_key: ''
            ssl_chain: ''
            ssl_certs_dir: ''
    mod_pagespeed: 0
    mod_spdy: 0
nginx:
    install: 0
    vhosts:
        ejsUD06Uwceq:
            server_name: awesome.dev
            server_aliases:
                - www.awesome.dev
            www_root: /var/www/awesome.dev
            listen_port: '80'
            index_files:
                - index.html
                - index.htm
                - index.php
            envvars:
                - 'APP_ENV dev'
            ssl_cert: ''
            ssl_key: ''
php:
    install: '1'
    version: '55'
    composer: '1'
    composer_home: ''
    modules:
        php:
            - cli
            - intl
            - mcrypt
        pear: {  }
        pecl:
            - pecl_http
    ini:
        display_errors: On
        error_reporting: '-1'
        session.save_path: /var/lib/php/session
    timezone: America/Chicago
xdebug:
    install: '1'
    settings:
        xdebug.default_enable: '1'
        xdebug.remote_autostart: '0'
        xdebug.remote_connect_back: '1'
        xdebug.remote_enable: '1'
        xdebug.remote_handler: dbgp
        xdebug.remote_port: '9000'
drush:
    install: 0
    settings:
        drush.tag_branch: 6.x
mysql:
    install: '1'
    root_password: '123'
    adminer: 0
    databases:
        JibsW5qqEtvE:
            grant:
                - ALL
            name: REDACTED
            host: localhost
            user: REDACTED
            password: '123'
            sql_file: ''
        TUQ8E8xUDyFi:
            grant:
                - ALL
            name: REDACTED
            host: localhost
            user: REDACTED
            password: '123'
            sql_file: ''
postgresql:
    install: '1'
    settings:
        root_password: '123'
        user_group: postgres
        encoding: UTF8
        version: '9.3'
    databases: {  }
    adminer: 0
mariadb:
    install: 0
    root_password: '123'
    adminer: 0
    databases: {  }
    version: '10.0'
mongodb:
    install: '1'
    settings:
        auth: 1
        port: '27017'
    databases: {  }
redis:
    install: '1'
    settings:
        conf_port: '6379'
beanstalkd:
    install: '1'
    settings:
        listenaddress: 0.0.0.0
        listenport: '13000'
        maxjobsize: '65535'
        maxconnections: '1024'
        binlogdir: /var/lib/beanstalkd/binlog
        binlogfsync: null
        binlogsize: '10485760'
    beanstalk_console: '1'
    binlogdir: /var/lib/beanstalkd/binlog
rabbitmq:
    install: 0
    settings:
        port: '5672'
elastic_search:
    install: '1'
    settings:
        java_install: true
        autoupgrade: true

Solution

The problem was that PuPHPet.com's CentOS vagrant boxes no longer come with a default set of firewall rules. So, in order to resolve this, boot up the VM using VirtualBox (or whichever hypervisor you are using), then modify and save /etc/sysconfig/ptables to allow your desired services.

In my personal opinion, the best way to edit your iptables' rules is by using a CLI tool which can easily be installed with yum: sudo yum install -y system-config-frewall-tui. This text-based user interface allows you to easily enable / disable standard services (httpd, bind, postfix, nfs, etc) using the wizard, as well as, create custom rules for non-standard ports / services (such as Webmin's port 10000). This tool is a true time saver.

If this doesn't resolve your connectivity issues, in order to rule out whether or not it is firewall related run sudo service iptables off on the guest and attempt SSH access from the host (ssh vagrant@my-host-name-or-ip).

If you have SSH access now, then the issue is going to be a FW problem and you will need to double check your iptables (did you save your changes?). If you are still unable to get SSH access, then it is most likely a networking problem and should be troubleshooted accordingly (check ifconfig, route, pinging from the host to the guest and vice versa, etc).

HTH.

Comments

[DroBuddy]

SELinux is disabled and I modified my iptables to accept tcp to dport 22. After restarting the service, I am now able to ssh [email protected] but vagrant up is still timing out. I am quite perplexed at this point...

[DroBuddy]

Well, after running sudo service iptables off I am now able to reach google, yum repos, etc. so it is going to be an iptables issue and not a networking issue. Now, the fun part, adding in all of my necessary iptables rules. Argh, this used to be pre-configured for vagrant boxes built using the PuPHPet tool. Apparently this is no longer the case.

[DroBuddy]

Ah, I forgot my beloved tool: system-config-firewall-tui. What a time saver. I highly recommend this to anyone else who happens to encounter this problem.

[Brian Adkins]

This site works best when there is a question and one or more answers. You are encouraged to add an answer to your own question.

[DroBuddy]

Good point. I'll do that. ;)