Weird network problems with Vagrant, VirtualBox, Ansible and Ubuntu Server box

Question

We are using Vagrant and Ansible to provision VirtualBox machine.

It was working great, but recently we've started to encounter some weird network problems with tasks that require access to the Internet, like: apt-get update, composer update, gem install etc. These problems emerge from time to time and they are unstable. Sometimes it can't connect to some server or domain name is not resolving.

Interestingly, is when I connect to the box itself via SSH (vagrant ssh) and issue such commands manually, like sudo apt-get update; sudo apt-get dist-upgrade - it's working fine all the time.

Here's the sample error with NPM:

failed: [localhost] => {"cmd": "/usr/bin/npm install --global npm-check-updates", "failed": true, "rc": 1}
stderr: npm http GET https://registry.npmjs.org/npm-check-updates
npm http GET https://registry.npmjs.org/npm-check-updates
npm http GET https://registry.npmjs.org/npm-check-updates
npm ERR! network getaddrinfo ENOTFOUND
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

Here's the sample error with APT:

failed: [localhost] => {"failed": true, "parsed": false}
invalid output was: SUDO-SUCCESS-qryiheatxowjxtcfbzrnzercnnwxtmyv
Traceback (most recent call last):
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1405611978.79-161922279792225/apt", line 1762, in <module>
    main()
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1405611978.79-161922279792225/apt", line 492, in main
    cache.update()
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 440, in update
    raise FetchFailedException(e)
apt.cache.FetchFailedException: W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/main/i18n/Translation-en_US  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/main/i18n/Translation-en  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/main/i18n/Translation-en_US.UTF-8  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/universe/i18n/Translation-en_US  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/universe/i18n/Translation-en  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/universe/i18n/Translation-en_US.UTF-8  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/i18n/Translation-en_US  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/i18n/Translation-en  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/i18n/Translation-en_US.UTF-8  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/i18n/Translation-en_US  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/i18n/Translation-en  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/i18n/Translation-en_US.UTF-8  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/main/source/Sources  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/universe/source/Sources  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/main/binary-i386/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/universe/binary-i386/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/source/Sources  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/source/Sources  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-i386/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, W:Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/binary-i386/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
, E:Some index files failed to download. They have been ignored, or old ones used instead.

Software versions:

• ansible 1.6.6
• Vagrant 1.6.3
• VirtualBox 4.3.14

We are using the following box:

http://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-i386-vagrant-disk1.box

Host OS is Ubuntu 14.04.

We encounter such problems with different computers in absolutely different networks (different cities).

What can be the reason for this? How do we debug it and fix it?

Update #1

Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# "2" is an API version here.
Vagrant.configure("2") do |config|

  config.vm.box = "trusty32"
  config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-i386-vagrant-disk1.box"

  config.vm.hostname = "wa.local"
  config.vm.network "private_network", ip: "192.168.2.2"

  # Using NFS for now, cause native sharing is broken
  config.vm.synced_folder ".", "/vagrant", nfs: true

  # Provisioning
  config.vm.provision "ansible" do |ansible|
    ansible.host_key_checking = false
    ansible.inventory_path = "ansible/inventory.ini"
    ansible.playbook = "ansible/vagrant.yml"
    ansible.limit = "all"
  end

  # Provider Configuration: VirtualBox
  config.vm.provider :virtualbox do |vb|
    # Setting RAM to 1 GB
    vb.memory = 1024
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
  end

end

Update #2

The problem is gone for now. I still don't know why it occurred in the first place. Maybe the line vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"] that I recently added to Vagrantfile helped.

Thanks for all suggestions!

Solution

Do you have any Intrusion Prevention Devices or Firewalls that could be causing problems? I've seen automation methods trigger rules designed to combat bots. Numerous connections can be misinterpreted as port scans, brute force attempts, etc. IDS/IPS/Firewalls will then start dropping packets, causing strange things to fail without any clear logic as to why. These rules won't trigger when you do things manually, so everything seems fine when you troubleshoot.

Comments

[MSumulong]

Can you post your Vagrantfile?

[Slava Fomin II]

@MSumulong, I've updated my question.

[Slava Fomin II]

Thanks @sjw for your suggestion! It's an interesting idea. I will try to investigate this if this problem occurs again.

[zuba]

I experience the same problem, and no, disabling ipv6 didn't resolve this issue

[zuba]

Slava, can you try to install some gem from inside your box? See my trouble here: serverfault.com/questions/626556/…

[Slava Fomin II]

Thank you for your reply! It's an interesting idea for sure. However I do not believe that we have any of such security systems.

[Dominik Ehrenberg]

So, we dug deeper and this seems to be an issue related to IPv6 and DS-lite. If you do have a provider that uses DS-lite, see if you can turn it off or use VPN to route all your traffic through it