apt.conf "Acquire::http:Proxy "proxyserver:port" seems not to be used (Ubuntu 13.04 under Virtual Box on Win7)

Question

This is for an installation of Ubuntu 13.04 Desktop in a VirtualBox VM. I'm on a corprate network with an HTTP proxy at x.y.z.251:9090. I've configured the guest OS with the autoconfiguration script, so Firefox works fine, but I can't get "apt-get" or "Ubuntu Software Center" to connect using the proxy.

I created the file "/etc/apt/apt.conf", it now contains a single line of text:

Acquire::http:Proxy "http://x.y.z.251:9090"

which is what the proxy.pac file that I specified in "System Settings/Network/Network Proxy" and applied to the entire system.

When I use Firefox, a packet trace shows that the TCP connection is done via the corporate proxy at port 9090, when I run "Ubuntu Software Center" and "apt-get", the packet trace shows TCP SYN packets with the eventual destination IP address and port 80. There, is of course, no response to the SYN packets, since the corporate firewall blocks TCP connections on port 80 from getting in or out.

With "sudo apt-get" in a terminal window, I always see:

Ign cdrom://Ubuntu 13.04 _Raring Ringtail_ - Release amd64 (20130424) raring/main Translation-en_US
Ign cdrom://Ubuntu 13.04 _Raring Ringtail_ - Release amd64 (20130424) raring/main Translation-en
Ign cdrom://Ubuntu 13.04 _Raring Ringtail_ - Release amd64 (20130424) raring/restricted Translation-en_US
Ign cdrom://Ubuntu 13.04 _Raring Ringtail_ - Release amd64 (20130424) raring/restricted Translation-en
Err http://extras.ubuntu.com raring Release.gpg 
  Could not connect to extras.ubuntu.com:80 (91.189.92.152), connection timed out
Err http://archive.canonical.com raring Release.gpg 
  Cannot initiate the connection to archive.canonical.com:80 (2001:67c:1360:8c01::1b). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::1b 80]
Err http://us.archive.ubuntu.com raring Release.gpg 
  Cannot initiate the connection to us.archive.ubuntu.com:80 (2001:67c:1562::14). - connect (101: Network is unreachable) [IP: 2001:67c:1562::14 80]
Err http://us.archive.ubuntu.com raring-updates Release.gpg
  Cannot initiate the connection to us.archive.ubuntu.com:80 (2001:67c:1562::14). - connect (101: Network is unreachable) [IP: 2001:67c:1562::14 80]
Err http://us.archive.ubuntu.com raring-backports Release.gpg
  Cannot initiate the connection to us.archive.ubuntu.com:80 (2001:67c:1562::14). - connect (101: Network is unreachable) [IP: 2001:67c:1562::14 80]
59% [Connecting to security.ubuntu.com (91.189.91.14)]

Eventually, I see:

Err http://security.ubuntu.com raring-security Release.gpg
  Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]
Reading package lists... Done                          
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/raring/Release.gpg  Cannot initiate the connection to us.archive.ubuntu.com:80 (2001:67c:1562::14). - connect (101: Network is unreachable) [IP: 2001:67c:1562::14 80]

W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/raring-updates/Release.gpg  Cannot initiate the connection to us.archive.ubuntu.com:80 (2001:67c:1562::14). - connect (101: Network is unreachable) [IP: 2001:67c:1562::14 80]

W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/raring-backports/Release.gpg  Cannot initiate the connection to us.archive.ubuntu.com:80 (2001:67c:1562::14). - connect (101: Network is unreachable) [IP: 2001:67c:1562::14 80]

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/raring-security/Release.gpg  Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8c01::18). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::18 80]

W: Failed to fetch http://archive.canonical.com/ubuntu/dists/raring/Release.gpg  Cannot initiate the connection to archive.canonical.com:80 (2001:67c:1360:8c01::1b). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::1b 80]

W: Failed to fetch http://extras.ubuntu.com/ubuntu/dists/raring/Release.gpg  Could not connect to extras.ubuntu.com:80 (91.189.92.152), connection timed out

W: Some index files failed to download. They have been ignored, or old ones used instead.

I'm not at all certain what I'm doing wrong.

--- Update ---- I have also tried:

Acquire::http:proxy "http://user:[email protected]:9090/"

"x.y.z" are, of course, place-holders; I doubt my employer's security people would look kindly on me posting the actual numbers. I see no difference in the packet trace -- "apt-get" is still using the real IP addresses and port 80. I made a deliberate mistake in the apt.conf file and "apt-get" exited with an error, so I'm certain that it sees this configuration line for the proxy, it just doesn't seem to honor it.

I am using the "NAT" networking connection, as it is against the IT policy here to bridge between networks, and the DHCP server only provides addresses to known MAC addresses. I don't want IT security banging on my cube (again).

Solution 1

Based on my past experience, overall Ubuntu (GNOME) doesn't work very well with PAC (Automatic Config). Pac (Proxy Auto-config) is basically JavaScript logic to determine the closest or most appropriate proxy for you, I strongly recommend using the proxy servers directly (in your case Network - Network Proxy).

A few things to check

Check your /etc/apt/apt.conf

Network -Network Proxy GUI will update /etc/apt/apt.conf and set the proxy info there.

Find out the proxy server info from pac

You can find the details by reading the proxy.pac file anyway.

So the /etc/apt/apt.conf should look like below (suppose the proxy server is => proxy.company.com port 80), do NOT use pac URL.

Acquire::http::proxy "http://proxy.company.com:80/";
Acquire::https::proxy "https://proxy.company.com:80/";
Acquire::ftp::proxy "ftp://proxy.company.com:80/";

NOTE: If no proxy is specified in the apt config files, apt-get will fall back to http_proxy environment variable.

From apt.conf man page, respond to the comment from @Braiam

   http
       HTTP URIs; http::Proxy is the default http proxy to use. It is in
       the standard form of http://[[user][:pass]@]host[:port]/. Per host
       proxies can also be specified by using the form http::Proxy::<host>
       with the special keyword DIRECT meaning to use no proxies. If no
       one of the above settings is specified, http_proxy environment
       variable will be used.

Check proxy environment variables

You can check proxy settings by running the following in terminal

echo $http_proxy
echo $https_proxy
echo $ftp_proxy

If you set env variables correctly, you should be should be able to update in CLI.

Prefer IPv4

I noticed apt-get was trying to connect to the update servers using their ipv6 address. This could cause problems (sometimes the IPV6 addresses expire or change).

You can edit /etc/gai.conf and add precedence ::ffff:0:0/96 100 to prefer ipv4 over ipv6.

More details

• https://unix.stackexchange.com/a/13263/30622
• https://serverfault.com/a/177649/145918

Solution 2

I believe I have had this very problem and the answer is simple. The syntax is critical. It must look like this:

Acquire::http::Proxy "http://x.y.z.251:9090";

(There is an extra ":" and a terminating ";")

Solution 3

@Braiam

I think it's better to start a separate answer to clarify things.

I did a quick test using Vagrant and a Ubuntu 13.04 Raring x86_64 VM behind firewall.

apt-get version is 0.9.7.7ubuntu4

root@raring:~# apt-get --version 
apt 0.9.7.7ubuntu4 for amd64 compiled on Apr 12 2013 23:49:05
Supported modules:
*Ver: Standard .deb
*Pkg:  Debian dpkg interface (Priority 30)
 Pkg:  Debian APT solver interface (Priority -1000)
 S.L: 'deb' Standard Debian binary tree
 S.L: 'deb-src' Standard Debian source tree
 Idx: Debian Source Index
 Idx: Debian Package Index
 Idx: Debian Translation Index
 Idx: Debian dpkg status file
 Idx: EDSP scenario file

Update: It works the same way on Precise 12.04.3

root@support:/etc/apt# uname -a
Linux support 3.8.0-30-generic #44~precise1-Ubuntu SMP Fri Aug 23 17:33:45 UTC 2013 i686 i686 i386 GNU/Linux
root@support:/etc/apt# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.3 LTS
Release:        12.04
Codename:       precise
root@support:/etc/apt# apt-get --version
apt 0.8.16~exp12ubuntu10.14 for i386 compiled on Sep  8 2013 03:26:42

1st - clear apt configuration files

I cleared the proxy settings in /etc/apt/apt.conf. I also did ack and ag scan of all files and sub-directories /etc/apt to make sure there is no proxy set in apt configuration files.

By default {http_proxy,https_proxy,ftp_proxy} env variables are not set.

apt-get was not able to connect to the update servers.

root@raring:~# apt-get -o Debug::Acquire::http=true update
0% [Connecting to au.archive.ubuntu.com (202.158.214.106)] [Connecting to security.ubuntu.com (91.189.91.13)] [Connecting to ppa.launchpad.net (91.189.95.83)]

2nd - set environment variables

Simply set the proxy variables

export {http_proxy,https_proxy,ftp_proxy}="http://10.xxx.xxx.231:80"

apt-get is now able to connect!

See the debug output (this is something that I have never tried).

root@raring:~# apt-get -o Debug::Acquire::http=true update
0% [Working]GET http://security.ubuntu.com/ubuntu/dists/raring-security/Release.gpg HTTP/1.1
Host: security.ubuntu.com
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (0.9.7.7ubuntu4)


GET http://au.archive.ubuntu.com/ubuntu/dists/raring/Release.gpg HTTP/1.1
Host: au.archive.ubuntu.com
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (0.9.7.7ubuntu4)


GET http://ppa.launchpad.net/git-core/ppa/ubuntu/dists/raring/Release.gpg HTTP/1.1
Host: ppa.launchpad.net
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (0.9.7.7ubuntu4)


HTTP/1.1 200 OK
Date: Fri, 27 Sep 2013 12:12:20 GMT
ETag: "16e20bb4-3a5-4db2e154a1dc0"
Server: Apache/2.2.3 (Red Hat)
Content-Type: text/plain; charset=UTF-8
Accept-Ranges: bytes
Last-Modified: Thu, 25 Apr 2013 11:54:39 GMT
Content-Length: 933
Proxy-Connection: Keep-Alive

Get:1 http://au.archive.ubuntu.com raring Release.gpg [933 B]
0% [1 Release.gpg 0 B/933 B 0%] [Waiting for headers] [Waiting for headers]GET http://au.archive.ubuntu.com/ubuntu/dists/raring-updates/Release.gpg HTTP/1.1
Host: au.archive.ubuntu.com
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (0.9.7.7ubuntu4)


99% [Waiting for headers] [Waiting for headers]HTTP/1.1 200 OK             
Date: Fri, 27 Sep 2013 12:12:20 GMT
ETag: "16e20ee7-3a5-4e7594ace9200"
Server: Apache/2.2.3 (Red Hat)
Content-Type: text/plain; charset=UTF-8
Accept-Ranges: bytes
Last-Modified: Fri, 27 Sep 2013 08:30:00 GMT
Content-Length: 933
Proxy-Connection: Keep-Alive

Get:2 http://au.archive.ubuntu.com raring-updates Release.gpg [933 B]
50% [2 Release.gpg 0 B/933 B 0%] [Waiting for headers] [Waiting for headers]GET http://au.archive.ubuntu.com/ubuntu/dists/raring-backports/Release.gpg HTTP/1.1
Host: au.archive.ubuntu.com
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (0.9.7.7ubuntu4)


100% [Waiting for headers] [Waiting for headers]HTTP/1.1 200 OK             
Date: Fri, 27 Sep 2013 12:12:21 GMT
ETag: "16e0083d-3a5-4e6844f477fc0"
Server: Apache/2.2.3 (Red Hat)
Content-Type: text/plain; charset=UTF-8
Accept-Ranges: bytes
Last-Modified: Mon, 16 Sep 2013 18:24:07 GMT
Content-Length: 933
Proxy-Connection: Keep-Alive

Get:3 http://au.archive.ubuntu.com raring-backports Release.gpg [933 B]
67% [3 Release.gpg 0 B/933 B 0%] [Waiting for headers] [Waiting for headers]HTTP/1.1 200 OK
Date: Fri, 27 Sep 2013 12:12:20 GMT
ETag: "3a5-4e75c5d969600"
Server: Apache/2.2.22 (Ubuntu)
Expires: Fri, 27 Sep 2013 13:05:00 GMT
Accept-Ranges: bytes
Cache-Control: max-age=3159, s-maxage=3300, proxy-revalidate
Last-Modified: Fri, 27 Sep 2013 12:10:00 GMT
Content-Length: 933
Proxy-Connection: Keep-Alive

......

3rd - unset the environment variables

unset the env vars => unset {http_proxy,https_proxy,ftp_proxy}

Now apt-get was unable to connect.

Conclusion

Just like what man apt.conf says, if http::Proxy is NOT set, http_proxy environment variable will be used.

BTW: I am also an Arch Linux user. Pacman works in a similar way, if I do not use wget or curl with proxy in pacman.conf, it'll use `{http_proxy,https_proxy,ftp_proxy}' environment variables.

Update sudo does NOT preserve environment variables. That's why sudo apt-get update fails. To workaround the issue (preserve proxy environment variables), use sudo -E apt-get update.

Comments

[Daniel Glasser]

Thanks for the response. The value I put in "/etc/apt/apt.conf" is the actual URI and socket that comes from the .pac file, not the URI of the .pac file itself (that's what I put in "system settings/networking/network proxy".

[Daniel Glasser]

I went into "System Settings/Networking/network proxy" and used manual configuration; this did update "/etc/apt/apt.conf". I also disabled IPv4 by editing "/etc/sysctl.conf". This seems to have done the trick. Thanks for your help.

[Braiam]

Not true, apt-get never uses http_proxy nor any environment variables.

[Terry Wang]

@Braiam do a man apt.conf and see http::Proxy. I have also updated the answer. Also I've been using http_proxy and https_proxy environment variables to use with apt-get for years. Do some research before voting down.

[Braiam]

Also, do research yourself! Run sudo apt-get -o Debug::Acquire::http=true update and check yourself if only setting the environment variable uses the proxy. And if you were using proxy for year, get yourself updated.

[Terry Wang]

@Braiam Thanks for the update. I'll look into the details, do test and update when I have time over the weekend. Calm down coz this is a place for technical Q&A.If more details had been added in your comment, I would have been more than happier to look into it when I saw it. I didn't know who down voted, it is not important and I don't care. What matters is that I knew something that I didn't know before. I'll update once I've done the test.

[Braiam]

Yeah, I'm wrong... sudo seems to eat up the http_proxy variable each time is run (which is why people seems like they won't use proxy at all), but apparently apt-get indeed import the variable.

[Braiam]

Btw, I do not get notified of ping in answers. Also, see my comment, try to use sudo instead login as root and see why I said that it didn't work.

[Terry Wang]

@Braiam I did some more test. sudo does NOT preserve envrionment variables, that's the problem! But you can workaround it by using sudo -E apt-get update && apt-get dist-upgrade ;-)

[Braiam]

I kinda said the same in my other comment :S

[Terry Wang]

@Braiam yep, by default sudo does not preserve environment variables. sudo -E does ;-) A good lesson taught.

[DaShaun]

I like how you clarified the answer. The solution to my problem was the "sudo -E" tip. I don't remember ever needing to use the -E option in the past.

[R-JANA]

The above solution worked very well with apt-get but I am facing issue with puppet installation. What I found is that whenever the download happens from the port 443 the download times out. the same url is able to download if i access the url from a browser. I assume that the browser is able to do some handshake which command line is not able to do. Any suggestions which i can try?

[pkm]

after this do we still need to configure url in the browser to access internet through proxy ?? ( eq. inside firefox settings > Configure proxies to access internet )

[FreeSoftwareServers]

Another reason redhat.com/sysadmin/sysadmins-dont-sudo . Just be root if your sysadmin!