Use ELPA (Emacs) behind a proxy requiring authentication

Question

I have read this and this question. In both they say Emacs can deal with authentication, but it does not work for me.

The question is: What is wrong?

The Emacs version is 24.0.97-1, and it is running on 64-bit Linux.

At work I have to use proxy server for any Internet connection. So I set the following environment variables:

http_proxy="http://username:password@ip:port
https_proxy="https://username:password@ip:port
ftp_proxy="ftp://username:password@ip:port

This works. I can download packages without any problem.

When I run M-x package-refresh-contents in Emacs it asks me for login and password for the proxy server, but it can not connect to the server. It even does not try to connect, i.e. after I type password and press Enter Emacs instantly reports: Failed to download 'marmalade' archive

The same happens if I remove username and password from http_proxy variable or if I set url-proxy-services in Emacs (even if I unset the system variable).

Solution 1

Emacs uses only HOST and PORT part from http_proxy.

I get authorization working without user interaction by:

(setq url-proxy-services
   '(("no_proxy" . "^\\(localhost\\|10.*\\)")
     ("http" . "proxy.com:8080")
     ("https" . "proxy.com:8080")))

(setq url-http-proxy-basic-auth-storage
    (list (list "proxy.com:8080"
                (cons "Input your LDAP UID !"
                      (base64-encode-string "LOGIN:PASSWORD")))))

This works for Emacs 24.3. It is based on non-public API tricks, so might not work is another Emacs versions...

Replace LOGIN and PASSWORD with your auth info...

Also there is url-http-proxy-digest-auth-storage. Just fill prompt with authentication data and check which var used by Emacs (by M-: var RET)...

Solution 2

It looks like Emacs has some troubles with authentication. So I have installed Squid and now use it as an intermediate between the external proxy server and all my applications. Squid is configured as a proxy without authentication and everything works well with it.

Many people recommend this solution but give no precise instructions. I made my /etc/squid/squid.conf from another one designed for different purpose. Probably it contains something that is not needed and/or misses something it should have. Any improvements are welcome:

# only access from localhost is allowed
acl localhost src 127.0.0.1/32
acl all src all
http_access allow localhost
http_access deny all
icp_access deny all

never_direct allow all

# turn off cache
cache_dir null /tmp
cache deny all

# logs
access_log /var/log/squid/access.log squid

# turn off proxy-headers (no idea what is it :))
via off
forwarded_for off

# describe external proxy server
cache_peer <proxy_ip> parent <proxy_port> 0 no-query default proxy-only login=<my_login>:<my_password>
http_port 10000
acl port10000 myport 10000
cache_peer_access <proxy_ip> allow port10000

This proxy has address 127.0.0.1:10000. In Emacs I have to execute the following code:

(setq url-proxy-services '(("http" . "127.0.0.1:10000")))

Solution 3

There are two bugs here - one is in url-http.el, and can be fixed with a patch I just sent to http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12069 This will stop Emacs from prompting you for the password on every attempt, and when it doesn't prompt you, it should work.

The other bug hasn't been tracked down yet, but it seems that when the proxy server requests authentication, the authentication is prompted for, then immediately the authentication request from the proxy server is processed by the package code. Meanwhile the real request continues in the background.

Solution 4

There is another related bug in emacs < 28.1 in url-http with the function url-https-proxy-connect, causing all https calls through an authenticating proxy to fail.

See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42422.

As a workaround for emacs < 28.1, override the function with the fixed version:

(with-eval-after-load 'url-http
  (defun url-https-proxy-connect (connection)
    (setq url-http-after-change-function 'url-https-proxy-after-change-function)
    (process-send-string connection (format (concat "CONNECT %s:%d HTTP/1.1\r\n"
                            "Host: %s\r\n"
                            (let ((proxy-auth (let ((url-basic-auth-storage
                                         'url-http-proxy-basic-auth-storage))
                                    (url-get-authentication url-http-proxy nil 'any nil))))
                              (if proxy-auth (concat "Proxy-Authorization: " proxy-auth "\r\n")))
                            "\r\n")
                        (url-host url-current-object)
                        (or (url-port url-current-object)
                        url-https-default-port)
                        (url-host url-current-object)))))

View more solutions

Comments

[Richard Gomes]

Have a look at proxydriver, which is an excellent tool for automagically configure proxy-related environment variables for you when NetworkManager connects. I'm using it in my laptops, since sometimes I'm behind a proxy server, sometimes not.

[Richard Gomes]

I've observed that (package-install 'something) was failing in my environment, when behind squid3. In a nutshell, the fix consisted on:<br/> via off<br/> forwarded_for transparent

[Richard Gomes]

I don't see need for littering .emacs with proxy related stuff. You can simply create environment variables for that. Have a look at proxydriver, which is an excellent tool for automagically configure proxy-related environment variables for you when NetworkManager connects. I'm using it in my laptops, since sometimes I'm behind a proxy server, sometimes not.

[Kamran Ahmed]

Do we have to replace "Input your LDAP UID !" with something or use it as it is?

[Giupo]

@KamranAhmed I'd put there my own user id. Am I wrong?

[Kamran Ahmed]

@Giupo Oh that was so obvious, I wonder if I was drunk when I commented that day... Thanks for clearing it out anyway. :D

[Alo]

How do I find my LDAP UID? user id of what? Help please!

[Welgriv]

This solution DOES NOT works at all, beside a lake of explanations. And its a security issue to store a password in plain text format

[Greg Whittier]

I also found this necessary with emacs 27.1 (windows).

[Joseph Tesfaye]

Doesn't work with me (Emcas 27.1 on Windows). When I input username and password it still prompts the error: Debugger entered--Lisp error: (file-error "http://mirrors.tuna.tsinghua.edu.cn/elpa/gnu/archi..." "Proxy authentication required")

[Joseph Tesfaye]

@GregWhittier Does this solution work for you? Are there extra setups other than this? Simply putting this in the init file along with url-proxy-services doesn't work for me.

[mosquito-magnet]

@GregWhittier: true, the fix is in emacs 28.1, I edited the post.

[mosquito-magnet]

@IvanHuang: that is a http url, it won't use url-https-proxy-connect but url-http-create-request. That should work out of the box in 27.1 though. Did you set url-http-proxy-basic-auth-storage, as specified in another answer here?

[Joseph Tesfaye]

It works when I change the mirror url to https

[mosquito-magnet]

Strange, but good that its working now ;) Maybe you have an old url-http library in your load path. You could call locate-library on it to check the load path, it should be in your emacs folder somewhere under share/emacs/27.1/lisp/url/url-http.elc.