Perl Package Manager via NTLM proxy

Question

I've installed ActivePerl and the include PPM at work, but I'm unable to get it connect to the module repository through the proxy. I've set up an environment variable

HTTP_PROXY = http://user:[email protected]:port

but PPM still shows a 407 error Proxy Authentication Required. I think it's an NTLM proxy, but I'm not sure.

After doing this, I get an Error 500: Bad hostname for ppm4.activestate.com. Turns out PPM asks my DNS server to resolve that name, but my local DNS server cannot resolve such external domain names. Only the proxy does such thing and only for HTTP traffic...

Here's the screenshot of what Wireshark (running on an BackTrack VM on the same Windows XP box) is showing:

My Windows box is 10.75.82.221 The DNS server is 10.7.3.110

This what TraceRoute shows from Windows:

G:\>tracert ppm4.activestate.com
No se puede resolver el nombre de destino ppm4.activestate.com.

Which means:

Cannot resolve ppm4.activestate.com

as expected.

As I said on the comments, the browser doesn't seem to launch any DNS query. Here's a Wireshark screenshot (from BT) that shows an access to www.google.es from Firefox (from WXP):

My Windows box is 10.75.82.221 The proxy is 10.7.8.46

As you can see, no DNS query whatsoever when accessing through browser...

To me, it just seems it's a security feature and I'll have to ask for permission to let me use ActiveState's PPM tool... which is not plausible, even if most (99%) of the scripts I make here are for speeding the work.

Solution 1

Solving error 407 is described in the perl.org Questions and Answers :

Q: Even when http_proxy is set to the correct server with the proper credentials (using the convention username:[email protected]:port) I'm still not able to install packages via cpan. "LWP failed with code[407] message[Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )]"

A: CPAN doesn't support NTLM authentication, and it's difficult to get Authen::NTLM working with CPAN. I ended up using ntlmaps as a local proxy for NTLM authentication.

The first thing you'll need to do is get the latest version of NTLMAPS from their svn repo, extract it somewhere. Next you'll need to get the latest version of python and install it. Open NTLMAPS' server.cfg, set PARENT_PROXY to your proxy's address or IP, PARENT_PROXY_PORT to the port the proxy runs on (usually 80 or 8080), NT_DOMAIN to your domain, and USER to your username.

Run runserver.bat. It will ask for your password, enter it and press enter. Next you have to configure cpan to use NTLMAPS as a proxy. Go to strawberry\perl\lib\CPAN and open Config.pm. Set http_proxy to http://localhost:5865. Run CPAN and use it as you normally would. Now you should only have to start up NTLMAPS before running CPAN to get it to run properly.

Just a quick note, you may need to force LM + NT authentication rather than LM; To do this, open the NTLMAPS server.cfg set NT_PART to 1 and NTLM_FLAGS to 07820000.

Removing HTTP_PROXY and modifiying CPAN/Config.pm as stated may solve the problem.

Summary of the comments below:

The study of wireshark dumps shows that ActivePerl issues DNS requests that are returned as unknown by the DNS server. The solutions possible are only :

1. Reconfiguration of the DNS server (impossible for the poster)
2. Launching a request on the ActiveState Community Site to stop these futile requests (no guarantee of any speedy relief)
3. Manually download and install the modules (only practical solution left).

Solution 2

If it's NTLM, you'll need the NTLM Perl module. The ActiveState PPM documentation says you can obtain a ppmx package from the repository, but I couldn't find any such link. You still can, however, download the ppd and tar.gz files and install them:

wget http://ppm4.activestate.com/MSWin32-x86/5.12/1205/N/NB/NBEBOUT/NTLM-1.09.ppd
wget http://ppm4.activestate.com/MSWin32-x86/5.12/1205/N/NB/NBEBOUT/NTLM-1.09.tar.gz
ppm install NTLM-1.09.ppd

Also note that, if your computer belongs to a Windows domain, you should not need to give the credentials explicitly - http://proxy.full.name:port should be enough for NTLM/Kerberos, in addition to being more secure.

---

Recently Microsoft software has been switching from NTLM to Kerberos – although usually proxies support both, it can happen that NTLM will be disabled in yours. There is not much documentation on Kerberos proxies with PPM, but HTTP-Negotiate might be worth trying, if you cannot get NTLM to work.

Solution 3

Here is an alternative to using NTLMAPS + Python (as described above by harrymc). The idea is very similar, but more "streamlined":

• Install Fiddler.

• Configure PPM to use Fiddler as a proxy. In the command prompt type:

  set http_proxy=http://127.0.0.1:8888
  

• Make sure to open any Internet page from any browser first (this will create an authenticated session between Fiddler and your corporate Proxy).

• Give PPM a try.

See the ActiveState Community Site for a bit more details.

If it still doesn't work, you may need to enable pac scripts in fiddler.

Tools -> Fiddler Options -> Connections

Check "Use a PAC Script"

Solution 4

This is a shot in the dark... nobody's mentioned trying this but it sounds like everyone's out of ideas... if this is true:

"Turns out PPM asks my DNS server to resolve that name, but my local DNS server cannot resolve such external domain names."

Then edit:

C:\Windows\System32\Drivers\etc\hosts

add the line:

204.244.102.19  ppm4.activestate.com

See if it makes a difference.

---

For troubleshooting ntlmaps:

There are two options in server.cfg DEBUG and BIN_DEBUG, if you have toubles with the server so set these options to DEBUG:1 and BIN_DEBUG:1 just before requesting a problem page (or resource). You have to restart proxy server to reread server.cfg. This will give you 3 log files per http request (per connection to be exact), like 127.0.0.1-1048, 127.0.0.1-1048.bin.client and 127.0.0.1-1048.bin.rserver. In the first one there is an info on what APS did, two others contain raw traffic from client and from proxy.

http://ntlmaps.sourceforge.net/

View more solutions

Comments

[harrymc]

Can you connect via a browser ? Does the handling of error 407 in this FAQ seem useful ?

[m0skit0]

I can connect via browser, but only to port 80. I didn't try that, gonna do it right now.

[m0skit0]

I tried that and now I get Error 500 ppm4.activestate.com bad hostname. I modified the corresponding Config.pm for ActivePerl.

[harrymc]

I think there is a wrong parameter somewhere. I fear you will have to debug the request/answer packets using fiddler2 or wireshark to find the error.

[m0skit0]

Ok, no wrong parameters. I see that my BOOTP configured DNS server cannot resolve external domains, and PPM is asking this DNS server directly. I'll try to see what's happening... Hmmm I cannot see any DNS request for external hostnames done through browser. I'm supposing HTTP requests are sent directly to proxy which filters and resolves the names...

[harrymc]

DNS mess-up is an excellent explanation. But that means that PPM ignores HTTP_PROXY or Config.pm.

[m0skit0]

But I removed HTTP_PROXY and modified CPAN/Config.pm as stated... In fact it seems to work because it doesn't spit 407 error anymore. It looks like you can only resolve external domains through the proxy and HTTP...

[harrymc]

Good. So I converted our discussion into an answer below. Please feel free to modify/add to it via edit.

[m0skit0]

Installed successfully but still no go. For example, I select Win32-EmailSend from PPM to install: Win32-EmailSend marked for install Installing package ... Downloading Win32-EmailSend-0.04 ... failed 407 Proxy Authentication Required Installing package failed ERROR: 407 Proxy Authentication Required

[m0skit0]

How can I setup HTTP::Negotiate for PPM? I just install it like the NTML module?

[m0skit0]

Thanks, I searched extensively on Google but didn't saw that. I already did most of that, except forcing LM + NT auth.Will try.

[m0skit0]

No go, still Error 500: bad hostname. As I said on the comments above: "[it's the] proxy which filters and resolves the names [for external domains]", so I directly cannot resolve external domain names into IP except through proxy and HTTP.

[harrymc]

Strange, this should have worked once configured. See this thread for a similar case (solved). For an alternative solution see this. If nothing works, what exactly does wireshark show?

[m0skit0]

I've attached a screenshot with minimal explanation in the question. Thanks for your time!

[harrymc]

Can you compare this wireshark trace with the DNS queries as when issued from the browser (which you say does work). The output of tracert ppm4.activestate.com will also be useful.

[harrymc]

Standard questions: Clearing the browser's cache; disabling firewalls. But anyway : Why for heaven's sake would perl issue dns requests instead of just trusting to the default resolver? The browser's behavior is normal, but ActivePerl is absolutely not normal. This looks more and more like an ActivePerl bug or bad implementation. The alternative solution I posted on Feb 8 seems to be a workaround for this bug, but may not apply to Windows. Which Windows version are you using?

[m0skit0]

Cleared Firefox cache, same result: no DNS query before sending the HTTP GET request. I'm using Windows XP. Btw, I see that Firefox has network.dns.disablePrefetch set to true. Maybe PPM has some similar option to avoid external DNS resolving?

[harrymc]

The wireshark output is unclear - needs an explanation of the IP addresses (which is what, especially the proxy). This can still be a firewall problem, maybe that of the proxy itself, as you said something about only being able to access via the browser on port 80. If worse comes to worse, you will need to manually download the modules and install.

[m0skit0]

Yes, sorry, added explanation of the IP addresses. And yes, only destination port 80 (HTTP) is allowed to pass through the proxy/firewall. I didn't test ALL ports to be sure, but I tested some like telnet, ftp and such.

[harrymc]

My tentative conclusion is that your proxy blocks DNS requests, which perl issues for some unknown reason. The solutions I can see are: (1) enable DNS in the proxy if possible (2) modify perl to stop this practice, (3) download and install modules manually. For (2) you could also launch a demand on the ActiveState Community Site.

[m0skit0]

I don't think the proxy blocks DNS requests. It's my local DNS server who doesn't resolve them, as you can see on the first Wireshark capture. (1) the proxy does DNS resolution, but only for HTTP requests AFAIK, and no, I can't change anything on the proxy, (2) I'll try to find time to do that and (3) seems most plausible solution. Thanks for your time.

[m0skit0]

Thanks, nice idea, didn't think about it. The problem is that if ppm4.activestate.com's IP ever changes, I would have to edit that file again... I'm testing it.

[m0skit0]

Weird: now PPM says: Downloading ActiveState Package Repository packlist ... failed 500 Can't connect to ppm4.activestate.com:80 (connect: timeout)

[harrymc]

My mistake: Your DNS server is indeed distinct from the proxy. So my point (1) above should be replaced by: Correctly configure your DNS server. If you don't control it, you are mostly left with point (3).

[mgjk]

Is the traffic still being redirected to the proxy? There are some debug options for ntlmaps which might show.

[m0skit0]

Wireshark does not show any requests being made. What debug options are you talking about?

[mgjk]

@m0skit0, I added new info about enabling ntlmaps debugging to my answer. Unless the logs say something really interesting, I'm out of ideas. Good luck :-)

[m0skit0]

Err... sorry but where does it generate the log files? Can't find any...

[m0skit0]

Thanks but won't work. An NTLM proxy is not the same than an HTTP proxy. It requires authentication.

[Vyankatesh Agrawal]

Oh yeah!! I didn't think of that. Since it was HTTP proxy for me, it worked. Thank you for educating me.