HttpClient 4.x how to use cookies?

Question

I'm trying to use the cookies I get in a response to my post method using HttpClient 4.0.3;

Here is my code:

public void generateRequest()
{
    DefaultHttpClient httpclient = new DefaultHttpClient();
    HttpPost httppost = new HttpPost("http://mysite.com/login");
    httpclient.getParams().setParameter("http.useragent", "Custom Browser");
    httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
            HttpVersion.HTTP_1_1);
    httpclient.getParams().setParameter(ClientPNames.COOKIE_POLICY,
            CookiePolicy.BROWSER_COMPATIBILITY);

    CookieStore cookieStore = new BasicCookieStore();
    HttpContext localContext = new BasicHttpContext();
    localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);

    try
    {
        LOG.info("Status Code: sending");
        List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
        nameValuePairs.add(new BasicNameValuePair("email", "john%40gmail.com"));
        nameValuePairs.add(new BasicNameValuePair("password", "mypassword"));
        httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
        httppost.setHeader("ContentType", "application/x-www-form-urlencoded");
        HttpResponse response = httpclient.execute(httppost, localContext);
        HttpEntity entity = response.getEntity();
        if (entity != null)
        {
            entity.consumeContent();
        }
        iterateCookies(httpclient);

    }
    catch (ClientProtocolException e)
    {
        LOG.error("ClientProtocolException", e);
    }
    catch (IOException e)
    {
        LOG.error("IOException", e);
    }
}

private void iterateCookies(DefaultHttpClient httpclient)
{
    List<Cookie> cookies = httpclient.getCookieStore().getCookies();
    if (cookies.isEmpty())
    {
        System.out.println("No cookies");
    }
    else
    {
        for (Cookie c : cookies)
        {
            System.out.println("-" + c.toString());
        }
    }
}

But I keep getting the No cookies logged out even though when I use web-sniffer.net, I get this response:

Status:            HTTP/1.1 302 Found
Cache-Control:     private, no-store    
Content-Type:      text/html; charset=utf-8 
Location:          http://www.mysite.com/loginok.html   
Server:            Microsoft-IIS/7.0    
X-AspNet-Version:  2.0.50727    
Set-Cookie:        USER=DDA5FF4E1C30661EC61CFA; domain=.mysite.com; expires=Tue, 08-Jan-2013     18:39:53 GMT; path=/   
Set-Cookie:        LOGIN=D6CC13A23DCF56AF81CFAF; domain=.mysite.com; path=/ Date: Mon, 09     Jan 2012 18:39:53 GMT 
Connection:        close    
Content-Length:    165

All the examples I've found online that make any sort of sense refer to HttpClient 3.x where you can set the CookiePolicy to IGNORE and handle the Set-Cookie header manually. I can't understand why this is so difficult in 4.x. I need access to the USER hash for a number of reasons. Can anyone please tell me how in the hell I can get access to it?

UPDATE

I have found the following C# code which does the same thing and works correctly.

private static string TryGetCookie(string user, string pass, string baseurl)
    {
        string body = string.Format("email={0}&password={1}", user, pass);
        byte[] bodyData = StringUtils.StringToASCIIBytes(body);

        HttpWebRequest req = WebRequest.Create(baseurl) as HttpWebRequest;

        if (null != req.Proxy)
        {
            req.Proxy.Credentials = CredentialCache.DefaultCredentials;
        }

        req.AllowAutoRedirect = false;
        req.Method = "Post";
        req.ContentType = "application/x-www-form-urlencoded";
        req.ContentLength = bodyData.Length;

        using (Stream reqBody = req.GetRequestStream())
        {
            reqBody.Write(bodyData, 0, bodyData.Length);
            reqBody.Close();
        }

        HttpWebResponse resp1 = req.GetResponse() as HttpWebResponse;

        string cookie = resp1.Headers["Set-Cookie"];

        if( string.IsNullOrEmpty(cookie))
        {
            if (0 < resp1.ContentLength)
            {
                // it's probably not an event day, and the server is returning a singlecharacter
                StreamReader stringReader = new StreamReader(resp1.GetResponseStream());

                return stringReader.ReadToEnd();
            }

            return null;
        }

        return ParseCookie(cookie);
    }

I believe my java code is not forming the post request correctly because when I use a URLConnection and print the request header from web-sniffer.net below:

POST /reg/login HTTP/1.1[CRLF]
Host: live-timing.formula1.com[CRLF]
Connection: close[CRLF]
User-Agent: Web-sniffer/1.0.37 (+http://web-sniffer.net/)[CRLF]
Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7[CRLF]
Cache-Control: no-cache[CRLF]
Accept-Language: de,en;q=0.7,en-us;q=0.3[CRLF]
Referer: http://web-sniffer.net/[CRLF]
Content-type: application/x-www-form-urlencoded[CRLF]
Content-length: 53[CRLF]
[CRLF]
email=john%40gmail.com&password=mypassword

I get a response from the server that contains the set-cookies header. Is my java code not generating the request the same as web-sniffer.net?

I have seen a post method generated using this code:

PostMethod authPost = new PostMethod("http://localhost:8000/webTest/j_security_check");
// authPost.setFollowRedirects(false);
NameValuePair[] data = {
new NameValuePair("email", "john%40gmail.com"),
new NameValuePair("password", "mypassword")
};
authPost.setRequestBody(data);
status = client.executeMethod(authPost); 

The main difference here being that the NameValuePair data is set in the request body rather than set as the entity. Does this make a difference? Would this produce the correct request header?

Solution 1

It's always a simple answer! After debugging the C# and another C program I found. I was jumping the gun and doing my own encoding on the email address to remove the @ character. This was the problem!!! Nothing else seemed to make a difference whether it was there or not! The code now looks like:

public void postData(final String email, final String password)
{
    DefaultHttpClient client = new DefaultHttpClient();
    HttpPost post = new HttpPost(LOGIN_URL);
    client.getParams().setParameter("http.useragent", "Custom Browser");
    client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);

    try
    {
        List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
        nameValuePairs.add(new BasicNameValuePair("email", email));
        nameValuePairs.add(new BasicNameValuePair("password", password));
        UrlEncodedFormEntity entity = new UrlEncodedFormEntity(nameValuePairs, HTTP.UTF_8);
        entity.setContentType("application/x-www-form-urlencoded");
        post.setEntity(entity);

        printHeaders(client.execute(post));
        printCookies(client.getCookieStore());
    }
    catch (ClientProtocolException e)
    {
        LOG.error("ClientProtocolException", e);
    }
    catch (IOException e)
    {
        LOG.error("IOException", e);
    }
}

and the output now looks like:

Response Headers:
-Cache-Control: private, no-store
-Content-Type: text/html; charset=utf-8
-Server: Microsoft-IIS/7.0
-X-AspNet-Version: 2.0.50727
-Set-Cookie: USER=3D907C0EB817FD7...92F79616E6E96026E24; domain=.mysite.com; expires=Thu, 10-Jan-2013 20:22:16 GMT; path=/
-Set-Cookie: LOGIN=7B7028DC2DA82...5CA6FB6CD6C2B1; domain=.mysite.com; path=/
-Date: Wed, 11 Jan 2012 20:22:16 GMT
-Content-Length: 165
-Cookie::   [version: 0][name: USER][value: 3D907C0E...E26E24][domain: .mysite.com][path: /][expiry: Thu Jan 10 20:22:16 GMT 2013]
-Cookie::   [version: 0][name: LOGIN][value: 7B7028D...D6C2B1][domain: .mysite.com][path: /][expiry: null]

Solution 2

Both cookies look suspicious. Both use outdated Netscape cookie draft format. Both have invalid domain attribute value. The LOGIN appears malformed (semicolon is missing after the path attribute) on top of that. So, most likely both cookies got rejected by HttpClient.

You can find out whether this is the case by running HttpClient with the context logging turned on as described here: http://hc.apache.org/httpcomponents-client-ga/logging.html

One last remark. Generally one should not meddle with cookie policies when using HttpClient 4.x. The default BEST_MATCH policy will automatically delegate processing of cookies to a particular cookie spec implementation based on the composition of the Set-Cookie header value. In order to disable cookie processing entirely one should remove cookie processing protocol interceptors from the protocol processing chain.

Hope this helps.

Comments

[Sonoman]

Thanks for the assistance. However using the BasicCookieStore in iterate cookies returns no cookies, as does using internal cookie store for DefaultHttpClient. I have no idea why...

[waxwing]

Strange. I used HttpClient in a project just a couple of days ago (version 4.1.2). My code is very similar to yours - with the external BasicCookieStore - but when I iterate over it, the cookies are there.

[Sonoman]

Hmmm... Possibly a bug with 4.0.3? I can't imagine that though as it's a pretty huge pitfall. Are there any other system variables that I need to set? I've seen them mentioned a few times in some docs.

[Sonoman]

Thanks for the insight. I'll remove the cookie policy stuff as soon as I get access to my dev machine. In reference to my edit to the original post above, I don't seem to get any cookies if I don't set the Content-type: application/x-www-form-urlencoded Content-length: 53 properties. However when I try so set Content-Length using HttpClient, I get an exception on the response header saying the ContentLength property has already been set. Any ideas if this could be causing the error and if so, how to fix it?

[ok2c]

@Chris Robinson: Do not meddle with the content-length value! It is quite easy to miscalculate it. Let HttpClient do its job and calculate the value automatically based on the length of content entity enclosed in the request message

[ok2c]

@Chris Robinson: HttpClient being a well behaved HttpClient also adds charset attribute to the Content-Type header value. Apparently the web app is too dumb to parse the Content-Type header value correctly. Use StringEntity#setCotnentType() method to override the default value and set it to 'application/x-www-form-urlencoded' (sans charset attribute)

[Sonoman]

Thanks for the help. I'm a little confused as to how I should create the StringEntity. i.e. what content String to create the entity with and then call entity.setContentType("application/x-www-form-urlencoded"); on. Could you perhaps clarify? Also, does my thinking of adding the NameValuePair data in the request body rather than set as an entity stack up?

[ok2c]

@Chris Robinson: UrlEncodedFormEntity is a subclass of StringEntity. Just set the content type on the UrlEncodedFormEntity instance you are using to submit the login form to the server.

[ok2c]

@Chris Robinson: By the way, I just noticed that if you had not misspelled the content-type header name in your original code it would likely have worked from the start. "httppost.setHeader("ContentType", ..." is obviously wrong.

[Sonoman]

Thanks for the help! I noticed that same thing a short while ago and corrected it but alas, no difference. Turns out I was just being a pillock. But it works now in any case!