Android: How to login into webpage programmatically, using HttpsURLConnection

Question

I'm new in Android (and in Java too), so sorry if my problem is a basic proposition! I have to write an Android app, whitch signs into an aspx webpage in the background, get some data from it, and after that logs out form the webpage. (and do that all programmatically)

Basicly, the procedure likes getting email-list from Gmail:
1. go to 'https://mail.google.com', and signs in
2. click to the "Contacts" (== go to "https://mail.google.com/mail/?shva=1&zx=dzi4xmuko5nz#contacts")
3. fetch the page using HttpsURLConnection (or something like this), and get emails in an (e.g. Map or String) object
4. click to the "Sign out" link

I hope, it's understandable. Looking the internet, I find the solution for only the "fetching part", so that's not a problem. But I don't have any idea about the "clicking part".

  ......
    // Get the connection
    URL myurl = new URL("https://mail.google.com");
    HttpsURLConnection con = (HttpsURLConnection) myurl.openConnection();

    // complete the fields
    con.setRequestProperty("Email","myacc");
    con.setRequestProperty("Passwd","mypass");

    /* 
     * in this part, should make sign in, and go directly to contacts... 
     * I don't have any idea how to do it...
     */

    // for the present, just write out the data
    InputStream ins = con.getInputStream();
    BufferedReader in = new BufferedReader(new InputStreamReader(ins));

    String inputLine;
    while ((inputLine = in.readLine()) != null) {
        Log.d("Page:"," "+inputLine);
    }

    in.close();

    /*
     * And here should be the "Sign out" part
     */
  ......

Any help would be great, Thank You for it! (and sorry, if my english isn't so well...)

EDIT: problem solved. Thank You!

 .......    
    String GMAIL_CONTACTS = "https://mail.google.com/mail/?shva=1#contacts";
    String GMAIL_LOGIN = "https://mail.google.com";

        DefaultHttpClient httpClient = new DefaultHttpClient();
        HttpPost httpPost = new HttpPost(GMAIL_LOGIN);

        List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(3);
        nameValuePairs.add(new BasicNameValuePair("Email", MY_ACC));
        nameValuePairs.add(new BasicNameValuePair("Passwd", MY_PASS));
        nameValuePairs.add(new BasicNameValuePair("signIn", "Sign In"));

        httpPost.setEntity(new UrlEncodedFormEntity(nameValuePairs));

        // Execute HTTP Post Request 
        HttpResponse response = httpClient.execute(httpPost);
        Log.d(TAG, "response stat code " + response.getStatusLine().getStatusCode());

        if (response.getStatusLine().getStatusCode() < 400) {

            String cookie = response.getFirstHeader("Set-Cookie")
                    .getValue();
            Log.d(TAG, "cookie: " + cookie);

            // get the contacts page 
            HttpGet getContacts = new HttpGet(GMAIL_CONTACTS);
            getContacts.setHeader("Cookie", cookie);
            response = httpClient.execute(getContacts);

            InputStream ins = response.getEntity().getContent();
            BufferedReader in = new BufferedReader(new InputStreamReader(
                    ins));

            String inputLine;
            while ((inputLine = in.readLine()) != null) {
                Log.d(TAG, " " + inputLine);
            }

            in.close();
        } else {
            Log.d(TAG, "Response error: "
                    + response.getStatusLine().getStatusCode());
        }
 .......

Solution 1

"Clicking" is basically sending a request to a server and displaying the return informations.

1/ find out what url to call for that request (if it is a web page, see firebug for example)

2/ find out what the parameters are, find out if the method is GET or POST

3/ reproduce programmatically.

4/ a "login" phase probably imply the use of a cookie, which the server gives you and that you must send back afterward for each request

However, your approach is wrong. You should not try to login directly to google via url connections. (Also you should use HttpClient). Moreover, request properties are not parameters. They are headers.

I strongly recommend you start with something simpler in order to get comfortable with HTTP in java, GETs, POSTs, parameters, headers, responses, cookies ...

edit

Once you receive the response, you'll want to check that

response.getStatusLine().getStatusCode() < 400

It will tell you that login was successful. (2xx are success, 3xx are moved and such. 4xx are errors in the request, 5xx are server side errors ; Gmail responds 302 to login to suggest redirection to inbox). Then, you'll notice that there is a particular header in the response "Set-Cookie" that contains the cookie you want for further connections so :

String cookie = response.getFistHeader("Set-Cookie");

Then, you should be able to call the request to get the contacts :

HttpGet getContacts = new HttpGet(GMAIL_CONTACTS);
getContacts.setHeader("Cookie", cookie);
response = httpClient.execute(getContacts);
InputStream ins = response.getEntity().getContent();

It should be something like that.

Solution 2

What you are trying to do is parse the Gmail html page. This is wrongn approach as Gmail uses javascript to build the page. Your code would have to emulate browser (execute javascript) for this to work.

If you only need read access to Gmail then use Gmail inbox feed API. This gives you access to unread messages in inbox.

If you need full access then see the Gmail IMAP access. As IMAP is a different protocol then HTTP you'd need to use separate IMAP library for java. See this tutorial.

Comments

[Lama]

Thank You for your answer, but it is not exactly about what I meant. I have to do this operations on a specific (private) website, not on Gmail. I mentioned Gmail just for example, because it's well known for (mostly) everyone, and it works the same as the page I have to fetch (login -> go some page -> logout).

[Lama]

Thank You for your answer! I tried it (create the client, post, excecute a response, and after that open a connection, directly to the 'inside-page'), but the problem is still on. It's still the login-page, what I get. So, the "clicking-part" is still not resolved.

[Snicolas]

there is no "clicking part", you should just load this url with the same client (apache httpcommons) that you used to log in. Also, don't forget to enable cookies on the client as the login is probably stored in some cookie.

[Lama]

1/ "you should just load this url with the same client (apache httpcommons)" 2/ "...enable cookies..." - but how could I do that? I'm new in Java, could You give me some example code, please?

[Lama]

Thank You for the answer! Following this train of thought: The code above uses HttpClient, and the GMAIL_LOGIN is the url, which needs 2 parameters, email, pass; and the method is POST . I think, it was your point (1/ 2/ 3/). My question is, how could I get Cookies in this case, and how could I "navigate bethween pages"? Could You give me some code about it? (I know my knowledge is poor about Http in Java, but the time is close to finish this task, and I'm stucked; that's why I'm asking for help.)

[Lama]

Thanks for the code! I tried it (lots of different type of it), but still not do, what I want. Maybe I miss something... I edit the first comment whit the code, please, if You have any ide, what is wrong, cooment it!