Sending HTTP Request GET/POST to form with Java?

Question

So I have this piece of code, and I got it to work, and now it basically allows me to send http post and get requests to almost any external website I want UNLESS the elements don't contain a name attribute. Here's an example:

This is the Java code:

    public static String sendPostRequest(String url) {

    StringBuffer sb = null;

    try {

        String data = URLEncoder.encode("user", "UTF-8") + "="
                + URLEncoder.encode("myUserName", "UTF-8") + "&"
                + URLEncoder.encode("submit", "UTF-8") + "="
                + URLEncoder.encode("Submit", "UTF-8");


        URL requestUrl = new URL(url);
        HttpURLConnection conn = (HttpURLConnection) requestUrl
                .openConnection();
        conn.setDoOutput(true);
        conn.setRequestMethod("GET");

        OutputStreamWriter osw = new OutputStreamWriter(
                conn.getOutputStream());
        osw.write(data);
        osw.flush();

        BufferedReader br = new BufferedReader(new InputStreamReader(
                conn.getInputStream()));

        String in = "";
        sb = new StringBuffer();

        while ((in = br.readLine()) != null) {
            sb.append(in + "\n");
        }

        osw.close();
        br.close();
    } catch (UnsupportedEncodingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (MalformedURLException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    return sb.toString();
}

This is the form I'm trying to send a request to (it's a form on the w3schools site, this being the site http://www.w3schools.com/html/html_forms.asp):

<form name="input0" target="_blank" action="html_form_action.asp" method="get">

Username: 

<input type="text" name="user" size="20" />

<input type="submit" value="Submit" />

</form>

Now because the Submit button doesn't have a name attribute, I can't send a proper HTTP Get/Post request to it (I know it's a get method in this case). What do I replace the String data with (what proper keys/values) so that it actually sends a request to this form?

Solution 1

You don't add the submit part to your data at all. This is just for the browser to know that "Submit" button triggers the action. Notice how the URL of the newly opened site looks: http://www.w3schools.com/html/html_form_action.asp?user=myUserName - no submit part here. So your data code should look like this:

String data = URLEncoder.encode("user", "UTF-8") + "="
            + URLEncoder.encode("myUserName", "UTF-8"); // end here

Solution 2

I am using HttpClient for generating http request

HttpClient is open source apache project. you can get widely code. HttpClient version 4.1 is good set of Http api

HttpClient Learning Artical

Comments

[ZimZim]

ehm that doesn't work, whether I use it on the html_form_action.asp site or the html_forms.asp site. And I've tried it with other sites, it doesn't work unless I add the submit button part to it...

[ZimZim]

How is this in any way relative to what I'm trying to do? I'm asking what keys/values I should be using to send an HTTP Post Request to a specific site... not about some kind of API

[Bartosz Moczulski]

This may be because GET requests are automatically replaced by POST when you call getOutputStream(). This is a reasonable thing to do because usually that's what the author meant. To truly issue a GET request remove entire OutputStreamWriter part and append your data to request URL like it should be in a valid GET request: URL requestUrl = new URL(url + "?" + data);. Of course you still don't add the submit part.

[ZimZim]

But that wouldn't be a GET request, that would just be directly getting the HTML source of a certain URL. I need to send a request. I tried to do this by removing the outputstream entirely, but it still just returns the wrong data.

[Ankur Loriya]

@user1007059 : HttpClient will give you better readability you can solve you problem... i am using this httpclien 4.1 in my project and its fetch data from any URL.

[Bartosz Moczulski]

Well, that's what a GET request indeed is - just a URL with query part (?someData=value&otherData=value...). You can just as well type it manually in your browser address bar and press enter. It's the same as if you filled the form and pressed the submit button. Think of GET as something like POST with all data moved from body into URL behind the question mark. It's a bit oversimplified but will do to catch the idea. Take a look here for full code: pastebin.com/Zmf6irXC

[mateuszb]

It should be noted that this uses Apache libraries.