301 Moved Permanently

Question

I'm trying to get HTML by URL in Java. But 301 Moved Permanently is all that I've got. Another URLs work. What's wrong? This is my code:

 hh= new URL("http://hh.ru");
        in = new BufferedReader(
                new InputStreamReader(hh.openStream()));


        while ((inputLine = in.readLine()) != null) {

            sb.append(inputLine).append("\n");
            str=sb.toString();//returns 301


        }

Solution 1

You're facing a redirect to other URL. It's quite normal and web site may have many reasons to redirect you. Just follow the redirect based on "Location" HTTP header like that:

URL hh= new URL("http://hh.ru");
URLConnection connection = hh.openConnection();
String redirect = connection.getHeaderField("Location");
if (redirect != null){
    connection = new URL(redirect).openConnection();
}
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
System.out.println();
while ((inputLine = in.readLine()) != null) {
    System.out.println(inputLine);
}

Your browser is following redirects automaticaly, but using URLConnection you should do it by your own. If it bothers you take a look at other Java HTTP client implementations, like Apache HTTP Client. Most of them are able to follow redirect automatically.

Solution 2

found this answer useful and improved a little due to the possibility of multiple redirections (e.g. 307 then 301).

URLConnection urlConnection = url.openConnection();
                String redirect = urlConnection.getHeaderField("Location");
                for (int i = 0; i < MAX_REDIRECTS ; i++) {
                    if (redirect != null) {
                        urlConnection = new URL(redirect).openConnection();
                        redirect = urlConnection.getHeaderField("Location");
                    } else {
                        break;
                    }
                }

Solution 3

There's nothing wrong with your code. The message means that hh.ru is permanently moved to another domain.

Comments

[Tony]

But this shows me HTML of mobile version. Look m.hh.ru. I want full version.

[Jk1]

It looks like version depends on user agent you're setting. This one gives a full version in my test: connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");

[Tony]

How can I test it next time if it doesn't work? I mean which strategy should I use to fix these parameters?

[Jk1]

If you want to fetch the same pages as browser does, then behave as similar to browser as possible: 1. Follow redirects and handle other HTTP codes 2. Set typical headers (User-agent, Accept, Referrer, etc) when making a request 3. Use cookies and maitain authentication, if necesary

[Arefe]

This simple answer helped me as the domain moved from the http to https

[gouessej]

@Jk1 Maybe it would be a good idea to look at the error code before considering that the header field "location" is a redirection, wouldn't it? connection.getResponseCode()?

[Jk1]

@gouessej the only legitimate use case for Location header other than 3XX is 201 Created, which is almost impossible to receive in response to the GET request. On the other hand, the response code check is always a good idea, whatever your client logic may be.