HttpWebRequest.GetResponse() returns error 500 Internal Server Error

Question

I'm using HttpWebRequest to make a request to a url:

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(urlAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

but it throws error 500 (Internal Server Error) but when i visit the URLAddress with browser it works fine, urlAddress= www.khademnews.com

it is a simple GET operation but it throws an exception for me how can I solve this?

Solution

You might need to set up the user agent as some sites might require it. Also you could use a WebClient to simplify your code:

using (var client = new WebClient())
{
    client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0";
    string result = client.DownloadString("http://www.khademnews.com");
}

The server might expect other headers as well. You could check with FireBug which headers are sent went you perform the request in your browser and add those headers.