How can I use VB.Net to read the content returned from a URL?

Question

Below is the example code I'm using to get this to work and it does work if I try to read yahoo.com.

Here is the problem. The address I need to read is a java servlet that processes parameters passed in, generates a text document on the server, and then redirects to another URL and returns the address of the text file on the server. I then need to download that text file and process it. I'm having problems connecting to the first URL with the parameters and I think it has to do with the redirect.

I'm using the WebRequest object and I've tried using the HttpWebRequest object. Are there any other objects that support redirects?

TIA

    Dim reader As StreamReader
    Dim request As WebRequest
    Dim response As WebResponse
    Dim data As String = ""

    Try
        request = WebRequest.Create("URL Here")
        request.Timeout = 30000
        response = request.GetResponse()
        reader = New StreamReader(response.GetResponseStream())
        data = reader.ReadToEnd()
    Catch ex As Exception
        MsgBox(ex.Message)
    End Try

    Return data

Solution 1

I think I found something that will work.

I used a WebBrowser control instead.

Have a button that runs this code...

WebBrowser1.Navigate("URL Here")

And this function to process once the request returns.

Private Sub WebBrowser1_Navigated(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserNavigatedEventArgs) Handles WebBrowser1.Navigated    
    MsgBox(WebBrowser1.DocumentText)    
End Sub

Solution 2

Edit

I just tested out HttpWebRequest.Create() and that does handle the 301 and 302 fine with out extra code.

Can you post the error you are seeing

---

You could cast the WebResponse to a HttpWebResponse:

I need to convert this to VB... but it might help you start:

var response = request.GetResponse() as HttpWebResponse;

if (response.StatusCode == HttpStatusCode.Moved || response.StatusCode == HttpStatusCode.Redirect)
{
    // Follow Redirect,  new request based off Redirect
}

// Read Data 

Comments

[BigBlondeViking]

What does the String you are using look like? What is the error message look like... Update you question with this info.

[BigBlondeViking]

So you redirect 2xtimes before getting the final url of the file?

[Dragn1821]

I think it has something to do with the request object because I get a 500 Internal Server error when using the correct address. I tried your suggestion, but I still get the error. Thanks

[Dragn1821]

The address is to a customer's server. The error occurs on the GetResponse() line so it must be coming from their server. But, like I said, if I replace the URL with www.yahoo.com, it works fine. So confused...

[Dragn1821]

See the end of this message for details on invoking just-in-time (JIT) debugging instead of this dialog box.

[Dragn1821]

************** Exception Text ************** System.Net.WebException: The remote server returned an error: (500) Internal Server Error. at System.Net.HttpWebRequest.GetResponse() at HOPRequirements.Form1.GetDataURL() in c:\my documents\My Projects\Repositories\HOPRequirements\HOPRequirements\Form1.‌​vb:line 64 at HOPRequirements.Form1.Button1_Click(Object sender, EventArgs e) in c:\my documents\My Projects\Repositories\HOPRequirements\HOPRequirements\Form1.‌​vb:line 22 at System.Windows.Forms.Control.OnClick(EventArgs e)

[Dragn1821]

at System.Windows.Forms.Button.WndProc(Message& m) at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(M‌​essage& m) at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Mes‌​sage& m) at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

[BigBlondeViking]

Do you know what a 500 error is? status-code.com, What ever you are requesting from the server is causing an error on the server... there is no redirect issues...

[BigBlondeViking]

If you control said server, You should check the error logs on that server... else you are in a hard place. you need to get some info from that server, to see why its throwing errors