Using WebClient or WebRequest to login to a website and access data

Question

I'm trying to access restricted data on a website using WebClient/WebRequest. There is no official API in that website, so what I'm trying to do is simply fill the HTML form and post the values to the server, so I'm logged in.

I tried this and this, but it doesn't look like the upcoming requests are logged in.

The latter example is much more appealing since I obviously prefer WebClient, but legacy WebRequest will do.

Anyway, in the first example I think it did login, but the upcoming requests that access the private data return a page with a message "This is member only content".

How to make a WebClient permanently logged in?

Solution 1

Update:

See my comment below.

---

Here's what I did and it works (credit).

Add this class first:

namespace System.Net
{
  using System.Collections.Specialized;
  using System.Linq;
  using System.Text;

  public class CookieAwareWebClient : WebClient
  {
    public void Login(string loginPageAddress, NameValueCollection loginData)
    {
      CookieContainer container;

      var request = (HttpWebRequest)WebRequest.Create(loginPageAddress);

      request.Method = "POST";
      request.ContentType = "application/x-www-form-urlencoded";

      var query = string.Join("&", 
        loginData.Cast<string>().Select(key => $"{key}={loginData[key]}"));

      var buffer = Encoding.ASCII.GetBytes(query);
      request.ContentLength = buffer.Length;
      var requestStream = request.GetRequestStream();
      requestStream.Write(buffer, 0, buffer.Length);
      requestStream.Close();

      container = request.CookieContainer = new CookieContainer();

      var response = request.GetResponse();
      response.Close();
      CookieContainer = container;
    }

    public CookieAwareWebClient(CookieContainer container)
    {
      CookieContainer = container;
    }

    public CookieAwareWebClient()
      : this(new CookieContainer())
    { }

    public CookieContainer CookieContainer { get; private set; }

    protected override WebRequest GetWebRequest(Uri address)
    {
      var request = (HttpWebRequest)base.GetWebRequest(address);
      request.CookieContainer = CookieContainer;
      return request;
    }
  }
}

Usage:

public static void Main()
{
  var loginAddress = "www.mywebsite.com/login";
  var loginData = new NameValueCollection
    {
      { "username", "shimmy" },
      { "password", "mypassword" }
    };

  var client = new CookieAwareWebClient();
  client.Login(loginAddress, loginData);
}

Solution 2

HTTP is stateless. So, you cannot WebClient permanently logged in. The concept of a session does not exist in HTTP. The server-side technologies such as ASP.NET simulate a stateful behavior through the concept of session using cookie or a query string parameter that gets sent back and forth in every request. Having said that, it is possible to emulate what a browser does from WebClient. If you have access to the website, connect to it using the right credentials and capture the traffic using Fiddler. Then, make sure WebClient sends out the right cookies, request headers, query strings, etc exactly same as the browser.

Comments

[Neshta]

@Anthony, use System.Text.Encoding instead of Encoding if you have the problem with non-static context.

[Kyle Gobel]

you can't loginData.ToString(), it will just be the type name.

[Herohtar]

As pointed out by Kyle, loginData.ToString() returns "System.Collections.Specialized.NameValueCollection".

[Shimmy Weitzhandler]

It used to work. Anyway guys, you can use FormUrlEncodedContent then ReadAsStringAsync instead. Anyway, this answer was good for when it was posted, I'd advise using HttpClient instead, providing a DelegatingHandler that overrides SendAsync to login. I hope to find time to refresh the answer.

[Shimmy Weitzhandler]

Anyway I've edited my answer, I hope it's working now. Don't forget that instead of downvoting and getting mad, this is an open website and you can just edit my answer in by that return a favor to the community and for other people looking at this answer in the future.

[Dennis van Gils]

Is there a way to test if the login was succesfull using this method?

[Shimmy Weitzhandler]

@DennisvanGils you can check the response message from the server by its HTTP status code etc.

[Dennis van Gils]

@Shimmy That didn't work for me as the server always returned a 200 ok code, however using this answer I could check if the container contained a cookie named .ASPXAUTH for the correct site, which did the job

[Sajitha Rathnayake]

what is GetWebRequest?

[Shimmy Weitzhandler]

@SajithaNilan this is an old post. Unless there is a reason you specifically want to use WebClient I'd advise you to use HttpClient instead, overriding the SendRequest method to achieve the Login.