HtmlAgilityPack Post Login

Question

I'm trying to login to a site using HtmlAgilityPack (site:http://html-agility-pack.net).

Now, I can't exactly figure out how to go about this.

I've tried setting the Html form values via

m_HtmlDoc.DocumentNode.SelectSingleNode("//input[@name='EMAIL']").SetAttributeValue("value", "myemail.com");

I then submit the form with

m_HtmlWeb.Load("http://example.com/", "POST");

This isn't working though. It's not logging in or anything. Does anyone else have any other insight?

Thank you

Solution 1

The HTML Agility Pack is used to parse HTML - you cannot use it to submit forms. Your first line of code changes the parsed nodes in memory. The second line does not post the page to the server - it loads the DOM again, but using the POST method instead of the default GET.

It doesn't look like you need to parse the page at all at this point, since you already know the name of the control. Use the HttpWebRequest class to send a post request to the server, with the string email=acb#example.com in the request.

Here's a sample I wrote when I needed something similar:

/// <summary>
/// Append a url parameter to a string builder, url-encodes the value
/// </summary>
/// <param name="sb"></param>
/// <param name="name"></param>
/// <param name="value"></param>
protected void AppendParameter(StringBuilder sb, string name, string value)
{
    string encodedValue = HttpUtility.UrlEncode(value);
    sb.AppendFormat("{0}={1}&", name, encodedValue);
}

private void SendDataToService()
{
    StringBuilder sb = new StringBuilder();
    AppendParameter(sb, "email", "[email protected]");

    byte[] byteArray = Encoding.UTF8.GetBytes(sb.ToString());

    string url = "http://example.com/"; //or: check where the form goes

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "POST";
    request.ContentType = "application/x-www-form-urlencoded";
    //request.Credentials = CredentialCache.DefaultNetworkCredentials; // ??

    using (Stream requestStream = request.GetRequestStream())
    {
        requestStream.Write(byteArray, 0, byteArray.Length);
    }

    HttpWebResponse response = (HttpWebResponse)request.GetResponse();

    // do something with response
}

Solution 2

If you want to do this with Html Agility Pack. Here's the code.

CookieCollection Cookies = new CookieCollection();
            var web = new HtmlWeb();
            web.OverrideEncoding = Encoding.Default;
            web.UseCookies = true;
            web.PreRequest += (request) =>
            {
                if (request.Method == "POST")
                {
                    string payload = request.Address.Query;
                    byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray());
                    request.ContentLength = buff.Length;
                    request.ContentType = "application/x-www-form-urlencoded";
                    System.IO.Stream reqStream = request.GetRequestStream();
                    reqStream.Write(buff, 0, buff.Length);
                }

                request.CookieContainer.Add(Cookies);

                return true;
            };

            web.PostResponse += (request, response) =>
            {
                if (request.CookieContainer.Count > 0 || response.Cookies.Count > 0)
                {
                    Cookies.Add(response.Cookies);
                }
            };

            string baseUrl = "Your Website URL";
            string urlToHit = baseUrl + "?QueryString with Login Credentials";
            HtmlDocument doc = web.Load(urlToHit, "POST");

Comments

[1poo]

Also if you are going to download and parse pages beyond the login page, you will likely have to pass the received cookies from the login response in you subsequent requests. See this question+answer for more info.

[Kobi]

@Anders - Good tip! I didn't even know it existed... Thanks!

[maplemale]

I don't see how this is an answer. You're assuming a UserName/Password Param exists, which for most security conscious web devs they don't.

[Igor]

I agree with @maplemale! User credentials on "query string" is bad idea(even encrypted). This is not an generic solution!