Converting Raw HTTP Request into HTTPWebRequest Object

Question

In .NET is it possible to convert a raw HTTP request to HTTPWebRequest object?

I'm sure .NET internally doing it. Any idea which part of the .NET is actually handling this? Can I call it or is there any external library which allows raw HTTP connections?

Solution 1

I dont believe there is an exposed method to do this. You may have to find or write a parser to break the request up and then write your own class that extends HttpWebRequest.

Here is what looks like a parser from CodeProject:

http://www.codeproject.com/KB/IP/CSHTTPServer.aspx

I looked at the rotor code for the HttpWebRequest (briefly) and I did not see anything that stood out as a silver bullet. Here is the link to the file:

http://www.123aspx.com/Rotor/RotorSrc.aspx?rot=40844

All of the rotor code is here for browsing online:

http://www.123aspx.com/Rotor/default.aspx

And here you can download it:

http://www.microsoft.com/downloads/details.aspx?FamilyId=8C09FD61-3F26-4555-AE17-3121B4F51D4D&displaylang=en

I know a bunch of links doesn't really answer your question, but I don't think the functionality that you are looking for is exposed in the framework. I would love to be proven wrong, so please update the post if you find a good way of doing it. I know tools out there must do it, anything written in .Net that logs raw requests and then lets you resubmit them is doing something similar. I believe fiddler (http://www.fiddler2.com) is written in .Net, you may want to shoot an email over to those guys and see if they can help.

Solution 2

Its possible now, but only with .Net Core 2.0+. Use Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser class:

using Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http;
using System;
using System.Buffers;
using System.Collections.Generic;
using System.Text;

public class Program : IHttpRequestLineHandler, IHttpHeadersHandler
{
    public static void Main(string[] args)
    {
        string requestString =
@"POST /resource/?query_id=0 HTTP/1.1
Host: example.com
User-Agent: custom
Accept: */*
Connection: close
Content-Length: 20
Content-Type: application/json

{""key1"":1, ""key2"":2}";
        byte[] requestRaw = Encoding.UTF8.GetBytes(requestString);
        ReadOnlySequence<byte> buffer = new ReadOnlySequence<byte>(requestRaw);
        HttpParser<Program> parser = new HttpParser<Program>();
        Program app = new Program();
        Console.WriteLine("Start line:");
        parser.ParseRequestLine(app, buffer, out var consumed, out var examined);
        buffer = buffer.Slice(consumed);
        Console.WriteLine("Headers:");
        parser.ParseHeaders(app, buffer, out consumed, out examined, out var b);
        buffer = buffer.Slice(consumed);
        string body = Encoding.UTF8.GetString(buffer.ToArray());
        Dictionary<string, int> bodyObject = Newtonsoft.Json.JsonConvert.DeserializeObject<Dictionary<string, int>>(body);
        Console.WriteLine("Body:");
        foreach (var item in bodyObject)
            Console.WriteLine($"key: {item.Key}, value: {item.Value}");
        Console.ReadKey();
    }

    public void OnHeader(Span<byte> name, Span<byte> value)
    {
        Console.WriteLine($"{Encoding.UTF8.GetString(name)}: {Encoding.UTF8.GetString(value)}");
    }

    public void OnStartLine(HttpMethod method, HttpVersion version, Span<byte> target, Span<byte> path, Span<byte> query, Span<byte> customMethod, bool pathEncoded)
    {
        Console.WriteLine($"method: {method}");
        Console.WriteLine($"version: {version}");
        Console.WriteLine($"target: {Encoding.UTF8.GetString(target)}");
        Console.WriteLine($"path: {Encoding.UTF8.GetString(path)}");
        Console.WriteLine($"query: {Encoding.UTF8.GetString(query)}");
        Console.WriteLine($"customMethod: {Encoding.UTF8.GetString(customMethod)}");
        Console.WriteLine($"pathEncoded: {pathEncoded}");
    }
}

Output:

Start line:
method: Post
version: Http11
target: /resource/?query_id=0
path: /resource/
query: ?query_id=0
customMethod:
pathEncoded: False
Headers:
Host: example.com
User-Agent: custom
Accept: */*
Connection: close
Content-Length: 20
Content-Type: application/json
Body:
key: key1, value: 1
key: key2, value: 2

Comments

[Chris Fong]

What do you need to do? An HttpRequest object is typically what you'd program against if you're doing ASP.Net stuff in a hosted enviroment. A HttpWebRequest is usually what you program against in client/server apps.

[Djonatas Tenfen]

This is a raw HTTP Request : GET /index.php HTTP/1.1 Host: www.example.com

[vapcguy]

And would look like this in code: var request = "GET /index.php HTTP/1.1\r\n" + "Host: www.example.com\r\n" + "Content-Length: 0\r\n" + "\r\n"; according to an answer in a related question, which seems to answer this one: stackoverflow.com/questions/11862890/…

[vapcguy]

Except that now the link is broken. People, stop just posting links and provide real answers! "Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline" stackoverflow.com/help/how-to-answer

[vapcguy]

Except that now the 123aspx.com link is broken. People, stop just posting links and provide real answers! "Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline" stackoverflow.com/help/how-to-answer

[derekantrican]

Practically all the links are broken in this answer

[Wasted_Coder]

The HttpParser from CodeProject is perfect! It is very well designed with multiple threading, but still, lite! A very good example if you need to write your own parser or HttpServer with complete control over the source (for any proprietor differences).

[Stanislav Prusac]

In version net5.0, the interface of this function, OnStartLine(..), has been changed.

[Ivan P.]

HttpParser on Code Project does not work well with large POST requests