How to capture HTTP packet with SharpPcap

Question

I would like to capture all incoming HTTP packets of my machine. To do that I'm using SharpPcap which is a WinPcap wrapper.

SharpPcap works very well but it captures TCP packets and this is too low level to do what I want. Does anyone know how can I easly get full HTTP requests/responses from all these TCP packets ?

Thanks

Solution 1

SharpPcap is already able to capture packets in the same manner that wireshark does (just in code rather than a GUI). And you can either parse them directly or you can dump them to the drive in the common .pcap file format.

The steps to parse a capture are:

• Pick an interface
• Open a connection in promiscuous mode
• Start capture either using a while loop or an event callback
• Parse the raw packet to the type that you want

If you're reading .pcap dump files the process is almost the same except you call an offline capture reader, don't need to pick an interface, and don't need to set promiscuous mode. All of the standard filters that wireshark, tcpdump, and most other Pcap frameworks use are supported in SharpPcap. For a reference to these check the tcpdump man.

Currently there is no support for parsing HTTP directly but parsing TCP packets is really easy.

When you receive the raw packet (non parsed) do this:

TCPPacket packet = TCPPacket.GetEncapsulated(rawPacket);

The Packet.Net (A separate and included component of SharpPcap) parser is capable of pulling out the TCP portion directly even if the communication is encapsulated by VPN, PPoE, or PPP.

Once you have the TCPPacket parsed just grab packet.PayloadBytes for the payload in a byte array that should contain the HTTP header in raw bytes that can be converted to the proper text format (I'm not really sure if HTTP headers use UTF-8 or ASCII encoding on that level). There should be plenty of freely available tools/libraries to parse HTTP headers.

---

To extract the HTTP packet from TCP:

You need to collect the tcp packets of the connection as they come in and if the data is fragmented (greater than 1500 bytes) you need to re-assemble the parts in memory. To discover which parts go in what order you need to carefully track the sequence/acknowledgement numbers.

This is a non-trivial thing to accomplish with SharpPcap because you're working with a much lower part of the stack and re-assembling the connection manually.

Wireshark has an interesting article on how to accomplish this in C.

As of right now, SharpPcap doesn't support TCP payload parsing.

---

If you're looking for easy-to-follow examples of how to use SharpPcap download the source tree and look at the example projects included. There is also a tutorial for SharpPcap on codeproject.

If you have more questions and/or you want to make any feature requests to the project, feel free to post on the SourceForge project. It is far from dead and continues to be under active development.

Note: Chris Morgan is the project lead and I'm one of the developers for SharpPcap/Packet.Net.

Update: The tutorial project on code project is now up-to-date to match the current API.

Solution 2

Decoding a TCP stream into HTTP request/response pairs is non-trivial. Tools like WireShark do this with considerable effort.

I wrote a WireShark wrapper for Ruby (not that that will help you), but before I wrote it I tried using tshark (the command-line version of WireShark). That didn't solve my problem but it may work for you. Here's how:

You capture the packets and write them to a pcap file (SharpPcap probably has a way to do this). At some point close the cap file and start another one, then on the old one run tshark with a filter for HTTP traffic, and a flag indicating you want the output in the PDML format. You'll find this is an XML format, easily parsed with the System.Xml tools, which contains the value of every HTTP field in a variety of formats. You can write C# code to spawn tshark, and pipe its StdOut stream into an XML reader so you get the packets out of tshark as they emerge. I don't recommend using the DOM parser as the PDML output for a large capture file can get crazy very quickly.

Unless your requirements are complex (as mine were), this may be all you need.

Comments

[Evan Plaice]

Interesting, is PDML a well-known standard format or an application-specific one-off. The cap in SharpPcap can be done 2 ways: for low/moderate traffic it's possible to capture/parse packets live; for high traffic captures (like a ftp transfer) the best option is to dump the raw cap to .pcap files and postprocess after. SharpPcap is basically just a cross platform framework for building wireshark-like captures into C# applications. Right now SharpPcap is the pcap wrapper and Packet.Net is the parser library.

[Evan Plaice]

Do you have the code posted somewhere (ex GitHub). If so, I'd like to take a look at it at some point. I'd like to see how much code/effort it took to manage the TCP re-assembly step.

[anelson]

I don't know if PDML is a standard or something specific to wireshark; in any case it's easy to process.

[anelson]

The code for my Ruby extension is at github.com/anelson/rcapdissector. Note that this was built against a version of WireShark from a couple years ago; it probably will not compile without modification using the latest WireShark sources.

[Evan Plaice]

Thanks. I don't need it to compile. I'd just like to see how it works out of personal curiosity. I have spent a lot of time reading RFCs and implementing parsers so I have an appreciation for working from (ambiguous) specifications to usable code.

[C4d]

GetEncapsulated equals to Packet.ParsePacket(e.Packet.LinkLayerType, e.Packet.Data); right?