What tool can I use to sniff HTTP/HTTPS traffic?

Question

I'm looking for a command line tool that can intercept HTTP/HTTPS requests, extract information such as: (content, destination, etc.), perform various analysis tasks, and finally determine if the request should be dropped or not. Legal requests must than be forwarded to the application.

A tool that is similar in nature to tcpdump, Wireshark, or Snort, but operates at the HTTP level.

References

• Intercept HTTP requests on Linux

Solution 1

Try mitmproxy.

• mitmproxy is an SSL-capable man-in-the-middle proxy for HTTP. It provides a console interface that allows traffic flows to be inspected and edited on the fly.

• mitmdump is the command-line version of mitmproxy, with the same functionality but without the user interface. Think tcpdump for HTTP.

Features

• Intercept HTTP requests and responses and modify them on the fly.
• Save complete HTTP conversations for later replay and analysis.
• Replay the client-side of an HTTP conversations.
• Replay HTTP responses of a previously recorded server.
• Reverse proxy mode to forward traffic to a specified server.
• Make scripted changes to HTTP traffic using Python.
• SSL certificates for interception are generated on the fly.

Screenshot

   

Example

I setup an example Jekyll Bootstrap app which is listening on port 4000 on my localhost. To intercept it's traffic I'd do the following:

% mitmproxy --mode reverse:http://localhost:4000 -p 4001

Then connect to my mitmproxy on port 4001 from my web browser (http://localhost:4001), resulting in this in mitmproxy:

   

You can then select any of the GET results to see the header info associated to that GET:

   

References

• mitmproxy documentation
• How mitmproxy works & Modes of Operation

Solution 2

mitmproxy/mitmdump

Equalivant to tcpdump for HTTPS is mitmdump. Here are the steps:

1. Install mitmproxy package (macOS: brew install mitmproxy).

2. Install mitmproxy CA certificate by the following commands:

   $ mitmdump --mode reverse:http://mitm.it/ -p 8080
   $ wget --content-disposition http://localhost:8080/cert/pem
   $ open mitmproxy-ca-cert.pem
   # Open, install and mark the certificate as trusted.
   

Now, here is the simple test on how to test reverse proxy:

1. Run: mitmdump --mode reverse:https://example.com/ -p 4433.

2. In another shell, run: curl https://localhost:4433.

   Now, you should see the page source and mitmdump command should produce the output like:

   Proxy server listening at http://*:4433
   [::1]:49446: clientconnect
   [::1]:49446: GET https://example.com/ HTTP/2.0
             << 200  1.24k
   [::1]:49446: clientdisconnect
   

For all traffic, just run: mitmdump or mitmproxy.

See: mitmproxy docs page for more details.

---

Charles Proxy

If you're on macOS, there is also Charles Proxy app (GUI) which allows view all of the HTTP and SSL/HTTPS traffic between the hosts.

Solution 3

For some situations, you can use a proxy that accepts incoming HTTP requests and makes outgoing HTTPS requests. As an example, I wanted to capture the traffic between git and github.com. I used mitmproxy:

mitmproxy -s httpser.py

where httpser.py is:

def request(context, flow):
  flow.request.scheme = 'https'
  flow.request.port = 443

I then ran git like so:

export http_proxy="http://127.0.0.1:8080/"
git clone http://github.com/oxplot/difftr

Now using wireshark listening on localhost, one can capture the plain traffic. Without the proxy, github would redirect git to use HTTPS.