Tcpdump on ec2 not seeing all packets

Question

I'm running tcpdump on an Amazon EC2 instance to monitor HTTP traffic going to Nginx (this is just a test box, the only resource is the example test page).

Running tcpdump with the command

# tcpdump -vn -i any port 80

shows the packets from a browser's request to the site, but shows nothing when accessing the page using a Python script (using the Requests library) or with manually crafted packets (Scapy).

• There's no local firewall running and the Security groups are properly set up.
• The scripts work properly: I can capture the transactions in tcpdump on the local end easily enough, and they return the page with status 200 OK.
• Saving the packets directly to a file (-w) also makes no difference, ruling out buffering issues (I believe?)
• I've also tried seeing if VLANs are causing the issue, but no luck; grepping for "80" still gives no results.

Questions:

1. What could be causing tcpdump to miss these quite specific packets that are definitely getting through the firewall to Nginx and back out again?
2. Why are packets from Firefox being seen whilst packets sent from the scripts slip by?

Thank you

Solution 1

Try using a specific interface instead of "any". That 'device' can't be used in promiscuous mode. See the tcpdump manpage: http://www.tcpdump.org/manpages/tcpdump.1.html

Solution 2

A technology that was released after this question was asked is VPC Flow Logs. This lets you look at network traffic in your VPC, filtered as you like.

VPC Flow Logs won't show you full packet contents, just basics like source, destination, port, protocol, size, time, and action like accept or reject, and a few others. It also won't show you traffic that originates and terminates inside an EC2 instance.

This isn't a direct answer to the question, as it doesn't solve the problem. It may however provide an alternate way to gather information to solve a similar problem.