How to capture "dropped packets" in tcpdump

Question

I have a problem with my networking performance. I am using Ubuntu 16.04 on VMware Cloud Server with NIC E1000. But I see some packets dropped in sections of ifconfig command:

root@ubuntu:~# ifconfig ens192
ens192    Link encap:Ethernet  HWaddr 00:50:56:03:25:14  
          inet addr:192.16.1.100  Bcast:192.16.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:574749 errors:0 dropped:83 overruns:0 frame:0
          TX packets:76478 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:44109471 (44.1 MB)  TX bytes:19484534 (19.4 MB)

Although it just some packets dropped but my server is running a real-time game online, so you know it impacts my clients that are connecting to it.

I have done in some researching and exploring information on Google, after that I tried to change config file for buffer ring, max windows size, and so on. But it still drops my packets.

So, now I want to capture packets that dropped for analyzing what type of packets exactly it is.

I also tried this capture for my view in wireshark:

sudo tcpdump -i ens192 -n -w /var/www/html/logs.pcap -C 1 -Z root

But I don't think I can see what packets are dropped! I think packets dropped is ignored before going to the filter of tcpdump.

Can you suggest me what method to capture "dropped packets" above (dropped:83)?

Thanks in advance!

Solution 1

The problem can be with tcpdump itself: If it doesn't respond quickly enough then old packets will be overwritten with new ones, which means that they are dropped.

If you capture all the bytes of each packet, it's very easy to overrun the kernel's packet capture buffer. The symptoms of this overrun are that your packet trace program will report that it dropped packets.

In the case of tcpdump, it prints a summary of how many packets were captured, filtered, and dropped when you stop the capture. For example:

$ sudo tcpdump -i en0 -w trace.pcap
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
94 packets captured
177 packets received by filter
0 packets dropped by kernel

If the dropped count is non-zero, you need to increase the packet capture buffer size by passing the -B option to tcpdump. Try it also without a capture file, to see if this improves the capture ratio.

Solution 2

Your questions seems to pertain to going back and finding out what might have caused a previous packet to drop rather than attempting to capture real time packets to try to find one that drops. For the latter, harrymc's answer should be able to eventually capture one that you're looking for.

To go back and look into what might be happening at that interface, you have to understand that those headers you're seeing for RX packets:574749 errors:0 dropped:83 overruns:0 frame:0 are simply summaries of underlying counters.

From my comment, I would recommend using the ethtool command/package from this answer to analyze some of the counters to see if you can find something that appears out of place.
ethtool -S ens192

From the broadcom drivers source file, we see tp->rx_dropped++; for a few separate occasions in the file. 1 2 3 Any one of these or more (depending on your exact NIC and underlying drivers) contribute to what could be causing dropped packets.

To ease your mind, your server is dropping less than 0.015% of packets received based on your output above. Your clients wouldn't notice any server interruption or even jitter until your drop rate/error rate climbs above 1%. Even then, it would be hardly noticeable. TCP will take care of any of the re-transmissions required.

Comments

[quixotic]

you probably need to hit up iptables for that information. see serverfault.com/a/126078 and serverfault.com/q/385937

[Kamil Maciorowski]

Read this answer. I can't verify it's true but if it is, then you cannot capture those packets at all because they never get to the kernel. My understanding is: if you could capture them, they wouldn't be considered "dropped" it the first place.

[harrymc]

The problem can be with tcpdump itself: If it doesn't respond quickly enough then old packets will be overwritten with new ones. You should maybe get tcpdump to go faster. Why are you limiting the log file size? You may find much more advice here.

[Will]

Consider taking a look at ethtool -S ens192 based on this answer. There can be any number of reasons your interface is dropping packets. May want to also check your outbound switchport to see if it is also detecting dropped/error packets in or out. Malformed and packets that fail checks can be dropped.

[Mihail Kuznesov]

Do you have access to VMware infrastructure? If yes, you can install one more vm, just for packet capturing, and make port mirroring into it. After, you can capture packet in 2 vm in same time. And after, you will need to compare it. If you don't have access to VMware, probably,you can ask your hoster to capture traffic for you. Technically, they can do it. But it can be non-technical problems

[Joey]

@Michael Kuznetsov: I have tested by creating another VM but not thing changed at all. It stills drop packets.

[Mihail Kuznesov]

No, you can mirror port in Vmware switch, and start capturing packet in second vm.

[Mihail Kuznesov]

pubs.vmware.com/vsphere-60/…

[Mihail Kuznesov]

And start tcpdump in your two vm. After, you can compare dumps for diff

[Joey]

Thanks for your comment, but TCPDUMP is just the way we troubleshoot for "What's exactly with "RX: dropped:83"? I want to know why my ubuntu dropped that packets, so I have to know what packets are they. :(

[harrymc]

Have you tried increasing -B? What is your output for tcpdump?

[Joey.Z]

For a real-time online game, 1% drops is sufficient to cause a noticeable lagging of game feel.

[kay27]

Sorry but seems the broadcom driver source had changed. There's only a single line with rx_dropped++ but it is now labelled as drop_it_no_recycle: and there are three gotos to it by the following conditions: 1) failure of build_skb(data, frag_size); 2) failure of netdev_alloc_skb(tp->dev, len + TG3_RAW_IP_ALIGN); 3) length is bigger than MTU. I think we can't just ignore them in some cases. As well as even a single packet lost can spoil the picture of video codec like H.264 or VP8 when you use RTP protocol.