Fragmentation and packet size,using tcpdump

Question

i am trying to understand concept of fragmentation:

i have two virtual machines with public ip connected to a switch.

tracepath shows packet not going through gateway

from vm1: Trying to send icmp with 65507 bytes to vm2.

ping -M want -s 65507 vm2 

but in tcpdump output on vm2: its showing

tcpdump -evvv icmp

12:48:44.635551 42:43:30:b4:89:0c (oui Unknown) > b6:7a:6b:7d:54:32 (oui Unknown), ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 10843, offset 1480, flags [+], proto ICMP (1), length 1500)
VM1 > VM2: icmp
12:48:44.635568 42:43:30:b4:89:0c (oui Unknown) > b6:7a:6b:7d:54:32 (oui Unknown), ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 10843, offset 2960, flags [+], proto ICMP (1), length 1500)
VM1 > Vm2: icmp
12:48:44.635572 42:43:30:b4:89:0c (oui Unknown) > b6:7a:6b:7d:54:32 (oui Unknown), ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 10843, offset 4440, flags [+], proto ICMP (1), length 1500)
VM1>VM2 icmp
12:48:44.635575 42:43:30:b4:89:0c (oui Unknown) > b6:7a:6b:7d:54:32 (oui Unknown), ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 10843, offset 5920, flags [+], proto ICMP (1), length 1500)
VM1>VM2: icmp
12:48:44.635578 42:43:30:b4:89:0c (oui Unknown) > b6:7a:6b:7d:54:32 (oui Unknown), ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 10843, offset 7400, flags [+], proto ICMP (1), length 1500)
**Vm1 > VM2**: icmp
12:48:44.635581 42:43:30:b4:89:0c (oui Unknown) > b6:7a:6b:7d:54:32 (oui Unknown), ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 10843, offset 8880, flags [+], proto ICMP (1), length 1500)

This is repeated 31 times until is received fully. full paste: http://pastebin.com/cnQhn8dK

So why it looks like total data received is 1500*31=46500 bytes and what happened to 65507-46500=19007 bytes.

Can some one please clarify this.

Solution 1

Yes, using tcpdump with -s option, the result is now right. As your result, we count 45 packet. 44 packets with 1500 bytes, 1 packet 415 bytes.

44*1500 + 415 = 66415

66415 - 65507 = 908

908 / 45 = 20 plus 8

You can see, each packet add 20 byte for ip header + 8 byte icmp header for the first packet.

Solution 2

Just to add on Gnouc's answer.

You are sending 65507 bytes of data. This doesn't include the ICMP headers which are 8 bytes.

The most common MTU size is 1500 . This size accounts for Layer 3 size that's why you see ethertype IPv4 (0x0800), length 1514: which means that the total frame size is actually 1514 bytes. These 14 bytes account for the Ethernet header. 12 bytes per mac address + 2 for the type.

The minimum and very common size for an IP Header is its minumum size 20 bytes (Maximum is 60 bytes).

So we have

1514 bytes - Ethernet header = 1500 bytes
1500 - IP header = 1480 bytes
1480 - ICMP header = 1472 bytes

You can send at most 1472 bytes without fragmentation.

BUT the way that IP fragments the packets it doesn't require to send the header for every packet but only for the first packet.

WITH fragmentation the maximum amount of bytes that you can send given an MTU of 1500 is 1480 bytes.

We know the total size of your data and what's the maximum you can send.

So it's going to take at least 65507 / 1480 ~= 44.2 packets . I.e. 44 packets of 1514 and then a final packet of less than half the size.

What happened to the rest of the packets?

But why 31 packets? Well it's all in the buffer size that you are capturing. At the end of your tcpdump you should be seeing

31 packets captured 
57 packets received by filter
14 packets dropped by kernel

If you add the packets captured + the packets dropped by kernel you will get to the right answer and that's what -s it alters the snap length.

From the tcpdump manual

-s Snarf snaplen bytes of data from each packet rather than the default of 65535 bytes. Packets truncated because of a limited snapshot are indicated in the output with ``[|proto]'', where proto is the name of the protocol level at which the truncation has occurred. Note that taking larger snapshots both increases the amount of time it takes to process packets and, effectively, decreases the amount of packet buffering. This may cause packets to be lost. You should limit snaplen to the smallest number that will capture the protocol information you're interested in. Setting snaplen to 0 sets it to the default of 65535, for backwards compatibility with recent older versions of tcpdump.

Why did the kernel drop the packets in the first place?

Again from the tcpdump manpage

packets dropped by kernel (this is the number of packets that were dropped, due to a lack of buffer space, by the packet capture mechanism in the OS on which tcpdump is running, if the OS reports that information to applications; if not, it will be reported as 0).

Comments

[cuonglm]

Please post more output of tcpdump, about 5 to 10 packets.

[Kevin Parker]

@Gnouc now can you please check.

[cuonglm]

What are parameters of tcpdump command you have used?

[Kevin Parker]

tcpdump -evvv icmp

[cuonglm]

Try: tcpdump -s 1514 -evvv and check result.

[Kevin Parker]

can u tell me what this "-s" will do?.With -s its showing a longer dump.

[cuonglm]

It changes the capture size of packets, you can man tcpdump for more details. What is your result?

[Kevin Parker]

pastebin.com/gXvfaVjg Now its showing 44 packets.

[Kevin Parker]

let us continue this discussion in chat

[cuonglm]

ICMP header for the last packet or the first?

[user]

@Gnouc you are right, fixed. The headers always go on the first packet.

[Kevin Parker]

you are right,with out s there are lots of dropped packets and filtered packets.So that means fragmented packets are reaching destination,but tcpdump can not show it due to low buffer space.Can you tell me who fragments these large icmp packet,since i am using L2 switch.

[user]

The linux kernel does. And any other known OS as well.