UDP multicast client does not see UDP multicast traffic generated by tcpreplay

Question

I have two programs:

• server ... it generates UDP traffic on a chosen multicast
• listener ... it prints UDP traffic on a chosen multicast (it subscribes to a multicast and prints whatever it receives).

When I run the server on one machine and listeners on some (other) machine(s), the listener sees UDP traffic and prints it correctly. So these programs should be in a good shape.

However, when I try to capture the traffic, on whatever machine, with tcpdump:

sudo tcpdump -i eth0 'dst 233.65.120.153' -w 0.pcap

and when I later try to replay it, on whatever machine, with tcpreplay:

sudo tcpreplay -i eth0 0.pcap

none of the listeners sees those captured packets:

09:38:40.975604 IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 32)
    172.27.6.176.53507 > 233.65.120.153.64968: [udp sum ok] UDP, length 4
    0x0000:  4500 0020 0000 4000 0111 6527 ac1b 06b0  [email protected]'....
    0x0010:  e941 7899 d103 fdc8 000c 579c 6162 6364  .Ax.......W.abcd
    0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
09:38:41.975709 IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 32)
    172.27.6.176.53507 > 233.65.120.153.64968: [udp sum ok] UDP, length 4
    0x0000:  4500 0020 0000 4000 0111 6527 ac1b 06b0  [email protected]'....
    0x0010:  e941 7899 d103 fdc8 000c 579c 6162 6364  .Ax.......W.abcd
    0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
09:38:42.975810 IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 32)
    172.27.6.176.53507 > 233.65.120.153.64968: [udp sum ok] UDP, length 4
    0x0000:  4500 0020 0000 4000 0111 6527 ac1b 06b0  [email protected]'....
    0x0010:  e941 7899 d103 fdc8 000c 579c 6162 6364  .Ax.......W.abcd
    0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

Note that even though none of the listeners sees UDP multicast traffic, I am still able to see it, on whatever machine, with tcpdump:

sudo tcpdump -i eth0 'dst 233.65.120.153' -X

My question: What should I do (differently) if I want to tcpreplay the UDP multicast traffic I am creating so that I can see it on application level (e.g. my listener program), not only by tcpdump?

$ cat sender.c

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <time.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>

#define PORT 64968
#define GROUP "233.65.120.153"

main(int argc, char *argv[])
{
     struct sockaddr_in addr;
     int fd, cnt;
     struct ip_mreq mreq;
     char *message="abcd";

     /* Create what looks like an ordinary UDP socket:
        AF_INET    ... IPv4
        SOCK_DGRAM ... UDP
        0          ... required constant
      */
     if ((fd=socket(AF_INET, SOCK_DGRAM, 0)) < 0) {
      perror("socket");
      exit(1);
     }

    /* Set up destination address:
        AF_INET ... IPv4
        GROUP   ... the IP-address of the multicast group
                    to which we want to multicast
        PORT    ... the UDP port that on which we want to multicast
      */
     memset(&addr, 0, sizeof(addr));
     addr.sin_family=AF_INET;
     addr.sin_addr.s_addr=inet_addr(GROUP);
     addr.sin_port=htons(PORT);

     /* now just sendto() our destination! */
     while (1) {
      if (sendto(fd, message, strlen(message), 0, (struct sockaddr *) &addr, sizeof(addr)) < 0) {
           perror("sendto");
           exit(1);
      }
      sleep(1);
     }
}

$ cat listener.c

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <time.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>

#define PORT 64968
#define GROUP "233.65.120.153"
#define MSGBUFSIZE 1000000

char msgbuf[MSGBUFSIZE];

main(int argc, char *argv[])
{
     struct sockaddr_in addr;
     int fd, nbytes,addrlen;
     struct ip_mreq mreq;

     u_int yes=1;

    /* Create what looks like an ordinary UDP socket:
       AF_INET ... IPv4
       SOCK_DGRAM ... UDP
       0 ... required constant
      */
    if ((fd=socket(AF_INET, SOCK_DGRAM, 0)) < 0) {
         perror("socket");
         exit(1);
    }

    /* Allow multiple sockets to use the same PORT number:
       SOL_SOCKET ... manipulate properties of the socket API itself
       SO_REUSEADDR ... Allow reuse of local addresses for bind
     */
    if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)) < 0) {
       perror("Reusing ADDR failed");
       exit(1);
       }

     /* set up destination address */
     memset(&addr,0,sizeof(addr));
     addr.sin_family=AF_INET;
     addr.sin_addr.s_addr=htonl(INADDR_ANY); /* N.B.: differs from sender */
     addr.sin_port=htons(PORT);

     /* bind to receive address */
     if (bind(fd,(struct sockaddr *) &addr,sizeof(addr)) < 0) {
      perror("bind");
      exit(1);
     }

     /* use setsockopt() to request that the kernel join a multicast group */
     mreq.imr_multiaddr.s_addr=inet_addr(GROUP);
     mreq.imr_interface.s_addr=htonl(INADDR_ANY);
     if (setsockopt(fd,IPPROTO_IP,IP_ADD_MEMBERSHIP,&mreq,sizeof(mreq)) < 0) {
      perror("setsockopt");
      exit(1);
     }

     /* now just enter a read-print loop */
     while (1) {
      addrlen=sizeof(addr);
      memset(msgbuf, 0, MSGBUFSIZE);
      if ((nbytes=recvfrom(fd, msgbuf, MSGBUFSIZE,0,
                           (struct sockaddr *) &addr, &addrlen)) < 0) {
           perror("recvfrom");
           exit(1);
      }
      printf("Incoming message size = %d\n", nbytes);
      int i;
      for (i=0; i < nbytes; i++)
              printf("%02x ", ((unsigned char) msgbuf[i]));
      printf("\n");
     }
}

Solution 1

We had the same problem. With tcpdump we saw the data; however, the multicast client/listener was not picking up the data. Then we realized that the Reverse Path Filter (rp_filter) was rejecting the packets.

After disabling the rp-filter, the client/listener application started picking up the packets. Use the below command to disable rp_filter:

echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter

In the above, replace 'eth0' with the interface receiving the multicast if other than eth0

Solution 2

In my case I needed to adjust the pcap file by setting the correct destination MAC address. Also the checksum should be recalculated. And yes, 2 hosts are required for "tcpreplay". Without these I was fighting for a long time but only "tcpdump" showed the replayed stream, not my multicast listening app :(

Here is the link to my article: Step by step instructions on Dump/Replay Multicast feed

Solution 3

To my knowledge, you can't do this on the same box ,tcpreplay bypasses the host's routing table and sends traffic out the interface.

you have to start your listener on a different box. and make sure multicast is enabled. because by default, switch discards multicast traffic.

Solution 4

This is just a theory, but it might be that the packets are discarded by the receiving side due to their checksums being wrong.

That could happen if the machine where you run tcpdump has IP or UDP checksum offloading enabled. That means the packages you capture locally haven't their checksums calculated yet, which the hardware does before sending them out. When you then tcpreplay those packets, the checksums are not calculated, as tcpreplay works on a lower level than the socket API you used to generate the packets.

In order to verify the correctness of the checksums (both those of the dump file as well as those of the packets spit out by the subsequent tcpreplay), tcpdump -v ... will warn you about wrong checksums. wireshark also colors wrongly checksummed frames differently (unless turned off in the wireshark settings).

Did you try to tcpdump the packets only on the sending host, or also on the receiving host? The latter would remove the checksum errors, if that is indeed your problem.

View more solutions

Comments

[Evgeniy Berezovsky]

It's strange you had to change the original, i.e. multicast, destination MAC address (or does your python code not result in a multicast MAC address, which is what you normally have with multicast?) to that of your NIC. This indicates an issue in your network setup. Perhaps a switch/router between your 2 hosts that blocks multicast ethernet frames? You also say in your blog "tcpreplay" cannot push the feed on localhost - but I'm using both tcpreplay (despite the warning it prints) and tcpdump successfully on localhost.

[bingen]

Ya, this happened to me. I could check it with Wireshark (after I enabled proper settings in it), and also doing: "tcrpewrite --fixcsum --infille in.pcap --outfile out.pcap", and then passing out.pcap to tcpreplay (it worked then).

[bingen]

I agree with @eugene-beresovsky. I guess it actually worked because you were passing -C option (= --fixcsum) to tcprewrite. I had a similar issue and changing destination MAC didn't work, but applying the fix checksum option did.

[pterodragon]

It seems like sudo sysctl net.ipv4.conf.all.rp_filter=0 must be enforced in tandem with sudo sysctl net.ipv4.conf.eth0.rp_filter=0 otherwise in my case it won't work

[Matthew Lundberg]

More specifically, it bypasses the network stack entirely, as it opens a raw socket to send. The network stack is where a copy is made to send to a local listener, and bypassing the network stack means that local listeners aren't informed.

[Znik]

In the question has displayed command sudo. It is in the most cases used under linux. then it can be assumed that it is Linux. of course, we can save packets by tcpdump, then we can analize that file under linux or windows by wireshark. this is much easier than analizing IP packet by packet.

[Evgeniy Berezovsky]

rp_filter is an important setting to know. But as the op successfully sends data with his server, it cannot have been the rp_filter in his case.