Timestamp outgoing packets

Question

I'm trying to get accurate timestamps for outgoing packets (sent using raw sockets). According to Linux/Documentation/networking/timestamping.txt, "For send time stamps the outgoing packet is looped back to the socket's error queue with the send time stamp(s) attached. It can be received with recvmsg(flags=MSG_ERRQUEUE).".

Unfortunately, recvmsg is always returning -1 when called on a raw socket (created with socket(PF_INET, SOCK_RAW, IPPROTO_RAW) and with SO_TIMESTAMP set to 1 with setsockopt). What am I doing wrong? Is there a better way of getting an accurate timestamp for an outgoing packet?

Addendum (information):

I also tried getting the timestamp from a packet sent through an UDP socket (source code below) and recvmsg returns -1: the error is "Resource temporarily unavailable" (EAGAIN).

Addendum (source code):

#include <arpa/inet.h>
#include <linux/net_tstamp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>

void die(char* s)
{
    perror(s);
    exit(1);
}

int main(int argc, char* argv[])
{
    char* destination_ip = "10.0.0.1";
    int destination_port = 1234;

    int sock;
    if ((sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
        die("socket()");
    }

    int timestamp_flags = SOF_TIMESTAMPING_TX_SOFTWARE;
    if (setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &timestamp_flags, sizeof(timestamp_flags)) < 0) {
        die("setsockopt()");
    }

    struct sockaddr_in si_server;
    memset(&si_server, 0, sizeof(si_server));
    si_server.sin_family = AF_INET;
    si_server.sin_port = htons(destination_port);
    if (inet_aton(destination_ip, &si_server.sin_addr) == 0) {
        die("inet_aton()");
    }

    const int buffer_len = 256;
    char buffer[buffer_len];

    const int n_packets = 10;
    for (int i = 0; i < n_packets; ++i) {
        sprintf(buffer, "Packet %d", i);
        if (sendto(sock, buffer, buffer_len, 0, (const sockaddr*) &si_server, sizeof(si_server)) < 0) {
            die("sendto()");
        }

        // Obtain the sent packet timestamp.
        char data[256];
        struct msghdr msg;
        struct iovec entry;
        struct sockaddr_in from_addr;
        struct {
            struct cmsghdr cm;
            char control[512];
        } control;
        int res;

        memset(&msg, 0, sizeof(msg));
        msg.msg_iov = &entry;
        msg.msg_iovlen = 1;
        entry.iov_base = data;
        entry.iov_len = sizeof(data);
        msg.msg_name = (caddr_t)&from_addr;
        msg.msg_namelen = sizeof(from_addr);
        msg.msg_control = &control;
        msg.msg_controllen = sizeof(control);        
        if (recvmsg(sock, &msg, MSG_ERRQUEUE) < 0) {
            die("recvmsg()");
        }
    }
    return 0;
}

Solution 1

Looking into the Linux kernel source code, I found that the function responsible for putting the message containing the timestamp of the packet on the error queue is skb_tx_timestamp. This function is supposed to be called by the NIC driver and unfortunately, the e1000 driver doesn't call it (there's a similar function for hardware timestamping, but this is obviously dependent on the NIC driver supporting it).

According to this NetDev discussion from last September, "no driver calls skb_tx_timestamp()" and "You'll need to tweak your NIC driver to play with this TX timestamps". After adding a call to skb_tx_timestamp to e1000_xmit_frame on e1000_main.c, I was able to obtain timestamps for outgoing packets (through an UDP socket). I wasn't able to obtain timestamps for outgoing packets on a RAW socket, though (I still get EAGAIN).

Solution 2

It's hard to know what you are doing wrong, since we cannot see your code.

However: The documentation says that SO_TIMESTAMP is for incoming packets, while SO_TIMESTAMPING is for outgoing packets.

The kernel documentation contains a full example which you could use as a base - though it's using UDP, but you should be able to adjust it to using a RAW socket. See the linux kernel Documentation/networking/timestamping/timestamping.c

EDIT: It seems transmit timestamping isn't universally supported, see e.g. here. Even today just a handful of nic drivers implement software support, and a few have hardware support.

Solution 3

sock_tx_timestamp is only called for SOCK_DGRAM sockets in current kernel code.

BTW, the document Documentation/networking/timestamping/timestamping.c isn't very accurate.

SO_TIMESTAMP / SO_TIMESTAMPNS / SO_TIMESTAMPING / SIOCGSTAMP / SIOCGSTAMPNS are similar. Anyone of them will enable application getting the timestamp of a received packet.

With SOF_TIMESTAMPING_TX_SOFTWARE, any one of the above flags will also provide the application a CMSG in MSG_ERRQUEUE, indicating the timestamp of a sent packet.

But SOF_TIMESTAMPING_RX_SOFTWARE is useless at all. It can not even be used to disable the reporting of timestamp of received packets.

Comments

[bruno nery]

My bad, I added the source code (trying to get the timestamp from UDP first). And yes, my code is based on timestamping.c. However, it still doesn't work (recvmsg tells me "Resource temporarily unavailable").

[nos]

That sounds like EAGAIN, which means there's nothing in the error queue (yet). So you'll have to re-try the recvmsg , or use select()/poll() to get notified when something's there. recvmsg with the MSG_ERRQUEUE flag does not block until something arrives. But see my last EDIT.

[bruno nery]

I just saw your last edit - please see my own response :). We both found the same, but I still have a problem: it doesn't seem to work with RAW sockets. Any idea why?

[nos]

Well, with the current code, it's just luck of you don't get EAGAIN, so at least make sure you do what I said in the last comment. Also, try to verify that your raw packet actually makes it onto the wire, e.g. debug it with wireshark.

[bruno nery]

You're right, I got lucky :). I started getting EAGAIN with the UDP socket (but select made it go away). As for the raw packet, select blocks (not using a timeout) and yes, the raw packet makes it into the wire (tcpdump shows them, and I also get the responses).

[notAChance]

I get strange behaviour with basically the above code - I am only returned a timestamp if the packet successfully arrives at its destination and only once the icmp response is received. This is contrary to the docs - lxr.free-electrons.com/source/Documentation/networking/… - at line 326.

[notAChance]

Nevermind - SOF_TIMESTAMPING_TX_SOFTWARE is not enabled on this kernel. I assume the transmit timestamp enabled kernels/drviers deal with the loopback error queue issue, explains why the select() was only unblocking when the ICMP response was received. Hope this can be of some use to someone somewhere...