[PATCH 0/1] Pass correct tun device's RX queue number to XDP program

[Gilberto Bertin <me@jibi.io>]

Hi,

I'm trying to run XDP on top of a tun device in multi queue mode as that allows
me to easily play on my laptop with an interface that supports multiple queues.

While testing it I found an issue: regardless of the the tun fd (i.e. RX queue)
I'm writing packets to, the XDP program thinks all packets are coming from RX
queue 0.

I'd like to propose a fix for that (see next mail) but let me first share some
context.

Here's what I'm using to setup a tun device with multiple RX/TX queues and write
some test packets into all of its RX queues:

        struct ifreq ifr = {
                .ifr_flags = IFF_TAP | IFF_NO_PI | IFF_MULTI_QUEUE, .ifr_name = "mytun0"
        };

        int fds[4];
        for (int i = 0; i < 4; i++) {
                if ((fds[i] = open("/dev/net/tun", O_RDWR)) < 0)
                        return -1;
                if (ioctl(fds[i], TUNSETIFF, (void *)&ifr))
                        return -1;
        }

	while (1) {
		for (int i = 0 ; i < 4; i++)
			write(fds[i], "\x12\x34\x56\x78\x90\xab\x12\x34\x56\x67\x90\xab\x08\x00", 14);
		sleep(2);
	}

Next I reproduced the problem with this XDP program attached to the tun device
created with the previous code snippet:

        $ cat xdp_kern.c
        #include <linux/bpf.h>
        #include <bpf_helpers.h>

        SEC("prog")
        unsigned int xdp_prog(struct xdp_md *xdp) {
                bpf_printk("rx queue: %d\n", xdp->rx_queue_index);
                return XDP_PASS;
        }

        char _license[] SEC("license") = "GPL";

        $ sudo ip link set dev mytun0 xdpdrv obj ./xdp_kern.o

        $ sudo bpftool prog tracelog
                   a.out-22189 [003] .... 14444.594609: 0: rx queue: 0
                   a.out-22189 [003] .... 14444.594645: 0: rx queue: 0
                   a.out-22189 [003] .... 14444.594653: 0: rx queue: 0
                   a.out-22189 [003] .... 14444.594661: 0: rx queue: 0

Here the XDP program is reporting all packets coming from RX queue 0 when they
should have been coming from RX queues 0..3.

After looking into the tun driver I ended up writing this stap script which
reveals the issue:

        $ cat rx_queue.stap
        probe kernel.function("tun_get_user") {
                printf("tun_get_user(): tfile->xdp_rxq.queue_index: %d\n", $tfile->xdp_rxq->queue_index);
        }

        probe kernel.function("netif_receive_generic_xdp") {
                printf("netif_receive_generic_xdp(): skb->queue_mapping: %d\n", $skb->queue_mapping);
                printf("\n");
        }

        $ sudo stap ./rx_queue.stap
        tun_get_user(): tfile->xdp_rxq.queue_index: 0
        netif_receive_generic_xdp(): skb->queue_mapping: 0

        tun_get_user(): tfile->xdp_rxq.queue_index: 1
        netif_receive_generic_xdp(): skb->queue_mapping: 0

        tun_get_user(): tfile->xdp_rxq.queue_index: 2
        netif_receive_generic_xdp(): skb->queue_mapping: 0

        tun_get_user(): tfile->xdp_rxq.queue_index: 3
        netif_receive_generic_xdp(): skb->queue_mapping: 0

It looks like the tun driver (specifically tun_get_user()) is not passing
correctly the information about the RX queue to do_xdp_generic().

Setting the queue_mapping field in the skb with skb_record_rx_queue() seems to
fix the issue.

Same test with the patched kernel:
        sudo bpftool prog tracelog
                   a.out-791   [004] ....  4826.499356: 0: rx queue: 0
                   a.out-791   [004] ....  4826.499532: 0: rx queue: 1
                   a.out-791   [004] ....  4826.499541: 0: rx queue: 2
                   a.out-791   [004] ....  4826.499549: 0: rx queue: 3

While at it, I also fixed tun_xdp_one(), which was calling skb_record_rx_queue()
after invoking do_xdp_generic().

Gilberto Bertin (1):
  net: tun: record RX queue in skb before do_xdp_generic()

 drivers/net/tun.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
2.20.1