Measuring/Debugging XDP Performance

Measuring/Debugging XDP Performance

[Christian Deacon]

Hey everyone,

I am new to XDP + AF_XDP (along with C programming in general), but I am 
very interested in it and I've been learning a lot recently. I own an 
Anycast network and our POP servers are running custom software our 
developer created that processes packets using XDP. This software 
basically forwards specific traffic to another machine via an IPIP 
tunnel. One issue I've been noticing is the packets our software is 
processing and forwarding to another machine keep dropping at higher 
traffic loads. I can't tell if this is dropping at the POP level or if 
the machine the software is forwarding this specific traffic to is. I've 
even tried upgrading the POP server from a two-core VPS (2.5 GHz CPUs) 
to a dedicated server (Intel E3-1230v6 @ 3.5 GHz, 4 cores, and 8 
threads). If this is being dropped at the POP level, I'm wondering if 
the software is being limited to one core on this specific POP (other 
POPs are able to use more than one core specifically). However, I have 
no way to confirm that. To my understanding XDP programs should be able 
to use more than one core.

My questions are the following:

1. Is there a way to see how much CPU the XDP program is using or the 
load of the NIC? To my understanding, you cannot tell the XDP program's 
CPU usage based off of something like `top` or `htop` due to that being 
in the user space (XDP happens at the NIC driver level in the kernel IIRC).

2. Are there any known bugs with XDP somehow being limited to one core 
(e.g. not multi-threaded)? The POP is running Ubuntu 18.04 on kernel 
'4.18.0-25'. The AF_XDP part of the software is also using `pthreads` 
and from what I saw, using them correctly. However, not much traffic 
gets forwarded to the AF_XDP program. Therefore, I doubt that's even an 
issue. I wasn't able to find any known bugs when researching this, but 
figured I'd ask this question just in case there's something I missed.

I would like to note that the NIC has 4 RX queues and the software makes 
use of all four. I'm not sure if that would make a difference or not.

If you need any additional information, please let me know.

Any help is highly appreciated!

Thank you for your time.

Re: Measuring/Debugging XDP Performance

[Vincent Li]

On Wed, 22 Jan 2020, Christian Deacon wrote:

> Hey everyone,
> 
> I am new to XDP + AF_XDP (along with C programming in general), but I am very
> interested in it and I've been learning a lot recently. I own an Anycast
> network and our POP servers are running custom software our developer created
> that processes packets using XDP. This software basically forwards specific
> traffic to another machine via an IPIP tunnel. One issue I've been noticing is
> the packets our software is processing and forwarding to another machine keep
> dropping at higher traffic loads. I can't tell if this is dropping at the POP
> level or if the machine the software is forwarding this specific traffic to
> is. I've even tried upgrading the POP server from a two-core VPS (2.5 GHz
> CPUs) to a dedicated server (Intel E3-1230v6 @ 3.5 GHz, 4 cores, and 8
> threads). If this is being dropped at the POP level, I'm wondering if the
> software is being limited to one core on this specific POP (other POPs are
> able to use more than one core specifically). However, I have no way to
> confirm that. To my understanding XDP programs should be able to use more than
> one core.
> 
> My questions are the following:
> 
> 1. Is there a way to see how much CPU the XDP program is using or the load of
> the NIC? To my understanding, you cannot tell the XDP program's CPU usage
> based off of something like `top` or `htop` due to that being in the user
> space (XDP happens at the NIC driver level in the kernel IIRC).

I am newbie in XDP too, maybe Linux 
Perf http://www.brendangregg.com/perf.html tool could help you figuring 
out which part of the code in your XDP app consuming CPU cycles (debug 
symbol needed)

Re: Measuring/Debugging XDP Performance

[Jesper Dangaard Brouer]

On Wed, 22 Jan 2020 14:11:07 -0800 (PST)
Vincent Li <mchun.li@gmail.com> wrote:

> On Wed, 22 Jan 2020, Christian Deacon wrote:
> 
> > Hey everyone,
> > 
> > I am new to XDP + AF_XDP (along with C programming in general), but I am very
> > interested in it and I've been learning a lot recently. I own an Anycast
> > network and our POP servers are running custom software our developer created
> > that processes packets using XDP. This software basically forwards specific
> > traffic to another machine via an IPIP tunnel. One issue I've been noticing is
> > the packets our software is processing and forwarding to another machine keep
> > dropping at higher traffic loads. I can't tell if this is dropping at the POP
> > level or if the machine the software is forwarding this specific traffic to
> > is. I've even tried upgrading the POP server from a two-core VPS (2.5 GHz
> > CPUs) to a dedicated server (Intel E3-1230v6 @ 3.5 GHz, 4 cores, and 8
> > threads). If this is being dropped at the POP level, I'm wondering if the
> > software is being limited to one core on this specific POP (other POPs are
> > able to use more than one core specifically). However, I have no way to
> > confirm that. To my understanding XDP programs should be able to use more than
> > one core.
> > 
> > My questions are the following:
> > 
> > 1. Is there a way to see how much CPU the XDP program is using or the load of
> > the NIC? To my understanding, you cannot tell the XDP program's CPU usage
> > based off of something like `top` or `htop` due to that being in the user
> > space (XDP happens at the NIC driver level in the kernel IIRC).  
> 
> I am newbie in XDP too, maybe Linux 
> Perf http://www.brendangregg.com/perf.html tool could help you figuring 
> out which part of the code in your XDP app consuming CPU cycles (debug 
> symbol needed)


I agree start with the 'perf' command line tool to look at the issue. 
As this is likely a CPU load distribution issue, let me give you are
couple of perf commands to use.

First record system wide (-a) entire system for 10 sec:

  perf record -g -a sleep 10

Look at what happened:

  perf report --no-children

Now you also want to look at this per CPU:

  perf report --no-children --sort cpu,comm,dso,symbol

If you want to send some info about your perf report results via email,
you can use the --stdio parameter to get the plain text output.

Once you have completed this quest, I'll help you with some more
advanced commands... does you distro have 'bpftrace' ?

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Measuring/Debugging XDP Performance

[Jesper Dangaard Brouer]

On Wed, 22 Jan 2020 12:02:23 -0600
Christian Deacon <gamemann@gflclan.com> wrote:

> Hey everyone,
> 
> I am new to XDP + AF_XDP (along with C programming in general), but I am 
> very interested in it and I've been learning a lot recently. I own an 
> Anycast network and our POP servers are running custom software our 
> developer created that processes packets using XDP. This software 
> basically forwards specific traffic to another machine via an IPIP 
> tunnel.

Why are you using AF_XDP to implement a simple IPIP tunnel header?

You can easily implement it in BPF.
Example here:
 https://github.com/torvalds/linux/blob/master/samples/bpf/xdp_tx_iptunnel_kern.c#L110-L112
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Measuring/Debugging XDP Performance

[Christian Deacon]

Hey Vincent and Jesper,

Thank you for the information and I will try these commands tonight when 
there is higher traffic load! The 'perf' command seems interesting.

Jesper, I do believe my kernel supports BPF Trace. I was able to use 
this function in our software while I was debugging it for a separate 
issue that I resolved earlier this week. With that said, the IPIP 
traffic is handled in the XDP program. We only redirect a certain packet 
to the AF_XDP program so it can be cached via a Redis server (needed in 
the user space). However, this doesn't happen that often and other 
packets handled by the XDP program itself are being dropped. Therefore, 
I doubt the AF_XDP program is causing anything.

Thank you!

On 1/23/2020 7:11 AM, Jesper Dangaard Brouer wrote:
> On Wed, 22 Jan 2020 12:02:23 -0600
> Christian Deacon <gamemann@gflclan.com> wrote:
>
>> Hey everyone,
>>
>> I am new to XDP + AF_XDP (along with C programming in general), but I am
>> very interested in it and I've been learning a lot recently. I own an
>> Anycast network and our POP servers are running custom software our
>> developer created that processes packets using XDP. This software
>> basically forwards specific traffic to another machine via an IPIP
>> tunnel.
> Why are you using AF_XDP to implement a simple IPIP tunnel header?
>
> You can easily implement it in BPF.
> Example here:
>   https://github.com/torvalds/linux/blob/master/samples/bpf/xdp_tx_iptunnel_kern.c#L110-L112

Re: Measuring/Debugging XDP Performance

[Jesper Dangaard Brouer]

On Thu, 23 Jan 2020 11:22:31 -0600
Christian Deacon <gamemann@gflclan.com> wrote:

> Thank you for the information and I will try these commands tonight when 
> there is higher traffic load! The 'perf' command seems interesting.

Thanks for the perf output in the private email thread. From the output
I can clearly see the problem (some copy pasted below signature).  You
are using the driver igb which does not support XDP-native, and is thus
using generic-XDP.

This driver only have 192 bytes headroom, where 256 is needed.  Thus,
running XDP-generic will be slower than running the normal network
stack, as every packet will get reallocated and copied.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

     2.88%  swapper          [kernel.kallsyms]         [k] igb_poll
            |          
             --2.82%--igb_poll
                       |          
                        --2.45%--net_rx_action
                                  __softirqentry_text_start
                                  irq_exit



    0.71%  006  swapper          [kernel.kallsyms]         [k] page_frag_free
            |
            ---page_frag_free
               |          
                --0.68%--skb_free_head
                          pskb_expand_head
                          do_xdp_generic
                          netif_receive_skb_internal
                          napi_gro_receive
                          igb_clean_rx_irq
                          igb_poll
                          net_rx_action

Re: Measuring/Debugging XDP Performance

[Christian Deacon]

Hey Jesper,


Thank you for the information and that makes a lot more sense! I didn't 
realize we had a NIC that doesn't support XDP-native. I will see if I 
can get a better NIC card for this dedicated server (I don't believe our 
current NIC card supports any other drivers besides `igb`).


In the meantime, I have stopped announcing my dedicated server to our 
Anycast network and have started announcing the four-core VPS that had 
similar issues before ordering the dedicated server (even though I feel 
this VPS should be able to handle the traffic load). The CPU is 
virtualized and running at 2.5 GHz per core. The VPS's OS is Ubuntu 
18.10 and the kernel is '4.18.0-20'. The NIC driver is "virtio-pci", 
which I believe is supported (please correct me if I'm wrong, I do see 
'virtio_net' in the list of supported XDP drivers which I'd assume is 
the same thing).


I will execute the 'perf' commands tonight when we have higher traffic 
load and reply with the output.


Thank you for your help!


On 1/23/2020 2:38 PM, Jesper Dangaard Brouer wrote:
> On Thu, 23 Jan 2020 11:22:31 -0600
> Christian Deacon <gamemann@gflclan.com> wrote:
>
>> Thank you for the information and I will try these commands tonight when
>> there is higher traffic load! The 'perf' command seems interesting.
> Thanks for the perf output in the private email thread. From the output
> I can clearly see the problem (some copy pasted below signature).  You
> are using the driver igb which does not support XDP-native, and is thus
> using generic-XDP.
>
> This driver only have 192 bytes headroom, where 256 is needed.  Thus,
> running XDP-generic will be slower than running the normal network
> stack, as every packet will get reallocated and copied.
>

Re: Measuring/Debugging XDP Performance

[Christian Deacon]

Hey everyone,


The only NIC our hosting provider could replace our old NIC with is 
using the `e1000e` driver (it's a one gigabit NIC card). I dug through 
some older mailing list threads and read XDP isn't very effective with 
one gigabit NICs (the preferred minimum is 10 gbps IIRC). These were 
threads from 2017 and I'm not sure if this is still the case. However, 
if it is, I'm still hoping I can use XDP, but hopefully have the 
performance equal to the network stack at the very least.


I understand the `e1000e` driver doesn't have XDP support upstream. 
Therefore, I tried implementing the driver patch that adds XDP support here:


https://github.com/adjavon/e1000e_xdp


Unfortunately, this patch was based off of kernel 4.10.2 (I'm using 
4.19). Therefore, I had to manually implement the patch code (this is my 
first time messing with NIC driver code). Sadly, it doesn't seem like 
the patch worked based off of the 'perf' results which are linked below. 
I still see "do_xdp_generic" calls:


```

xxx@xxx/xxxx/xxxx# cat results_2.log | grep "xdp"
                           do_xdp_generic
      0.33%  006  swapper          [kernel.kallsyms]         [k] 
do_xdp_generic                         -      -
      0.09%  006  swapper          [kernel.kallsyms]         [k] 
bpf_xdp_adjust_head                    -      -
      0.09%  005  swapper          [kernel.kallsyms]         [k] 
do_xdp_generic                         -      -
      0.07%  006  swapper          [kernel.kallsyms]         [k] 
generic_xdp_tx                         -      -
      0.07%  004  swapper          [kernel.kallsyms]         [k] 
do_xdp_generic                         -      -
      0.05%  003  swapper          [kernel.kallsyms]         [k] 
do_xdp_generic                         -      -
      0.02%  004  swapper          [kernel.kallsyms]         [k] 
generic_xdp_tx                         -      -
      0.01%  005  swapper          [kernel.kallsyms]         [k] 
generic_xdp_tx                         -      -
```


With that said, it appears my other POP servers that are using the 
"virtio_net" NIC driver are using XDP generic according to the 'perf' 
results as well (they're similar to the above). They're also on kernel 4.18.


My question is, how would you know if you're using XDP-native or not? Is 
it possible you're still using XDP-native, even when there are XDP 
generic calls from the 'perf' command?


Here's a link to the outputs of the 'perf' command discussed above for 
anyone interested:


https://g.gflclan.com/6sHmHPlo3W.zip


Any help is highly appreciated and thank you for your time!


On 1/23/2020 3:38 PM, Christian Deacon wrote:
> Hey Jesper,
>
>
> Thank you for the information and that makes a lot more sense! I 
> didn't realize we had a NIC that doesn't support XDP-native. I will 
> see if I can get a better NIC card for this dedicated server (I don't 
> believe our current NIC card supports any other drivers besides `igb`).
>
>
> In the meantime, I have stopped announcing my dedicated server to our 
> Anycast network and have started announcing the four-core VPS that had 
> similar issues before ordering the dedicated server (even though I 
> feel this VPS should be able to handle the traffic load). The CPU is 
> virtualized and running at 2.5 GHz per core. The VPS's OS is Ubuntu 
> 18.10 and the kernel is '4.18.0-20'. The NIC driver is "virtio-pci", 
> which I believe is supported (please correct me if I'm wrong, I do see 
> 'virtio_net' in the list of supported XDP drivers which I'd assume is 
> the same thing).
>
>
> I will execute the 'perf' commands tonight when we have higher traffic 
> load and reply with the output.
>
>
> Thank you for your help!
>
>
> On 1/23/2020 2:38 PM, Jesper Dangaard Brouer wrote:
>> On Thu, 23 Jan 2020 11:22:31 -0600
>> Christian Deacon <gamemann@gflclan.com> wrote:
>>
>>> Thank you for the information and I will try these commands tonight 
>>> when
>>> there is higher traffic load! The 'perf' command seems interesting.
>> Thanks for the perf output in the private email thread. From the output
>> I can clearly see the problem (some copy pasted below signature).  You
>> are using the driver igb which does not support XDP-native, and is thus
>> using generic-XDP.
>>
>> This driver only have 192 bytes headroom, where 256 is needed. Thus,
>> running XDP-generic will be slower than running the normal network
>> stack, as every packet will get reallocated and copied.
>>

Re: Measuring/Debugging XDP Performance

[Matheus Salgueiro Castanho]

Hi Christian,

On Tue, Jan 28, 2020 at 4:01 PM Christian Deacon <gamemann@gflclan.com> wrote:
>
> Hey everyone,

[...]

>
> My question is, how would you know if you're using XDP-native or not?

I believe the 'ip link' command is able to tell you that. If you have
an XDP program attached to interface eth0, you can do something like:

$ ip link show dev eth0
6: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc
mq state UP mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
prog/xdp id 27 tag f95672269956c10d jited

Note the 'xdp' after the MTU size. IIRC, that indicates XDP Native.
XDP generic would show `xdpgeneric` instead. `xdpoffload` is also a
possible output if your program is offloaded to a SmartNIC.

>
>
>
> Here's a link to the outputs of the 'perf' command discussed above for
> anyone interested:
>
>
> https://g.gflclan.com/6sHmHPlo3W.zip
>
>
> Any help is highly appreciated and thank you for your time!
>

--
Matheus Castanho

Re: Measuring/Debugging XDP Performance

[Jesper Dangaard Brouer]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Tue, 28 Jan 2020 12:42:21 -0600
Christian Deacon <gamemann@gflclan.com> wrote:

> I understand the `e1000e` driver doesn't have XDP support upstream. 
> Therefore, I tried implementing the driver patch that adds XDP support 
> here <https://github.com/adjavon/e1000e_xdp>. Unfortunately, this patch 
> was based off of kernel 4.10.2 (I'm using 4.19). Therefore, I had to 
> manually implement the patch code (this is my first time messing with 
> NIC driver code). Sadly, it doesn't seem like the patch worked based off 
> of the 'perf' results which are attached. I still see "do_xdp_generic" 

I do think it would be more valuable to implement XDP for driver igb.

Given igb is a "slow" 1Gbit/s interface, we could also just fix
XDP-generic to avoid the reallocation of the SKB for every packet...
I've attached two patches which does exactly that. It sounds like you
know howto apply patches and recompile your kernel(?)

I've tested this on my testlab.  The igb hardware seems is limited
to 1.2Mpps.  Using an XDP_DROP test program, I observe the CPU
processing packets use 50% CPU time before, and 36% CPU time after the
patches.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

[-- Attachment #2: 01-04-xdp-headroom --]
[-- Type: application/octet-stream, Size: 1808 bytes --]

xdp: accept that XDP headroom isn't always equal XDP_PACKET_HEADROOM

From: Jesper Dangaard Brouer <brouer@redhat.com>

The Intel based drivers (ixgbe + i40e) have implemented XDP with
headroom 192 bytes and not the recommended 256 bytes defined by
XDP_PACKET_HEADROOM.  For generic-XDP, accept that this headroom
is also a valid size.

Still for generic-XDP if headroom is less, still expand headroom to
XDP_PACKET_HEADROOM as this is the default in most XDP drivers.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
 include/uapi/linux/bpf.h |    1 +
 net/core/dev.c           |    4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index f1d74a2bd234..2d93cb1f9433 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -3291,6 +3291,7 @@ struct bpf_xdp_sock {
 };
 
 #define XDP_PACKET_HEADROOM 256
+#define XDP_PACKET_HEADROOM_MIN 192
 
 /* User return codes for XDP prog type.
  * A valid XDP program must return one of these defined values. All other
diff --git a/net/core/dev.c b/net/core/dev.c
index 38bc35da39f7..c54ed9066724 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4531,11 +4531,11 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 		return XDP_PASS;
 
 	/* XDP packets must be linear and must have sufficient headroom
-	 * of XDP_PACKET_HEADROOM bytes. This is the guarantee that also
+	 * of XDP_PACKET_HEADROOM_MIN bytes. This is the guarantee that also
 	 * native XDP provides, thus we need to do it here as well.
 	 */
 	if (skb_is_nonlinear(skb) ||
-	    skb_headroom(skb) < XDP_PACKET_HEADROOM) {
+	    skb_headroom(skb) < XDP_PACKET_HEADROOM_MIN) {
 		int hroom = XDP_PACKET_HEADROOM - skb_headroom(skb);
 		int troom = skb->tail + skb->data_len - skb->end;
 

[-- Attachment #3: 02-igb-adjust-headroom --]
[-- Type: application/octet-stream, Size: 2669 bytes --]

igb: increase headroom to cooperate with XDP-generic

From: Jesper Dangaard Brouer <brouer@redhat.com>

Increase the packet headroom to 192 bytes.

This works, but due to NIC TimeStamp header (IGB_TS_HDR_LEN = 16),
providing 192 bytes headroom, cause the driver to use order-1 pages
(8192 bytes) to store packets.  This is only a problem if the TS
header is combined with a VLAN header and is a MTU sized packets.

A solution would be to detect if HW timestamp is enabled, but I
couldn't find an easy test.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
 drivers/net/ethernet/intel/igb/igb.h      |    9 +++++++--
 drivers/net/ethernet/intel/igb/igb_main.c |    3 ++-
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h
index 49b5fa9d4783..4c689fe947d1 100644
--- a/drivers/net/ethernet/intel/igb/igb.h
+++ b/drivers/net/ethernet/intel/igb/igb.h
@@ -137,10 +137,15 @@ struct vf_mac_filter {
 #define IGB_RX_HDR_LEN		IGB_RXBUFFER_256
 #define IGB_TS_HDR_LEN		16
 
-#define IGB_SKB_PAD		(NET_SKB_PAD + NET_IP_ALIGN)
+/* Headroom 192 fit MTU size Ethernet frame into 2K with skb_shared_info */
+#define IGB_SKB_PAD		192
+// FIXME: What about NET_IP_ALIGN?
+
 #if (PAGE_SIZE < 8192)
 #define IGB_MAX_FRAME_BUILD_SKB \
 	(SKB_WITH_OVERHEAD(IGB_RXBUFFER_2048) - IGB_SKB_PAD - IGB_TS_HDR_LEN)
+	 // Hmm... this -IGB_TS_HDR_LEN might be a problem
+	// 2048 - 320 - 192 - 16 = 1520
 #else
 #define IGB_MAX_FRAME_BUILD_SKB (IGB_RXBUFFER_2048 - IGB_TS_HDR_LEN)
 #endif
@@ -348,7 +353,7 @@ static inline unsigned int igb_rx_pg_order(struct igb_ring *ring)
 {
 #if (PAGE_SIZE < 8192)
 	if (ring_uses_large_buffer(ring))
-		return 1;
+		return 1; // page order-1 = 8192 bytes
 #endif
 	return 0;
 }
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index b46bff8fe056..8ed3db27d4d2 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -4567,7 +4567,7 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter,
 #if (PAGE_SIZE < 8192)
 	if (adapter->max_frame_size <= IGB_MAX_FRAME_BUILD_SKB)
 		return;
-
+	// Issue (1522 <= 1520) --> set_ring_uses_large_buffer
 	set_ring_uses_large_buffer(rx_ring);
 #endif
 }
@@ -6256,6 +6256,7 @@ static int igb_change_mtu(struct net_device *netdev, int new_mtu)
 {
 	struct igb_adapter *adapter = netdev_priv(netdev);
 	int max_frame = new_mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
+	// 1500 + 14 + 4 + 4 = 1522
 
 	/* adjust max frame to be at least the size of a standard frame */
 	if (max_frame < (ETH_FRAME_LEN + ETH_FCS_LEN))

Re: Measuring/Debugging XDP Performance

[Jesper Dangaard Brouer]

On Tue, 28 Jan 2020 12:42:21 -0600
Christian Deacon <gamemann@gflclan.com> wrote:

> With that said, it appears my other POP servers that are using the 
> "virtio_net" NIC driver are using XDP generic according to the 'perf' 
> results as well (they're similar to the above). They're also on
> kernel 4.18.

First of all 'virtio_net' is tricky to get working with 'native-XDP',
because you need to disable a number of offloads.  This have gotten a
bit easier on newer kernels.

That said, it is important to understand that getting raw "fast" frames
into your "virtio_net" virtual netcard is a two-stage "rocket".   The
owner of the hardware box also need a native-XDP netcard that runs XDP
which XDP-redirect frames into your virtual NIC, else there is not
performance advantage.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Measuring/Debugging XDP Performance

[Christian Deacon]

Hey Jesper and Matheus,

Thank you for the information and assistance! I really appreciate it!

Matheus, I was able to confirm that my POP servers (running as VPSs with 
the "virtio_net" driver) are indeed using XDP generic based off of the 
command you provided. I didn't realize it included "xdpgeneric" on the 
link list. This is definitely an easier way to tell if you're using XDP 
generic compared to looking at the 'perf' results.

Jesper, thank you for those two patches! I've recompiled the 4.19 kernel 
with those implemented and have it running on my POP server now. I've 
started announcing the POP server to the network again and I will let 
you know how it goes along with getting results from the 'perf' command. 
Do you think it would be worth trying to implement XDP support into the 
`igb` driver? I understand if it can't be supported officially, but once 
I gain more experience, etc. I might give it a try if it would increase 
performance.

I figured the physical NIC card used to host the VPSs would need to 
support XDP-native, but I just wanted to make sure. I tried asking my 
hosting provider if the NIC card uses a driver that is on the XDP-native 
supported list. They came back stating some nodes do and some do not, 
sadly. What type of offloads would need to be disabled in order to get 
it working (in the case the node's NIC driver does support XDP-native)?

Thank you for your time.


On 1/29/2020 8:26 AM, Jesper Dangaard Brouer wrote:
> On Tue, 28 Jan 2020 12:42:21 -0600
> Christian Deacon <gamemann@gflclan.com> wrote:
>
>> With that said, it appears my other POP servers that are using the
>> "virtio_net" NIC driver are using XDP generic according to the 'perf'
>> results as well (they're similar to the above). They're also on
>> kernel 4.18.
> First of all 'virtio_net' is tricky to get working with 'native-XDP',
> because you need to disable a number of offloads.  This have gotten a
> bit easier on newer kernels.
>
> That said, it is important to understand that getting raw "fast" frames
> into your "virtio_net" virtual netcard is a two-stage "rocket".   The
> owner of the hardware box also need a native-XDP netcard that runs XDP
> which XDP-redirect frames into your virtual NIC, else there is not
> performance advantage.
>