[RFC bpf-next PATCH] samples/bpf: xdp1 add XDP hardware offload option

[RFC bpf-next PATCH] samples/bpf: xdp1 add XDP hardware offload option

[Jesper Dangaard Brouer]

Trying to use XDP hardware offloading via XDP_FLAGS_HW_MODE
and setting the ifindex in prog_load_attr.ifindex before
loading the BPF code via bpf_prog_load_xattr().

This unfortunately does not seem to work...
- Am I doing something wrong?

Notice, I also disable the map BPF_MAP_TYPE_PERCPU_ARRAY
to make sure it was not related to the map (not supporting
offloading).

Failed with:
 # ./xdp1 -O $(</sys/class/net/enp130s0np1/ifindex)
 libbpf: load bpf program failed: Invalid argument
 libbpf: failed to load program 'xdp1'
 libbpf: failed to load object './xdp1_kern.o'

Tested on kernel 4.18.0-2.el8.x86_64 with driver nfp
 Ethernet controller: Netronome Systems, Inc. Device 4000
---
 samples/bpf/xdp1_kern.c |    8 +++++---
 samples/bpf/xdp1_user.c |   12 ++++++++++--
 2 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/samples/bpf/xdp1_kern.c b/samples/bpf/xdp1_kern.c
index 219742106bfd..1b577b7dba1e 100644
--- a/samples/bpf/xdp1_kern.c
+++ b/samples/bpf/xdp1_kern.c
@@ -14,12 +14,14 @@
 #include <linux/ipv6.h>
 #include "bpf_helpers.h"
 
+/*
 struct bpf_map_def SEC("maps") rxcnt = {
 	.type = BPF_MAP_TYPE_PERCPU_ARRAY,
 	.key_size = sizeof(u32),
 	.value_size = sizeof(long),
 	.max_entries = 256,
 };
+*/
 
 static int parse_ipv4(void *data, u64 nh_off, void *data_end)
 {
@@ -83,9 +85,9 @@ int xdp_prog1(struct xdp_md *ctx)
 	else
 		ipproto = 0;
 
-	value = bpf_map_lookup_elem(&rxcnt, &ipproto);
-	if (value)
-		*value += 1;
+//	value = bpf_map_lookup_elem(&rxcnt, &ipproto);
+//	if (value)
+//		*value += 1;
 
 	return rc;
 }
diff --git a/samples/bpf/xdp1_user.c b/samples/bpf/xdp1_user.c
index b02c531510ed..a362b5adfcf0 100644
--- a/samples/bpf/xdp1_user.c
+++ b/samples/bpf/xdp1_user.c
@@ -64,7 +64,8 @@ static void usage(const char *prog)
 		"usage: %s [OPTS] IFINDEX\n\n"
 		"OPTS:\n"
 		"    -S    use skb-mode\n"
-		"    -N    enforce native mode\n",
+		"    -N    enforce native mode\n"
+		"    -O    offload mode\n",
 		prog);
 }
 
@@ -74,7 +75,7 @@ int main(int argc, char **argv)
 	struct bpf_prog_load_attr prog_load_attr = {
 		.prog_type	= BPF_PROG_TYPE_XDP,
 	};
-	const char *optstr = "SN";
+	const char *optstr = "SNO";
 	int prog_fd, map_fd, opt;
 	struct bpf_object *obj;
 	struct bpf_map *map;
@@ -88,6 +89,9 @@ int main(int argc, char **argv)
 		case 'N':
 			xdp_flags |= XDP_FLAGS_DRV_MODE;
 			break;
+		case 'O':
+			xdp_flags |= XDP_FLAGS_HW_MODE;
+			break;
 		default:
 			usage(basename(argv[0]));
 			return 1;
@@ -109,6 +113,10 @@ int main(int argc, char **argv)
 	snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
 	prog_load_attr.file = filename;
 
+	/* For HW offload provide ifindex when loading BPF code */
+	if (xdp_flags & XDP_FLAGS_HW_MODE) {
+		prog_load_attr.ifindex = ifindex;
+	}
 	if (bpf_prog_load_xattr(&prog_load_attr, &obj, &prog_fd))
 		return 1;
 

Re: [RFC bpf-next PATCH] samples/bpf: xdp1 add XDP hardware offload option

[Jakub Kicinski]

On Tue, 04 Sep 2018 16:59:19 +0200, Jesper Dangaard Brouer wrote:
> Trying to use XDP hardware offloading via XDP_FLAGS_HW_MODE
> and setting the ifindex in prog_load_attr.ifindex before
> loading the BPF code via bpf_prog_load_xattr().
> 
> This unfortunately does not seem to work...
> - Am I doing something wrong?
> 
> Notice, I also disable the map BPF_MAP_TYPE_PERCPU_ARRAY
> to make sure it was not related to the map (not supporting
> offloading).
> 
> Failed with:
>  # ./xdp1 -O $(</sys/class/net/enp130s0np1/ifindex)
>  libbpf: load bpf program failed: Invalid argument
>  libbpf: failed to load program 'xdp1'
>  libbpf: failed to load object './xdp1_kern.o'
> 
> Tested on kernel 4.18.0-2.el8.x86_64 with driver nfp
>  Ethernet controller: Netronome Systems, Inc. Device 4000

Are you running the BPF capable FW?

https://help.netronome.com/support/solutions/articles/36000050009-agilio-ebpf-2-0-6-extended-berkeley-packet-filter

Re: [RFC bpf-next PATCH] samples/bpf: xdp1 add XDP hardware offload option

[Jesper Dangaard Brouer]

On Tue, 4 Sep 2018 17:09:12 +0200
Jakub Kicinski <jakub.kicinski@netronome.com> wrote:

> On Tue, 04 Sep 2018 16:59:19 +0200, Jesper Dangaard Brouer wrote:
> > Trying to use XDP hardware offloading via XDP_FLAGS_HW_MODE
> > and setting the ifindex in prog_load_attr.ifindex before
> > loading the BPF code via bpf_prog_load_xattr().
> > 
> > This unfortunately does not seem to work...
> > - Am I doing something wrong?
> > 
> > Notice, I also disable the map BPF_MAP_TYPE_PERCPU_ARRAY
> > to make sure it was not related to the map (not supporting
> > offloading).
> > 
> > Failed with:
> >  # ./xdp1 -O $(</sys/class/net/enp130s0np1/ifindex)
> >  libbpf: load bpf program failed: Invalid argument
> >  libbpf: failed to load program 'xdp1'
> >  libbpf: failed to load object './xdp1_kern.o'
> > 
> > Tested on kernel 4.18.0-2.el8.x86_64 with driver nfp
> >  Ethernet controller: Netronome Systems, Inc. Device 4000  
> 
> Are you running the BPF capable FW?
> 
> https://help.netronome.com/support/solutions/articles/36000050009-agilio-ebpf-2-0-6-extended-berkeley-packet-filter

I'm likely not running the correct firmware...

Can you tell me, with the ethtool -i output, if I'm running the
appropriate firmware?

# ethtool -i enp129s0np1
driver: nfp
version: 4.18.0-2.el8.x86_64 SMP mod_unl
firmware-version: 0.0.3.5 0.21 nic-2.0.7 nic
expansion-rom-version: 
bus-info: 0000:81:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

If this is a firmware version case, then we should really improve the
errors we are giving the user, the -EINVAL can be anything.

 "libbpf: load bpf program failed: Invalid argument"

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [oss-drivers] Re: [RFC bpf-next PATCH] samples/bpf: xdp1 add XDP hardware offload option

[Jakub Kicinski]

On Tue, 4 Sep 2018 18:49:33 +0200, Jesper Dangaard Brouer wrote:
> On Tue, 4 Sep 2018 17:09:12 +0200
> Jakub Kicinski <jakub.kicinski@netronome.com> wrote:
> 
> > On Tue, 04 Sep 2018 16:59:19 +0200, Jesper Dangaard Brouer wrote:  
> > > Trying to use XDP hardware offloading via XDP_FLAGS_HW_MODE
> > > and setting the ifindex in prog_load_attr.ifindex before
> > > loading the BPF code via bpf_prog_load_xattr().
> > > 
> > > This unfortunately does not seem to work...
> > > - Am I doing something wrong?
> > > 
> > > Notice, I also disable the map BPF_MAP_TYPE_PERCPU_ARRAY
> > > to make sure it was not related to the map (not supporting
> > > offloading).
> > > 
> > > Failed with:
> > >  # ./xdp1 -O $(</sys/class/net/enp130s0np1/ifindex)
> > >  libbpf: load bpf program failed: Invalid argument
> > >  libbpf: failed to load program 'xdp1'
> > >  libbpf: failed to load object './xdp1_kern.o'
> > > 
> > > Tested on kernel 4.18.0-2.el8.x86_64 with driver nfp
> > >  Ethernet controller: Netronome Systems, Inc. Device 4000    
> > 
> > Are you running the BPF capable FW?
> > 
> > https://help.netronome.com/support/solutions/articles/36000050009-agilio-ebpf-2-0-6-extended-berkeley-packet-filter  
> 
> I'm likely not running the correct firmware...
> 
> Can you tell me, with the ethtool -i output, if I'm running the
> appropriate firmware?
> 
> # ethtool -i enp129s0np1
> driver: nfp
> version: 4.18.0-2.el8.x86_64 SMP mod_unl
> firmware-version: 0.0.3.5 0.21 nic-2.0.7 nic
> expansion-rom-version: 
> bus-info: 0000:81:00.0
> supports-statistics: yes
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: yes
> supports-priv-flags: no

Yup, the BPF firmware says bpf in firmware version.

> If this is a firmware version case, then we should really improve the
> errors we are giving the user, the -EINVAL can be anything.
> 
>  "libbpf: load bpf program failed: Invalid argument"

That is true.

Re: [oss-drivers] Re: [RFC bpf-next PATCH] samples/bpf: xdp1 add XDP hardware offload option

[Jesper Dangaard Brouer]

On Tue, 4 Sep 2018 18:59:57 +0200
Jakub Kicinski <jakub.kicinski@netronome.com> wrote:

> On Tue, 4 Sep 2018 18:49:33 +0200, Jesper Dangaard Brouer wrote:
> > On Tue, 4 Sep 2018 17:09:12 +0200
> > Jakub Kicinski <jakub.kicinski@netronome.com> wrote:
> >   
> > > On Tue, 04 Sep 2018 16:59:19 +0200, Jesper Dangaard Brouer wrote:    
> > > > Trying to use XDP hardware offloading via XDP_FLAGS_HW_MODE
> > > > and setting the ifindex in prog_load_attr.ifindex before
> > > > loading the BPF code via bpf_prog_load_xattr().
> > > > 
> > > > This unfortunately does not seem to work...
> > > > - Am I doing something wrong?
> > > > 
> > > > Notice, I also disable the map BPF_MAP_TYPE_PERCPU_ARRAY
> > > > to make sure it was not related to the map (not supporting
> > > > offloading).
> > > > 
> > > > Failed with:
> > > >  # ./xdp1 -O $(</sys/class/net/enp130s0np1/ifindex)
> > > >  libbpf: load bpf program failed: Invalid argument
> > > >  libbpf: failed to load program 'xdp1'
> > > >  libbpf: failed to load object './xdp1_kern.o'
> > > > 
> > > > Tested on kernel 4.18.0-2.el8.x86_64 with driver nfp
> > > >  Ethernet controller: Netronome Systems, Inc. Device 4000      
> > > 
> > > Are you running the BPF capable FW?
> > > 
> > > https://help.netronome.com/support/solutions/articles/36000050009-agilio-ebpf-2-0-6-extended-berkeley-packet-filter    
> > 
> > I'm likely not running the correct firmware...
> > 
> > Can you tell me, with the ethtool -i output, if I'm running the
> > appropriate firmware?
> > 
> > # ethtool -i enp129s0np1
> > driver: nfp
> > version: 4.18.0-2.el8.x86_64 SMP mod_unl
> > firmware-version: 0.0.3.5 0.21 nic-2.0.7 nic
> > expansion-rom-version: 
> > bus-info: 0000:81:00.0
> > supports-statistics: yes
> > supports-test: no
> > supports-eeprom-access: no
> > supports-register-dump: yes
> > supports-priv-flags: no  
> 
> Yup, the BPF firmware says bpf in firmware version.

Downloaded: agilio-bpf-firmware-2.0.6.121-1.noarch.rpm

RPM install it:
 rpm -hiv  agilio-bpf-firmware-2.0.6.121-1.noarch.rpm

# rpm -ql agilio-bpf-firmware
/opt/netronome/firmware/agilio-bpf
/opt/netronome/firmware/agilio-bpf/nic_AMDA0058-0011_2x40.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0058-0012_2x40.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0078-0011_1x100.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0081-0001_1x40.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0081-0001_4x10.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0096-0001_2x10.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0097-0001_2x40.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0097-0001_4x10_1x40.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0097-0001_8x10.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0099-0001_1x10_1x25.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0099-0001_2x10.nffw
/opt/netronome/firmware/agilio-bpf/nic_AMDA0099-0001_2x25.nffw

Netronome: Basic Firmware User Guide
 https://help.netronome.com/support/solutions/articles/36000049975-basic-firmware-user-guide

A section says after installing the firmware, unload and reload the
driver kernel module will upgrade the firmware.

 ## reload driver to load new firmware
 rmmod nfp; modprobe nfp

Firmware upgrade worked! :-) This is by-far the easiest firmware
upgrade I've experienced.  And I don't have to reboot the machine :-)

 # ethtool -i enp130s0np1  | grep firmware-version
 firmware-version: 0.0.3.5 0.21 bpf-2.0.6.121 ebpf

Notice the bpf/ebpf strings in firmware-version.

 
> > If this is a firmware version case, then we should really improve the
> > errors we are giving the user, the -EINVAL can be anything.
> > 
> >  "libbpf: load bpf program failed: Invalid argument"  
> 
> That is true.

Same goes for improving the error message, when loading a map type that
offload cannot handle.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer