Packet forwarding in XDP

Packet forwarding in XDP

[Anmol Panda]

Dear all,
     We are trying to write a program that receives a packet from a 
client and is supposed to return the packet to the source. We tried to 
swap the source and dest and send it using the XDP_TX action. But it 
doesn't work.

    We have compiled a new kernel and bound the NIC to xdp successfully. 
After binding, the system stops responding to ping and other packet 
generating scripts (like sendip). We are using kernel  4.13.12 (Ubuntu 
17.10) from the main linux kernel tree and binding with Intel I217-LM 
NIC. This NIC is using e1000e driver. Can xdp / eBPF be used for this 
NIC? Moreover, do we need to patch the driver for additional support?

     Our aim is to benchmark various forwarding applications (L2 , L3, 
etc.) run using xdp. Please point to some code samples to achieve this. 
We will be truly grateful.



Thanks,
Sincerely,

Re: Packet forwarding in XDP

[Jesper Dangaard Brouer]

On Sun, 12 Nov 2017 14:27:25 +0530
Anmol Panda <anmol07.visitor@cse.iitd.ac.in> wrote:

> Dear all,
>      We are trying to write a program that receives a packet from a 
> client and is supposed to return the packet to the source. We tried to 
> swap the source and dest and send it using the XDP_TX action. But it 
> doesn't work.
> 
>     We have compiled a new kernel and bound the NIC to xdp successfully. 
> After binding, the system stops responding to ping and other packet 
> generating scripts (like sendip). We are using kernel  4.13.12 (Ubuntu 
> 17.10) from the main linux kernel tree and binding with Intel I217-LM 
> NIC. This NIC is using e1000e driver. Can xdp / eBPF be used for this 
> NIC? Moreover, do we need to patch the driver for additional support?

Not all drivers have "native' XDP support. See slide 11 of [1]

[1] http://people.netfilter.org/hawk/presentations/driving-IT2017/driving-IT-2017_XDP_eBPF_technology_Jesper_Brouer.pdf

There is something called "Generic" XDP (v4.12) for ease of
development. It allow you to attach an XDP program any net_device. I
call it --skb-mode in my sample programs. See [2]

[2] https://prototype-kernel.readthedocs.io/en/latest/blogposts/xdp25_eval_generic_xdp_tx.html

>      Our aim is to benchmark various forwarding applications (L2 , L3, 
> etc.) run using xdp. Please point to some code samples to achieve this. 
> We will be truly grateful.

Kernel have directory: samples/bpf/

I maintain some more example programs here:
 https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf


Do notice that e1000e is only 1Gbit/s (which is very slow).  The normal
Linux kernel can handle 1Gbit/s without any issues... I hope you are
planning to use something faster, else it does not make sense to use XDP.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Packet forwarding in XDP

[Anmol Panda]

On 12.11.2017 17:30, Jesper Dangaard Brouer wrote:
> On Sun, 12 Nov 2017 14:27:25 +0530
> Anmol Panda <anmol07.visitor@cse.iitd.ac.in> wrote:
> 
>> Dear all,
>>      We are trying to write a program that receives a packet from a
>> client and is supposed to return the packet to the source. We tried to
>> swap the source and dest and send it using the XDP_TX action. But it
>> doesn't work.
>> 
>>     We have compiled a new kernel and bound the NIC to xdp 
>> successfully.
>> After binding, the system stops responding to ping and other packet
>> generating scripts (like sendip). We are using kernel  4.13.12 (Ubuntu
>> 17.10) from the main linux kernel tree and binding with Intel I217-LM
>> NIC. This NIC is using e1000e driver. Can xdp / eBPF be used for this
>> NIC? Moreover, do we need to patch the driver for additional support?
> 
> Not all drivers have "native' XDP support. See slide 11 of [1]
> 
> [1]
> http://people.netfilter.org/hawk/presentations/driving-IT2017/driving-IT-2017_XDP_eBPF_technology_Jesper_Brouer.pdf
> 
> There is something called "Generic" XDP (v4.12) for ease of
> development. It allow you to attach an XDP program any net_device. I
> call it --skb-mode in my sample programs. See [2]
> 
> [2]
> https://prototype-kernel.readthedocs.io/en/latest/blogposts/xdp25_eval_generic_xdp_tx.html
> 
>>      Our aim is to benchmark various forwarding applications (L2 , L3,
>> etc.) run using xdp. Please point to some code samples to achieve 
>> this.
>> We will be truly grateful.
> 
> Kernel have directory: samples/bpf/
> 
> I maintain some more example programs here:
>  
> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
> 
> 
> Do notice that e1000e is only 1Gbit/s (which is very slow).  The normal
> Linux kernel can handle 1Gbit/s without any issues... I hope you are
> planning to use something faster, else it does not make sense to use 
> XDP.

Re: Packet forwarding in XDP

[Anmol Panda]

On 12.11.2017 17:30, Jesper Dangaard Brouer wrote:
> On Sun, 12 Nov 2017 14:27:25 +0530
> Anmol Panda <anmol07.visitor@cse.iitd.ac.in> wrote:
> 
>> Dear all,
>>      We are trying to write a program that receives a packet from a
>> client and is supposed to return the packet to the source. We tried to
>> swap the source and dest and send it using the XDP_TX action. But it
>> doesn't work.
>> 
>>     We have compiled a new kernel and bound the NIC to xdp 
>> successfully.
>> After binding, the system stops responding to ping and other packet
>> generating scripts (like sendip). We are using kernel  4.13.12 (Ubuntu
>> 17.10) from the main linux kernel tree and binding with Intel I217-LM
>> NIC. This NIC is using e1000e driver. Can xdp / eBPF be used for this
>> NIC? Moreover, do we need to patch the driver for additional support?
> 
> Not all drivers have "native' XDP support. See slide 11 of [1]
> 
> [1]
> http://people.netfilter.org/hawk/presentations/driving-IT2017/driving-IT-2017_XDP_eBPF_technology_Jesper_Brouer.pdf
> 
> There is something called "Generic" XDP (v4.12) for ease of
> development. It allow you to attach an XDP program any net_device. I
> call it --skb-mode in my sample programs. See [2]
> 
> [2]
> https://prototype-kernel.readthedocs.io/en/latest/blogposts/xdp25_eval_generic_xdp_tx.html
> 
>>      Our aim is to benchmark various forwarding applications (L2 , L3,
>> etc.) run using xdp. Please point to some code samples to achieve 
>> this.
>> We will be truly grateful.
> 
> Kernel have directory: samples/bpf/
> 
> I maintain some more example programs here:
>  
> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
> 
> 
> Do notice that e1000e is only 1Gbit/s (which is very slow).  The normal
> Linux kernel can handle 1Gbit/s without any issues... I hope you are
> planning to use something faster, else it does not make sense to use 
> XDP.

Hello,
    Thank you Jesper for your quick response.
    We are trying to run the programs ~/linux-4.13.12/samples/bpf/ and we 
get the following error (also attached as a snapshot):

'
root@compute5:~/workspace/XDP/kernel/linux-4.13.12/samples/bpf# 
./tc_l2_redirect.sh
l2_to_ipip ingress: Note: 8 bytes struct bpf_elf_map fixup performed due 
to size mismatch!
No bpf syscall, kernel headers too old?
No bpf syscall, kernel headers too old?
Map object 'tun_iface' rejected: Function not implemented (38)!
  - Type:         2
  - Identifier:   0
  - Pinning:      2
  - Size key:     4
  - Size value:   4
  - Max elems:    1
  - Flags:        0

Error loading maps into kernel!
Error fetching ELF ancillary data!
'
     We are now using the Intel x520 dual port NICs (total 40 Gbps b/w). 
Please guide us regarding the error above.

Thanks,
Sincerely,

Re: Packet forwarding in XDP

[David Miller]

From: Anmol Panda <anmol07.visitor@cse.iitd.ac.in>
Date: Tue, 14 Nov 2017 12:03:23 +0530

> No bpf syscall, kernel headers too old?

Please enable CONFIG_BPF_SYSCALL in your kernel config.

Re: Packet forwarding in XDP

[Anmol Panda]

On 14.11.2017 12:07, David Miller wrote:
> From: Anmol Panda <anmol07.visitor@cse.iitd.ac.in>
> Date: Tue, 14 Nov 2017 12:03:23 +0530
> 
>> No bpf syscall, kernel headers too old?
> 
> Please enable CONFIG_BPF_SYSCALL in your kernel config.

Hello David,

    Thank you for your response. We have downloaded the latest stable 
version of the Linux kernel (4.13.12) and we compiled it with these 
flags enabled as per the directions here: 
http://cilium.readthedocs.io/en/stable/bpf/  .

CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_CLS_ACT=y
CONFIG_BPF_JIT=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_TEST_BPF=m

    We also enabled bpf_jit_enable and after the new kernel installation, 
we verified the setup using the ./test_verifier script int eh 
tools/testing/selftests/bpf/ folder and the all 506 tests were passed. 
When we tried the xdp1 program in the sample/tests/ we got the error:

No bpf syscall, kernel headers too old?


    Is there anything we are missing here? Please guide us.

Thank you,
Sincerely,

Re: Packet forwarding in XDP

[Anmol Panda]

On 14.11.2017 12:27, Anmol Panda wrote:
> On 14.11.2017 12:07, David Miller wrote:
>> From: Anmol Panda <anmol07.visitor@cse.iitd.ac.in>
>> Date: Tue, 14 Nov 2017 12:03:23 +0530
>> 
>>> No bpf syscall, kernel headers too old?
>> 
>> Please enable CONFIG_BPF_SYSCALL in your kernel config.
> 
> Hello David,
> 
>    Thank you for your response. We have downloaded the latest stable
> version of the Linux kernel (4.13.12) and we compiled it with these
> flags enabled as per the directions here:
> http://cilium.readthedocs.io/en/stable/bpf/  .
> 
> CONFIG_CGROUP_BPF=y
> CONFIG_BPF=y
> CONFIG_BPF_SYSCALL=y
> CONFIG_NET_SCH_INGRESS=m
> CONFIG_NET_CLS_BPF=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_BPF_JIT=y
> CONFIG_LWTUNNEL_BPF=y
> CONFIG_HAVE_EBPF_JIT=y
> CONFIG_BPF_EVENTS=y
> CONFIG_TEST_BPF=m
> 
>    We also enabled bpf_jit_enable and after the new kernel
> installation, we verified the setup using the ./test_verifier script
> int eh tools/testing/selftests/bpf/ folder and the all 506 tests were
> passed. When we tried the xdp1 program in the sample/tests/ we got the
> error:
> 
> No bpf syscall, kernel headers too old?
> 
> 
>    Is there anything we are missing here? Please guide us.
> 
> Thank you,
> Sincerely,

Sorry, once correction. The error encountered was in the 
tc_redirect_l2.sh script and not the xdp1.

Sincerely,

Re: Packet forwarding in XDP

[Daniel Borkmann]

On 11/14/2017 07:57 AM, Anmol Panda wrote:
> On 14.11.2017 12:07, David Miller wrote:
>> From: Anmol Panda <anmol07.visitor@cse.iitd.ac.in>
>> Date: Tue, 14 Nov 2017 12:03:23 +0530
>>
>>> No bpf syscall, kernel headers too old?
>>
>> Please enable CONFIG_BPF_SYSCALL in your kernel config.
> 
> Hello David,
> 
>    Thank you for your response. We have downloaded the latest stable version of the Linux kernel (4.13.12) and we compiled it with these flags enabled as per the directions here: http://cilium.readthedocs.io/en/stable/bpf/  .
> 
> CONFIG_CGROUP_BPF=y
> CONFIG_BPF=y
> CONFIG_BPF_SYSCALL=y
> CONFIG_NET_SCH_INGRESS=m
> CONFIG_NET_CLS_BPF=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_BPF_JIT=y
> CONFIG_LWTUNNEL_BPF=y
> CONFIG_HAVE_EBPF_JIT=y
> CONFIG_BPF_EVENTS=y
> CONFIG_TEST_BPF=m
> 
>    We also enabled bpf_jit_enable and after the new kernel installation, we verified the setup using the ./test_verifier script int eh tools/testing/selftests/bpf/ folder and the all 506 tests were passed. When we tried the xdp1 program in the sample/tests/ we got the error:
> 
> No bpf syscall, kernel headers too old?
> 
>    Is there anything we are missing here? Please guide us.

Yes, please make sure that you have a reasonably recent iproute2, if not,
just compile it from the git master branch from:

https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/

You kernel headers installed on your machine seem too old, that is why
__NR_bpf is not defined and the iproute2 loader throws this error message
on bpf syscall.

Two options:

1) Install recent kernel headers, for example in your kernel source tree,
   run: make headers_install INSTALL_HDR_PATH=/usr/

or

2) Apply the iproute2 patch with fallback definitions for the syscall
   number manually: https://patchwork.ozlabs.org/patch/776015/ and then
   compile iproute2 again (patch is not in upstream iproute2).

Hope that helps,
Daniel

Re: Packet forwarding in XDP

[Anmol Panda]

On 14.11.2017 17:41, Daniel Borkmann wrote:
> On 11/14/2017 07:57 AM, Anmol Panda wrote:
>> On 14.11.2017 12:07, David Miller wrote:
>>> From: Anmol Panda <anmol07.visitor@cse.iitd.ac.in>
>>> Date: Tue, 14 Nov 2017 12:03:23 +0530
>>> 
>>>> No bpf syscall, kernel headers too old?
>>> 
>>> Please enable CONFIG_BPF_SYSCALL in your kernel config.
>> 
>> Hello David,
>> 
>>    Thank you for your response. We have downloaded the latest stable 
>> version of the Linux kernel (4.13.12) and we compiled it with these 
>> flags enabled as per the directions here: 
>> http://cilium.readthedocs.io/en/stable/bpf/  .
>> 
>> CONFIG_CGROUP_BPF=y
>> CONFIG_BPF=y
>> CONFIG_BPF_SYSCALL=y
>> CONFIG_NET_SCH_INGRESS=m
>> CONFIG_NET_CLS_BPF=m
>> CONFIG_NET_CLS_ACT=y
>> CONFIG_BPF_JIT=y
>> CONFIG_LWTUNNEL_BPF=y
>> CONFIG_HAVE_EBPF_JIT=y
>> CONFIG_BPF_EVENTS=y
>> CONFIG_TEST_BPF=m
>> 
>>    We also enabled bpf_jit_enable and after the new kernel 
>> installation, we verified the setup using the ./test_verifier script 
>> int eh tools/testing/selftests/bpf/ folder and the all 506 tests were 
>> passed. When we tried the xdp1 program in the sample/tests/ we got the 
>> error:
>> 
>> No bpf syscall, kernel headers too old?
>> 
>>    Is there anything we are missing here? Please guide us.
> 
> Yes, please make sure that you have a reasonably recent iproute2, if 
> not,
> just compile it from the git master branch from:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/
> 
> You kernel headers installed on your machine seem too old, that is why
> __NR_bpf is not defined and the iproute2 loader throws this error 
> message
> on bpf syscall.
> 
> Two options:
> 
> 1) Install recent kernel headers, for example in your kernel source 
> tree,
>    run: make headers_install INSTALL_HDR_PATH=/usr/
> 
> or
> 
> 2) Apply the iproute2 patch with fallback definitions for the syscall
>    number manually: https://patchwork.ozlabs.org/patch/776015/ and then
>    compile iproute2 again (patch is not in upstream iproute2).
> 
> Hope that helps,
> Daniel

Hello Daniel,
     Thank you for your response. We have completed both the above steps. 
We aren't getting the sys_call error but now it reports the following 
issues with the xdp1 program from /samples/bpf/.

  ./xdp1 -N 6
bpf_load_program() err=1


On checking with 'ip link list', we get the following output indicating 
the program was jited to the device.
'
6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq state 
UP mode DEFAULT group default qlen 1000
     link/ether a0:36:9f:42:9d:54 brd ff:ff:ff:ff:ff:ff
     prog/xdp id 14 tag 5c65c4da7b254d8a jited
'

On running the tc_redirect_l2 script, we found this message (its an 
error/warning we receive on several programs, including some of the 
p4c-xdp examples). The output after running the tc_redirect_l2 script.sh 
is

'
./tc_l2_redirect.sh

l2_to_ipip ingress: Note: 8 bytes struct bpf_elf_map fixup performed due 
to size mismatch!
Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
OK
l2_to_ipip egress: Note: 8 bytes struct bpf_elf_map fixup performed due 
to size mismatch!
Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
OK
l2_to_ip6tnl ingress: Note: 8 bytes struct bpf_elf_map fixup performed 
due to size mismatch!
Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
'

   On running the xdp11.o file from p4c/extensions/p4c-xdp/tests/ we get:
'
  ip link set dev p1p1 xdp obj xdp11.o

Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
RTNETLINK answers: Device or resource busy
'

    We have two error here. The first one being the bpf_load_program() 
and the other being the bpf_elf_map_fixup issue. Lastly, the device is 
reported to be busy although the programs are jited successfully to the 
device (in this case p1p1). Please guide us.

Thank you,
Sincerely,

Re: Packet forwarding in XDP

[Daniel Borkmann]

On 11/15/2017 06:55 AM, Anmol Panda wrote:
> On 14.11.2017 17:41, Daniel Borkmann wrote:
>> On 11/14/2017 07:57 AM, Anmol Panda wrote:
>>> On 14.11.2017 12:07, David Miller wrote:
>>>> From: Anmol Panda <anmol07.visitor@cse.iitd.ac.in>
>>>> Date: Tue, 14 Nov 2017 12:03:23 +0530
>>>>
>>>>> No bpf syscall, kernel headers too old?
>>>>
>>>> Please enable CONFIG_BPF_SYSCALL in your kernel config.
>>>
>>> Hello David,
>>>
>>>    Thank you for your response. We have downloaded the latest stable version of the Linux kernel (4.13.12) and we compiled it with these flags enabled as per the directions here: http://cilium.readthedocs.io/en/stable/bpf/  .
>>>
>>> CONFIG_CGROUP_BPF=y
>>> CONFIG_BPF=y
>>> CONFIG_BPF_SYSCALL=y
>>> CONFIG_NET_SCH_INGRESS=m
>>> CONFIG_NET_CLS_BPF=m
>>> CONFIG_NET_CLS_ACT=y
>>> CONFIG_BPF_JIT=y
>>> CONFIG_LWTUNNEL_BPF=y
>>> CONFIG_HAVE_EBPF_JIT=y
>>> CONFIG_BPF_EVENTS=y
>>> CONFIG_TEST_BPF=m
>>>
>>>    We also enabled bpf_jit_enable and after the new kernel installation, we verified the setup using the ./test_verifier script int eh tools/testing/selftests/bpf/ folder and the all 506 tests were passed. When we tried the xdp1 program in the sample/tests/ we got the error:
>>>
>>> No bpf syscall, kernel headers too old?
>>>
>>>    Is there anything we are missing here? Please guide us.
>>
>> Yes, please make sure that you have a reasonably recent iproute2, if not,
>> just compile it from the git master branch from:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/
>>
>> You kernel headers installed on your machine seem too old, that is why
>> __NR_bpf is not defined and the iproute2 loader throws this error message
>> on bpf syscall.
>>
>> Two options:
>>
>> 1) Install recent kernel headers, for example in your kernel source tree,
>>    run: make headers_install INSTALL_HDR_PATH=/usr/
>>
>> or
>>
>> 2) Apply the iproute2 patch with fallback definitions for the syscall
>>    number manually: https://patchwork.ozlabs.org/patch/776015/ and then
>>    compile iproute2 again (patch is not in upstream iproute2).
>>
>> Hope that helps,
>> Daniel
> 
> Hello Daniel,
>     Thank you for your response. We have completed both the above steps. We aren't getting the sys_call error but now it reports the following issues with the xdp1 program from /samples/bpf/.
> 
>  ./xdp1 -N 6
> bpf_load_program() err=1
> 
> 
> On checking with 'ip link list', we get the following output indicating the program was jited to the device.
> '
> 6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DEFAULT group default qlen 1000
>     link/ether a0:36:9f:42:9d:54 brd ff:ff:ff:ff:ff:ff
>     prog/xdp id 14 tag 5c65c4da7b254d8a jited
> '

Okay, so the XDP program got loaded just fine.

> On running the tc_redirect_l2 script, we found this message (its an error/warning we receive on several programs, including some of the p4c-xdp examples). The output after running the tc_redirect_l2 script.sh is
> 
> '
> ./tc_l2_redirect.sh
> 
> l2_to_ipip ingress: Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> OK
> l2_to_ipip egress: Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> OK
> l2_to_ip6tnl ingress: Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> '

That note is harmless and not an issue at all - it just means that the obj
file was compiled with an older struct bpf_elf_map and loader fixed it up;
maybe I should just remove it to be potentially less confusing.

>   On running the xdp11.o file from p4c/extensions/p4c-xdp/tests/ we get:
> '
>  ip link set dev p1p1 xdp obj xdp11.o
> 
> Note: 8 bytes struct bpf_elf_map fixup performed due to size mismatch!
> RTNETLINK answers: Device or resource busy
> '

See `man ip-link`. It's replaced as:

  ip -force link set dev p1p1 xdp obj xdp11.o

>    We have two error here. The first one being the bpf_load_program() and the other being the bpf_elf_map_fixup issue. Lastly, the device is reported to be busy although the programs are jited successfully to the device (in this case p1p1). Please guide us.
> 
> Thank you,
> Sincerely,
> 
> 
> 
>