Using AF_XDP To Modify Outgoing Packets

Using AF_XDP To Modify Outgoing Packets

[Christian Deacon]

Hey everyone,


I am fairly new to XDP and AF_XDP programming. Therefore, I apologize if 
this question sounds silly.


To my understanding, an XDP program cannot process outgoing packets 
since it doesn't support the TX path. However, I read that AF_XDP 
sockets support both receiving and transmitting packets inside the user 
space while achieving zero-copy. I am trying to create a C program that 
modifies outgoing packets on an interface and trying to do this the 
fastest way possible. Unfortunately, I cannot use IPTables for this.


I wanted to know if there is any way to use AF_XDP sockets to modify 
outgoing packets on an interface. I wasn't sure if the AF_XDP sockets 
only support receiving traffic from the XDP program when using the 
redirect function or not.


If this isn't possible, are there any plans to add TX path support into 
XDP in the future? I'm also not sure what else I can do to achieve fast 
packet processing for this. I looked into using standard AF_PACKET 
sockets. However, since that makes a copy of the packet from the kernel, 
I'd assume that's pretty slow and I'd have to find a way to block the 
original packets (probably doing something with IPTables). I might also 
look into DPDK, but I want to see if using AF_XDP sockets for outgoing 
packet processing is possible first because it looks like it's going to 
take a while to learn DPDK.


Any help is highly appreciated and thank you for your time!

Re: Using AF_XDP To Modify Outgoing Packets

[David Ahern]

On 3/30/20 10:11 AM, Christian Deacon wrote:
> If this isn't possible, are there any plans to add TX path support into
> XDP in the future?

I am working on it. Latest set is here:
   https://github.com/dsahern/linux/commits/xdp/egress-rfc5-06

does not include support for AF_XDP, but hopefully that can be added
later by someone with more knowledge of socket needs.

Re: Using AF_XDP To Modify Outgoing Packets

[Jakub Kicinski]

On Mon, 30 Mar 2020 11:11:49 -0500 Christian Deacon wrote:
> Hey everyone,
> 
> 
> I am fairly new to XDP and AF_XDP programming. Therefore, I apologize if 
> this question sounds silly.
> 
> 
> To my understanding, an XDP program cannot process outgoing packets 
> since it doesn't support the TX path. However, I read that AF_XDP 
> sockets support both receiving and transmitting packets inside the user 
> space while achieving zero-copy. I am trying to create a C program that 
> modifies outgoing packets on an interface and trying to do this the 
> fastest way possible. Unfortunately, I cannot use IPTables for this.
> 
> 
> I wanted to know if there is any way to use AF_XDP sockets to modify 
> outgoing packets on an interface. I wasn't sure if the AF_XDP sockets 
> only support receiving traffic from the XDP program when using the 
> redirect function or not.
> 
> 
> If this isn't possible, are there any plans to add TX path support into 
> XDP in the future? I'm also not sure what else I can do to achieve fast 
> packet processing for this. I looked into using standard AF_PACKET 
> sockets. However, since that makes a copy of the packet from the kernel, 
> I'd assume that's pretty slow and I'd have to find a way to block the 
> original packets (probably doing something with IPTables). I might also 
> look into DPDK, but I want to see if using AF_XDP sockets for outgoing 
> packet processing is possible first because it looks like it's going to 
> take a while to learn DPDK.
> 
> 
> Any help is highly appreciated and thank you for your time!

Could you give us some more info on the use case? Where do the packets
originate? The main advantage of XDP is that it processes packets
before the metadata needed by the networking stack is allocated (skb).
This is only possible for RX or when packets are redirected from
another interface with XDP. All packets which pass through the stack
anyway can be efficiently modified using cls_bpf.

Re: Using AF_XDP To Modify Outgoing Packets

[Christian Deacon]

Hey David and Jakub,

Thank you for your replies!

David, it's good to know that egress support is being added to XDP and I 
appreciate all the work you and others are putting into this! Do you 
know if there is any ETA on when this will be officially 
available/supported? With that said, will this be faster than most 
solutions available now for processing/modifying outbound packets such 
as using standard AF_PACKET sockets, IPTables, or DPDK?

Jakub, thank you for that information! As for my project, I have a 
program forwarding traffic to a server via IPIP packets. The destination 
server has multiple network namespaces along with the IPIP tunnel 
endpoints and the application sitting inside each namespace. As of right 
now, the destination machine replies back through the IPIP tunnel (to 
the forwarding server) and the forwarding server has to send the replies 
back to the client. My goal is to make it so the application sends 
traffic back to the client directly by spoofing the source address as 
the forwarding server's IP address. This would result in less load on 
the forwarding server along with less latency in my case. Currently, the 
IPIP tunnel endpoints inside the namespaces are set as the default 
devices and all IPIP packets go out the main interface on the 
destination machine.

Initially, I tried creating a veth pair and put the peer inside the 
namespace. I then created a bridge on the main namespace and bridged the 
veth on the main namespace. I assigned the bridge an IP and had an SNAT 
rule in the IPTables POSTROUTING chain to source all traffic out as the 
forwarding server IP. I set the veth pair inside the network namespace 
as the default device on the network namespace and set the next hop to 
the bridge IP. The networking part of this worked fine, traffic sent out 
from the application (through the default route in the network 
namespace) was reaching the clients directly and the clients were 
replying back to the forwarding server. However, this still didn't work 
and I believe the cause is due to the application not supporting two 
separate interfaces (one for receiving and one for sending). 
Unfortunately, the application is closed-source and I doubt support for 
using two separate interfaces will be added.

With the above said, I've been trying to look into creating a program 
that would receive all outgoing packets on the main interface. It would 
check the outer IP header's protocol to ensure it's IPPROTO_IPIP. If 
this is the case, it would then check if the outer IP header's source 
address is the same as the main interface's IP address. If this matches, 
it would save the outer IP header's destination address and remove the 
outer IP header. It would then replace the inner IP header's source 
address with the saved address (outer IP header's destination address) 
which should be the IP of the forwarding server. Afterwards, it would 
recalculate the IP and transport header's checksums and continue sending 
the packet. I believe in theory this should work.

I am trying to find the best way to achieve the above. I don't believe 
IPTables supports changing the packet's contents to the same extent as 
the above.

I made an XDP program yesterday that would do this, but later found out 
XDP doesn't support egress at the moment. I still plan to use the code 
for when TX path/egress support is added. I'd like to come up with 
another solution in the meantime to achieve the above, though.

I hope this helps clear up the situation and thank you again!


On 3/30/2020 1:10 PM, Jakub Kicinski wrote:
> On Mon, 30 Mar 2020 11:11:49 -0500 Christian Deacon wrote:
>> Hey everyone,
>>
>>
>> I am fairly new to XDP and AF_XDP programming. Therefore, I apologize if
>> this question sounds silly.
>>
>>
>> To my understanding, an XDP program cannot process outgoing packets
>> since it doesn't support the TX path. However, I read that AF_XDP
>> sockets support both receiving and transmitting packets inside the user
>> space while achieving zero-copy. I am trying to create a C program that
>> modifies outgoing packets on an interface and trying to do this the
>> fastest way possible. Unfortunately, I cannot use IPTables for this.
>>
>>
>> I wanted to know if there is any way to use AF_XDP sockets to modify
>> outgoing packets on an interface. I wasn't sure if the AF_XDP sockets
>> only support receiving traffic from the XDP program when using the
>> redirect function or not.
>>
>>
>> If this isn't possible, are there any plans to add TX path support into
>> XDP in the future? I'm also not sure what else I can do to achieve fast
>> packet processing for this. I looked into using standard AF_PACKET
>> sockets. However, since that makes a copy of the packet from the kernel,
>> I'd assume that's pretty slow and I'd have to find a way to block the
>> original packets (probably doing something with IPTables). I might also
>> look into DPDK, but I want to see if using AF_XDP sockets for outgoing
>> packet processing is possible first because it looks like it's going to
>> take a while to learn DPDK.
>>
>>
>> Any help is highly appreciated and thank you for your time!
> Could you give us some more info on the use case? Where do the packets
> originate? The main advantage of XDP is that it processes packets
> before the metadata needed by the networking stack is allocated (skb).
> This is only possible for RX or when packets are redirected from
> another interface with XDP. All packets which pass through the stack
> anyway can be efficiently modified using cls_bpf.
>

Re: Using AF_XDP To Modify Outgoing Packets

[Toke Høiland-Jørgensen]

Christian Deacon <gamemann@gflclan.com> writes:

> Hey David and Jakub,
>
> Thank you for your replies!
>
> David, it's good to know that egress support is being added to XDP and I 
> appreciate all the work you and others are putting into this! Do you 
> know if there is any ETA on when this will be officially 
> available/supported? With that said, will this be faster than most 
> solutions available now for processing/modifying outbound packets such 
> as using standard AF_PACKET sockets, IPTables, or DPDK?
>
> Jakub, thank you for that information! As for my project, I have a 
> program forwarding traffic to a server via IPIP packets. The destination 
> server has multiple network namespaces along with the IPIP tunnel 
> endpoints and the application sitting inside each namespace. As of right 
> now, the destination machine replies back through the IPIP tunnel (to 
> the forwarding server) and the forwarding server has to send the replies 
> back to the client. My goal is to make it so the application sends 
> traffic back to the client directly by spoofing the source address as 
> the forwarding server's IP address. This would result in less load on 
> the forwarding server along with less latency in my case. Currently, the 
> IPIP tunnel endpoints inside the namespaces are set as the default 
> devices and all IPIP packets go out the main interface on the 
> destination machine.
>
> Initially, I tried creating a veth pair and put the peer inside the 
> namespace. I then created a bridge on the main namespace and bridged the 
> veth on the main namespace. I assigned the bridge an IP and had an SNAT 
> rule in the IPTables POSTROUTING chain to source all traffic out as the 
> forwarding server IP. I set the veth pair inside the network namespace 
> as the default device on the network namespace and set the next hop to 
> the bridge IP. The networking part of this worked fine, traffic sent out 
> from the application (through the default route in the network 
> namespace) was reaching the clients directly and the clients were 
> replying back to the forwarding server. However, this still didn't work 
> and I believe the cause is due to the application not supporting two 
> separate interfaces (one for receiving and one for sending). 
> Unfortunately, the application is closed-source and I doubt support for 
> using two separate interfaces will be added.
>
> With the above said, I've been trying to look into creating a program 
> that would receive all outgoing packets on the main interface. It would 
> check the outer IP header's protocol to ensure it's IPPROTO_IPIP. If 
> this is the case, it would then check if the outer IP header's source 
> address is the same as the main interface's IP address. If this matches, 
> it would save the outer IP header's destination address and remove the 
> outer IP header. It would then replace the inner IP header's source 
> address with the saved address (outer IP header's destination address) 
> which should be the IP of the forwarding server. Afterwards, it would 
> recalculate the IP and transport header's checksums and continue sending 
> the packet. I believe in theory this should work.
>
> I am trying to find the best way to achieve the above. I don't believe 
> IPTables supports changing the packet's contents to the same extent as 
> the above.
>
> I made an XDP program yesterday that would do this, but later found out 
> XDP doesn't support egress at the moment. I still plan to use the code 
> for when TX path/egress support is added. I'd like to come up with 
> another solution in the meantime to achieve the above, though.

I think you could do this with the TC hook? You can install BPF programs
there that have then same ability to modify the program as XDP does. And
since the packets are coming from an application, you don't gain any
speedup from XDP anyway (since the kernel has already built its packet
data structures).

-Toke

Re: Using AF_XDP To Modify Outgoing Packets

[Christian Deacon]

Hey Toke,


Thank you for your reply and information!


After looking into the TC Hook, it looks like this will work for my 
case. I'll try to get a program made in the next few days or so.


Thank you for the help!


On 3/31/2020 4:29 AM, Toke Høiland-Jørgensen wrote:
> Christian Deacon <gamemann@gflclan.com> writes:
>
>> Hey David and Jakub,
>>
>> Thank you for your replies!
>>
>> David, it's good to know that egress support is being added to XDP and I
>> appreciate all the work you and others are putting into this! Do you
>> know if there is any ETA on when this will be officially
>> available/supported? With that said, will this be faster than most
>> solutions available now for processing/modifying outbound packets such
>> as using standard AF_PACKET sockets, IPTables, or DPDK?
>>
>> Jakub, thank you for that information! As for my project, I have a
>> program forwarding traffic to a server via IPIP packets. The destination
>> server has multiple network namespaces along with the IPIP tunnel
>> endpoints and the application sitting inside each namespace. As of right
>> now, the destination machine replies back through the IPIP tunnel (to
>> the forwarding server) and the forwarding server has to send the replies
>> back to the client. My goal is to make it so the application sends
>> traffic back to the client directly by spoofing the source address as
>> the forwarding server's IP address. This would result in less load on
>> the forwarding server along with less latency in my case. Currently, the
>> IPIP tunnel endpoints inside the namespaces are set as the default
>> devices and all IPIP packets go out the main interface on the
>> destination machine.
>>
>> Initially, I tried creating a veth pair and put the peer inside the
>> namespace. I then created a bridge on the main namespace and bridged the
>> veth on the main namespace. I assigned the bridge an IP and had an SNAT
>> rule in the IPTables POSTROUTING chain to source all traffic out as the
>> forwarding server IP. I set the veth pair inside the network namespace
>> as the default device on the network namespace and set the next hop to
>> the bridge IP. The networking part of this worked fine, traffic sent out
>> from the application (through the default route in the network
>> namespace) was reaching the clients directly and the clients were
>> replying back to the forwarding server. However, this still didn't work
>> and I believe the cause is due to the application not supporting two
>> separate interfaces (one for receiving and one for sending).
>> Unfortunately, the application is closed-source and I doubt support for
>> using two separate interfaces will be added.
>>
>> With the above said, I've been trying to look into creating a program
>> that would receive all outgoing packets on the main interface. It would
>> check the outer IP header's protocol to ensure it's IPPROTO_IPIP. If
>> this is the case, it would then check if the outer IP header's source
>> address is the same as the main interface's IP address. If this matches,
>> it would save the outer IP header's destination address and remove the
>> outer IP header. It would then replace the inner IP header's source
>> address with the saved address (outer IP header's destination address)
>> which should be the IP of the forwarding server. Afterwards, it would
>> recalculate the IP and transport header's checksums and continue sending
>> the packet. I believe in theory this should work.
>>
>> I am trying to find the best way to achieve the above. I don't believe
>> IPTables supports changing the packet's contents to the same extent as
>> the above.
>>
>> I made an XDP program yesterday that would do this, but later found out
>> XDP doesn't support egress at the moment. I still plan to use the code
>> for when TX path/egress support is added. I'd like to come up with
>> another solution in the meantime to achieve the above, though.
> I think you could do this with the TC hook? You can install BPF programs
> there that have then same ability to modify the program as XDP does. And
> since the packets are coming from an application, you don't gain any
> speedup from XDP anyway (since the kernel has already built its packet
> data structures).
>
> -Toke
>

Re: Using AF_XDP To Modify Outgoing Packets

[Toke Høiland-Jørgensen]

Christian Deacon <gamemann@gflclan.com> writes:

> Hey Toke,
>
>
> Thank you for your reply and information!
>
>
> After looking into the TC Hook, it looks like this will work for my 
> case. I'll try to get a program made in the next few days or so.
>
> Thank you for the help!

Awesome! You're welcome :)

-Toke

Re: Using AF_XDP To Modify Outgoing Packets

[Jesper Dangaard Brouer]

On Tue, 31 Mar 2020 18:22:53 -0500
Christian Deacon <gamemann@gflclan.com> wrote:

> After looking into the TC Hook, it looks like this will work for my 
> case. I'll try to get a program made in the next few days or so.

If you want code examples of TC egress hook with BPF look at[1]:
 [1] https://github.com/xdp-project/xdp-cpumap-tc/

Notice that map pinning with libbpf have gotten easier (thanks to
Toke). This example does work (runs in production), it shares and reuse
maps between XDP and TC-bpf.  The example uses the old way to define
maps, and I recommend switching to the new BTF-based syntax instead
(requires LLVM 10), which defines ".maps" SEC, see example[2], and
notice "pinning" option, which obsoletes a lot of the map code in[1].

Attaching TC-bpf progs are different that XDP.  I've hidden the detail
in C-code function tc_egress_attach_bpf() (see[3]), which actually just
calls the "tc" command.  I was hoping that this would be replaced with a
libbpf call, like we have for XDP, but that have not happened.


[2] https://github.com/xdp-project/xdp-tools/blob/master/xdp-filter/xdpfilt_prog.h#L124-L131
[3] https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/common_user.c#L386
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

google keywords: tc qdisc clsact cls_bpf egress filter

Re: Using AF_XDP To Modify Outgoing Packets

[Christian Deacon]

Hey Jesper,


Thank you for the information and examples! They helped a lot!


I've started developing the TC egress program today and I have it nearly 
finished. For some reason, when I modify the source address of the inner 
IP header after removing the outer IP header, the packet is never 
received from the destination host after returning 'TC_ACT_OK'. I did a 
tcpdump and can see the packets with the changed source IP along with 
all the checksums being calculated properly. They also have the correct 
MAC addresses in the Ethernet header. I plan to continue looking into it 
tomorrow and to my understanding, this mailing list is for XDP-specific 
threads. Therefore, I'll probably be creating a thread on the BPF 
mailing list if I can't figure things out (I didn't see any TC-specific 
mailing list).


I just wanted to thank everyone who has replied for the information and 
help! I really appreciate it.


On 4/1/2020 4:55 AM, Jesper Dangaard Brouer wrote:
> On Tue, 31 Mar 2020 18:22:53 -0500
> Christian Deacon <gamemann@gflclan.com> wrote:
>
>> After looking into the TC Hook, it looks like this will work for my
>> case. I'll try to get a program made in the next few days or so.
> If you want code examples of TC egress hook with BPF look at[1]:
>   [1] https://github.com/xdp-project/xdp-cpumap-tc/
>
> Notice that map pinning with libbpf have gotten easier (thanks to
> Toke). This example does work (runs in production), it shares and reuse
> maps between XDP and TC-bpf.  The example uses the old way to define
> maps, and I recommend switching to the new BTF-based syntax instead
> (requires LLVM 10), which defines ".maps" SEC, see example[2], and
> notice "pinning" option, which obsoletes a lot of the map code in[1].
>
> Attaching TC-bpf progs are different that XDP.  I've hidden the detail
> in C-code function tc_egress_attach_bpf() (see[3]), which actually just
> calls the "tc" command.  I was hoping that this would be replaced with a
> libbpf call, like we have for XDP, but that have not happened.
>
>
> [2] https://github.com/xdp-project/xdp-tools/blob/master/xdp-filter/xdpfilt_prog.h#L124-L131
> [3] https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/common_user.c#L386