PFCP support in kernel

PFCP support in kernel

[Drewek, Wojciech]

Hi Harald,

First of all I want to thank you for your revision of our GTP changes, we've learned a lot from your
comments.

So, as you may know our changes were focused around implementing offload of GTP traffic.
We wanted to introduce a consistent solution so we followed the approach used for geneve/vxlan.
In general this approach looks like that:
- create tunnel device (geneve/vxlan/GTP)
- use that device in tc filter command
- based on the type of the device used in tc filter, our driver knows what traffic should be offloaded
Going to the point, now we want to do the same with PFCP. 
The question is: does it even make sense to create PFCP device and parse PFCP messages in kernel space?
My understanding is that PFCP is some kind of equivalent of GTP-C and since GTP-C was purposely not
implemented in the kernel I feel like PFCP wouldn't fit there either.
Maybe there is something that PFCP module could implement and it would be useful.

Lastly, if you are wrong person to ask such question then I'm sorry.
Maybe you know someone else who could help us?

Regards,
Wojtek

Re: PFCP support in kernel

[Harald Welte]

Hi Wojciech,

On Wed, Mar 09, 2022 at 12:27:01PM +0000, Drewek, Wojciech wrote:
> First of all I want to thank you for your revision of our GTP changes,
> we've learned a lot from your comments.

Happy to help!

> So, as you may know our changes were focused around implementing offload of GTP traffic.

Of course, that was what the kernel GTP driver always was about.
Unfortunately it didn't receive a lot of love from the telecom industry,
and hence it is stuck in "2G/3G land", i.e. at a time before the EPS
with its dedicated bearers.  So you cannot use it for IMS/VoLTE, for
example, as it can only match on the source IP address and doesn't have
the capability of using packet classification to put packets in
different tunnels based on [inner IP] port numbers, etc.

> We wanted to introduce a consistent solution so we followed the approach used for geneve/vxlan.
> In general this approach looks like that:
> - create tunnel device (geneve/vxlan/GTP)
> - use that device in tc filter command
> - based on the type of the device used in tc filter, our driver knows what traffic should be offloaded

I'm sorry, I have very limited insight into geneve/vxlan.  It may
be of interest to you that within Osmocom we are currently implementing
a UPF that uses nftables as the backend.  The UPF runs in userspace,
handles a minimal subset of PFCP (no qos/shaping, for example), and then
installs rules into nftables to perform packet matching and
manipulation.  Contrary to the old kernel GTP driver, this approach is
more flexible as it can also cover the TEID mapping case which you find
at SGSN/S-GW or in roaming hubs.  We currently are just about to
complete a prof-of-concept of that.

> Going to the point, now we want to do the same with PFCP. 
> The question is: does it even make sense to create PFCP device and
> parse PFCP messages in kernel space?

I don't think so.  IMHO, PFCP is a rather complex prootocol and it
should be handled in a userspace program, and that userspace program can
then install whatever kernel configuration - whether you want to use
nftables, or tc, or ebpf, or whatever other old or new subsystem in the
kernel network stack.

> My understanding is that PFCP is some kind of equivalent of GTP-C and since GTP-C was purposely not
> implemented in the kernel I feel like PFCP wouldn't fit there either.

I'm sorry, but PFCP is not a replacement of GTP-C.  It serves a rather
different purpose, i.e. to act as protocol between control and user
plane.  GTP-C is control plane, GTP-U is user plane.  PFCP is used in
between the control and user plane entities to configure the user plane.
It's purpose is primarily to be able to mix and match control plane
implementations with user plane implementations. - and to be able to
reuse the same user plane implementation from multiple different network
elements,  like SGW, PGW, HNBGW, roaming hubs, ...

On an abstract / architectural level, PFCP can be compared a bit to
NETLINK:  A protocol between the control plane (linux userspace, routing
daemons, etc.) and the data plane (kernel network stack).

Not sure if there is anything new in it for you, but a while ago in the
osmocom developer call covered CUPS, see the following video recording:
https://media.ccc.de/v/osmodevcall-20211125-laforge-cups-pfcp

> Lastly, if you are wrong person to ask such question then I'm sorry.
> Maybe you know someone else who could help us?

I'm a bit overloaded, but happy to help as far as time permits.

Regards,
	Harald

-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

RE: PFCP support in kernel

[Drewek, Wojciech]

Hi Harald,

Thx for your reply!

> -----Original Message-----
> From: Harald Welte <laforge@gnumonks.org>
> Sent: środa, 9 marca 2022 19:16
> To: Drewek, Wojciech <wojciech.drewek@intel.com>
> Cc: netdev@vger.kernel.org; michal.swiatkowski@linux.intel.com; Marcin Szycik <marcin.szycik@linux.intel.com>
> Subject: Re: PFCP support in kernel
> 
> Hi Wojciech,
> 
> On Wed, Mar 09, 2022 at 12:27:01PM +0000, Drewek, Wojciech wrote:
> > First of all I want to thank you for your revision of our GTP changes,
> > we've learned a lot from your comments.
> 
> Happy to help!
> 
> > So, as you may know our changes were focused around implementing offload of GTP traffic.
> 
> Of course, that was what the kernel GTP driver always was about.
> Unfortunately it didn't receive a lot of love from the telecom industry,
> and hence it is stuck in "2G/3G land", i.e. at a time before the EPS
> with its dedicated bearers.  So you cannot use it for IMS/VoLTE, for
> example, as it can only match on the source IP address and doesn't have
> the capability of using packet classification to put packets in
> different tunnels based on [inner IP] port numbers, etc.
> 
> > We wanted to introduce a consistent solution so we followed the approach used for geneve/vxlan.
> > In general this approach looks like that:
> > - create tunnel device (geneve/vxlan/GTP)
> > - use that device in tc filter command
> > - based on the type of the device used in tc filter, our driver knows what traffic should be offloaded
> 
> I'm sorry, I have very limited insight into geneve/vxlan.  It may
> be of interest to you that within Osmocom we are currently implementing
> a UPF that uses nftables as the backend.  The UPF runs in userspace,
> handles a minimal subset of PFCP (no qos/shaping, for example), and then
> installs rules into nftables to perform packet matching and
> manipulation.  Contrary to the old kernel GTP driver, this approach is
> more flexible as it can also cover the TEID mapping case which you find
> at SGSN/S-GW or in roaming hubs.  We currently are just about to
> complete a prof-of-concept of that.

That's interesting, I have two questions:
- is it going to be possible to math packets based on SEID?
- any options for offloading this nftables  filters to the hardware?
> 
> > Going to the point, now we want to do the same with PFCP.
> > The question is: does it even make sense to create PFCP device and
> > parse PFCP messages in kernel space?
> 
> I don't think so.  IMHO, PFCP is a rather complex prootocol and it
> should be handled in a userspace program, and that userspace program can
> then install whatever kernel configuration - whether you want to use
> nftables, or tc, or ebpf, or whatever other old or new subsystem in the
> kernel network stack.

Ok, we will then rethink our approach in this matter.
> 
> > My understanding is that PFCP is some kind of equivalent of GTP-C and since GTP-C was purposely not
> > implemented in the kernel I feel like PFCP wouldn't fit there either.
> 
> I'm sorry, but PFCP is not a replacement of GTP-C.  It serves a rather
> different purpose, i.e. to act as protocol between control and user
> plane.  GTP-C is control plane, GTP-U is user plane.  PFCP is used in
> between the control and user plane entities to configure the user plane.
> It's purpose is primarily to be able to mix and match control plane
> implementations with user plane implementations. - and to be able to
> reuse the same user plane implementation from multiple different network
> elements,  like SGW, PGW, HNBGW, roaming hubs, ...

Sorry for my lack of knowledge and thanks for explanation.
> 
> On an abstract / architectural level, PFCP can be compared a bit to
> NETLINK:  A protocol between the control plane (linux userspace, routing
> daemons, etc.) and the data plane (kernel network stack).
> 
> Not sure if there is anything new in it for you, but a while ago in the
> osmocom developer call covered CUPS, see the following video recording:
> https://media.ccc.de/v/osmodevcall-20211125-laforge-cups-pfcp
> 
> > Lastly, if you are wrong person to ask such question then I'm sorry.
> > Maybe you know someone else who could help us?
> 
> I'm a bit overloaded, but happy to help as far as time permits.
> 
> Regards,
> 	Harald
> 
> --
> - Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
> ============================================================================
> "Privacy in residential applications is a desirable marketing option."
>                                                   (ETSI EN 300 175-7 Ch. A6)

Regards,
Wojtek

Re: PFCP support in kernel

[Harald Welte]

Hi Wojciech,

On Thu, Mar 10, 2022 at 03:24:07PM +0000, Drewek, Wojciech wrote:

> > I'm sorry, I have very limited insight into geneve/vxlan.  It may
> > be of interest to you that within Osmocom we are currently implementing
> > a UPF that uses nftables as the backend.  The UPF runs in userspace,
> > handles a minimal subset of PFCP (no qos/shaping, for example), and then
> > installs rules into nftables to perform packet matching and
> > manipulation.  Contrary to the old kernel GTP driver, this approach is
> > more flexible as it can also cover the TEID mapping case which you find
> > at SGSN/S-GW or in roaming hubs.  We currently are just about to
> > complete a prof-of-concept of that.
> 
> That's interesting, I have two questions:
> - is it going to be possible to math packets based on SEID?

I'm sorry, I'm not following you.  The SEID I know (TS 29.244 Section 5.6.2)
has only significance on the PFCP session between control and user plane.

The PFCP peers (e.g. SMF and UPF in a PGW use case) use the SEID to
differentiate between different PFCP sessions.

IMHO this has nothing to do with matching of user plane packets in the
actual UFP?

> - any options for offloading this nftables  filters to the hardware?

You would have to talk to the netfilter project if there are any related
approaches for nftables hardware offload, I am no longer involved in
netfilter development for more than a decade by now.

In the context of the "osmo-upf" proof-of-concept we're working on at
sysmocom, the task is explicitly to avoid any type of hardware
acceleration and to see what kind of performance we can reach with a
current mainline kernel in pure software.

-- 
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

RE: PFCP support in kernel

[Drewek, Wojciech]

Hi Harald,

> -----Original Message-----
> From: Harald Welte <laforge@gnumonks.org>
> Sent: czwartek, 10 marca 2022 18:49
> To: Drewek, Wojciech <wojciech.drewek@intel.com>
> Cc: netdev@vger.kernel.org; michal.swiatkowski@linux.intel.com; Marcin Szycik <marcin.szycik@linux.intel.com>
> Subject: Re: PFCP support in kernel
> 
> Hi Wojciech,
> 
> On Thu, Mar 10, 2022 at 03:24:07PM +0000, Drewek, Wojciech wrote:
> 
> > > I'm sorry, I have very limited insight into geneve/vxlan.  It may
> > > be of interest to you that within Osmocom we are currently implementing
> > > a UPF that uses nftables as the backend.  The UPF runs in userspace,
> > > handles a minimal subset of PFCP (no qos/shaping, for example), and then
> > > installs rules into nftables to perform packet matching and
> > > manipulation.  Contrary to the old kernel GTP driver, this approach is
> > > more flexible as it can also cover the TEID mapping case which you find
> > > at SGSN/S-GW or in roaming hubs.  We currently are just about to
> > > complete a prof-of-concept of that.
> >
> > That's interesting, I have two questions:
> > - is it going to be possible to math packets based on SEID?
> 
> I'm sorry, I'm not following you.  The SEID I know (TS 29.244 Section 5.6.2)
> has only significance on the PFCP session between control and user plane.
> 
> The PFCP peers (e.g. SMF and UPF in a PGW use case) use the SEID to
> differentiate between different PFCP sessions.
> 
> IMHO this has nothing to do with matching of user plane packets in the
> actual UFP?

Ok, I think I got it. Thanks for explanation!
> 
> > - any options for offloading this nftables  filters to the hardware?
> 
> You would have to talk to the netfilter project if there are any related
> approaches for nftables hardware offload, I am no longer involved in
> netfilter development for more than a decade by now.
> 
> In the context of the "osmo-upf" proof-of-concept we're working on at
> sysmocom, the task is explicitly to avoid any type of hardware
> acceleration and to see what kind of performance we can reach with a
> current mainline kernel in pure software.

Thanks for answer, so I think we will try with TC tool for now.
> 
> --
> - Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
> ============================================================================
> "Privacy in residential applications is a desirable marketing option."
>                                                   (ETSI EN 300 175-7 Ch. A6)