[RFC] staging: octeon: upstream suitability of CN50xx forwarding acceleration

[RFC] staging: octeon: upstream suitability of CN50xx forwarding acceleration

[Markus Krause]

Hello all,

I have worked on and privately tested packet forwarding acceleration for 
the Cavium Octeon CN5020 SoC
used in the Ubiquiti UniFi USG-3P and EdgeRouter Lite. The changes
are currently submitted for OpenWRT review, where I was asked to probe 
the prospects
of this being upstreamed into the Kernel.
I was sceptical as this is essentially a discontinued SoC - but there 
are many devices
with it out there and I felt this work might be useful to the community 
considering
Unifi is dropping their controller support for them and otherwise they 
become e-waste.

Main motivation is that without this hw offloading, achieving actual 
Gigabit line rate is impossible on these devices.

Before preparing an upstream patch series, I would appreciate
guidance on whether the approach is suitable for the Linux Kernel and which
tree and review process should be used. (I have - ZERO - Kernel contribution
experience and hence I am a bit intimidated by this^^)

The routed path integrates with Netfilter hardware flowtable
callbacks. It caches established TCP/UDP flows and performs the
required header rewrites in the Octeon driver's receive path,
reusing the receive buffer and existing hardware PKO transmit path.

The CN5020 has no autonomous NAT engine: lookup and rewriting still
run on the CPU. The implementation supports IPv4/IPv6, NAT, PPPoE
and a single VLAN tag, with unsupported packets falling back to
the normal stack. This should cover the most critical use-cases,
especially also PPPoE with encapsulated VLAN which is something used
by many ISPs (e.g. famously Telekom with VLAN7 for DSL).

On the USG, single-stream plain-NAT throughput improved from
approximately 560–600 Mbit/s with software flow offload to
930–940 Mbit/s. PPPoE over VLAN 7 also reaches approximately
930 Mbit/s. The implementation and benchmark details are here:

https://github.com/openwrt/openwrt/pull/25645

There is also a companion bridge forwarding shortcut using
switchdev FDB notifications, which could be discussed here or separately:

https://github.com/openwrt/openwrt/pull/25823

This essentially takes care of same-subnet switching, e.g. on the USG 
that would
be the case if the third port is used as a second LAN port. It also 
supports VLANs.
(basically everything that does not terminate on the USG like a VPN 
connection and has
no fw table entry)

My main questions are:

- Is this driver-local forwarding approach acceptable in principle,
   including its use of the hardware-flowtable interface?
- Would you require a different integration or work toward moving
   the driver out of staging first?
- Which tree should an RFC implementation target, and who else
   should be included in the review?

I can provide the code as an inline RFC patch series if requested.
Benchmarks are available at the linked Github PR.

The bottom line is that this is what allows devices to operate at 
Gigabit line rate. Without
making use of the SoCs hardware-accelerated buffers and transmit-paths
it's essentially impossible to saturate the Link and maximum packet count is
also very limited.

Thank you,
Markus Krause