Re: [RFC net-next 0/6] psp: use virt cookie as Rx steering hint

[Cosmin Ratiu <cratiu@nvidia.com>]

On Mon, 2026-08-24 at 17:05 +0200, Cosmin Ratiu wrote:
> On Sun, 2026-08-23 at 13:48 -0400, Willem de Bruijn wrote:
> > Jakub Kicinski wrote:
> > > Hi!
> > > 
> > > This PoC series uses a field of the PSP header intended for
> > > tunnels
> > > to auto-steer Rx traffic. Various attempts have been made at
> > > trying
> > > to get Rx traffic to land close to the core where the application
> > > runs.
> > > By default RSS picks the Rx queue based on the flow hash.
> > > I'm not going to cover all previous solutions in detail but
> > > broadly
> > > - we have RFS in SW which looks on which CPU Tx happens and
> > > backlogs
> > > Rx packets there, it is quite efficient. aRFS is built on top
> > > of RFS but tries to program flows into the NIC. Some NICs have
> > > a "cache" and try to automatically remember the flow to queue
> > > association.
> > > 
> > > All those solutions are entirely local to the receiver.
> > > Ideally we would want the solution to look something like
> > > TCP timestamp option - we send an opaque cookie to the peer,
> > > and the peer echoes it back to us. Our NIC can steer based
> > > on that echoed cookie.
> > 
> > Another option is to reverse RSS entropy. This requires knowledge
> > of
> > the RSS secret.
> > 
> > Especially with PSP, the outer UDP source port is defined as flow
> > hash, so can be used for this.
> > 
> > Have the receiver compute a 4-tuple hash such that it knows the RSS
> > block will select the intended queue. The only free variable here
> > in
> > general is the source port. Then communicate this preferred source
> > port to the sender.
> 
> I was thinking of something along these lines as an alternative.
> When initializing PSP for a connection, look at what the device uses
> with .get_rxfh(), and find a value sport (either by linear algebra or
> linear probing) such that the 4-tuple hash results in the desired
> queue. This might require some non-trivial code changes to
> reimplement
> the hashes in SW though, but it's local to this sport selection only.
> 
> psp_dev_encapsulate() and psp_write_headers() both conveniently
> already
> have a sport argument, which is currently ignored, so plumbing this
> is
> easy.
> 
> This would result in zero RX changes for driver implementors.
> 
> An easier alternative to avoid reverse engineering RSS hash
> implementations would be to vary the desired sport across multiple
> packets and until packets start coming to the desired queue.
> Presumably
> this could converge to the desired queue in O(num_queues) TX packets
> +
> acks. Maybe could be done with TCP keep alives at connection setup?
> 

Another issue we have with a potential mlx5 implementation is that HW
GRO cannot work with this feature as proposed, but there are options.

For HW GRO, mlx5 NICs need to decapsulate packets before sending them
to the HW object which does GRO. The SPI and PSP version are passed as
CQE fields. There's no more CQE space to pass req_qid alongside those.

Option 1 would be to only set req_qid on _changes_ (edge transitions)
and keep it 0 otherwise. Also make it mean queue = req_qid - 1 to allow
the use of queue 0. Ideally, only the first packet in each direction
would have req_qid != 0. These packets cannot be decapped and cannot go
through HW GRO, but all subsequent ones with req_qid == 0 may. dst_qid
is of course != 0 in all packets and is honored.

But then you need to make sure those packets aren't lost, and maybe
retransmit req_qid until it gets received. I thought about it a bit,
maybe using TCP acks as an indication? Or just trying best-effort a
couple of times...

Option 2 would be to just disallow HW GRO with req_qid.

And option 3: sport selection to get the desired RSS result makes this
interaction non-existent.

Cosmin.