GRO after RPS?

[David Miller <davem@davemloft.net>]

Herbert, after thinking about some ideas we've been discussing and
some suggestions from folks like Tom Herbert, I'm thinking of changing
it such that we do GRO after RPS sends the packet to a remove cpu.

The idea being that, this way, if we have a device provided ->rxhash
we can elide touching the packet headers entirely.

Initially I wanted to defer the eth_type_trans() by adding a state bit
to sk_buff, and making ->ndo_type_trans() a new netdev op.  The state
bit exists so that we can have a transition period and avoid doing
the type_trans multiple times if the driver still does it early.

So, in this way, when RPS doesn't even need to touch the packet
headers, due to a device provided skb->rxhash, we can defer the
type_trans all the way to the remote cpu.

The only thing getting in the way of this is GRO, since it wants to
parse the packet headers too for flow matching.

At first I thought this was a bad idea to defer GRO to the remote cpu,
since if we do batch things up it means we have less packets to queue
up to the remote cpu.

But upon further consideration it doesn't matter, because GRO is going
to fuddle with the packets and link them up into a list anyways.

So if the list handling is a wash, then it's a real win to defer GRO
and thus potentially all packet header touching to the remote cpu.

Also, we can add the quick ->rxhash check to the GRO flow matcher
like we discussed several times in the past.  And guess what?  Now
that RPS runs first, we'll always have a valid skb->rhash available
for this purpose since RPS will compute one in software for us :-)

So Herbert, any objections before I start hacking on this?