Re: GRO: can't force packet up stack immediately?

[John Ousterhout <ouster@cs.stanford.edu>]

Homa uses GRO to collect batches of packets for protocol processing,
but there are times when it wants to push a batch of packet up through
the stack immediately (it doesn't want any more packets to be
processed at NAPI level before pushing the batch up). However, I can't
see a way to achieve this goal. I can return a packet pointer as the
result of homa_gro_receive (and this used to be sufficient to push the
packet up the stack). What happens now is:
* dev_gro_receive calls napi_gro_complete (same as before)
* napi_gro_complete calls gro_normal_one, whereas it used to call
netif_receive_skb_internal
* gro_normal_one just adds the packet to napi->rx_list.

Then NAPI-level packet processing continues, until eventually
napi_complete_done is called; it invokes gro_normal_list, which calls
netif_receive_skb_list_internal.

Because of this, packets can be delayed several microseconds before
they are pushed up the stack. Homa is trying to squeeze out latency,
so the extra delay is undesirable.

-John-

On Thu, Dec 3, 2020 at 11:35 AM Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
>
>
> On 12/3/20 8:03 PM, John Ousterhout wrote:
> > I recently upgraded my kernel module implementing the Homa transport
> > protocol from 4.15.18 to 5.4.80, and a GRO feature available in the
> > older version seems to have gone away in the newer version. In
> > particular, it used to be possible for a protocol's xxx_gro_receive
> > function to force a packet up the stack immediately by returning that
> > skb as the result of xxx_gro_receive. However, in the newer kernel
> > version, these packets simply get queued on napi->rx_list; the queue
> > doesn't get flushed up-stack until napi_complete_done is called or
> > gro_normal_batch packets accumulate. For Homa, this extra level of
> > queuing gets in the way.
>
>
> Could you describe what the issue is ?
>
> >
> > Is there any way for a xxx_gro_receive function to force a packet (in
> > particular, one of those in the list passed as first argument to
> > xxx_gro_receive) up the protocol stack immediately? I suppose I could
> > set gro_normal_batch to 1, but that might interfere with other
> > protocols that really want the batching.
> >
> > -John-
> >