Re: [PATCH] Software receive packet steering

[Tom Herbert <therbert@google.com>]

>
> There are some things I've been brainstorming about.
>
> One thought I keep coming back to is the hack the block layer
> is using right now.  It remembers which CPU a block I/O request
> comes in on, and it makes sure the completion runs on that
> cpu too.
>
> We could remember the cpu that the last socket level operation
> occurred upon, and use that as a target for packets.  This requires a
> bit of work.
>
> First we'd need some kind of pre-demux at netif_receive_skb()
> time to look up the cpu target, and reference this blob from
> the socket somehow, and keep it uptodate at various specific
> locations (read/write/poll, whatever...).
>
> Or we could pre-demux the real socket.  That could be exciting.
>

We are doing the pre-demux, and it works well.  The additional benefit
is that the hash result or the the sk itself could be cached in the
skb for the upper layer protocol.

One caveat though is that if the device provides a hash, ie. Toeplitz,
we really want to use that in the CPU look-up to avoid the cache miss
on the header.  We considered using the Toeplitz hash as the inet
hash, but it's incredibly expensive to do in software being about 20x
slower than inet_ehashfn is best we could do.  Our (naive) solution is
to maintain a big array of CPU indices where we write the CPU ids from
recvmsg and sendmsg, and then read it using the hash provided on
incoming packets.  This is lockless and allows very fast operations,
but doesn't take collisions into account (probably allows a slim
possibility of thrashing a connection between CPUs).  The other option
we considered was maintaining a secondary cnx lookup table based on
Toeplitz hash, but that seemed to be rather involved.

> But then we come back to the cpu number changing issue.  There is a
> cool way to handle this, because it seems that we can just keep
> queueing to the previous cpu and it can check the socket cpu cookie.
> If that changes, the old target can push the rest of it's queue to
> that cpu and then update the cpu target blob.
>
> Anyways, just some ideas.
>
Thanks for your thoughts.