Re: [RFC Patch net-next] net_sched: make classifying lockless on ingress

[John Fastabend <john.fastabend@gmail.com>]

On 12/23/2013 04:56 PM, Cong Wang wrote:
> On Sat, Dec 21, 2013 at 3:09 PM, John Fastabend
> <john.fastabend@gmail.com> wrote:
>>
>> I solved this by making them per CPU and synchronizing when I hit
>> an operation that required sync'ing them. Going forward if folks
>> have the time to write SMP aware qdisc's that work with eventually
>> consistent counters that would be great.
>>
>
> Interesting, then you have to copy the same filters and actions
> to all per-cpu-ingress-qdisc, right? Also you need to handle
> CPU online/offline event.
>
> The number of CPU's grows fast today, so the total size
> of such ingress qdisc would be huge if I install lots
> of filters and action.
>

In this case I was specifically talking about statistics so the
bstats and qstats.

As long as the qdisc's do not require global state this works well
enough. However as Jamal keeps pointing out the problem is any qdisc
which requires global state requires locking (I paraphrase but I think
replicate the spirit correctly) and this doesn't work well with many
CPUs. So you either replicate the qdiscs one per queue like we do in
the mq and mqprio case effectively removing any global state or you
develop qdiscs that don't require global state or at least work with
eventually consistent data to avoid the constant syncing of data.

I think though a qdisc per nic queue is really not as bad as you think.
For example we do this on the tx side and it works OK. Note its per
RX queue and not per CPU.

.John



-- 
John Fastabend         Intel Corporation