Re: RFS configuration questions

Eric Dumazet
2010-12-02T21:40:46Z

Le jeudi 02 décembre 2010 à 15:16 -0600, Shawn Bohrer a écrit :
> I've been playing around with RPS/RFS on my multiqueue 10g Chelsio NIC
> and I've got some questions about configuring RFS.
>
> I've enabled RPS with:
>
> for x in $(seq 0 7); do
> echo FFFFFFFF,FFFFFFFF > /sys/class/net/vlan816/queues/rx-${x}/rps_cpus
> done
>
> This appears to work when I watch 'mpstat -P ALL 1' as I can see the
> softirq load is now getting distributed across all of the CPUs instead
> of just the four (the card is a two port card and assigns four queues
> per port) original hw receive queues which I have bound to CPUs
> 0-3.
>
> To enable RFS I've run:
>
> echo 16384 > /proc/sys/net/core/rps_sock_flow_entries
>
> Is there any explanation of what this sysctl actually does? Is this
> the max number of sockets/flows that the kernel can steer? Is this a
> system wide max, a per interface max, or a per receive queue max?
>

Yes, some doc is missing...

Its a system wide and shared limit.

> Next I ran:
>
> for x in $(seq 0 7); do
> echo 16384 > /sys/class/net/vlan816/queues/rx-${x}/rps_flow_cnt
> done
>
> Is this correct? Is these the max number of sockets/flows that can be
> steered per receive queue? Does the sum of these values need to add
> up to rps_sock_flow_entries (I also tried 2048)? Is this all that is
> needed to enable RFS?
>

Yes thats all.

> With these settings I can watch 'mpstat -P ALL 1' and it doesn't
> appear RFS has changed the softirq load. To get a better idea if it
> was working I used taskset to bind my receiving processes to a set of
> cores, yet mpstat still shows the softirq load getting distributed
> across all cores, not just the ones where my receiving processes are
> bound. Is there a better way to determine if RFS is actually working?
> Have I configured RFS incorrectly?

It seems fine to me, but what kind of workload do you have, and what
version of kernel do you run ?