Re: RFS configuration questions

[Shawn Bohrer <sbohrer@rgmadvisors.com>]

On Thu, Dec 02, 2010 at 10:40:41PM +0100, Eric Dumazet wrote:
> Le jeudi 02 décembre 2010 à 15:16 -0600, Shawn Bohrer a écrit :
> > I've been playing around with RPS/RFS on my multiqueue 10g Chelsio NIC
> > and I've got some questions about configuring RFS.
> > 
> > I've enabled RPS with:
> > 
> > for x in $(seq 0 7); do
> >     echo FFFFFFFF,FFFFFFFF > /sys/class/net/vlan816/queues/rx-${x}/rps_cpus
> > done
> > 
> > This appears to work when I watch 'mpstat -P ALL 1' as I can see the
> > softirq load is now getting distributed across all of the CPUs instead
> > of just the four (the card is a two port card and assigns four queues
> > per port) original hw receive queues which I have bound to CPUs
> > 0-3.
> > 
> > To enable RFS I've run:
> > 
> > echo 16384 > /proc/sys/net/core/rps_sock_flow_entries
> > 
> > Is there any explanation of what this sysctl actually does?  Is this
> > the max number of sockets/flows that the kernel can steer?  Is this a
> > system wide max, a per interface max, or a per receive queue max?
> > 
> 
> Yes, some doc is missing...
> 
> Its a system wide and shared limit.

So the sum of /sys/class/net/*/queues/rx-*/rps_flow_cnt should be less
than or equal to rps_sock_flow_entries?

> > Next I ran:
> > 
> > for x in $(seq 0 7); do
> >     echo 16384 > /sys/class/net/vlan816/queues/rx-${x}/rps_flow_cnt
> > done
> > 
> > Is this correct?  Is these the max number of sockets/flows that can be
> > steered per receive queue?  Does the sum of these values need to add
> > up to rps_sock_flow_entries (I also tried 2048)? Is this all that is
> > needed to enable RFS?
> > 
> 
> Yes thats all.

Same as above... I should be using 2048 if I have 8 queues and have
set rps_sock_flow_entries to 16384?  Out of curiosity what happens
when you open more sockets than you have rps_flow_cnt?

> > With these settings I can watch 'mpstat -P ALL 1' and it doesn't
> > appear RFS has changed the softirq load.  To get a better idea if it
> > was working I used taskset to bind my receiving processes to a set of
> > cores, yet mpstat still shows the softirq load getting distributed
> > across all cores, not just the ones where my receiving processes are
> > bound.  Is there a better way to determine if RFS is actually working?
> > Have I configured RFS incorrectly?
> 
> It seems fine to me, but what kind of workload do you have, and what
> version of kernel do you run ?

I just did some more testing on 2.6.36.1.  Using netperf UDP_STREAM
and TCP_STREAM I was able to see that the softirq load would run on
the CPU where netperf was bound so it appears that RFS is working.

However if I run one of my applications which is a single process
listening to ~30 multicast addresses the softirq load does not run on
the CPU where the application is bound.  Does RFS not support
receiving multicast?

--
Shawn