RFS configuration questions

RFS configuration questions

[Shawn Bohrer]

I've been playing around with RPS/RFS on my multiqueue 10g Chelsio NIC
and I've got some questions about configuring RFS.

I've enabled RPS with:

for x in $(seq 0 7); do
    echo FFFFFFFF,FFFFFFFF > /sys/class/net/vlan816/queues/rx-${x}/rps_cpus
done

This appears to work when I watch 'mpstat -P ALL 1' as I can see the
softirq load is now getting distributed across all of the CPUs instead
of just the four (the card is a two port card and assigns four queues
per port) original hw receive queues which I have bound to CPUs
0-3.

To enable RFS I've run:

echo 16384 > /proc/sys/net/core/rps_sock_flow_entries

Is there any explanation of what this sysctl actually does?  Is this
the max number of sockets/flows that the kernel can steer?  Is this a
system wide max, a per interface max, or a per receive queue max?

Next I ran:

for x in $(seq 0 7); do
    echo 16384 > /sys/class/net/vlan816/queues/rx-${x}/rps_flow_cnt
done

Is this correct?  Is these the max number of sockets/flows that can be
steered per receive queue?  Does the sum of these values need to add
up to rps_sock_flow_entries (I also tried 2048)? Is this all that is
needed to enable RFS?

With these settings I can watch 'mpstat -P ALL 1' and it doesn't
appear RFS has changed the softirq load.  To get a better idea if it
was working I used taskset to bind my receiving processes to a set of
cores, yet mpstat still shows the softirq load getting distributed
across all cores, not just the ones where my receiving processes are
bound.  Is there a better way to determine if RFS is actually working?
Have I configured RFS incorrectly?

Thanks,
Shawn

Re: RFS configuration questions

[Eric Dumazet]

Le jeudi 02 décembre 2010 à 15:16 -0600, Shawn Bohrer a écrit :
> I've been playing around with RPS/RFS on my multiqueue 10g Chelsio NIC
> and I've got some questions about configuring RFS.
> 
> I've enabled RPS with:
> 
> for x in $(seq 0 7); do
>     echo FFFFFFFF,FFFFFFFF > /sys/class/net/vlan816/queues/rx-${x}/rps_cpus
> done
> 
> This appears to work when I watch 'mpstat -P ALL 1' as I can see the
> softirq load is now getting distributed across all of the CPUs instead
> of just the four (the card is a two port card and assigns four queues
> per port) original hw receive queues which I have bound to CPUs
> 0-3.
> 
> To enable RFS I've run:
> 
> echo 16384 > /proc/sys/net/core/rps_sock_flow_entries
> 
> Is there any explanation of what this sysctl actually does?  Is this
> the max number of sockets/flows that the kernel can steer?  Is this a
> system wide max, a per interface max, or a per receive queue max?
> 

Yes, some doc is missing...

Its a system wide and shared limit.

> Next I ran:
> 
> for x in $(seq 0 7); do
>     echo 16384 > /sys/class/net/vlan816/queues/rx-${x}/rps_flow_cnt
> done
> 
> Is this correct?  Is these the max number of sockets/flows that can be
> steered per receive queue?  Does the sum of these values need to add
> up to rps_sock_flow_entries (I also tried 2048)? Is this all that is
> needed to enable RFS?
> 

Yes thats all.

> With these settings I can watch 'mpstat -P ALL 1' and it doesn't
> appear RFS has changed the softirq load.  To get a better idea if it
> was working I used taskset to bind my receiving processes to a set of
> cores, yet mpstat still shows the softirq load getting distributed
> across all cores, not just the ones where my receiving processes are
> bound.  Is there a better way to determine if RFS is actually working?
> Have I configured RFS incorrectly?

It seems fine to me, but what kind of workload do you have, and what
version of kernel do you run ?

Re: RFS configuration questions

[Shawn Bohrer]

On Thu, Dec 02, 2010 at 10:40:41PM +0100, Eric Dumazet wrote:
> Le jeudi 02 décembre 2010 à 15:16 -0600, Shawn Bohrer a écrit :
> > I've been playing around with RPS/RFS on my multiqueue 10g Chelsio NIC
> > and I've got some questions about configuring RFS.
> > 
> > I've enabled RPS with:
> > 
> > for x in $(seq 0 7); do
> >     echo FFFFFFFF,FFFFFFFF > /sys/class/net/vlan816/queues/rx-${x}/rps_cpus
> > done
> > 
> > This appears to work when I watch 'mpstat -P ALL 1' as I can see the
> > softirq load is now getting distributed across all of the CPUs instead
> > of just the four (the card is a two port card and assigns four queues
> > per port) original hw receive queues which I have bound to CPUs
> > 0-3.
> > 
> > To enable RFS I've run:
> > 
> > echo 16384 > /proc/sys/net/core/rps_sock_flow_entries
> > 
> > Is there any explanation of what this sysctl actually does?  Is this
> > the max number of sockets/flows that the kernel can steer?  Is this a
> > system wide max, a per interface max, or a per receive queue max?
> > 
> 
> Yes, some doc is missing...
> 
> Its a system wide and shared limit.

So the sum of /sys/class/net/*/queues/rx-*/rps_flow_cnt should be less
than or equal to rps_sock_flow_entries?

> > Next I ran:
> > 
> > for x in $(seq 0 7); do
> >     echo 16384 > /sys/class/net/vlan816/queues/rx-${x}/rps_flow_cnt
> > done
> > 
> > Is this correct?  Is these the max number of sockets/flows that can be
> > steered per receive queue?  Does the sum of these values need to add
> > up to rps_sock_flow_entries (I also tried 2048)? Is this all that is
> > needed to enable RFS?
> > 
> 
> Yes thats all.

Same as above... I should be using 2048 if I have 8 queues and have
set rps_sock_flow_entries to 16384?  Out of curiosity what happens
when you open more sockets than you have rps_flow_cnt?

> > With these settings I can watch 'mpstat -P ALL 1' and it doesn't
> > appear RFS has changed the softirq load.  To get a better idea if it
> > was working I used taskset to bind my receiving processes to a set of
> > cores, yet mpstat still shows the softirq load getting distributed
> > across all cores, not just the ones where my receiving processes are
> > bound.  Is there a better way to determine if RFS is actually working?
> > Have I configured RFS incorrectly?
> 
> It seems fine to me, but what kind of workload do you have, and what
> version of kernel do you run ?

I just did some more testing on 2.6.36.1.  Using netperf UDP_STREAM
and TCP_STREAM I was able to see that the softirq load would run on
the CPU where netperf was bound so it appears that RFS is working.

However if I run one of my applications which is a single process
listening to ~30 multicast addresses the softirq load does not run on
the CPU where the application is bound.  Does RFS not support
receiving multicast?

--
Shawn

Re: RFS configuration questions

[Ben Hutchings]

On Fri, 2010-12-03 at 10:00 -0600, Shawn Bohrer wrote:
> On Thu, Dec 02, 2010 at 10:40:41PM +0100, Eric Dumazet wrote:
> > Le jeudi 02 décembre 2010 à 15:16 -0600, Shawn Bohrer a écrit :
> > > I've been playing around with RPS/RFS on my multiqueue 10g Chelsio NIC
> > > and I've got some questions about configuring RFS.
> > > 
> > > I've enabled RPS with:
> > > 
> > > for x in $(seq 0 7); do
> > >     echo FFFFFFFF,FFFFFFFF > /sys/class/net/vlan816/queues/rx-${x}/rps_cpus
> > > done
> > > 
> > > This appears to work when I watch 'mpstat -P ALL 1' as I can see the
> > > softirq load is now getting distributed across all of the CPUs instead
> > > of just the four (the card is a two port card and assigns four queues
> > > per port) original hw receive queues which I have bound to CPUs
> > > 0-3.
> > > 
> > > To enable RFS I've run:
> > > 
> > > echo 16384 > /proc/sys/net/core/rps_sock_flow_entries
> > > 
> > > Is there any explanation of what this sysctl actually does?  Is this
> > > the max number of sockets/flows that the kernel can steer?  Is this a
> > > system wide max, a per interface max, or a per receive queue max?
> > > 
> > 
> > Yes, some doc is missing...
> > 
> > Its a system wide and shared limit.
> 
> So the sum of /sys/class/net/*/queues/rx-*/rps_flow_cnt should be less
> than or equal to rps_sock_flow_entries?

No, that is not required.

> > > Next I ran:
> > > 
> > > for x in $(seq 0 7); do
> > >     echo 16384 > /sys/class/net/vlan816/queues/rx-${x}/rps_flow_cnt
> > > done
> > > 
> > > Is this correct?  Is these the max number of sockets/flows that can be
> > > steered per receive queue?  Does the sum of these values need to add
> > > up to rps_sock_flow_entries (I also tried 2048)? Is this all that is
> > > needed to enable RFS?
> > > 
> > 
> > Yes thats all.
> 
> Same as above... I should be using 2048 if I have 8 queues and have
> set rps_sock_flow_entries to 16384?  Out of curiosity what happens
> when you open more sockets than you have rps_flow_cnt?
[...]

There will be a hash collision - multiple flows will hash to the same
value and must therefore be handled on the same processor.  (In fact
this is likely to happen long before you fill the table completely.)

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: RFS configuration questions

[Eric Dumazet]

Le vendredi 03 décembre 2010 à 10:00 -0600, Shawn Bohrer a écrit :
> On Thu, Dec 02, 2010 at 10:40:41PM +0100, Eric Dumazet wrote:
> > Le jeudi 02 décembre 2010 à 15:16 -0600, Shawn Bohrer a écrit :
> > > I've been playing around with RPS/RFS on my multiqueue 10g Chelsio NIC
> > > and I've got some questions about configuring RFS.
> > > 
> > > I've enabled RPS with:
> > > 
> > > for x in $(seq 0 7); do
> > >     echo FFFFFFFF,FFFFFFFF > /sys/class/net/vlan816/queues/rx-${x}/rps_cpus
> > > done
> > > 
> > > This appears to work when I watch 'mpstat -P ALL 1' as I can see the
> > > softirq load is now getting distributed across all of the CPUs instead
> > > of just the four (the card is a two port card and assigns four queues
> > > per port) original hw receive queues which I have bound to CPUs
> > > 0-3.
> > > 
> > > To enable RFS I've run:
> > > 
> > > echo 16384 > /proc/sys/net/core/rps_sock_flow_entries
> > > 
> > > Is there any explanation of what this sysctl actually does?  Is this
> > > the max number of sockets/flows that the kernel can steer?  Is this a
> > > system wide max, a per interface max, or a per receive queue max?
> > > 
> > 
> > Yes, some doc is missing...
> > 
> > Its a system wide and shared limit.
> 
> So the sum of /sys/class/net/*/queues/rx-*/rps_flow_cnt should be less
> than or equal to rps_sock_flow_entries?
> 

I always used same count, but you probably can use lower values
for /sys/class/net/*/queues/rx-*/rps_flow_cnt if you dont have much
memory.

> > > Next I ran:
> > > 
> > > for x in $(seq 0 7); do
> > >     echo 16384 > /sys/class/net/vlan816/queues/rx-${x}/rps_flow_cnt
> > > done
> > > 
> > > Is this correct?  Is these the max number of sockets/flows that can be
> > > steered per receive queue?  Does the sum of these values need to add
> > > up to rps_sock_flow_entries (I also tried 2048)? Is this all that is
> > > needed to enable RFS?
> > > 
> > 
> > Yes thats all.
> 
> Same as above... I should be using 2048 if I have 8 queues and have
> set rps_sock_flow_entries to 16384?  Out of curiosity what happens
> when you open more sockets than you have rps_flow_cnt?

Its a hash table, so collisions can happen. Nothing bad happens.

> 
> > > With these settings I can watch 'mpstat -P ALL 1' and it doesn't
> > > appear RFS has changed the softirq load.  To get a better idea if it
> > > was working I used taskset to bind my receiving processes to a set of
> > > cores, yet mpstat still shows the softirq load getting distributed
> > > across all cores, not just the ones where my receiving processes are
> > > bound.  Is there a better way to determine if RFS is actually working?
> > > Have I configured RFS incorrectly?
> > 
> > It seems fine to me, but what kind of workload do you have, and what
> > version of kernel do you run ?
> 
> I just did some more testing on 2.6.36.1.  Using netperf UDP_STREAM
> and TCP_STREAM I was able to see that the softirq load would run on
> the CPU where netperf was bound so it appears that RFS is working.

Be careful about softirq times, they most of the time are wrong.
You can do "cat /proc/net/softnet_stat" and check last column to see if
packets are really distributed to "other cpus"

> 
> However if I run one of my applications which is a single process
> listening to ~30 multicast addresses the softirq load does not run on
> the CPU where the application is bound.  Does RFS not support
> receiving multicast?

No because :

net/ipv4/udp.c

static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
{
	int rc;

	if (inet_sk(sk)->inet_daddr)
		sock_rps_save_rxhash(sk, skb->rxhash);


Rationale is : RFS is implemented for _connected_ sockets only (TCP or
connected UDP)

(You can check commit changelog : 

http://git2.kernel.org/?p=linux/kernel/git/davem/net-next-2.6.git;a=commit;h=fec5e652e58fa6017b2c9e06466cb2a6538de5b4


This is because rxhash value is different for each src address.
If we allowed your process to call sock_rps_save_rxhash(sk,
skb->rxhash); with many different rxhash, it would blow the hash table.

You might try to remove the  "if (inet_sk(sk)->inet_daddr)" test for
your benches, or add a new logic (socket flag maybe ?) to really trigger
RFS on your UDP sockets.