2.6.35-rc2, CONFIG_RPS is filling the dmesg log

2.6.35-rc2, CONFIG_RPS is filling the dmesg log

[Tim Gardner]

With 2.6.35-rc2 my dmesg log is being flooded with messages like this:

br0 received packet on queue 4, but number of RX queues is 1

This machine is bridged for KVM and has 2 igb network adapters.

The root cause appears to be CONFIG_RPS=y and the fact that none of the 
drivers that call skb_record_rx_queue() perform their net device 
allocation using alloc_netdev_mq(), thereby initializing num_rx_queues 
to a maximum of 1.

Given that this is early RPS days, is the warning in get_rps_cpu() 
really necessary? It would appear that _all_ of the multi-receive queue 
devices that call skb_record_rx_queue() will cause this log noise.

By the way, how do you turn off CONFIG_RPS? The only way I could get it 
disabled was to change the default in net/Kconfig to 'n'.

rtg
-- 
Tim Gardner tim.gardner@canonical.com

Re: 2.6.35-rc2, CONFIG_RPS is filling the dmesg log

[Tim Gardner]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

On 06/08/2010 02:55 PM, Tim Gardner wrote:
> With 2.6.35-rc2 my dmesg log is being flooded with messages like this:
>
> br0 received packet on queue 4, but number of RX queues is 1
>
> This machine is bridged for KVM and has 2 igb network adapters.
>
> The root cause appears to be CONFIG_RPS=y and the fact that none of the
> drivers that call skb_record_rx_queue() perform their net device
> allocation using alloc_netdev_mq(), thereby initializing num_rx_queues
> to a maximum of 1.
>
> Given that this is early RPS days, is the warning in get_rps_cpu()
> really necessary? It would appear that _all_ of the multi-receive queue
> devices that call skb_record_rx_queue() will cause this log noise.
>
> By the way, how do you turn off CONFIG_RPS? The only way I could get it
> disabled was to change the default in net/Kconfig to 'n'.
>
> rtg

This is the route that I'm taking with Ubuntu in the short term. I'll 
have lots of server testers complaining pretty soon if I don't take care 
of this now. It does keep my server logs from filling.

rtg

-- 
Tim Gardner tim.gardner@canonical.com

[-- Attachment #2: 0001-net-Print-num_rx_queues-imbalance-warning-only-when-.patch --]
[-- Type: text/x-patch, Size: 1210 bytes --]

>From 02598ea1409568654a554fae3ac2c22ecc2474d0 Mon Sep 17 00:00:00 2001
From: Tim Gardner <tim.gardner@canonical.com>
Date: Tue, 8 Jun 2010 17:51:27 -0600
Subject: [PATCH] net: Print num_rx_queues imbalance warning only when there are allocated queues

BugLink: http://bugs.launchpad.net/bugs/591416

So far no users of skb_record_rx_queue() use alloc_netdev_mq() for
network device initialization, so don't print a warning about num_rx_queues
imbalances in get_rps_cpu() unless they have actually been allocated.

Signed-off-by: Tim Gardner <tim.gardner@canonical.com>
---
 net/core/dev.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index d03470f..0852608 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2253,7 +2253,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 	if (skb_rx_queue_recorded(skb)) {
 		u16 index = skb_get_rx_queue(skb);
 		if (unlikely(index >= dev->num_rx_queues)) {
-			if (net_ratelimit()) {
+			if (dev->num_rx_queues > 1 && net_ratelimit()) {
 				pr_warning("%s received packet on queue "
 					"%u, but number of RX queues is %u\n",
 					dev->name, index, dev->num_rx_queues);
-- 
1.7.0.4

Re: 2.6.35-rc2, CONFIG_RPS is filling the dmesg log

[Eric Dumazet]

Le mercredi 09 juin 2010 à 07:27 -0600, Tim Gardner a écrit :
> On 06/08/2010 02:55 PM, Tim Gardner wrote:
> > With 2.6.35-rc2 my dmesg log is being flooded with messages like this:
> >
> > br0 received packet on queue 4, but number of RX queues is 1
> >
> > This machine is bridged for KVM and has 2 igb network adapters.
> >
> > The root cause appears to be CONFIG_RPS=y and the fact that none of the
> > drivers that call skb_record_rx_queue() perform their net device
> > allocation using alloc_netdev_mq(), thereby initializing num_rx_queues
> > to a maximum of 1.
> >
> > Given that this is early RPS days, is the warning in get_rps_cpu()
> > really necessary? It would appear that _all_ of the multi-receive queue
> > devices that call skb_record_rx_queue() will cause this log noise.
> >
> > By the way, how do you turn off CONFIG_RPS? The only way I could get it
> > disabled was to change the default in net/Kconfig to 'n'.
> >
> > rtg
> 
> This is the route that I'm taking with Ubuntu in the short term. I'll 
> have lots of server testers complaining pretty soon if I don't take care 
> of this now. It does keep my server logs from filling.
> 
> rtg
> 

Probably fine, but your commit message is not exact :

  So far no users of skb_record_rx_queue() use alloc_netdev_mq() for
  network device initialization, so don't print a warning about num_rx_queues
  imbalances in get_rps_cpu() unless they have actually been allocated.

In fact, drivers that use skb_record_rx_queue() did use alloc_netdev_mq().

Problem is : packets going thru bridge/bonding that are not yet
multiqueue enabled. If R[PF]S enabled for these "virtual devices",
we trigger the get_rps_cpu() warning.

Also, in a bonding setup, we still have a problem
because all tx packets will go thru tx queue 0 (dev_pick_tx() job)

(That might be good to know that for Ubuntu server testers)

Re: 2.6.35-rc2, CONFIG_RPS is filling the dmesg log

[Tim Gardner]

[-- Attachment #1: Type: text/plain, Size: 2011 bytes --]

On 06/09/2010 07:42 AM, Eric Dumazet wrote:
> Le mercredi 09 juin 2010 à 07:27 -0600, Tim Gardner a écrit :
>> On 06/08/2010 02:55 PM, Tim Gardner wrote:
>>> With 2.6.35-rc2 my dmesg log is being flooded with messages like this:
>>>
>>> br0 received packet on queue 4, but number of RX queues is 1
>>>
>>> This machine is bridged for KVM and has 2 igb network adapters.
>>>
>>> The root cause appears to be CONFIG_RPS=y and the fact that none of the
>>> drivers that call skb_record_rx_queue() perform their net device
>>> allocation using alloc_netdev_mq(), thereby initializing num_rx_queues
>>> to a maximum of 1.
>>>
>>> Given that this is early RPS days, is the warning in get_rps_cpu()
>>> really necessary? It would appear that _all_ of the multi-receive queue
>>> devices that call skb_record_rx_queue() will cause this log noise.
>>>
>>> By the way, how do you turn off CONFIG_RPS? The only way I could get it
>>> disabled was to change the default in net/Kconfig to 'n'.
>>>
>>> rtg
>>
>> This is the route that I'm taking with Ubuntu in the short term. I'll
>> have lots of server testers complaining pretty soon if I don't take care
>> of this now. It does keep my server logs from filling.
>>
>> rtg
>>
>
> Probably fine, but your commit message is not exact :
>
>    So far no users of skb_record_rx_queue() use alloc_netdev_mq() for
>    network device initialization, so don't print a warning about num_rx_queues
>    imbalances in get_rps_cpu() unless they have actually been allocated.
>
> In fact, drivers that use skb_record_rx_queue() did use alloc_netdev_mq().
>
> Problem is : packets going thru bridge/bonding that are not yet
> multiqueue enabled. If R[PF]S enabled for these "virtual devices",
> we trigger the get_rps_cpu() warning.
>
> Also, in a bonding setup, we still have a problem
> because all tx packets will go thru tx queue 0 (dev_pick_tx() job)
>
> (That might be good to know that for Ubuntu server testers)
>

How about this?

-- 
Tim Gardner tim.gardner@canonical.com

[-- Attachment #2: 0001-UBUNTU-SAUCE-net-Print-num_rx_queues-imbalance-warni.patch --]
[-- Type: text/x-patch, Size: 1443 bytes --]

>From ad76786a1a0c7b7b3c9bfeb4116fa0e2742f6328 Mon Sep 17 00:00:00 2001
From: Tim Gardner <tim.gardner@canonical.com>
Date: Tue, 8 Jun 2010 17:51:27 -0600
Subject: [PATCH] net: Print num_rx_queues imbalance warning only when there are allocated queues

BugLink: http://bugs.launchpad.net/bugs/591416

There are a number of network drivers (bridge, bonding, etc) that are not yet
receive multi-queue enabled and use alloc_netdev(), so don't print a
num_rx_queues imbalance warning in that case.

Also, only print the warning once for those drivers that _are_ multi-queue
enabled.

Signed-off-by: Tim Gardner <tim.gardner@canonical.com>
---
 net/core/dev.c |    8 +++-----
 1 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index d03470f..14a8568 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2253,11 +2253,9 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 	if (skb_rx_queue_recorded(skb)) {
 		u16 index = skb_get_rx_queue(skb);
 		if (unlikely(index >= dev->num_rx_queues)) {
-			if (net_ratelimit()) {
-				pr_warning("%s received packet on queue "
-					"%u, but number of RX queues is %u\n",
-					dev->name, index, dev->num_rx_queues);
-			}
+			WARN_ONCE(dev->num_rx_queues > 1, "%s received packet "
+				"on queue %u, but number of RX queues is %u\n",
+				dev->name, index, dev->num_rx_queues);
 			goto done;
 		}
 		rxqueue = dev->_rx + index;
-- 
1.7.0.4

Re: 2.6.35-rc2, CONFIG_RPS is filling the dmesg log

[Eric Dumazet]

Le mercredi 09 juin 2010 à 09:22 -0600, Tim Gardner a écrit :
> On 06/09/2010 07:42 AM, Eric Dumazet wrote:
> > Le mercredi 09 juin 2010 à 07:27 -0600, Tim Gardner a écrit :
> >> On 06/08/2010 02:55 PM, Tim Gardner wrote:
> >>> With 2.6.35-rc2 my dmesg log is being flooded with messages like this:
> >>>
> >>> br0 received packet on queue 4, but number of RX queues is 1
> >>>
> >>> This machine is bridged for KVM and has 2 igb network adapters.
> >>>
> >>> The root cause appears to be CONFIG_RPS=y and the fact that none of the
> >>> drivers that call skb_record_rx_queue() perform their net device
> >>> allocation using alloc_netdev_mq(), thereby initializing num_rx_queues
> >>> to a maximum of 1.
> >>>
> >>> Given that this is early RPS days, is the warning in get_rps_cpu()
> >>> really necessary? It would appear that _all_ of the multi-receive queue
> >>> devices that call skb_record_rx_queue() will cause this log noise.
> >>>
> >>> By the way, how do you turn off CONFIG_RPS? The only way I could get it
> >>> disabled was to change the default in net/Kconfig to 'n'.
> >>>
> >>> rtg
> >>
> >> This is the route that I'm taking with Ubuntu in the short term. I'll
> >> have lots of server testers complaining pretty soon if I don't take care
> >> of this now. It does keep my server logs from filling.
> >>
> >> rtg
> >>
> >
> > Probably fine, but your commit message is not exact :
> >
> >    So far no users of skb_record_rx_queue() use alloc_netdev_mq() for
> >    network device initialization, so don't print a warning about num_rx_queues
> >    imbalances in get_rps_cpu() unless they have actually been allocated.
> >
> > In fact, drivers that use skb_record_rx_queue() did use alloc_netdev_mq().
> >
> > Problem is : packets going thru bridge/bonding that are not yet
> > multiqueue enabled. If R[PF]S enabled for these "virtual devices",
> > we trigger the get_rps_cpu() warning.
> >
> > Also, in a bonding setup, we still have a problem
> > because all tx packets will go thru tx queue 0 (dev_pick_tx() job)
> >
> > (That might be good to know that for Ubuntu server testers)
> >
> 
> How about this?
> 

Acked-by: Eric Dumazet <eric.dumazet@gmail.com>

Thanks !