WARNING at dev_disable_lro when enabling ip_forward

WARNING at dev_disable_lro when enabling ip_forward

[Sergey Kononenko]

Hello,

I have system with two Intel 82575EB 1Gb/s NICs and two Intel 82598EB
10Gb/s NICs (motherboard Supermicro X7DWU). I'm using kernel 2.6.29.3
with compiled in drivers for both types of NICs with LRO enabled:
CONFIG_INET_LRO=y
CONFIG_IGB=y
CONFIG_IGB_LRO=y
CONFIG_IGB_DCA=y
CONFIG_IXGBE=y
CONFIG_IXGBE_DCA=y

After boot process is completed according to ethtool LRO is disabled on igb
and enabled on ixgbe. After I'm trying to turn on forwarding by
sysctl -w net.ipv4.ip_forward=1 I get following in dmesg:
------------[ cut here ]------------
WARNING: at net/core/dev.c:1199 dev_disable_lro+0x7c/0x80()
Hardware name: X7DWU
Modules linked in: ipt_REJECT ipt_ttl xt_multiport xt_hashlimit xt_length xt_NOTRACK xt_tcpudp xt_state ipt_set ip_set_i
ptree ip_set iptable_raw iptable_filter ip_tables x_tables nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 8021q msr cpuid
 ipmi_devintf ipmi_si ipmi_msghandler coretemp w83793 hwmon_vid i2c_dev loop i2c_i801 psmouse pcspkr i2c_core evdev i5k_
amb i5400_edac edac_core container button usbhid hid usb_storage sg sr_mod cdrom ehci_hcd uhci_hcd usbcore thermal proce
ssor fan thermal_sys hwmon
Pid: 5869, comm: sysctl Not tainted 2.6.29.3 #1
Call Trace:
 [<ffffffff8023b5d0>] warn_slowpath+0xd0/0x130
 [<ffffffff802872b3>] ? __alloc_pages_internal+0xe3/0x4f0
 [<ffffffff8028a3b0>] ? lru_cache_add_lru+0x20/0x50
 [<ffffffff802a12d5>] ? page_add_new_anon_rmap+0x85/0x90
 [<ffffffff80298003>] ? handle_mm_fault+0x553/0x770
 [<ffffffff8029c9bc>] ? vma_adjust+0xfc/0x4e0
 [<ffffffff804c4e8c>] dev_disable_lro+0x7c/0x80
 [<ffffffff8051c2e0>] devinet_sysctl_forward+0x140/0x160
 [<ffffffff8030b7c7>] proc_sys_call_handler+0xc7/0xe0
 [<ffffffff8030b7ef>] proc_sys_write+0xf/0x20
 [<ffffffff802b3f2b>] vfs_write+0xcb/0x190
 [<ffffffff802b40e0>] sys_write+0x50/0x90
 [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
---[ end trace 2907c917776795f3 ]---
------------[ cut here ]------------
WARNING: at net/core/dev.c:1199 dev_disable_lro+0x7c/0x80()
Hardware name: X7DWU
Modules linked in: ipt_REJECT ipt_ttl xt_multiport xt_hashlimit xt_length xt_NOTRACK xt_tcpudp xt_state ipt_set ip_set_iptree ip_set iptable_raw iptable_filter ip_tables x_tables nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 8021q msr cpuid ipmi_devintf ipmi_si ipmi_msghandler coretemp w83793 hwmon_vid i2c_dev loop i2c_i801 psmouse pcspkr i2c_core evdev i5k_amb i5400_edac edac_core container button usbhid hid usb_storage sg sr_mod cdrom ehci_hcd uhci_hcd usbcore thermal processor fan thermal_sys hwmon
Pid: 5869, comm: sysctl Tainted: G        W  2.6.29.3 #1
Call Trace:
 [<ffffffff8023b5d0>] warn_slowpath+0xd0/0x130
 [<ffffffff802872b3>] ? __alloc_pages_internal+0xe3/0x4f0
 [<ffffffff8028a3b0>] ? lru_cache_add_lru+0x20/0x50
 [<ffffffff802a12d5>] ? page_add_new_anon_rmap+0x85/0x90
 [<ffffffff80298003>] ? handle_mm_fault+0x553/0x770
 [<ffffffff8029c9bc>] ? vma_adjust+0xfc/0x4e0
 [<ffffffff804c4e8c>] dev_disable_lro+0x7c/0x80
 [<ffffffff8051c2e0>] devinet_sysctl_forward+0x140/0x160
 [<ffffffff8030b7c7>] proc_sys_call_handler+0xc7/0xe0
 [<ffffffff8030b7ef>] proc_sys_write+0xf/0x20
 [<ffffffff802b3f2b>] vfs_write+0xcb/0x190
 [<ffffffff802b40e0>] sys_write+0x50/0x90
 [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
---[ end trace 2907c917776795f4 ]---


Disabling ethtool -K eth0 lro off (and eth1 too) before turning on
forwarding don't help. I still get this warning but looks like kernel
works normally after it. I don't want jsut to disable LRO in kernel
config because I use same build of kernel on another machines there are
no need in ip forwarding. So may be this warning can be fixed somehow?

With best regards,
Sergey Kononenko.

[IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[David Miller]

From: Sergey Kononenko <sergk@sergk.org.ua>
Date: Tue, 19 May 2009 01:01:47 +0300

> I have system with two Intel 82575EB 1Gb/s NICs and two Intel 82598EB
> 10Gb/s NICs (motherboard Supermicro X7DWU). I'm using kernel 2.6.29.3
> with compiled in drivers for both types of NICs with LRO enabled:
> CONFIG_INET_LRO=y
> CONFIG_IGB=y
> CONFIG_IGB_LRO=y
> CONFIG_IGB_DCA=y
> CONFIG_IXGBE=y
> CONFIG_IXGBE_DCA=y
> 
> After boot process is completed according to ethtool LRO is disabled on igb
> and enabled on ixgbe. After I'm trying to turn on forwarding by
> sysctl -w net.ipv4.ip_forward=1 I get following in dmesg:

The problem is that the IGB driver in 2.6.29.3 is buggy, it supports
LRO but does not hook up a ->set_flags ethtool_ops function.  That's
why this warning triggers.

It should be pretty easy to fix, since there is a generic
implementation provided in net/core/ethtool.c, and IXGB properly
hooks up this method as well.

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[Stephen Hemminger]

Does this fix it?

diff --git a/drivers/net/igb/igb_ethtool.c b/drivers/net/igb/igb_ethtool.c
index 4bdfc2e..a425d85 100644
--- a/drivers/net/igb/igb_ethtool.c
+++ b/drivers/net/igb/igb_ethtool.c
@@ -2028,6 +2028,10 @@ static struct ethtool_ops igb_ethtool_ops = {
 	.get_ethtool_stats      = igb_get_ethtool_stats,
 	.get_coalesce           = igb_get_coalesce,
 	.set_coalesce           = igb_set_coalesce,
+#ifdef CONFIG_IGB_LRO
+	.get_flags		= ethtool_op_get_flags,
+	.set_flags		= ethtool_op_set_flags,
+#endif
 };
 
 void igb_set_ethtool_ops(struct net_device *netdev)

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[David Miller]

From: Stephen Hemminger <shemminger@vyatta.com>
Date: Mon, 18 May 2009 15:57:14 -0700

> Does this fix it?

It should, but I think the ifdef is probably superfluous.

> diff --git a/drivers/net/igb/igb_ethtool.c b/drivers/net/igb/igb_ethtool.c
> index 4bdfc2e..a425d85 100644
> --- a/drivers/net/igb/igb_ethtool.c
> +++ b/drivers/net/igb/igb_ethtool.c
> @@ -2028,6 +2028,10 @@ static struct ethtool_ops igb_ethtool_ops = {
>  	.get_ethtool_stats      = igb_get_ethtool_stats,
>  	.get_coalesce           = igb_get_coalesce,
>  	.set_coalesce           = igb_set_coalesce,
> +#ifdef CONFIG_IGB_LRO
> +	.get_flags		= ethtool_op_get_flags,
> +	.set_flags		= ethtool_op_set_flags,
> +#endif
>  };
>  
>  void igb_set_ethtool_ops(struct net_device *netdev)

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[Jeff Kirsher]

On Mon, May 18, 2009 at 4:00 PM, David Miller <davem@davemloft.net> wrote:
> From: Stephen Hemminger <shemminger@vyatta.com>
> Date: Mon, 18 May 2009 15:57:14 -0700
>
>> Does this fix it?
>
> It should, but I think the ifdef is probably superfluous.
>

I agree with Dave, the #ifdef is not necessary.  Thanks Stephen for
the quick patch.

>> diff --git a/drivers/net/igb/igb_ethtool.c b/drivers/net/igb/igb_ethtool.c
>> index 4bdfc2e..a425d85 100644
>> --- a/drivers/net/igb/igb_ethtool.c
>> +++ b/drivers/net/igb/igb_ethtool.c
>> @@ -2028,6 +2028,10 @@ static struct ethtool_ops igb_ethtool_ops = {
>>       .get_ethtool_stats      = igb_get_ethtool_stats,
>>       .get_coalesce           = igb_get_coalesce,
>>       .set_coalesce           = igb_set_coalesce,
>> +#ifdef CONFIG_IGB_LRO
>> +     .get_flags              = ethtool_op_get_flags,
>> +     .set_flags              = ethtool_op_set_flags,
>> +#endif
>>  };
>>
>>  void igb_set_ethtool_ops(struct net_device *netdev)

If this resolves Sergey's issue, I will push this to the stable tree
(minus the #ifdef).

-- 
Cheers,
Jeff

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[Stephen Hemminger]

On Mon, 18 May 2009 16:00:29 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:

> From: Stephen Hemminger <shemminger@vyatta.com>
> Date: Mon, 18 May 2009 15:57:14 -0700
> 
> > Does this fix it?
> 
> It should, but I think the ifdef is probably superfluous.
> 
> > diff --git a/drivers/net/igb/igb_ethtool.c b/drivers/net/igb/igb_ethtool.c
> > index 4bdfc2e..a425d85 100644
> > --- a/drivers/net/igb/igb_ethtool.c
> > +++ b/drivers/net/igb/igb_ethtool.c
> > @@ -2028,6 +2028,10 @@ static struct ethtool_ops igb_ethtool_ops = {
> >  	.get_ethtool_stats      = igb_get_ethtool_stats,
> >  	.get_coalesce           = igb_get_coalesce,
> >  	.set_coalesce           = igb_set_coalesce,
> > +#ifdef CONFIG_IGB_LRO
> > +	.get_flags		= ethtool_op_get_flags,
> > +	.set_flags		= ethtool_op_set_flags,
> > +#endif
> >  };
> >  
> >  void igb_set_ethtool_ops(struct net_device *netdev)

For get, the ifdef is superflous, but for set you don't want
the user to turn LRO on if not configured.


-- 

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[David Miller]

From: Stephen Hemminger <shemminger@vyatta.com>
Date: Mon, 18 May 2009 16:08:04 -0700

> On Mon, 18 May 2009 16:00:29 -0700 (PDT)
> David Miller <davem@davemloft.net> wrote:
> 
>> > @@ -2028,6 +2028,10 @@ static struct ethtool_ops igb_ethtool_ops = {
>> >  	.get_ethtool_stats      = igb_get_ethtool_stats,
>> >  	.get_coalesce           = igb_get_coalesce,
>> >  	.set_coalesce           = igb_set_coalesce,
>> > +#ifdef CONFIG_IGB_LRO
>> > +	.get_flags		= ethtool_op_get_flags,
>> > +	.set_flags		= ethtool_op_set_flags,
>> > +#endif
>> >  };
>> >  
>> >  void igb_set_ethtool_ops(struct net_device *netdev)
> 
> For get, the ifdef is superflous, but for set you don't want
> the user to turn LRO on if not configured.

Stephen's right, ->set_flags has to be ifdef protected.

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[Sergey Kononenko]

> Does this fix it?
> 
> diff --git a/drivers/net/igb/igb_ethtool.c
> b/drivers/net/igb/igb_ethtool.c index 4bdfc2e..a425d85 100644
> --- a/drivers/net/igb/igb_ethtool.c
> +++ b/drivers/net/igb/igb_ethtool.c
> @@ -2028,6 +2028,10 @@ static struct ethtool_ops igb_ethtool_ops = {
>  	.get_ethtool_stats      = igb_get_ethtool_stats,
>  	.get_coalesce           = igb_get_coalesce,
>  	.set_coalesce           = igb_set_coalesce,
> +#ifdef CONFIG_IGB_LRO
> +	.get_flags		= ethtool_op_get_flags,
> +	.set_flags		= ethtool_op_set_flags,
> +#endif
>  };
Yes, patch have fixed the problem. Thank you.

With best regards,
Sergey Kononenko.

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[Jeff Kirsher]

On Tue, May 19, 2009 at 1:46 PM, Sergey Kononenko <sergk@sergk.org.ua> wrote:
>> Does this fix it?
>>
>> diff --git a/drivers/net/igb/igb_ethtool.c
>> b/drivers/net/igb/igb_ethtool.c index 4bdfc2e..a425d85 100644
>> --- a/drivers/net/igb/igb_ethtool.c
>> +++ b/drivers/net/igb/igb_ethtool.c
>> @@ -2028,6 +2028,10 @@ static struct ethtool_ops igb_ethtool_ops = {
>>       .get_ethtool_stats      = igb_get_ethtool_stats,
>>       .get_coalesce           = igb_get_coalesce,
>>       .set_coalesce           = igb_set_coalesce,
>> +#ifdef CONFIG_IGB_LRO
>> +     .get_flags              = ethtool_op_get_flags,
>> +     .set_flags              = ethtool_op_set_flags,
>> +#endif
>>  };
> Yes, patch have fixed the problem. Thank you.
>
> With best regards,
> Sergey Kononenko.
> --

Thanks Sergey and Stephen, I will get this queued up for the current
and stable trees.

-- 
Cheers,
Jeff

Re: [IGB 2.6.29.3 bug] Re: WARNING at dev_disable_lro when enabling ip_forward

[David Miller]

From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Date: Tue, 19 May 2009 15:06:57 -0700

> Thanks Sergey and Stephen, I will get this queued up for the current
> and stable trees.

current doesn't need it, since igb uses GRO now.