Bug related to bonding

Bug related to bonding

[Heiko Gerstung]

Hi!

I am at a total loss with this one.  My vanilla 2.4.32 crashes when I
try to use bonding together with my rtl8150 based (USB-Ethernet) NICs.
If this is a known error, I apologize for bothering the list and would
appreciate any pointers to a working solution/workaround.

Reproduce:
# modprobe bonding mode=1 miimon=100 maxbonds=4
# ifconfig bond0 172.16.10.111 netmask 255.255.255.0 up
# ifenslave bond0 eth1 eth2
Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
bonding: MII link monitoring set to 100 ms
00:60:6E:30:07:Scheduling in interrupt
kernel BUG at sched.c:564!
invalid operand: 0000
CPU:     0
EIP:      0010:[<c011461d>]    Not tainted
EFLAGS:  00010282
...(following the CPU registers and Call Trace)....

Please let me know which details I have to provide from the bug message
(I have to type it in manually, no copy'n'paste possible:-)).

It is not clear to me whether this is a bug in the bonding module, in
the network driver or in the kernel itself.
All 2.6.x kernels I tried worked fine, but I am currently bound to a
2.4.x kernel and all 2.4.x kernels I tried (2.4.20, 2.4.29) showed
similiar problems when activating bonding.

Thank you in advance,
kind regards,
Heiko

Re: Bug related to bonding

[Willy Tarreau]

Hi,

On Mon, May 15, 2006 at 11:35:28AM +0200, Heiko Gerstung wrote:
> Hi!
> 
> I am at a total loss with this one.  My vanilla 2.4.32 crashes when I
> try to use bonding together with my rtl8150 based (USB-Ethernet) NICs.
> If this is a known error, I apologize for bothering the list and would
> appreciate any pointers to a working solution/workaround.
> 
> Reproduce:
> # modprobe bonding mode=1 miimon=100 maxbonds=4
> # ifconfig bond0 172.16.10.111 netmask 255.255.255.0 up
> # ifenslave bond0 eth1 eth2
> Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
> bonding: MII link monitoring set to 100 ms
> 00:60:6E:30:07:Scheduling in interrupt
> kernel BUG at sched.c:564!
> invalid operand: 0000
> CPU:     0
> EIP:      0010:[<c011461d>]    Not tainted
> EFLAGS:  00010282
> ...(following the CPU registers and Call Trace)....

It looks like what causes trouble is the link state monitoring. Please
try to disable 'miimon' just to ensure that it does not crash. If it
does not crash, a quick workaround would be to use another network
driver/card, but a good fix would be to send us the complete oops and
its decoding through ksymoops.

In the mean time, if you absolutely need to use this card and the link
monitoring, you might use the ARP monitoring instead. Just point it to
a valid IP on the same segment to detect its reachability through your
NICs.

> Please let me know which details I have to provide from the bug message
> (I have to type it in manually, no copy'n'paste possible:-)).

If you can use a serial console, it will help you. If you don't have
enough time to copy it by hand, boot with 'panic=180' to get 3 minutes
before the automatic reboot. It is very important to get the other
registers, and particularly the stack dump to know what function called
schedule().
 
> It is not clear to me whether this is a bug in the bonding module, in
> the network driver or in the kernel itself.

Possibly both. The bonding driver checks the link status regularly through
the use of MII ioctls. It's possible that the driver does nasty things during
this call.

> All 2.6.x kernels I tried worked fine, but I am currently bound to a
> 2.4.x kernel and all 2.4.x kernels I tried (2.4.20, 2.4.29) showed
> similiar problems when activating bonding.

That's interesting, I'll try to diff the bonding driver between 2.4 and
2.6. For info, I have multiple production machines running it on 2.4 with
e1000 and tg3 drivers which never had a single problem during years of
uptime.

> Thank you in advance,
> kind regards,
> Heiko

Regards,
Willy

Re: Bug related to bonding

[Heiko Gerstung]

Good morning/afternoon/evening (please choose your favourtite time of
day:-))!

Willy Tarreau wrote:
> Hi,
>
> On Mon, May 15, 2006 at 11:35:28AM +0200, Heiko Gerstung wrote:
>   
>> Hi!
>>
>> I am at a total loss with this one.  My vanilla 2.4.32 crashes when I
>> try to use bonding together with my rtl8150 based (USB-Ethernet) NICs.
>> If this is a known error, I apologize for bothering the list and would
>> appreciate any pointers to a working solution/workaround.
>>
>> Reproduce:
>> # modprobe bonding mode=1 miimon=100 maxbonds=4
>> # ifconfig bond0 172.16.10.111 netmask 255.255.255.0 up
>> # ifenslave bond0 eth1 eth2
>> Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
>> bonding: MII link monitoring set to 100 ms
>> 00:60:6E:30:07:Scheduling in interrupt
>> kernel BUG at sched.c:564!
>> invalid operand: 0000
>> CPU:     0
>> EIP:      0010:[<c011461d>]    Not tainted
>> EFLAGS:  00010282
>> ...(following the CPU registers and Call Trace)....
>>     
>
> It looks like what causes trouble is the link state monitoring. Please
> try to disable 'miimon' just to ensure that it does not crash. If it
> does not crash, a quick workaround would be to use another network
> driver/card, but a good fix would be to send us the complete oops and
> its decoding through ksymoops.
>
> In the mean time, if you absolutely need to use this card and the link
> monitoring, you might use the ARP monitoring instead. Just point it to
> a valid IP on the same segment to detect its reachability through your
> NICs.
>   

Thank you very much for the hint. I was able to track this down to the
driver which seems to crash when trying to serve a ETHTOOL_GSET ioctl
from the bonding driver. When I comment that out, it does not crash
anymore but then there is link detection available, of course.

>> Please let me know which details I have to provide from the bug message
>> (I have to type it in manually, no copy'n'paste possible:-)).
>>     
>
> If you can use a serial console, it will help you. If you don't have
> enough time to copy it by hand, boot with 'panic=180' to get 3 minutes
> before the automatic reboot. It is very important to get the other
> registers, and particularly the stack dump to know what function called
> schedule().
>   

Another nice hint, but unfortunately the device I have to deal with has
no serial ports available.
>> It is not clear to me whether this is a bug in the bonding module, in
>> the network driver or in the kernel itself.
>>     
> Possibly both. The bonding driver checks the link status regularly through
> the use of MII ioctls. It's possible that the driver does nasty things during
> this call.
>   
For me it looks like the driver (rtl8150.c) seems to always call
"copy_to_user" when it handles such a ETHTOOL_GSET call. I would assume
that this causes trouble when the bonding module uses the ioctl, because
there is no userspace memory that has to be addressed in such a case.
Mmmh, it should find out from the pointer (I get with the ETHTOOL ioctl)
if this is kernelspace or userspace and then call copy_to_user only if
applicable.

Anyone here who can tell me how to handle this (or point me to a driver
which already does that)?
>> All 2.6.x kernels I tried worked fine, but I am currently bound to a
>> 2.4.x kernel and all 2.4.x kernels I tried (2.4.20, 2.4.29) showed
>> similiar problems when activating bonding.
>>     
>
> That's interesting, I'll try to diff the bonding driver between 2.4 and
> 2.6. For info, I have multiple production machines running it on 2.4 with
> e1000 and tg3 drivers which never had a single problem during years of
> uptime.
>   
We had a number of problems with bonding and that particular driver, but
we are bound to it due to hardware design decisions. For instance, in a
2.4.20 driver the kernel crashed when we used NET-SNMP to get the
interface statistics for the bond0 interface.

Thanks again for your support,
kind regards,
Heiko

Re: Bug related to bonding

[Heiko Gerstung]

Heiko Gerstung wrote:
> Thank you very much for the hint. I was able to track this down to the
> driver which seems to crash when trying to serve a ETHTOOL_GSET ioctl
> from the bonding driver. When I comment that out, it does not crash
> anymore but then there is link detection available, of course.
>
>   

"... but then there is _no_ link detection available, of course" was
what I wanted to say...

Early in the morning Typo (tm)


Sorry!

Regards,
Heiko

Re: Bug related to bonding

[Willy Tarreau]

On Tue, May 16, 2006 at 07:03:38AM +0200, Heiko Gerstung wrote:
> Good morning/afternoon/evening (please choose your favourtite time of
> day:-))!

it was the morning by then :-)

> Thank you very much for the hint. I was able to track this down to the
> driver which seems to crash when trying to serve a ETHTOOL_GSET ioctl
> from the bonding driver. When I comment that out, it does not crash
> anymore but then there is link detection available, of course.

OK, this is a significative finding.

> > If you can use a serial console, it will help you. If you don't have
> > enough time to copy it by hand, boot with 'panic=180' to get 3 minutes
> > before the automatic reboot. It is very important to get the other
> > registers, and particularly the stack dump to know what function called
> > schedule().
> >   
> 
> Another nice hint, but unfortunately the device I have to deal with has
> no serial ports available.

In this case, hand-copying is still available. We've all done that a lot
of times and it does not take as long as you can imagine.

> For me it looks like the driver (rtl8150.c) seems to always call
> "copy_to_user" when it handles such a ETHTOOL_GSET call. I would assume
> that this causes trouble when the bonding module uses the ioctl, because
> there is no userspace memory that has to be addressed in such a case.
> Mmmh, it should find out from the pointer (I get with the ETHTOOL ioctl)
> if this is kernelspace or userspace and then call copy_to_user only if
> applicable.

Well, at first, I did not find this driver, then I realized that it was
and USB one ! Its ethtool ioctl calls usb_control_msg() which is defined
in usb/usb.c. Look at the header :

 *      Don't use this function from within an interrupt context, like a
 *      bottom half handler.  If you need a asyncronous message, or need to send
 *      a message from within interrupt context, use usb_submit_urb()
 */
int usb_control_msg(struct usb_device *dev, unsigned int pipe, __u8 request,

So I think that the rtl8150 driver is simply buggy, or at least does not
expect to be used this way.

> Anyone here who can tell me how to handle this (or point me to a driver
> which already does that)?

May be you can try to change the 2 usb_control_msg() calls for a
combination of FILL_CONTROL_URB() + usb_submit_urb ? Hmmm reading the
code, it looks like nearly everything is already provided. In
rtl8150_ethtool_ioctl(), you should try to replaces occurences of
get_registers() by async_get_registers() that you will write by
comparing set_registers() with async_set_registers().

> >> All 2.6.x kernels I tried worked fine, but I am currently bound to a
> >> 2.4.x kernel and all 2.4.x kernels I tried (2.4.20, 2.4.29) showed
> >> similiar problems when activating bonding.
> >>     
> >
> > That's interesting, I'll try to diff the bonding driver between 2.4 and
> > 2.6. For info, I have multiple production machines running it on 2.4 with
> > e1000 and tg3 drivers which never had a single problem during years of
> > uptime.
> >   
> We had a number of problems with bonding and that particular driver, but
> we are bound to it due to hardware design decisions. For instance, in a
> 2.4.20 driver the kernel crashed when we used NET-SNMP to get the
> interface statistics for the bond0 interface.

Older versions were awful, but it ahs quickly stabilized and seems OK to me
in *recent* kernels.

> Thanks again for your support,
> kind regards,
> Heiko

Regards,
Willy

Re: Bug related to bonding

[Heiko Gerstung]

Hi, Willy:

Willy Tarreau wrote:
> So I think that the rtl8150 driver is simply buggy, or at least does not
> expect to be used this way.
>
>   
Yes, the original author (Petko) confirmed a few minutes ago that this
is the case.
>> Anyone here who can tell me how to handle this (or point me to a driver
>> which already does that)?
>>     
>
> May be you can try to change the 2 usb_control_msg() calls for a
> combination of FILL_CONTROL_URB() + usb_submit_urb ? Hmmm reading the
> code, it looks like nearly everything is already provided. In
> rtl8150_ethtool_ioctl(), you should try to replaces occurences of
> get_registers() by async_get_registers() that you will write by
> comparing set_registers() with async_set_registers().
>   
I tried that and it seems that it segfaults now when trying to execute
the following line:

dev->ctrl_urb->transfer_buffer_length = size;

with the following panic message:

Unable to handle kernel NULL pointer dereference at virtual address 00000028
printing eip:
c48a229a
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0010:[<c48a229a>] Not tainted
EFLAGS: 00010286
eax: 00000000 ebx: c3aae000 ecx: 00000001 edx: 00000001
esi: 0000012e edi: 00000001 ebp: 00000001 esp: c3b3fe28
ds: 0018 es: 0018 ss: 0018
Process modprobe.old (pid: 55, stackpage=c3b3f000)
Stack: c48a3b05 c48a3af5 c48a3ae5 c3aae08c c030b1e0 00000064 c3b3fe63
c3aae000
00000000 c48a2575 c3aae000 0000012e 00000001 c3b3fe63 10000000 c3aae000
c10e3200 c10e3000 c48a3399 c3aae000 c39af220 c0286a00 c02e03c8 00001708
Call Trace: [<c48a3b05>] [<c48a3af5>] [<c48a3ae5>] [<c48a2575>] [<c48a3399>]
[<c48a3ce0>] [<c48a3d60>] [<c01c7339>] [<c48a3ce0>] [<c48a3d40>]
[<c01c7044>]
[<c01c7066>] [<c01c6691>] [<c48a3d40>] [<c01c6635>] [<c48a34a4>]
[<c48a3d40>]
[<c48a3aa0>] [<c0117ce2>] [<c48a2060>] [<c0108903>]

Code: 89 78 28 68 15 3b 8a c4 e8 69 4c 87 fb 8b 43 04 8b 93 88 00
/etc/myscript: line 43: 55 Segmentation fault modprobe rtl8150 >
/dev/null 2>&1

(wow! hand copying that was what I needed at the end of a working day :-))

Please find below the complete async_get_registers function I set up, I
hope it's OK to post it here. A kernel hacker will immediately spot the
error, no? :-)

Kind regards,
Heiko


static int get_registers(rtl8150_t * dev, u16 indx, u16 size, void *data)
{
int ret;
char *buffer;

printk("get_registers dev=%08X dev->dr=%08X indx=%d size=%d\n",(unsigned
long) dev, (unsigned long) &dev->dr, indx,size);
buffer = kmalloc(size, GFP_DMA);
if (!buffer) {
warn("%s: looks like we're out of memory", __FUNCTION__);
return -ENOMEM;
}


if (test_bit(RX_REG_SET, &dev->flags))
return -EAGAIN;

dev->dr.bRequestType = RTL8150_REQT_READ;
dev->dr.bRequest = RTL8150_REQ_GET_REGS;
dev->dr.wValue = cpu_to_le16(indx);
dev->dr.wIndex = cpu_to_le16p(&indx);
dev->dr.wLength = cpu_to_le16p(&size);

dev->ctrl_urb->transfer_buffer_length = size;

FILL_CONTROL_URB(dev->ctrl_urb, dev->udev,
usb_rcvctrlpipe(dev->udev, 0), (char *) &dev->dr,
buffer, size, ctrl_callback, dev);

if ((ret = usb_submit_urb(dev->ctrl_urb)))
err("control request submission failed: %d", ret);

return ret;
}

Re: Bug related to bonding

[Willy Tarreau]

Hi Heiko,

On Tue, May 16, 2006 at 05:09:38PM +0200, Heiko Gerstung wrote:
> Hi, Willy:
> 
> Willy Tarreau wrote:
> > So I think that the rtl8150 driver is simply buggy, or at least does not
> > expect to be used this way.
> >
> >   
> Yes, the original author (Petko) confirmed a few minutes ago that this
> is the case.
> >> Anyone here who can tell me how to handle this (or point me to a driver
> >> which already does that)?
> >>     
> >
> > May be you can try to change the 2 usb_control_msg() calls for a
> > combination of FILL_CONTROL_URB() + usb_submit_urb ? Hmmm reading the
> > code, it looks like nearly everything is already provided. In
> > rtl8150_ethtool_ioctl(), you should try to replaces occurences of
> > get_registers() by async_get_registers() that you will write by
> > comparing set_registers() with async_set_registers().
> >   
> I tried that and it seems that it segfaults now when trying to execute
> the following line:
> 
> dev->ctrl_urb->transfer_buffer_length = size;
> 
> with the following panic message:
> 
> Unable to handle kernel NULL pointer dereference at virtual address 00000028
> printing eip:
> c48a229a
> *pde = 00000000
> Oops: 0002
> CPU: 0
> EIP: 0010:[<c48a229a>] Not tainted
> EFLAGS: 00010286
> eax: 00000000 ebx: c3aae000 ecx: 00000001 edx: 00000001
> esi: 0000012e edi: 00000001 ebp: 00000001 esp: c3b3fe28
> ds: 0018 es: 0018 ss: 0018
> Process modprobe.old (pid: 55, stackpage=c3b3f000)
> Stack: c48a3b05 c48a3af5 c48a3ae5 c3aae08c c030b1e0 00000064 c3b3fe63
> c3aae000
> 00000000 c48a2575 c3aae000 0000012e 00000001 c3b3fe63 10000000 c3aae000
> c10e3200 c10e3000 c48a3399 c3aae000 c39af220 c0286a00 c02e03c8 00001708
> Call Trace: [<c48a3b05>] [<c48a3af5>] [<c48a3ae5>] [<c48a2575>] [<c48a3399>]
> [<c48a3ce0>] [<c48a3d60>] [<c01c7339>] [<c48a3ce0>] [<c48a3d40>]
> [<c01c7044>]
> [<c01c7066>] [<c01c6691>] [<c48a3d40>] [<c01c6635>] [<c48a34a4>]
> [<c48a3d40>]
> [<c48a3aa0>] [<c0117ce2>] [<c48a2060>] [<c0108903>]
> 
> Code: 89 78 28 68 15 3b 8a c4 e8 69 4c 87 fb 8b 43 04 8b 93 88 00
> /etc/myscript: line 43: 55 Segmentation fault modprobe rtl8150 >
> /dev/null 2>&1
> 
> (wow! hand copying that was what I needed at the end of a working day :-))

You should really pass the oops through ksymoops. Of course, you'll need
some dump of ksyms and /proc/modules, but to get those, you simply have to
reload the bonding driver WITHOUT miimon to ensure that it does not crash,
and your memory should be in the exact same conditions. If only rtl8150 is
involved, you will not need bonding at all, only the rtl8150 driver will
be enough.

> Please find below the complete async_get_registers function I set up, I
> hope it's OK to post it here. A kernel hacker will immediately spot the
> error, no? :-)

Just a guess : I suspect that dev->ctrl_urb is NULL. I don't know how you
are supposed to initialize it though.

> Kind regards,
> Heiko

Regards,
Willy

> static int get_registers(rtl8150_t * dev, u16 indx, u16 size, void *data)
> {
> int ret;
> char *buffer;
> 
> printk("get_registers dev=%08X dev->dr=%08X indx=%d size=%d\n",(unsigned
> long) dev, (unsigned long) &dev->dr, indx,size);
> buffer = kmalloc(size, GFP_DMA);
> if (!buffer) {
> warn("%s: looks like we're out of memory", __FUNCTION__);
> return -ENOMEM;
> }
> 
> 
> if (test_bit(RX_REG_SET, &dev->flags))
> return -EAGAIN;
> 
> dev->dr.bRequestType = RTL8150_REQT_READ;
> dev->dr.bRequest = RTL8150_REQ_GET_REGS;
> dev->dr.wValue = cpu_to_le16(indx);
> dev->dr.wIndex = cpu_to_le16p(&indx);
> dev->dr.wLength = cpu_to_le16p(&size);
> 
> dev->ctrl_urb->transfer_buffer_length = size;
> 
> FILL_CONTROL_URB(dev->ctrl_urb, dev->udev,
> usb_rcvctrlpipe(dev->udev, 0), (char *) &dev->dr,
> buffer, size, ctrl_callback, dev);
> 
> if ((ret = usb_submit_urb(dev->ctrl_urb)))
> err("control request submission failed: %d", ret);
> 
> return ret;
> }
> 

USB net driver hacker wanted (was Re: Bug related to bonding)

[Heiko Gerstung]

Hi List!

I am still having problems with rtl8150.c on Linux 2.4.32 ...

Willy Tarreau wrote:
>> Please find below the complete async_get_registers function I set up, I
>> hope it's OK to post it here. A kernel hacker will immediately spot the
>> error, no? :-)
>>     
> Just a guess : I suspect that dev->ctrl_urb is NULL. I don't know how you
> are supposed to initialize it though.
>   
You were right :-) It was a question of how things get initialized.
My current state is that I am able to load the module but using my
async_get_register function does not succeed due to timeouts. I am no
kernel hacker and therefore it is like running through the darkest night
with my sunglasses on :-) ... I sincerely hope, the maintainer of the
driver helps me out...

It is interesting to see that with a 2.4.20 kernel I have no problems
with loading the driver, it only crashes when I run net-snmp on it and
then fire up an snmpwalk (which, I guess, queries the driver for some
interface statistics or something like that).

Deadline is coming and still no hope, yet. We'll see...


Kind regards,
Heiko


>   
>> Kind regards,
>> Heiko
>>     
>
> Regards,
> Willy
>
>   
>> static int get_registers(rtl8150_t * dev, u16 indx, u16 size, void *data)
>> {
>> int ret;
>> char *buffer;
>>
>> printk("get_registers dev=%08X dev->dr=%08X indx=%d size=%d\n",(unsigned
>> long) dev, (unsigned long) &dev->dr, indx,size);
>> buffer = kmalloc(size, GFP_DMA);
>> if (!buffer) {
>> warn("%s: looks like we're out of memory", __FUNCTION__);
>> return -ENOMEM;
>> }
>>
>>
>> if (test_bit(RX_REG_SET, &dev->flags))
>> return -EAGAIN;
>>
>> dev->dr.bRequestType = RTL8150_REQT_READ;
>> dev->dr.bRequest = RTL8150_REQ_GET_REGS;
>> dev->dr.wValue = cpu_to_le16(indx);
>> dev->dr.wIndex = cpu_to_le16p(&indx);
>> dev->dr.wLength = cpu_to_le16p(&size);
>>
>> dev->ctrl_urb->transfer_buffer_length = size;
>>
>> FILL_CONTROL_URB(dev->ctrl_urb, dev->udev,
>> usb_rcvctrlpipe(dev->udev, 0), (char *) &dev->dr,
>> buffer, size, ctrl_callback, dev);
>>
>> if ((ret = usb_submit_urb(dev->ctrl_urb)))
>> err("control request submission failed: %d", ret);
>>
>> return ret;
>> }
>>
>>     
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>   

Re: USB net driver hacker wanted (was Re: Bug related to bonding)

[Willy Tarreau]

On Wed, May 17, 2006 at 08:31:20PM +0200, Heiko Gerstung wrote:
> Hi List!
> 
> I am still having problems with rtl8150.c on Linux 2.4.32 ...
> 
> Willy Tarreau wrote:
> >> Please find below the complete async_get_registers function I set up, I
> >> hope it's OK to post it here. A kernel hacker will immediately spot the
> >> error, no? :-)
> >>     
> > Just a guess : I suspect that dev->ctrl_urb is NULL. I don't know how you
> > are supposed to initialize it though.
> >   
> You were right :-) It was a question of how things get initialized.
> My current state is that I am able to load the module but using my
> async_get_register function does not succeed due to timeouts. I am no
> kernel hacker and therefore it is like running through the darkest night
> with my sunglasses on :-) ... I sincerely hope, the maintainer of the
> driver helps me out...

I have no clue in this area either, unfortunately.

> It is interesting to see that with a 2.4.20 kernel I have no problems
> with loading the driver, it only crashes when I run net-snmp on it and
> then fire up an snmpwalk (which, I guess, queries the driver for some
> interface statistics or something like that).

You mean the bonding driver ? Older versions did not support ethtool ioctl(),
but only MII calls. I don't remember at what version it changed, but I suspect
from what you're saying that your driver might work if you disable ethtool.

Take a look at the bonding driver, there's a place where it first tries if
ethtool is supported on the driver, and if not it uses MII. I think that
commenting out one "if" statement would be close to enough.

> Deadline is coming and still no hope, yet. We'll see...

There's always hope, you have all the code :-)

> 
> Kind regards,
> Heiko

Regards,
Willy