Re: oops in skbuff.c

Re: oops in skbuff.c

[James Morris]

On Thu, 16 Oct 2003, Sebastian Piecha wrote:

>  panics. Copying more than 4 GB to the samba share also lets the 
> kernel panic with an OOPS.

Does the above cause a panic for the 2.6-test kernels?
Do you have netfilter enabled, and if so, any iptables modules loaded?


- James
-- 
James Morris
<jmorris@redhat.com>

Re: oops in skbuff.c

[Sebastian Piecha]

On 16 Oct 2003 at 15:15, James Morris wrote:

> On Thu, 16 Oct 2003, Sebastian Piecha wrote:
> 
> >  panics. Copying more than 4 GB to the samba share also lets the 
> > kernel panic with an OOPS.
> 
> Does the above cause a panic for the 2.6-test kernels?
> Do you have netfilter enabled, and if so, any iptables modules loaded?
> 
No. Only in 2.4-kernels. No netfilter. No iptables. Just samba 
sharing some reiserfs directories on a Promise attached ide harddisk.

Mit freundlichen Gruessen/Best regards,
Sebastian Piecha

EMail: spi@gmxpro.de

Re: oops in skbuff.c

[James Morris]

On Thu, 16 Oct 2003, Sebastian Piecha wrote:

> On 16 Oct 2003 at 15:15, James Morris wrote:
> > On Thu, 16 Oct 2003, Sebastian Piecha wrote:
> > 
> > Does the above cause a panic for the 2.6-test kernels?
> > Do you have netfilter enabled, and if so, any iptables modules loaded?
> > 
> No. Only in 2.4-kernels. No netfilter. No iptables. Just samba 
> sharing some reiserfs directories on a Promise attached ide harddisk.

Are you able to test with a different NIC to the onboard 3C905B?


- James
-- 
James Morris
<jmorris@redhat.com>

Re: oops in skbuff.c

[Sebastian Piecha]

On 16 Oct 2003 at 16:37, James Morris wrote:

> On Thu, 16 Oct 2003, Sebastian Piecha wrote:
> 
> > On 16 Oct 2003 at 15:15, James Morris wrote:
> > > On Thu, 16 Oct 2003, Sebastian Piecha wrote:
> > > 
> > > Does the above cause a panic for the 2.6-test kernels?
> > > Do you have netfilter enabled, and if so, any iptables modules loaded?
> > > 
> > No. Only in 2.4-kernels. No netfilter. No iptables. Just samba 
> > sharing some reiserfs directories on a Promise attached ide harddisk.
> 
> Are you able to test with a different NIC to the onboard 3C905B?
> 
> 

Unfortunately I only have 3com NICs. Do you think it's the adapter? 
But why it's working in kernel 2.6? There are different changes in 
skbuff.c between kernel 2.4.22-ac4 and 2.6.0-test7. Is there any 
slight chance to port it back to 2.4.x?

Mit freundlichen Gruessen/Best regards,
Sebastian Piecha

EMail: spi@gmxpro.de

oops in skbuff.c

[Sebastian Piecha]

Hello,

this is a copy of a mail I sent to the linux-net mailing list.

Every time accessing an huge amount of data from a Win XP client to a 
linux server via samba I'm getting an oops. Different mailings to the 
linux kernel mailling list or samba bugzilla didn't help. Any help 
would be appreciated. Please CC me on all further mail traffic.

I'm getting the oops in kernel 2.4.20, 2.4.22-ac4 and 2.4.23pre1. No 
oops occurred in 2.6.0test1 and 2.6.0test7.

I'm using samba 2.2.8a (samba 2.2.7a showed same behaviour).

The oops happened all time in skb_drop_fraglist.

I run memtest for about 25 hours without any error.

I have the dim feeling that skbuff.c is the source of evil. Comparing 
the Promise drivers in 2.4.22-ac4 and 2.6.0-test7 shows only few 
differences (but maybe the important one). In skbuff.c a lot of 
things changed. I don't have a clue what the changes are for. If just 
for fixing bugs skbuff.c could maybe easily be ported back to 2.4?

Capture of an oops in 2.4.22-ac4:

Oops: 0000
CPU:    0
EIP:    0010:[<c02518a3>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010206
eax: c525e660   ebx: 00200000   ecx: 00000000   edx: 00200000
esi: c5c97d40   edi: c5c97da0   ebp: c038ba24   esp: c034ff18
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0, stackpage=c034f000)
Stack: c5c97d40 c025194b c5c97d40 c5c97d40 c5c97d40 c025196c c5c97d40 

c5c97d40
       c5c97d40 c0251ab4 c5c97d40 c57acd00 00000000 c0255103 c5c97d40 

c038b4c8
       00000001 fffffffd c03ac328 d3051000 c011e2cd c038b4c8 c03ac320 

c03729c0
Call Trace:    [<c025194b>] [<c025196c>] [<c0251ab4>] [<c0255103>] 
[<c011e2cd>]
  [<c010a32d>] [<c0106d70>] [<c0105000>] [<c010c7d8>] [<c0106d70>] 
[<c0105000>]
  [<c0106d9c>] [<c0106deb>] [<c0105049>]
Code: 8b 1b 8b 42 74 83 f8 01 74 0b f0 ff 4a 74 0f 94 c0 84 c0 74


>>EIP; c02518a3 <skb_drop_fraglist+17/3c>   <=====

>>eax; c525e660 <_end+4e6fe3c/14cc883c>
>>esi; c5c97d40 <_end+58a951c/14cc883c>
>>edi; c5c97da0 <_end+58a957c/14cc883c>
>>ebp; c038ba24 <softnet_data+24/3400>
>>esp; c034ff18 <init_task_union+1f18/2000>

Trace; c025194b <skb_release_data+5f/74>
Trace; c025196c <kfree_skbmem+c/68>
Trace; c0251ab4 <__kfree_skb+ec/f4>
Trace; c0255103 <net_tx_action+5f/11c>
Trace; c011e2cd <do_softirq+7d/dc>
Trace; c010a32d <do_IRQ+dd/ec>
Trace; c0106d70 <default_idle+0/34>
Trace; c0105000 <_stext+0/0>
Trace; c010c7d8 <call_do_IRQ+5/d>
Trace; c0106d70 <default_idle+0/34>
Trace; c0105000 <_stext+0/0>
Trace; c0106d9c <default_idle+2c/34>
Trace; c0106deb <cpu_idle+27/34>
Trace; c0105049 <rest_init+49/4c>

Code;  c02518a3 <skb_drop_fraglist+17/3c>
00000000 <_EIP>:
Code;  c02518a3 <skb_drop_fraglist+17/3c>   <=====
   0:   8b 1b                     mov    (%ebx),%ebx   <=====
Code;  c02518a5 <skb_drop_fraglist+19/3c>
   2:   8b 42 74                  mov    0x74(%edx),%eax
Code;  c02518a8 <skb_drop_fraglist+1c/3c>
   5:   83 f8 01                  cmp    $0x1,%eax
Code;  c02518ab <skb_drop_fraglist+1f/3c>
   8:   74 0b                     je     15 <_EIP+0x15>
Code;  c02518ad <skb_drop_fraglist+21/3c>
   a:   f0 ff 4a 74               lock decl 0x74(%edx)
Code;  c02518b1 <skb_drop_fraglist+25/3c>
   e:   0f 94 c0                  sete   %al
Code;  c02518b4 <skb_drop_fraglist+28/3c>
  11:   84 c0                     test   %al,%al
Code;  c02518b6 <skb_drop_fraglist+2a/3c>
  13:   74 00                     je     15 <_EIP+0x15>

 <0>Kernel panic: Aiee, killing interrupt handler!


Full description:

I'm using Samba to distribute some shares to Windows clients. One of 
the shares is an Image-directory where I'm storing PQDI Images of 
Windows clients. One of the created images is about 40GB of size and 
is split up to 56 files each of same size. When verifying this image 
from a Win XP client, PQDI  stops with an error (error 1811, "Could 
not read from image file") and the Linux kernel panics. Verifying 
this image from DOS (with MS network client) is done without any 
error. Also verifying smaller images is done without any error. 
Another PQDI version (7.0) also reports an error and the Linux Kernel 

 panics. Copying more than 4 GB to the samba share also lets the 
kernel panic with an OOPS. Copying data locally from the Linux 
console is done without an error.

In the beginning I thought that the Promise controller is the source 
of problem, now I'm not sure. Maybe it's samba or the combination of 
samba and kernel version (--> skbuff.c?).  

The share is lying in a directory on a Reiser filesystem:  

share Images
ReiserFS
LVM (on /dev/md0 only, 120GB)
RAID1 /dev/md0 (120GB)
/dev/hda1 + /dev/hde1 (one primary partition of 120GB on each drive)
/dev/hda + /dev/hde (each 120GB) IDE UDMA133-controller  

As IDE-controller I first used a Promise FastTrak TX2000 (which 
supports "hardware"-RAID). I tried the binary Promise-driver 
(1.03.0.1) and the source code-driver (1.02.0.25), both without 
success. All time the OOPS occurred. Then I replaced the controller 
and both Samsung SP1203N-hard drives (each  120GB) against a Promise 
UltraTrak 133 TX2 and two Maxtor drives (6Y120P0, each 120GB) and 
installed a Linux native software-RAID without any Promise-driver. 
But again the OOPS occurred. Of course I updated the Promise-firmware 

to the latest level.  

To eliminate the RAID and LVM-drivers as the source of problem I 
installed just a Reiser FS on one 120GB-primary partition on one of 
both Maxtor disks (after removing the drive from the RAID). But again 

the Linux kernel panicked. Trying ext3 instead of reiserfs didn't 
help. As I do not have enough space on my scsi-disks I can't verify 
this big image from a scsi-disk.  

Sometimes the Linux kernel panic occurs immediately some minutes 
after starting the verify, sometimes it happens after reading half of 

all image files. Samba doesn't report any error. I also tried a 
different PCI-slot for the Promise-adapter without any success.

Environment:
# Dell Optiplex GX1 400MTbr+, Intel II 400 MHz, 320 MB RAM
# Adaptec AHA 2940UW as PCI-adapter with two hard drives (20GB and 
4GB, /boot is on the first scsi-drive) and a Plextor CD-writer
# onboard LAN (3com 3C905B)
# Promise Ultra133 TX2 as PCI-adapter with two Maxtor-drives (each 
120GB)
# DVD-ROM at the onboard-IDE 


--
Mit freundlichen Gruessen/Best regards,
Sebastian Piecha

EMail: spi@gmxpro.de

Re: oops in skbuff.c

[Sebastian Piecha]

On 16 Oct 2003 at 15:15, James Morris wrote:

> On Thu, 16 Oct 2003, Sebastian Piecha wrote:
> 
> >  panics. Copying more than 4 GB to the samba share also lets the 
> > kernel panic with an OOPS.
> 
> Does the above cause a panic for the 2.6-test kernels?
> Do you have netfilter enabled, and if so, any iptables modules loaded?
> 
> 
I did a mistake. iptables is installed but no rules added. I'll 
remove iptables and rerun the check.

--
Mit freundlichen Gruessen/Best regards,
Sebastian Piecha

EMail: spi@gmxpro.de

Re: oops in skbuff.c

[James Morris]

On Thu, 16 Oct 2003, Sebastian Piecha wrote:

> > Are you able to test with a different NIC to the onboard 3C905B?
> > 
> Unfortunately I only have 3com NICs. Do you think it's the adapter? 

Not sure, just trying to rule it out.


- James
-- 
James Morris
<jmorris@redhat.com>

Re: oops in skbuff.c

[Sebastian Piecha]

> > On Thu, 16 Oct 2003, Sebastian Piecha wrote:
> > 
> > >  panics. Copying more than 4 GB to the samba share also lets the 
> > > kernel panic with an OOPS.
> > 
> > Does the above cause a panic for the 2.6-test kernels?
> > Do you have netfilter enabled, and if so, any iptables modules loaded?
> > 
> > 
> I did a mistake. iptables is installed but no rules added. I'll 
> remove iptables and rerun the check.

For being sure I removed iptables but again the oops happened. I also 
tried kernel 2.4.18 without success.

Does anybody have a clue how to go on?

--
Mit freundlichen Gruessen/Best regards,
Sebastian Piecha

EMail: spi@gmxpro.de

Re: oops in skbuff.c

[Sebastian Piecha]

> > > On Thu, 16 Oct 2003, Sebastian Piecha wrote:
> > > 
> > > >  panics. Copying more than 4 GB to the samba share also lets the 
> > > > kernel panic with an OOPS.
> > > 
> > > Does the above cause a panic for the 2.6-test kernels?
> > > Do you have netfilter enabled, and if so, any iptables modules loaded?
> > > 
> > > 
> > I did a mistake. iptables is installed but no rules added. I'll 
> > remove iptables and rerun the check.
> 
> For being sure I removed iptables but again the oops happened. I also 
> tried kernel 2.4.18 without success.
> 
> Does anybody have a clue how to go on?

I reinstalled my system from scratch. Debian testing, kernel 2.4.22, 
samba 2.2.8a. Again the oops is occuring in skbuff.

With kernel 2.6.0-test7 I didn't get any oops. But I'd prefer using a 
stable 2.4.x kernel. There are modifications in skbuff.c in kernel 
2.6.0-test7.
For what reason skbuff.c has changed?
Due to bug fixes or general changes in the kernel?
Could skbuff.c easily be backported to 2.4.20 or 2.4.22?

And here's a copy of the last oops:

Oops: 0000
CPU:    0
EIP:    0010:[<c01c1402>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010282
eax: c44ca6e0   ebx: 00200000   ecx: c44ca6e0   edx: 00200000
esi: cf80ae20   edi: fffffffd   ebp: 00000046   esp: c029bf10
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0, stackpage=c029b000)
Stack: 00000030 cf80ae20 c01c14c9 cf80ae20 cf80ae20 00000030 c01c1513 
cf80ae20
       cf80ae20 cf80ae20 c01c1697 cf80ae20 00000000 00000030 cf8105c0 
cf80ae20
       c01c65e3 cf80ae20 0000000e 00000001 c02ba568 c0121b95 c02ba568 
00000002
Call Trace:    [<c01c14c9>] [<c01c1513>] [<c01c1697>] [<c01c65e3>] 
[<c0121b95>]
  [<c010a88b>] [<c0107080>] [<c010cde8>] [<c0107080>] [<c01070a3>] 
[<c0107132>]
  [<c0105000>]
Code: 8b 42 78 8b 1b 48 74 0a ff 4a 78 0f 94 c0 84 c0 74 08 89 14


>>EIP; c01c1402 <skb_drop_fraglist+22/50>   <=====

>>eax; c44ca6e0 <_end+41d3b18/14516498>
>>ecx; c44ca6e0 <_end+41d3b18/14516498>
>>esi; cf80ae20 <_end+f514258/14516498>
>>esp; c029bf10 <init_task_union+1f10/2000>

Trace; c01c14c9 <skb_release_data+69/a0>
Trace; c01c1513 <kfree_skbmem+13/70>
Trace; c01c1697 <__kfree_skb+127/1c0>
Trace; c01c65e3 <net_tx_action+43/c0>
Trace; c0121b95 <do_softirq+95/a0>
Trace; c010a88b <do_IRQ+9b/a0>
Trace; c0107080 <default_idle+0/40>
Trace; c010cde8 <call_do_IRQ+5/d>
Trace; c0107080 <default_idle+0/40>
Trace; c01070a3 <default_idle+23/40>
Trace; c0107132 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>

Code;  c01c1402 <skb_drop_fraglist+22/50>
00000000 <_EIP>:
Code;  c01c1402 <skb_drop_fraglist+22/50>   <=====
   0:   8b 42 78                  mov    0x78(%edx),%eax   <=====
Code;  c01c1405 <skb_drop_fraglist+25/50>
   3:   8b 1b                     mov    (%ebx),%ebx
Code;  c01c1407 <skb_drop_fraglist+27/50>
   5:   48                        dec    %eax
Code;  c01c1408 <skb_drop_fraglist+28/50>
   6:   74 0a                     je     12 <_EIP+0x12>
Code;  c01c140a <skb_drop_fraglist+2a/50>
   8:   ff 4a 78                  decl   0x78(%edx)
Code;  c01c140d <skb_drop_fraglist+2d/50>
   b:   0f 94 c0                  sete   %al
Code;  c01c1410 <skb_drop_fraglist+30/50>
   e:   84 c0                     test   %al,%al
Code;  c01c1412 <skb_drop_fraglist+32/50>
  10:   74 08                     je     1a <_EIP+0x1a>
Code;  c01c1414 <skb_drop_fraglist+34/50>
  12:   89 14 00                  mov    %edx,(%eax,%eax,1)

 <0>Kernel panic: Aiee, killing interrupt handler!



--
Mit freundlichen Gruessen/Best regards,
Sebastian Piecha

EMail: spi@gmxpro.de

Re: oops in skbuff.c

[Sebastian Piecha]

Everything works now. skbuff.c seems to be ok.

At the end it seems to be an issue between one of my three memory 
modules and the installed Promise ide controller. Without the Promise 
controller everythink works and with a changed memory stick also 
everything works. I did a lot of memtests but no error was shown. I 
think one of the memory modules has some electrical specialties when 
used together with the Promise controller.

Thanks a lot to everybody who tried to shed some light on this.

--
Mit freundlichen Gruessen/Best regards,
Sebastian Piecha

EMail: spi@gmxpro.de