sym53c8xx: WARNING on module unload, crashes on further usage

sym53c8xx: WARNING on module unload, crashes on further usage

[Nikola Ciprich]

Hi,
we're experiencing problems using changer connected on LSI Logic / Symbios Logic 53c875 adaptor (using sym53c8xx module).
Sometimes changer gets stuck, and reloading sym53c8xx module is the only way to help. But it alwas issues following warning,
and on subsequent mtx usage, the whole machine usually crashes...

the warning issued on module unload:

WARNING: at /home/src/redhat/BUILD/linux-2.6.29/arch/x86/include/asm/dma-mapping.h:297 ___free_dma_mem_cluster+0xba/0xc0 [sym53c8xx]()
Hardware name: PDSM4+
Modules linked in: ipmi_si ipmi_devintf ipmi_msghandler nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs ipv6 reiserfs dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod sbs sbshc fan battery ac parport_pc lp parport nvram st osst joydev sg usbhid container piix pata_acpi ide_pci_generic sym53c8xx(-) scsi_transport_spi arcmsr thermal processor thermal_sys buttone1000e rng_core i2c_i801 ata_piix i3000_edac iTCO_wdt ata_generic i2c_core shpchp edac_core pci_hotplug pcspkr rtc ide_gd_mod ide_core ahci libata sd_mod scsi_mod raid456 async_xor async_memcpy async_tx xor raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_tabl
e]

Pid: 5495, comm: rmmod Not tainted 2.6.29lb.00_01_PRE09 #1
Call Trace:
 [<c022e7d6>] warn_slowpath+0x86/0xa0
 [<c02e312f>] ? free_proc_entry+0x5f/0x80
 [<c02e312f>] ? free_proc_entry+0x5f/0x80
 [<f80ae38a>] ___free_dma_mem_cluster+0xba/0xc0 [sym53c8xx]
 [<f80ae231>] __sym_mfree+0x91/0xe0 [sym53c8xx]
 [<c046d82c>] ? _spin_lock_irqsave+0x4c/0x60
 [<f80ae741>] __sym_mfree_dma+0x51/0x80 [sym53c8xx]
 [<f80a7851>] sym_hcb_free+0x51/0x190 [sym53c8xx]
 [<f80a5659>] sym_free_resources+0x39/0x80 [sym53c8xx]
 [<c03352b9>] ? __delay+0x9/0x10
 [<f80a5735>] sym_detach+0x95/0xb0 [sym53c8xx]
 [<f80a5775>] sym2_remove+0x25/0x50 [sym53c8xx]
 [<c0340c99>] pci_device_remove+0x19/0x40
 [<c03954e9>] __device_release_driver+0x49/0x80
 [<c03955b7>] driver_detach+0x97/0xa0
 [<c03948b3>] bus_remove_driver+0x73/0x90
 [<c0395a5f>] driver_unregister+0x2f/0x40
 [<c0340ef1>] pci_unregister_driver+0x21/0x80
 [<f80af0dd>] sym2_exit+0xd/0x20 [sym53c8xx]
 [<c0258b06>] sys_delete_module+0x126/0x1f0
 [<c021aa76>] ? do_page_fault+0x1f6/0x630
 [<c0268da0>] ? audit_syscall_exit+0x310/0x380
 [<c028d699>] ? sys_munmap+0x39/0x50
 [<c0203415>] sysenter_do_call+0x12/0x2c

We're having those problems on 2.6.29, but it was happening also on older kernels (2.6.26, ...)
Could somebody have a look on this?
Should more information be needed, I'll gladly provide it.
thanks a lot in advance.
best regards
nik

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: sym53c8xx: WARNING on module unload, crashes on further usage

[Matthew Wilcox]

On Thu, Apr 02, 2009 at 09:54:04AM +0200, Nikola Ciprich wrote:
> Hi,
> we're experiencing problems using changer connected on LSI Logic / Symbios Logic 53c875 adaptor (using sym53c8xx module).
> Sometimes changer gets stuck, and reloading sym53c8xx module is the only way to help. But it alwas issues following warning,
> and on subsequent mtx usage, the whole machine usually crashes...

The warning is actually harmless.  What error messages do you get when
the machine crashes?

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: sym53c8xx: WARNING on module unload, crashes on further usage

[Nikola Ciprich]

Hi Matthew,
well, I've got VERY crappy screen photo, my collegue made using his cellphone, You can check
it on http://nik.lbox.cz/downloads/sym53c8xx-crash.jpg
But we can quite easily reproduce the problem, I'll try to do it today evening and make a
better screenshot...
I'll report further then.
Regarding warning, couldn't it then be surpressed, if it's harmless? Kernel then gets tainted,
which makes me a bit nervous :-)
regards
nik

On Thu, Apr 02, 2009 at 05:23:38AM -0600, Matthew Wilcox wrote:
> On Thu, Apr 02, 2009 at 09:54:04AM +0200, Nikola Ciprich wrote:
> > Hi,
> > we're experiencing problems using changer connected on LSI Logic / Symbios Logic 53c875 adaptor (using sym53c8xx module).
> > Sometimes changer gets stuck, and reloading sym53c8xx module is the only way to help. But it alwas issues following warning,
> > and on subsequent mtx usage, the whole machine usually crashes...
> 
> The warning is actually harmless.  What error messages do you get when
> the machine crashes?
> 
> -- 
> Matthew Wilcox				Intel Open Source Technology Centre
> "Bill, look, we understand that you're interested in selling us this
> operating system, but compare it to ours.  We can't possibly take such
> a retrograde step."
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: sym53c8xx: WARNING on module unload, crashes on further usage

[Nikola Ciprich]

Hmm,
maybe a bit usefull can be begin of crash message, he got on console:

databox [488754.946366] BUG: spinlock recursion on CPU#1, swapper/0
databox [488754.950343]  lock: f70ddbe0, .magic: dead4ead, .owner: swapper/0, .owner_cpu: 1

n.


On Thu, Apr 02, 2009 at 02:26:10PM +0200, Nikola Ciprich wrote:
> Hi Matthew,
> well, I've got VERY crappy screen photo, my collegue made using his cellphone, You can check
> it on http://nik.lbox.cz/downloads/sym53c8xx-crash.jpg
> But we can quite easily reproduce the problem, I'll try to do it today evening and make a
> better screenshot...
> I'll report further then.
> Regarding warning, couldn't it then be surpressed, if it's harmless? Kernel then gets tainted,
> which makes me a bit nervous :-)
> regards
> nik
> 
> On Thu, Apr 02, 2009 at 05:23:38AM -0600, Matthew Wilcox wrote:
> > On Thu, Apr 02, 2009 at 09:54:04AM +0200, Nikola Ciprich wrote:
> > > Hi,
> > > we're experiencing problems using changer connected on LSI Logic / Symbios Logic 53c875 adaptor (using sym53c8xx module).
> > > Sometimes changer gets stuck, and reloading sym53c8xx module is the only way to help. But it alwas issues following warning,
> > > and on subsequent mtx usage, the whole machine usually crashes...
> > 
> > The warning is actually harmless.  What error messages do you get when
> > the machine crashes?
> > 
> > -- 
> > Matthew Wilcox				Intel Open Source Technology Centre
> > "Bill, look, we understand that you're interested in selling us this
> > operating system, but compare it to ours.  We can't possibly take such
> > a retrograde step."
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> -- 
> -------------------------------------
> Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
> 
> tel.:   +420 596 603 142
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@linuxbox.cz
> -------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

crash unrelated to sym53c8xx

[Matthew Wilcox]

On Thu, Apr 02, 2009 at 02:28:08PM +0200, Nikola Ciprich wrote:
> Hmm,
> maybe a bit usefull can be begin of crash message, he got on console:
> 
> databox [488754.946366] BUG: spinlock recursion on CPU#1, swapper/0
> databox [488754.950343]  lock: f70ddbe0, .magic: dead4ead, .owner: swapper/0, .owner_cpu: 1

Yes, that's somewhat more useful ... the tail of the stack trace in the
photo indicates this isn't (at least directly) a bug in the sym53c8xx
driver.  Can you get the rest of the crash message?  netconsole might
help here if you don't have a serial console.

> 
> On Thu, Apr 02, 2009 at 02:26:10PM +0200, Nikola Ciprich wrote:
> > Hi Matthew,
> > well, I've got VERY crappy screen photo, my collegue made using his cellphone, You can check
> > it on http://nik.lbox.cz/downloads/sym53c8xx-crash.jpg
> > But we can quite easily reproduce the problem, I'll try to do it today evening and make a
> > better screenshot...
> > I'll report further then.
> > Regarding warning, couldn't it then be surpressed, if it's harmless? Kernel then gets tainted,
> > which makes me a bit nervous :-)

There's a few reasons this warning hasn't been fixed yet:

 - It only appears on module unload.  Most people don't unload sym53c8xx.
 - The problem that it's warning about is only a problem on ARM.
   There's not a huge userbase of ARM users with sym53c8xx cards.
 - Fixing it requires a fairly large rewrite of how sym2 manages its
   memory.  I've done about half the work required, but I've basically
   run out of interest.

> > regards
> > nik
> > 
> > On Thu, Apr 02, 2009 at 05:23:38AM -0600, Matthew Wilcox wrote:
> > > On Thu, Apr 02, 2009 at 09:54:04AM +0200, Nikola Ciprich wrote:
> > > > Hi,
> > > > we're experiencing problems using changer connected on LSI Logic / Symbios Logic 53c875 adaptor (using sym53c8xx module).
> > > > Sometimes changer gets stuck, and reloading sym53c8xx module is the only way to help. But it alwas issues following warning,
> > > > and on subsequent mtx usage, the whole machine usually crashes...
> > > 
> > > The warning is actually harmless.  What error messages do you get when
> > > the machine crashes?
> > > 
> > > -- 
> > > Matthew Wilcox				Intel Open Source Technology Centre
> > > "Bill, look, we understand that you're interested in selling us this
> > > operating system, but compare it to ours.  We can't possibly take such
> > > a retrograde step."
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > -- 
> > -------------------------------------
> > Nikola CIPRICH
> > LinuxBox.cz, s.r.o.
> > 28. rijna 168, 709 01 Ostrava
> > 
> > tel.:   +420 596 603 142
> > fax:    +420 596 621 273
> > mobil:  +420 777 093 799
> > www.linuxbox.cz
> > 
> > mobil servis: +420 737 238 656
> > email servis: servis@linuxbox.cz
> > -------------------------------------
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> -- 
> -------------------------------------
> Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
> 
> tel.:   +420 596 603 142
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@linuxbox.cz
> -------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: crash unrelated to sym53c8xx

[Nikola Ciprich]

> Yes, that's somewhat more useful ... the tail of the stack trace in the
> photo indicates this isn't (at least directly) a bug in the sym53c8xx
> driver.  Can you get the rest of the crash message?  netconsole might
> help here if you don't have a serial console.
OK, I'll try to get the whole message using IPMI console.
I'll post it in ~2 hours...

> There's a few reasons this warning hasn't been fixed yet:
> 
>  - It only appears on module unload.  Most people don't unload sym53c8xx.
>  - The problem that it's warning about is only a problem on ARM.
>    There's not a huge userbase of ARM users with sym53c8xx cards.
>  - Fixing it requires a fairly large rewrite of how sym2 manages its
>    memory.  I've done about half the work required, but I've basically
>    run out of interest.
Hmm, maybe I don't understand what You mean by ARM, but it's i686 machine,
not arm (actually it's opteron, but running i686 OS).
but if it's reallu harmless, I don't have that big problem with it. only the crashes
really worry me :)

Re: crash unrelated to sym53c8xx

[Matthew Wilcox]

On Thu, Apr 02, 2009 at 02:59:54PM +0200, Nikola Ciprich wrote:
> > There's a few reasons this warning hasn't been fixed yet:
> > 
> >  - It only appears on module unload.  Most people don't unload sym53c8xx.
> >  - The problem that it's warning about is only a problem on ARM.
> >    There's not a huge userbase of ARM users with sym53c8xx cards.
> >  - Fixing it requires a fairly large rewrite of how sym2 manages its
> >    memory.  I've done about half the work required, but I've basically
> >    run out of interest.
> Hmm, maybe I don't understand what You mean by ARM, but it's i686 machine,
> not arm (actually it's opteron, but running i686 OS).
> but if it's reallu harmless, I don't have that big problem with it. only the crashes
> really worry me :)

The warning is there on x86 to warn us that we're doing something that
will cause problems on ARM.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: crash unrelated to sym53c8xx

[Nikola Ciprich]

OK, now I understand, thanks for explaining...
unfortunately I'm now unable to reproduce the crash including backtrace, machine always
hangs completely without any error reported :(
I've setup netconsole logging and I'll wait if it crashes and report something during normal
usage.
I'll report if it happens.
thanks for Your time.
have a nice day
nik

On Thu, Apr 02, 2009 at 07:01:56AM -0600, Matthew Wilcox wrote:
> On Thu, Apr 02, 2009 at 02:59:54PM +0200, Nikola Ciprich wrote:
> > > There's a few reasons this warning hasn't been fixed yet:
> > > 
> > >  - It only appears on module unload.  Most people don't unload sym53c8xx.
> > >  - The problem that it's warning about is only a problem on ARM.
> > >    There's not a huge userbase of ARM users with sym53c8xx cards.
> > >  - Fixing it requires a fairly large rewrite of how sym2 manages its
> > >    memory.  I've done about half the work required, but I've basically
> > >    run out of interest.
> > Hmm, maybe I don't understand what You mean by ARM, but it's i686 machine,
> > not arm (actually it's opteron, but running i686 OS).
> > but if it's reallu harmless, I don't have that big problem with it. only the crashes
> > really worry me :)
> 
> The warning is there on x86 to warn us that we're doing something that
> will cause problems on ARM.
> 
> -- 
> Matthew Wilcox				Intel Open Source Technology Centre
> "Bill, look, we understand that you're interested in selling us this
> operating system, but compare it to ours.  We can't possibly take such
> a retrograde step."
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: crash unrelated to sym53c8xx

[Nikola Ciprich]

Hi Matthew,
finally we caught the crash even with backtrace, here it goes:

[65468.555522] BUG: spinlock lockup on CPU#1, scsi_eh_9/5912, ec5625e0
[65468.555522] Pid: 5912, comm: scsi_eh_9 Tainted: G        W  2.6.28lb.10 #1
[65468.555522] Call Trace:
[65468.555522]  [<c0333654>] _raw_spin_lock+0x144/0x180
[65468.555522]  [<c046178c>] _spin_lock_irqsave+0x4c/0x60
[65468.555522]  [<c031d111>] ? blk_put_request+0x21/0x50
[65468.555522]  [<c031d111>] blk_put_request+0x21/0x50
[65468.555522]  [<f827163c>] sg_finish_rem_req+0x9c/0xf0 [sg]
[65468.555522]  [<f8272589>] sg_rq_end_io+0x1a9/0x2b0 [sg]
[65468.555522]  [<c0461afc>] ? _spin_unlock_irqrestore+0x2c/0x50
[65468.555522]  [<c031cda0>] end_that_request_last+0x50/0x230
[65468.555522]  [<c046178c>] ? _spin_lock_irqsave+0x4c/0x60
[65468.555522]  [<c031d05e>] blk_end_io+0x5e/0x90
[65468.555522]  [<c031d0ea>] blk_end_request+0x1a/0x20
[65468.555522]  [<f81a9305>] scsi_end_request+0x25/0x90 [scsi_mod]
[65468.555522]  [<f81aa072>] scsi_io_completion+0x92/0x420 [scsi_mod]
[65468.555522]  [<f81a973b>] ? scsi_device_unbusy+0x8b/0xb0 [scsi_mod]
[65468.555522]  [<f81a3538>] scsi_finish_command+0x98/0x100 [scsi_mod]
[65468.555522]  [<f81a790c>] ? scsi_eh_ready_devs+0x54c/0x720 [scsi_mod]
[65468.555522]  [<f81a6eba>] scsi_eh_flush_done_q+0x4a/0x100 [scsi_mod]
[65468.555522]  [<f81a815d>] scsi_error_handler+0x31d/0x520 [scsi_mod]
[65468.555522]  [<c0461afc>] ? _spin_unlock_irqrestore+0x2c/0x50
[65468.555522]  [<c021eda9>] ? complete+0x49/0x60
[65468.555522]  [<f81a7e40>] ? scsi_error_handler+0x0/0x520 [scsi_mod]
[65468.555522]  [<c023fda2>] kthread+0x42/0x70
[65468.555522]  [<c023fd60>] ? kthread+0x0/0x70
[65468.555522]  [<c02041c7>] kernel_thread_helper+0x7/0x10

Will it help?
BR
nik
x
-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: crash unrelated to sym53c8xx

[FUJITA Tomonori]

On Wed, 8 Apr 2009 10:30:02 +0200
Nikola Ciprich <extmaillist@linuxbox.cz> wrote:

> Hi Matthew,
> finally we caught the crash even with backtrace, here it goes:
> 
> [65468.555522] BUG: spinlock lockup on CPU#1, scsi_eh_9/5912, ec5625e0
> [65468.555522] Pid: 5912, comm: scsi_eh_9 Tainted: G        W  2.6.28lb.10 #1
> [65468.555522] Call Trace:
> [65468.555522]  [<c0333654>] _raw_spin_lock+0x144/0x180
> [65468.555522]  [<c046178c>] _spin_lock_irqsave+0x4c/0x60
> [65468.555522]  [<c031d111>] ? blk_put_request+0x21/0x50
> [65468.555522]  [<c031d111>] blk_put_request+0x21/0x50
> [65468.555522]  [<f827163c>] sg_finish_rem_req+0x9c/0xf0 [sg]
> [65468.555522]  [<f8272589>] sg_rq_end_io+0x1a9/0x2b0 [sg]
> [65468.555522]  [<c0461afc>] ? _spin_unlock_irqrestore+0x2c/0x50
> [65468.555522]  [<c031cda0>] end_that_request_last+0x50/0x230
> [65468.555522]  [<c046178c>] ? _spin_lock_irqsave+0x4c/0x60
> [65468.555522]  [<c031d05e>] blk_end_io+0x5e/0x90
> [65468.555522]  [<c031d0ea>] blk_end_request+0x1a/0x20
> [65468.555522]  [<f81a9305>] scsi_end_request+0x25/0x90 [scsi_mod]
> [65468.555522]  [<f81aa072>] scsi_io_completion+0x92/0x420 [scsi_mod]
> [65468.555522]  [<f81a973b>] ? scsi_device_unbusy+0x8b/0xb0 [scsi_mod]
> [65468.555522]  [<f81a3538>] scsi_finish_command+0x98/0x100 [scsi_mod]
> [65468.555522]  [<f81a790c>] ? scsi_eh_ready_devs+0x54c/0x720 [scsi_mod]
> [65468.555522]  [<f81a6eba>] scsi_eh_flush_done_q+0x4a/0x100 [scsi_mod]
> [65468.555522]  [<f81a815d>] scsi_error_handler+0x31d/0x520 [scsi_mod]
> [65468.555522]  [<c0461afc>] ? _spin_unlock_irqrestore+0x2c/0x50
> [65468.555522]  [<c021eda9>] ? complete+0x49/0x60
> [65468.555522]  [<f81a7e40>] ? scsi_error_handler+0x0/0x520 [scsi_mod]
> [65468.555522]  [<c023fda2>] kthread+0x42/0x70
> [65468.555522]  [<c023fd60>] ? kthread+0x0/0x70
> [65468.555522]  [<c02041c7>] kernel_thread_helper+0x7/0x10
> 
> Will it help?

This looks like a sg bug.

2.6.30-rc1 includes the following commit to fix this bug:

commit 015640edb1f346e0b2eda703587c4cd1c310ec1d
Author: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Date:   Fri Apr 3 19:28:06 2009 +0900

    [SCSI] sg: fix q->queue_lock on scsi_error_handler path


It's tagged 'Cc: stable@kernel.org' so it will be merged into stable
trees too.