* call_usermodehelper hang
@ 2004-04-06 17:03 Brian King
2004-04-16 8:52 ` Heiko Carstens
0 siblings, 1 reply; 13+ messages in thread
From: Brian King @ 2004-04-06 17:03 UTC (permalink / raw)
To: linux-scsi; +Cc: brking
I have been running into some kernel hangs due to call_usermodehelper. Looking
at the backtrace, it looks to me like there are deadlock issues with adding
devices from work queues. Attached is a sample backtrace from one of the
hangs I experienced. My question is why does call_usermodehelper do 2 different
things depending on whether or not it is called from the kevent task? It appears
that the simple way to fix the hang would be to never have call_usermodehelper
use a work_queue since it must be called from process context anyway, or
am I missing something?
0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0 swapper
SP(esp) PC(eip) Function(args)
0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace
0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4
0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138
0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104
0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4
0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134
0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70
0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60
0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290
0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94
0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c
0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110
0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38
0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28
0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c
0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c
0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830 events/0
SP(esp) PC(eip) Function(args)
0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace
0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4
0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c
0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c
0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0
0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c
0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04
0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90
0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60
0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268
0xc00000003faafee0 0xc0000000000839cc .kthread +0x160
0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
--
Brian King
eServer Storage I/O
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: call_usermodehelper hang
2004-04-06 17:03 call_usermodehelper hang Brian King
@ 2004-04-16 8:52 ` Heiko Carstens
0 siblings, 0 replies; 13+ messages in thread
From: Heiko Carstens @ 2004-04-16 8:52 UTC (permalink / raw)
To: rusty; +Cc: brking, linux-scsi, linux-scsi-owner, linux-kernel
> I have been running into some kernel hangs due to call_usermodehelper.
Looking
> at the backtrace, it looks to me like there are deadlock issues with
adding
> devices from work queues. Attached is a sample backtrace from one of the
> hangs I experienced. My question is why does call_usermodehelper do
> 2 different
> things depending on whether or not it is called from the kevent
> task? It appears
> that the simple way to fix the hang would be to never have
call_usermodehelper
> use a work_queue since it must be called from process context anyway, or
> am I missing something?
Do you have a solution for this problem? As it appears we have the very
same
problem with the zfcp lldd, since we also trigger a scsi_add_device call
by
using schedule_work. IMO the right way to solve this problem would be to
not
trigger any hotplug events while holding a semaphore.
> 0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0
swapper
> SP(esp) PC(eip) Function(args)
> 0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace
> 0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4
> 0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138
> 0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104
> 0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4
> 0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134
> 0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70
> 0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60
> 0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290
> 0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94
> 0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c
> 0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110
> 0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38
> 0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28
> 0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c
> 0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c
> 0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
>
> 0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830
events/0
> SP(esp) PC(eip) Function(args)
> 0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace
> 0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4
> 0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c
> 0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c
> 0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0
> 0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c
> 0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04
> 0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90
> 0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60
> 0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268
> 0xc00000003faafee0 0xc0000000000839cc .kthread +0x160
> 0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: call_usermodehelper hang
@ 2004-04-16 8:52 ` Heiko Carstens
0 siblings, 0 replies; 13+ messages in thread
From: Heiko Carstens @ 2004-04-16 8:52 UTC (permalink / raw)
To: Brian King, rusty; +Cc: brking, linux-scsi, linux-scsi-owner, linux-kernel
> I have been running into some kernel hangs due to call_usermodehelper.
Looking
> at the backtrace, it looks to me like there are deadlock issues with
adding
> devices from work queues. Attached is a sample backtrace from one of the
> hangs I experienced. My question is why does call_usermodehelper do
> 2 different
> things depending on whether or not it is called from the kevent
> task? It appears
> that the simple way to fix the hang would be to never have
call_usermodehelper
> use a work_queue since it must be called from process context anyway, or
> am I missing something?
Do you have a solution for this problem? As it appears we have the very
same
problem with the zfcp lldd, since we also trigger a scsi_add_device call
by
using schedule_work. IMO the right way to solve this problem would be to
not
trigger any hotplug events while holding a semaphore.
> 0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0
swapper
> SP(esp) PC(eip) Function(args)
> 0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace
> 0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4
> 0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138
> 0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104
> 0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4
> 0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134
> 0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70
> 0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60
> 0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290
> 0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94
> 0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c
> 0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110
> 0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38
> 0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28
> 0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c
> 0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c
> 0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
>
> 0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830
events/0
> SP(esp) PC(eip) Function(args)
> 0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace
> 0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4
> 0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c
> 0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c
> 0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0
> 0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c
> 0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04
> 0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90
> 0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60
> 0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268
> 0xc00000003faafee0 0xc0000000000839cc .kthread +0x160
> 0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: call_usermodehelper hang
2004-04-16 8:52 ` Heiko Carstens
(?)
@ 2004-04-16 14:06 ` Brian King
-1 siblings, 0 replies; 13+ messages in thread
From: Brian King @ 2004-04-16 14:06 UTC (permalink / raw)
To: Heiko Carstens; +Cc: rusty, linux-scsi, linux-scsi-owner, linux-kernel
The fix is in Andrew Morton's tree. The solution was to add a
call_usermodehelper_async, which can be called with semaphores held.
Grab the following patches:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.5/2.6.5-mm6/broken-out/call_usermodehelper_async.patch
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.5/2.6.5-mm6/broken-out/call_usermodehelper_async-always.patch
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.5/2.6.5-mm6/broken-out/kstrdup-and-friends.patch
You will get a compile error since the mm tree has another patch in it.
Change the offending (system_state != SYSTEM_RUNNING) to (!system_running)
-Brian
Heiko Carstens wrote:
>>I have been running into some kernel hangs due to call_usermodehelper.
>
> Looking
>
>>at the backtrace, it looks to me like there are deadlock issues with
>
> adding
>
>>devices from work queues. Attached is a sample backtrace from one of the
>>hangs I experienced. My question is why does call_usermodehelper do
>>2 different
>>things depending on whether or not it is called from the kevent
>>task? It appears
>>that the simple way to fix the hang would be to never have
>
> call_usermodehelper
>
>>use a work_queue since it must be called from process context anyway, or
>>am I missing something?
>
>
> Do you have a solution for this problem? As it appears we have the very
> same
> problem with the zfcp lldd, since we also trigger a scsi_add_device call
> by
> using schedule_work. IMO the right way to solve this problem would be to
> not
> trigger any hotplug events while holding a semaphore.
>
>
>>0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0
>
> swapper
>
>> SP(esp) PC(eip) Function(args)
>>0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace
>>0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4
>>0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138
>>0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104
>>0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4
>>0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134
>>0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70
>>0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60
>>0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290
>>0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94
>>0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c
>>0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110
>>0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38
>>0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28
>>0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c
>>0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c
>>0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
>>
>>0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830
>
> events/0
>
>> SP(esp) PC(eip) Function(args)
>>0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace
>>0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4
>>0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c
>>0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c
>>0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0
>>0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c
>>0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04
>>0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90
>>0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60
>>0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268
>>0xc00000003faafee0 0xc0000000000839cc .kthread +0x160
>>0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
>
>
>
--
Brian King
eServer Storage I/O
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 13+ messages in thread
* call_usermodehelper hang
@ 2004-04-06 18:11 Brian King
2004-04-07 0:29 ` Andrew Morton
2004-04-07 0:41 ` Chris Wright
0 siblings, 2 replies; 13+ messages in thread
From: Brian King @ 2004-04-06 18:11 UTC (permalink / raw)
To: linux-kernel
I have been running into some kernel hangs due to call_usermodehelper. Looking
at the backtrace, it looks to me like there are deadlock issues with adding
devices from work queues. Attached is a sample backtrace from one of the
hangs I experienced. My question is why does call_usermodehelper do 2 different
things depending on whether or not it is called from the kevent task? It appears
that the simple way to fix the hang would be to never have call_usermodehelper
use a work_queue since it must be called from process context anyway, or
am I missing something?
0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0 swapper
SP(esp) PC(eip) Function(args)
0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace
0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4
0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138
0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104
0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4
0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134
0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70
0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60
0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290
0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94
0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c
0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110
0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38
0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28
0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c
0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c
0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830 events/0
SP(esp) PC(eip) Function(args)
0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace
0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4
0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c
0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c
0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0
0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c
0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04
0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90
0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60
0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268
0xc00000003faafee0 0xc0000000000839cc .kthread +0x160
0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
--
Brian King
eServer Storage I/O
IBM Linux Technology Center
--
Brian King
eServer Storage I/O
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: call_usermodehelper hang
2004-04-06 18:11 Brian King
@ 2004-04-07 0:29 ` Andrew Morton
2004-04-07 6:11 ` Greg KH
2004-04-07 0:41 ` Chris Wright
1 sibling, 1 reply; 13+ messages in thread
From: Andrew Morton @ 2004-04-07 0:29 UTC (permalink / raw)
To: Brian King; +Cc: linux-kernel, Greg KH
Brian King <brking@us.ibm.com> wrote:
>
> I have been running into some kernel hangs due to call_usermodehelper. Looking
> at the backtrace, it looks to me like there are deadlock issues with adding
> devices from work queues. Attached is a sample backtrace from one of the
> hangs I experienced. My question is why does call_usermodehelper do 2 different
> things depending on whether or not it is called from the kevent task? It appears
> that the simple way to fix the hang would be to never have call_usermodehelper
> use a work_queue since it must be called from process context anyway, or
> am I missing something?
>
swapper is running call_usermodehelper() while holding
down_write(&bus->subsys.rwsem); via bus_add_driver().
Meanwhile, keventd is blocked on the same lock in bus_add_device().
I'd say that the bug lies in the kobject code - we should not call
call_usermodehelper() while holding any locks which keventd may ever
acquire.
> 0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0 swapper
> SP(esp) PC(eip) Function(args)
> 0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace
> 0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4
> 0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138
> 0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104
> 0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4
> 0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134
> 0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70
> 0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60
> 0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290
> 0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94
> 0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c
> 0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110
> 0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38
> 0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28
> 0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c
> 0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c
> 0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
>
> 0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830 events/0
> SP(esp) PC(eip) Function(args)
> 0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace
> 0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4
> 0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c
> 0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c
> 0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0
> 0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c
> 0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04
> 0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90
> 0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60
> 0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268
> 0xc00000003faafee0 0xc0000000000839cc .kthread +0x160
> 0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: call_usermodehelper hang
2004-04-07 0:29 ` Andrew Morton
@ 2004-04-07 6:11 ` Greg KH
2004-04-07 14:00 ` Brian King
0 siblings, 1 reply; 13+ messages in thread
From: Greg KH @ 2004-04-07 6:11 UTC (permalink / raw)
To: Andrew Morton; +Cc: Brian King, linux-kernel
On Tue, Apr 06, 2004 at 05:29:03PM -0700, Andrew Morton wrote:
> Brian King <brking@us.ibm.com> wrote:
> >
> > I have been running into some kernel hangs due to call_usermodehelper. Looking
> > at the backtrace, it looks to me like there are deadlock issues with adding
> > devices from work queues. Attached is a sample backtrace from one of the
> > hangs I experienced. My question is why does call_usermodehelper do 2 different
> > things depending on whether or not it is called from the kevent task? It appears
> > that the simple way to fix the hang would be to never have call_usermodehelper
> > use a work_queue since it must be called from process context anyway, or
> > am I missing something?
> >
>
> swapper is running call_usermodehelper() while holding
> down_write(&bus->subsys.rwsem); via bus_add_driver().
>
> Meanwhile, keventd is blocked on the same lock in bus_add_device().
>
> I'd say that the bug lies in the kobject code - we should not call
> call_usermodehelper() while holding any locks which keventd may ever
> acquire.
How is keventd calling sysfs code? Is scsi using it to drive device
detection somehow? I don't see how the kobject core code itself can do
this on its own.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: call_usermodehelper hang
2004-04-07 6:11 ` Greg KH
@ 2004-04-07 14:00 ` Brian King
0 siblings, 0 replies; 13+ messages in thread
From: Brian King @ 2004-04-07 14:00 UTC (permalink / raw)
To: Greg KH; +Cc: Andrew Morton, linux-kernel
Greg KH wrote:
> On Tue, Apr 06, 2004 at 05:29:03PM -0700, Andrew Morton wrote:
>
>>Brian King <brking@us.ibm.com> wrote:
>>
>>>I have been running into some kernel hangs due to call_usermodehelper. Looking
>>>at the backtrace, it looks to me like there are deadlock issues with adding
>>>devices from work queues. Attached is a sample backtrace from one of the
>>>hangs I experienced. My question is why does call_usermodehelper do 2 different
>>>things depending on whether or not it is called from the kevent task? It appears
>>>that the simple way to fix the hang would be to never have call_usermodehelper
>>>use a work_queue since it must be called from process context anyway, or
>>>am I missing something?
>>>
>>
>>swapper is running call_usermodehelper() while holding
>>down_write(&bus->subsys.rwsem); via bus_add_driver().
>>
>>Meanwhile, keventd is blocked on the same lock in bus_add_device().
>>
>>I'd say that the bug lies in the kobject code - we should not call
>>call_usermodehelper() while holding any locks which keventd may ever
>>acquire.
>
>
> How is keventd calling sysfs code? Is scsi using it to drive device
> detection somehow? I don't see how the kobject core code itself can do
> this on its own.
scsi_add_device is being called from keventd by the ipr LLD. I am assuming this
is a reasonable thing to do. If not, then I would argue the scsi_add_device API
is broken as it must be called from task level, and if a LLD cannot use keventd,
then every LLD that wants to use it would need to create its own workqueue thread.
--
Brian King
eServer Storage I/O
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: call_usermodehelper hang
2004-04-06 18:11 Brian King
2004-04-07 0:29 ` Andrew Morton
@ 2004-04-07 0:41 ` Chris Wright
2004-04-07 1:46 ` Brian King
1 sibling, 1 reply; 13+ messages in thread
From: Chris Wright @ 2004-04-07 0:41 UTC (permalink / raw)
To: Brian King; +Cc: linux-kernel
* Brian King (brking@us.ibm.com) wrote:
> I have been running into some kernel hangs due to call_usermodehelper. Looking
> at the backtrace, it looks to me like there are deadlock issues with adding
> devices from work queues. Attached is a sample backtrace from one of the
> hangs I experienced. My question is why does call_usermodehelper do 2 different
> things depending on whether or not it is called from the kevent task? It appears
> that the simple way to fix the hang would be to never have call_usermodehelper
> use a work_queue since it must be called from process context anyway, or
> am I missing something?
It does two different things because it's trying to run from keventd.
In the case that current is not keventd, it schedules the work, so
keventd will pick that work up later to run in it's process context.
How early is this hang? It looks like init thread adds work and waits
for it's completion while holding a semaphore. It is never woken up by
keventd which is sleeping waiting for wakeup from semaphore that init
thread took.
Seems troubling to hold the sem while calling call_usermodehelper, as that
could go off for a long time.
swapper events/0
------- --------
bus_add_driver() ipr_worker_thread()
down_write(&bus->subsys.rwsem) /* here */ scsi_add_device()
driver_attach() bus_add_device()
... /* sem is already taken */
sd_probe() down_write(&bus->subsys.rwsem)
add_disk() schedule() /* bye! */
...
call_usermodehelper()
wait_for_completion() /* never awoken by keventd */
thanks,
-chris
--
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: call_usermodehelper hang
2004-04-07 0:41 ` Chris Wright
@ 2004-04-07 1:46 ` Brian King
0 siblings, 0 replies; 13+ messages in thread
From: Brian King @ 2004-04-07 1:46 UTC (permalink / raw)
To: Chris Wright; +Cc: linux-kernel
Chris Wright wrote:
> * Brian King (brking@us.ibm.com) wrote:
>
>>I have been running into some kernel hangs due to call_usermodehelper. Looking
>>at the backtrace, it looks to me like there are deadlock issues with adding
>>devices from work queues. Attached is a sample backtrace from one of the
>>hangs I experienced. My question is why does call_usermodehelper do 2 different
>>things depending on whether or not it is called from the kevent task? It appears
>>that the simple way to fix the hang would be to never have call_usermodehelper
>>use a work_queue since it must be called from process context anyway, or
>>am I missing something?
>
>
> It does two different things because it's trying to run from keventd.
> In the case that current is not keventd, it schedules the work, so
> keventd will pick that work up later to run in it's process context.
>
> How early is this hang?
Pretty early. Boot time, loading scsi drivers. While initializing the
third scsi adapter on the system a device shows up dynamically on the
first adapter, the LLD schedules work to call scsi_add_device and we
end up with the hang.
It looks like init thread adds work and waits
> for it's completion while holding a semaphore. It is never woken up by
> keventd which is sleeping waiting for wakeup from semaphore that init
> thread took.
>
> Seems troubling to hold the sem while calling call_usermodehelper, as that
> could go off for a long time.
I agree. There is another similar hang I ran into recently with the scsi
core in that I was calling scsi_add_device from keventd, which ended up
sleeping on the scan_mutex since the module load process was calling
scsi_scan_host. scsi_scan_host was in the same kobject hotplug code,
calling call_usermodehelper. I figured my LLD shouldn't be doing that,
so I synchronized the events. Perhaps that was the wrong fix... I
suppose I could have changed the LLD to always call
scsi_scan_host/scsi_add_device/etc. from keventd...
-Brian
^ permalink raw reply [flat|nested] 13+ messages in thread
* call_usermodehelper hang
@ 2004-04-06 21:58 Brian King
0 siblings, 0 replies; 13+ messages in thread
From: Brian King @ 2004-04-06 21:58 UTC (permalink / raw)
To: linux-hotplug
I just posted this to linux-kernel and it was suggested that I post here as well.
I have been running into some kernel hangs due to call_usermodehelper. Looking
at the backtrace, it looks to me like there are deadlock issues with adding
devices from work queues. Attached is a sample backtrace from one of the
hangs I experienced. My question is why does call_usermodehelper do 2 different
things depending on whether or not it is called from the kevent task? It appears
that the simple way to fix the hang would be to never have call_usermodehelper
use a work_queue since it must be called from process context anyway, or
am I missing something?
0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0 swapper
SP(esp) PC(eip) Function(args)
0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace
0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4
0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138
0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104
0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4
0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134
0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70
0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60
0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290
0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94
0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c
0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110
0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38
0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28
0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c
0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c
0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830 events/0
SP(esp) PC(eip) Function(args)
0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace
0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4
0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c
0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c
0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0
0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c
0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04
0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90
0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60
0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268
0xc00000003faafee0 0xc0000000000839cc .kthread +0x160
0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
--
Brian King
eServer Storage I/O
IBM Linux Technology Center
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 13+ messages in thread* call_usermodehelper hang
@ 2005-02-25 17:17 Payasam Manohar
2005-02-25 18:06 ` Lee Revell
0 siblings, 1 reply; 13+ messages in thread
From: Payasam Manohar @ 2005-02-25 17:17 UTC (permalink / raw)
To: linux-kernel
hai all,
I want to call a user program(script) from linux kernel.
I am using Redhat linux 9( kernel version 2.4.20-8). With the help of
call_usermodehelper I am calling the user level program from one of the
kernel driver. I am setting the parameters correctly.
int call_usermodehelper(char *path, char *argv, char *envp);
The system is hanging after giving a call trace and the error
Code:
<0> Kernel Panic : Aiee,Killing interrupt handler
In interrupt handler- not syncing.
Any help is welcome.
Thanks&Regards,
P.Manohar.
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: call_usermodehelper hang
2005-02-25 17:17 Payasam Manohar
@ 2005-02-25 18:06 ` Lee Revell
0 siblings, 0 replies; 13+ messages in thread
From: Lee Revell @ 2005-02-25 18:06 UTC (permalink / raw)
To: Payasam Manohar; +Cc: linux-kernel
On Fri, 2005-02-25 at 22:47 +0530, Payasam Manohar wrote:
> hai all,
> I want to call a user program(script) from linux kernel.
> I am using Redhat linux 9( kernel version 2.4.20-8). With the help of
> call_usermodehelper I am calling the user level program from one of the
> kernel driver. I am setting the parameters correctly.
> int call_usermodehelper(char *path, char *argv, char *envp);
>
> The system is hanging after giving a call trace and the error
> Code:
> <0> Kernel Panic : Aiee,Killing interrupt handler
> In interrupt handler- not syncing.
>
> Any help is welcome.
I don't think you can do that from interrupt context.
Lee
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2005-02-25 18:06 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-04-06 17:03 call_usermodehelper hang Brian King
2004-04-16 8:52 ` Heiko Carstens
2004-04-16 8:52 ` Heiko Carstens
2004-04-16 14:06 ` Brian King
-- strict thread matches above, loose matches on Subject: below --
2004-04-06 18:11 Brian King
2004-04-07 0:29 ` Andrew Morton
2004-04-07 6:11 ` Greg KH
2004-04-07 14:00 ` Brian King
2004-04-07 0:41 ` Chris Wright
2004-04-07 1:46 ` Brian King
2004-04-06 21:58 Brian King
2005-02-25 17:17 Payasam Manohar
2005-02-25 18:06 ` Lee Revell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.