* pci: kernel crash in bus_find_device
@ 2014-05-20 19:17 Francesco Ruggeri
2014-05-20 19:50 ` Guenter Roeck
0 siblings, 1 reply; 5+ messages in thread
From: Francesco Ruggeri @ 2014-05-20 19:17 UTC (permalink / raw)
To: linux-pci, linux-kernel; +Cc: Francesco Ruggeri
I posted this about a week ago but I did not get any replies.
Re-trying.
While traversing devices on pci_bus_type I ran into the crash below.
The immediate cause of the crash is that bus_find_device is trying to resume
a scan starting from a device that has been unregistered (and whose knode_bus
has already been klist_del' ed).
The main issue seems to be that when resuming a scan the caller should
be holding a
reference to the klist_node, but instead it relies on holding a
reference to the device.
I played with a couple of narrow fixes, but a clean solution would
affect quite a bit of code.
Has anybody run into this before?
Thanks,
Francesco Ruggeri
------------[ cut here ]------------
WARNING: at /bld/EosKernel/Artools-rpmbuild/linux-3.4/include/linux/kref.h:41
klist_iter_init_node+0x30/0x38()
Modules linked in: pci_scan(O) sch_prio sand_dma(PO) arista_bde(PO)
macvlan ip6table_mangle iptable_mangle msr nf_conntrack_ipv6
nf_defrag_ipv6 ip6t_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG
xt_limit ipt_REJECT xt_hl xt_state xt_multiport xt_tcpudp kbfd(O)
8021q garp stp llc tun scd_em_driver(O) nf_conntrack_tftp iptable_raw
iptable_filter ip_tables xt_NOTRACK nf_conntrack xt_mark ip6table_raw
ip6table_filter ip6_tables x_tables scd(O) k8temp amd64_edac_mod hwmon
kvm_amd kvm
Pid: 6861, comm: pci_scan_0 Tainted: P O
3.4.43.Ar-1797671.flbocafruggeri #1
Call Trace:
[<ffffffff81029dc4>] warn_slowpath_common+0x80/0x98
[<ffffffff811b57f1>] ? pci_do_find_bus+0x49/0x49
[<ffffffff81029df1>] warn_slowpath_null+0x15/0x17
[<ffffffff813a43ce>] klist_iter_init_node+0x30/0x38
[<ffffffff8120e57e>] bus_find_device+0x48/0x90
[<ffffffff811b5908>] pci_get_dev_by_id+0x5e/0x81
[<ffffffff811b5a6a>] pci_get_subsys+0x5c/0x7f
[<ffffffff811b5a9e>] pci_get_device+0x11/0x13
[<ffffffffa00b2087>] pci_scan+0x39/0x8a [pci_scan]
[<ffffffffa00b204e>] ? init_module+0x3c/0x3c [pci_scan]
[<ffffffff81040e6e>] kthread+0x84/0x8c
[<ffffffff813c8b14>] kernel_thread_helper+0x4/0x10
[<ffffffff81040dea>] ? __init_kthread_worker+0x37/0x37
[<ffffffff813c8b10>] ? gs_change+0xb/0xb
---[ end trace 79cea1ec476672fe ]---
------------[ cut here ]------------
WARNING: at /bld/EosKernel/Artools-rpmbuild/linux-3.4/lib/klist.c:189
klist_release+0x2b/0xeb()
Modules linked in: pci_scan(O) sch_prio sand_dma(PO) arista_bde(PO)
macvlan ip6table_mangle iptable_mangle msr nf_conntrack_ipv6
nf_defrag_ipv6 ip6t_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG
xt_limit ipt_REJECT xt_hl xt_state xt_multiport xt_tcpudp kbfd(O)
8021q garp stp llc tun scd_em_driver(O) nf_conntrack_tftp iptable_raw
iptable_filter ip_tables xt_NOTRACK nf_conntrack xt_mark ip6table_raw
ip6table_filter ip6_tables x_tables scd(O) k8temp amd64_edac_mod hwmon
kvm_amd kvm
Pid: 6861, comm: pci_scan_0 Tainted: P W O
3.4.43.Ar-1797671.flbocafruggeri #1
Call Trace:
[<ffffffff81029dc4>] warn_slowpath_common+0x80/0x98
[<ffffffff8120de13>] ? bus_get_device_klist+0x10/0x10
[<ffffffff81029df1>] warn_slowpath_null+0x15/0x17
[<ffffffff813a440e>] klist_release+0x2b/0xeb
[<ffffffff813a44ec>] klist_dec_and_del+0x1e/0x25
[<ffffffff813a4528>] klist_next+0x35/0xc9
[<ffffffff811b57f1>] ? pci_do_find_bus+0x49/0x49
[<ffffffff8120deb3>] next_device+0x9/0x19
[<ffffffff8120e5a2>] bus_find_device+0x6c/0x90
[<ffffffff811b5908>] pci_get_dev_by_id+0x5e/0x81
[<ffffffff811b5a6a>] pci_get_subsys+0x5c/0x7f
[<ffffffff811b5a9e>] pci_get_device+0x11/0x13
[<ffffffffa00b2087>] pci_scan+0x39/0x8a [pci_scan]
[<ffffffffa00b204e>] ? init_module+0x3c/0x3c [pci_scan]
[<ffffffff81040e6e>] kthread+0x84/0x8c
[<ffffffff813c8b14>] kernel_thread_helper+0x4/0x10
[<ffffffff81040dea>] ? __init_kthread_worker+0x37/0x37
[<ffffffff813c8b10>] ? gs_change+0xb/0xb
---[ end trace 79cea1ec476672ff ]---
general protection fault: 0000 [#1] PREEMPT SMP
CPU 1
Modules linked in: pci_scan(O) sch_prio sand_dma(PO) arista_bde(PO)
macvlan ip6table_mangle iptable_mangle msr nf_conntrack_ipv6
nf_defrag_ipv6 ip6t_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG
xt_limit ipt_REJECT xt_hl xt_state xt_multiport xt_tcpudp kbfd(O)
8021q garp stp llc tun scd_em_driver(O) nf_conntrack_tftp iptable_raw
iptable_filter ip_tables xt_NOTRACK nf_conntrack xt_mark ip6table_raw
ip6table_filter ip6_tables x_tables scd(O) k8temp amd64_edac_mod hwmon
kvm_amd kvm
Pid: 6861, comm: pci_scan_0 Tainted: P W O
3.4.43.Ar-1797671.flbocafruggeri #1
RIP: 0010:[<ffffffff813a442c>] [<ffffffff813a442c>] klist_release+0x49/0xeb
RSP: 0018:ffff88001c55bd50 EFLAGS: 00010293
RAX: dead000000200200 RBX: ffff880030949e78 RCX: ffff880000000010
RDX: dead000000100100 RSI: 0000000000000000 RDI: dead000000200200
RBP: ffff88001c55bd70 R08: dead000000100100 R09: 000000000000000a
R10: 0000000000000000 R11: ffffffff81619920 R12: ffff880030949e90
R13: ffff880030949e78 R14: ffffffff8120de13 R15: ffff880027e717e0
FS: 0000000000000000(0000) GS:ffff88013fb00000(0000) knlGS:00000000f73bc6d0
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000009012644 CR3: 0000000069f9e000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process pci_scan_0 (pid: 6861, threadinfo ffff88001c55a000, task
ffff880032ffd340)
Stack:
ffff880030949e78 ffff88001c55bde0 dead000000100100 ffff880030949e78
ffff88001c55bd80 ffffffff813a44ec ffff88001c55bdc0 ffffffff813a4528
ffff88001c55bde0 ffff880027e717e0 ffffffff811b57f1 ffff88001c55bde0
Call Trace:
[<ffffffff813a44ec>] klist_dec_and_del+0x1e/0x25
[<ffffffff813a4528>] klist_next+0x35/0xc9
[<ffffffff811b57f1>] ? pci_do_find_bus+0x49/0x49
[<ffffffff8120deb3>] next_device+0x9/0x19
[<ffffffff8120e5a2>] bus_find_device+0x6c/0x90
[<ffffffff811b5908>] pci_get_dev_by_id+0x5e/0x81
[<ffffffff811b5a6a>] pci_get_subsys+0x5c/0x7f
[<ffffffff811b5a9e>] pci_get_device+0x11/0x13
[<ffffffffa00b2087>] pci_scan+0x39/0x8a [pci_scan]
[<ffffffffa00b204e>] ? init_module+0x3c/0x3c [pci_scan]
[<ffffffff81040e6e>] kthread+0x84/0x8c
[<ffffffff813c8b14>] kernel_thread_helper+0x4/0x10
[<ffffffff81040dea>] ? __init_kthread_worker+0x37/0x37
[<ffffffff813c8b10>] ? gs_change+0xb/0xb
Code: 00 48 c7 c7 a1 01 51 81 e8 ce 59 c8 ff 49 8b 54 24 f0 49 8b 44
24 f8 49 b8 00 01 10 00 00 00 ad de 48 bf 00 02 20 00 00 00 ad de <48>
89 42 08 48 89 10 49 89 7c 24 f8 4d 89 44 24 f0 48 c7 c7 30
RIP [<ffffffff813a442c>] klist_release+0x49/0xeb
RSP <ffff88001c55bd50>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: pci: kernel crash in bus_find_device
2014-05-20 19:17 pci: kernel crash in bus_find_device Francesco Ruggeri
@ 2014-05-20 19:50 ` Guenter Roeck
2014-05-20 22:35 ` Francesco Ruggeri
0 siblings, 1 reply; 5+ messages in thread
From: Guenter Roeck @ 2014-05-20 19:50 UTC (permalink / raw)
To: Francesco Ruggeri; +Cc: linux-pci, linux-kernel, Francesco Ruggeri
On Tue, May 20, 2014 at 12:17:57PM -0700, Francesco Ruggeri wrote:
> I posted this about a week ago but I did not get any replies.
> Re-trying.
>
> While traversing devices on pci_bus_type I ran into the crash below.
> The immediate cause of the crash is that bus_find_device is trying to resume
> a scan starting from a device that has been unregistered (and whose knode_bus
> has already been klist_del' ed).
> The main issue seems to be that when resuming a scan the caller should
> be holding a
> reference to the klist_node, but instead it relies on holding a
> reference to the device.
> I played with a couple of narrow fixes, but a clean solution would
> affect quite a bit of code.
>
> Has anybody run into this before?
>
Hi Francesco,
I may be missing something, but I don't find a pci_scan symbol in the 3.4
kernel. Also, the process name suggests that you may possibly trigger pci
rescans from user space. Both suggest that you may possibly run third party
code in your kernel.
Either case, I ran into similar problems myself with pci rescans triggered
from user space. The 3.4 kernel has no synchronization for rescans triggered
from user space with those triggered from the kernel. In a nutshell, when
triggering rescans and removals from user space you must ensure that only
one such rescan/removal is active at any given time. Under no circumstances
trigger rescans from user space if a rescan can also be triggered from the
kernel. Obviously that also applies if rescans can be triggered multiple times
in parallel by some third party kernel module. Maybe that explains your
problem ?
The problem has been addressed recently with commit 9d16947 (PCI: Add
global pci_lock_rescan_remove) and several subsequent patches.
Guenter
> Thanks,
> Francesco Ruggeri
>
>
> ------------[ cut here ]------------
> WARNING: at /bld/EosKernel/Artools-rpmbuild/linux-3.4/include/linux/kref.h:41
> klist_iter_init_node+0x30/0x38()
> Modules linked in: pci_scan(O) sch_prio sand_dma(PO) arista_bde(PO)
> macvlan ip6table_mangle iptable_mangle msr nf_conntrack_ipv6
> nf_defrag_ipv6 ip6t_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG
> xt_limit ipt_REJECT xt_hl xt_state xt_multiport xt_tcpudp kbfd(O)
> 8021q garp stp llc tun scd_em_driver(O) nf_conntrack_tftp iptable_raw
> iptable_filter ip_tables xt_NOTRACK nf_conntrack xt_mark ip6table_raw
> ip6table_filter ip6_tables x_tables scd(O) k8temp amd64_edac_mod hwmon
> kvm_amd kvm
> Pid: 6861, comm: pci_scan_0 Tainted: P O
> 3.4.43.Ar-1797671.flbocafruggeri #1
> Call Trace:
> [<ffffffff81029dc4>] warn_slowpath_common+0x80/0x98
> [<ffffffff811b57f1>] ? pci_do_find_bus+0x49/0x49
> [<ffffffff81029df1>] warn_slowpath_null+0x15/0x17
> [<ffffffff813a43ce>] klist_iter_init_node+0x30/0x38
> [<ffffffff8120e57e>] bus_find_device+0x48/0x90
> [<ffffffff811b5908>] pci_get_dev_by_id+0x5e/0x81
> [<ffffffff811b5a6a>] pci_get_subsys+0x5c/0x7f
> [<ffffffff811b5a9e>] pci_get_device+0x11/0x13
> [<ffffffffa00b2087>] pci_scan+0x39/0x8a [pci_scan]
> [<ffffffffa00b204e>] ? init_module+0x3c/0x3c [pci_scan]
> [<ffffffff81040e6e>] kthread+0x84/0x8c
> [<ffffffff813c8b14>] kernel_thread_helper+0x4/0x10
> [<ffffffff81040dea>] ? __init_kthread_worker+0x37/0x37
> [<ffffffff813c8b10>] ? gs_change+0xb/0xb
> ---[ end trace 79cea1ec476672fe ]---
> ------------[ cut here ]------------
> WARNING: at /bld/EosKernel/Artools-rpmbuild/linux-3.4/lib/klist.c:189
> klist_release+0x2b/0xeb()
> Modules linked in: pci_scan(O) sch_prio sand_dma(PO) arista_bde(PO)
> macvlan ip6table_mangle iptable_mangle msr nf_conntrack_ipv6
> nf_defrag_ipv6 ip6t_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG
> xt_limit ipt_REJECT xt_hl xt_state xt_multiport xt_tcpudp kbfd(O)
> 8021q garp stp llc tun scd_em_driver(O) nf_conntrack_tftp iptable_raw
> iptable_filter ip_tables xt_NOTRACK nf_conntrack xt_mark ip6table_raw
> ip6table_filter ip6_tables x_tables scd(O) k8temp amd64_edac_mod hwmon
> kvm_amd kvm
> Pid: 6861, comm: pci_scan_0 Tainted: P W O
> 3.4.43.Ar-1797671.flbocafruggeri #1
> Call Trace:
> [<ffffffff81029dc4>] warn_slowpath_common+0x80/0x98
> [<ffffffff8120de13>] ? bus_get_device_klist+0x10/0x10
> [<ffffffff81029df1>] warn_slowpath_null+0x15/0x17
> [<ffffffff813a440e>] klist_release+0x2b/0xeb
> [<ffffffff813a44ec>] klist_dec_and_del+0x1e/0x25
> [<ffffffff813a4528>] klist_next+0x35/0xc9
> [<ffffffff811b57f1>] ? pci_do_find_bus+0x49/0x49
> [<ffffffff8120deb3>] next_device+0x9/0x19
> [<ffffffff8120e5a2>] bus_find_device+0x6c/0x90
> [<ffffffff811b5908>] pci_get_dev_by_id+0x5e/0x81
> [<ffffffff811b5a6a>] pci_get_subsys+0x5c/0x7f
> [<ffffffff811b5a9e>] pci_get_device+0x11/0x13
> [<ffffffffa00b2087>] pci_scan+0x39/0x8a [pci_scan]
> [<ffffffffa00b204e>] ? init_module+0x3c/0x3c [pci_scan]
> [<ffffffff81040e6e>] kthread+0x84/0x8c
> [<ffffffff813c8b14>] kernel_thread_helper+0x4/0x10
> [<ffffffff81040dea>] ? __init_kthread_worker+0x37/0x37
> [<ffffffff813c8b10>] ? gs_change+0xb/0xb
> ---[ end trace 79cea1ec476672ff ]---
> general protection fault: 0000 [#1] PREEMPT SMP
> CPU 1
> Modules linked in: pci_scan(O) sch_prio sand_dma(PO) arista_bde(PO)
> macvlan ip6table_mangle iptable_mangle msr nf_conntrack_ipv6
> nf_defrag_ipv6 ip6t_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG
> xt_limit ipt_REJECT xt_hl xt_state xt_multiport xt_tcpudp kbfd(O)
> 8021q garp stp llc tun scd_em_driver(O) nf_conntrack_tftp iptable_raw
> iptable_filter ip_tables xt_NOTRACK nf_conntrack xt_mark ip6table_raw
> ip6table_filter ip6_tables x_tables scd(O) k8temp amd64_edac_mod hwmon
> kvm_amd kvm
>
> Pid: 6861, comm: pci_scan_0 Tainted: P W O
> 3.4.43.Ar-1797671.flbocafruggeri #1
> RIP: 0010:[<ffffffff813a442c>] [<ffffffff813a442c>] klist_release+0x49/0xeb
> RSP: 0018:ffff88001c55bd50 EFLAGS: 00010293
> RAX: dead000000200200 RBX: ffff880030949e78 RCX: ffff880000000010
> RDX: dead000000100100 RSI: 0000000000000000 RDI: dead000000200200
> RBP: ffff88001c55bd70 R08: dead000000100100 R09: 000000000000000a
> R10: 0000000000000000 R11: ffffffff81619920 R12: ffff880030949e90
> R13: ffff880030949e78 R14: ffffffff8120de13 R15: ffff880027e717e0
> FS: 0000000000000000(0000) GS:ffff88013fb00000(0000) knlGS:00000000f73bc6d0
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000009012644 CR3: 0000000069f9e000 CR4: 00000000000007e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process pci_scan_0 (pid: 6861, threadinfo ffff88001c55a000, task
> ffff880032ffd340)
> Stack:
> ffff880030949e78 ffff88001c55bde0 dead000000100100 ffff880030949e78
> ffff88001c55bd80 ffffffff813a44ec ffff88001c55bdc0 ffffffff813a4528
> ffff88001c55bde0 ffff880027e717e0 ffffffff811b57f1 ffff88001c55bde0
> Call Trace:
> [<ffffffff813a44ec>] klist_dec_and_del+0x1e/0x25
> [<ffffffff813a4528>] klist_next+0x35/0xc9
> [<ffffffff811b57f1>] ? pci_do_find_bus+0x49/0x49
> [<ffffffff8120deb3>] next_device+0x9/0x19
> [<ffffffff8120e5a2>] bus_find_device+0x6c/0x90
> [<ffffffff811b5908>] pci_get_dev_by_id+0x5e/0x81
> [<ffffffff811b5a6a>] pci_get_subsys+0x5c/0x7f
> [<ffffffff811b5a9e>] pci_get_device+0x11/0x13
> [<ffffffffa00b2087>] pci_scan+0x39/0x8a [pci_scan]
> [<ffffffffa00b204e>] ? init_module+0x3c/0x3c [pci_scan]
> [<ffffffff81040e6e>] kthread+0x84/0x8c
> [<ffffffff813c8b14>] kernel_thread_helper+0x4/0x10
> [<ffffffff81040dea>] ? __init_kthread_worker+0x37/0x37
> [<ffffffff813c8b10>] ? gs_change+0xb/0xb
> Code: 00 48 c7 c7 a1 01 51 81 e8 ce 59 c8 ff 49 8b 54 24 f0 49 8b 44
> 24 f8 49 b8 00 01 10 00 00 00 ad de 48 bf 00 02 20 00 00 00 ad de <48>
> 89 42 08 48 89 10 49 89 7c 24 f8 4d 89 44 24 f0 48 c7 c7 30
> RIP [<ffffffff813a442c>] klist_release+0x49/0xeb
> RSP <ffff88001c55bd50>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: pci: kernel crash in bus_find_device
2014-05-20 19:50 ` Guenter Roeck
@ 2014-05-20 22:35 ` Francesco Ruggeri
2014-05-20 23:38 ` Guenter Roeck
2014-05-21 17:39 ` Guenter Roeck
0 siblings, 2 replies; 5+ messages in thread
From: Francesco Ruggeri @ 2014-05-20 22:35 UTC (permalink / raw)
To: Guenter Roeck, linux-pci; +Cc: linux-kernel, Francesco Ruggeri
Hi Guenter,
thank you for your reply. I will check out the changes that you pointed to.
The problem we are seeing is a race condition between for_each_pci_dev
(or similar) and device_unregisters. I am not sure if use of the new
lock should be extended to all code using for_each_pci_dev as well.
pci_scan is a kernel thread that I used for testing purposes, to
mimick the dynamics that we saw in our crashes in
edac_pci_clear_parity_errors:
for (;;) {
pci_dev = NULL;
while ((pci_dev = pci_get_device(PCI_ANY_ID,
PCI_ANY_ID, pci_dev)) != NULL)
;
}
It keeps traversing klist_devices in pci_bus_type using
bus_find_device, costantly resuming its search for the next element
starting from the one it got in the previous round.
There are several loops of this kind in linux. In case of this thread
no action is taken on the elements as they are "found".
The race condition occurs when bus_find_device resumes its search from
a device that has been unregistered. Because device_unregister resets
klist_bus in the device, bus_find device cannot resume from where it
left off in the klist.
The sequence is device_unregister, device_del, bus_remove_device,
klist_del(&dev->p->knode_bus.).
Francesco
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: pci: kernel crash in bus_find_device
2014-05-20 22:35 ` Francesco Ruggeri
@ 2014-05-20 23:38 ` Guenter Roeck
2014-05-21 17:39 ` Guenter Roeck
1 sibling, 0 replies; 5+ messages in thread
From: Guenter Roeck @ 2014-05-20 23:38 UTC (permalink / raw)
To: Francesco Ruggeri; +Cc: linux-pci, linux-kernel, Francesco Ruggeri
On Tue, May 20, 2014 at 03:35:15PM -0700, Francesco Ruggeri wrote:
> Hi Guenter,
> thank you for your reply. I will check out the changes that you pointed to.
> The problem we are seeing is a race condition between for_each_pci_dev
> (or similar) and device_unregisters. I am not sure if use of the new
> lock should be extended to all code using for_each_pci_dev as well.
>
> pci_scan is a kernel thread that I used for testing purposes, to
> mimick the dynamics that we saw in our crashes in
> edac_pci_clear_parity_errors:
>
> for (;;) {
> pci_dev = NULL;
> while ((pci_dev = pci_get_device(PCI_ANY_ID,
> PCI_ANY_ID, pci_dev)) != NULL)
> ;
> }
>
> It keeps traversing klist_devices in pci_bus_type using
> bus_find_device, costantly resuming its search for the next element
> starting from the one it got in the previous round.
> There are several loops of this kind in linux. In case of this thread
> no action is taken on the elements as they are "found".
>
> The race condition occurs when bus_find_device resumes its search from
> a device that has been unregistered. Because device_unregister resets
> klist_bus in the device, bus_find device cannot resume from where it
> left off in the klist.
> The sequence is device_unregister, device_del, bus_remove_device,
> klist_del(&dev->p->knode_bus.).
>
Hmmm ... sounds more like a generic problem, not specifically related to pci.
Essentially everything calling bus_find_device() with a starting device which
has been removed (though only pci and scsi seem to be doing that in practice).
Can you reproduce the problem with the latest kernel ? Also, can you
send me the entire file with the kernel thread you mentioned above ?
Maybe I can reproduce the problem here.
Thanks,
Guenter
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: pci: kernel crash in bus_find_device
2014-05-20 22:35 ` Francesco Ruggeri
2014-05-20 23:38 ` Guenter Roeck
@ 2014-05-21 17:39 ` Guenter Roeck
1 sibling, 0 replies; 5+ messages in thread
From: Guenter Roeck @ 2014-05-21 17:39 UTC (permalink / raw)
To: Francesco Ruggeri; +Cc: linux-pci, linux-kernel, Francesco Ruggeri
On Tue, May 20, 2014 at 03:35:15PM -0700, Francesco Ruggeri wrote:
> Hi Guenter,
> thank you for your reply. I will check out the changes that you pointed to.
> The problem we are seeing is a race condition between for_each_pci_dev
> (or similar) and device_unregisters. I am not sure if use of the new
> lock should be extended to all code using for_each_pci_dev as well.
>
> pci_scan is a kernel thread that I used for testing purposes, to
> mimick the dynamics that we saw in our crashes in
> edac_pci_clear_parity_errors:
>
> for (;;) {
> pci_dev = NULL;
> while ((pci_dev = pci_get_device(PCI_ANY_ID,
> PCI_ANY_ID, pci_dev)) != NULL)
> ;
> }
>
> It keeps traversing klist_devices in pci_bus_type using
> bus_find_device, costantly resuming its search for the next element
> starting from the one it got in the previous round.
> There are several loops of this kind in linux. In case of this thread
> no action is taken on the elements as they are "found".
>
> The race condition occurs when bus_find_device resumes its search from
> a device that has been unregistered. Because device_unregister resets
> klist_bus in the device, bus_find device cannot resume from where it
> left off in the klist.
> The sequence is device_unregister, device_del, bus_remove_device,
> klist_del(&dev->p->knode_bus.).
>
Problem is confirmed to exist in 3.14, and can be reproduced easily
with the following dummy driver, courtesy to Francesco. I added
usleep_range() to make it easier to reproduce. It took only about
half a dozen hot insertion/removal events to make it happen.
Here are the tracebacks:
------------[ cut here ]------------
WARNING: at /home/p2020/linux-freescale/include/linux/kref.h:47
Modules linked in: jnx_connector leds_gpio sam_flash gpio_sam i2c_sam sam_core uio_pci_hostif pci_scan [last unloaded: sam_core]
CPU: 0 PID: 2641 Comm: pci_scan Not tainted 3.14.4-juniper-00422-gf428c34 #47
task: e7ce8ea0 ti: e73e6000 task.ti: e73e6000
NIP: c04e0988 LR: c02baa28 CTR: c0268ca4
REGS: e73e7da0 TRAP: 0700 Not tainted (3.14.4-juniper-00422-gf428c34)
MSR: 00029000 <CE,EE,ME> CR: 24038382 XER: 00000000
GPR00: c0268b38 e73e7e50 e7ce8ea0 e7c96f94 e73e7e58 e725a264 c0268a38 eedaa2c0
GPR08: 00000002 00000001 00000000 00021000 2403d382 00000000 c00576f8 e7377750
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
GPR24: 00000000 00000000 f170f000 00000000 c056c33c c0268a38 e73e7ea8 eedaa2c0
NIP [c04e0988] klist_iter_init_node.part.0+0xc/0x684
LR [c02baa28] bus_find_device+0x48/0xac
Call Trace:
[e73e7e80] [c0268b38] pci_get_dev_by_id+0x5c/0x94
[e73e7ea0] [c0268c94] pci_get_subsys+0x38/0x48
[e73e7ed0] [f170f02c] pci_scan+0x2c/0x64 [pci_scan]
[e73e7ee0] [c00577bc] kthread+0xc4/0xd8
[e73e7f40] [c000f004] ret_from_kernel_thread+0x5c/0x64
and:
------------[ cut here ]------------
WARNING: at /home/p2020/linux-freescale/lib/klist.c:189
Modules linked in: jnx_connector leds_gpio sam_flash gpio_sam i2c_sam sam_core uio_pci_hostif pci_scan [last unloaded: sam_core]
CPU: 0 PID: 2641 Comm: pci_scan Tainted: G W 3.14.4-juniper-00422-gf428c34 #47
task: e7ce8ea0 ti: e73e6000 task.ti: e73e6000
NIP: c04d7ad0 LR: c04d7be4 CTR: c0268ca4
REGS: e73e7d30 TRAP: 0700 Tainted: G W (3.14.4-juniper-00422-gf428c34)
MSR: 00029000 <CE,EE,ME> CR: 24038382 XER: 00000000
GPR00: c04d7be4 e73e7de0 e7ce8ea0 e725a264 e73e7e58 e725a264 c0268a38 eedaa2c0
GPR08: 00000002 00000001 00000001 00021000 24038384 00000000 c00576f8 e7377750
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
GPR24: 00000000 00000000 f170f000 00000000 c02ba364 e725a258 e725a258 e73e7e58
NIP [c04d7ad0] klist_release+0x20/0xec
LR [c04d7be4] klist_dec_and_del+0x48/0x5c
Call Trace:
[e73e7e10] [c04d7be4] klist_dec_and_del+0x48/0x5c
[e73e7e20] [c04d7c3c] klist_next+0x44/0x138
[e73e7e40] [c02ba444] next_device+0x10/0x34
[e73e7e50] [c02baa30] bus_find_device+0x50/0xac
[e73e7e80] [c0268b38] pci_get_dev_by_id+0x5c/0x94
[e73e7ea0] [c0268c94] pci_get_subsys+0x38/0x48
[e73e7ed0] [f170f02c] pci_scan+0x2c/0x64 [pci_scan]
[e73e7ee0] [c00577bc] kthread+0xc4/0xd8
[e73e7f40] [c000f004] ret_from_kernel_thread+0x5c/0x64
Francesco, I'll test the patches you sent me next.
Guenter
---
/*
* PCI scan test driver
*/
#include <linux/delay.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/sched.h>
#include <linux/kprobes.h>
#include <linux/kallsyms.h>
#include <linux/kthread.h>
#include <linux/pci.h>
#include <linux/pcieport_if.h>
static struct task_struct *pci_scan_task = NULL;
static int pci_scan(void *unused)
{
for (;;) {
struct pci_dev *dev = NULL;
while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL)
usleep_range(1000, 2000);
schedule();
if (kthread_should_stop())
break;
}
return 0;
}
static int __init pci_scan_init(void)
{
pci_scan_task = kthread_create(pci_scan, NULL, "pci_scan");
if (!pci_scan_task)
return -ENODEV;
wake_up_process(pci_scan_task);
return 0;
}
static void __exit pci_scan_exit(void)
{
if (pci_scan_task)
kthread_stop(pci_scan_task);
}
module_init(pci_scan_init);
module_exit(pci_scan_exit);
MODULE_LICENSE("GPL");
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-05-21 17:39 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-05-20 19:17 pci: kernel crash in bus_find_device Francesco Ruggeri
2014-05-20 19:50 ` Guenter Roeck
2014-05-20 22:35 ` Francesco Ruggeri
2014-05-20 23:38 ` Guenter Roeck
2014-05-21 17:39 ` Guenter Roeck
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox