Re: [RFC PATCH linux-2.6.18-xen] pciback: clean up (MSI-X vec, entrynr) list when resetting PCI device

[Laszlo Ersek <lersek@redhat.com>]

(Perhaps it's best to write a separate mail about this.)

On 06/01/11 12:05, Laszlo Ersek wrote:
> Hi,
> 
> this is more of an RFC than a patch now for linux-2.6.18-xen. Describing the situation captured in RHBZ#688673, in a nutshell:
> 
> - let's say we have an Intel 82576 card (igb),
> - the card exports virtual functions (igbvf),
> - one virtual function is passed through to a PV guest,
> - the igbvf driver, co-operating with pcifront, pciback, and the hypervisor, sets up three MSI-X vectors for the virtual function PCI device,
> - when the domU is shut down, the MSI-X vectors are "leaked" in dom0, because nobody ever reaches dom0's pci_disable_msix() / msi_remove_pci_irq_vectors() during shutdown,
> - therefore configuration of the VF during the next boot of such a guest doesn't succeed (dom0 thinks, based on its stale list, that MSI-X vectors are already allocated/mapped for the device)

I tried to install a Fedora 15 PV guest as well (the host/hv was RHEL-5 kernel-xen, -264 build). Surprisingly, when the igbvf driver called pci_enable_msix() in the PV guest at initialization time, it bugged out:

$ addr2line -p -f -i -e /usr/lib/debug/lib/modules/2.6.38.6-26.rc1.fc15.x86_64/vmlinux <<<ffffffff8125658d
readl at /usr/src/debug/kernel-2.6.38.fc15/linux-2.6.38.x86_64/arch/x86/include/asm/io.h:58
 (inlined by) msix_program_entries at /usr/src/debug/kernel-2.6.38.fc15/linux-2.6.38.x86_64/drivers/pci/msi.c:527
 (inlined by) msix_capability_init at /usr/src/debug/kernel-2.6.38.fc15/linux-2.6.38.x86_64/drivers/pci/msi.c:578
 (inlined by) pci_enable_msix at /usr/src/debug/kernel-2.6.38.fc15/linux-2.6.38.x86_64/drivers/pci/msi.c:806
 (inlined by) pci_enable_msix at /usr/src/debug/kernel-2.6.38.fc15/linux-2.6.38.x86_64/drivers/pci/msi.c:773

Thanks,
lacos

[168711.841672] BUG: unable to handle kernel paging request at ffffc90001a9600c
[168711.841984] IP: [<ffffffff8125658d>] pci_enable_msix+0x270/0x354
[168711.842343] PGD 9c838067 PUD 9c839067 PMD 3216067 PTE 0
[168711.842675] Oops: 0000 [#1] SMP [168711.843016] last sysfs file: /sys/devices/virtual/block/dm-0/dev
[168711.843486] CPU 0 [168711.843507] Modules linked in: igbvf(+) joydev xen_pcifront xen_netfront ipv6 xen_blkfront
[168711.844504] [168711.845030] Pid: 449, comm: modprobe Not tainted 2.6.38.6-26.rc1.fc15.x86_64 #1  [168711.845722] RIP: e030:[<ffffffff8125658d>]  [<ffffffff8125658d>] pci_enable_msix+0x270/0x354
[168711.846440] RSP: e02b:ffff880098b4bc58  EFLAGS: 00010286
[168711.847172] RAX: ffffc90001a9600c RBX: ffff880003304000 RCX: 0000000000000006
[168711.847960] RDX: 0000000000000006 RSI: ffff88009c800200 RDI: ffff88009c800200
[168711.848775] RBP: ffff880098b4bcc8 R08: ffff88009c80c000 R09: 0000000000000002
[168711.849634] R10: ffff880098361400 R11: 0000000000000003 R12: ffff880002ceac80
[168711.850531] R13: ffff880096278280 R14: 0000000000000000 R15: ffff8800033048f0
[168711.851452] FS:  00007f2ee5360720(0000) GS:ffff88009fdf5000(0000) knlGS:0000000000000000
[168711.852434] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[168711.853454] CR2: ffffc90001a9600c CR3: 00000000033ce000 CR4: 0000000000002620
[168711.854529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[168711.855638] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
[168711.856776] Process modprobe (pid: 449, threadinfo ffff880098b4a000, task ffff880003acae40)
[168711.857954] Stack:
[168711.859139]  ffff880096278280 0000000000000002 0000000000000000 ffff8800033048f0
[168711.860445]  0000007298b4bcc8 ffffc9000000000c ffff880000000008 c002ffff00000000
[168711.861777]  ffff880003304000 ffff880098350000 ffff880003304000 ffff880003304090
[168711.863149] Call Trace:
[168711.864527]  [<ffffffffa0083400>] igbvf_probe+0x308/0x85b [igbvf]
[168711.865978]  [<ffffffff81006c3f>] ? xen_restore_fl_direct_end+0x0/0x1
[168711.867469]  [<ffffffff81080b0f>] ? arch_local_irq_restore+0xb/0xd
[168711.868984]  [<ffffffff812463ce>] local_pci_probe+0x44/0x75
[168711.870531]  [<ffffffff81246e60>] pci_device_probe+0xd0/0xff
[168711.872170]  [<ffffffff812e6af9>] driver_probe_device+0x124/0x1ff
[168711.873804]  [<ffffffff812e6c2e>] __driver_attach+0x5a/0x7e
[168711.875436]  [<ffffffff812e6bd4>] ? __driver_attach+0x0/0x7e
[168711.875441]  [<ffffffff812e5b9a>] bus_for_each_dev+0x53/0x89
[168711.875446]  [<ffffffff812e6706>] driver_attach+0x1e/0x20
[168711.875455]  [<ffffffff812e6318>] bus_add_driver+0xe2/0x235
[168711.875462]  [<ffffffffa0089000>] ? igbvf_init_module+0x0/0x51 [igbvf]
[168711.875468]  [<ffffffff812e6e17>] driver_register+0x98/0x105
[168711.875474]  [<ffffffffa0089000>] ? igbvf_init_module+0x0/0x51 [igbvf]
[168711.875480]  [<ffffffff81247723>] __pci_register_driver+0x56/0xc3
[168711.875486]  [<ffffffffa0089000>] ? igbvf_init_module+0x0/0x51 [igbvf]
[168711.875493]  [<ffffffffa008904f>] igbvf_init_module+0x4f/0x51 [igbvf]
[168711.875500]  [<ffffffff810020a4>] do_one_initcall+0x7f/0x137
[168711.875506]  [<ffffffff810852dd>] sys_init_module+0xa6/0x1e4
[168711.875511]  [<ffffffff81009bc2>] system_call_fastpath+0x16/0x1b
[168711.875514] Code: 00 0f b7 42 04 c1 e0 04 83 c0 0c 89 45 b8 41 8b 44 24 0c 89 02 41 8b 7c 24 0c 89 4d a0 e8 d0 80 e5 ff 48 63 45 b8 49 03 44 24 20 <8b> 00 be 01 00 00 00 41 89 44 24 08 4c 89 e7 e8 c1 f2 ff ff 4d [168711.875552] RIP  [<ffffffff8125658d>] pci_enable_msix+0x270/0x354
[168711.875558]  RSP <ffff880098b4bc58>
[168711.875560] CR2: ffffc90001a9600c
[168711.875564] ---[ end trace c82c0d02bae884a2 ]---
[168711.875570] BUG: sleeping function called from invalid context at kernel/rwsem.c:21
[168711.875573] in_atomic(): 0, irqs_disabled(): 1, pid: 449, name: modprobe
[168711.875577] Pid: 449, comm: modprobe Tainted: G      D     2.6.38.6-26.rc1.fc15.x86_64 #1
[168711.875580] Call Trace:
[168711.875586]  [<ffffffff81047d33>] __might_sleep+0xeb/0xf0
[168711.875593]  [<ffffffff81474eee>] down_read+0x21/0x38
[168711.875598]  [<ffffffff8108be35>] acct_collect+0x4a/0x182
[168711.875603]  [<ffffffff81058627>] do_exit+0x216/0x732
[168711.875608]  [<ffffffff8147588c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[168711.875613]  [<ffffffff81006c52>] ? check_events+0x12/0x20
[168711.875618]  [<ffffffff81476b8e>] oops_end+0xbc/0xc5
[168711.875623]  [<ffffffff8146c0b4>] no_context+0x203/0x212
[168711.875628]  [<ffffffff81004415>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[168711.875633]  [<ffffffff8146c257>] __bad_area_nosemaphore+0x194/0x1b7
[168711.875637]  [<ffffffff8146b8e4>] ? pmd_val+0x10/0x12
[168711.875642]  [<ffffffff8146b945>] ? pte_offset_kernel+0x19/0x3f
[168711.875647]  [<ffffffff8146c28d>] bad_area_nosemaphore+0x13/0x15
[168711.875651]  [<ffffffff81478c5d>] do_page_fault+0x1c5/0x37a
[168711.875657]  [<ffffffff81476095>] page_fault+0x25/0x30
[168711.875662]  [<ffffffff8125658d>] ? pci_enable_msix+0x270/0x354
[168711.875667]  [<ffffffff81256584>] ? pci_enable_msix+0x267/0x354
[168711.875675]  [<ffffffffa0083400>] igbvf_probe+0x308/0x85b [igbvf]
[168711.875680]  [<ffffffff81006c3f>] ? xen_restore_fl_direct_end+0x0/0x1
[168711.875684]  [<ffffffff81080b0f>] ? arch_local_irq_restore+0xb/0xd
[168711.875689]  [<ffffffff812463ce>] local_pci_probe+0x44/0x75
[168711.875693]  [<ffffffff81246e60>] pci_device_probe+0xd0/0xff
[168711.875698]  [<ffffffff812e6af9>] driver_probe_device+0x124/0x1ff
[168711.875703]  [<ffffffff812e6c2e>] __driver_attach+0x5a/0x7e
[168711.875708]  [<ffffffff812e6bd4>] ? __driver_attach+0x0/0x7e
[168711.875713]  [<ffffffff812e5b9a>] bus_for_each_dev+0x53/0x89
[168711.875718]  [<ffffffff812e6706>] driver_attach+0x1e/0x20
[168711.875722]  [<ffffffff812e6318>] bus_add_driver+0xe2/0x235
[168711.875729]  [<ffffffffa0089000>] ? igbvf_init_module+0x0/0x51 [igbvf]
[168711.875734]  [<ffffffff812e6e17>] driver_register+0x98/0x105
[168711.875741]  [<ffffffffa0089000>] ? igbvf_init_module+0x0/0x51 [igbvf]
[168711.875746]  [<ffffffff81247723>] __pci_register_driver+0x56/0xc3
[168711.875752]  [<ffffffffa0089000>] ? igbvf_init_module+0x0/0x51 [igbvf]
[168711.875759]  [<ffffffffa008904f>] igbvf_init_module+0x4f/0x51 [igbvf]
[168711.875764]  [<ffffffff810020a4>] do_one_initcall+0x7f/0x137
[168711.875769]  [<ffffffff810852dd>] sys_init_module+0xa6/0x1e4
[168711.875773]  [<ffffffff81009bc2>] system_call_fastpath+0x16/0x1b
[168711.879089] udevd-work[367]: '/sbin/modprobe -bv pci:v00008086d000010CAsv000015D9sd000010C9bc02sc00i00' unexpected exit with status 0x0009