From mboxrd@z Thu Jan 1 00:00:00 1970 From: Laszlo Ersek Subject: Re: [RFC PATCH linux-2.6.18-xen] pciback: clean up (MSI-X vec, entrynr) list when resetting PCI device Date: Wed, 01 Jun 2011 13:02:02 +0200 Message-ID: <4DE61C2A.2000006@redhat.com> References: <4DE60EF8.5060902@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4DE60EF8.5060902@redhat.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "xen-devel@lists.xensource.com" Cc: Paolo Bonzini , konrad.wilk@oracle.com List-Id: xen-devel@lists.xenproject.org (Perhaps it's best to write a separate mail about this.) On 06/01/11 12:05, Laszlo Ersek wrote: > Hi, > > this is more of an RFC than a patch now for linux-2.6.18-xen. Describing the situation captured in RHBZ#688673, in a nutshell: > > - let's say we have an Intel 82576 card (igb), > - the card exports virtual functions (igbvf), > - one virtual function is passed through to a PV guest, > - the igbvf driver, co-operating with pcifront, pciback, and the hypervisor, sets up three MSI-X vectors for the virtual function PCI device, > - when the domU is shut down, the MSI-X vectors are "leaked" in dom0, because nobody ever reaches dom0's pci_disable_msix() / msi_remove_pci_irq_vectors() during shutdown, > - therefore configuration of the VF during the next boot of such a guest doesn't succeed (dom0 thinks, based on its stale list, that MSI-X vectors are already allocated/mapped for the device) I tried to install a Fedora 15 PV guest as well (the host/hv was RHEL-5 kernel-xen, -264 build). Surprisingly, when the igbvf driver called pci_enable_msix() in the PV guest at initialization time, it bugged out: $ addr2line -p -f -i -e /usr/lib/debug/lib/modules/2.6.38.6-26.rc1.fc15.x86_64/vmlinux <<] pci_enable_msix+0x270/0x354 [168711.842343] PGD 9c838067 PUD 9c839067 PMD 3216067 PTE 0 [168711.842675] Oops: 0000 [#1] SMP [168711.843016] last sysfs file: /sys/devices/virtual/block/dm-0/dev [168711.843486] CPU 0 [168711.843507] Modules linked in: igbvf(+) joydev xen_pcifront xen_netfront ipv6 xen_blkfront [168711.844504] [168711.845030] Pid: 449, comm: modprobe Not tainted 2.6.38.6-26.rc1.fc15.x86_64 #1 [168711.845722] RIP: e030:[] [] pci_enable_msix+0x270/0x354 [168711.846440] RSP: e02b:ffff880098b4bc58 EFLAGS: 00010286 [168711.847172] RAX: ffffc90001a9600c RBX: ffff880003304000 RCX: 0000000000000006 [168711.847960] RDX: 0000000000000006 RSI: ffff88009c800200 RDI: ffff88009c800200 [168711.848775] RBP: ffff880098b4bcc8 R08: ffff88009c80c000 R09: 0000000000000002 [168711.849634] R10: ffff880098361400 R11: 0000000000000003 R12: ffff880002ceac80 [168711.850531] R13: ffff880096278280 R14: 0000000000000000 R15: ffff8800033048f0 [168711.851452] FS: 00007f2ee5360720(0000) GS:ffff88009fdf5000(0000) knlGS:0000000000000000 [168711.852434] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [168711.853454] CR2: ffffc90001a9600c CR3: 00000000033ce000 CR4: 0000000000002620 [168711.854529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [168711.855638] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000 [168711.856776] Process modprobe (pid: 449, threadinfo ffff880098b4a000, task ffff880003acae40) [168711.857954] Stack: [168711.859139] ffff880096278280 0000000000000002 0000000000000000 ffff8800033048f0 [168711.860445] 0000007298b4bcc8 ffffc9000000000c ffff880000000008 c002ffff00000000 [168711.861777] ffff880003304000 ffff880098350000 ffff880003304000 ffff880003304090 [168711.863149] Call Trace: [168711.864527] [] igbvf_probe+0x308/0x85b [igbvf] [168711.865978] [] ? xen_restore_fl_direct_end+0x0/0x1 [168711.867469] [] ? arch_local_irq_restore+0xb/0xd [168711.868984] [] local_pci_probe+0x44/0x75 [168711.870531] [] pci_device_probe+0xd0/0xff [168711.872170] [] driver_probe_device+0x124/0x1ff [168711.873804] [] __driver_attach+0x5a/0x7e [168711.875436] [] ? __driver_attach+0x0/0x7e [168711.875441] [] bus_for_each_dev+0x53/0x89 [168711.875446] [] driver_attach+0x1e/0x20 [168711.875455] [] bus_add_driver+0xe2/0x235 [168711.875462] [] ? igbvf_init_module+0x0/0x51 [igbvf] [168711.875468] [] driver_register+0x98/0x105 [168711.875474] [] ? igbvf_init_module+0x0/0x51 [igbvf] [168711.875480] [] __pci_register_driver+0x56/0xc3 [168711.875486] [] ? igbvf_init_module+0x0/0x51 [igbvf] [168711.875493] [] igbvf_init_module+0x4f/0x51 [igbvf] [168711.875500] [] do_one_initcall+0x7f/0x137 [168711.875506] [] sys_init_module+0xa6/0x1e4 [168711.875511] [] system_call_fastpath+0x16/0x1b [168711.875514] Code: 00 0f b7 42 04 c1 e0 04 83 c0 0c 89 45 b8 41 8b 44 24 0c 89 02 41 8b 7c 24 0c 89 4d a0 e8 d0 80 e5 ff 48 63 45 b8 49 03 44 24 20 <8b> 00 be 01 00 00 00 41 89 44 24 08 4c 89 e7 e8 c1 f2 ff ff 4d [168711.875552] RIP [] pci_enable_msix+0x270/0x354 [168711.875558] RSP [168711.875560] CR2: ffffc90001a9600c [168711.875564] ---[ end trace c82c0d02bae884a2 ]--- [168711.875570] BUG: sleeping function called from invalid context at kernel/rwsem.c:21 [168711.875573] in_atomic(): 0, irqs_disabled(): 1, pid: 449, name: modprobe [168711.875577] Pid: 449, comm: modprobe Tainted: G D 2.6.38.6-26.rc1.fc15.x86_64 #1 [168711.875580] Call Trace: [168711.875586] [] __might_sleep+0xeb/0xf0 [168711.875593] [] down_read+0x21/0x38 [168711.875598] [] acct_collect+0x4a/0x182 [168711.875603] [] do_exit+0x216/0x732 [168711.875608] [] ? _raw_spin_unlock_irqrestore+0x17/0x19 [168711.875613] [] ? check_events+0x12/0x20 [168711.875618] [] oops_end+0xbc/0xc5 [168711.875623] [] no_context+0x203/0x212 [168711.875628] [] ? __raw_callee_save_xen_pmd_val+0x11/0x1e [168711.875633] [] __bad_area_nosemaphore+0x194/0x1b7 [168711.875637] [] ? pmd_val+0x10/0x12 [168711.875642] [] ? pte_offset_kernel+0x19/0x3f [168711.875647] [] bad_area_nosemaphore+0x13/0x15 [168711.875651] [] do_page_fault+0x1c5/0x37a [168711.875657] [] page_fault+0x25/0x30 [168711.875662] [] ? pci_enable_msix+0x270/0x354 [168711.875667] [] ? pci_enable_msix+0x267/0x354 [168711.875675] [] igbvf_probe+0x308/0x85b [igbvf] [168711.875680] [] ? xen_restore_fl_direct_end+0x0/0x1 [168711.875684] [] ? arch_local_irq_restore+0xb/0xd [168711.875689] [] local_pci_probe+0x44/0x75 [168711.875693] [] pci_device_probe+0xd0/0xff [168711.875698] [] driver_probe_device+0x124/0x1ff [168711.875703] [] __driver_attach+0x5a/0x7e [168711.875708] [] ? __driver_attach+0x0/0x7e [168711.875713] [] bus_for_each_dev+0x53/0x89 [168711.875718] [] driver_attach+0x1e/0x20 [168711.875722] [] bus_add_driver+0xe2/0x235 [168711.875729] [] ? igbvf_init_module+0x0/0x51 [igbvf] [168711.875734] [] driver_register+0x98/0x105 [168711.875741] [] ? igbvf_init_module+0x0/0x51 [igbvf] [168711.875746] [] __pci_register_driver+0x56/0xc3 [168711.875752] [] ? igbvf_init_module+0x0/0x51 [igbvf] [168711.875759] [] igbvf_init_module+0x4f/0x51 [igbvf] [168711.875764] [] do_one_initcall+0x7f/0x137 [168711.875769] [] sys_init_module+0xa6/0x1e4 [168711.875773] [] system_call_fastpath+0x16/0x1b [168711.879089] udevd-work[367]: '/sbin/modprobe -bv pci:v00008086d000010CAsv000015D9sd000010C9bc02sc00i00' unexpected exit with status 0x0009