* PCI-Passthrough: VF ends up with no IRQ assigned on the physical machine
@ 2012-03-26 16:34 Alexander Lyakas
2012-03-26 22:38 ` Alex Williamson
0 siblings, 1 reply; 4+ messages in thread
From: Alexander Lyakas @ 2012-03-26 16:34 UTC (permalink / raw)
To: kvm
Greetings everybody,
I am running stock ubuntu-natty 2.6.38-8, with KVM 0.14.0 and libvirt
0.8.8. I am using Intel 82599EB dual-port 10Gb NIC with VFs spawned by
each one of the PFs. Those VFs are further assigned to KVM instances
with PCI-passthrough using libvirt. Each instance receives 4 VFs (two
VFs from each PF).
Usually when KVM starts, messages like these appear in the kernel log
of the physical machine:
syslog.ubuntu-sata-51 146907 Mar 25 18:19:51 ubuntu-sata-51 kernel:
[51267.029108] pci-stub 0000:03:13.1: claimed by stub
syslog.ubuntu-sata-51 146908 Mar 25 18:19:51 ubuntu-sata-51 kernel:
[51267.398569] pci-stub 0000:03:12.3: claimed by stub
syslog.ubuntu-sata-51 146909 Mar 25 18:19:52 ubuntu-sata-51 kernel:
[51267.628108] pci-stub 0000:03:12.4: claimed by stub
syslog.ubuntu-sata-51 146910 Mar 25 18:19:52 ubuntu-sata-51 kernel:
[51267.887635] pci-stub 0000:03:11.6: claimed by stub
syslog.ubuntu-sata-51 146916 Mar 25 18:19:54 ubuntu-sata-51 kernel:
[51270.133238] pci-stub 0000:03:13.1: enabling device (0000 -> 0002)
syslog.ubuntu-sata-51 146918 Mar 25 18:19:56 ubuntu-sata-51 kernel:
[51271.848116] pci-stub 0000:03:12.3: enabling device (0000 -> 0002)
syslog.ubuntu-sata-51 146920 Mar 25 18:19:56 ubuntu-sata-51 kernel:
[51271.950604] pci-stub 0000:03:12.4: enabling device (0000 -> 0002)
syslog.ubuntu-sata-51 146922 Mar 25 18:19:56 ubuntu-sata-51 kernel:
[51272.060462] pci-stub 0000:03:11.6: enabling device (0000 -> 0002)
syslog.ubuntu-sata-51 146924 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.051439] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X
syslog.ubuntu-sata-51 146925 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.051451] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X
syslog.ubuntu-sata-51 146926 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.051458] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X
syslog.ubuntu-sata-51 146927 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.260257] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X
syslog.ubuntu-sata-51 146928 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.260272] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X
syslog.ubuntu-sata-51 146929 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.260283] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X
syslog.ubuntu-sata-51 146931 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.470006] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X
syslog.ubuntu-sata-51 146932 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.470020] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X
syslog.ubuntu-sata-51 146933 Mar 25 18:20:03 ubuntu-sata-51 kernel:
[51279.470031] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X
syslog.ubuntu-sata-51 146934 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.709613] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X
syslog.ubuntu-sata-51 146935 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.709626] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X
syslog.ubuntu-sata-51 146936 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.709638] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X
syslog.ubuntu-sata-51 146937 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.830499] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X
syslog.ubuntu-sata-51 146938 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.830512] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X
syslog.ubuntu-sata-51 146939 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.830527] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X
syslog.ubuntu-sata-51 146940 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.979145] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X
syslog.ubuntu-sata-51 146941 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.979158] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X
syslog.ubuntu-sata-51 146942 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51279.979170] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X
syslog.ubuntu-sata-51 146943 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51280.069021] pci-stub 0000:03:11.6: irq 192 for MSI/MSI-X
syslog.ubuntu-sata-51 146944 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51280.069035] pci-stub 0000:03:11.6: irq 193 for MSI/MSI-X
syslog.ubuntu-sata-51 146945 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51280.069048] pci-stub 0000:03:11.6: irq 194 for MSI/MSI-X
syslog.ubuntu-sata-51 146946 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51280.208841] pci-stub 0000:03:11.6: irq 192 for MSI/MSI-X
syslog.ubuntu-sata-51 146947 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51280.208854] pci-stub 0000:03:11.6: irq 193 for MSI/MSI-X
syslog.ubuntu-sata-51 146948 Mar 25 18:20:04 ubuntu-sata-51 kernel:
[51280.208866] pci-stub 0000:03:11.6: irq 194 for MSI/MSI-X
After the instance starts, it modifies udev rules for consistent
interface naming and reloads the ixgbevf driver, in order for the new
udev rules to apply. In some cases this leads to a situation, in which
one of the VFs ends up with no IRQ assigned:
syslog.ubuntu-sata-51 146952 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.618421] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X
syslog.ubuntu-sata-51 146953 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.618436] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X
syslog.ubuntu-sata-51 146954 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.618448] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X
syslog.ubuntu-sata-51 146955 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.818060] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X
syslog.ubuntu-sata-51 146956 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.818074] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X
syslog.ubuntu-sata-51 146957 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.818086] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X
syslog.ubuntu-sata-51 146958 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.957754] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X
syslog.ubuntu-sata-51 146959 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.957768] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X
syslog.ubuntu-sata-51 146960 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51293.957779] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X
syslog.ubuntu-sata-51 146961 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.237367] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X
syslog.ubuntu-sata-51 146962 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.237380] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X
syslog.ubuntu-sata-51 146963 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.237391] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X
syslog.ubuntu-sata-51 146964 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.309212] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X
syslog.ubuntu-sata-51 146965 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.309227] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X
syslog.ubuntu-sata-51 146966 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.309242] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X
syslog.ubuntu-sata-51 146967 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.516813] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X
syslog.ubuntu-sata-51 146968 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.516826] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X
syslog.ubuntu-sata-51 146969 Mar 25 18:20:18 ubuntu-sata-51 kernel:
[51294.516838] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X
The VF at 03:11.6 at this point has no IRQ assigned, and is not
functional at all within the instance.
At this point, I don't know how to further debug this issue. In one
case after instance reloaded ixgbevf, I saw that same IRQ was assigned
to two different VFs - see IRQ 144/5/6:
syslog.ccmaster 75318 Mar 22 17:25:49 ccmaster kernel: [12300.850946]
pci-stub 0000:03:12.7: irq 81 for MSI/MSI-X
syslog.ccmaster 75319 Mar 22 17:25:49 ccmaster kernel: [12300.850966]
pci-stub 0000:03:12.7: irq 82 for MSI/MSI-X
syslog.ccmaster 75320 Mar 22 17:25:49 ccmaster kernel: [12300.850982]
pci-stub 0000:03:12.7: irq 83 for MSI/MSI-X
syslog.ccmaster 75321 Mar 22 17:25:50 ccmaster kernel: [12301.060561]
pci-stub 0000:03:12.7: irq 81 for MSI/MSI-X
syslog.ccmaster 75322 Mar 22 17:25:50 ccmaster kernel: [12301.060576]
pci-stub 0000:03:12.7: irq 82 for MSI/MSI-X
syslog.ccmaster 75323 Mar 22 17:25:50 ccmaster kernel: [12301.060586]
pci-stub 0000:03:12.7: irq 83 for MSI/MSI-X
syslog.ccmaster 75324 Mar 22 17:25:50 ccmaster kernel: [12301.150203]
pci-stub 0000:03:14.2: irq 120 for MSI/MSI-X
syslog.ccmaster 75325 Mar 22 17:25:50 ccmaster kernel: [12301.150220]
pci-stub 0000:03:14.2: irq 121 for MSI/MSI-X
syslog.ccmaster 75326 Mar 22 17:25:50 ccmaster kernel: [12301.150235]
pci-stub 0000:03:14.2: irq 122 for MSI/MSI-X
syslog.ccmaster 75327 Mar 22 17:25:50 ccmaster kernel: [12301.339964]
pci-stub 0000:03:14.2: irq 120 for MSI/MSI-X
syslog.ccmaster 75328 Mar 22 17:25:50 ccmaster kernel: [12301.339977]
pci-stub 0000:03:14.2: irq 121 for MSI/MSI-X
syslog.ccmaster 75329 Mar 22 17:25:50 ccmaster kernel: [12301.339988]
pci-stub 0000:03:14.2: irq 122 for MSI/MSI-X
syslog.ccmaster 75330 Mar 22 17:25:50 ccmaster kernel: [12301.509569]
pci-stub 0000:03:11.4: irq 144 for MSI/MSI-X
syslog.ccmaster 75331 Mar 22 17:25:50 ccmaster kernel: [12301.509582]
pci-stub 0000:03:11.4: irq 145 for MSI/MSI-X
syslog.ccmaster 75332 Mar 22 17:25:50 ccmaster kernel: [12301.509593]
pci-stub 0000:03:11.4: irq 146 for MSI/MSI-X
syslog.ccmaster 75333 Mar 22 17:25:50 ccmaster kernel: [12301.799293]
pci-stub 0000:03:10.1: irq 144 for MSI/MSI-X
syslog.ccmaster 75334 Mar 22 17:25:50 ccmaster kernel: [12301.799309]
pci-stub 0000:03:10.1: irq 145 for MSI/MSI-X
syslog.ccmaster 75335 Mar 22 17:25:50 ccmaster kernel: [12301.799324]
pci-stub 0000:03:10.1: irq 146 for MSI/MSI-X
syslog.ccmaster 75336 Mar 22 17:25:51 ccmaster kernel: [12301.948971]
pci-stub 0000:03:10.1: irq 144 for MSI/MSI-X
syslog.ccmaster 75337 Mar 22 17:25:51 ccmaster kernel: [12301.948984]
pci-stub 0000:03:10.1: irq 145 for MSI/MSI-X
syslog.ccmaster 75338 Mar 22 17:25:51 ccmaster kernel: [12301.948996]
pci-stub 0000:03:10.1: irq 146 for MSI/MSI-X
At this point, the 03:11.4 VF became non-functional within the
instance, and it has no IRQ assigned on the physical machine.
Can anybody please advise how can I debug this issue further?
Thanks!
P.S.: I am not subscribed to the list, so pls reply directly
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PCI-Passthrough: VF ends up with no IRQ assigned on the physical machine
2012-03-26 16:34 PCI-Passthrough: VF ends up with no IRQ assigned on the physical machine Alexander Lyakas
@ 2012-03-26 22:38 ` Alex Williamson
2012-03-27 11:01 ` Alexander Lyakas
0 siblings, 1 reply; 4+ messages in thread
From: Alex Williamson @ 2012-03-26 22:38 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: kvm
On Mon, 2012-03-26 at 18:34 +0200, Alexander Lyakas wrote:
> Greetings everybody,
> I am running stock ubuntu-natty 2.6.38-8, with KVM 0.14.0 and libvirt
Any chance you can try upstream qemu-kvm? Interrupts, including msi-x
have been given an overhaul recently. Thanks,
Alex
> 0.8.8. I am using Intel 82599EB dual-port 10Gb NIC with VFs spawned by
> each one of the PFs. Those VFs are further assigned to KVM instances
> with PCI-passthrough using libvirt. Each instance receives 4 VFs (two
> VFs from each PF).
> Usually when KVM starts, messages like these appear in the kernel log
> of the physical machine:
> syslog.ubuntu-sata-51 146907 Mar 25 18:19:51 ubuntu-sata-51 kernel:
> [51267.029108] pci-stub 0000:03:13.1: claimed by stub
> syslog.ubuntu-sata-51 146908 Mar 25 18:19:51 ubuntu-sata-51 kernel:
...
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PCI-Passthrough: VF ends up with no IRQ assigned on the physical machine
2012-03-26 22:38 ` Alex Williamson
@ 2012-03-27 11:01 ` Alexander Lyakas
2012-03-27 13:57 ` Alex Williamson
0 siblings, 1 reply; 4+ messages in thread
From: Alexander Lyakas @ 2012-03-27 11:01 UTC (permalink / raw)
To: Alex Williamson; +Cc: kvm
Hi Alex,
chances are not too high, but I can try.
Should I try version 0.15.1? Will it be compatible with my kernel
(2.6.38-8)? Or should I also take 'kvm-kmod' or 'kvm'?
Alex.
On Tue, Mar 27, 2012 at 12:38 AM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Mon, 2012-03-26 at 18:34 +0200, Alexander Lyakas wrote:
>> Greetings everybody,
>> I am running stock ubuntu-natty 2.6.38-8, with KVM 0.14.0 and libvirt
>
> Any chance you can try upstream qemu-kvm? Interrupts, including msi-x
> have been given an overhaul recently. Thanks,
>
> Alex
>
>> 0.8.8. I am using Intel 82599EB dual-port 10Gb NIC with VFs spawned by
>> each one of the PFs. Those VFs are further assigned to KVM instances
>> with PCI-passthrough using libvirt. Each instance receives 4 VFs (two
>> VFs from each PF).
>> Usually when KVM starts, messages like these appear in the kernel log
>> of the physical machine:
>> syslog.ubuntu-sata-51 146907 Mar 25 18:19:51 ubuntu-sata-51 kernel:
>> [51267.029108] pci-stub 0000:03:13.1: claimed by stub
>> syslog.ubuntu-sata-51 146908 Mar 25 18:19:51 ubuntu-sata-51 kernel:
> ...
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PCI-Passthrough: VF ends up with no IRQ assigned on the physical machine
2012-03-27 11:01 ` Alexander Lyakas
@ 2012-03-27 13:57 ` Alex Williamson
0 siblings, 0 replies; 4+ messages in thread
From: Alex Williamson @ 2012-03-27 13:57 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: kvm
On Tue, 2012-03-27 at 13:01 +0200, Alexander Lyakas wrote:
> Hi Alex,
> chances are not too high, but I can try.
> Should I try version 0.15.1? Will it be compatible with my kernel
> (2.6.38-8)? Or should I also take 'kvm-kmod' or 'kvm'?
Not new enough, try qemu-kvm.git. It should be compatible with your
kernel. Thanks,
Alex
> On Tue, Mar 27, 2012 at 12:38 AM, Alex Williamson
> <alex.williamson@redhat.com> wrote:
> > On Mon, 2012-03-26 at 18:34 +0200, Alexander Lyakas wrote:
> >> Greetings everybody,
> >> I am running stock ubuntu-natty 2.6.38-8, with KVM 0.14.0 and libvirt
> >
> > Any chance you can try upstream qemu-kvm? Interrupts, including msi-x
> > have been given an overhaul recently. Thanks,
> >
> > Alex
> >
> >> 0.8.8. I am using Intel 82599EB dual-port 10Gb NIC with VFs spawned by
> >> each one of the PFs. Those VFs are further assigned to KVM instances
> >> with PCI-passthrough using libvirt. Each instance receives 4 VFs (two
> >> VFs from each PF).
> >> Usually when KVM starts, messages like these appear in the kernel log
> >> of the physical machine:
> >> syslog.ubuntu-sata-51 146907 Mar 25 18:19:51 ubuntu-sata-51 kernel:
> >> [51267.029108] pci-stub 0000:03:13.1: claimed by stub
> >> syslog.ubuntu-sata-51 146908 Mar 25 18:19:51 ubuntu-sata-51 kernel:
> > ...
> >
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-03-27 13:57 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-03-26 16:34 PCI-Passthrough: VF ends up with no IRQ assigned on the physical machine Alexander Lyakas
2012-03-26 22:38 ` Alex Williamson
2012-03-27 11:01 ` Alexander Lyakas
2012-03-27 13:57 ` Alex Williamson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox