pci-stub error and MSI-X for KVM guest

pci-stub error and MSI-X for KVM guest

[Fischer, Anna]

I am running Fedora Core 12 with a 2.6.31 kernel. I use the Intel 82576 SR-IOV network card and want to assign its Virtual Functions (VFs) to separate KVM guests. My guests also run Fedora Core 12 with a 2.6.31 kernel. I use the latest igb driver in the host OS and load it with 2 VFs activated. Then I assign those to my KVM guests. I use virt-manager to do this which then takes care of configuring pci-stub.

This works fine in principle and I can see the PCI device in the guest under lspci. However, the 82576 VF driver requires the OS to support MSI-X. My Fedora installation is configured with MSI-X, e.g. CONFIG_PCI_MSI is 'y'. When I load the driver it tells me it cannot initialize MSI-X for the device, and under /proc/interrupts I can see that MSI-X does not seem to work. Is this a KVM/QEMU limitation? It works for me when running the VF driver under a non-virtualized Linux system.

Also, when I do an lspci on the KVM guest, that is fine, but when I do an lspci -v then the guest crashes down. In the host OS under dmesg I can see this:

pci-stub 0000:03:10.0: restoring config space at offset 0x1 (was 0x100000, writing 0x100004)

Is this a known issue? My qemu-kvm version is 2:0.11.0.

Thanks,
Anna

Re: pci-stub error and MSI-X for KVM guest

[Chris Wright]

* Fischer, Anna (anna.fischer@hp.com) wrote:
> I am running Fedora Core 12 with a 2.6.31 kernel. I use the Intel
> 82576 SR-IOV network card and want to assign its Virtual Functions (VFs)
> to separate KVM guests. My guests also run Fedora Core 12 with a 2.6.31
> kernel. I use the latest igb driver in the host OS and load it with 2
> VFs activated. Then I assign those to my KVM guests. I use virt-manager
> to do this which then takes care of configuring pci-stub.

By 2.6.31 are you referring to the stock Fedora 12 kernel package?

> This works fine in principle and I can see the PCI device in the
> guest under lspci. However, the 82576 VF driver requires the OS
> to support MSI-X. My Fedora installation is configured with MSI-X,
> e.g. CONFIG_PCI_MSI is 'y'. When I load the driver it tells me it cannot
> initialize MSI-X for the device, and under /proc/interrupts I can see
> that MSI-X does not seem to work. Is this a KVM/QEMU limitation? It works
> for me when running the VF driver under a non-virtualized Linux system.

No, this should work fine.  QEMU/KVM supports MSI-X to guest as well as
VFs.

> Also, when I do an lspci on the KVM guest, that is fine, but when I
> do an lspci -v then the guest crashes down. In the host OS under dmesg
> I can see this:
> 
> pci-stub 0000:03:10.0: restoring config space at offset 0x1 (was 0x100000, writing 0x100004)
> 
> Is this a known issue? My qemu-kvm version is 2:0.11.0.

No, I've not seen the crash before.  What do you mean "the guest crashes
down"?

This looks like a Fedora specific version (rpm version).  Can you verify
this is from Fedora packages vs. upstream source?  If it's Fedora,
would be useful to open a bug there.

thanks,
-chris

RE: pci-stub error and MSI-X for KVM guest

[Fischer, Anna]

> Subject: Re: pci-stub error and MSI-X for KVM guest
> 
> * Fischer, Anna (anna.fischer@hp.com) wrote:
> > I am running Fedora Core 12 with a 2.6.31 kernel. I use the Intel
> > 82576 SR-IOV network card and want to assign its Virtual Functions
> (VFs)
> > to separate KVM guests. My guests also run Fedora Core 12 with a
> 2.6.31
> > kernel. I use the latest igb driver in the host OS and load it with 2
> > VFs activated. Then I assign those to my KVM guests. I use virt-
> manager
> > to do this which then takes care of configuring pci-stub.
> 
> By 2.6.31 are you referring to the stock Fedora 12 kernel package?

Yes.


> > This works fine in principle and I can see the PCI device in the
> > guest under lspci. However, the 82576 VF driver requires the OS
> > to support MSI-X. My Fedora installation is configured with MSI-X,
> > e.g. CONFIG_PCI_MSI is 'y'. When I load the driver it tells me it
> cannot
> > initialize MSI-X for the device, and under /proc/interrupts I can see
> > that MSI-X does not seem to work. Is this a KVM/QEMU limitation? It
> works
> > for me when running the VF driver under a non-virtualized Linux system.
> 
> No, this should work fine.  QEMU/KVM supports MSI-X to guest as well as
> VFs.

Actually, I just got this to work. However, it only works if I call qemu-kvm from the command line, while it doesn't work when I start the guest via the virt-manager. So this seems to be an issue with Fedora's virt-manager rather than with KVM/QEMU. If I call qemu-kvm from the command line then I get the pci-stub messages saying 'irq xx for MSI/MSI-x' when the guest boots up and the VF device works just fine inside the guest. When I start the guest using virt-manager then I don't see any of these irq allocation messages from pci-stub. Any idea what the problem could be here?

 
> > Also, when I do an lspci on the KVM guest, that is fine, but when I
> > do an lspci -v then the guest crashes down. In the host OS under dmesg
> > I can see this:
> >
> > pci-stub 0000:03:10.0: restoring config space at offset 0x1 (was
> 0x100000, writing 0x100004)
> >
> > Is this a known issue? My qemu-kvm version is 2:0.11.0.
> 
> No, I've not seen the crash before.  What do you mean "the guest crashes
> down"?

So this also only happens when starting the guest using virt-manager. It works fine when starting qemu-kvm from the command line. This is weird as I call it with the same parameters as I can see virt-manager uses under 'ps -ef | grep qemu'. The guest crashes down means that the QEMU process is terminated. I don't see anything in the logs. It just disappears.

 
> This looks like a Fedora specific version (rpm version).  Can you verify
> this is from Fedora packages vs. upstream source?  If it's Fedora,
> would be useful to open a bug there.

Yes, I am using KVM/QEMU which ships with the Fedora Core 12 distribution.

Thanks for your help,
Anna

Re: pci-stub error and MSI-X for KVM guest

[Chris Wright]

* Fischer, Anna (anna.fischer@hp.com) wrote:
> > Subject: Re: pci-stub error and MSI-X for KVM guest
> > > This works fine in principle and I can see the PCI device in the
> > > guest under lspci. However, the 82576 VF driver requires the OS
> > > to support MSI-X. My Fedora installation is configured with MSI-X,
> > > e.g. CONFIG_PCI_MSI is 'y'. When I load the driver it tells me it
> > cannot
> > > initialize MSI-X for the device, and under /proc/interrupts I can see
> > > that MSI-X does not seem to work. Is this a KVM/QEMU limitation? It
> > works
> > > for me when running the VF driver under a non-virtualized Linux system.
> > 
> > No, this should work fine.  QEMU/KVM supports MSI-X to guest as well as
> > VFs.
> 
> Actually, I just got this to work. However, it only works if I call
> qemu-kvm from the command line, while it doesn't work when I start
> the guest via the virt-manager. So this seems to be an issue with
> Fedora's virt-manager rather than with KVM/QEMU. If I call qemu-kvm
> from the command line then I get the pci-stub messages saying 'irq xx
> for MSI/MSI-x' when the guest boots up and the VF device works just fine
> inside the guest. When I start the guest using virt-manager then I don't
> see any of these irq allocation messages from pci-stub. Any idea what
> the problem could be here?

No, sounds odd.  Can you:

  # virsh dumpxml [domain]

and show the output of the hostdev XML section?

> > > Also, when I do an lspci on the KVM guest, that is fine, but when I
> > > do an lspci -v then the guest crashes down. In the host OS under dmesg
> > > I can see this:
> > >
> > > pci-stub 0000:03:10.0: restoring config space at offset 0x1 (was
> > 0x100000, writing 0x100004)
> > >
> > > Is this a known issue? My qemu-kvm version is 2:0.11.0.
> > 
> > No, I've not seen the crash before.  What do you mean "the guest crashes
> > down"?
> 
> So this also only happens when starting the guest using virt-manager. It
> works fine when starting qemu-kvm from the command line. This is weird as
> I call it with the same parameters as I can see virt-manager uses under
> 'ps -ef | grep qemu'. The guest crashes down means that the QEMU process
> is terminated. I don't see anything in the logs. It just disappears.

Ouch.  Can you do "debuginfo-install qemu-system-x86" to get the debug
packages, then attach gdb to the QEMU process so that when you do lspci -v
in the guest (assuming this is QEMU segfaulting) you'll get a backtrace?

> > This looks like a Fedora specific version (rpm version).  Can you verify
> > this is from Fedora packages vs. upstream source?  If it's Fedora,
> > would be useful to open a bug there.
> 
> Yes, I am using KVM/QEMU which ships with the Fedora Core 12 distribution.

OK, please file a bug there (and include the backtrace info).

thanks,
-chris

RE: pci-stub error and MSI-X for KVM guest

[Fischer, Anna]

> Subject: Re: pci-stub error and MSI-X for KVM guest
> 
> * Fischer, Anna (anna.fischer@hp.com) wrote:
> > > Subject: Re: pci-stub error and MSI-X for KVM guest
> > > > This works fine in principle and I can see the PCI device in the
> > > > guest under lspci. However, the 82576 VF driver requires the OS
> > > > to support MSI-X. My Fedora installation is configured with MSI-X,
> > > > e.g. CONFIG_PCI_MSI is 'y'. When I load the driver it tells me it
> > > cannot
> > > > initialize MSI-X for the device, and under /proc/interrupts I can
> see
> > > > that MSI-X does not seem to work. Is this a KVM/QEMU limitation?
> It
> > > works
> > > > for me when running the VF driver under a non-virtualized Linux
> system.
> > >
> > > No, this should work fine.  QEMU/KVM supports MSI-X to guest as well
> as
> > > VFs.
> >
> > Actually, I just got this to work. However, it only works if I call
> > qemu-kvm from the command line, while it doesn't work when I start
> > the guest via the virt-manager. So this seems to be an issue with
> > Fedora's virt-manager rather than with KVM/QEMU. If I call qemu-kvm
> > from the command line then I get the pci-stub messages saying 'irq xx
> > for MSI/MSI-x' when the guest boots up and the VF device works just
> fine
> > inside the guest. When I start the guest using virt-manager then I
> don't
> > see any of these irq allocation messages from pci-stub. Any idea what
> > the problem could be here?
> 
> No, sounds odd.  Can you:
> 
>   # virsh dumpxml [domain]
> 
> and show the output of the hostdev XML section?

<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
	</source>
</hostdev>

The device to assign is at 0000:03:10.3, dmesg shows:

pci-stub 0000:03:10.3: enabling device (0000 -> 0002)
assign device: host bdf = 3:10:3


 
> > > > Also, when I do an lspci on the KVM guest, that is fine, but when
> I
> > > > do an lspci -v then the guest crashes down. In the host OS under
> dmesg
> > > > I can see this:
> > > >
> > > > pci-stub 0000:03:10.0: restoring config space at offset 0x1 (was
> > > 0x100000, writing 0x100004)
> > > >
> > > > Is this a known issue? My qemu-kvm version is 2:0.11.0.
> > >
> > > No, I've not seen the crash before.  What do you mean "the guest
> crashes
> > > down"?
> >
> > So this also only happens when starting the guest using virt-manager.
> It
> > works fine when starting qemu-kvm from the command line. This is weird
> as
> > I call it with the same parameters as I can see virt-manager uses
> under
> > 'ps -ef | grep qemu'. The guest crashes down means that the QEMU
> process
> > is terminated. I don't see anything in the logs. It just disappears.
> 
> Ouch.  Can you do "debuginfo-install qemu-system-x86" to get the debug
> packages, then attach gdb to the QEMU process so that when you do lspci
> -v
> in the guest (assuming this is QEMU segfaulting) you'll get a backtrace?

I don't know how I can tell virt-manager through the GUI to enable debug mode, e.g. call virt-manager with '-s'. From the command line I can attach gdb like this, but when running virt-manager from the GUI then I cannot connect to localhost:1234. However, the issues only arise when starting virt-manager from the GUI. I can't find the configuration option to somehow tell that I want it to be launched with '-s'?

 
> > > This looks like a Fedora specific version (rpm version).  Can you
> verify
> > > this is from Fedora packages vs. upstream source?  If it's Fedora,
> > > would be useful to open a bug there.
> >
> > Yes, I am using KVM/QEMU which ships with the Fedora Core 12
> distribution.
> 
> OK, please file a bug there (and include the backtrace info).

I will file a bug once I get the full information. Currently my guess is actually that I might have package mismatches or so with libvirt or virt-manager or QEMU related software. The is my only explanation for why it works from the command line, but not from the GUI. Some path variables must be set differently and perhaps pointing to different libraries or packages or so, otherwise there is no way it can behave differently when calling virt-manager with exactly the same parameters...

Cheers,
Anna

RE: pci-stub error and MSI-X for KVM guest

[Fischer, Anna]

> Subject: RE: pci-stub error and MSI-X for KVM guest
> 
> > Subject: Re: pci-stub error and MSI-X for KVM guest
> >
> > * Fischer, Anna (anna.fischer@hp.com) wrote:
> > > > Subject: Re: pci-stub error and MSI-X for KVM guest
> > > > > This works fine in principle and I can see the PCI device in the
> > > > > guest under lspci. However, the 82576 VF driver requires the OS
> > > > > to support MSI-X. My Fedora installation is configured with MSI-
> X,
> > > > > e.g. CONFIG_PCI_MSI is 'y'. When I load the driver it tells me
> it
> > > > cannot
> > > > > initialize MSI-X for the device, and under /proc/interrupts I
> can
> > see
> > > > > that MSI-X does not seem to work. Is this a KVM/QEMU limitation?
> > It
> > > > works
> > > > > for me when running the VF driver under a non-virtualized Linux
> > system.
> > > >
> > > > No, this should work fine.  QEMU/KVM supports MSI-X to guest as
> well
> > as
> > > > VFs.
> > >
> > > Actually, I just got this to work. However, it only works if I call
> > > qemu-kvm from the command line, while it doesn't work when I start
> > > the guest via the virt-manager. So this seems to be an issue with
> > > Fedora's virt-manager rather than with KVM/QEMU. If I call qemu-kvm
> > > from the command line then I get the pci-stub messages saying 'irq
> xx
> > > for MSI/MSI-x' when the guest boots up and the VF device works just
> > fine
> > > inside the guest. When I start the guest using virt-manager then I
> > don't
> > > see any of these irq allocation messages from pci-stub. Any idea
> what
> > > the problem could be here?
> >
> > No, sounds odd.  Can you:
> >
> >   # virsh dumpxml [domain]
> >
> > and show the output of the hostdev XML section?
> 
> <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
> 	</source>
> </hostdev>
> 
> The device to assign is at 0000:03:10.3, dmesg shows:
> 
> pci-stub 0000:03:10.3: enabling device (0000 -> 0002)
> assign device: host bdf = 3:10:3

I forgot, here is the process that the virt-manager GUI creates, e.g. this is the one that does not work.

qemu      3072     1  4 11:26 ?        00:00:33 /usr/bin/qemu-kvm -S -M pc-0.11 -m 1024 -smp 1 -name FC10-2 -uuid b811b278-fae2-a3cc-d51d-8f5b078b2477 -monitor unix:/var/lib/libvirt/qemu/FC10-2.monitor,server,nowait -boot c -drive file=/var/lib/libvirt/images/FC10-2.img,if=virtio,index=0,boot=on -drive file=/home/af/Download/Fedora-12-x86_64-Live-KDE.iso,if=ide,media=cdrom,index=2 -net none -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-gb -vga cirrus -soundhw es1370 -pcidevice host=03:10.3

Note that this one does work from the command line, but not via the GUI.

For the debugging to work, I need the '-s' option to be added too...

Cheers,
Anna

Re: pci-stub error and MSI-X for KVM guest

[Chris Wright]

* Fischer, Anna (anna.fischer@hp.com) wrote:
> > Ouch.  Can you do "debuginfo-install qemu-system-x86" to get the debug
> > packages, then attach gdb to the QEMU process so that when you do lspci
> > -v
> > in the guest (assuming this is QEMU segfaulting) you'll get a backtrace?
> 
> I don't know how I can tell virt-manager through the GUI to enable debug mode, e.g. call virt-manager with '-s'. From the command line I can attach gdb like this, but when running virt-manager from the GUI then I cannot connect to localhost:1234. However, the issues only arise when starting virt-manager from the GUI. I can't find the configuration option to somehow tell that I want it to be launched with '-s'?

Just looking for a backtrace of the qemu-kvm process itself.  So after
you launch it via virt-manager, gdb /usr/bin/qemu-kvm $(pidof qemu-kvm)
should be sufficient.

thanks,
-chris

RE: pci-stub error and MSI-X for KVM guest

[Fischer, Anna]

[-- Attachment #1: Type: text/plain, Size: 4093 bytes --]

> Subject: Re: pci-stub error and MSI-X for KVM guest
> 
> * Fischer, Anna (anna.fischer@hp.com) wrote:
> > > Ouch.  Can you do "debuginfo-install qemu-system-x86" to get the
> debug
> > > packages, then attach gdb to the QEMU process so that when you do
> lspci
> > > -v
> > > in the guest (assuming this is QEMU segfaulting) you'll get a
> backtrace?
> >
> > I don't know how I can tell virt-manager through the GUI to enable
> debug mode, e.g. call virt-manager with '-s'. From the command line I
> can attach gdb like this, but when running virt-manager from the GUI
> then I cannot connect to localhost:1234. However, the issues only arise
> when starting virt-manager from the GUI. I can't find the configuration
> option to somehow tell that I want it to be launched with '-s'?
> 
> Just looking for a backtrace of the qemu-kvm process itself.  So after
> you launch it via virt-manager, gdb /usr/bin/qemu-kvm $(pidof qemu-kvm)
> should be sufficient.

So, when setting a breakpoint for the exit() call I'm getting a bit closer to figuring where it kills my guest.

Breakpoint 1, exit (status=1) at exit.c:99
99	{
Current language:  auto
The current source language is "auto; currently c".
(gdb) bt
#0  exit (status=1) at exit.c:99
#1  0x0000000000470c6e in assigned_dev_pci_read_config (d=0x259c6f0, address=64, len=4)
    at /usr/src/debug/qemu-kvm-0.11.0/hw/device-assignment.c:349
#2  0x000000000042419d in handle_io (vcpu=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.11.0/qemu-kvm.c:784
#3  kvm_run (vcpu=<value optimized out>) at /usr/src/debug/qemu-kvm-0.11.0/qemu-kvm.c:1017
#4  0x0000000000424273 in kvm_cpu_exec (env=0x3f)
    at /usr/src/debug/qemu-kvm-0.11.0/qemu-kvm.c:1686
#5  0x0000000000425856 in kvm_main_loop_cpu (env=0x255a150)
    at /usr/src/debug/qemu-kvm-0.11.0/qemu-kvm.c:1868
#6  ap_main_loop (env=0x255a150) at /usr/src/debug/qemu-kvm-0.11.0/qemu-kvm.c:1905
#7  0x00000035aac06a3a in start_thread (arg=<value optimized out>) at pthread_create.c:297
#8  0x00000035aa0ddf3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()
(gdb) p assigned_dev_pci_read_config::address
$1 = 64
(gdb) p assigned_dev_pci_read_config::val
$2 = 0
(gdb) p assigned_dev_pci_read_config::len
$3 = 4
(gdb) p assigned_dev_pci_read_config::ret
$4 = <value optimized out>
(gdb) p assigned_dev_pci_read_config::fd
$5 = 13
(gdb) p assigned_dev_pci_read_config::pci_dev
$6 = (AssignedDevice *) 0x259c6f0
(gdb) p assigned_dev_pci_read_config::pci_dev->real_device
$7 = {bus = 0 '\000', dev = 0 '\000', func = 0 '\000', irq = 0, region_number = 7, regions = {{
      type = 512, valid = 1, base_addr = 4077142016, size = 16384, resource_fd = 15}, {type = 0, 
      valid = 0, base_addr = 0, size = 0, resource_fd = 0}, {type = 0, valid = 0, base_addr = 0, 
      size = 0, resource_fd = 0}, {type = 512, valid = 1, base_addr = 4077273088, size = 16384, 
      resource_fd = 16}, {type = 0, valid = 0, base_addr = 0, size = 0, resource_fd = 0}, {
      type = 0, valid = 0, base_addr = 0, size = 0, resource_fd = 0}, {type = 0, valid = 0, 
      base_addr = 0, size = 0, resource_fd = 0}}, config_fd = 13}
(gdb) p assigned_dev_pci_read_config::d
$8 = (PCIDevice *) 0x259c6f0

So the function assigned_dev_pci_read_config fails to read the PCI configuration of the device and then the exit(1) call kills my guest. I don't know enough about the internals of KVM PCI device assignment and furthermore I don't quite know why this works when starting virt-manager from the command line and not when starting it from the GUI.

>From the dmesg logs I would still guess that the problem is that pci-stub is not initialized properly, and perhaps this is also why the PCI read fails here? pci-stub tells me in the logs "enabling device" but I don't see any messages about enabling/assigning interrupts as I do when running from the command line. 

Let me know if you need any further information.

Attached a list of virt packages I run under Fedora Core 12.

Thanks,
Anna


[-- Attachment #2: packages.log --]
[-- Type: application/octet-stream, Size: 1904 bytes --]

Installed Packages
fence-virtd.x86_64                             0.1.3-1.fc12             @updates
fence-virtd-libvirt.x86_64                     0.1.3-1.fc12             @updates
libvirt.x86_64                                 0.7.1-15.fc12            @fedora 
libvirt-cim.x86_64                             0.5.7-1.fc12             @fedora 
libvirt-client.x86_64                          0.7.1-15.fc12            @fedora 
libvirt-devel.x86_64                           0.7.1-15.fc12            @fedora 
libvirt-java.noarch                            0.3.0-1.fc12             @fedora 
libvirt-java-devel.noarch                      0.3.0-1.fc12             @fedora 
libvirt-java-javadoc.noarch                    0.3.0-1.fc12             @fedora 
libvirt-python.x86_64                          0.7.1-15.fc12            @fedora 
libvirt-qpid.x86_64                            0.2.17-3.fc12            @updates
ocaml-libvirt.x86_64                           0.6.1.0-6.fc12           @fedora 
ocaml-libvirt-devel.x86_64                     0.6.1.0-6.fc12           @fedora 
python-virtinst.noarch                         0.500.0-5.fc12           @fedora 
ruby-libvirt.x86_64                            0.1.0-5.fc12             @fedora 
virt-manager.noarch                            0.8.0-7.fc12             @fedora 
virt-viewer.x86_64                             0.2.0-1.fc12             @fedora 
virtaal.noarch                                 0.5.0-1.fc12             @updates
virtuoso-opensource.x86_64                     5.0.12-1.fc12            @fedora 
virtuoso-opensource-apps.x86_64                5.0.12-1.fc12            @fedora 
virtuoso-opensource-conductor.noarch           5.0.12-1.fc12            @fedora 
virtuoso-opensource-doc.noarch                 5.0.12-1.fc12            @fedora 
virtuoso-opensource-utils.x86_64               5.0.12-1.fc12            @fedora 

Re: pci-stub error and MSI-X for KVM guest

[Chris Wright]

* Fischer, Anna (anna.fischer@hp.com) wrote:
> So, when setting a breakpoint for the exit() call I'm getting a bit closer to figuring where it kills my guest.

Thanks, this helps clarify what is happening.

> Breakpoint 1, exit (status=1) at exit.c:99
> 99	{
> Current language:  auto
> The current source language is "auto; currently c".
> (gdb) bt
> #0  exit (status=1) at exit.c:99
> #1  0x0000000000470c6e in assigned_dev_pci_read_config (d=0x259c6f0, address=64, len=4)

assigned_dev_pci_read_config(..., 64, 4)
                                  ^^
This is a libvirt issue.  When you use virt-manager it has libvirtd
fork/exec qemu-kvm.  libvirtd will drop privileges and run qemu-kvm as
user qemu (or perhaps root if you've edited qemu.conf).  Regardless of
the user, it clears capabilities.  Reading PCI config space beyond just
the header requires CAP_SYS_ADMIN.  The above is reading the first 4
bytes of device dependent config space, and the kernel is returning 0
because qemu doesn't have CAP_SYS_ADMIN.

Basically, this means that device assignment w/ libvirt will break
MSI/MSI-X because qemu will never be able to see that the host device
has those PCI capabilities.  This, in turn, renders VF device assignment
useless (since a VF is required to support MSI and/or MSI-X).

Granting CAP_SYS_ADMIN for each qemu instance that does device assignment
would render the privilege reduction useless (CAP_SYS_ADMIN is the
kitchen sink catchall of the Linux capability system).

Hmmph...

Re: pci-stub error and MSI-X for KVM guest

[Daniel P. Berrange]

On Thu, Jan 07, 2010 at 04:50:03PM -0800, Chris Wright wrote:
> * Fischer, Anna (anna.fischer@hp.com) wrote:
> > So, when setting a breakpoint for the exit() call I'm getting a bit closer to figuring where it kills my guest.
> 
> Thanks, this helps clarify what is happening.
> 
> > Breakpoint 1, exit (status=1) at exit.c:99
> > 99	{
> > Current language:  auto
> > The current source language is "auto; currently c".
> > (gdb) bt
> > #0  exit (status=1) at exit.c:99
> > #1  0x0000000000470c6e in assigned_dev_pci_read_config (d=0x259c6f0, address=64, len=4)
> 
> assigned_dev_pci_read_config(..., 64, 4)
>                                   ^^
> This is a libvirt issue.  When you use virt-manager it has libvirtd
> fork/exec qemu-kvm.  libvirtd will drop privileges and run qemu-kvm as
> user qemu (or perhaps root if you've edited qemu.conf).  Regardless of
> the user, it clears capabilities.  Reading PCI config space beyond just
> the header requires CAP_SYS_ADMIN.  The above is reading the first 4
> bytes of device dependent config space, and the kernel is returning 0
> because qemu doesn't have CAP_SYS_ADMIN.

Hmm, libvirt also chown()'s the files in /sys/bus/pci/devices/<DEVICE>/*
to 'qemu' (and sets SELinux context) so that the unprivileged QEMU process
can have full read/write access to them. I would have hoped that would
avoid the need to have any capabilities like CAP_SYS_ADMIN :-(

> Basically, this means that device assignment w/ libvirt will break
> MSI/MSI-X because qemu will never be able to see that the host device
> has those PCI capabilities.  This, in turn, renders VF device assignment
> useless (since a VF is required to support MSI and/or MSI-X).
> 
> Granting CAP_SYS_ADMIN for each qemu instance that does device assignment
> would render the privilege reduction useless (CAP_SYS_ADMIN is the
> kitchen sink catchall of the Linux capability system).

Yeah that's pretty troublesome, even when libvirt runs QEMU as 'root', it will
remove all capabilities. Why is the 'CAP_SYS_ADMIN' check there - is it a
mistakenly over-zealous permission check that could be removed, just relying
on access controls on the sysfs /sys/bus/pci/devices/<DEVICE>/config
file ?


Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: pci-stub error and MSI-X for KVM guest

[Chris Wright]

* Daniel P. Berrange (berrange@redhat.com) wrote:
> On Thu, Jan 07, 2010 at 04:50:03PM -0800, Chris Wright wrote:
> > Basically, this means that device assignment w/ libvirt will break
> > MSI/MSI-X because qemu will never be able to see that the host device
> > has those PCI capabilities.  This, in turn, renders VF device assignment
> > useless (since a VF is required to support MSI and/or MSI-X).
> > 
> > Granting CAP_SYS_ADMIN for each qemu instance that does device assignment
> > would render the privilege reduction useless (CAP_SYS_ADMIN is the
> > kitchen sink catchall of the Linux capability system).
> 
> Yeah that's pretty troublesome, even when libvirt runs QEMU as 'root', it will
> remove all capabilities. Why is the 'CAP_SYS_ADMIN' check there - is it a
> mistakenly over-zealous permission check that could be removed, just relying
> on access controls on the sysfs /sys/bus/pci/devices/<DEVICE>/config
> file ?

Tools like lspci expect the ability to read a pci function's config
space header.  In the past there have been devices that wedge when reading
areas of device dependent config space that are not implemented, so this
portion of config space is protected (hence the need for privileges to
do lspci -v).

thanks,
-chris