PCI Device Assignment status

PCI Device Assignment status

[Rodrigo Campos]

Hi!

I wanted to know the status of PCI device assignment.

As far as I can see in the webpage and in the mailing list, it seems to be
working ok if you have VT-d support on the motherboard and cpu. But if it isn't
too much trouble, I wanted some confirmation about this, since I'm not sure and
I don't want to buy hardware to test this when there is no way it's going to
work :)

Also, I wanted to confirm that there is no limit in the maximun PCI cards that
could be given to a guest (like VMWare which only lets you give at most 2 cards
to a guest, if I'm not wrong) and which version of qemu-kvm/Linux kernel I need
for propper support.




Thanks a lot,
Rodrigo

Re: PCI Device Assignment status

[Alex Williamson]

On Fri, Sep 3, 2010 at 9:54 AM, Rodrigo Campos <rodrigo@sdfg.com.ar> wrote:
> Hi!
>
> I wanted to know the status of PCI device assignment.
>
> As far as I can see in the webpage and in the mailing list, it seems to be
> working ok if you have VT-d support on the motherboard and cpu. But if it isn't
> too much trouble, I wanted some confirmation about this, since I'm not sure and
> I don't want to buy hardware to test this when there is no way it's going to
> work :)

Yes, it works if you have VT-d support (Intel) or AMD IOMMU (note this
is different than the AMD GART that's often used as an IOMMU).  The
Intel boxes are a lot easier to find.

> Also, I wanted to confirm that there is no limit in the maximun PCI cards that
> could be given to a guest (like VMWare which only lets you give at most 2 cards
> to a guest, if I'm not wrong) and which version of qemu-kvm/Linux kernel I need
> for propper support.

The only limit I'm aware of is the number of slots on a bus (32), a
few of which are used by qemu emulated devices and can't be removed.
Any of the 0.12 releases should work fine.  Also note that there are
also device and driver dependencies that can be far more subtle in
whether the device can work with passthrough or not.  Things like
whether it has device dependent registers in pci config space, if the
driver expects specific capabilities at certain offsets, if it's a
multi-function device that shares nvram and uses the function as an
index offset, drivers that assume a bridge topology for a given
vid/did, etc...   SR-IOV devices are designed for this, so should
work, otherwise you might also post the cards you'd like to use and
see if anyone has experience with them.

Alex

Re: PCI Device Assignment status

[Rodrigo Campos]

On Fri, Sep 03, 2010 at 04:38:00PM -0600, Alex Williamson wrote:
> On Fri, Sep 3, 2010 at 9:54 AM, Rodrigo Campos <rodrigo@sdfg.com.ar> wrote:
> > Also, I wanted to confirm that there is no limit in the maximun PCI cards that
> > could be given to a guest (like VMWare which only lets you give at most 2 cards
> > to a guest, if I'm not wrong) and which version of qemu-kvm/Linux kernel I need
> > for propper support.
> 
> The only limit I'm aware of is the number of slots on a bus (32), a
> few of which are used by qemu emulated devices and can't be removed.

Great, so it shouldn't be a problem I guess =)

> Any of the 0.12 releases should work fine.  Also note that there are

Cool, thanks!

> also device and driver dependencies that can be far more subtle in
> whether the device can work with passthrough or not.  Things like
> whether it has device dependent registers in pci config space, if the
> driver expects specific capabilities at certain offsets, if it's a
> multi-function device that shares nvram and uses the function as an
> index offset, drivers that assume a bridge topology for a given
> vid/did, etc...   SR-IOV devices are designed for this, so should
> work, otherwise you might also post the cards you'd like to use and
> see if anyone has experience with them.

Ahh, I see. Thanks a lot for the information :)

I will need, for sure, at least this card:
http://www.dialogic.com/products/serial_protocol_adapters/Eiconcard_C_Series.htm?techspec=1&regID=4599

But I don't really know what the driver does. I will use it on QNX (yeah, I know
it wouldn't be hard real-time :)), so it's not the driver/software it says
there. It's this one, in case it matters/gives anyone some idea:
http://www.gandacar.ca/index.php?option=com_content&view=article&id=60&Itemid=89

I think that this card does almost everything in hardware and the driver only
does some communication (although I don't know how)

Have anyone used this card with passthrough or have some intuition if it should
work ? :)




Thanks a lot,
Rodrigo

Re: PCI Device Assignment status

[Joerg Roedel]

On Fri, Sep 03, 2010 at 04:38:00PM -0600, Alex Williamson wrote:
> On Fri, Sep 3, 2010 at 9:54 AM, Rodrigo Campos <rodrigo@sdfg.com.ar> wrote:
> > Hi!
> >
> > I wanted to know the status of PCI device assignment.
> >
> > As far as I can see in the webpage and in the mailing list, it seems to be
> > working ok if you have VT-d support on the motherboard and cpu. But if it isn't
> > too much trouble, I wanted some confirmation about this, since I'm not sure and
> > I don't want to buy hardware to test this when there is no way it's going to
> > work :)
> 
> Yes, it works if you have VT-d support (Intel) or AMD IOMMU (note this
> is different than the AMD GART that's often used as an IOMMU).  The
> Intel boxes are a lot easier to find.

For an AMD IOMMU you just need to buy a Mainboard with the AMD 890FX
chipset. As far as I know all available boards support IOMMU with the
latest BIOS.
On the server-side, look for an AMD SR56x0 chipset [where x=(5|7|9)].

	Joerg

Re: PCI Device Assignment status

[Rodrigo Campos]

On Sun, Sep 05, 2010 at 07:00:08PM +0200, Joerg Roedel wrote:
> On Fri, Sep 03, 2010 at 04:38:00PM -0600, Alex Williamson wrote:
> > On Fri, Sep 3, 2010 at 9:54 AM, Rodrigo Campos <rodrigo@sdfg.com.ar> wrote:
> > > Hi!
> > >
> > > I wanted to know the status of PCI device assignment.
> > >
> > > As far as I can see in the webpage and in the mailing list, it seems to be
> > > working ok if you have VT-d support on the motherboard and cpu. But if it isn't
> > > too much trouble, I wanted some confirmation about this, since I'm not sure and
> > > I don't want to buy hardware to test this when there is no way it's going to
> > > work :)
> > 
> > Yes, it works if you have VT-d support (Intel) or AMD IOMMU (note this
> > is different than the AMD GART that's often used as an IOMMU).  The
> > Intel boxes are a lot easier to find.
> 
> For an AMD IOMMU you just need to buy a Mainboard with the AMD 890FX
> chipset. As far as I know all available boards support IOMMU with the
> latest BIOS.
> On the server-side, look for an AMD SR56x0 chipset [where x=(5|7|9)].

Great, thanks a lot!

Re: PCI Device Assignment status

[Kenni Lund]

2010/9/3 Rodrigo Campos <rodrigo@sdfg.com.ar>:
> Hi!
>
> I wanted to know the status of PCI device assignment.
>
> As far as I can see in the webpage and in the mailing list, it seems to be
> working ok if you have VT-d support on the motherboard and cpu. But if it isn't
> too much trouble, I wanted some confirmation about this, since I'm not sure and
> I don't want to buy hardware to test this when there is no way it's going to
> work :)

It highly depends on what you want to passthrough...if it's some
well-known SR-IOV server NIC, then sure, it will probably work. If you
want to passthrough various PCI devices in a regular desktop system or
workstation, then forget it.

I've been playing that "various PCI devices in a regular desktop
system" game for the last 8 months on my HTPC. I've been running with
both stable versions of KVM as well as self-compiled versions from git
during this time. The tests I have performed were done on two
different VT-d capable boards (a Gigabyte EQ45M-S2 and a Intel
DQ57TM). I've tried to passthrough 2 PCI TV tuners (Hauppauge 500 +
Hauppauge 1300), 2 PCI USB 2.0 cards (can't remember the brands), 1
onboard USB controller (on a Gigabyte EQ45M-S2), and finally a PCI
Express USB 3.0 card (Asrock). The only device which works correctly,
is the Hauppauge 500 card. I bought the USB 3.0 card ONLY to see if it
would help with a PCI Express card instead of the regular PCI cards I
had tested...but no, it didn't change anything. In my tests the cards
"almost works" in some cases - eg. the card gets correctly initialized
in the guest and you can start to use it, but then you'll have timing
issues, driver crashes, client program crashes, etc. etc.

Right now I'm awaiting RHEL 6 to see if passthrough has been improved
here. If not, then as much as I would hate it, I'll switch back to
Xen...people could passthrough all kinds of weird PCI cards 5 years
ago with paravirtualization on Xen. Even though I see Xen as a dying
platform, at least it works.

Best Regards
Kenni

Re: PCI Device Assignment status

[Rodrigo Campos]

On Sat, Sep 04, 2010 at 02:48:46AM +0200, Kenni Lund wrote:
> 2010/9/3 Rodrigo Campos <rodrigo@sdfg.com.ar>:
> > Hi!
> >
> > I wanted to know the status of PCI device assignment.
> >
> > As far as I can see in the webpage and in the mailing list, it seems to be
> > working ok if you have VT-d support on the motherboard and cpu. But if it isn't
> > too much trouble, I wanted some confirmation about this, since I'm not sure and
> > I don't want to buy hardware to test this when there is no way it's going to
> > work :)
> 
> It highly depends on what you want to passthrough...if it's some
> well-known SR-IOV server NIC, then sure, it will probably work. If you
> want to passthrough various PCI devices in a regular desktop system or
> workstation, then forget it.

It's none of this. It's an X.25 card (there are more details on the mail I've
just sent to Alex)

> I've been playing that "various PCI devices in a regular desktop
> system" game for the last 8 months on my HTPC. I've been running with
> both stable versions of KVM as well as self-compiled versions from git
> during this time. The tests I have performed were done on two
> different VT-d capable boards (a Gigabyte EQ45M-S2 and a Intel
> DQ57TM). I've tried to passthrough 2 PCI TV tuners (Hauppauge 500 +
> Hauppauge 1300), 2 PCI USB 2.0 cards (can't remember the brands), 1
> onboard USB controller (on a Gigabyte EQ45M-S2), and finally a PCI
> Express USB 3.0 card (Asrock). The only device which works correctly,
> is the Hauppauge 500 card. I bought the USB 3.0 card ONLY to see if it
> would help with a PCI Express card instead of the regular PCI cards I
> had tested...but no, it didn't change anything. In my tests the cards
> "almost works" in some cases - eg. the card gets correctly initialized
> in the guest and you can start to use it, but then you'll have timing
> issues, driver crashes, client program crashes, etc. etc.

Ahhh, thanks for sharing your experience. I guess I will have to try then :)



Thanks,
Rodrigo

Re: PCI Device Assignment status

[Konstantin Khlebnikov]

On Fri, 3 Sep 2010 19:54:00 +0400
Rodrigo Campos <rodrigo@sdfg.com.ar> wrote:

> I wanted to know the status of PCI device assignment.

I successfully use kvm for assignment second GPU in my notebook
(Thinkpad T500) into guest OS, two months ago. But it require several
hacks for correct vga-bios passthrough and it work only for linux
guests -- all windows guests hung whole system at early boot. Currently
I haven't enough free time to finish this investigation.

Re: PCI Device Assignment status

[Rodrigo Campos]

On Mon, Sep 06, 2010 at 03:11:32PM +0400, Konstantin Khlebnikov wrote:
> On Fri, 3 Sep 2010 19:54:00 +0400
> Rodrigo Campos <rodrigo@sdfg.com.ar> wrote:
> 
> > I wanted to know the status of PCI device assignment.
> 
> I successfully use kvm for assignment second GPU in my notebook
> (Thinkpad T500) into guest OS, two months ago. But it require several
> hacks for correct vga-bios passthrough and it work only for linux
> guests -- all windows guests hung whole system at early boot. Currently
> I haven't enough free time to finish this investigation.

Thanks, its good to know. Luckily I don't need to assign a second GPU, so I
won't need those hacks :)




Thanks a lot,
Rodrigo