Nvidia GPU passthrough and Device IDs

Nvidia GPU passthrough and Device IDs

[Gordan Bobic]

 This has recently come to my attention:

 http://www.nvidia.com/object/user-selectable-machines.html

 along with this snippet of information:

 - Physical Function PCIid: 10ed:11bf - GK104GL [GRID K2]
 - Virtual Function selectable from
 PCIid: 10ed:118b - GK104 [GeForce K2 USM]
 PCIid: 10ed:118c - GK104 [NVS K2 USM]
 PCIid: 10ed: 11b0 - GK104GL [Quadro K2 USM]
 PCIid: 10ed:11b1 - GK104GL [Tesla K2 USM]

 The way this reads to me is that it means that a different
 device ID is exposed to domU than what runs on the host.

 Therefore, it must be possible to change the device ID
 visible to domU to something other than what is visible
 on the host.

 So, as per a question that has recently been asked, is
 there a way to apply a software shim to expose a different
 device ID to the domU when doing PCI passthrough? This
 might completely avoid the need to modify the GeForce
 cards in ways discussed here recently to make them
 work in PCI passthrough mode by simply faking the
 device ID exposed.

 Gordan

Re: Nvidia GPU passthrough and Device IDs

[Andrew Cooper]

On 28/11/13 11:21, Gordan Bobic wrote:
> This has recently come to my attention:
>
> http://www.nvidia.com/object/user-selectable-machines.html
>
> along with this snippet of information:
>
> - Physical Function PCIid: 10ed:11bf - GK104GL [GRID K2]
> - Virtual Function selectable from
> PCIid: 10ed:118b - GK104 [GeForce K2 USM]
> PCIid: 10ed:118c - GK104 [NVS K2 USM]
> PCIid: 10ed: 11b0 - GK104GL [Quadro K2 USM]
> PCIid: 10ed:11b1 - GK104GL [Tesla K2 USM]
>
> The way this reads to me is that it means that a different
> device ID is exposed to domU than what runs on the host.
>
> Therefore, it must be possible to change the device ID
> visible to domU to something other than what is visible
> on the host.
>
> So, as per a question that has recently been asked, is
> there a way to apply a software shim to expose a different
> device ID to the domU when doing PCI passthrough? This
> might completely avoid the need to modify the GeForce
> cards in ways discussed here recently to make them
> work in PCI passthrough mode by simply faking the
> device ID exposed.
>
> Gordan

All config space accesses get vetted by Qemu or pcifront.

We have had one bug for a particular graphics card with PCI passthrough
which was fixed by faking up the wrong class code.  (There was a windows
XPDM/WDDM stack issue with the affected card, and the cirrus emulated
graphics card was detected as "better" than the passed-through one)

~Andrew

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Nvidia GPU passthrough and Device IDs

[Gordan Bobic]

 On Thu, 28 Nov 2013 11:27:09 +0000, Andrew Cooper 
 <andrew.cooper3@citrix.com> wrote:
> On 28/11/13 11:21, Gordan Bobic wrote:
>> This has recently come to my attention:
>>
>> http://www.nvidia.com/object/user-selectable-machines.html
>>
>> along with this snippet of information:
>>
>> - Physical Function PCIid: 10ed:11bf - GK104GL [GRID K2]
>> - Virtual Function selectable from
>> PCIid: 10ed:118b - GK104 [GeForce K2 USM]
>> PCIid: 10ed:118c - GK104 [NVS K2 USM]
>> PCIid: 10ed: 11b0 - GK104GL [Quadro K2 USM]
>> PCIid: 10ed:11b1 - GK104GL [Tesla K2 USM]
>>
>> The way this reads to me is that it means that a different
>> device ID is exposed to domU than what runs on the host.
>>
>> Therefore, it must be possible to change the device ID
>> visible to domU to something other than what is visible
>> on the host.
>>
>> So, as per a question that has recently been asked, is
>> there a way to apply a software shim to expose a different
>> device ID to the domU when doing PCI passthrough? This
>> might completely avoid the need to modify the GeForce
>> cards in ways discussed here recently to make them
>> work in PCI passthrough mode by simply faking the
>> device ID exposed.
>>
>> Gordan
>
> All config space accesses get vetted by Qemu or pcifront.
>
> We have had one bug for a particular graphics card with PCI 
> passthrough
> which was fixed by faking up the wrong class code.  (There was a 
> windows
> XPDM/WDDM stack issue with the affected card, and the cirrus emulated
> graphics card was detected as "better" than the passed-through one)

 I'll take that to be a resounding "yes" in answer to my
 question. :)

 Could you please point me in the rough direction of the code
 and process flow that handles this?

 Gordan

Re: Nvidia GPU passthrough and Device IDs

[Konrad Rzeszutek Wilk]

On Thu, Nov 28, 2013 at 11:29:49AM +0000, Gordan Bobic wrote:
> On Thu, 28 Nov 2013 11:27:09 +0000, Andrew Cooper
> <andrew.cooper3@citrix.com> wrote:
> >On 28/11/13 11:21, Gordan Bobic wrote:
> >>This has recently come to my attention:
> >>
> >>http://www.nvidia.com/object/user-selectable-machines.html
> >>
> >>along with this snippet of information:
> >>
> >>- Physical Function PCIid: 10ed:11bf - GK104GL [GRID K2]
> >>- Virtual Function selectable from
> >>PCIid: 10ed:118b - GK104 [GeForce K2 USM]
> >>PCIid: 10ed:118c - GK104 [NVS K2 USM]
> >>PCIid: 10ed: 11b0 - GK104GL [Quadro K2 USM]
> >>PCIid: 10ed:11b1 - GK104GL [Tesla K2 USM]
> >>
> >>The way this reads to me is that it means that a different
> >>device ID is exposed to domU than what runs on the host.
> >>
> >>Therefore, it must be possible to change the device ID
> >>visible to domU to something other than what is visible
> >>on the host.
> >>
> >>So, as per a question that has recently been asked, is
> >>there a way to apply a software shim to expose a different
> >>device ID to the domU when doing PCI passthrough? This
> >>might completely avoid the need to modify the GeForce
> >>cards in ways discussed here recently to make them
> >>work in PCI passthrough mode by simply faking the
> >>device ID exposed.
> >>
> >>Gordan
> >
> >All config space accesses get vetted by Qemu or pcifront.
> >
> >We have had one bug for a particular graphics card with PCI
> >passthrough
> >which was fixed by faking up the wrong class code.  (There was a
> >windows
> >XPDM/WDDM stack issue with the affected card, and the cirrus emulated
> >graphics card was detected as "better" than the passed-through one)
> 
> I'll take that to be a resounding "yes" in answer to my
> question. :)
> 
> Could you please point me in the rough direction of the code
> and process flow that handles this?

You would be wanting to look in /git/xen/tools/qemu-xen-dir/hw/xen_pt_config_init.c

There is this code:

591     /* Device ID reg */                                                         
 592     {                                                                           
 593         .offset     = PCI_DEVICE_ID,                                            
 594         .size       = 2,                                                        
 595         .init_val   = 0x0000,                                                   
 596         .ro_mask    = 0xFFFF,                                                   
 597         .emu_mask   = 0xFFFF,                                                   
 598         .init       = xen_pt_device_reg_init,                                   
 599         .u.w.read   = xen_pt_word_reg_read,                                     
 600         .u.w.write  = xen_pt_word_reg_write,                                    
 601     },                                                                          

And the xen_pt_... are the default ones. You want to over-write 
xen_pt_device_reg_init with your own function that sets
the init_val to something. Or modify xen_pt_device_reg_init a bit.

something that provides a different value.

Looking forward to hearing how it goes!