GTX 760 passed through

GTX 760 passed through

[Nvidia Reverse]

[-- Attachment #1.1.1: Type: text/plain, Size: 284 bytes --]

Hello,
I've successfully passed a unmodified GTX 760 to Win7 x64. It involved some
driver patching on the client side but I'm close to getting the required
steps on the server side, too. Are there any legal issues that might arise
from releasing a patch?
Bob

[image: Inline image 1]

[-- Attachment #1.1.2: Type: text/html, Size: 397 bytes --]

[-- Attachment #1.2: gtx760.png --]
[-- Type: image/png, Size: 25549 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: GTX 760 passed through

[Gordan Bobic]

 On Mon, 2 Dec 2013 13:18:22 +0100, Nvidia Reverse 
 <aidivnreverse@gmail.com> wrote:
> Hello,
> I've successfully passed a unmodified GTX 760 to Win7 x64. It 
> involved
> some driver patching on the client side but I'm close to getting the
> required steps on the server side, too. Are there any legal issues
> that might arise from releasing a patch?

 Considering we have everything up to and including a Titan/780 easily
 modifiable into Quadros/Teslas/Grids to make them work? :)

 I for one would welcome not having to break out my soldering iron.

 When you say driver patching, what are we talking about? Unless
 something changed very recently, just modifying the .inf file
 isn't sufficient (unless your server-side patch does some
 device ID faking - which it probably doesn't since the card
 is still showing up as a 780 in domU).

 Gordan

Re: GTX 760 passed through

[Nvidia Reverse]

[-- Attachment #1.1: Type: text/plain, Size: 1135 bytes --]

The nvlddmkm.sys needs to be patched removing the whitelist for the device
ids allowed to be virtualized.
But the interesting part is how NVIDIA detects that the GPU is
virtualized...


On Mon, Dec 2, 2013 at 5:12 PM, Gordan Bobic <gordan@bobich.net> wrote:

> On Mon, 2 Dec 2013 13:18:22 +0100, Nvidia Reverse <aidivnreverse@gmail.com>
> wrote:
>
>> Hello,
>> I've successfully passed a unmodified GTX 760 to Win7 x64. It involved
>> some driver patching on the client side but I'm close to getting the
>> required steps on the server side, too. Are there any legal issues
>> that might arise from releasing a patch?
>>
>
> Considering we have everything up to and including a Titan/780 easily
> modifiable into Quadros/Teslas/Grids to make them work? :)
>
> I for one would welcome not having to break out my soldering iron.
>
> When you say driver patching, what are we talking about? Unless
> something changed very recently, just modifying the .inf file
> isn't sufficient (unless your server-side patch does some
> device ID faking - which it probably doesn't since the card
> is still showing up as a 780 in domU).
>
> Gordan
>

[-- Attachment #1.2: Type: text/html, Size: 2043 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: GTX 760 passed through

[Konrad Rzeszutek Wilk]

On Mon, Dec 02, 2013 at 06:36:48PM +0100, Nvidia Reverse wrote:
> The nvlddmkm.sys needs to be patched removing the whitelist for the device
> ids allowed to be virtualized.
> But the interesting part is how NVIDIA detects that the GPU is
> virtualized...

Interesting. I was thinking that the BIOS/firmware would run itself
in the virtualized or non-virtualized code depending on the device id.
But you seem to imply that it is all in the OS driver code.

At which point the idea of just modifying in QEMU the PCI device ID
to be different would .. well, make it possible to do a lot of
neat stuff as Gordan pointed out.

In terms of legal issues of patching up a windows kernel driver and
showing other folks how to do it?

No idea. Presumarily there is some license thing that you had agreed
when you installed the Nvidia driver - check to see what it says.

> 
> 
> On Mon, Dec 2, 2013 at 5:12 PM, Gordan Bobic <gordan@bobich.net> wrote:
> 
> > On Mon, 2 Dec 2013 13:18:22 +0100, Nvidia Reverse <aidivnreverse@gmail.com>
> > wrote:
> >
> >> Hello,
> >> I've successfully passed a unmodified GTX 760 to Win7 x64. It involved
> >> some driver patching on the client side but I'm close to getting the
> >> required steps on the server side, too. Are there any legal issues
> >> that might arise from releasing a patch?
> >>
> >
> > Considering we have everything up to and including a Titan/780 easily
> > modifiable into Quadros/Teslas/Grids to make them work? :)
> >
> > I for one would welcome not having to break out my soldering iron.
> >
> > When you say driver patching, what are we talking about? Unless
> > something changed very recently, just modifying the .inf file
> > isn't sufficient (unless your server-side patch does some
> > device ID faking - which it probably doesn't since the card
> > is still showing up as a 780 in domU).
> >
> > Gordan
> >

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: GTX 760 passed through

[Gordan Bobic]

On 12/02/2013 08:16 PM, Konrad Rzeszutek Wilk wrote:
> On Mon, Dec 02, 2013 at 06:36:48PM +0100, Nvidia Reverse wrote:
>> The nvlddmkm.sys needs to be patched removing the whitelist for the device
>> ids allowed to be virtualized.
>> But the interesting part is how NVIDIA detects that the GPU is
>> virtualized...
>
> Interesting. I was thinking that the BIOS/firmware would run itself
> in the virtualized or non-virtualized code depending on the device id.
> But you seem to imply that it is all in the OS driver code.

Certainly not the case with the BIOS part - Quadro BIOS is quite 
different. GeForce BIOS, for example, doesn't have ECC control code and 
a few other things. For example, flash a 4GB GTX680 with a K5000 BIOS 
and the ECC control options start to show up in the control panel. If 
they went to the trouble of stripping all that out of the GeForce BIOS 
I'd rather like to think that they would have tripped out any magic 
required to run virtualized.

Since we know that isn't the case (since removing a single resistor on a 
680 makes it work just fine virtualized as a Tesla K10), the only 
logical conclusion there is nothing that the BIOS does that makes any 
difference to virtualization.

> At which point the idea of just modifying in QEMU the PCI device ID
> to be different would .. well, make it possible to do a lot of
> neat stuff as Gordan pointed out.

Only on pre-Kepler GPUs. As far as I can tell, there is a register in 
the GPU where the hard-strapped device ID is kept, and this gets set at 
boot time BEFORE the BIOS runs. It is the pre-BIOS device ID that gets 
stuck in the GPU, and the soft-strap only affects the PCI device ID - 
yes, the two can differ on Kepler. Driver keys off the hard-strapped ID.

> In terms of legal issues of patching up a windows kernel driver and
> showing other folks how to do it?
>
> No idea. Presumarily there is some license thing that you had agreed
> when you installed the Nvidia driver - check to see what it says.

I think the OP was more referring to specifically applying a patch to 
Xen to prevent the Nvidia driver in domU from being able to figure out 
it's running in a VM. If it doesn't think it's running in a VM, it 
doesn't disable the device.

BTW, Konrad, did your GTX460 work with PCI passthrough after modifying 
it to a Q4000M?

Gordan

Re: GTX 760 passed through

[Konrad Rzeszutek Wilk]

On Mon, Dec 02, 2013 at 09:04:51PM +0000, Gordan Bobic wrote:
> On 12/02/2013 08:16 PM, Konrad Rzeszutek Wilk wrote:
> >On Mon, Dec 02, 2013 at 06:36:48PM +0100, Nvidia Reverse wrote:
> >>The nvlddmkm.sys needs to be patched removing the whitelist for the device
> >>ids allowed to be virtualized.
> >>But the interesting part is how NVIDIA detects that the GPU is
> >>virtualized...
> >
> >Interesting. I was thinking that the BIOS/firmware would run itself
> >in the virtualized or non-virtualized code depending on the device id.
> >But you seem to imply that it is all in the OS driver code.
> 
> Certainly not the case with the BIOS part - Quadro BIOS is quite
> different. GeForce BIOS, for example, doesn't have ECC control code
> and a few other things. For example, flash a 4GB GTX680 with a K5000
> BIOS and the ECC control options start to show up in the control
> panel. If they went to the trouble of stripping all that out of the
> GeForce BIOS I'd rather like to think that they would have tripped
> out any magic required to run virtualized.
> 
> Since we know that isn't the case (since removing a single resistor
> on a 680 makes it work just fine virtualized as a Tesla K10), the
> only logical conclusion there is nothing that the BIOS does that
> makes any difference to virtualization.
> 
> >At which point the idea of just modifying in QEMU the PCI device ID
> >to be different would .. well, make it possible to do a lot of
> >neat stuff as Gordan pointed out.
> 
> Only on pre-Kepler GPUs. As far as I can tell, there is a register
> in the GPU where the hard-strapped device ID is kept, and this gets
> set at boot time BEFORE the BIOS runs. It is the pre-BIOS device ID
> that gets stuck in the GPU, and the soft-strap only affects the PCI
> device ID - yes, the two can differ on Kepler. Driver keys off the

HA! Schizophrenic card.

> hard-strapped ID.
> 
> >In terms of legal issues of patching up a windows kernel driver and
> >showing other folks how to do it?
> >
> >No idea. Presumarily there is some license thing that you had agreed
> >when you installed the Nvidia driver - check to see what it says.
> 
> I think the OP was more referring to specifically applying a patch
> to Xen to prevent the Nvidia driver in domU from being able to
> figure out it's running in a VM. If it doesn't think it's running in
> a VM, it doesn't disable the device.

Oh, Xen is open-source so you can do whatever you want (to a certain
extent of course - you can't package and resell it without providing
a means to get the source code).
> 
> BTW, Konrad, did your GTX460 work with PCI passthrough after
> modifying it to a Q4000M?

Yes, pretty fantastic! And it seems to work OK - I am loading some
games on the Windows guest to double-check. The audio part does not
seem to work - oddly enough.

I also have a Radeon 4870  that in the past worked when passing
in the guest but one had to 'unplug' it in Windows to be able to
restart it again. Will try that once I get a machine that can have
three GPUs in it.

Now I hadn't tried to start the Windows guest with more than 2GB -
but when I do that I should have some patches to make that work (based
on your work).

Re: GTX 760 passed through

[Gordan Bobic]

 On Tue, 3 Dec 2013 11:09:34 -0500, Konrad Rzeszutek Wilk 
 <konrad.wilk@oracle.com> wrote:
> On Mon, Dec 02, 2013 at 09:04:51PM +0000, Gordan Bobic wrote:
>> On 12/02/2013 08:16 PM, Konrad Rzeszutek Wilk wrote:
>> >On Mon, Dec 02, 2013 at 06:36:48PM +0100, Nvidia Reverse wrote:
>> >>The nvlddmkm.sys needs to be patched removing the whitelist for 
>> the device
>> >>ids allowed to be virtualized.
>> >>But the interesting part is how NVIDIA detects that the GPU is
>> >>virtualized...
>> >
>> >Interesting. I was thinking that the BIOS/firmware would run itself
>> >in the virtualized or non-virtualized code depending on the device 
>> id.
>> >But you seem to imply that it is all in the OS driver code.
>>
>> Certainly not the case with the BIOS part - Quadro BIOS is quite
>> different. GeForce BIOS, for example, doesn't have ECC control code
>> and a few other things. For example, flash a 4GB GTX680 with a K5000
>> BIOS and the ECC control options start to show up in the control
>> panel. If they went to the trouble of stripping all that out of the
>> GeForce BIOS I'd rather like to think that they would have tripped
>> out any magic required to run virtualized.
>>
>> Since we know that isn't the case (since removing a single resistor
>> on a 680 makes it work just fine virtualized as a Tesla K10), the
>> only logical conclusion there is nothing that the BIOS does that
>> makes any difference to virtualization.
>>
>> >At which point the idea of just modifying in QEMU the PCI device ID
>> >to be different would .. well, make it possible to do a lot of
>> >neat stuff as Gordan pointed out.
>>
>> Only on pre-Kepler GPUs. As far as I can tell, there is a register
>> in the GPU where the hard-strapped device ID is kept, and this gets
>> set at boot time BEFORE the BIOS runs. It is the pre-BIOS device ID
>> that gets stuck in the GPU, and the soft-strap only affects the PCI
>> device ID - yes, the two can differ on Kepler. Driver keys off the
>
> HA! Schizophrenic card.

 I think it's to do with the way Grid cards are designed to work.
 The idea being that the card itself is a Grid, but you can run
 it in different modes in the guest (at least on ESXi), e.g.
 GeForce, Quadro or Tesla. So the guest sees the PCI device ID
 of a GeForce card, and the driver for the GeForce card loads,
 but it only works because it checks the GPU's device ID, and
 only works if the GPU's device ID is one that it is supposed
 to work on.

 The PCI device IDs for the virtual GPUs are different from
 the physical ones, though (see the thing I pasted in the
 other thread.

>> hard-strapped ID.
>>
>> >In terms of legal issues of patching up a windows kernel driver and
>> >showing other folks how to do it?
>> >
>> >No idea. Presumarily there is some license thing that you had 
>> agreed
>> >when you installed the Nvidia driver - check to see what it says.
>>
>> I think the OP was more referring to specifically applying a patch
>> to Xen to prevent the Nvidia driver in domU from being able to
>> figure out it's running in a VM. If it doesn't think it's running in
>> a VM, it doesn't disable the device.
>
> Oh, Xen is open-source so you can do whatever you want (to a certain
> extent of course - you can't package and resell it without providing
> a means to get the source code).

 I for one am very much looking forward to seeing aidivn's patch
 for this. :)

>> BTW, Konrad, did your GTX460 work with PCI passthrough after
>> modifying it to a Q4000M?
>
> Yes, pretty fantastic! And it seems to work OK - I am loading some
> games on the Windows guest to double-check. The audio part does not
> seem to work - oddly enough.

 Splendid! I never tried HDMI audio - I don't have any HDMI audio
 hardware, so I have no way of testing whether it works. In fairness
 I find PCI audio passthrough to be problematic anyway. Sound gets
 choppy with both Intel HDA (ICH10) and Creative Labs PCIe audio
 at times (both run with intel-snd-hda driver on Linux). OTOH, a
 £3 USB audio adapter (USB host passed via PCI passthrough) works
 lovely.

> I also have a Radeon 4870  that in the past worked when passing
> in the guest but one had to 'unplug' it in Windows to be able to
> restart it again. Will try that once I get a machine that can have
> three GPUs in it.

 I have seen major problems with a setup where I have a dom0 ATI
 card and want another ATI card for domU. With 4850 in dom0 and a
 7970 for domU, when the domU starts up the host crashes solid.
 As far as I can tell, something happens with BIOS on one card
 initializing the other, and the dom0 GPU gets wiped out in
 the process. I just gave up on ATI for this decade. Maybe next
 decade.

> Now I hadn't tried to start the Windows guest with more than 2GB -
> but when I do that I should have some patches to make that work 
> (based
> on your work).

 If your IOMMU and PCIe bridges work fine you shouldn't need
 them. Remember that those (unfinished) patches were only
 useful when running with secondary PCIe bridges that bypas
 the root PCIe bridge and do DMA directly (thus preventing
 the IOMMU from translating the memory addresses). If your
 motherboard doesn't engage in such crazyness, you shouldn't
 need those patches.

 I've put the work on those patches on hold because
 everything I've read about PVH thus far looks suspiciously
 like it might solve the problem anyway (in fact, some of the
 patches in the set you mention came straight out of the PVH
 implementation, IIRC). So I think I'll try PVH when the next
 version comes out and see how that fares. For now I can live
 with the fact that my incomplete patch set results in 2.5GB
 or RAM going "missing" for each domU.

 Either way, even if you need the patches, you should be OK
 with passing the amount of RAM to domU that only goes up to
 the address where your lowest mapped BAR is on the host.

 Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel