IOMMU Domain for Dom0

IOMMU Domain for Dom0

[Alex Merritt]

Hello,

I'm looking to enable the NVIDIA CUDA driver/runtime stack to work in
Dom0 on Xen. I've contacted NVIDIA through various capacities and have
gotten replies essentially saying they cannot provide assistance, and
have been following the nvnews.com forums. However, now that I have
IOMMU-capable processors (with VT-d) and a version of Xen which can
successfully program this hardware, I am interested to determine if it
is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0,
not an HVM/pvops guest).

I have successfully been able to launch an HVM guest, passing a few
GPUs through. Installing an unmodified developer driver from NVIDIA
works using the standard method (no SYSSRC=/-OUT= or anything) and
CUDA applications execute as expected. Can this be done for Dom0,
achieving the same result without requiring any involvement from
NVIDIA or modifications to the driver?

My immediate interest is more to see if it "can be done" via a hack or
something, not necessarily whether it would make sense for Xen to
support this in the future. My goal with this email is to get feedback
on two fronts: 1) is there a limitation due to the architecture,
meaning that as Dom0 is pvops it cannot use VT-d, or is it a
implementation addition to Xen that would be needed? 2) If the latter
is true (programming needed), how much effort (code/time) would
you/anyone estimate it would take to enable this to work, and could
you provide some starting pointers for me to do so? I am untrained
when it comes to the Xen sourcecode.

I'm using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an
Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but
that needn't be static.

I originally asked this on xen-users, but was informed this mailing
list would be better suited:
http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html

Many thanks in advance!
Alex

Re: IOMMU Domain for Dom0

[Sander Eikelenboom]

Hello Alex,

What nvidia cards did you use to succesfully passthrough to a HVM and have CUDA working ?
This could be of interest for others trying to achieve the same ...

--
Sander

Monday, June 27, 2011, 5:35:17 AM, you wrote:

> Hello,

> I'm looking to enable the NVIDIA CUDA driver/runtime stack to work in
> Dom0 on Xen. I've contacted NVIDIA through various capacities and have
> gotten replies essentially saying they cannot provide assistance, and
> have been following the nvnews.com forums. However, now that I have
> IOMMU-capable processors (with VT-d) and a version of Xen which can
> successfully program this hardware, I am interested to determine if it
> is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0,
> not an HVM/pvops guest).

> I have successfully been able to launch an HVM guest, passing a few
> GPUs through. Installing an unmodified developer driver from NVIDIA
> works using the standard method (no SYSSRC=/-OUT= or anything) and
> CUDA applications execute as expected. Can this be done for Dom0,
> achieving the same result without requiring any involvement from
> NVIDIA or modifications to the driver?

> My immediate interest is more to see if it "can be done" via a hack or
> something, not necessarily whether it would make sense for Xen to
> support this in the future. My goal with this email is to get feedback
> on two fronts: 1) is there a limitation due to the architecture,
> meaning that as Dom0 is pvops it cannot use VT-d, or is it a
> implementation addition to Xen that would be needed? 2) If the latter
> is true (programming needed), how much effort (code/time) would
> you/anyone estimate it would take to enable this to work, and could
> you provide some starting pointers for me to do so? I am untrained
> when it comes to the Xen sourcecode.

> I'm using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an
> Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but
> that needn't be static.

> I originally asked this on xen-users, but was informed this mailing
> list would be better suited:
> http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html

> Many thanks in advance!
> Alex





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: IOMMU Domain for Dom0

[Alex Merritt]

On Mon, Jun 27, 2011 at 00:01, Sander Eikelenboom <linux@eikelenboom.it> wrote:
> Hello Alex,
>
> What nvidia cards did you use to succesfully passthrough to a HVM and have CUDA working ?

NVIDIA Tesla ("Fermi") C2070
http://www.nvidia.com/object/personal-supercomputing.html

Using the latest driver available, 270.41.19
http://developer.nvidia.com/cuda-toolkit-40#Linux

Motherboard - Supermicro X8DTG-D

> This could be of interest for others trying to achieve the same ...

HVM pass-through you mean?

I'm unsure how much demand there still is for the toolstack to work in
Dom0, but I can say I'm still pushing.

>
> --
> Sander
>
> Monday, June 27, 2011, 5:35:17 AM, you wrote:
>
>> Hello,
>
>> I'm looking to enable the NVIDIA CUDA driver/runtime stack to work in
>> Dom0 on Xen. I've contacted NVIDIA through various capacities and have
>> gotten replies essentially saying they cannot provide assistance, and
>> have been following the nvnews.com forums. However, now that I have
>> IOMMU-capable processors (with VT-d) and a version of Xen which can
>> successfully program this hardware, I am interested to determine if it
>> is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0,
>> not an HVM/pvops guest).
>
>> I have successfully been able to launch an HVM guest, passing a few
>> GPUs through. Installing an unmodified developer driver from NVIDIA
>> works using the standard method (no SYSSRC=/-OUT= or anything) and
>> CUDA applications execute as expected. Can this be done for Dom0,
>> achieving the same result without requiring any involvement from
>> NVIDIA or modifications to the driver?
>
>> My immediate interest is more to see if it "can be done" via a hack or
>> something, not necessarily whether it would make sense for Xen to
>> support this in the future. My goal with this email is to get feedback
>> on two fronts: 1) is there a limitation due to the architecture,
>> meaning that as Dom0 is pvops it cannot use VT-d, or is it a
>> implementation addition to Xen that would be needed? 2) If the latter
>> is true (programming needed), how much effort (code/time) would
>> you/anyone estimate it would take to enable this to work, and could
>> you provide some starting pointers for me to do so? I am untrained
>> when it comes to the Xen sourcecode.
>
>> I'm using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an
>> Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but
>> that needn't be static.
>
>> I originally asked this on xen-users, but was informed this mailing
>> list would be better suited:
>> http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html
>
>> Many thanks in advance!
>> Alex
>
>
>
>
>
> --
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it
>
>

Re: IOMMU Domain for Dom0

[Konrad Rzeszutek Wilk]

On Sun, Jun 26, 2011 at 09:35:17PM -0600, Alex Merritt wrote:
> Hello,
> 
> I'm looking to enable the NVIDIA CUDA driver/runtime stack to work in
> Dom0 on Xen. I've contacted NVIDIA through various capacities and have

I've been running the NVIDIA binary driver with 2.6.39 / 3.0 kernel.

> gotten replies essentially saying they cannot provide assistance, and
> have been following the nvnews.com forums. However, now that I have
> IOMMU-capable processors (with VT-d) and a version of Xen which can
> successfully program this hardware, I am interested to determine if it
> is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0,
> not an HVM/pvops guest).
> 
> I have successfully been able to launch an HVM guest, passing a few
> GPUs through. Installing an unmodified developer driver from NVIDIA
> works using the standard method (no SYSSRC=/-OUT= or anything) and
> CUDA applications execute as expected. Can this be done for Dom0,
> achieving the same result without requiring any involvement from
> NVIDIA or modifications to the driver?

It should be no trouble.. albeit you might need to update the Linux
kernel to 2.6.39 or 3.0 to take advantage of the 1-1 P2M mapping code.

> 
> My immediate interest is more to see if it "can be done" via a hack or
> something, not necessarily whether it would make sense for Xen to
> support this in the future. My goal with this email is to get feedback
> on two fronts: 1) is there a limitation due to the architecture,
> meaning that as Dom0 is pvops it cannot use VT-d, or is it a
> implementation addition to Xen that would be needed? 2) If the latter
> is true (programming needed), how much effort (code/time) would
> you/anyone estimate it would take to enable this to work, and could
> you provide some starting pointers for me to do so? I am untrained
> when it comes to the Xen sourcecode.
> 
> I'm using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an
> Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but
> that needn't be static.
> 
> I originally asked this on xen-users, but was informed this mailing
> list would be better suited:
> http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html
> 
> Many thanks in advance!
> Alex
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: IOMMU Domain for Dom0

[Alex Merritt]

Konrad,

On Mon, Jun 27, 2011 at 10:11, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Sun, Jun 26, 2011 at 09:35:17PM -0600, Alex Merritt wrote:
>> Hello,
>>
>> I'm looking to enable the NVIDIA CUDA driver/runtime stack to work in
>> Dom0 on Xen. I've contacted NVIDIA through various capacities and have
>
> I've been running the NVIDIA binary driver with 2.6.39 / 3.0 kernel.

To avoid confusion here, with 2.6.39* I have been able to successfully
build, install and load the NVIDIA binary driver. Executing the SDK
applications fail (except for deviceQuery), telling me devices are not
available or don't exist. I had to manually create the device files
(/dev/nvidiactl, /dev/nvidia[0-9]) and while in runlevel 3 I had to
create a persistent client with 'nvidia-smi -pm 1' to avoid the
initial driver state caching so that applications don't have an
initial slow-down when the CUDA runtime library opens the devices
files for the first time. Have you been able to do so otherwise? I'm
not sure what exactly you mean by "running the NVIDIA binary driver".

However, installing the driver on Dom0/3.0.0-rc5 fails for me. The
build process executes a configuration test (conftest.sh:1488 after
extracting the driver sources) to determine the which Makefile to use,
then fails.

conftest.sh:1517 in <extracted nvidia sources>/kernel/ removing the
stdout/stderr redirection:

<src>/include/linux/rcupdate.h: In function '__kfree_rcu':
<src>/include/linux/rcupdate.h:822: error: size of array 'type name' is negative

--------------------------------
$ sh devdriver_4.0_linux_64_270.41.19.run -x
$ cd NVIDIA-Linux-x86_64-270.41.19/kernel
$ make module
If you are using a Linux 2.4 kernel, please make sure
you either have configured kernel sources matching your
kernel or the correct set of kernel headers installed
on your system.

If you are using a Linux 2.6 kernel, please make sure
you have configured kernel sources matching your kernel
installed on your system. If you specified a separate
output directory using either the "KBUILD_OUTPUT" or
the "O" KBUILD parameter, make sure to specify this
directory with the SYSOUT environment variable or with
the equivalent nvidia-installer command line option.

Depending on where and how the kernel sources (or the
kernel headers) were installed, you may need to specify
their location with the SYSSRC environment variable or
the equivalent nvidia-installer command line option.

*** Unable to determine the target kernel version. ***

make: *** [select_makefile] Error 1
--------------------------------

This cannot be true as I just built it and booted into it.
/lib/modules/$(uname -r)/build points to the correct sources. You did
not encounter this hiccup?

>
>> gotten replies essentially saying they cannot provide assistance, and
>> have been following the nvnews.com forums. However, now that I have
>> IOMMU-capable processors (with VT-d) and a version of Xen which can
>> successfully program this hardware, I am interested to determine if it
>> is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0,
>> not an HVM/pvops guest).
>>
>> I have successfully been able to launch an HVM guest, passing a few
>> GPUs through. Installing an unmodified developer driver from NVIDIA
>> works using the standard method (no SYSSRC=/-OUT= or anything) and
>> CUDA applications execute as expected. Can this be done for Dom0,
>> achieving the same result without requiring any involvement from
>> NVIDIA or modifications to the driver?
>
> It should be no trouble.. albeit you might need to update the Linux
> kernel to 2.6.39 or 3.0 to take advantage of the 1-1 P2M mapping code.

I will look into this once the module can build properly...

>
>>
>> My immediate interest is more to see if it "can be done" via a hack or
>> something, not necessarily whether it would make sense for Xen to
>> support this in the future. My goal with this email is to get feedback
>> on two fronts: 1) is there a limitation due to the architecture,
>> meaning that as Dom0 is pvops it cannot use VT-d, or is it a
>> implementation addition to Xen that would be needed? 2) If the latter
>> is true (programming needed), how much effort (code/time) would
>> you/anyone estimate it would take to enable this to work, and could
>> you provide some starting pointers for me to do so? I am untrained
>> when it comes to the Xen sourcecode.
>>
>> I'm using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an
>> Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but
>> that needn't be static.
>>
>> I originally asked this on xen-users, but was informed this mailing
>> list would be better suited:
>> http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html
>>
>> Many thanks in advance!
>> Alex
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: IOMMU Domain for Dom0

[Konrad Rzeszutek Wilk]

> >> I'm looking to enable the NVIDIA CUDA driver/runtime stack to work in
> >> Dom0 on Xen. I've contacted NVIDIA through various capacities and have
> >
> > I've been running the NVIDIA binary driver with 2.6.39 / 3.0 kernel.
> 
> To avoid confusion here, with 2.6.39* I have been able to successfully
..snip..
> However, installing the driver on Dom0/3.0.0-rc5 fails for me. The
> build process executes a configuration test (conftest.sh:1488 after
> extracting the driver sources) to determine the which Makefile to use,
> then fails.
> 
> conftest.sh:1517 in <extracted nvidia sources>/kernel/ removing the
> stdout/stderr redirection:
> 
> <src>/include/linux/rcupdate.h: In function '__kfree_rcu':
> <src>/include/linux/rcupdate.h:822: error: size of array 'type name' is negative
.. snip
> *** Unable to determine the target kernel version. ***
> 
> make: *** [select_makefile] Error 1
> --------------------------------
> 
> This cannot be true as I just built it and booted into it.
> /lib/modules/$(uname -r)/build points to the correct sources. You did
> not encounter this hiccup?

Um, actually I did. I completly failed to notice it as 2.6.39 worked and
under 3.0 the nouveau driver kicked in.

Looks like NVIDIA needs to fix their script to deal with the 3.0 kernels.