Direct access to GPGPU do-able?

Direct access to GPGPU do-able?

[Carsten Aulbert]

Hi,

I'll start with a one-off question here, so please cc me on the reply. 

We are running a largish cluster and are currently buying GPGPU systems (Tesla 
and soon Fermi based). We will have at least 2 possibly 4 of these cards per 
box and have the problem that some codes need different CUDA kernel drivers to 
run. As these boxes have 4 CPU cores, 12 GB of memory and CPU-VT support we 
thought that this might be solvable by creating (para-) virtualized guests on 
the boxes and passing one GPGPU device into a guest at a time. In there we 
then can run any kernel/driver combo necessary.

But since my current virtualization experience only stretches to OpenVZ and 
VirtualBox (tinkering with Xen a couple of years back), I don't know if KVM is 
the right approach here. We need something which we can automatically set-up 
via CLI, i.e. starting and stopping the guests need to be fully automatic, we 
don't need a graphical environment within the guests, just plain text is good 
enough.

What do you think, is looking at KVM the right choice for this? Can we pass a 
device directly into a guest?

Cheers

Carsten

-- 
Dr. Carsten Aulbert - Max Planck Institute for Gravitational Physics
Callinstrasse 38, 30167 Hannover, Germany
Phone/Fax: +49 511 762-17185 / -17193
http://www.top500.org/system/9234 | http://www.top500.org/connfam/6/list/3

Re: Direct access to GPGPU do-able?

[Weidong Han]

Carsten Aulbert wrote:
> Hi,
>
> I'll start with a one-off question here, so please cc me on the reply. 
>
> We are running a largish cluster and are currently buying GPGPU systems (Tesla 
> and soon Fermi based). We will have at least 2 possibly 4 of these cards per 
> box and have the problem that some codes need different CUDA kernel drivers to 
> run. As these boxes have 4 CPU cores, 12 GB of memory and CPU-VT support we 
> thought that this might be solvable by creating (para-) virtualized guests on 
> the boxes and passing one GPGPU device into a guest at a time. In there we 
> then can run any kernel/driver combo necessary.
>
> But since my current virtualization experience only stretches to OpenVZ and 
> VirtualBox (tinkering with Xen a couple of years back), I don't know if KVM is 
> the right approach here. We need something which we can automatically set-up 
> via CLI, i.e. starting and stopping the guests need to be fully automatic, we 
> don't need a graphical environment within the guests, just plain text is good 
> enough.
>
> What do you think, is looking at KVM the right choice for this? Can we pass a 
> device directly into a guest?
>   
KVM already supports device assignment (e.g. NIC) to guest, but graphics 
card assignment is not supported yet.

Federico, you said you were porting xen graphics assignment patches to 
kvm, what's your progress?

Regards,
Weidong

> Cheers
>
> Carsten
>
>   

Re: Direct access to GPGPU do-able?

[Fede]

I analyzed both XEN patches and determined where this patch must be
done in KVM. It's just a matter of writing the code.

Currently I'm studying for some final exams, so I will take some time
off before I continue with this port.

And because of hardware related problems, if it's not done in two or
three weeks, then it will be finished in february.

Regards,
Federico

On Tue, Dec 1, 2009 at 05:45, Weidong Han <weidong.han@intel.com> wrote:
> Carsten Aulbert wrote:
>>
>> Hi,
>>
>> I'll start with a one-off question here, so please cc me on the reply.
>> We are running a largish cluster and are currently buying GPGPU systems
>> (Tesla and soon Fermi based). We will have at least 2 possibly 4 of these
>> cards per box and have the problem that some codes need different CUDA
>> kernel drivers to run. As these boxes have 4 CPU cores, 12 GB of memory and
>> CPU-VT support we thought that this might be solvable by creating (para-)
>> virtualized guests on the boxes and passing one GPGPU device into a guest at
>> a time. In there we then can run any kernel/driver combo necessary.
>>
>> But since my current virtualization experience only stretches to OpenVZ
>> and VirtualBox (tinkering with Xen a couple of years back), I don't know if
>> KVM is the right approach here. We need something which we can automatically
>> set-up via CLI, i.e. starting and stopping the guests need to be fully
>> automatic, we don't need a graphical environment within the guests, just
>> plain text is good enough.
>>
>> What do you think, is looking at KVM the right choice for this? Can we
>> pass a device directly into a guest?
>>
>
> KVM already supports device assignment (e.g. NIC) to guest, but graphics
> card assignment is not supported yet.
>
> Federico, you said you were porting xen graphics assignment patches to kvm,
> what's your progress?
>
> Regards,
> Weidong
>
>> Cheers
>>
>> Carsten
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Direct access to GPGPU do-able?

[André Weidemann]

Hi Federico,

On 01.12.2009 19:33, Fede wrote:

> I analyzed both XEN patches and determined where this patch must be
> done in KVM. It's just a matter of writing the code.
>
> Currently I'm studying for some final exams, so I will take some time
> off before I continue with this port.
>
> And because of hardware related problems, if it's not done in two or
> three weeks, then it will be finished in february.


Did you find some time to start working on the patches for KVM, so 
graphic cards will be initialized properly inside VMs?

Regards
  André