[Qemu-devel] approaches to 3D virtualisation

[Qemu-devel] approaches to 3D virtualisation

[Dave Airlie]

So I've been musing on the addition of some sort of 3D passthrough for
qemu (as I'm sure have lots of ppl)

But I think the goals of such an addition need to be discussed prior
to anyone writing a line of code.

Current existing solutions in the area:
a) VMware virtual graphics adapter - based on DX9, has an open
KMS/Gallium3D driver stack recently released by vmware, has certified
Windows drivers and has a documented vGPU interface (it could be
documented a lot better)

b) VirtualBox - seems to be GL based passthrough based on a Chromium
backend. DX9 support, looks to be done via Wine DX->GL converter built
into a Windows driver (not confirmed code base is crazy) I'm assuming
chromium is being used to stream GL through the passthru but this
definitely requires more investigation.

Now both of these seem to be local only rendering solutions, though I
might be underselling the vbox/chromium fun.

Now to add a remoting 3D rendering protocol is where things get ugly
fast, and very different from current 2D rendering with offscreen
pixmaps. The major difference is most server farms will not contain
any 3D hardware and will not want to contain any 3D hw due to power
considerations. Now if you have a remote protocol, and the client
disconnects, you have to keep some sort of transaction log, and either
replay the transactions when a client reconnects, or have some sort of
sw rendering kick in on the server as a fallback. Now 3D sw rendering
solutions are insanely slow and quite CPU intensive. VMware are
working on an llvm based 3D renderer for something like this situation
but I'm not convinced of how useable it will be.

Also with remoting you need to come up with a minimum acceptable level
of 3D you want to expose to guests OSes depending on what the
capabilities of the client side hw, or refuse connections unless the
client can run all the features that you've exposed in side the guest.
This ranges from DX7->DX11, and GL1.5 to GL3.0 situations.

I'm not sure how ppl who work with VNC see VNC fitting in with this
sort of protocol, I expect SPICE is better placed with proper design
to address this. Though it would firstly be nice to design a vGPU
interface for the guest OS to work against, I lean towards the VMware
because of the availability of guest drivers, however the Virtualbox
one is probably also acceptable if anyone can find where it is
documented.

This is just a sort of a brain dump from me, but I'd like to get ppl
talking about this, since 3D is not to be considered some simple
extension of 2D functionality,
its a whole different world, modern GPUs are 98% of silicon dedicated
to 3D processing, most of the current 2D type accel such as QXL
interface provides are done using 3D engines on most GPUs, and the
inability to render 3D on the server side of a link with any useful
speed in current server hw.

So I suppose the main questions to answer up front are:
1) Rendering done on same hw as VM is running?
just use VNC to dump the final answer over the network.
2) Rendering done on the client viewing end of the link where

[Qemu-devel] Re: approaches to 3D virtualisation

[Dave Airlie]

Oops gmail send this, silly laptop has a mind of its own sometimes.

On Sat, Dec 12, 2009 at 11:58 AM, Dave Airlie <airlied@gmail.com> wrote:
> So I've been musing on the addition of some sort of 3D passthrough for
> qemu (as I'm sure have lots of ppl)
>
> But I think the goals of such an addition need to be discussed prior
> to anyone writing a line of code.
>
> Current existing solutions in the area:
> a) VMware virtual graphics adapter - based on DX9, has an open
> KMS/Gallium3D driver stack recently released by vmware, has certified
> Windows drivers and has a documented vGPU interface (it could be
> documented a lot better)
>
> b) VirtualBox - seems to be GL based passthrough based on a Chromium
> backend. DX9 support, looks to be done via Wine DX->GL converter built
> into a Windows driver (not confirmed code base is crazy) I'm assuming
> chromium is being used to stream GL through the passthru but this
> definitely requires more investigation.
>
> Now both of these seem to be local only rendering solutions, though I
> might be underselling the vbox/chromium fun.
>
> Now to add a remoting 3D rendering protocol is where things get ugly
> fast, and very different from current 2D rendering with offscreen
> pixmaps. The major difference is most server farms will not contain
> any 3D hardware and will not want to contain any 3D hw due to power
> considerations. Now if you have a remote protocol, and the client
> disconnects, you have to keep some sort of transaction log, and either
> replay the transactions when a client reconnects, or have some sort of
> sw rendering kick in on the server as a fallback. Now 3D sw rendering
> solutions are insanely slow and quite CPU intensive. VMware are
> working on an llvm based 3D renderer for something like this situation
> but I'm not convinced of how useable it will be.
>
> Also with remoting you need to come up with a minimum acceptable level
> of 3D you want to expose to guests OSes depending on what the
> capabilities of the client side hw, or refuse connections unless the
> client can run all the features that you've exposed in side the guest.
> This ranges from DX7->DX11, and GL1.5 to GL3.0 situations.
>
> I'm not sure how ppl who work with VNC see VNC fitting in with this
> sort of protocol, I expect SPICE is better placed with proper design
> to address this. Though it would firstly be nice to design a vGPU
> interface for the guest OS to work against, I lean towards the VMware
> because of the availability of guest drivers, however the Virtualbox
> one is probably also acceptable if anyone can find where it is
> documented.
>
> This is just a sort of a brain dump from me, but I'd like to get ppl
> talking about this, since 3D is not to be considered some simple
> extension of 2D functionality,
> its a whole different world, modern GPUs are 98% of silicon dedicated
> to 3D processing, most of the current 2D type accel such as QXL
> interface provides are done using 3D engines on most GPUs, and the
> inability to render 3D on the server side of a link with any useful
> speed in current server hw.
>
> So I suppose the main questions to answer up front are:
> 1) Rendering done on same hw as VM is running?
> just use VNC to dump the final answer over the network.

This requires 3D rendering hw on the client, this could be your laptop
or desktop in the office, in which case its probably fine nto so good
if its server hw in a rack.

> 2) Rendering done on the client viewing end of the link where the viewer is,

Have to transport 3D to the client, but more likely the client end
will have 3D hardware to actually render stuff at a reasonable speed.

So I suppose lets discuss ;-)

Dave.
(btw I work for Red Hat but not in this capacity or near virt teams,
this is purely a personal itch).

[Qemu-devel] Re: approaches to 3D virtualisation

[Dave Airlie]

>>
>> Current existing solutions in the area:
>> a) VMware virtual graphics adapter - based on DX9, has an open
>> KMS/Gallium3D driver stack recently released by vmware, has certified
>> Windows drivers and has a documented vGPU interface (it could be
>> documented a lot better)

http://vmware-svga.svn.sourceforge.net/viewvc/vmware-svga/trunk/doc/gpu-wiov.pdf?revision=1

is a good whitepaper on the different 3D virtualisation approaches and why
vmware picked what they did also.

Dave.

[Qemu-devel] Re: approaches to 3D virtualisation

[Juan Quintela]

Dave Airlie <airlied@gmail.com> wrote:
>>>
>>> Current existing solutions in the area:
>>> a) VMware virtual graphics adapter - based on DX9, has an open
>>> KMS/Gallium3D driver stack recently released by vmware, has certified
>>> Windows drivers and has a documented vGPU interface (it could be
>>> documented a lot better)
>
> http://vmware-svga.svn.sourceforge.net/viewvc/vmware-svga/trunk/doc/gpu-wiov.pdf?revision=1
>
> is a good whitepaper on the different 3D virtualisation approaches and why
> vmware picked what they did also.
>
> Dave.

I have zero clue of 3D, but for the qemu part, vmware_vga is the "nicer"
driver.

On the size department:

(virtio-2*)$ wc -l vga.c cirrus_vga.c vmware_vga.c
  2380 vga.c
  3235 cirrus_vga.c
  1211 vmware_vga.c

(notice that part of vga.c is used by both vwmare_vga and cirrus_vga).

On the features, vmware_vga has a nice feature: xrandr already works
(i.e. you can have any modern resolution that you like), cirrus and
vga_std have much less options here.

vmware_vga also has Xvideo (or something) that improves video support.

once telling that, the 3 drivers need a big cleanup.

- vga.c -> VBE is optional (#ifdef'd) but you can't compile it out.

- cirrus_vga -> uses all vga.c code minus the vbe code.  Replicates
  several big functions (switch) for only changing a couple of lines
  there.

- vmware_vga.  Lets say that the way that it embeds an vga is
  _interesting_ to say the less.  Also the vga can be compiled out, but
  not bios/os on earth will boot without its support.

I did some cleanups of the three drivers some months ago, but they need
much more care.  I didn't follow due to time constraints and my zero
knowledge of graphics (hardware/software/...).

Later, Juan.

Re: [Qemu-devel] Re: approaches to 3D virtualisation

[Anthony Liguori]

Juan Quintela wrote:
> Dave Airlie <airlied@gmail.com> wrote:
>   
>>>> Current existing solutions in the area:
>>>> a) VMware virtual graphics adapter - based on DX9, has an open
>>>> KMS/Gallium3D driver stack recently released by vmware, has certified
>>>> Windows drivers and has a documented vGPU interface (it could be
>>>> documented a lot better)
>>>>         
>> http://vmware-svga.svn.sourceforge.net/viewvc/vmware-svga/trunk/doc/gpu-wiov.pdf?revision=1
>>
>> is a good whitepaper on the different 3D virtualisation approaches and why
>> vmware picked what they did also.
>>
>> Dave.
>>     
>
> I have zero clue of 3D, but for the qemu part, vmware_vga is the "nicer"
> driver.
>   

I like the design of the vmware_vga driver but it has one critical 
flaw.  The Windows drivers has a EULA that prohibits their use outside 
of VMware.

Without reasonably licensed Windows drivers, I don't think it's viable.

I'm hoping QXL can fill this niche.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: approaches to 3D virtualisation

[Carl-Daniel Hailfinger]

On 12.12.2009 13:08, Juan Quintela wrote:
> - vmware_vga.  Lets say that the way that it embeds an vga is
>   _interesting_ to say the less.  Also the vga can be compiled out, but
>   not bios/os on earth will boot without its support.
>   

The coreboot developers can boot Linux on all VIA chipsets without VGA
support (no VGA ROM, no VGA interface) and X works just fine. Note that
this works on pretty much every 2.6.x Linux kernel and X with the
Unichrome driver.

Regards,
Carl-Daniel

-- 
Developer quote of the month: 
"We are juggling too many chainsaws and flaming arrows and tigers."

Re: [Qemu-devel] Re: approaches to 3D virtualisation

[Dave Airlie]

On Sun, Dec 13, 2009 at 1:23 AM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Juan Quintela wrote:
>>
>> Dave Airlie <airlied@gmail.com> wrote:
>>
>>>>>
>>>>> Current existing solutions in the area:
>>>>> a) VMware virtual graphics adapter - based on DX9, has an open
>>>>> KMS/Gallium3D driver stack recently released by vmware, has certified
>>>>> Windows drivers and has a documented vGPU interface (it could be
>>>>> documented a lot better)
>>>>>
>>>
>>>
>>> http://vmware-svga.svn.sourceforge.net/viewvc/vmware-svga/trunk/doc/gpu-wiov.pdf?revision=1
>>>
>>> is a good whitepaper on the different 3D virtualisation approaches and
>>> why
>>> vmware picked what they did also.
>>>
>>> Dave.
>>>
>>
>> I have zero clue of 3D, but for the qemu part, vmware_vga is the "nicer"
>> driver.
>>
>
> I like the design of the vmware_vga driver but it has one critical flaw.
>  The Windows drivers has a EULA that prohibits their use outside of VMware.

Good point, I hadn't read the vmware EULA, this does put a spanner in the works,
unless someone is willing to write alternate drivers.

My reason for liking vmware is its not a redesign it all at once
solution, you can bring
up the emulated adapter using known working drivers (the Linux ones
have no EULA),
and confirm it works, if you then want to write Windows drivers
outside the EULA, at
least you have two platforms to validate them on, Someone could in theory write
Windows drivers under VMware itself and we could work in parallel

>
> Without reasonably licensed Windows drivers, I don't think it's viable.
>
> I'm hoping QXL can fill this niche.

It would be nice, its just the design everything at once approach
never sits well
with me, having to do the host side interface, guest vGPU and guest drivers
all at once requires a lot of blame hunting, i.e. where correct fixes go etc.

But yes ideally an open QXL that can challenge VMware would be coolest

Maybe the QXL interface can leverage some of the VMware design at least
rather than reinventing the wheel

Dave.

Re: [Qemu-devel] approaches to 3D virtualisation

[Mark Williamson]

On Saturday 12 December 2009 01:58:02 Dave Airlie wrote:
> So I've been musing on the addition of some sort of 3D passthrough for
> qemu (as I'm sure have lots of ppl)
> 
> But I think the goals of such an addition need to be discussed prior
> to anyone writing a line of code.
> 
> Current existing solutions in the area:
> a) VMware virtual graphics adapter - based on DX9, has an open
> KMS/Gallium3D driver stack recently released by vmware, has certified
> Windows drivers and has a documented vGPU interface (it could be
> documented a lot better)
> 
> b) VirtualBox - seems to be GL based passthrough based on a Chromium
> backend. DX9 support, looks to be done via Wine DX->GL converter built
> into a Windows driver (not confirmed code base is crazy) I'm assuming
> chromium is being used to stream GL through the passthru but this
> definitely requires more investigation.

There may be an option c) in existence.  The OpenTC project (EU funded) 
included a secure UI project and part of that involved a prototype 
paravirtualised 3D driver.  It was done primarily by a research assistant at 
Cambridge University Computer Lab and basically added a paravirtual GPU device 
that essentially proxied Gallium-level commands across a shared memory 
transport.

There's a short paper on the work here, it was implemented for Xen but I doubt 
it's that Xen-specific:
http://www.xen.org/files/xensummit_oracle09/xensummit_chris.pdf

I think there's a longer paper that might be available somewhere.

I assume this is similar to the VMware approach, however as implemented it did 
require a Gallium 3D stack available on the host also in order to actually 
render the stuff.  ISTR the guy who worked on it mentioning that VMware may 
have implemented a Gallium->high level 3D API translator on the host side 
instead, since most hosts aren't using Gallium drivers yet.

The main reason I mention this is that I *thought* the code was meant to be 
open sourced, in which case it might at least be an interesting starting 
point.  I'm not quite sure where it *is* though!

Cheers,
Mark

PS: Whilst I'm braindumping, I also remember a guy implementing an OpenGL 
guest->host passthrough for Qemu back when Fabrice was maintainer, so that 
code should be in the archives somewhere.  Obviously that's not so useful for 
Windows guests, though Fabrice suggested maybe investigating Wine's DX->GL 
translation code.  Another GL virtualisation project is here, which is also 
GL-only (but despite the name is not Xen-only): 
http://www.cs.toronto.edu/~andreslc/xen-gl/

> Now both of these seem to be local only rendering solutions, though I
> might be underselling the vbox/chromium fun.
> 
> Now to add a remoting 3D rendering protocol is where things get ugly
> fast, and very different from current 2D rendering with offscreen
> pixmaps. The major difference is most server farms will not contain
> any 3D hardware and will not want to contain any 3D hw due to power
> considerations. Now if you have a remote protocol, and the client
> disconnects, you have to keep some sort of transaction log, and either
> replay the transactions when a client reconnects, or have some sort of
> sw rendering kick in on the server as a fallback. Now 3D sw rendering
> solutions are insanely slow and quite CPU intensive. VMware are
> working on an llvm based 3D renderer for something like this situation
> but I'm not convinced of how useable it will be.
> 
> Also with remoting you need to come up with a minimum acceptable level
> of 3D you want to expose to guests OSes depending on what the
> capabilities of the client side hw, or refuse connections unless the
> client can run all the features that you've exposed in side the guest.
> This ranges from DX7->DX11, and GL1.5 to GL3.0 situations.
> 
> I'm not sure how ppl who work with VNC see VNC fitting in with this
> sort of protocol, I expect SPICE is better placed with proper design
> to address this. Though it would firstly be nice to design a vGPU
> interface for the guest OS to work against, I lean towards the VMware
> because of the availability of guest drivers, however the Virtualbox
> one is probably also acceptable if anyone can find where it is
> documented.
> 
> This is just a sort of a brain dump from me, but I'd like to get ppl
> talking about this, since 3D is not to be considered some simple
> extension of 2D functionality,
> its a whole different world, modern GPUs are 98% of silicon dedicated
> to 3D processing, most of the current 2D type accel such as QXL
> interface provides are done using 3D engines on most GPUs, and the
> inability to render 3D on the server side of a link with any useful
> speed in current server hw.
> 
> So I suppose the main questions to answer up front are:
> 1) Rendering done on same hw as VM is running?
> just use VNC to dump the final answer over the network.
> 2) Rendering done on the client viewing end of the link where
> 

Re: [Qemu-devel] approaches to 3D virtualisation

[Paul Brook]

On Saturday 12 December 2009, Dave Airlie wrote:
> So I've been musing on the addition of some sort of 3D passthrough for
> qemu (as I'm sure have lots of ppl)

IIUC a typical graphics system consists of several operations:

1) Allocate space for data objects[2] on server[1].
2) Upload data from client to server
3) Issue data processing commands that manipulate (combine/modify) data 
objects. For example a 3D rasterizer takes an image and sets of coordinates, 
and writes pixels to an image.
4) display data object to user.
5) Read data back to client. In modern systems this should almost never 
happen.

I'd expect this to be the same for both 2D and 3D subsystems. The only real 
wart is that some 2D systems do not provide sufficient offload, and some 
processing is still done by the guest CPU. This means (5) is common, and 
you're effectively limited to a local implementation.

With remote rendering the main difference is that you have a relatively high 
latency connection between client and server. If you have more than a few 
round-trips per frame you probably aren't going to get acceptable performance. 
IIUC this is why remote X connections perform so poorly, the protocol is 
effectively synchronous so the client must wait for a response from the server 
before sending the next command.

In practical terms this means that the state of the graphics pipeline should 
not be guest visible. Considering the above pipeline, the only place where 
guest state is visible is (5).  I'd expect that this almost never happens in 
normal circumstances. The fact the SLI/Crossfire setups can operate in AFR 
mode supports this theory.

One prerequisite for isolating the graphics pipeline is that commands may not 
fail. I guess this may require step (1) be a synchronous operation. However 
steps (2), (3) and (4) should be fire-and-forget operations.

If step (2) is defined as completing any time between the issue of the upload 
and the actual use then this allows both local zero-copy and remote explicit-
upload implementations.

A protocol that meets these requirements should be largely transport agnostic. 
While a full paravirtual interface may be desirable to squeeze the last bits 
of performance out, it should be possible to get acceptable performance over 
e.g. TCP, in the same way that the main benefit of virtio block/net drivers is 
simplicity and consistency rather than actual performance[3].

My understanding is that Chromium effectively implements the system described 
above, and I guess the VirtualBox implementation is just a custom transport 
backend and some modesetting tweaks. I have no specific knowledge of the 
VMware implementation.


Once you have remote rendering the next problem is hotplug.
IMO transparently migrating state is not an realistic option. This effectively 
required mirroring all of the server data on the guest. For source data (i.e. 
textures) this is fairly trivial. However for intermediate images (think 
redirected rendering of a 3D application window in a composited environment) 
this is not feasible. You could try to record the commands used to generate 
all intermediate data, however this also becomes infeasible. Command sequences 
may be large, and the original source data may no longer be available.

Instead I suggest adding some sort of "damage" notification whereby the server 
can inform the client that data objects have been lost. When hotplug 
(switching to a different terminal) occurs we immediately complete all pending 
commands and report that all objects have been lost. The guest should then re-
upload and regenerate as necessary and proceed to render the next frame. I'd 
expect that clients already have logic to do this as part of the VRAM handling 
for local video cards.

As long as the guest never tries to read back the image, a null implementation 
is also trivial.


Obviously all this is predicated on having a virtual display driver in the 
guest.

For simple framebuffer devices, and actual VGA hardware, our initial premise 
that GPU state is not guest visible fails. In practice this means that there's 
little scope for doing remote server size acceleration, and you're reduced to 
implementing everything in the client and trying to optimize (2).

Paul

[1] I'm using X client/server terminology. The client is the guest OS and the 
server is the user's terminal.
[2] Data objects include textures/bitmaps, vertex buffers, fragment programs, 
and probably command buffers.
[3] Obviously if you emulate lame hardware like ne2k or IDE then performance 
will suck. However emulation of a high-end NIC of SCSI HBA should get within 
spitting distance of virtio.

Re: [Qemu-devel] approaches to 3D virtualisation

[Stefano Stabellini]

On Sun, 13 Dec 2009, Mark Williamson wrote:
> There may be an option c) in existence.  The OpenTC project (EU funded) 
> included a secure UI project and part of that involved a prototype 
> paravirtualised 3D driver.  It was done primarily by a research assistant at 
> Cambridge University Computer Lab and basically added a paravirtual GPU device 
> that essentially proxied Gallium-level commands across a shared memory 
> transport.
> 
> There's a short paper on the work here, it was implemented for Xen but I doubt 
> it's that Xen-specific:
> http://www.xen.org/files/xensummit_oracle09/xensummit_chris.pdf
> 
> I think there's a longer paper that might be available somewhere.
> 
> I assume this is similar to the VMware approach, however as implemented it did 
> require a Gallium 3D stack available on the host also in order to actually 
> render the stuff.  ISTR the guy who worked on it mentioning that VMware may 
> have implemented a Gallium->high level 3D API translator on the host side 
> instead, since most hosts aren't using Gallium drivers yet.
> 
> The main reason I mention this is that I *thought* the code was meant to be 
> open sourced, in which case it might at least be an interesting starting 
> point.  I'm not quite sure where it *is* though!
> 

Yes, the code is available somewhere and might be a good starting point;
I can retrieve it if you guys are really interested.
However you should be aware that the Windows drivers are completely
missing in this case too.

I think this discussion is really interesting and it should be
split it in two parts: the VGPU interface with 3d support,
and 3d remoting.
Both are difficult topics to discuss and both can (should?) be used
independently from one another: a VGPU infrastructure can be used to
render guest 3d commands on the host (see the VMware paper); 3d remoting
can be used on native too.

I am curious about the VMware MKS thread mentioned in the paper, that is
the backend that receives rendering requests from the guest and executes
them on the host: is that available somewhere?
If it is closed source and cannot be reused for Qemu\KVM\Xen, we might
as well design our own VGPU interface as we like.

Re: [Qemu-devel] approaches to 3D virtualisation

[Avi Kivity]

On 12/14/2009 02:03 PM, Paul Brook wrote:
> [3] Obviously if you emulate lame hardware like ne2k or IDE then 
> performance
> will suck. However emulation of a high-end NIC of SCSI HBA should get within
> spitting distance of virtio.
>    

Practically the only difference performance-wise between virtio and 
high-end controllers is that the ring index is located in a chip 
register on real hardware, and in memory for virtio (plus hardware can 
ask virtio not to notify).

Real hardware assumes guest->controller traffic is cheap, which isn't 
true for virtio/kvm.

-- 
error compiling committee.c: too many arguments to function