[RFC] Xen Virtual Framebuffer

[RFC] Xen Virtual Framebuffer

[Anthony Liguori]

Now that's 3.0.0's out, I thought it would be a good time to bring up 
the topic of framebuffer virtualization.

I threw together a proof-of-concept over the weekend of a simple virtual 
framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d 
buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's 
also a simple message system for keyboard/mouse events.

The first frontend is a GTK widget with python bindings (so it can 
easily be embedded in a larger management app) and a small python app.  
Right now, the VT system seems to work fine and X starts quite happily 
(using the fbdev driver).  Clicking in the app captures the 
mouse/keyboard and ctrl+alt will release the capture.

There's a readme and an hg bundle in the tarball below that explains how 
to set things up.

Some interesting topics in this area are acceleration, whether we should 
implement our own X driver (or just enhance the fbdev driver since it 
uses no acceleration right now), and how to properly expose it over 
something like VNC.

As always, feedback is greatly appreciated.

http://www.cs.utexas.edu/users/aliguori/xen-vfb-20051205.tar.gz

Regards,

Anthony Liguori

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

I've made a page in the Wiki with more details:

http://wiki.xensource.com/xenwiki/VirtualFramebuffer

Regards,

Anthony Liguori

Anthony Liguori wrote:

> Now that's 3.0.0's out, I thought it would be a good time to bring up 
> the topic of framebuffer virtualization.
>
> I threw together a proof-of-concept over the weekend of a simple 
> virtual framebuffer/keyboard/mouse.  The basic design is have a 
> vmalloc()'d buffer in the guest exposed as /dev/fb0 and mmap()'d in 
> dom0.  There's also a simple message system for keyboard/mouse events.
>
> The first frontend is a GTK widget with python bindings (so it can 
> easily be embedded in a larger management app) and a small python 
> app.  Right now, the VT system seems to work fine and X starts quite 
> happily (using the fbdev driver).  Clicking in the app captures the 
> mouse/keyboard and ctrl+alt will release the capture.
>
> There's a readme and an hg bundle in the tarball below that explains 
> how to set things up.
>
> Some interesting topics in this area are acceleration, whether we 
> should implement our own X driver (or just enhance the fbdev driver 
> since it uses no acceleration right now), and how to properly expose 
> it over something like VNC.
>
> As always, feedback is greatly appreciated.
>
> http://www.cs.utexas.edu/users/aliguori/xen-vfb-20051205.tar.gz
>
> Regards,
>
> Anthony Liguori
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

I haven't tried playing with X and Xen, but why doesn't it work to
just treat the multiple domains like a network? You run X in dom0 and
give it full access to the video hardware. Then you ssh into each
domain and start X apps, just like you do when using X remotely.
OpenGL will even work this way and be accelerated (as soon as X fixes
indirect acceleration). This model should let you get apps up from
each domain simultaneously on the X display in dom0.

--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

Hi Jon,

The basic problem is running something like gdm or nautilus which want 
to own the root X window.  You'd have to use something like Xnest.  It 
doesn't help much with VT switching either which would be a really nice 
feature to has.

However, using remote X is a good option for a number of use-cases so 
it's certainly appropriate for certain environments.

Regards,

Anthony Liguori

Jon Smirl wrote:

>I haven't tried playing with X and Xen, but why doesn't it work to
>just treat the multiple domains like a network? You run X in dom0 and
>give it full access to the video hardware. Then you ssh into each
>domain and start X apps, just like you do when using X remotely.
>OpenGL will even work this way and be accelerated (as soon as X fixes
>indirect acceleration). This model should let you get apps up from
>each domain simultaneously on the X display in dom0.
>
>--
>Jon Smirl
>jonsmirl@gmail.com
>
>  
>

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/5/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> Hi Jon,
>
> The basic problem is running something like gdm or nautilus which want
> to own the root X window.  You'd have to use something like Xnest.  It
> doesn't help much with VT switching either which would be a really nice
> feature to has.

Xnest is the best solution for multiple root windows. With virtual
framebuffers you are just building Xnest again via a different
mechanism. Xnest has the advantage that you can use hardware
acceleration from the X server in dom0.

> However, using remote X is a good option for a number of use-cases so
> it's certainly appropriate for certain environments.
>
> Regards,
>
> Anthony Liguori
>
> Jon Smirl wrote:
>
> >I haven't tried playing with X and Xen, but why doesn't it work to
> >just treat the multiple domains like a network? You run X in dom0 and
> >give it full access to the video hardware. Then you ssh into each
> >domain and start X apps, just like you do when using X remotely.
> >OpenGL will even work this way and be accelerated (as soon as X fixes
> >indirect acceleration). This model should let you get apps up from
> >each domain simultaneously on the X display in dom0.
> >
> >--
> >Jon Smirl
> >jonsmirl@gmail.com
> >
> >
> >
>
>


--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

Jon Smirl wrote:

>On 12/5/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
>  
>
>>Hi Jon,
>>
>>The basic problem is running something like gdm or nautilus which want
>>to own the root X window.  You'd have to use something like Xnest.  It
>>doesn't help much with VT switching either which would be a really nice
>>feature to has.
>>    
>>
>
>Xnest is the best solution for multiple root windows. With virtual
>framebuffers you are just building Xnest again via a different
>mechanism. Xnest has the advantage that you can use hardware
>acceleration from the X server in dom0.
>  
>
I like Xnest quite a lot.  We need something that's available when X 
isn't though (for the VT console for instance).

You certainly don't have to run X on /dev/fb0.  It's mostly to make 
text-mode and distro installation a whole lot more user friendly.

Regards,

Anthony Liguori

>>However, using remote X is a good option for a number of use-cases so
>>it's certainly appropriate for certain environments.
>>
>>Regards,
>>
>>Anthony Liguori
>>
>>Jon Smirl wrote:
>>
>>    
>>
>>>I haven't tried playing with X and Xen, but why doesn't it work to
>>>just treat the multiple domains like a network? You run X in dom0 and
>>>give it full access to the video hardware. Then you ssh into each
>>>domain and start X apps, just like you do when using X remotely.
>>>OpenGL will even work this way and be accelerated (as soon as X fixes
>>>indirect acceleration). This model should let you get apps up from
>>>each domain simultaneously on the X display in dom0.
>>>
>>>--
>>>Jon Smirl
>>>jonsmirl@gmail.com
>>>
>>>
>>>
>>>      
>>>
>>    
>>
>
>
>--
>Jon Smirl
>jonsmirl@gmail.com
>
>  
>

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/5/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> I like Xnest quite a lot.  We need something that's available when X
> isn't though (for the VT console for instance).
>
> You certainly don't have to run X on /dev/fb0.  It's mostly to make
> text-mode and distro installation a whole lot more user friendly.

Can't distro install into a domain be handled on a virtual serial line
and Linux terminal emulation? This seems like a lot of work to go
through to get something like Redhat's graphical boot working.

>
> Regards,
>
> Anthony Liguori
>
> >>However, using remote X is a good option for a number of use-cases so
> >>it's certainly appropriate for certain environments.
> >>
> >>Regards,
> >>
> >>Anthony Liguori
> >>
> >>Jon Smirl wrote:
> >>
> >>
> >>
> >>>I haven't tried playing with X and Xen, but why doesn't it work to
> >>>just treat the multiple domains like a network? You run X in dom0 and
> >>>give it full access to the video hardware. Then you ssh into each
> >>>domain and start X apps, just like you do when using X remotely.
> >>>OpenGL will even work this way and be accelerated (as soon as X fixes
> >>>indirect acceleration). This model should let you get apps up from
> >>>each domain simultaneously on the X display in dom0.
> >>>
> >>>--
> >>>Jon Smirl
> >>>jonsmirl@gmail.com
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
> >--
> >Jon Smirl
> >jonsmirl@gmail.com
> >
> >
> >
>
>


--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

Jon Smirl wrote:

>On 12/5/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
>  
>
>>I like Xnest quite a lot.  We need something that's available when X
>>isn't though (for the VT console for instance).
>>
>>You certainly don't have to run X on /dev/fb0.  It's mostly to make
>>text-mode and distro installation a whole lot more user friendly.
>>    
>>
>
>Can't distro install into a domain be handled on a virtual serial line
>and Linux terminal emulation? This seems like a lot of work to go
>through to get something like Redhat's graphical boot working.
>  
>
I added a section to the Wiki page for discussion of various approachs 
and the pros/cons.  Feel free to extend this information.

So far, the framebuffer has not been all that much code (only a few 
hundred lines) so I think the benefit from the increased usuability is 
well worth it.

Thanks for the feedback,

Anthony Liguori

>>Regards,
>>
>>Anthony Liguori
>>
>>    
>>
>>>>However, using remote X is a good option for a number of use-cases so
>>>>it's certainly appropriate for certain environments.
>>>>
>>>>Regards,
>>>>
>>>>Anthony Liguori
>>>>
>>>>Jon Smirl wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>I haven't tried playing with X and Xen, but why doesn't it work to
>>>>>just treat the multiple domains like a network? You run X in dom0 and
>>>>>give it full access to the video hardware. Then you ssh into each
>>>>>domain and start X apps, just like you do when using X remotely.
>>>>>OpenGL will even work this way and be accelerated (as soon as X fixes
>>>>>indirect acceleration). This model should let you get apps up from
>>>>>each domain simultaneously on the X display in dom0.
>>>>>
>>>>>--
>>>>>Jon Smirl
>>>>>jonsmirl@gmail.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>        
>>>>
>>>--
>>>Jon Smirl
>>>jonsmirl@gmail.com
>>>
>>>
>>>
>>>      
>>>
>>    
>>
>
>
>--
>Jon Smirl
>jonsmirl@gmail.com
>
>  
>

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

After thinking about this for a while, wouldn't Xen be better off with
a virtual VGA device instead of a virtual fbdev? The virtual VGA
device will work for other operating systems as well as Linux. 
Implementing a VESA BIOS may be better than emulating VGA.
http://www.vesa.org/public/VBE/vbecore3.pdf

I'm not exactly sure how Redhat install boots, but when it first comes
up it has to be using VGA mode or vesafb since it doesn't know what
video hardware the box has. (What does it do if there is no video card
installed? serial line?) After a couple of questions in VGA mode it
quickly starts an X server and goes graphical.

On a server box with only a serial line and no video hardware, the
install has to stay in text mode using terminal emulation.

The problem is when the distro wants to start the X sever and go into
graphics mode. That isn't going to work very well with the Xen model.
You have three choices:
1) Treat it like a server and use the virtual serial line with
terminal emulation
2) virtual VGA - Let it start the xserver and run the xserver on the
virtual VGA device.
3) or create a new install mode where the installer starts a network
device, waits for an ssh connect, and then run a remote X based
installation.

The third mode would be best for Xen but I don't believe an
distribution works that way currently. #3 is a minor change to
existing installers. One problem I see is knowing what address to ssh
into. The address could be sent to the serial console as part of the
boot process. If you want to get fancy implement a new terminal escape
sequence that could be sent down the serial line which will trigger
the ssh connect and run the graphical installer.

--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

Jon Smirl wrote:

>After thinking about this for a while, wouldn't Xen be better off with
>a virtual VGA device instead of a virtual fbdev? The virtual VGA
>device will work for other operating systems as well as Linux. 
>Implementing a VESA BIOS may be better than emulating VGA.
>http://www.vesa.org/public/VBE/vbecore3.pdf
>  
>
This is how Qemu does it so it's what we do for VT.  The VMI spec 
(VMware paravirtual spec) assumes that you'll be doing device emulation 
and calls for an emulated PCI bus.  Xen achieves really good performance 
though by avoiding device emulation.  Native speed device emulation is 
an active area of research though so this might not always be the case :-)

We don't currently do that in Xen though so it would be a considerable 
amount of work to emulate a PCI bus and a VGA device (not to mention an 
emu86 to be able to run the BIOS).

Regards,

Anthony Liguori

>--
>Jon Smirl
>jonsmirl@gmail.com
>
>  
>

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/5/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> Jon Smirl wrote:
>
> >After thinking about this for a while, wouldn't Xen be better off with
> >a virtual VGA device instead of a virtual fbdev? The virtual VGA
> >device will work for other operating systems as well as Linux.
> >Implementing a VESA BIOS may be better than emulating VGA.
> >http://www.vesa.org/public/VBE/vbecore3.pdf
> >
> >
> This is how Qemu does it so it's what we do for VT.  The VMI spec
> (VMware paravirtual spec) assumes that you'll be doing device emulation
> and calls for an emulated PCI bus.  Xen achieves really good performance
> though by avoiding device emulation.  Native speed device emulation is
> an active area of research though so this might not always be the case :-)
>
> We don't currently do that in Xen though so it would be a considerable
> amount of work to emulate a PCI bus and a VGA device (not to mention an
> emu86 to be able to run the BIOS).

VGA devices don't live on the PCI bus, they are ISA legacy devices at
fixed IO ports/RAM address. But VESA is a better solution than the VGA
device.

Does Xen supply a BIOS into the virtual machine? If so, just implement
the VESA entry points. Most of the Xen-based VESA entry points will do
nothing, but they can't return the not-implemented error. This is very
similar to what you are doing with a virtual fbdev, but the VESA
scheme will work for Windows too. Linux already has a vesafb driver
that will use the entry points you provide.

However, I am assuming that Windows/Linux will fallback to using VESA
calls if they don't find a physical VGA device. There isn't a simple
way to test this since every video card that implements VESA also
implements VGA. If you want to get errors from Linux while it is still
in real mode you need to implement the BIOS INT 10 interface.

--
Jon Smirl
jonsmirl@gmail.com

答复: [Xen-devel] [RFC] Xen Virtual Framebuffer

[苗枫]

I think this topic must be subject to Para-Virtualized Guest Linux (or some
other os), for unmodified guest os on a VTx box, do you have any suggestion
for graphic acceleration?

And, what if we adjust the qemu-dm to accommodate both Para-dom and VTx-dom?

 

Miao Feng

Star SoftComm(China) Ltd


-----邮件原件-----
发件人: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] 代表 Jon Smirl
发送时间: 2005年12月6日 11:13
收件人: Anthony Liguori
抄送: xen-devel; Antonino A. Daplas
主题: Re: [Xen-devel] [RFC] Xen Virtual Framebuffer

On 12/5/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> Jon Smirl wrote:
>
> >After thinking about this for a while, wouldn't Xen be better off with
> >a virtual VGA device instead of a virtual fbdev? The virtual VGA
> >device will work for other operating systems as well as Linux.
> >Implementing a VESA BIOS may be better than emulating VGA.
> >http://www.vesa.org/public/VBE/vbecore3.pdf
> >
> >
> This is how Qemu does it so it's what we do for VT.  The VMI spec
> (VMware paravirtual spec) assumes that you'll be doing device emulation
> and calls for an emulated PCI bus.  Xen achieves really good performance
> though by avoiding device emulation.  Native speed device emulation is
> an active area of research though so this might not always be the case :-)
>
> We don't currently do that in Xen though so it would be a considerable
> amount of work to emulate a PCI bus and a VGA device (not to mention an
> emu86 to be able to run the BIOS).

VGA devices don't live on the PCI bus, they are ISA legacy devices at
fixed IO ports/RAM address. But VESA is a better solution than the VGA
device.

Does Xen supply a BIOS into the virtual machine? If so, just implement
the VESA entry points. Most of the Xen-based VESA entry points will do
nothing, but they can't return the not-implemented error. This is very
similar to what you are doing with a virtual fbdev, but the VESA
scheme will work for Windows too. Linux already has a vesafb driver
that will use the entry points you provide.

However, I am assuming that Windows/Linux will fallback to using VESA
calls if they don't find a physical VGA device. There isn't a simple
way to test this since every video card that implements VESA also
implements VGA. If you want to get errors from Linux while it is still
in real mode you need to implement the BIOS INT 10 interface.

--
Jon Smirl
jonsmirl@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: 答复: [Xen-devel] [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

苗枫 wrote:

>I think this topic must be subject to Para-Virtualized Guest Linux (or some
>other os), for unmodified guest os on a VTx box, do you have any suggestion
>for graphic acceleration?
>  
>
You're right, this is PV guest framebuffer virtualization.

There was a discussion recently on #qemu about acceleration. There are
few possibilities. The first would be to use 2d acceleration instead of
emulating in software (the cirrus card has all the standard fill, copy,
etc 2d ops). Then there's the topic of 3d acceleration.

This is a pretty big task and probably something you want to do with a
paravirtual driver. If we had our own X driver, we could implement the
appropriate pass-thru stuff. I don't know enough about Windows internals
to comment intelligently on it.

>And, what if we adjust the qemu-dm to accommodate both Para-dom and VTx-dom?
>  
>
This is something I'm very interested in. Unifying VT/PV management will
definitely be an important usability feature for future versions of Xen.
Haven't though much about it myself though.

Regards,

Anthony Liguori

> 
>
>Miao Feng
>
>Star SoftComm(China) Ltd
>
>
>-----邮件原件-----
>发件人: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] 代表 Jon Smirl
>发送时间: 2005年12月6日 11:13
>收件人: Anthony Liguori
>抄送: xen-devel; Antonino A. Daplas
>主题: Re: [Xen-devel] [RFC] Xen Virtual Framebuffer
>
>On 12/5/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
>  
>
>>Jon Smirl wrote:
>>
>>    
>>
>>>After thinking about this for a while, wouldn't Xen be better off with
>>>a virtual VGA device instead of a virtual fbdev? The virtual VGA
>>>device will work for other operating systems as well as Linux.
>>>Implementing a VESA BIOS may be better than emulating VGA.
>>>http://www.vesa.org/public/VBE/vbecore3.pdf
>>>
>>>
>>>      
>>>
>>This is how Qemu does it so it's what we do for VT.  The VMI spec
>>(VMware paravirtual spec) assumes that you'll be doing device emulation
>>and calls for an emulated PCI bus.  Xen achieves really good performance
>>though by avoiding device emulation.  Native speed device emulation is
>>an active area of research though so this might not always be the case :-)
>>
>>We don't currently do that in Xen though so it would be a considerable
>>amount of work to emulate a PCI bus and a VGA device (not to mention an
>>emu86 to be able to run the BIOS).
>>    
>>
>
>VGA devices don't live on the PCI bus, they are ISA legacy devices at
>fixed IO ports/RAM address. But VESA is a better solution than the VGA
>device.
>
>Does Xen supply a BIOS into the virtual machine? If so, just implement
>the VESA entry points. Most of the Xen-based VESA entry points will do
>nothing, but they can't return the not-implemented error. This is very
>similar to what you are doing with a virtual fbdev, but the VESA
>scheme will work for Windows too. Linux already has a vesafb driver
>that will use the entry points you provide.
>
>However, I am assuming that Windows/Linux will fallback to using VESA
>calls if they don't find a physical VGA device. There isn't a simple
>way to test this since every video card that implements VESA also
>implements VGA. If you want to get errors from Linux while it is still
>in real mode you need to implement the BIOS INT 10 interface.
>
>--
>Jon Smirl
>jonsmirl@gmail.com
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>
>
>
>  
>

Re: [RFC] Xen Virtual Framebuffer

[Tracy R Reed]

Jon Smirl wrote:
> I haven't tried playing with X and Xen, but why doesn't it work to
> just treat the multiple domains like a network? You run X in dom0 and

I currently have one big machine (dual core AMD64, 2G RAM, etc) which I
use for my desktop, my webserver, my mailserver, and I may add other
domains eventually. Dom0 very special in terms of security and system
maintainability. Ideally I would like to be able to put a mimimal
install of a long-term stable distro like CentOS4 or RHEL4 in dom0 and
use that domain only for managing my domU's. In fact I can see someone
someday creating a special little distro just for running in dom0. I
would like to be able to run my desktop/workstation environment in a
domU with X and everything. I currently have to run X in dom0 for
reasons already mentioned in this thread which means I clutter up my
dom0 with a zillion files and do unsafe things in it like read email and
browse the web. When I upgrade my desktop install it dramatically
affects dom0 which potentially destabilizes everything.

Ideally I would install a dom0 when I get a new piece of hardware (or
better yet, manufacturers ship it with dom0 preconfigured standard on
all of their machines and then optionally the OS of your choice in a
domU and you never know dom0 is there unless you care about such
things!) and then never touch it for the life of the hardware.

The beauty of Xen is that your hardware becomes a shared resource to run
whatever combination of OS's you want on allowing you to get better
utilization of all that precious CPU/RAM and you get good security
separation as a bonus. Aside from being able to run X in a domU there is
only one other thing I can think of asking for out of Xen which is the
subject of my next email...

-- 
Tracy R Reed
http://copilotconsulting.com
1-877-MY-COPILOT

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/9/05, Tracy R Reed <treed@copilotconsulting.com> wrote:
> Jon Smirl wrote:
> > I haven't tried playing with X and Xen, but why doesn't it work to
> > just treat the multiple domains like a network? You run X in dom0 and
>
> I currently have one big machine (dual core AMD64, 2G RAM, etc) which I
> use for my desktop, my webserver, my mailserver, and I may add other
> domains eventually. Dom0 very special in terms of security and system
> maintainability. Ideally I would like to be able to put a mimimal
> install of a long-term stable distro like CentOS4 or RHEL4 in dom0 and
> use that domain only for managing my domU's. In fact I can see someone
> someday creating a special little distro just for running in dom0. I
> would like to be able to run my desktop/workstation environment in a
> domU with X and everything. I currently have to run X in dom0 for
> reasons already mentioned in this thread which means I clutter up my
> dom0 with a zillion files and do unsafe things in it like read email and
> browse the web. When I upgrade my desktop install it dramatically
> affects dom0 which potentially destabilizes everything.
>
> Ideally I would install a dom0 when I get a new piece of hardware (or
> better yet, manufacturers ship it with dom0 preconfigured standard on
> all of their machines and then optionally the OS of your choice in a
> domU and you never know dom0 is there unless you care about such
> things!) and then never touch it for the life of the hardware.
>
> The beauty of Xen is that your hardware becomes a shared resource to run
> whatever combination of OS's you want on allowing you to get better
> utilization of all that precious CPU/RAM and you get good security
> separation as a bonus. Aside from being able to run X in a domU there is
> only one other thing I can think of asking for out of Xen which is the
> subject of my next email...

You don't have to run the X server in dom0. You can run it in any
domain that has sole control of the graphics hardware.  The core
problem is that only one domain can have control of the video
hardware, there is no realistic way to virtualize a 3D engine.

My knowledge of Xen is limited, is there currently a way to assign
control of a specific PCI device to a domain other than zero? If so,
assign the graphics card to that domain and run X there. Then use the
remote X protocol to access the other domains.

Linux implements unix domain sockets for high performance on a local
machine, does Xen provide an equivalent for interdomain communication?
If so X will perform better over this kind of connection instead of a
full TCP stack.

If you have assigned control of the video device to a domain other
that zero that implies that dom0 does not have a display anymore. It
there a way to use a network connection to talk to dom0 and control
the user domains? If so use xterm from the user domain running X to
connect to the network interface of dom0 and run the control program.

--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Jacob Gorm Hansen]

On 12/11/05, Jon Smirl <jonsmirl@gmail.com> wrote:

> You don't have to run the X server in dom0. You can run it in any
> domain that has sole control of the graphics hardware.  The core
> problem is that only one domain can have control of the video
> hardware, there is no realistic way to virtualize a 3D engine.

Even if it is impossible, I am currently attempting to do just that.
At http://www.diku.dk/~jacobg/gfx you can see a screen shot of 3 Xen
VM's displaying hardware-accelerated OpenGL to a shared server in
dom0.

best regards,
Jacob

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/16/05, Jacob Gorm Hansen <jacobg@diku.dk> wrote:
> On 12/11/05, Jon Smirl <jonsmirl@gmail.com> wrote:
>
> > You don't have to run the X server in dom0. You can run it in any
> > domain that has sole control of the graphics hardware.  The core
> > problem is that only one domain can have control of the video
> > hardware, there is no realistic way to virtualize a 3D engine.
>
> Even if it is impossible, I am currently attempting to do just that.
> At http://www.diku.dk/~jacobg/gfx you can see a screen shot of 3 Xen
> VM's displaying hardware-accelerated OpenGL to a shared server in
> dom0.

You can virtualize a single vendor's card. You can't really virtualize
3D in general because the cards are so different, there is no common
3D interface to virtualize. If you are going to design a common
virtual 3D interface you might as well use the exiting OpenGL/GLX
protocol rather than design something new. You'll find that GLX is
more communications efficient than virtualizing at the low level.

For example on Linux all of the 3D drivers use the DRM interface. But
the commands sent to the DRM interface are completely different for
each vendor's hardware. dom0 would be forced into decoding vendor
specific commands to stop a user DOM from moving the scanout buffer or
changing the mode. The Linux DRM code does contain a command verifier
which would help in decoding the packets. Another big, complex problem
is that the 3D hardware uses AGP space and DMA. These will have to be
virtualized too.

Another approach would be to use the concepts from Linux direct
rendering and to apply them to the set of VMs. With direct rendering
all of the domains are allowed direct access to the video hardware and
a sharing protocol is provided. In this scheme there is shared memory
between all of the VMs. The shared memory holds the DRM lock and video
memory management data. Each of the VMs would direct render into a
non-visible framebuffer. A master domain would then run composite and
build the screen. You still have to virtualize DMA and AGP. This isn't
really a virtual device, it is a shared device. With virtual devices a
misbehaving domain can't take down the device, in this model a
misbehaving domain can crash the device for all other domains.

> best regards,
> Jacob
>

--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Jacob Gorm Hansen]

On 12/16/05, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 12/16/05, Jacob Gorm Hansen <jacobg@diku.dk> wrote:
> > On 12/11/05, Jon Smirl <jonsmirl@gmail.com> wrote:
> >
> > > You don't have to run the X server in dom0. You can run it in any
> > > domain that has sole control of the graphics hardware.  The core
> > > problem is that only one domain can have control of the video
> > > hardware, there is no realistic way to virtualize a 3D engine.
> >
> > Even if it is impossible, I am currently attempting to do just that.
> > At http://www.diku.dk/~jacobg/gfx you can see a screen shot of 3 Xen
> > VM's displaying hardware-accelerated OpenGL to a shared server in
> > dom0.
>
> You can virtualize a single vendor's card. You can't really virtualize
> 3D in general because the cards are so different, there is no common
> 3D interface to virtualize. If you are going to design a common
> virtual 3D interface you might as well use the exiting OpenGL/GLX
> protocol rather than design something new. You'll find that GLX is
> more communications efficient than virtualizing at the low level.

My software so far runs on top of pure OpenGL, though this far I have
only gotten the ATI drivers running under Xen (mostly a case of
changing virt_to_phys() into virt_to_bus() a few random places in the
open source wrapper).

Though I was inspired by GLX I have opted not to base my work on it,
because I would prefer not to rely on anything X11-related, and
because it is a wire protocol which seems inefficient compared to a
shared memory protocol, especially when sending large objects such as
textures.

Jacob

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/19/05, Jacob Gorm Hansen <jacobg@diku.dk> wrote:
> On 12/16/05, Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 12/16/05, Jacob Gorm Hansen <jacobg@diku.dk> wrote:
> > > On 12/11/05, Jon Smirl <jonsmirl@gmail.com> wrote:
> > >
> > > > You don't have to run the X server in dom0. You can run it in any
> > > > domain that has sole control of the graphics hardware.  The core
> > > > problem is that only one domain can have control of the video
> > > > hardware, there is no realistic way to virtualize a 3D engine.
> > >
> > > Even if it is impossible, I am currently attempting to do just that.
> > > At http://www.diku.dk/~jacobg/gfx you can see a screen shot of 3 Xen
> > > VM's displaying hardware-accelerated OpenGL to a shared server in
> > > dom0.
> >
> > You can virtualize a single vendor's card. You can't really virtualize
> > 3D in general because the cards are so different, there is no common
> > 3D interface to virtualize. If you are going to design a common
> > virtual 3D interface you might as well use the exiting OpenGL/GLX
> > protocol rather than design something new. You'll find that GLX is
> > more communications efficient than virtualizing at the low level.
>
> My software so far runs on top of pure OpenGL, though this far I have
> only gotten the ATI drivers running under Xen (mostly a case of
> changing virt_to_phys() into virt_to_bus() a few random places in the
> open source wrapper).

Are you aware of the EGL API proposal?
http://www.freedesktop.org/wiki/EGL

There has also been a lot of discussion around writing an new display
server centered around OpenGL instead of X.  It is discussed at the
bottom of this article:
http://www.freedesktop.org/~jonsmirl/graphics.html

A standalone X server could still use the GLX protocol. If you look at
the mesa source code you will find that it is all automatically
generated. It is certainly possible to build a new OpenGL only server
that reuses the GLX code.

>
> Though I was inspired by GLX I have opted not to base my work on it,
> because I would prefer not to rely on anything X11-related, and
> because it is a wire protocol which seems inefficient compared to a
> shared memory protocol, especially when sending large objects such as
> textures.

X uses the same base protocol on the wire as local. But the protocol
can be extended and parts of it overridden, for example the shared
memory extension is used to avoid the texture copies.

The advantage to this scheme is that any pair of machines can always
communicate using the base protocol. It the client/server are both
local they will be able to negotiate the shared memory extension and
upgrade their communications.

>
> Jacob
>


--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[M.A. Williamson]

Wow!  That was fast work ;-)

Sounds like really good stuff. Is there any chance of a couple of 
screenshots?

Have you had a chance to check out comparable X.org drivers? How hairy are 
they to put together? I agree it seems nicest (in terms of compatibility 
with Xen-unaware X servers) to use a kernel-based framebuffer as a basis 
for things (as opposed to having X talk to the virtual display directly). 
It also allows us automagic compatibility with toolkits and apps that can 
render to the framebuffer directly.

Side note: it'd be really nice to be able to have an option to export the 
guest framebuffer display over VNC using libvncserver. The intent would be 
to allow management software on another node to access domain consoles, 
without needing the domain to mess about running VNC (or indeed, having a 
working network setup). I'm sure you've thought of that, though ;-)

Other side note: it'd be really neat to be able to support copy-and-paste 
operations from text mode consoles to other windows in dom0. Does the 
kernel have hooks in the right places to make this workable?

Cheers,
Mark

On Dec 5 2005, Anthony Liguori wrote:

>Now that's 3.0.0's out, I thought it would be a good time to bring up 
>the topic of framebuffer virtualization.
>
>I threw together a proof-of-concept over the weekend of a simple virtual 
>framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d 
>buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's 
>also a simple message system for keyboard/mouse events.
>
>The first frontend is a GTK widget with python bindings (so it can 
>easily be embedded in a larger management app) and a small python app.  
>Right now, the VT system seems to work fine and X starts quite happily 
>(using the fbdev driver).  Clicking in the app captures the 
>mouse/keyboard and ctrl+alt will release the capture.
>
>There's a readme and an hg bundle in the tarball below that explains how 
>to set things up.
>
>Some interesting topics in this area are acceleration, whether we should 
>implement our own X driver (or just enhance the fbdev driver since it 
>uses no acceleration right now), and how to properly expose it over 
>something like VNC.
>
>As always, feedback is greatly appreciated.
>
>http://www.cs.utexas.edu/users/aliguori/xen-vfb-20051205.tar.gz
>
>Regards,
>
>Anthony Liguori
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

M.A. Williamson wrote:

> Wow!  That was fast work ;-)

Linux rocks :-)  The fb driver infrastructure is really easy to work with.

> Sounds like really good stuff. Is there any chance of a couple of 
> screenshots?

Yeah, I can post some (I'll link on the wiki page) tomorrow as I forgot 
to re-enable VNC on my dev machine so I have to wait to head back in the 
office.  It looks a lot like VT or Qemu right now as you can imagine :-)

> Have you had a chance to check out comparable X.org drivers? How hairy 
> are they to put together? I agree it seems nicest (in terms of 
> compatibility with Xen-unaware X servers) to use a kernel-based 
> framebuffer as a basis for things (as opposed to having X talk to the 
> virtual display directly). It also allows us automagic compatibility 
> with toolkits and apps that can render to the framebuffer directly.

What's nice is that the fbdev device maps the memory directly to 
userspace, so you don't take a hit at all by going through the kernel 
interface.  My current line of thinking is to use an fb device with 
special device-specific ioctls for the couple non-standard accelerations 
we could implement.

The X.org code doesn't look so bad.  Interestingly, VMware has a driver 
in the X.org tree so I don't think we'll have a problem just working 
with those guys to get our own driver :-)

> Side note: it'd be really nice to be able to have an option to export 
> the guest framebuffer display over VNC using libvncserver. The intent 
> would be to allow management software on another node to access domain 
> consoles, without needing the domain to mess about running VNC (or 
> indeed, having a working network setup). I'm sure you've thought of 
> that, though ;-)

Absolutely.  libvncserver is not a pleasant library but a reasonable 
place to start.  We really need to implement a few accelerations first 
before VNC is a reasonable option (at least copyrect) since otherwise 
we'll have to do a lot of computation to figure out what portions of the 
screen have changed.

> Other side note: it'd be really neat to be able to support 
> copy-and-paste operations from text mode consoles to other windows in 
> dom0. Does the kernel have hooks in the right places to make this 
> workable?

I'm not sure.  I think there may be some clever approachs to it.  If 
nothing else, some patches to gpm could achieve the desired effect.  
copy/paste would be an awesome feature to have so I'm definitely 
interested in making it work.

The same is true for drag and drop :-)  There are a lot of cool things 
we can do...

Regards,

Anthony Liguori

> Cheers,
> Mark
>
> On Dec 5 2005, Anthony Liguori wrote:
>
>> Now that's 3.0.0's out, I thought it would be a good time to bring up 
>> the topic of framebuffer virtualization.
>>
>> I threw together a proof-of-concept over the weekend of a simple 
>> virtual framebuffer/keyboard/mouse.  The basic design is have a 
>> vmalloc()'d buffer in the guest exposed as /dev/fb0 and mmap()'d in 
>> dom0.  There's also a simple message system for keyboard/mouse events.
>>
>> The first frontend is a GTK widget with python bindings (so it can 
>> easily be embedded in a larger management app) and a small python 
>> app.  Right now, the VT system seems to work fine and X starts quite 
>> happily (using the fbdev driver).  Clicking in the app captures the 
>> mouse/keyboard and ctrl+alt will release the capture.
>>
>> There's a readme and an hg bundle in the tarball below that explains 
>> how to set things up.
>>
>> Some interesting topics in this area are acceleration, whether we 
>> should implement our own X driver (or just enhance the fbdev driver 
>> since it uses no acceleration right now), and how to properly expose 
>> it over something like VNC.
>>
>> As always, feedback is greatly appreciated.
>>
>> http://www.cs.utexas.edu/users/aliguori/xen-vfb-20051205.tar.gz
>>
>> Regards,
>>
>> Anthony Liguori
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>

Re: [RFC] Xen Virtual Framebuffer

[Gerd Knorr]

> I threw together a proof-of-concept over the weekend of a simple virtual 
> framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d 
> buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's 
> also a simple message system for keyboard/mouse events.

Cool.

> Some interesting topics in this area are acceleration, whether we should 
> implement our own X driver (or just enhance the fbdev driver since it 
> uses no acceleration right now), and how to properly expose it over 
> something like VNC.

Hmm, I don't think acceleration is that important here.  2D acceleration 
likely disappeares in graphics hardware soon.  I'd say a simple, stupid 
framebuffer for installs is perfectly fine, for accelerated graphics its 
probably more useful to skip 2D altogether and look at OpenGL 
virtualization instead.

just my 2 cent,

   Gerd

Re: [RFC] Xen Virtual Framebuffer

[Jacob Gorm Hansen]

On 12/6/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> Now that's 3.0.0's out, I thought it would be a good time to bring up
> the topic of framebuffer virtualization.
>
> I threw together a proof-of-concept over the weekend of a simple virtual
> framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d
> buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's
> also a simple message system for keyboard/mouse events.

hi,

A virtual framebuffer is probably a good idea, as this is an easy way
of getting legacy software like X11 running on Xen. My personal
interest is in getting xenified applications running with features and
performance comparable to modern systems such as OSX Aqua or the new
display stuff in Vista.

For the last couple of weeks I have been trying to build an
accelerated OpenGL multiplexer for Xen. I serialize a stream of OpenGL
commands to one or more page ranges that I share with a viewer in
dom0, and there I interpret the streams, so that the result ends up on
the shared screen. Clients can utilize OpenGL display lists to avoid
the cost of interpretation.

Currently, I can run glGears and similar demos, including stuff that
uses texturing, at very decent speeds. I use the closed ATI drivers
for the FireGL in my T42p laptop. I have a Xen patch for the open
parts of the driver available if anyone is interested. It would also
be great to get the open source r300 driver working under Xen, but so
far I have been unable to make that happen (the old monolithic x.org
tree was a major pain to work with, hopefully things will be better in
the future).

This work is in some ways inspired by an older 2D display system I
built, for which I cannot release the source for legal reasons. With
that system, I was able to support 12 concurrent MPlayers showing a
512x304 DIVX movie at 25fps, on a 3.2GHz machine with a 128MB Radeon
9600SE card. I expect my new system to perform at least a little
better than that, while also supporting 3D.

It seems at this point the only way to get HW-accelerated OpenGL
running on Linux is through X11 / GLX, hopefully in the longer term I
will be able to remove X from dom0, in order to reduce the size and
complexity of the base installation.

Best regards,
Jacob

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

When Ian gets finished fixing the X server to properly handle remote
OpenGL with accelerated calls (instead of software as it now does) you
won't need to write something custom like this.. I added Ian to the CC
list, maybe he will give us an update on when direct rendering for
remote connections will be ready.

On 12/6/05, Jacob Gorm Hansen <jacobg@diku.dk> wrote:
> On 12/6/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> > Now that's 3.0.0's out, I thought it would be a good time to bring up
> > the topic of framebuffer virtualization.
> >
> > I threw together a proof-of-concept over the weekend of a simple virtual
> > framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d
> > buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's
> > also a simple message system for keyboard/mouse events.
>
> hi,
>
> A virtual framebuffer is probably a good idea, as this is an easy way
> of getting legacy software like X11 running on Xen. My personal
> interest is in getting xenified applications running with features and
> performance comparable to modern systems such as OSX Aqua or the new
> display stuff in Vista.
>
> For the last couple of weeks I have been trying to build an
> accelerated OpenGL multiplexer for Xen. I serialize a stream of OpenGL
> commands to one or more page ranges that I share with a viewer in
> dom0, and there I interpret the streams, so that the result ends up on
> the shared screen. Clients can utilize OpenGL display lists to avoid
> the cost of interpretation.
>
> Currently, I can run glGears and similar demos, including stuff that
> uses texturing, at very decent speeds. I use the closed ATI drivers
> for the FireGL in my T42p laptop. I have a Xen patch for the open
> parts of the driver available if anyone is interested. It would also
> be great to get the open source r300 driver working under Xen, but so
> far I have been unable to make that happen (the old monolithic x.org
> tree was a major pain to work with, hopefully things will be better in
> the future).
>
> This work is in some ways inspired by an older 2D display system I
> built, for which I cannot release the source for legal reasons. With
> that system, I was able to support 12 concurrent MPlayers showing a
> 512x304 DIVX movie at 25fps, on a 3.2GHz machine with a 128MB Radeon
> 9600SE card. I expect my new system to perform at least a little
> better than that, while also supporting 3D.
>
> It seems at this point the only way to get HW-accelerated OpenGL
> running on Linux is through X11 / GLX, hopefully in the longer term I
> will be able to remove X from dom0, in order to reduce the size and
> complexity of the base installation.
>
> Best regards,
> Jacob
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>


--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

It occurred to me that multiple Linux domains can all share the same
framebuffer. Just treat domain swaps as VT swap events.  You could
even set the different domains up so that something like
Ctrl-Alt-Shift-1, 2, 3 jumps between domains.

dom0 would catch the key stroke, send the current domain a VT save
event, switch to the new domain and send it a VT restore.  This
assumes that all domains have direct access to the video hardware.

This scheme will probably work for multiple basic Xservers one in each
domain, but it will not work for multiple OpenGL or composite users
since they will stomp on each other's use of video RAM.

--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Gerd Knorr]

Anthony Liguori wrote:
> Now that's 3.0.0's out, I thought it would be a good time to bring up 
> the topic of framebuffer virtualization.
> 
> I threw together a proof-of-concept over the weekend of a simple virtual 
> framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d 
> buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's 
> also a simple message system for keyboard/mouse events.

Browsed through the code, looked at the wiki page now ...

First, I've hacked up something very simliar for user-mode-linux, 
patches are at http://www.suse.de/~kraxel/uml/patches/2.6.15-rc5/, you 
might want to have a look and steel some ideas (unless you already did).
Note that the input system got some sysfs-related changes recently, so 
the patches really work on very latest kernels only.

Some more comments:

  * why do you put the mfn and evtchn into the start page?  IMHO that
    should be handled by xenstore, like it is done for virtual network
    and disk.

  * I would create two different input devices for mouse/keyboard
    (although strictly seen it isn't really needed) to avoid confusing
    userspace apps.

  * I would simply reuse the input event struct from
    include/linux/input.h for dom0 -> domU input event delivery
    instead of creating something new.

  * i18n keyboard is really tricky (that is a huge mess in vmx btw).
    I think the only sane approach is to let the guest (domU) do
    all the i18n handling and try to avoid doing any key mapping
    in the host (dom0).  If you have *two* instances (both host
    and guest) doing keyboard mapping and both not even knowing
    what the other one is doing you have absolutely no chance to
    get it right ...

Keeping track of the in-kernel screen changes (fbcon) is easy, tracking 
userspace is more difficuilt but doable, the uml framebuffer driver does 
that by watching page faults: The no_page handler maps the page and 
marks it "dirty", the update function does zap_pte_range to kill the 
mappings so the no_page handler is called on the next update again.

cheers,
   Gerd

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/16/05, Gerd Knorr <kraxel@suse.de> wrote:
> Anthony Liguori wrote:
> > Now that's 3.0.0's out, I thought it would be a good time to bring up
> > the topic of framebuffer virtualization.
> >
> > I threw together a proof-of-concept over the weekend of a simple virtual
> > framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d
> > buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's
> > also a simple message system for keyboard/mouse events.
>
> Browsed through the code, looked at the wiki page now ...
>
> First, I've hacked up something very simliar for user-mode-linux,
> patches are at http://www.suse.de/~kraxel/uml/patches/2.6.15-rc5/, you
> might want to have a look and steel some ideas (unless you already did).
> Note that the input system got some sysfs-related changes recently, so
> the patches really work on very latest kernels only.
>
> Some more comments:
>
>   * why do you put the mfn and evtchn into the start page?  IMHO that
>     should be handled by xenstore, like it is done for virtual network
>     and disk.
>
>   * I would create two different input devices for mouse/keyboard
>     (although strictly seen it isn't really needed) to avoid confusing
>     userspace apps.

Another strategy for input would be to use USB input devices and the
USB over IP stack (or Xen USB support) to shift the devices between
domains.

Installing multiple keyboards, USB audio devices, mice and multiple
video cards is one way to achieve multiuser computing.

>
>   * I would simply reuse the input event struct from
>     include/linux/input.h for dom0 -> domU input event delivery
>     instead of creating something new.
>
>   * i18n keyboard is really tricky (that is a huge mess in vmx btw).
>     I think the only sane approach is to let the guest (domU) do
>     all the i18n handling and try to avoid doing any key mapping
>     in the host (dom0).  If you have *two* instances (both host
>     and guest) doing keyboard mapping and both not even knowing
>     what the other one is doing you have absolutely no chance to
>     get it right ...
>
> Keeping track of the in-kernel screen changes (fbcon) is easy, tracking
> userspace is more difficuilt but doable, the uml framebuffer driver does
> that by watching page faults: The no_page handler maps the page and
> marks it "dirty", the update function does zap_pte_range to kill the
> mappings so the no_page handler is called on the next update again.
>
> cheers,
>    Gerd
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>


--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Gerd Knorr]

>>   * I would create two different input devices for mouse/keyboard
>>     (although strictly seen it isn't really needed) to avoid confusing
>>     userspace apps.
> 
> Another strategy for input would be to use USB input devices and the
> USB over IP stack (or Xen USB support) to shift the devices between
> domains.

To handle remote display you need a xenconsoled-like helper application 
in domain0 anyway (which for example could export the display via vnc). 
That gives you remote input (via vnc or whatever protocol) almost for 
free as well, and I think application-wise it is much more convinient to 
take this route instead of using hid-over-virtualusb-over-ip.

> Installing multiple keyboards, USB audio devices, mice and multiple
> video cards is one way to achieve multiuser computing.

It's kida silly though to assign a domain some physical keyboard and a 
virtual framebuffer.

It might suddenly make much more sense once we can assign pci devices to 
domU's again, so you can setup a domU with both real display and 
keyboard, but thats completely unrelated to the virtual framebuffer ;)

cheers,

   Gerd

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

Gerd Knorr wrote:

>> Another strategy for input would be to use USB input devices and the
>> USB over IP stack (or Xen USB support) to shift the devices between
>> domains.
>
>
> To handle remote display you need a xenconsoled-like helper 
> application in domain0 anyway (which for example could export the 
> display via vnc). That gives you remote input (via vnc or whatever 
> protocol) almost for free as well, and I think application-wise it is 
> much more convinient to take this route instead of using 
> hid-over-virtualusb-over-ip.

http://hg.codemonkey.ws/vncfb/

Is my first pass at a VNC server btw.

Regards,

Anthony Liguori

>> Installing multiple keyboards, USB audio devices, mice and multiple
>> video cards is one way to achieve multiuser computing.
>
>
> It's kida silly though to assign a domain some physical keyboard and a 
> virtual framebuffer.
>
> It might suddenly make much more sense once we can assign pci devices 
> to domU's again, so you can setup a domU with both real display and 
> keyboard, but thats completely unrelated to the virtual framebuffer ;)
>
> cheers,
>
>   Gerd
>
>

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

Gerd Knorr wrote:

> Anthony Liguori wrote:
>
>> Now that's 3.0.0's out, I thought it would be a good time to bring up 
>> the topic of framebuffer virtualization.
>>
>> I threw together a proof-of-concept over the weekend of a simple 
>> virtual framebuffer/keyboard/mouse.  The basic design is have a 
>> vmalloc()'d buffer in the guest exposed as /dev/fb0 and mmap()'d in 
>> dom0.  There's also a simple message system for keyboard/mouse events.
>
>
> Browsed through the code, looked at the wiki page now ...

Excellent, I actually have made some major updates but not posted the 
bundle yet.

> First, I've hacked up something very simliar for user-mode-linux, 
> patches are at http://www.suse.de/~kraxel/uml/patches/2.6.15-rc5/, you 
> might want to have a look and steel some ideas (unless you already did).

Excellent, I'll check it out.

> Note that the input system got some sysfs-related changes recently, so 
> the patches really work on very latest kernels only.
>
> Some more comments:
>
>  * why do you put the mfn and evtchn into the start page?  IMHO that
>    should be handled by xenstore, like it is done for virtual network
>    and disk.

xenstore is not available at start-of-day.  This is why the console 
doesn't use it.  I'm not tied to using the start_info_t though--just 
figured the same arguement could be made for the framebuffer as the console.

>  * I would create two different input devices for mouse/keyboard
>    (although strictly seen it isn't really needed) to avoid confusing
>    userspace apps.

That's what I'm doing right now.  The first drop of the code was a bit 
of a mess wrt how the ring queue was used.

>  * I would simply reuse the input event struct from
>    include/linux/input.h for dom0 -> domU input event delivery
>    instead of creating something new.

Is this too Linux specific though?  I could just copy the structure.

>  * i18n keyboard is really tricky (that is a huge mess in vmx btw).
>    I think the only sane approach is to let the guest (domU) do
>    all the i18n handling and try to avoid doing any key mapping
>    in the host (dom0).  If you have *two* instances (both host
>    and guest) doing keyboard mapping and both not even knowing
>    what the other one is doing you have absolutely no chance to
>    get it right ...

How would you imagine this working?  I'll be getting X11 events in dom0 
so would you just pass them to domU and let it do the mapping?  I know 
QEmu has to use keyboard maps depending on locality.  We wouldn't push 
that into the domU kernel right?

i18n is hard and the thing I really don't know how to do right :-)  Help 
here is greatly appreciated.

> Keeping track of the in-kernel screen changes (fbcon) is easy, 
> tracking userspace is more difficuilt but doable, the uml framebuffer 
> driver does that by watching page faults: The no_page handler maps the 
> page and marks it "dirty", the update function does zap_pte_range to 
> kill the mappings so the no_page handler is called on the next update 
> again.

Yes, the latest code just implements all the 2d ops and passes a message 
through the framebuffer ring queue for updates.  VNC in console mode is 
quite snappy.

I'll take a look at the uml framebuffer to see how it handles page 
faults.  I was thinking of something similiar but didn't know how to 
approach it.

The latest Xen-code is at 
http://www.cs.utexas.edu/users/aliguori/vfb-20051216.bundle

Thanks Gerd!

Regards,

Anthony Liguori

> cheers,
>   Gerd
>

Re: [RFC] Xen Virtual Framebuffer

[Gerd Knorr]

>>  * why do you put the mfn and evtchn into the start page?  IMHO that
>>    should be handled by xenstore, like it is done for virtual network
>>    and disk.
> 
> xenstore is not available at start-of-day.  This is why the console 
> doesn't use it.

Yep, for the xenstore page we clearly need it (bootstrap issue), and for 
the console this is quite handy debug tool as well to easily get early 
boot messages in case the kernel crashes.  I think that are enougth 
special cases.

> I'm not tied to using the start_info_t though--just 
> figured the same arguement could be made for the framebuffer as the 
> console.

Not nessesarely.  The kernel messages are buffered and printed as soon 
as the first console device is available.  In case of the framebuffer 
console this usually happens quite late in the boot sequence, with all 
subsystems being initialized already.  Just do a "dmesg" on a native 
linux machine and look where the vesafb initialization and the "Console: 
switching to colour frame buffer device 160x64" message is ;)

>>  * I would simply reuse the input event struct from
>>    include/linux/input.h for dom0 -> domU input event delivery
>>    instead of creating something new.
> 
> Is this too Linux specific though?  I could just copy the structure.

Well, yes, it is Linux specific.  So other OS guests will have to adapt 
to that.  I don't think that is a big problem though, they have to adapt 
to xenbus too ;)

Hmm, well, maybe, GPL and BSD don't mix without problems, so it could be 
a problem to simply copy the include/linux/input.h file into the FreeBSD 
tree for example ...

>>  * i18n keyboard is really tricky (that is a huge mess in vmx btw).
>>    I think the only sane approach is to let the guest (domU) do
>>    all the i18n handling and try to avoid doing any key mapping
>>    in the host (dom0).  If you have *two* instances (both host
>>    and guest) doing keyboard mapping and both not even knowing
>>    what the other one is doing you have absolutely no chance to
>>    get it right ...
> 
> How would you imagine this working?  I'll be getting X11 events in dom0 
> so would you just pass them to domU and let it do the mapping?

I'd try to build a synthetic "struct input_event" out of whatever input
I get and pass that to the domU.

> I know 
> QEmu has to use keyboard maps depending on locality.  We wouldn't push 
> that into the domU kernel right?

Certainly not.

> i18n is hard and the thing I really don't know how to do right :-)  Help 
> here is greatly appreciated.

Ok, I'll try to explain things a bit and how I think the design could 
look like, taking the german keyboard as example.

The "de" kbd layout for the letters a-z looks like the us layout with z 
and y swapped (there are more differences but thats enougth for now to 
explain things a bit ...).

The key code in "struct input_event" referes to the untranslated keys, 
i.e. KEY_Y doesn't mean "the key labeled with 'y'" but means "the key in 
the top row, 7th from left".  That key is 'z' on a german keyboard layout.

The easiest job would have some linux-specific frontend which simply 
uses the dom0 framebuffer for output and reads input from 
/dev/input/event<x>, it could simply forward whatever input events it 
gets as-is ;)

X11 basically has two kinds of key ids:  One are the keycodes, which 
reference untranslated keys too.  The other ones are keysyms, which are 
  translated.  "xev" is nice to play around with that a bit, best 
together with some keyboard layout switcher, KDE has one for example.

Problem of the keycodes is that they are not portable.  On a sparc with 
solaris you have other keycodes than on a PC with Linux.  For the UML 
Framebuffer I decided to ignore this portability problem because UML is 
linux-on-linux only and a remote display doesn't work too.  So I've 
build a fixed table to map the (pc-specific) X11 keycodes to linux 
keycodes.  The uml machine comes up with a us keyboard layout, no matter 
what layout the host uses.  Then I have to use "loadkeys <whatever>" to 
get the (console) keyboard layout I want, like on a real machine.

Another possible approach for X11-based frontends would be to use the 
keysyms and translate them backwards to get untranslated keys.  Maybe 
this is what qemu does and what it needs the locale-specific keytables 
for ...

cheers,

   Gerd

RE: [RFC] Xen Virtual Framebuffer

[James Harper]

Some things just work better when you can enable shared memory
extensions under X, which obviously can't be done over the network.

Also, X isn't the only thing that can make use of a framebuffer.

James

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Jon Smirl
> Sent: Tuesday, 6 December 2005 11:36
> To: Anthony Liguori
> Cc: xen-devel
> Subject: Re: [Xen-devel] [RFC] Xen Virtual Framebuffer
> 
> I haven't tried playing with X and Xen, but why doesn't it work to
> just treat the multiple domains like a network? You run X in dom0 and
> give it full access to the video hardware. Then you ssh into each
> domain and start X apps, just like you do when using X remotely.
> OpenGL will even work this way and be accelerated (as soon as X fixes
> indirect acceleration). This model should let you get apps up from
> each domain simultaneously on the X display in dom0.
> 
> --
> Jon Smirl
> jonsmirl@gmail.com
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: [RFC] Xen Virtual Framebuffer

[M.A. Williamson]

And as another also: a big usability benefit of having the framebuffer is 
that users don't have to have a working network or install any extra 
software. It's just a transparency thing; you don't really *need* it but it 
makes domUs behave more like "proper" machines.

It should also give better performance, as James mentioned. Eventually, 
it'd be nice to support accelerated OpenGL in domUs but that may be some 
way off.

Cheers,
Mark

On Dec 6 2005, James Harper wrote:

>Some things just work better when you can enable shared memory
>extensions under X, which obviously can't be done over the network.
>
>Also, X isn't the only thing that can make use of a framebuffer.
>
>James
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Jon Smirl
>> Sent: Tuesday, 6 December 2005 11:36
>> To: Anthony Liguori
>> Cc: xen-devel
>> Subject: Re: [Xen-devel] [RFC] Xen Virtual Framebuffer
>> 
>> I haven't tried playing with X and Xen, but why doesn't it work to
>> just treat the multiple domains like a network? You run X in dom0 and
>> give it full access to the video hardware. Then you ssh into each
>> domain and start X apps, just like you do when using X remotely.
>> OpenGL will even work this way and be accelerated (as soon as X fixes
>> indirect acceleration). This model should let you get apps up from
>> each domain simultaneously on the X display in dom0.
>> 
>> --
>> Jon Smirl
>> jonsmirl@gmail.com
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 06 Dec 2005 00:53:14 +0000, M.A. Williamson <maw48@cam.ac.uk> wrote:
> And as another also: a big usability benefit of having the framebuffer is
> that users don't have to have a working network or install any extra
> software. It's just a transparency thing; you don't really *need* it but it
> makes domUs behave more like "proper" machines.

You don't need a network outside of the box; you just treat all of the
domains as being wired together with a virtual network inside of the
box.  X already uses Unix domain sockets to talk between apps and the
X server. I would expect a virtualized network running inside the box
to have about the same performance as Unix domain sockets. The X
server already support this socket model today, no new code needs to
be written, Xen just needs to provide the internal virtual network.

> It should also give better performance, as James mentioned. Eventually,
> it'd be nice to support accelerated OpenGL in domUs but that may be some
> way off.

Virtual framebuffer is not going to give you better performance than
the X socket system. I wrote this paper a while ago, it should give
you a good feel of the complexities involved.
http://dri.freedesktop.org/~jonsmirl/graphics.html

>
> Cheers,
> Mark
>
> On Dec 6 2005, James Harper wrote:
>
> >Some things just work better when you can enable shared memory
> >extensions under X, which obviously can't be done over the network.
> >
> >Also, X isn't the only thing that can make use of a framebuffer.
> >
> >James
> >
> >> -----Original Message-----
> >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> >> bounces@lists.xensource.com] On Behalf Of Jon Smirl
> >> Sent: Tuesday, 6 December 2005 11:36
> >> To: Anthony Liguori
> >> Cc: xen-devel
> >> Subject: Re: [Xen-devel] [RFC] Xen Virtual Framebuffer
> >>
> >> I haven't tried playing with X and Xen, but why doesn't it work to
> >> just treat the multiple domains like a network? You run X in dom0 and
> >> give it full access to the video hardware. Then you ssh into each
> >> domain and start X apps, just like you do when using X remotely.
> >> OpenGL will even work this way and be accelerated (as soon as X fixes
> >> indirect acceleration). This model should let you get apps up from
> >> each domain simultaneously on the X display in dom0.
> >>
> >> --
> >> Jon Smirl
> >> jonsmirl@gmail.com
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel
> >
>


--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[M.A. Williamson]

>You don't need a network outside of the box; you just treat all of the
>domains as being wired together with a virtual network inside of the
>box.

Sure thing; I'm just expecting users to break their network configuration 
with alacrity ;-) Something that's zero-config would be nice to have 
available as a baseline.

As a bonus it also gives us the opportunity to support things like QT 
Embedded, GTK, elinks, etc when those are configured to render direct to 
the framebuffer device. To be fair, these users are likely to be relatively 
uncommon.

> X already uses Unix domain sockets to talk between apps and the
>X server. I would expect a virtualized network running inside the box
>to have about the same performance as Unix domain sockets. The X
>server already support this socket model today, no new code needs to
>be written, Xen just needs to provide the internal virtual network.
>
>> It should also give better performance, as James mentioned. Eventually,
>> it'd be nice to support accelerated OpenGL in domUs but that may be some
>> way off.
>
>Virtual framebuffer is not going to give you better performance than
>the X socket system. I wrote this paper a while ago, it should give
>you a good feel of the complexities involved.
>http://dri.freedesktop.org/~jonsmirl/graphics.html

It seems relatively unfortunate to have to transfer data over the internal 
network when we could just use framebuffer memory which would be shared 
dom0 <-> domU (although I'm not sure if that overhead would actually matter 
to us). Surely it'd also be beneficial to enable X<->app shared memory 
tricks etc, rather than running over a network socket?

Btw, that paper had been on my "must read" list for a while, thanks for 
reminding me. Kudos for your work on Xgl, too bad you weren't supported 
more.

Cheers,
Mark

>
>>
>> Cheers,
>> Mark
>>
>> On Dec 6 2005, James Harper wrote:
>>
>> >Some things just work better when you can enable shared memory
>> >extensions under X, which obviously can't be done over the network.
>> >
>> >Also, X isn't the only thing that can make use of a framebuffer.
>> >
>> >James
>> >
>> >> -----Original Message-----
>> >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> >> bounces@lists.xensource.com] On Behalf Of Jon Smirl
>> >> Sent: Tuesday, 6 December 2005 11:36
>> >> To: Anthony Liguori
>> >> Cc: xen-devel
>> >> Subject: Re: [Xen-devel] [RFC] Xen Virtual Framebuffer
>> >>
>> >> I haven't tried playing with X and Xen, but why doesn't it work to
>> >> just treat the multiple domains like a network? You run X in dom0 and
>> >> give it full access to the video hardware. Then you ssh into each
>> >> domain and start X apps, just like you do when using X remotely.
>> >> OpenGL will even work this way and be accelerated (as soon as X fixes
>> >> indirect acceleration). This model should let you get apps up from
>> >> each domain simultaneously on the X display in dom0.
>> >>
>> >> --
>> >> Jon Smirl
>> >> jonsmirl@gmail.com
>> >>
>> >> _______________________________________________
>> >> Xen-devel mailing list
>> >> Xen-devel@lists.xensource.com
>> >> http://lists.xensource.com/xen-devel
>> >
>> >
>> >_______________________________________________
>> >Xen-devel mailing list
>> >Xen-devel@lists.xensource.com
>> >http://lists.xensource.com/xen-devel
>> >
>>
>
>
>--
>Jon Smirl
>jonsmirl@gmail.com
>

Re: [RFC] Xen Virtual Framebuffer

[Jon Smirl]

On 12/5/05, James Harper <james.harper@bendigoit.com.au> wrote:
> Some things just work better when you can enable shared memory
> extensions under X, which obviously can't be done over the network.
>
> Also, X isn't the only thing that can make use of a framebuffer.

What is an example of something that uses framebuffer?

Console is not a good example since that is effectively addressed by
using remote xterms.

>
> James

--
Jon Smirl
jonsmirl@gmail.com

Re: [RFC] Xen Virtual Framebuffer

[Charles Duffy]

Jon Smirl wrote:
> On 12/5/05, James Harper <james.harper@bendigoit.com.au> wrote:
> 
>>Some things just work better when you can enable shared memory
>>extensions under X, which obviously can't be done over the network.
>>
>>Also, X isn't the only thing that can make use of a framebuffer.
> 
> What is an example of something that uses framebuffer?

Any application based on Qt/Embedded or Gtk/FB. Several distributions' 
graphical installers, I think. Trolltech's palmtop application 
environment, I'm quite sure. Any number of custom embedded apps intended 
to run in environments too small for an X server. Quite some time back 
there were a decent number of games written to target the framebuffer, 
as well.

Being able to use Xen to prototype embedded applications (at least when 
host and target architectures match) is a very, very big win.

RE: [RFC] Xen Virtual Framebuffer

[Petersson, Mats]

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Gerd Knorr
> Sent: 06 December 2005 10:51
> To: Anthony Liguori
> Cc: xen-devel
> Subject: Re: [Xen-devel] [RFC] Xen Virtual Framebuffer
> 
> > I threw together a proof-of-concept over the weekend of a simple 
> > virtual framebuffer/keyboard/mouse.  The basic design is have a 
> > vmalloc()'d buffer in the guest exposed as /dev/fb0 and mmap()'d in 
> > dom0.  There's also a simple message system for 
> keyboard/mouse events.
> 
> Cool.
> 
> > Some interesting topics in this area are acceleration, whether we 
> > should implement our own X driver (or just enhance the fbdev driver 
> > since it uses no acceleration right now), and how to 
> properly expose 
> > it over something like VNC.
> 
> Hmm, I don't think acceleration is that important here.  2D 
> acceleration likely disappeares in graphics hardware soon.  
> I'd say a simple, stupid framebuffer for installs is 
> perfectly fine, for accelerated graphics its probably more 
> useful to skip 2D altogether and look at OpenGL 
> virtualization instead.

Actually, since I worked 2 years at a graphics chip manufacturer, doing
2D driver work, I would say that the "pure" 2D accelleration may well be
gone, but a lot of accelleration can still be had from using the 3D
engine to do 2D work... 

But then again, most 2D work is pretty quick even if you do it by pretty
simple means (memcpy-type operations), so yes, doing OpenGL/3D
accelleration would probably be more meaningfull. 

--
Mats
> 
> just my 2 cent,
> 
>    Gerd
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

Re: [RFC] Xen Virtual Framebuffer

[John Byrne]

> Now that's 3.0.0's out, I thought it would be a good time to bring up 
> the topic of framebuffer virtualization.
> 
> I threw together a proof-of-concept over the weekend of a simple virtual 
> framebuffer/keyboard/mouse.  The basic design is have a vmalloc()'d 
> buffer in the guest exposed as /dev/fb0 and mmap()'d in dom0.  There's 
> also a simple message system for keyboard/mouse events.
> 
> The first frontend is a GTK widget with python bindings (so it can 
> easily be embedded in a larger management app) and a small python app.  
> Right now, the VT system seems to work fine and X starts quite happily 
> (using the fbdev driver).  Clicking in the app captures the 
> mouse/keyboard and ctrl+alt will release the capture.
> 
> There's a readme and an hg bundle in the tarball below that explains how 
> to set things up.
> 
> Some interesting topics in this area are acceleration, whether we should 
> implement our own X driver (or just enhance the fbdev driver since it 
> uses no acceleration right now), and how to properly expose it over 
> something like VNC.
> 
> As always, feedback is greatly appreciated.
> 
> http://www.cs.utexas.edu/users/aliguori/xen-vfb-20051205.tar.gz
> 
> Regards,
> 
> Anthony Liguori
> 

Anthony,

Looking at Xen from the perspective of managing a farm of machines, what 
I want is a single method for remotely accessing the console/graphics 
for a guest domain, regardless of whether the domain is PV/VT. (A method 
of connect a VNC client to dom0 port 5900 + domid would be just fine. 
[From the code, this seems to be way qemu does it for VNC.]) The current 
model of "xm console" and Xvnc for PV domains is annoying. Unless, I am 
misunderstanding what you are doing, once you add the VNC support, your 
work provides what I want.

So, what are the prospects of getting it finished and in, in the near 
future? Is there any chance of getting it into the 3.0.x code base?

Thanks,

John Byrne

Re: [RFC] Xen Virtual Framebuffer

[Anthony Liguori]

John Byrne wrote:

> Anthony,
>
> Looking at Xen from the perspective of managing a farm of machines, 
> what I want is a single method for remotely accessing the 
> console/graphics for a guest domain, regardless of whether the domain 
> is PV/VT.

Yes, this is what the driving use-case for me has been.  I've done a lot 
of experiment with various VNC methods until I settled on this as the 
best approach.

> (A method of connect a VNC client to dom0 port 5900 + domid would be 
> just fine. [From the code, this seems to be way qemu does it for 
> VNC.]) The current model of "xm console" and Xvnc for PV domains is 
> annoying. Unless, I am misunderstanding what you are doing, once you 
> add the VNC support, your work provides what I want.

Yes, that's the goal.

> So, what are the prospects of getting it finished and in, in the near 
> future? Is there any chance of getting it into the 3.0.x code base?

The first step is to get the local display stuff working.  Then we can 
look at integrating that in Xen.  I have a Task list on the wiki and 
it's not really that much more work.

The second step is proof-of-concept VNC support which will probably be 
with libvncserver.  The display will be sluggish but it will be a start.

The final step would be to implement an accelerated X driver so that we 
could get the high-level operations info (CopyRect for window dragging, 
hw cursor, etc.) and then we can implement an efficient VNC server.

As for what goes into Xen when, that's up to Ian.  This is of course 
pretty early though so it's probably premature to really ask that until 
there's more progress made and some more discussion.

Regards,

Anthony Liguori

> Thanks,
>
> John Byrne
>
>
>
>
>
>
>