[Qemu-devel] paravirtual mouse/tablet

[Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 108 bytes --]

   Hi,

Just throwing a quick writeup into the ring to kickstart the design 
discussion ;)

cheers,
   Gerd

[-- Attachment #2: pvtablet.h --]
[-- Type: text/x-chdr, Size: 1725 bytes --]

#ifndef __QEMU_PVTABLET__
#define __QEMU_PVTABLET__ 1

/*
 * qemu patavirtual tablet interface
 */

#include <inttypes.h>

/* our virtio-serial channel */
#define QEMU_PVTABLET_NAME "org.qemu.pvtablet.0"

enum qemu_pvtablet_buttons {
    QEMU_PVTABLET_BTN_LEFT,
    QEMU_PVTABLET_BTN_RIGHT,
    QEMU_PVTABLET_BTN_MIDDLE,
    /*
     * we can handle up to 32 buttons although
     * not every has a individual name ;)
     */
};

/* send button down+up events */
typedef struct qemu_pvtablet_button {
    uint32_t button;
    uint32_t mask;
} qemu_pvtablet_button;

/* send pointer move events */
typedef struct qemu_pvtablet_position {
    uint32_t pos_x;  /* range: 0 -> 0xffff */
    uint32_t pos_y;  /* range: 0 -> 0xffff */
} qemu_pvtablet_position;

/*
 * specify which display all following events refering to
 * (for multihead setups).
 */
typedef struct qemu_pvtablet_display {
    /*
     * FIXME: spice just uses the display channel number, but that
     * doesn't fly in general ...
     */
    uint32_t spice_display_channel;
} qemu_pvtablet_display;

enum qemu_pvtablet_type {
    QEMU_PVTABLET_MSG_MOVE,     /* qemu_pvtablet_position */
    QEMU_PVTABLET_MSG_BTN_DOWN, /* qemu_pvtablet_button */
    QEMU_PVTABLET_MSG_BTN_UP,   /* qemu_pvtablet_button */
    QEMU_PVTABLET_MSG_DISPLAY,  /* qemu_pvtablet_display */
};

typedef struct qemu_pvtablet_message {
    uint32_t size;            /* whole message size */
    uint32_t type;            /* qemu_pvtablet_type */
    uint64_t tv_secs;
    uint64_t tv_usecs;
    union {
        qemu_pvtablet_position position;
        qemu_pvtablet_button   button;
        qemu_pvtablet_display  display;
    } m;
} qemu_pvtablet_message;

#endif /* __QEMU_PVTABLET__ */

[Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Stefan Hajnoczi]

Can you elaborate how the spice display channel comes into play?  On a
physical machine you just have input devices with no notion of
display.  It's up to the windowing system to process input events and
handle multihead.  Why does a pv tablet tie itself to a display
channel?

What about mouse wheel (aka z axis motion)?

Stefan

[Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

On 01/13/11 12:01, Stefan Hajnoczi wrote:

> Can you elaborate how the spice display channel comes into play?  On a
> physical machine you just have input devices with no notion of
> display.  It's up to the windowing system to process input events and
> handle multihead.  Why does a pv tablet tie itself to a display
> channel?

You have two qxl devices, each linked to a spice display channel.  The 
spice client will open one window for each channel.  The mouse position 
is a triple consisting of (x, y, window/channel).

Only the guest knows how it configured the displays, so only the guest 
is able to create a correct pointer position out of this data.  So the 
X-Server would combine this data with its virtual display configuration 
(basically adding the display offset to the coordinates), then send this 
as event to the X clients.

> What about mouse wheel (aka z axis motion)?

It's covered.  That are really just button presses at mouse protocol 
level (buttons 4+5).  We can add them to the enum to have fancy names 
for them, that is just the sugar on top though.

cheers,
   Gerd

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/13/2011 05:51 AM, Gerd Hoffmann wrote:
> On 01/13/11 12:01, Stefan Hajnoczi wrote:
>
>> Can you elaborate how the spice display channel comes into play?  On a
>> physical machine you just have input devices with no notion of
>> display.  It's up to the windowing system to process input events and
>> handle multihead.  Why does a pv tablet tie itself to a display
>> channel?
>
> You have two qxl devices, each linked to a spice display channel.  The 
> spice client will open one window for each channel.  The mouse 
> position is a triple consisting of (x, y, window/channel).
>

Would be slightly cleaner to have multiple devices.  That avoids 
breaking the abstraction and sets us up to handling multiple mice 
properly in the future.  That maybe implies that we need an offscreen 
coordinate for the mouse so that you can hide the mouse when it leaves 
one window.

> Only the guest knows how it configured the displays, so only the guest 
> is able to create a correct pointer position out of this data.  So the 
> X-Server would combine this data with its virtual display 
> configuration (basically adding the display offset to the 
> coordinates), then send this as event to the X clients.
>
>> What about mouse wheel (aka z axis motion)?
>
> It's covered.  That are really just button presses at mouse protocol 
> level (buttons 4+5).  We can add them to the enum to have fancy names 
> for them, that is just the sugar on top though.

If we add features, then we can deal with things like an additional axis 
to cover pressure sensitive touchpads.  Not worth designing now but a 
feature negotiation gives us the option later.

Regards,

Anthony Liguori

> cheers,
>   Gerd
>
>

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

On 01/13/11 16:55, Anthony Liguori wrote:
> On 01/13/2011 05:51 AM, Gerd Hoffmann wrote:
>> On 01/13/11 12:01, Stefan Hajnoczi wrote:
>>
>>> Can you elaborate how the spice display channel comes into play? On a
>>> physical machine you just have input devices with no notion of
>>> display. It's up to the windowing system to process input events and
>>> handle multihead. Why does a pv tablet tie itself to a display
>>> channel?
>>
>> You have two qxl devices, each linked to a spice display channel. The
>> spice client will open one window for each channel. The mouse position
>> is a triple consisting of (x, y, window/channel).
>>
>
> Would be slightly cleaner to have multiple devices.

Yea, right, that would work too.  If we have some way to assign a 
display to a pvmouse device this works equally well.

> That maybe implies that we need an offscreen coordinate for the
> mouse so that you can hide the mouse when it leaves one window.

Hmm?  I fail to see why multihead is special here.

cheers,
   Gerd

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/13/2011 11:08 AM, Gerd Hoffmann wrote:
> On 01/13/11 16:55, Anthony Liguori wrote:
>> On 01/13/2011 05:51 AM, Gerd Hoffmann wrote:
>>> On 01/13/11 12:01, Stefan Hajnoczi wrote:
>>>
>>>> Can you elaborate how the spice display channel comes into play? On a
>>>> physical machine you just have input devices with no notion of
>>>> display. It's up to the windowing system to process input events and
>>>> handle multihead. Why does a pv tablet tie itself to a display
>>>> channel?
>>>
>>> You have two qxl devices, each linked to a spice display channel. The
>>> spice client will open one window for each channel. The mouse position
>>> is a triple consisting of (x, y, window/channel).
>>>
>>
>> Would be slightly cleaner to have multiple devices.
>
> Yea, right, that would work too.  If we have some way to assign a 
> display to a pvmouse device this works equally well.

I think that has to be outside of the device.  There are so many ways to 
map mice to multi heads.  In fact, one mouse could easily map to a 
single device.  It's almost something that really should be part of the 
guest configuration.

>> That maybe implies that we need an offscreen coordinate for the
>> mouse so that you can hide the mouse when it leaves one window.
>
> Hmm?  I fail to see why multihead is special here.

If you show two mice in the guest, would X render two cursors?

Regards,

Anthony Liguori

> cheers,
>   Gerd
>

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

   Hi,

> I think that has to be outside of the device. There are so many ways to
> map mice to multi heads. In fact, one mouse could easily map to a single
> device. It's almost something that really should be part of the guest
> configuration.

Ok, reasonable.  Multihead with spice pretty much requires a guest agent 
anyway, so I'm sure we'll can figure a way to handle this.

>>> That maybe implies that we need an offscreen coordinate for the
>>> mouse so that you can hide the mouse when it leaves one window.
>>
>> Hmm? I fail to see why multihead is special here.
>
> If you show two mice in the guest, would X render two cursors?

With two mice you still have just one cursor.  Just plug a usb mouse 
into your laptop, you can move the single pointer with both usb mouse 
and touchpad then.  Didn't try what happens with two tablets, but I 
expect it wouldn't be different ...

cheers,
   Gerd

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Daniel P. Berrange]

On Fri, Jan 14, 2011 at 09:49:40AM +0100, Gerd Hoffmann wrote:
> >>>That maybe implies that we need an offscreen coordinate for the
> >>>mouse so that you can hide the mouse when it leaves one window.
> >>
> >>Hmm? I fail to see why multihead is special here.
> >
> >If you show two mice in the guest, would X render two cursors?
> 
> With two mice you still have just one cursor.  Just plug a usb mouse
> into your laptop, you can move the single pointer with both usb
> mouse and touchpad then.  Didn't try what happens with two tablets,
> but I expect it wouldn't be different ...

NB having all mice bound to one cursor is merely the historical
default behaviour. IIUC the recent "Multi-Pointer X" feature
lets you now have multiple cursors, one per pointing device.

https://secure.wikimedia.org/wikipedia/en/wiki/Multi-Pointer_X

Regards,
Daniel

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/14/2011 04:48 AM, Daniel P. Berrange wrote:
> NB having all mice bound to one cursor is merely the historical
> default behaviour. IIUC the recent "Multi-Pointer X" feature
> lets you now have multiple cursors, one per pointing device.
>
> https://secure.wikimedia.org/wikipedia/en/wiki/Multi-Pointer_X
>    

Yeah, that's exactly what I had in mind and why I suggested an offscreen 
coordinate.

Regards,

Anthony Liguori

> Regards,
> Daniel
>    

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Alon Levy]

On Fri, Jan 14, 2011 at 10:48:50AM +0000, Daniel P. Berrange wrote:
> On Fri, Jan 14, 2011 at 09:49:40AM +0100, Gerd Hoffmann wrote:
> > >>>That maybe implies that we need an offscreen coordinate for the
> > >>>mouse so that you can hide the mouse when it leaves one window.
> > >>
> > >>Hmm? I fail to see why multihead is special here.
> > >
> > >If you show two mice in the guest, would X render two cursors?
> > 
> > With two mice you still have just one cursor.  Just plug a usb mouse
> > into your laptop, you can move the single pointer with both usb
> > mouse and touchpad then.  Didn't try what happens with two tablets,
> > but I expect it wouldn't be different ...
> 
> NB having all mice bound to one cursor is merely the historical
> default behaviour. IIUC the recent "Multi-Pointer X" feature
> lets you now have multiple cursors, one per pointing device.
> 
> https://secure.wikimedia.org/wikipedia/en/wiki/Multi-Pointer_X

Yes, you can enable that, and if we want to have multiple pointers then
they shouldn't disappear - that would be the case if multiple users
where connected, each with their own cursor. You expect the non active cursor
to remain (maybe X hides it after some seconds of inactivity, guest
specific).

> 
> Regards,
> Daniel
> 

Re: [Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Daniel P. Berrange]

On Fri, Jan 14, 2011 at 04:28:52PM +0200, Alon Levy wrote:
> On Fri, Jan 14, 2011 at 10:48:50AM +0000, Daniel P. Berrange wrote:
> > On Fri, Jan 14, 2011 at 09:49:40AM +0100, Gerd Hoffmann wrote:
> > > >>>That maybe implies that we need an offscreen coordinate for the
> > > >>>mouse so that you can hide the mouse when it leaves one window.
> > > >>
> > > >>Hmm? I fail to see why multihead is special here.
> > > >
> > > >If you show two mice in the guest, would X render two cursors?
> > > 
> > > With two mice you still have just one cursor.  Just plug a usb mouse
> > > into your laptop, you can move the single pointer with both usb
> > > mouse and touchpad then.  Didn't try what happens with two tablets,
> > > but I expect it wouldn't be different ...
> > 
> > NB having all mice bound to one cursor is merely the historical
> > default behaviour. IIUC the recent "Multi-Pointer X" feature
> > lets you now have multiple cursors, one per pointing device.
> > 
> > https://secure.wikimedia.org/wikipedia/en/wiki/Multi-Pointer_X
> 
> Yes, you can enable that, and if we want to have multiple pointers then
> they shouldn't disappear - that would be the case if multiple users
> where connected, each with their own cursor. You expect the non active cursor
> to remain (maybe X hides it after some seconds of inactivity, guest
> specific).

Multi-Pointer X isn't just about allowing different users
to have their own pointer. It is explicitly designed to allow
one person to control multiple pointers at the same time and
to allow applications to receive events from multiple devices
concurrently.

Regards,
Daniel

Re: [Spice-devel] [Qemu-devel] Re: paravirtual mouse/tablet

[Alon Levy]

On Fri, Jan 14, 2011 at 02:52:35PM +0000, Daniel P. Berrange wrote:
> On Fri, Jan 14, 2011 at 04:28:52PM +0200, Alon Levy wrote:
> > On Fri, Jan 14, 2011 at 10:48:50AM +0000, Daniel P. Berrange wrote:
> > > On Fri, Jan 14, 2011 at 09:49:40AM +0100, Gerd Hoffmann wrote:
> > > > >>>That maybe implies that we need an offscreen coordinate for the
> > > > >>>mouse so that you can hide the mouse when it leaves one window.
> > > > >>
> > > > >>Hmm? I fail to see why multihead is special here.
> > > > >
> > > > >If you show two mice in the guest, would X render two cursors?
> > > > 
> > > > With two mice you still have just one cursor.  Just plug a usb mouse
> > > > into your laptop, you can move the single pointer with both usb
> > > > mouse and touchpad then.  Didn't try what happens with two tablets,
> > > > but I expect it wouldn't be different ...
> > > 
> > > NB having all mice bound to one cursor is merely the historical
> > > default behaviour. IIUC the recent "Multi-Pointer X" feature
> > > lets you now have multiple cursors, one per pointing device.
> > > 
> > > https://secure.wikimedia.org/wikipedia/en/wiki/Multi-Pointer_X
> > 
> > Yes, you can enable that, and if we want to have multiple pointers then
> > they shouldn't disappear - that would be the case if multiple users
> > where connected, each with their own cursor. You expect the non active cursor
> > to remain (maybe X hides it after some seconds of inactivity, guest
> > specific).
> 
> Multi-Pointer X isn't just about allowing different users
> to have their own pointer. It is explicitly designed to allow
> one person to control multiple pointers at the same time and
> to allow applications to receive events from multiple devices
> concurrently.

Yes, of course, I didn't suggest otherwise. I don't even think most setup's
will be able to distinguish these two scenarios.

> 
> Regards,
> Daniel
> _______________________________________________
> Spice-devel mailing list
> Spice-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel

Re: [Qemu-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/13/2011 04:19 AM, Gerd Hoffmann wrote:
>   Hi,
>
> Just throwing a quick writeup into the ring to kickstart the design 
> discussion ;)
>
> cheers,
>   Gerd
> #ifndef __QEMU_PVTABLET__
> #define __QEMU_PVTABLET__ 1
>
> /*
>   * qemu patavirtual tablet interface
>   */
>
> #include<inttypes.h>
>
> /* our virtio-serial channel */
> #define QEMU_PVTABLET_NAME "org.qemu.pvtablet.0"
>
> enum qemu_pvtablet_buttons {
>      QEMU_PVTABLET_BTN_LEFT,
>      QEMU_PVTABLET_BTN_RIGHT,
>      QEMU_PVTABLET_BTN_MIDDLE,
>      /*
>       * we can handle up to 32 buttons although
>       * not every has a individual name;)
>       */
> };
>
> /* send button down+up events */
> typedef struct qemu_pvtablet_button {
>      uint32_t button;
>      uint32_t mask;
> } qemu_pvtablet_button;
>
> /* send pointer move events */
> typedef struct qemu_pvtablet_position {
>      uint32_t pos_x;  /* range: 0 ->  0xffff */
>      uint32_t pos_y;  /* range: 0 ->  0xffff */
> } qemu_pvtablet_position;
>
>    

Let's add feature negotiation and not hardcode the resolution.

typedef enum qemu_pvtablet_features {
     /* None yet */
};

/* host->guest, sent before any other events */
typedef struct qemu_pvtablet_init {
     uint32_t res_x;   /* x axis resolution */
     uint32_t res_y;   /* y axis resolution */
     uint32_t features;  /* qemu_pvtablet_features */
} qemu_pvtablet_init;

/* guest->host, sent after pvtablet_init.  host will not send additional 
messages until this is received */
typedef struct qemu_pvtablet_ack {
     uint32_t features; /* qemu_pvtable_features */
};


> enum qemu_pvtablet_type {
>      QEMU_PVTABLET_MSG_MOVE,     /* qemu_pvtablet_position */
>      QEMU_PVTABLET_MSG_BTN_DOWN, /* qemu_pvtablet_button */
>      QEMU_PVTABLET_MSG_BTN_UP,   /* qemu_pvtablet_button */
>    
        QEMU_PVTABLET_MSG_INIT,   /* qemu_pvtable_init */
        QEMU_PVTABLE_MSG_ACK, /* qemu_pvtable_ack */

Regards,

Anthony Liguori

> };
>
> typedef struct qemu_pvtablet_message {
>      uint32_t size;            /* whole message size */
>      uint32_t type;            /* qemu_pvtablet_type */
>      uint64_t tv_secs;
>      uint64_t tv_usecs;
>      union {
>          qemu_pvtablet_position position;
>          qemu_pvtablet_button   button;
>          qemu_pvtablet_display  display;
>      } m;
> } qemu_pvtablet_message;
>    



> #endif /* __QEMU_PVTABLET__ */
>    

Re: [Qemu-devel] paravirtual mouse/tablet

[Avi Kivity]

On 01/13/2011 05:52 PM, Anthony Liguori wrote:
>
> /* host->guest, sent before any other events */
> typedef struct qemu_pvtablet_init {
>     uint32_t res_x;   /* x axis resolution */
>     uint32_t res_y;   /* y axis resolution */
>     uint32_t features;  /* qemu_pvtablet_features */

uint32_t available_buttons; /* bitmask */

> } qemu_pvtablet_init;

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/13/2011 10:14 AM, Avi Kivity wrote:
> On 01/13/2011 05:52 PM, Anthony Liguori wrote:
>>
>> /* host->guest, sent before any other events */
>> typedef struct qemu_pvtablet_init {
>>     uint32_t res_x;   /* x axis resolution */
>>     uint32_t res_y;   /* y axis resolution */
>>     uint32_t features;  /* qemu_pvtablet_features */
>
> uint32_t available_buttons; /* bitmask */

Yes, I had intended to do that but left it out.

Should it be a bitmask or just a button count?  Buttons really have no 
standard meaning so usually a button count is sufficient.

Regards,

Anthony Liguori

>
>> } qemu_pvtablet_init;
>

Re: [Qemu-devel] paravirtual mouse/tablet

[Paolo Bonzini]

On 01/13/2011 05:39 PM, Anthony Liguori wrote:
> On 01/13/2011 10:14 AM, Avi Kivity wrote:
>> On 01/13/2011 05:52 PM, Anthony Liguori wrote:
>>>
>>> /* host->guest, sent before any other events */
>>> typedef struct qemu_pvtablet_init {
>>> uint32_t res_x; /* x axis resolution */
>>> uint32_t res_y; /* y axis resolution */
>>> uint32_t features; /* qemu_pvtablet_features */
>>
>> uint32_t available_buttons; /* bitmask */
>
> Yes, I had intended to do that but left it out.
>
> Should it be a bitmask or just a button count? Buttons really have no
> standard meaning so usually a button count is sufficient.

3/4 are the mouse wheel, so if you had a mouse with 5 buttons and no 
wheel those would be buttons 0/1/2/5/6.

Paolo

Re: [Qemu-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/13/2011 11:09 AM, Paolo Bonzini wrote:
> On 01/13/2011 05:39 PM, Anthony Liguori wrote:
>> On 01/13/2011 10:14 AM, Avi Kivity wrote:
>>> On 01/13/2011 05:52 PM, Anthony Liguori wrote:
>>>>
>>>> /* host->guest, sent before any other events */
>>>> typedef struct qemu_pvtablet_init {
>>>> uint32_t res_x; /* x axis resolution */
>>>> uint32_t res_y; /* y axis resolution */
>>>> uint32_t features; /* qemu_pvtablet_features */
>>>
>>> uint32_t available_buttons; /* bitmask */
>>
>> Yes, I had intended to do that but left it out.
>>
>> Should it be a bitmask or just a button count? Buttons really have no
>> standard meaning so usually a button count is sufficient.
>
> 3/4 are the mouse wheel, so if you had a mouse with 5 buttons and no 
> wheel those would be buttons 0/1/2/5/6.

It can still be reported as a 7 button mouse but I don't really mind 
either way.

Regards,

Anthony Liguori

> Paolo
>
>

Re: [Qemu-devel] paravirtual mouse/tablet

[Alexander Graf]

On 13.01.2011, at 17:39, Anthony Liguori wrote:

> On 01/13/2011 10:14 AM, Avi Kivity wrote:
>> On 01/13/2011 05:52 PM, Anthony Liguori wrote:
>>> 
>>> /* host->guest, sent before any other events */
>>> typedef struct qemu_pvtablet_init {
>>>    uint32_t res_x;   /* x axis resolution */
>>>    uint32_t res_y;   /* y axis resolution */
>>>    uint32_t features;  /* qemu_pvtablet_features */
>> 
>> uint32_t available_buttons; /* bitmask */
> 
> Yes, I had intended to do that but left it out.
> 
> Should it be a bitmask or just a button count?  Buttons really have no standard meaning so usually a button count is sufficient.

Some random thoughts:

  * multitouch capabilities would be good to design in a mouse protocol for 2011, so having say 16 x/y pairs would be better
  * on mac os at least scrolling is not done by pressing virtual buttons, but by having a separate scroll interface that knows about velocity and such - maybe worth adding that to the protocol from the beginning too.


Alex

Re: [Qemu-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/13/2011 11:13 AM, Alexander Graf wrote:
>
>> Yes, I had intended to do that but left it out.
>>
>> Should it be a bitmask or just a button count?  Buttons really have no standard meaning so usually a button count is sufficient.
>>      
> Some random thoughts:
>
>    * multitouch capabilities would be good to design in a mouse protocol for 2011, so having say 16 x/y pairs would be better
>    * on mac os at least scrolling is not done by pressing virtual buttons, but by having a separate scroll interface that knows about velocity and such - maybe worth adding that to the protocol from the beginning too.
>    

Features negotiation takes care of this.

Regards,

Anthony Liguori

>
> Alex
>
>    

Re: [Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

   Hi,

> * multitouch capabilities would be good to design in a mouse protocol
> for 2011, so having say 16 x/y pairs would be better

Point.  What do we need here?  Finger $n down, finger $n up, finger $n 
moved to $x,$y?  Does it make sense to just add this to the 
move+buttonup/down structs?  Or should it better be separate?

> * on mac os at least scrolling is not done by pressing virtual
> buttons, but by having a separate scroll interface that knows about
> velocity and such - maybe worth adding that to the protocol from the
> beginning too.

Is that actually done by the hardware?  I'd expect macos processes the 
multitouch events, then provides this (finger $n moves this fast into 
that direction) as high-level interface, so this isn't something we have 
to care about at virtual hardware level.

cheers,
   Gerd

Re: [Qemu-devel] paravirtual mouse/tablet

[Alexander Graf]

On 14.01.2011, at 12:27, Gerd Hoffmann wrote:

>  Hi,
> 
>> * multitouch capabilities would be good to design in a mouse protocol
>> for 2011, so having say 16 x/y pairs would be better
> 
> Point.  What do we need here?  Finger $n down, finger $n up, finger $n moved to $x,$y?  Does it make sense to just add this to the move+buttonup/down structs?  Or should it better be separate?

I guess a mere tuple of (x,y,down) N times should be enough for the protocol, no? This would even be useful for single-point tablets which usually also have a proximity detector. Thinking about this a bit more - there are proximity sensors in tablets :). So they know how hard you pressed. If we want to be able to forward that to the guest from a real world tablet, we need a pressure field.

So it'd end up being (x,y,pressure) N times (I think 16 is fine for the foreseeable future). The details of what exactly that means should be figured out by the guest driver. If the guest is multitouch capable, it'd forward those events to the respective system. If it's tablet compatible, take the first entry and if it only supports plain mice, just make it (x,y) when pressure > 0.

> 
>> * on mac os at least scrolling is not done by pressing virtual
>> buttons, but by having a separate scroll interface that knows about
>> velocity and such - maybe worth adding that to the protocol from the
>> beginning too.
> 
> Is that actually done by the hardware?  I'd expect macos processes the multitouch events, then provides this (finger $n moves this fast into that direction) as high-level interface, so this isn't something we have to care about at virtual hardware level.

I'm not familiar with the hardware interface, but in order to support that the background interface must be a lot more complex than a simple button press. I'd frankly assume they do that in software with their own multitouch detection code though, so it should be fine to have the buttons as legacy interface as long as there is multitouch to implement the scrolling.

But then again - how would we forward fine-grained scrolling to the guest if we only know that it's scrolling, but not what the actual presses on the touchpad looked like? Ugh.


Alex

Re: [Spice-devel] [Qemu-devel] paravirtual mouse/tablet

[Frédéric Grelot]

> So it'd end up being (x,y,pressure) N times (I think 16 is fine for
> the foreseeable future).

It's highly speculative, but Microsoft just released Surface 2, and it is able to handle up to 40 different pressure points... It is aimed at people in reunion meeting around a table and manipulation/exchanging documents with each other...
If there is no hard/soft limit at 16, maybe it would be a good idea to allow some more from the beginning...

Frederic.

Re: [Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

   Hi,

> So it'd end up being (x,y,pressure) N times (I think 16 is fine for
> the foreseeable future).

I'd tend to extend MOVE to (x,y,pressure,index) and send N events with 
the same timestamp.  Needs to send only as many events as it finds 
fingers on the touchpad, i.e. usually just one or two, even if the 
protocol can easily handle alot more than 16 ;)

For a simple tablet pressure and index would just be 0.

> The details of what exactly that means
> should be figured out by the guest driver.

Agree.

> I'm not familiar with the hardware interface, but in order to support
> that the background interface must be a lot more complex than a
> simple button press.

Buttons events are for buttons.  Real ones, which apple lost ;)

Of course a tap on the trackpad is usually interpreted as mouse click. 
But that is the job of the guest OS, our virtual hardware doesn't care.

> But then again - how would we forward fine-grained scrolling to the
> guest if we only know that it's scrolling, but not what the actual
> presses on the touchpad looked like? Ugh.

There must be an interface to get (more or less) the raw touchpad data, 
for apps which want implement their own multitouch gestures?

cheers,
   Gerd

Re: [Qemu-devel] paravirtual mouse/tablet

[Alexander Graf]

On 14.01.2011, at 16:13, Gerd Hoffmann wrote:

>  Hi,
> 
>> So it'd end up being (x,y,pressure) N times (I think 16 is fine for
>> the foreseeable future).
> 
> I'd tend to extend MOVE to (x,y,pressure,index) and send N events with the same timestamp.  Needs to send only as many events as it finds fingers on the touchpad, i.e. usually just one or two, even if the protocol can easily handle alot more than 16 ;)
> 
> For a simple tablet pressure and index would just be 0.

For a simple tablet pressure would be MAX and index would be 0. But yes, I like the idea :).

> 
>> The details of what exactly that means
>> should be figured out by the guest driver.
> 
> Agree.
> 
>> I'm not familiar with the hardware interface, but in order to support
>> that the background interface must be a lot more complex than a
>> simple button press.
> 
> Buttons events are for buttons.  Real ones, which apple lost ;)
> 
> Of course a tap on the trackpad is usually interpreted as mouse click. But that is the job of the guest OS, our virtual hardware doesn't care.

With an interface the way you mention it below (self-implemented gesture support), this shouldn't be an issue. A tap should just get forwarded to the guest as a tap and then the guest can configure what a tap means - which is what users want.

> 
>> But then again - how would we forward fine-grained scrolling to the
>> guest if we only know that it's scrolling, but not what the actual
>> presses on the touchpad looked like? Ugh.
> 
> There must be an interface to get (more or less) the raw touchpad data, for apps which want implement their own multitouch gestures?

I'm not sure there is, but that seems like the only viable solution. This way the guest can define its own multitouch gestures that the host doesn't even have to know about (think of osx guests on linux hosts for example).


Alex

Re: [Qemu-devel] paravirtual mouse/tablet

[Alexander Graf]

On 14.01.2011, at 16:28, Alexander Graf wrote:

> 
> On 14.01.2011, at 16:13, Gerd Hoffmann wrote:
> 
>> Hi,
>> 
>>> So it'd end up being (x,y,pressure) N times (I think 16 is fine for
>>> the foreseeable future).
>> 
>> I'd tend to extend MOVE to (x,y,pressure,index) and send N events with the same timestamp.  Needs to send only as many events as it finds fingers on the touchpad, i.e. usually just one or two, even if the protocol can easily handle alot more than 16 ;)
>> 
>> For a simple tablet pressure and index would just be 0.
> 
> For a simple tablet pressure would be MAX and index would be 0. But yes, I like the idea :).

0 for non-presses of course :). Sorry


Alex

Re: [Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

   Hi,

>>> For a simple tablet pressure and index would just be 0.
>>
>> For a simple tablet pressure would be MAX and index would be 0. But yes, I like the idea :).
>
> 0 for non-presses of course :). Sorry

Right ;)

There are three cases:

(1) no pressure supported (i.e. your mouse moving around in the vnc
     window and qemu reporting this as tablet coordinates).
(2) just pen/finger present/not present supported.  pressure jumps
     between 0 and max (and we can make max == 1 in that case).
(3) real pressure measurement.

Will send out v3 shortly.

cheers,
   Gerd

Re: [Qemu-devel] paravirtual mouse/tablet

[Alexander Graf]

On 14.01.2011, at 17:31, Gerd Hoffmann wrote:

>  Hi,
> 
>>>> For a simple tablet pressure and index would just be 0.
>>> 
>>> For a simple tablet pressure would be MAX and index would be 0. But yes, I like the idea :).
>> 
>> 0 for non-presses of course :). Sorry
> 
> Right ;)
> 
> There are three cases:
> 
> (1) no pressure supported (i.e. your mouse moving around in the vnc
>    window and qemu reporting this as tablet coordinates).
> (2) just pen/finger present/not present supported.  pressure jumps
>    between 0 and max (and we can make max == 1 in that case).

Phew - that's one of the bits where touchpads and tablets behave differently IIUC. For touchpads, pressing means movement happens. For tablets, pressing means an actual press, as if you would press your pencil on a sheet of paper. So for tablets, pressure==1 basically means click.

Now the thing is that multitouch gestures might take non-click pressure into account for their calculations. So we need pressure for movements on touchpads, but with tablets, we only need pressure when clicks happen. For everyone else, clicks would just be a button press.

Tricky.


Alex

Re: [Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

>> There are three cases:
>>
>> (1) no pressure supported (i.e. your mouse moving around in the vnc
>> window and qemu reporting this as tablet coordinates).
>> (2) just pen/finger present/not present supported.  pressure jumps
>> between 0 and max (and we can make max == 1 in that case).
>
> Phew - that's one of the bits where touchpads and tablets behave
> differently IIUC. For touchpads, pressing means movement happens.
> For tablets, pressing means an actual press, as if you would press
> your pencil on a sheet of paper. So for tablets, pressure==1
> basically means click.

I wouldn't make a difference from the virtual hardware perspective.  If 
something touches the pad/tablet surface we'll report the position where 
it happened (and the pressure if supported).  Whenever this is 
interpreted as click or not is up to the guest.

> Now the thing is that multitouch gestures might take non-click
> pressure into account for their calculations. So we need pressure
> for movements on touchpads, but with tablets, we only need pressure
> when clicks happen. For everyone else, clicks would just be a button
> press.

-EPARSE.  What is "non-click pressure" ?

cheers,
   Gerd

Re: [Qemu-devel] paravirtual mouse/tablet

[Alexander Graf]

On 17.01.2011, at 08:48, Gerd Hoffmann wrote:

>>> There are three cases:
>>> 
>>> (1) no pressure supported (i.e. your mouse moving around in the vnc
>>> window and qemu reporting this as tablet coordinates).
>>> (2) just pen/finger present/not present supported.  pressure jumps
>>> between 0 and max (and we can make max == 1 in that case).
>> 
>> Phew - that's one of the bits where touchpads and tablets behave
>> differently IIUC. For touchpads, pressing means movement happens.
>> For tablets, pressing means an actual press, as if you would press
>> your pencil on a sheet of paper. So for tablets, pressure==1
>> basically means click.
> 
> I wouldn't make a difference from the virtual hardware perspective.  If something touches the pad/tablet surface we'll report the position where it happened (and the pressure if supported).  Whenever this is interpreted as click or not is up to the guest.

There is a major difference. "Touching" means on

Touchpad: movement of cursor
Tablet: pressing down a pen

This difference needs to be reflected somehow. Maybe in the future there might even be devices that combine both. Imagine a tablet device that has a capacitive touchpad, but also accepts pen input, so you can draw on it. This would be separate events. Pressing down a pen would be interpreted completely differently from touching something on the device.


Alex

Re: [Qemu-devel] paravirtual mouse/tablet

[Alon Levy]

On Fri, Jan 14, 2011 at 04:13:29PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> >So it'd end up being (x,y,pressure) N times (I think 16 is fine for
> >the foreseeable future).
> 
> I'd tend to extend MOVE to (x,y,pressure,index) and send N events
> with the same timestamp.  Needs to send only as many events as it
> finds fingers on the touchpad, i.e. usually just one or two, even if
> the protocol can easily handle alot more than 16 ;)

Doesn't that lose the coincidence? I expect mostly presses happen
sequentially (milliseconds apart maybe, but still), so probably not
an issue at all.

> 
> For a simple tablet pressure and index would just be 0.
> 
> >The details of what exactly that means
> >should be figured out by the guest driver.
> 
> Agree.
> 
> >I'm not familiar with the hardware interface, but in order to support
> >that the background interface must be a lot more complex than a
> >simple button press.
> 
> Buttons events are for buttons.  Real ones, which apple lost ;)
> 
> Of course a tap on the trackpad is usually interpreted as mouse
> click. But that is the job of the guest OS, our virtual hardware
> doesn't care.
> 
> >But then again - how would we forward fine-grained scrolling to the
> >guest if we only know that it's scrolling, but not what the actual
> >presses on the touchpad looked like? Ugh.
> 
> There must be an interface to get (more or less) the raw touchpad
> data, for apps which want implement their own multitouch gestures?
> 
> cheers,
>   Gerd
> 
> 

Re: [Qemu-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/14/2011 08:36 AM, Alexander Graf wrote:
> On 14.01.2011, at 12:27, Gerd Hoffmann wrote:
>
>    
>>   Hi,
>>
>>      
>>> * multitouch capabilities would be good to design in a mouse protocol
>>> for 2011, so having say 16 x/y pairs would be better
>>>        
>> Point.  What do we need here?  Finger $n down, finger $n up, finger $n moved to $x,$y?  Does it make sense to just add this to the move+buttonup/down structs?  Or should it better be separate?
>>      
> I guess a mere tuple of (x,y,down) N times should be enough for the protocol, no?

Surely, evdev has an interface to support this already.  Let's just do 
what it does instead of inventing something that none of us can validate 
actually works.

Or better yet, delay implementing it until someone actually knows how to 
support it.

Regards,

Anthony Liguori

>   This would even be useful for single-point tablets which usually also have a proximity detector. Thinking about this a bit more - there are proximity sensors in tablets :). So they know how hard you pressed. If we want to be able to forward that to the guest from a real world tablet, we need a pressure field.
>
> So it'd end up being (x,y,pressure) N times (I think 16 is fine for the foreseeable future). The details of what exactly that means should be figured out by the guest driver. If the guest is multitouch capable, it'd forward those events to the respective system. If it's tablet compatible, take the first entry and if it only supports plain mice, just make it (x,y) when pressure>  0.
>
>    
>>      
>>> * on mac os at least scrolling is not done by pressing virtual
>>> buttons, but by having a separate scroll interface that knows about
>>> velocity and such - maybe worth adding that to the protocol from the
>>> beginning too.
>>>        
>> Is that actually done by the hardware?  I'd expect macos processes the multitouch events, then provides this (finger $n moves this fast into that direction) as high-level interface, so this isn't something we have to care about at virtual hardware level.
>>      
> I'm not familiar with the hardware interface, but in order to support that the background interface must be a lot more complex than a simple button press. I'd frankly assume they do that in software with their own multitouch detection code though, so it should be fine to have the buttons as legacy interface as long as there is multitouch to implement the scrolling.
>
> But then again - how would we forward fine-grained scrolling to the guest if we only know that it's scrolling, but not what the actual presses on the touchpad looked like? Ugh.
>
>
> Alex
>
>    

Re: [Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

   Hi,

>> I guess a mere tuple of (x,y,down) N times should be enough for the
>> protocol, no?
>
> Surely, evdev has an interface to support this already. Let's just do
> what it does instead of inventing something that none of us can validate
> actually works.

http://www.mjmwired.net/kernel/Documentation/input/multi-touch-protocol.txt

Looks like the linux input layer does effectively the same: send series 
of events for multitouch.

A sync event to mark the end of the batch is obviously a pretty good 
idea I'm going to steal.

Some devices seem to be able to attach IDs to the touchpoints.  Then you 
don't have to send the whole set each time but can limit yourself to the 
ones which did actually change.  We should support that too.

> Or better yet, delay implementing it until someone actually knows how to
> support it.

I think we can get sane support into the protocol right from start, and 
that is much better than adding something afterwards.  Just look at the 
linux input layer document -- some of the loops they have to hop though 
are there just because multitouch is a extension which came later.

cheers,
   Gerd

Re: [Qemu-devel] paravirtual mouse/tablet

[Peter Maydell]

On 14 January 2011 09:37, Anthony Liguori <anthony@codemonkey.ws> wrote:
[multitouch]
> Surely, evdev has an interface to support this already.  Let's just do what
> it does instead of inventing something that none of us can validate actually
> works.
>
> Or better yet, delay implementing it until someone actually knows how to
> support it.

I was just talking to Chase Douglas here at the ubuntu sprint about this, and
he pointed me at https://launchpad.net/rinput which is a program which
can forward remote events including multitouch over the network (including
forwarding from both Linux and MacOS X clients). Maybe that would be
worth looking at since it's a working implementation of forwarding
multitouch events between systems?

-- PMM

Re: [Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

   Hi,

> I was just talking to Chase Douglas here at the ubuntu sprint about this, and
> he pointed me at https://launchpad.net/rinput which is a program which
> can forward remote events including multitouch over the network (including
> forwarding from both Linux and MacOS X clients). Maybe that would be
> worth looking at since it's a working implementation of forwarding
> multitouch events between systems?

Looks like they are basically using the linux input layer protocol.

cheers,
   Gerd

Re: [Qemu-devel] paravirtual mouse/tablet

[Avi Kivity]

On 01/13/2011 12:19 PM, Gerd Hoffmann wrote:
>   Hi,
>
> Just throwing a quick writeup into the ring to kickstart the design 
> discussion ;)
>
>
> typedef struct qemu_pvtablet_message {
>      uint32_t size;            /* whole message size */
>      uint32_t type;            /* qemu_pvtablet_type */
>      uint64_t tv_secs;
>      uint64_t tv_usecs;

time relative to what base?

can we actually provide it? if the tablet is remote, there may not be a 
synchronized time source.

>      union {
>          qemu_pvtablet_position position;
>          qemu_pvtablet_button   button;
>          qemu_pvtablet_display  display;
>      } m;
> } qemu_pvtablet_message;
>

So the message size for a qemu_pvtablet_position would only include the 
position member?


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] paravirtual mouse/tablet

[Anthony Liguori]

On 01/13/2011 10:18 AM, Avi Kivity wrote:
> On 01/13/2011 12:19 PM, Gerd Hoffmann wrote:
>>   Hi,
>>
>> Just throwing a quick writeup into the ring to kickstart the design 
>> discussion ;)
>>
>>
>> typedef struct qemu_pvtablet_message {
>>      uint32_t size;            /* whole message size */
>>      uint32_t type;            /* qemu_pvtablet_type */
>>      uint64_t tv_secs;
>>      uint64_t tv_usecs;
>
> time relative to what base?
>
> can we actually provide it? if the tablet is remote, there may not be 
> a synchronized time source.

I had the same thought, but if it's just treated as time since the init 
message, the guest is capable of working it out (minus drift).

>>      union {
>>          qemu_pvtablet_position position;
>>          qemu_pvtablet_button   button;
>>          qemu_pvtablet_display  display;
>>      } m;
>> } qemu_pvtablet_message;
>>
>
> So the message size for a qemu_pvtablet_position would only include 
> the position member?

I didn't comment on this because I was treating this as a protocol 
proposal and not actual code but usually the following is clearer:

struct qemu_pvtablet_hdr {
     uint32_t size;
     uint32_t type;
     uint64_t tv_secs;
     uint64_t tv_usecs;
};

struct qemu_pvtablet_position {
     struct qemu_pvtablet_hdr hdr;
     uint32_t x;
     uint32_t y;
};

union qemu_pvtablet_message {
     struct qemu_pvtablet_hdr hdr;
     struct qemu_pvtablet_position position;
     ...
};

Regards,

Anthony Liguori

Re: [Qemu-devel] paravirtual mouse/tablet

[Gerd Hoffmann]

On 01/13/11 17:18, Avi Kivity wrote:
> On 01/13/2011 12:19 PM, Gerd Hoffmann wrote:
>> Hi,
>>
>> Just throwing a quick writeup into the ring to kickstart the design
>> discussion ;)
>>
>>
>> typedef struct qemu_pvtablet_message {
>> uint32_t size; /* whole message size */
>> uint32_t type; /* qemu_pvtablet_type */
>> uint64_t tv_secs;
>> uint64_t tv_usecs;
>
> time relative to what base?

Guess that needs to be refined ;)

Just something relative (if available) should good enougth.  The 
intended purpose is being able to figure how much time passed between 
two events, so one can figure whenever two mouse clicks should be 
considered a double-click or not.  Ideally the timestamps from the 
original mouse event in the vnc/spice client would be passed all the way 
through to the guest.  Didn't check the protocols whenever they actually 
support that, but I think we should have this in the protocol even if 
they don't ...

cheers,
   Gerd

Re: [Qemu-devel] paravirtual mouse/tablet

[Avi Kivity]

On 01/13/2011 07:19 PM, Gerd Hoffmann wrote:
> On 01/13/11 17:18, Avi Kivity wrote:
>> On 01/13/2011 12:19 PM, Gerd Hoffmann wrote:
>>> Hi,
>>>
>>> Just throwing a quick writeup into the ring to kickstart the design
>>> discussion ;)
>>>
>>>
>>> typedef struct qemu_pvtablet_message {
>>> uint32_t size; /* whole message size */
>>> uint32_t type; /* qemu_pvtablet_type */
>>> uint64_t tv_secs;
>>> uint64_t tv_usecs;
>>
>> time relative to what base?
>
> Guess that needs to be refined ;)
>
> Just something relative (if available) should good enougth.  The 
> intended purpose is being able to figure how much time passed between 
> two events, so one can figure whenever two mouse clicks should be 
> considered a double-click or not.  Ideally the timestamps from the 
> original mouse event in the vnc/spice client would be passed all the 
> way through to the guest.  Didn't check the protocols whenever they 
> actually support that, but I think we should have this in the protocol 
> even if they don't ...

Ok.  We should then specify that the base is arbitrary (and get rid of 
tv_secs - 2^64 usecs is half a million years, which should be sufficient 
time to get a fully threaded qemu.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

[Qemu-devel] Re: [Spice-devel] paravirtual mouse/tablet

[Alon Levy]

On Thu, Jan 13, 2011 at 11:19:52AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> Just throwing a quick writeup into the ring to kickstart the design
> discussion ;)

I apologize for not reading everything before sending, I just don't want
to forget this detail: Whatever we design needs to address having multiple
mice in a single guest. This will be good for multiple user scenario (multiple
vnc connections already possible, multiple spice not yet but planned) where
each owns it's own mouse. This is supported already in X for a while (multi
pointer, different from the multi touch which is harder and not yet there).

> 
> cheers,
>   Gerd

> #ifndef __QEMU_PVTABLET__
> #define __QEMU_PVTABLET__ 1
> 
> /*
>  * qemu patavirtual tablet interface
>  */
> 
> #include <inttypes.h>
> 
> /* our virtio-serial channel */
> #define QEMU_PVTABLET_NAME "org.qemu.pvtablet.0"
> 
> enum qemu_pvtablet_buttons {
>     QEMU_PVTABLET_BTN_LEFT,
>     QEMU_PVTABLET_BTN_RIGHT,
>     QEMU_PVTABLET_BTN_MIDDLE,
>     /*
>      * we can handle up to 32 buttons although
>      * not every has a individual name ;)
>      */
> };
> 
> /* send button down+up events */
> typedef struct qemu_pvtablet_button {
>     uint32_t button;
>     uint32_t mask;
> } qemu_pvtablet_button;
> 
> /* send pointer move events */
> typedef struct qemu_pvtablet_position {
>     uint32_t pos_x;  /* range: 0 -> 0xffff */
>     uint32_t pos_y;  /* range: 0 -> 0xffff */
> } qemu_pvtablet_position;
> 
> /*
>  * specify which display all following events refering to
>  * (for multihead setups).
>  */
> typedef struct qemu_pvtablet_display {
>     /*
>      * FIXME: spice just uses the display channel number, but that
>      * doesn't fly in general ...
>      */
>     uint32_t spice_display_channel;
> } qemu_pvtablet_display;
> 
> enum qemu_pvtablet_type {
>     QEMU_PVTABLET_MSG_MOVE,     /* qemu_pvtablet_position */
>     QEMU_PVTABLET_MSG_BTN_DOWN, /* qemu_pvtablet_button */
>     QEMU_PVTABLET_MSG_BTN_UP,   /* qemu_pvtablet_button */
>     QEMU_PVTABLET_MSG_DISPLAY,  /* qemu_pvtablet_display */
> };
> 
> typedef struct qemu_pvtablet_message {
>     uint32_t size;            /* whole message size */
>     uint32_t type;            /* qemu_pvtablet_type */
>     uint64_t tv_secs;
>     uint64_t tv_usecs;
>     union {
>         qemu_pvtablet_position position;
>         qemu_pvtablet_button   button;
>         qemu_pvtablet_display  display;
>     } m;
> } qemu_pvtablet_message;
> 
> #endif /* __QEMU_PVTABLET__ */

> _______________________________________________
> Spice-devel mailing list
> Spice-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel