[Qemu-devel] Support for VNC LED state extension proposal

[Qemu-devel] Support for VNC LED state extension proposal

[Lei Li]

Hi,

This proposal aims to add support for VNC LED state extension to
Qemu VNC server. It also contains the proposal of this VNC extension,
althrough it should be sent to some other mailing, but we'd like to
send to Qemu mailing list first to have your suggestion. Please
let me know your ideas and thoughts.

When access a guest by console through VNC, there might be
mismatch between the lock keys notification LED on the computer
running the VNC client session and the current status of the lock
keys on the guest machine. This happens because the VNC protocol
does not have any support to deal with setting led state.

To solve this, we plan to:
1) Add LED state extension to VNC proposal (As the description in
   'Proposal for adding LED state extensions to VNC protocol').

2) Support VNC LED state extension in Qemu.

The general flow is:
   
1) A client that supports the LED state extension starts by requesting
   the led flags Psuedo-encoding.

2) Server then send a message to the client with the current LED state
   if this extension is supported.

3) Clinet receive this message and sync with server by setting the local
   LED state.

4) Everytime the LED state is changed, server will send an update of the
   current LED state to client, and client will resync with server.

For instance, if the change of LED state is requested from remote client
through VNC, client will send scancode to VNC server (This is base on the
extension 'QEMU Extended Key Event Message' Anthony have already added to
VNC protocol as reference link below), server pass the scancode to OS/X11
and then OS/X11 handle the setting of keyboard LED. After the setting
job, server receive the LED state event from OS/X11, will send current LED state
message to client. Then, client will resync the LED state with server
based on this message by setting the local LED state.

http://tigervnc.svn.sourceforge.net/viewvc/tigervnc/rfbproto/rfbproto.rst?content-type=text/plain
   

------

Proposal for adding LED state extensions to VNC protocol

QEMU LED state extension Psuedo-encoding
----------------------------------------

A client that supports the LED extension starts by requesting the led
flags pseudo-encoding declaring that it is capable of accepting the
`QEMU LED state extensions Server Message`_. The server, in turn, will
send the current led flags in an update `FrameBufferUpdate`_ with the
matching psuedo-encoding and the *num-rectangles* field set to 1, however,
no rectangles will actually be sent. After sending this notification,
server will update the led flags everytime the led state is changed in
the update, client could resync the led state with the server by setting
the local led state accordingly.

=========================== =================== =======================
No. of bytes                Type                Description
=========================== =================== =======================
1                           ``U8``              *led-flags*
=========================== =================== =======================

The example psuedo-encodings for *led-flags* defined as following:

======= ===============================================================
Code    Description
======= ===============================================================
100     CapsLock is set
010     NumLock is set
001     ScrollLock is set
110     CapsLock and NumLock are set
111     CapsLock, NumLock and ScrollLock are set
======= ===============================================================

QEMU LED state extensions Server Message
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This submessage allows the server to send the current led state for the
lock keys (CapsLock, NumLock and ScrollLock) to client.

=============== ==================== ========== =======================
No. of bytes    Type                 [Value]    Description
=============== ==================== ========== =======================
1               ``U8``               255        *message-type*
1               ``U8``               2          *submessage-type*
1               ``U8``                          *led-flags*
=============== ==================== ========== =======================

The *led-flags* fields take the same values as described for the 'QEMU
LEDs state extension Psuedo-encoding'_.

Re: [Qemu-devel] Support for VNC LED state extension proposal

[Gerd Hoffmann]

  Hi,

> When access a guest by console through VNC, there might be
> mismatch between the lock keys notification LED on the computer
> running the VNC client session and the current status of the lock
> keys on the guest machine. This happens because the VNC protocol
> does not have any support to deal with setting led state.

What is the exact issue you are trying to fix here?

The vnc server code in qemu already tracks the keyboard led state and
will insert synthetic key events if needed to make sure vnc client and
guest machine stay in sync.

cheers,
  Gerd

Re: [Qemu-devel] Support for VNC LED state extension proposal

[Anthony Liguori]

Gerd Hoffmann <kraxel@redhat.com> writes:

>   Hi,
>
>> When access a guest by console through VNC, there might be
>> mismatch between the lock keys notification LED on the computer
>> running the VNC client session and the current status of the lock
>> keys on the guest machine. This happens because the VNC protocol
>> does not have any support to deal with setting led state.
>
> What is the exact issue you are trying to fix here?
>
> The vnc server code in qemu already tracks the keyboard led state and
> will insert synthetic key events if needed to make sure vnc client and
> guest machine stay in sync.

If the guest changes LED state, we want to the client to update the
physical keyboard LED state.

Think thin client without an X server.

Regards,

Anthony Liguori

>
> cheers,
>   Gerd

Re: [Qemu-devel] Support for VNC LED state extension proposal

[Gerd Hoffmann]

On 04/15/13 16:41, Anthony Liguori wrote:
> Gerd Hoffmann <kraxel@redhat.com> writes:
> 
>>   Hi,
>>
>>> When access a guest by console through VNC, there might be
>>> mismatch between the lock keys notification LED on the computer
>>> running the VNC client session and the current status of the lock
>>> keys on the guest machine. This happens because the VNC protocol
>>> does not have any support to deal with setting led state.
>>
>> What is the exact issue you are trying to fix here?
>>
>> The vnc server code in qemu already tracks the keyboard led state and
>> will insert synthetic key events if needed to make sure vnc client and
>> guest machine stay in sync.
> 
> If the guest changes LED state, we want to the client to update the
> physical keyboard LED state.

Guests usually change the LED state in response to a keyboard event, in
which case the physical keyboard LEDs already have the correct state.

There are exceptions of course.  Guest linux kernel panic-ing, then
blinking capslock.  Multiple vnc clients connected at the same time.

> Think thin client without an X server.

Should be even less an issue there.

With X & multiple windows guest+host capslock/numlock state go out of
sync because of sequences like move-kbd-focus-away-from-vncclient-window
+ press-{caps,num}lock + move-focus-back (and thats why we have the
logic to insert synthetic key events to resync in the first place).

What is your vnc client?  Does it support VNC_ENCODING_EXT_KEY_EVENT?

cheers,
  Gerd

Re: [Qemu-devel] Support for VNC LED state extension proposal

[Anthony Liguori]

Gerd Hoffmann <kraxel@redhat.com> writes:

> On 04/15/13 16:41, Anthony Liguori wrote:
>> Gerd Hoffmann <kraxel@redhat.com> writes:
>> 
>>>   Hi,
>>>
>>>> When access a guest by console through VNC, there might be
>>>> mismatch between the lock keys notification LED on the computer
>>>> running the VNC client session and the current status of the lock
>>>> keys on the guest machine. This happens because the VNC protocol
>>>> does not have any support to deal with setting led state.
>>>
>>> What is the exact issue you are trying to fix here?
>>>
>>> The vnc server code in qemu already tracks the keyboard led state and
>>> will insert synthetic key events if needed to make sure vnc client and
>>> guest machine stay in sync.
>> 
>> If the guest changes LED state, we want to the client to update the
>> physical keyboard LED state.
>
> Guests usually change the LED state in response to a keyboard event, in
> which case the physical keyboard LEDs already have the correct state.

Ack.

> There are exceptions of course.  Guest linux kernel panic-ing, then
> blinking capslock.  Multiple vnc clients connected at the same time.

Ack.

>> Think thin client without an X server.
>
> Should be even less an issue there.
>
> With X & multiple windows guest+host capslock/numlock state go out of
> sync because of sequences like move-kbd-focus-away-from-vncclient-window
> + press-{caps,num}lock + move-focus-back (and thats why we have the
> logic to insert synthetic key events to resync in the first place).
>
> What is your vnc client?  Does it support VNC_ENCODING_EXT_KEY_EVENT?

It's uses gvnc as a protocol library and renders via fbdev.  It reads
keyboard events by putting /dev/tty into mediumraw mode and uses ext key
events exclusively.  It has no knowledge of the guest keymap.

It is possible to attempt to keep the LED state in sync by tracking the
lock key state in the client.  However, there are drawbacks to this.

Since the client isn't aware of the guest keymap, it technically doesn't
know which keys are the lock keys.  Admittedly, this is a solvable
problem but the solution is certainly not elegant.

There's also the problem with guest initiated LED changes that don't
correspond to lock key presses.  Admittedly not a huge problem but still
there nonetheless.

I believe VMware already has a VNC extension for passing LED state
changes and I think having an open extension for this is a Good Thing.
It's a pretty obvious missing piece in the VNC protocol.

Regards,

Anthony Liguori

>
> cheers,
>   Gerd

Re: [Qemu-devel] Support for VNC LED state extension proposal

[Gerd Hoffmann]

  Hi,

>> What is your vnc client?  Does it support VNC_ENCODING_EXT_KEY_EVENT?
> 
> It's uses gvnc as a protocol library and renders via fbdev.  It reads
> keyboard events by putting /dev/tty into mediumraw mode and uses ext key
> events exclusively.  It has no knowledge of the guest keymap.

Neat.  URL?

Does the linux kernel keep track of {caps,num}lock state (+leds) with
the keyboard in mediumraw mode?  Or does it expect the userspace app set
the led state then?

No UI client (neither vnc nor spice) knows the guest keymap btw.

> I believe VMware already has a VNC extension for passing LED state
> changes and I think having an open extension for this is a Good Thing.
> It's a pretty obvious missing piece in the VNC protocol.

Anyone tried to bug vmware to open the specs for the existing extension?

cheers,
  Gerd

Re: [Qemu-devel] Support for VNC LED state extension proposal

[Anthony Liguori]

Gerd Hoffmann <kraxel@redhat.com> writes:

>   Hi,
>
>>> What is your vnc client?  Does it support VNC_ENCODING_EXT_KEY_EVENT?
>> 
>> It's uses gvnc as a protocol library and renders via fbdev.  It reads
>> keyboard events by putting /dev/tty into mediumraw mode and uses ext key
>> events exclusively.  It has no knowledge of the guest keymap.
>
> Neat.  URL?

We haven't published yet but I keep meaning too...  It's really quite simple.

> Does the linux kernel keep track of {caps,num}lock state (+leds) with
> the keyboard in mediumraw mode?

Nope.

> Or does it expect the userspace app set the led state then?

Yup.

> No UI client (neither vnc nor spice) knows the guest keymap btw.

Well, they do actually.

gtk-vnc has key modifier tracking code.  I assume gtk-spice does too.
This code looks at the GdkEventKey::keysym which has gone through the
host keymap.  The assumption is that the host keymap == the guest keymap
at least as far as modifiers are concerned.

Take a look at key_event in vncdisplay.c in gtk-vnc.

>> I believe VMware already has a VNC extension for passing LED state
>> changes and I think having an open extension for this is a Good Thing.
>> It's a pretty obvious missing piece in the VNC protocol.
>
> Anyone tried to bug vmware to open the specs for the existing
> extension?

I have in the past but that was many years ago at this point.

Regards,

Anthony Liguori

>
> cheers,
>   Gerd