[Qemu-devel] Request for comment - dynamic VNC keyboard mapping

[Qemu-devel] Request for comment - dynamic VNC keyboard mapping

[Mario]

Hi all,

I have a question related to the VNC server keyboard settings. Currently the user of qemu has to decide before VM startup which language is used for VNC keyboard mapping. If no keyboard is configured, the en-us keyboard will be loaded for keysym to scancode conversion. A later reconfiguration of the keyboard is not possible at present.

So here is my idea:

The VNC Display context stores a list of loaded keyboards (like vs->vd->kbds). If no keyboard was specified, at least the default will be added to the list ("en-us"?). If a client connects, a copy of the (latest configured) pointer will be stored within the VncState structure. For any keys sent by this client the own keyboard reference will be used.

As (in theory) every VNC client may have a differen keyboard layout, in my point of view it makes sense to attach the keyboard used for keysym2scancode() to the client context rather to the VNC server context. However as (most likely) all clients do have the same keyboard mapping it would be an overhead to parse and store the mapping list each time the client connects.

Now in my idea two mechanisms may change the behavior of the client keyboard.

1.) a qmp command was sent, that sets /configures for a particular/default keyboard.
2.) within the VNC protocol a "VNC_MSG_CLIENT_QEMU/VNC_MSG_CLIENT_QEMU_EXT_KEYBOARD_LAYOUT" message arrives

If the new keyboard was not already loaded and stored inside vs->vd->kbds , qemu initializes the new mapping. The affected VNC client (VncState) again stores the reference only.

This enables switching the keyboard layout during runtime as well as using multiple different keyboard layouts at atime.

Together with a corresponding patch I would also add a patch for libvncclient to have a reference implmentation for the integrated keyboard layout switching.

What are your thoughts about this kind of improvement?

Thanks,
Mario

[Qemu-devel] Request for comment - dynamic VNC keyboard mapping

[Mario]

Hi all,

I have a question related to the VNC server keyboard settings. Currently the user of qemu has to decide before VM startup which language is used for VNC keyboard mapping. If no keyboard is configured, the en-us keyboard will be loaded for keysym to scancode conversion. A later reconfiguration of the keyboard is not possible at present.

So here is my idea:

The VNC Display context stores a list of loaded keyboards (like vs->vd->kbds). If no keyboard was specified, at least the default will be added to the list ("en-us"?). If a client connects, a copy of the (latest configured) pointer will be stored within the VncState structure. For any keys sent by this client the own keyboard reference will be used.

As (in theory) every VNC client may have a differen keyboard layout, in my point of view it makes sense to attach the keyboard used for keysym2scancode() to the client context rather to the VNC server context. However as (most likely) all clients do have the same keyboard mapping it would be an overhead to parse and store the mapping list each time the client connects.

Now in my idea two mechanisms may change the behavior of the client keyboard.

1.) a qmp command was sent, that sets /configures for a particular/default keyboard.
2.) within the VNC protocol a "VNC_MSG_CLIENT_QEMU/VNC_MSG_CLIENT_QEMU_EXT_KEYBOARD_LAYOUT" message arrives

If the new keyboard was not already loaded and stored inside vs->vd->kbds , qemu initializes the new mapping. The affected VNC client (VncState) again stores the reference only.

This enables switching the keyboard layout during runtime as well as using multiple different keyboard layouts at atime.

Together with a corresponding patch I would also add a patch for libvncclient to have a reference implmentation for the integrated keyboard layout switching.

What are your thoughts about this kind of improvement?

Thanks,
Mario

Re: [Qemu-devel] Request for comment - dynamic VNC keyboard mapping

[Daniel P. Berrangé]

On Fri, May 03, 2019 at 01:47:15PM +0200, Mario wrote:
> Hi all,
> 
> I have a question related to the VNC server keyboard settings. Currently
> the user of qemu has to decide before VM startup which language is used
> for VNC keyboard mapping. If no keyboard is configured, the en-us keyboard
> will be loaded for keysym to scancode conversion. A later reconfiguration
> of the keyboard is not possible at present.

That isn't actually correct. In fact best practice is to *not* set and
"-k keymap" arg when launching QEMU.

Any kind of keysym to scancode conversion done by QEMU is inherantly
lossy and fragile as it requires a 100% matching keymap between QEMU
and the guest OS. QEMU keymaps are much better than the past, but
they're never going to be guaranteed to match what the guest OS is
using because they're potentialy using completely different data
sources.

By not setting "-k", you let the QEMU VNC server activate its keyboard
scancode VNC protocol extension. With this active, a compatible VNC
client will then send raw hardware scancodes to QEMU alongside keysyms.

QEMU will thus ignore the keysyms and send the scancodes straight to
the guest with no keymap conversion performed at all. 

With this working you can just dynamically change your guest OS keymap
as needed to match your VNC client machine's keyboard and QEMU stays
out of the way entirely. 

Obviously the important thing here is to have a VNC client that supports
this scancode extension. This was originally done in the GTK-VNC widget,
and thus virt-viewer, remote-viewer, vinagre, virt-manager, GNOME Boxes
all support this.  More recently the TigerVNC client app also supports
this, as does the in browser noVNC client.

> So here is my idea:
> 
> The VNC Display context stores a list of loaded keyboards (like
> vs->vd->kbds). If no keyboard was specified, at least the default
> will be added to the list ("en-us"?). If a client connects, a copy
> of the (latest configured) pointer will be stored within the
> VncState structure. For any keys sent by this client the own
> keyboard reference will be used.
> 
> As (in theory) every VNC client may have a differen keyboard layout,
> in my point of view it makes sense to attach the keyboard used for
> keysym2scancode() to the client context rather to the VNC server
> context. However as (most likely) all clients do have the same
> keyboard mapping it would be an overhead to parse and store the
> mapping list each time the client connects.
> 
> Now in my idea two mechanisms may change the behavior of the client
> keyboard.
> 
> 1.) a qmp command was sent, that sets /configures for a particular/
>     default keyboard.
> 2.) within the VNC protocol a "VNC_MSG_CLIENT_QEMU/
>     VNC_MSG_CLIENT_QEMU_EXT_KEYBOARD_LAYOUT" message arrives
> 
> If the new keyboard was not already loaded and stored inside
> vs->vd->kbds , qemu initializes the new mapping. The affected VNC
> client (VncState) again stores the reference only.
> 
> This enables switching the keyboard layout during runtime as well
> as using multiple different keyboard layouts at atime.
> 
> Together with a corresponding patch I would also add a patch for
> libvncclient to have a reference implmentation for the integrated
> keyboard layout switching.
> 
> What are your thoughts about this kind of improvement?

I don't think it is worth creating a new VNC protocol extension for this,
since any kind of keymap conversion done on the server is inherantly
lossy so can never be bug free. Since a new extension requires modifying
VNC clients regardless, it is better to just modify them to support the
raw scancode extension instead.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Request for comment - dynamic VNC keyboard mapping

[Daniel P. Berrangé]

On Fri, May 03, 2019 at 01:47:15PM +0200, Mario wrote:
> Hi all,
> 
> I have a question related to the VNC server keyboard settings. Currently
> the user of qemu has to decide before VM startup which language is used
> for VNC keyboard mapping. If no keyboard is configured, the en-us keyboard
> will be loaded for keysym to scancode conversion. A later reconfiguration
> of the keyboard is not possible at present.

That isn't actually correct. In fact best practice is to *not* set and
"-k keymap" arg when launching QEMU.

Any kind of keysym to scancode conversion done by QEMU is inherantly
lossy and fragile as it requires a 100% matching keymap between QEMU
and the guest OS. QEMU keymaps are much better than the past, but
they're never going to be guaranteed to match what the guest OS is
using because they're potentialy using completely different data
sources.

By not setting "-k", you let the QEMU VNC server activate its keyboard
scancode VNC protocol extension. With this active, a compatible VNC
client will then send raw hardware scancodes to QEMU alongside keysyms.

QEMU will thus ignore the keysyms and send the scancodes straight to
the guest with no keymap conversion performed at all. 

With this working you can just dynamically change your guest OS keymap
as needed to match your VNC client machine's keyboard and QEMU stays
out of the way entirely. 

Obviously the important thing here is to have a VNC client that supports
this scancode extension. This was originally done in the GTK-VNC widget,
and thus virt-viewer, remote-viewer, vinagre, virt-manager, GNOME Boxes
all support this.  More recently the TigerVNC client app also supports
this, as does the in browser noVNC client.

> So here is my idea:
> 
> The VNC Display context stores a list of loaded keyboards (like
> vs->vd->kbds). If no keyboard was specified, at least the default
> will be added to the list ("en-us"?). If a client connects, a copy
> of the (latest configured) pointer will be stored within the
> VncState structure. For any keys sent by this client the own
> keyboard reference will be used.
> 
> As (in theory) every VNC client may have a differen keyboard layout,
> in my point of view it makes sense to attach the keyboard used for
> keysym2scancode() to the client context rather to the VNC server
> context. However as (most likely) all clients do have the same
> keyboard mapping it would be an overhead to parse and store the
> mapping list each time the client connects.
> 
> Now in my idea two mechanisms may change the behavior of the client
> keyboard.
> 
> 1.) a qmp command was sent, that sets /configures for a particular/
>     default keyboard.
> 2.) within the VNC protocol a "VNC_MSG_CLIENT_QEMU/
>     VNC_MSG_CLIENT_QEMU_EXT_KEYBOARD_LAYOUT" message arrives
> 
> If the new keyboard was not already loaded and stored inside
> vs->vd->kbds , qemu initializes the new mapping. The affected VNC
> client (VncState) again stores the reference only.
> 
> This enables switching the keyboard layout during runtime as well
> as using multiple different keyboard layouts at atime.
> 
> Together with a corresponding patch I would also add a patch for
> libvncclient to have a reference implmentation for the integrated
> keyboard layout switching.
> 
> What are your thoughts about this kind of improvement?

I don't think it is worth creating a new VNC protocol extension for this,
since any kind of keymap conversion done on the server is inherantly
lossy so can never be bug free. Since a new extension requires modifying
VNC clients regardless, it is better to just modify them to support the
raw scancode extension instead.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Request for comment - dynamic VNC keyboard mapping

[Daniel Henrique Barboza]

Hi Mario,

Sorry for the delayed answer.  As Daniel already said, any solution based
on how "-k <keymap>" works are too daunting to be maintained - there
are a lot of known keymaps to be aware of.

There's also the case of changing keymaps during the same session.
Speaking from experience: my laptop has a regular en-us keyboard, but
I also use it with an external Brazilian keyboard (which has the "Ç" key and
other punctuation keys). I happen to use some VMs in this hardware too.
It is not uncommon in my usage, in the same VNC session, a change in the
current keyboard layout en_us -> pt_br back and forth.

This is only possible because gtk-vnc does not care about my layout, but for
the keycodes being pressed. In my case, instead of sending the "Ç" symbol,
it would send the keycode of the 11th keyboard key located at the forth 
line.
The translation of keycode to keysym happens up to the OS layer.

gtk-vnc implements the RFB scancode extension that allows for the keycode to
be sent to VNC. This implementation isn't complicated to pull it off, as 
Daniel
also mentioned in his reply. Many VNC clients already supports it. If 
you want
something to start from, I've implemented this extension in a browser VNC
client called noVNC a few years ago and I talked about it here:

https://danielhb.github.io/article/2019/05/06/noVNC-QEMU-RFB.html

You may also want to check Daniel's article about all this scancode/keycode/
keysym theory here:

https://www.berrange.com/tags/rfb/



Thanks,


DHB

On 5/3/19 8:47 AM, Mario wrote:
> Hi all,
>
> I have a question related to the VNC server keyboard settings. Currently the user of qemu has to decide before VM startup which language is used for VNC keyboard mapping. If no keyboard is configured, the en-us keyboard will be loaded for keysym to scancode conversion. A later reconfiguration of the keyboard is not possible at present.
>
> So here is my idea:
>
> The VNC Display context stores a list of loaded keyboards (like vs->vd->kbds). If no keyboard was specified, at least the default will be added to the list ("en-us"?). If a client connects, a copy of the (latest configured) pointer will be stored within the VncState structure. For any keys sent by this client the own keyboard reference will be used.
>
> As (in theory) every VNC client may have a differen keyboard layout, in my point of view it makes sense to attach the keyboard used for keysym2scancode() to the client context rather to the VNC server context. However as (most likely) all clients do have the same keyboard mapping it would be an overhead to parse and store the mapping list each time the client connects.
>
> Now in my idea two mechanisms may change the behavior of the client keyboard.
>
> 1.) a qmp command was sent, that sets /configures for a particular/default keyboard.
> 2.) within the VNC protocol a "VNC_MSG_CLIENT_QEMU/VNC_MSG_CLIENT_QEMU_EXT_KEYBOARD_LAYOUT" message arrives
>
> If the new keyboard was not already loaded and stored inside vs->vd->kbds , qemu initializes the new mapping. The affected VNC client (VncState) again stores the reference only.
>
> This enables switching the keyboard layout during runtime as well as using multiple different keyboard layouts at atime.
>
> Together with a corresponding patch I would also add a patch for libvncclient to have a reference implmentation for the integrated keyboard layout switching.
>
> What are your thoughts about this kind of improvement?
>
> Thanks,
> Mario