[Qemu-devel] Re: KVM usability

[Qemu-devel] Re: KVM usability

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 8577 bytes --]

[ adding qemu-devel to CC as most issues concern that project as well ]

Ingo Molnar wrote:
> * Avi Kivity <avi@redhat.com> wrote:
> 
>> On 02/26/2010 01:17 PM, Ingo Molnar wrote:
>>> Nobody is really 'in charge' of how KVM gets delivered to the user. You
>>> isolated the fun kernel part for you and pushed out the boring bits to
>>> user-space. So if mundane things like mouse integration sucks 'hey that's a
>>> user-space tooling problem', if file integration sucks then 'hey, that's an
>>> admin problem', if it cannot be used over the network 'hey, that's an Xorg
>>> problem', etc. etc.
>> btw, mouse integration works with -usbdevice tablet and recent
>> Fedoras, 'it was an X.org driver problem'.
>>
>> Really, I don't understand your problems.
> 
> I run bleeding edge rawhide on my main desktop so i just tried latest & 
> greatest KVM and Qemu bits and started up kvm-qemu with some Fedora and XP 
> images i had around:
> 
>   2.6.33-0.44.rc8.git0.fc13.x86_64
>   qemu-system-x86-0.12.2-6.fc13.x86_64
> 
> Here's my experience with it:
> 
>  - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
>    1680x1050 laptop screen. It's so small that initially i even overlooked 
>    that i started it. It should multiplex pixels up to a reasonable screen 
>    size by default.
> 
>  - The mouse is trapped straight away by default if you click into it. That's 
>    very annoying if you actually try to integrate a guest OS into your desktop 
>    - it's not just 'another, slightly weird app' but a sticky, opinionated GUI 
>    component that you have to fight with all the time.
> 
>  - Once trapped it's not obvious how to untrap the mouse. The qemu window 
>    title says:
> 
>           QEMU: Press Ctrl-ALT to exit grab
> 
>    Of course once you _know_ what a 'grab' is, you'll know where to look.
>    At minimum it should say:
> 
>           QEMU: Press Ctrl-ALT to exit mouse grab
> 
>    But to first-time users it's an annoying trap of the mouse and with no
>    obvious place to look for help. [besides, it doesnt tell which Ctrl and
>    which ALT to use - it's the left side. The right side Ctrl does not work.]
> 
>  - Graphics performance is awful even with the 640x480 miniature version.
>    During bootup I can see it drawing single characters. This is a Core2 
>    2.8GHz.
> 
>  - Sound does not work by default. I have to go dig into command-line options
>    to see that i need to add: "-soundhw all". Why isnt sound enabled by 
>    default?
> 
>  - Qemu images are not integrated into the rest of the desktop. If i click on 
>    a Qemu image it says:
> 
>      Could not display "/home/mingo/qemu/hda.img".
> 
>      The file is of an unknown type.
> 
>    10 years of Qemu and its base image format is still 'unknown' ?
> 
>  - Random bugs. I tried to boot some old Fedora image i had around, it says:
> 
>      spirit:~/qemu> qemu-kvm ./hda-fedora.img 
>      kvm: unhandled exit 80000021
>      kvm_run returned -22
> 
>    Bugs happen, but more important is what a user can do with it. To a user, 
>    what does this tell? Is it actionable? Does it give any URL to check? Any bug
>    submit mechanism to follow? Does it even tell what the code itself thinks
>    that happened? Nope - it just prints that error message (on the console, so 
>    to anyone starting it via a clicky interface wouldnt notice that there's 
>    something wrong) - and the guest session hanging indefinitely.
> 
>  - When it boots up, the Qemu window flips around its size crazily, as the BIOS,
>    the bootloader and the OS sets different screen resolutions. To the user that
>    technical detail is immaterial, what matters is an amateurish-looking app 
>    that flips its window size as if it was an adware popup window trying to 
>    avoid being caught.
> 
>  - There's no obvious way to activate paravirt drivers on the Windows side.
>    There's no friendly "install guest drivers" button to click on with Qemu.
> 
>    _Of course_ you will end up emulating hardware in KVM (and passing it through
>    to the guest once it's clear that emulation performance sucks) and sooner 
>    or later you will end up requesting unreasonable things of the host kernel 
>    to achieve that ...
> 
>  - Another small detail: there is zero on-screen help (beyond the Ctrl-ALT 
>    line) for a newbie to quickly find his way around it. No wiki address, no 
>    help, no nothing. There's not even any hint about what this window does. 
>    Which guest is it? In what state is that guest?
> 
>  - But i'm a more advanced user so i dont need help screens, i knew that the 
>    "go full screen" hotkey is:
> 
>            LeftCtrl-LeftALT-F
> 
>    ... except that it is a one-way road: pressing it for a second time does 
>    not restore the window, trapping me in the guest altogether! Ctrl-ALT does 
>    not exit the trap either. I had to shut down the guest to get back my X 
>    desktop.
> 
> etc., etc.
> 
> ( I could go on and on about finer details of good integration, like the 
>   difficulty of integrating host/guest files, networking, no way to quickly 
>   attach ISOs via that window, no way to activate networking, sound and no way 
>   to snapshot, no way to increase memory size except a command line option. )
> 
> etc - but that's not the point really: i only spent 10 minutes on this and i 
> didnt try hard at all - _11_ bugs/annoyances from all across the functionality 
> spectrum.
> 
> And the thing is _me_ reporting bugs does not matter at all in this picture, 
> so please dont come with "why didnt you report this?".
> 
> _Anyone_ with half a brain who takes a critical look at this virtualization 
> solution would notice the same. Still, it's essentially unchanged from 5 years 
> ago.
> 
> Why is that so? I have outlined my opinion that this is due to the artificial 
> package separation / over-modularization and no-one really being in charge of 
> "KVM quality as a whole" - and i'm wondering what your theory is how such a 
> state of affairs became possible.
> 
> I'm not trolling you at all: is it _really_ not obvious to you that the 
> KVM/qemu usability status quo honestly sucks, to an unbiased observer?
> 
> And AFAICS KVM developers keep concentrating on all the wrong things due to 
> that bad split/packaging: writing newer and newer low level kernel patches and 
> optimizations which are nice but in large part irrelevant because the 
> _fundamental basics_ of usability suck so much ... But to you it's probably 
> just another external package so not really something you can do much about, 
> right?

Yes, there are quite a few frontends around, also for desktop
virtualization. AQEMU [1], e.g., should address at least some of your
valuable usability criticisms. If GUIs are not yet optimally integrated
with QEMU and if there is something that can be done inside QEMU or even
KVM, let's discuss that together with the related communities. However,
that is definitely not a valid reason for KVM developers to stop
improving the virtualization core for demanding use cases and start
writing GUIs.

Likely it is unfortunate that there is not The One And Only graphical
frontend for the desktop scenario we can focus on (But is it unfortunate
for desktop Linux that there is still Gnome vs. KDE? I bet there are
zillion opinions on that.). But maybe this discussion here will have the
positive effect that the QEMU/KVM community starts giving more feedback
to some of those projects, and vice versa.

> 
> Really, the KVM design is so nice in many regards and Qemu has come forward 
> leaps and bounds in the past few years as well, how can you miss such basic 
> areas of weakness? 'First impression' is the thing that gives you new 
> developers - it's any OSS project's bread and butter.

'First impression' mostly gives us more users - a worthwhile goal as well!

But developers are rather attracted by rich features, a flexible,
extensible design, a working community and - of course - a proper
licensing model. Look at Virtualbox, they probably meet your usability
requirements out of the box. Do they have a developer community? Not at
all because they miss some of that key requirements. Also for those
reasons, Vbox dropped from the list of candidates for many
virtualization projects @work, and we decided to invest in the true
Linux stack.

Jan

[1] http://sourceforge.net/projects/aqemu/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Chris Webb]

Avi Kivity <avi@redhat.com> writes:

> On 03/02/2010 11:34 AM, Jernej Simončič wrote:
> >On Tuesday, March 2, 2010, 9:21:18, Chris Webb wrote:
> >
> >>I remember about a year ago, someone asserting on the list that -usbdevice
> >>tablet was very CPU intensive even when not in use, and should be avoided if
> >>mouse support wasn't needed, e.g. on non-graphical VMs. Was that actually a
> >>significant hit, and is it still true today?
> >It would appear that this is still the case, at least on slower hosts
> >- on Atom Z530 (1,6GHz), the XP VM uses ~30% CPU when idle with
> >-usbdevice tablet, but only ~4% without it. However, on a faster host
> >(Core2 Quad 2,66GHz), there's practically no difference (Vista x64 VM
> >uses ~1% CPU when idle regardless of -usbdevice tablet).
> 
> Looks like the tablet is set to 100 Hz polling rate.  We may be able
> to get away with 30 Hz or even less (ep_bInterval, in ms, in
> hw/usb-wacom.c).

Hi Avi. Sorry for the very late follow-up, but I decided to experiment with
this. The cpu impact of the usb tablet device shows up fairly clearly on a
crude test on my (relatively low-spec) desktop. Running an idle Fedora 11
livecd on qemu-kvm 0.12.3, top shows around 0.1% of my cpu in use, but this
increases to roughly 5% when specifying -usbdevice tablet, and more detailed
examination with perf record/report suggests about a factor of thirty too.

It's actually a more general symptom with USB or at least HID devices by the
look of things: although -usb doesn't increase CPU use on its own, the same
increase in load can also be triggered by -usbdevice keyboard or mouse.
However, running with all three of -usbdevice mouse, keyboard and tablet
doesn't increase load any more than just one of these.

Changing the USB tablet polling interval from 10ms to 100ms in both
hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase in
bInterval shown in lsusb -v in the guest and the hint of jerky mouse
movement I expected from setting this value so high. A similar change to the
polling interval for the keyboard and mouse also made no difference to their
performance impact.

Taking the FRAME_TIMER_FREQ down to 100 in hw/usb-uhci.c does seem to reduce
the CPU load quite a bit, but at the expense of making the USB tablet (and
presumably all other USB devices) very laggy.

Could there be some bug here that causes the usb hid devices to wake qemu at
the maximum rate possible (FRAME_TIMER_FREQ?) rather than the configured
polling interval?

Best wishes,

Chris.

PS Vmmouse works fine as an absolute pointing device in the place of
-usbdevice tablet without the performance impact, but this isn't supported
out of the box with typical linux live CDs (e.g. Fedora 11 and 12 or
Knoppix) so unfortunately it's probably less suitable as a default
configuration to expose to end-users.

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > Looks like the tablet is set to 100 Hz polling rate.  We may be able
> > to get away with 30 Hz or even less (ep_bInterval, in ms, in
> > hw/usb-wacom.c).
>
> Changing the USB tablet polling interval from 10ms to 100ms in both
> hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase
>  in bInterval shown in lsusb -v in the guest and the hint of jerky mouse
>  movement I expected from setting this value so high. A similar change to
>  the polling interval for the keyboard and mouse also made no difference to
>  their performance impact.

The USB HID devices implement the SET_IDLE command, so the polling interval 
will have no real effect on performance.

My guess is that the overhead you're seeing is entirely from the USB host 
adapter having to wake up and check the transport descriptor lists. This will 
only result in the guest being woken if a device actually responds (as 
mentioned above it should not).

>Taking the FRAME_TIMER_FREQ down to 100 in hw/usb-uhci.c does seem to reduce
>the CPU load quite a bit, but at the expense of making the USB tablet (and
>presumably all other USB devices) very laggy.

The guest USB driver explicitly decides which devices to poll each frame. 
Slowing down the frame rate will effectively change the polling period by the 
same factor. e.g. the HID device requests a polling rate of 10ms, you slowed 
down frame rate by 10x, so you're efectively only polling every 100ms.

If you want a quick and nasty hack then you can probably make the device wake 
up less often, and process multiple frames every wakeup.  However this is 
probably going to do bad things (at best extremely poor performance) when 
using actual USB devices.

Fixing this properly is hard because the transport descriptor lists are stores 
in system RAM, and polled by the host adapter.  The first step is to read the 
whole table of descriptors, and calculate when the next event is due. However 
the guest will not explicitly notify the HBA when these tables are modified, 
so you also need some sort of MMU trap to trigger recalculation.

This only gets you down to the base polling interval requested by the device.  
Increasing this interval causes significant user visible latency, so 
increasing it is not an option. The guest is also likely to distribute polling 
events evenly, further reducing the effective sleep interval.  To fix this you 
need additional APIs so that a device can report when an endpoint will become 
unblocked, rather than just waiting to be polled and NAKing the request.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Avi Kivity]

On 04/04/2010 05:25 PM, Paul Brook wrote:
>>> Looks like the tablet is set to 100 Hz polling rate.  We may be able
>>> to get away with 30 Hz or even less (ep_bInterval, in ms, in
>>> hw/usb-wacom.c).
>>>        
>> Changing the USB tablet polling interval from 10ms to 100ms in both
>> hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase
>>   in bInterval shown in lsusb -v in the guest and the hint of jerky mouse
>>   movement I expected from setting this value so high. A similar change to
>>   the polling interval for the keyboard and mouse also made no difference to
>>   their performance impact.
>>      
> The USB HID devices implement the SET_IDLE command, so the polling interval
> will have no real effect on performance.
>    

On a Linux guest (F12), I see 125 USB interrupts per second with no 
mouse movement, so something is broken (on the guest or host).

> My guess is that the overhead you're seeing is entirely from the USB host
> adapter having to wake up and check the transport descriptor lists. This will
> only result in the guest being woken if a device actually responds (as
> mentioned above it should not).
>    

A quick profile on the host side doesn't show this.  Instead, I see a 
lot of select() overhead.  Surprising as there are ~10 descriptors being 
polled, so ~1200 polls per second.  Maybe epoll will help here.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > The USB HID devices implement the SET_IDLE command, so the polling
> > interval will have no real effect on performance.
> 
> On a Linux guest (F12), I see 125 USB interrupts per second with no
> mouse movement, so something is broken (on the guest or host).

Turns out to be a a bug in the UHCI emulation. It should only raise an 
interrupt if the transfer actually completes (i.e. the active bit is set to 
zero). Fixed by 5bd2c0d7.

I was testing with an OHCI controller, which does not have this bug.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > My guess is that the overhead you're seeing is entirely from the USB host
> > adapter having to wake up and check the transport descriptor lists. This
> > will only result in the guest being woken if a device actually responds
> > (as mentioned above it should not).
> 
> A quick profile on the host side doesn't show this.  Instead, I see a
> lot of select() overhead.

This actually confirms my hypothesis. After fixing the UHCI bug the guest is 
completely idle, but the host still needs to wake up at 1ms intervals to do 
UHCI emulation. I can believe that the most visible part of this is the 
select() syscall.

> Surprising as there are ~10 descriptors being
> polled, so ~1200 polls per second.  Maybe epoll will help here.

I'm not sure where you get 1200 from.  select will be called once per host 
wakeup. i.e. if the USB controller is enabled then 1k times per second due to 
the frame tick.

Are you sure there are actually 10 descriptors being polled? Remember that the 
nfds argument is the value of the largest fd in the set (+1), not the number 
of descriptors in the set.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Avi Kivity]

On 04/05/2010 12:53 AM, Paul Brook wrote:
>
>> Surprising as there are ~10 descriptors being
>> polled, so ~1200 polls per second.  Maybe epoll will help here.
>>      
> I'm not sure where you get 1200 from.  select will be called once per host
> wakeup. i.e. if the USB controller is enabled then 1k times per second due to
> the frame tick.
>    

That was 125 interrupts/sec x 10 fds ('poll' here was the per-file 
callback in the kernel, not the select call).  With your numbers, it's 
more like 10k polls/sec which can be eliminated.

> Are you sure there are actually 10 descriptors being polled? Remember that the
> nfds argument is the value of the largest fd in the set (+1), not the number
> of descriptors in the set.
>    

I was estimating from the strace parsed output, the real number is 7.  
So the kernel calls 7k callbacks/sec only to return with a timeout.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.