[Qemu-devel] qemu guest agent spins in poll/nanosleep(100ms) when nothing is listening on host

[Qemu-devel] qemu guest agent spins in poll/nanosleep(100ms) when nothing is listening on host

[Daniel P. Berrange]

I've been doing some experimentation with the QEMU guest agent and have
noticed that when nothing is connected on the host side of the virtio
serial channel, the guest agent just spins in a pool/sleep(100ms) loop.
I know you'd ordinarily expect some mgmt app in the host to be listening
to the other end of the channel, but it still seems suboptimal to have
to spin in a loop like this when nothing is listening, constantly causing
wakeups in an otherwise idle guest.

Looking at the qemu-ga.c code I see two places where it might handle
a poll event and then sleep, when nothing is on the other end of the
virtio serial socket.


   case G_IO_STATUS_AGAIN:
        /* virtio causes us to spin here when no process is attached to
         * host-side chardev. sleep a bit to mitigate this
         */
        if (s->virtio) {
            usleep(100*1000);
        }
        return true;

   ....


    } else if (strcmp(s->method, "virtio-serial") == 0) {
        /* we spin on EOF for virtio-serial, so back off a bit. also,
         * dont close the connection in this case, it'll resume normal
         * operation when another process connects to host chardev
         */
        usleep(100*1000);
        goto out_noclose;
    }


I get the feeling that this kind of problem inherant in the use of any
virtio-serial channel, in the same way you can't detect EOF for a regular
serial device channel either. Given that virtio-serial is a nice paravirt
device, is there anything we can do to it, to allow better handling of
EOF by applications ?

Or perhaps there is some way to make use of epoll() in edge-triggered
mode to detect it already, because IIUC, edge-triggered mode should only
fire once for the EOF condition, and then not fire again until something
in the host actually sends some data ?

Of course glib's event loop doesn't support edge-triggered events/epoll,
but perhaps we could just call epoll() directly in the event handler,
instead of the usleep() call ?

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] qemu guest agent spins in poll/nanosleep(100ms) when nothing is listening on host

[Michael Roth]

On Thu, 6 Oct 2011 12:31:05 +0100, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> I've been doing some experimentation with the QEMU guest agent and have
> noticed that when nothing is connected on the host side of the virtio
> serial channel, the guest agent just spins in a pool/sleep(100ms) loop.
> I know you'd ordinarily expect some mgmt app in the host to be listening
> to the other end of the channel, but it still seems suboptimal to have
> to spin in a loop like this when nothing is listening, constantly causing
> wakeups in an otherwise idle guest.
> 
> Looking at the qemu-ga.c code I see two places where it might handle
> a poll event and then sleep, when nothing is on the other end of the
> virtio serial socket.
> 
> 
>    case G_IO_STATUS_AGAIN:
>         /* virtio causes us to spin here when no process is attached to
>          * host-side chardev. sleep a bit to mitigate this
>          */
>         if (s->virtio) {
>             usleep(100*1000);
>         }
>         return true;
> 
>    ....
> 
> 
>     } else if (strcmp(s->method, "virtio-serial") == 0) {
>         /* we spin on EOF for virtio-serial, so back off a bit. also,
>          * dont close the connection in this case, it'll resume normal
>          * operation when another process connects to host chardev
>          */
>         usleep(100*1000);
>         goto out_noclose;
>     }
> 
> 
> I get the feeling that this kind of problem inherant in the use of any
> virtio-serial channel, in the same way you can't detect EOF for a regular
> serial device channel either. Given that virtio-serial is a nice paravirt
> device, is there anything we can do to it, to allow better handling of
> EOF by applications ?

Indeed, and there was a discussion a while back where I think we had tentative
agreement on a path forward for this. Unfortunately there doesn't seem to be
a clear solution for doing it purely in guest-userspace:

http://www.mail-archive.com/qemu-devel@nongnu.org/msg57002.html

The gist of it is basically making the (guest-side) virtio-serial chardev
behave more like a unix socket, i.e. if the host hangs up you get a single EOF
and then your FD becomes invalid, at which point you need to re-open the
chardev to get a valid FD. This could potentially be done with via a new set of
-chardev/-device flags.

> 
> Or perhaps there is some way to make use of epoll() in edge-triggered
> mode to detect it already, because IIUC, edge-triggered mode should only
> fire once for the EOF condition, and then not fire again until something
> in the host actually sends some data ?
> 
> Of course glib's event loop doesn't support edge-triggered events/epoll,
> but perhaps we could just call epoll() directly in the event handler,
> instead of the usleep() call ?

That's definitely worth looking into. Has the 100ms sleep been causing any
issues though? My main concern with the polling behavior was less a matter of
performance than being able to provide a "session" where the start and end
of a stream could be reliably determined, which we don't have currently. But
the guest agent has since been reworked to persist state between host
connects/disconnects so it didn't seem to be a major issue anymore.


> 
> Regards,
> Daniel
> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
> 

-- 
Sincerely,
Mike Roth
IBM Linux Technology Center

Re: [Qemu-devel] qemu guest agent spins in poll/nanosleep(100ms) when nothing is listening on host

[Daniel P. Berrange]

On Thu, Oct 06, 2011 at 04:15:07PM -0500, Michael Roth wrote:
> On Thu, 6 Oct 2011 12:31:05 +0100, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> > I get the feeling that this kind of problem inherant in the use of any
> > virtio-serial channel, in the same way you can't detect EOF for a regular
> > serial device channel either. Given that virtio-serial is a nice paravirt
> > device, is there anything we can do to it, to allow better handling of
> > EOF by applications ?
> 
> Indeed, and there was a discussion a while back where I think we had tentative
> agreement on a path forward for this. Unfortunately there doesn't seem to be
> a clear solution for doing it purely in guest-userspace:
> 
> http://www.mail-archive.com/qemu-devel@nongnu.org/msg57002.html
> 
> The gist of it is basically making the (guest-side) virtio-serial chardev
> behave more like a unix socket, i.e. if the host hangs up you get a single EOF
> and then your FD becomes invalid, at which point you need to re-open the
> chardev to get a valid FD. This could potentially be done with via a new set of
> -chardev/-device flags.

Ah interesting idea.

> > Or perhaps there is some way to make use of epoll() in edge-triggered
> > mode to detect it already, because IIUC, edge-triggered mode should only
> > fire once for the EOF condition, and then not fire again until something
> > in the host actually sends some data ?
> > 
> > Of course glib's event loop doesn't support edge-triggered events/epoll,
> > but perhaps we could just call epoll() directly in the event handler,
> > instead of the usleep() call ?
> 
> That's definitely worth looking into. Has the 100ms sleep been causing any
> issues though? My main concern with the polling behavior was less a matter of
> performance than being able to provide a "session" where the start and end
> of a stream could be reliably determined, which we don't have currently. But
> the guest agent has since been reworked to persist state between host
> connects/disconnects so it didn't seem to be a major issue anymore.

We're intending to have the agent installed in all Fedora 16 guests and
later guests by default, and used by libvirt for shutdown/reboot. so I
was just looking at how it was working to ensure there are no surprises
and happened to notice the wakeups when disconnected on the host. So it
hasn't actually caused any problems for me, I just have a general desire
to ensure any code doesn't do frequent wakeups when there's no work todo.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|