[Qemu-devel] QEMU websockets support is laggy?

[Qemu-devel] QEMU websockets support is laggy?

[Brian Rak]

We've been considering switching over to using qemu's built in 
websockets support (to avoid the overhead of needing websockify 
running).  We've been seeing very poor performance after the switch (it 
takes the console 4-5 seconds to update after pressing a key).  So far, 
I haven't been able to find any indication of why this is happening.  
The exact same configuration works perfectly when running with 
websockify, but laggy when hitting qemu directly.

I've tried a few things (disabling encryption, bypassing our usual nginx 
proxy, even connecting via a ssh tunnel), and haven't made any sort of 
progress here.  Has anyone else seen this?  Any suggestions as to where 
I should start looking?

Re: [Qemu-devel] QEMU websockets support is laggy?

[Dr. David Alan Gilbert]

* Brian Rak (brak@gameservers.com) wrote:
> We've been considering switching over to using qemu's built in websockets
> support (to avoid the overhead of needing websockify running).  We've been
> seeing very poor performance after the switch (it takes the console 4-5
> seconds to update after pressing a key).  So far, I haven't been able to
> find any indication of why this is happening.  The exact same configuration
> works perfectly when running with websockify, but laggy when hitting qemu
> directly.
> 
> I've tried a few things (disabling encryption, bypassing our usual nginx
> proxy, even connecting via a ssh tunnel), and haven't made any sort of
> progress here.  Has anyone else seen this?  Any suggestions as to where I
> should start looking?


copying in Dan Berrange who knows the websocket code.

Dave

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU websockets support is laggy?

[Daniel P. Berrange]

On Tue, Jan 24, 2017 at 05:02:25PM -0500, Brian Rak wrote:
> We've been considering switching over to using qemu's built in websockets
> support (to avoid the overhead of needing websockify running).  We've been
> seeing very poor performance after the switch (it takes the console 4-5
> seconds to update after pressing a key).  So far, I haven't been able to
> find any indication of why this is happening.  The exact same configuration
> works perfectly when running with websockify, but laggy when hitting qemu
> directly.
> 
> I've tried a few things (disabling encryption, bypassing our usual nginx
> proxy, even connecting via a ssh tunnel), and haven't made any sort of
> progress here.  Has anyone else seen this?  Any suggestions as to where I
> should start looking?

Can you clarify the exact setup you have ?  As mentioned on IRC, I don't
see any degradation in performance betweeen builtin websockets vs a
websockets proxy - if anything the builtin websockets is marginally less
laggy.  I was connecting over TCP localhost, however, so would not see
any effects of network latency. My test was QEMU git master, with noVNC
git master.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] QEMU websockets support is laggy?

[Daniel P. Berrange]

On Fri, Jan 27, 2017 at 09:35:38AM +0000, Daniel P. Berrange wrote:
> On Tue, Jan 24, 2017 at 05:02:25PM -0500, Brian Rak wrote:
> > We've been considering switching over to using qemu's built in websockets
> > support (to avoid the overhead of needing websockify running).  We've been
> > seeing very poor performance after the switch (it takes the console 4-5
> > seconds to update after pressing a key).  So far, I haven't been able to
> > find any indication of why this is happening.  The exact same configuration
> > works perfectly when running with websockify, but laggy when hitting qemu
> > directly.
> > 
> > I've tried a few things (disabling encryption, bypassing our usual nginx
> > proxy, even connecting via a ssh tunnel), and haven't made any sort of
> > progress here.  Has anyone else seen this?  Any suggestions as to where I
> > should start looking?
> 
> Can you clarify the exact setup you have ?  As mentioned on IRC, I don't
> see any degradation in performance betweeen builtin websockets vs a
> websockets proxy - if anything the builtin websockets is marginally less
> laggy.  I was connecting over TCP localhost, however, so would not see
> any effects of network latency. My test was QEMU git master, with noVNC
> git master.

It turns out that to see the problem you have a wait a while - the
connection is initially fine, but gets worse over time. Using a TCP
connection seems to make it get worse more quickly.

After a painful debugging session I've discovered the problem is that
QEMU is reading data off the socket, decoding a websockets packet and
then processing it. The problem is if the read off the socket gets
multiple websockets packets at once, it only decodes the first packet.
The remaining packets aren't decoded until more data arrives on the
socket. This gets progressively worse & worse. I'll send a patch for
this next week...


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] QEMU websockets support is laggy?

[Daniel P. Berrange]

On Fri, Jan 27, 2017 at 06:08:20PM +0000, Daniel P. Berrange wrote:
> On Fri, Jan 27, 2017 at 09:35:38AM +0000, Daniel P. Berrange wrote:
> > On Tue, Jan 24, 2017 at 05:02:25PM -0500, Brian Rak wrote:
> > > We've been considering switching over to using qemu's built in websockets
> > > support (to avoid the overhead of needing websockify running).  We've been
> > > seeing very poor performance after the switch (it takes the console 4-5
> > > seconds to update after pressing a key).  So far, I haven't been able to
> > > find any indication of why this is happening.  The exact same configuration
> > > works perfectly when running with websockify, but laggy when hitting qemu
> > > directly.
> > > 
> > > I've tried a few things (disabling encryption, bypassing our usual nginx
> > > proxy, even connecting via a ssh tunnel), and haven't made any sort of
> > > progress here.  Has anyone else seen this?  Any suggestions as to where I
> > > should start looking?
> > 
> > Can you clarify the exact setup you have ?  As mentioned on IRC, I don't
> > see any degradation in performance betweeen builtin websockets vs a
> > websockets proxy - if anything the builtin websockets is marginally less
> > laggy.  I was connecting over TCP localhost, however, so would not see
> > any effects of network latency. My test was QEMU git master, with noVNC
> > git master.
> 
> It turns out that to see the problem you have a wait a while - the
> connection is initially fine, but gets worse over time. Using a TCP
> connection seems to make it get worse more quickly.
> 
> After a painful debugging session I've discovered the problem is that
> QEMU is reading data off the socket, decoding a websockets packet and
> then processing it. The problem is if the read off the socket gets
> multiple websockets packets at once, it only decodes the first packet.
> The remaining packets aren't decoded until more data arrives on the
> socket. This gets progressively worse & worse. I'll send a patch for
> this next week...

I've just sent a patch & would appreciate your testing to confirm that
it fixes the problem for you too.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] QEMU websockets support is laggy?

[Brian Rak]

On 1/30/2017 5:51 AM, Daniel P. Berrange wrote:
> On Fri, Jan 27, 2017 at 06:08:20PM +0000, Daniel P. Berrange wrote:
>> On Fri, Jan 27, 2017 at 09:35:38AM +0000, Daniel P. Berrange wrote:
>>> On Tue, Jan 24, 2017 at 05:02:25PM -0500, Brian Rak wrote:
>>>> We've been considering switching over to using qemu's built in websockets
>>>> support (to avoid the overhead of needing websockify running).  We've been
>>>> seeing very poor performance after the switch (it takes the console 4-5
>>>> seconds to update after pressing a key).  So far, I haven't been able to
>>>> find any indication of why this is happening.  The exact same configuration
>>>> works perfectly when running with websockify, but laggy when hitting qemu
>>>> directly.
>>>>
>>>> I've tried a few things (disabling encryption, bypassing our usual nginx
>>>> proxy, even connecting via a ssh tunnel), and haven't made any sort of
>>>> progress here.  Has anyone else seen this?  Any suggestions as to where I
>>>> should start looking?
>>> Can you clarify the exact setup you have ?  As mentioned on IRC, I don't
>>> see any degradation in performance betweeen builtin websockets vs a
>>> websockets proxy - if anything the builtin websockets is marginally less
>>> laggy.  I was connecting over TCP localhost, however, so would not see
>>> any effects of network latency. My test was QEMU git master, with noVNC
>>> git master.
>> It turns out that to see the problem you have a wait a while - the
>> connection is initially fine, but gets worse over time. Using a TCP
>> connection seems to make it get worse more quickly.
>>
>> After a painful debugging session I've discovered the problem is that
>> QEMU is reading data off the socket, decoding a websockets packet and
>> then processing it. The problem is if the read off the socket gets
>> multiple websockets packets at once, it only decodes the first packet.
>> The remaining packets aren't decoded until more data arrives on the
>> socket. This gets progressively worse & worse. I'll send a patch for
>> this next week...
> I've just sent a patch & would appreciate your testing to confirm that
> it fixes the problem for you too.
>
> Regards,
> Daniel
Confirmed, with the patch the built in websockets support performs just 
as well as vnc + websockify.  Thanks for the help!