Network performance regression between kvm-29 and kvm-31

Network performance regression between kvm-29 and kvm-31

[Aurelien Jarno]

Hi,

I have just noticed that there is a huge performance regression in
network transfers between kvm-29 and kvm-31. Using the RTL8139 emulated
card, I have seen the network speed has been divided by 27!!! I haven't
made precise speed measurements with other emulated card, but the 
network speed has been reduced by the same order.

Those tests have been done on a Core 2 CPU.

After some tests I have seen that the problem lies in the user part. I
will do a git bisect tomorrow to find the patch that has caused the
problem, unless somebody already find it.

Cheers,
Aurelien

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Network performance regression between kvm-29 and kvm-31

[Aurelien Jarno]

On Sat, Jul 21, 2007 at 12:25:45AM +0200, Aurelien Jarno wrote:
> Hi,
> 
> I have just noticed that there is a huge performance regression in
> network transfers between kvm-29 and kvm-31. Using the RTL8139 emulated
> card, I have seen the network speed has been divided by 27!!! I haven't
> made precise speed measurements with other emulated card, but the 
> network speed has been reduced by the same order.
> 
> Those tests have been done on a Core 2 CPU.
> 
> After some tests I have seen that the problem lies in the user part. I
> will do a git bisect tomorrow to find the patch that has caused the
> problem, unless somebody already find it.

I finally didn't resist to run git bisect before going to bed...

The patch that causes the regression is:

commit 7d2e674835492040c9baddf0989a95abea9a8d0f
Author: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>
Date:   Sun Jul 15 15:08:23 2007 +0300

    kvm: qemu: synchronous signal handling
    
    in order to ensure that signals are handled with the global lock held, we
    block them, and dequeue them with sigtimedwait().  we can then call the
    handlers directly.
    
    this ensures that the device model and internal qemu data structures are not
    corrupted by cpu 0 thread handling a signal while cpu 1 is servicing a pio
    or mmio request.

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Network performance regression between kvm-29 and kvm-31

[Avi Kivity]

Aurelien Jarno wrote:
> On Sat, Jul 21, 2007 at 12:25:45AM +0200, Aurelien Jarno wrote:
>   
>> Hi,
>>
>> I have just noticed that there is a huge performance regression in
>> network transfers between kvm-29 and kvm-31. Using the RTL8139 emulated
>> card, I have seen the network speed has been divided by 27!!! I haven't
>> made precise speed measurements with other emulated card, but the 
>> network speed has been reduced by the same order.
>>
>> Those tests have been done on a Core 2 CPU.
>>
>> After some tests I have seen that the problem lies in the user part. I
>> will do a git bisect tomorrow to find the patch that has caused the
>> problem, unless somebody already find it.
>>     
>
> I finally didn't resist to run git bisect before going to bed...
>
> The patch that causes the regression is:
>
> commit 7d2e674835492040c9baddf0989a95abea9a8d0f
> Author: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>
> Date:   Sun Jul 15 15:08:23 2007 +0300
>
>     kvm: qemu: synchronous signal handling
>     
>     in order to ensure that signals are handled with the global lock held, we
>     block them, and dequeue them with sigtimedwait().  we can then call the
>     handlers directly.
>     
>     this ensures that the device model and internal qemu data structures are not
>     corrupted by cpu 0 thread handling a signal while cpu 1 is servicing a pio
>     or mmio request.
>
>   

Is this with -net user or -net tap?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Network performance regression between kvm-29 and kvm-31

[Aurelien Jarno]

Avi Kivity a écrit :
> Aurelien Jarno wrote:
>> On Sat, Jul 21, 2007 at 12:25:45AM +0200, Aurelien Jarno wrote:
>>   
>>> Hi,
>>>
>>> I have just noticed that there is a huge performance regression in
>>> network transfers between kvm-29 and kvm-31. Using the RTL8139 emulated
>>> card, I have seen the network speed has been divided by 27!!! I haven't
>>> made precise speed measurements with other emulated card, but the 
>>> network speed has been reduced by the same order.
>>>
>>> Those tests have been done on a Core 2 CPU.
>>>
>>> After some tests I have seen that the problem lies in the user part. I
>>> will do a git bisect tomorrow to find the patch that has caused the
>>> problem, unless somebody already find it.
>>>     
>> I finally didn't resist to run git bisect before going to bed...
>>
>> The patch that causes the regression is:
>>
>> commit 7d2e674835492040c9baddf0989a95abea9a8d0f
>> Author: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>
>> Date:   Sun Jul 15 15:08:23 2007 +0300
>>
>>     kvm: qemu: synchronous signal handling
>>     
>>     in order to ensure that signals are handled with the global lock held, we
>>     block them, and dequeue them with sigtimedwait().  we can then call the
>>     handlers directly.
>>     
>>     this ensures that the device model and internal qemu data structures are not
>>     corrupted by cpu 0 thread handling a signal while cpu 1 is servicing a pio
>>     or mmio request.
>>
>>   
> 
> Is this with -net user or -net tap?
> 

With -net tap.

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Network performance regression between kvm-29 and kvm-31

[Avi Kivity]

Aurelien Jarno wrote:
>>>
>>> I finally didn't resist to run git bisect before going to bed...
>>>
>>> The patch that causes the regression is:
>>>
>>> commit 7d2e674835492040c9baddf0989a95abea9a8d0f
>>> Author: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>
>>> Date:   Sun Jul 15 15:08:23 2007 +0300
>>>
>>>     kvm: qemu: synchronous signal handling
>>>     
>>>     in order to ensure that signals are handled with the global lock held, we
>>>     block them, and dequeue them with sigtimedwait().  we can then call the
>>>     handlers directly.
>>>     
>>>     this ensures that the device model and internal qemu data structures are not
>>>     corrupted by cpu 0 thread handling a signal while cpu 1 is servicing a pio
>>>     or mmio request.
>>>
>>>   
>>>       
>> Is this with -net user or -net tap?
>>
>>     
>
> With -net tap.
>
>   

I pushed a commit which fixes a latency issue with that commit.  However 
with my test (ping -f against the host) I only get a x2 performance 
increase.  Can you test kvm-userspace.git?  And can you detail your 
benchmark?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Network performance regression between kvm-29 and kvm-31

[Aurelien Jarno]

On Sun, Jul 22, 2007 at 01:30:14PM +0300, Avi Kivity wrote:
> Aurelien Jarno wrote:
> >>>
> >>> I finally didn't resist to run git bisect before going to bed...
> >>>
> >>> The patch that causes the regression is:
> >>>
> >>> commit 7d2e674835492040c9baddf0989a95abea9a8d0f
> >>> Author: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>
> >>> Date:   Sun Jul 15 15:08:23 2007 +0300
> >>>
> >>>     kvm: qemu: synchronous signal handling
> >>>     
> >>>     in order to ensure that signals are handled with the global lock held, we
> >>>     block them, and dequeue them with sigtimedwait().  we can then call the
> >>>     handlers directly.
> >>>     
> >>>     this ensures that the device model and internal qemu data structures are not
> >>>     corrupted by cpu 0 thread handling a signal while cpu 1 is servicing a pio
> >>>     or mmio request.
> >>>
> >>>   
> >>>       
> >> Is this with -net user or -net tap?
> >>
> >>     
> >
> > With -net tap.
> >
> >   
> 
> I pushed a commit which fixes a latency issue with that commit.  However 
> with my test (ping -f against the host) I only get a x2 performance 
> increase.  Can you test kvm-userspace.git?  And can you detail your 
> benchmark?

Yes, that fixes the problem. However it seems it has broken SMP
support, I am not able to boot a linux guest with 2 CPU with this 
patch.

About my benchmark, I am just copying a big file using scp between the
guest and the host, and I saw a drop from 7,5 MiB/s to 280 kiB/s.

Aurelien

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/