system call time increase when turning on CONFIG_PARAVIRT

system call time increase when turning on CONFIG_PARAVIRT

[Tim Chen]

With CONFIG_PARAVIRT turned on, I've found that time invoking
system_call jumped up quite a lot.  Using TCP streaming test as a
workload and running on 32-bit 2.6.20 kernel, system_call goes up from
0.00025% all the way to 1.6% in the oprofile data.  There is a drop of
about 4% in overall throughput for this particular workload. 

With lmbench's null system call test, the call time goes up from 0.10
usec to 0.225 usec.

I'm testing on dual socket Intel core 2 processor running at 2.67 GHz
with 4 GB RAM.

Tim 

Re: system call time increase when turning on CONFIG_PARAVIRT

[Jeremy Fitzhardinge]

Tim Chen wrote:
> With CONFIG_PARAVIRT turned on, I've found that time invoking
> system_call jumped up quite a lot.  Using TCP streaming test as a
> workload and running on 32-bit 2.6.20 kernel, system_call goes up from
> 0.00025% all the way to 1.6% in the oprofile data.  There is a drop of
> about 4% in overall throughput for this particular workload. 
>
> With lmbench's null system call test, the call time goes up from 0.10
> usec to 0.225 usec.
>
> I'm testing on dual socket Intel core 2 processor running at 2.67 GHz
> with 4 GB RAM.
[ I assume you're talking about running on native hardware. ]

In the current paravirt changes in the kernel, many of the
paravirtualized operations are implemented as (expensive) indirect calls
via paravirt_ops.  Among the changes in the paravirt patches I posted
yesterday is an enhanced patching mechanism which inlines a lot of the
common operations, and converts the rest into direct calls.

I haven't done any detailed measurements on what effect this will have,
but it does bring the actual executed instruction stream much closer to
the !CONFIG_PARAVIRT case, and so I would hope it would recover most or
all of the performance loss you've noticed.

    J

Re: system call time increase when turning on CONFIG_PARAVIRT

[Tim Chen]

On Fri, 2007-03-02 at 13:54 -0800, Jeremy Fitzhardinge wrote:

> [ I assume you're talking about running on native hardware. ]
> 
That's correct.

> I haven't done any detailed measurements on what effect this will have,
> but it does bring the actual executed instruction stream much closer to
> the !CONFIG_PARAVIRT case, and so I would hope it would recover most or
> all of the performance loss you've noticed.
> 
I also hope that the performance can be recovered as this option could
enabled in distributions' kernels in future.

Tim

Re: system call time increase when turning on CONFIG_PARAVIRT

[Jeremy Fitzhardinge]

Tim Chen wrote:
> I also hope that the performance can be recovered as this option could
> enabled in distributions' kernels in future.

Yes, the intent is that running a CONFIG_PARAVIRT kernel on native
hardware will have negligible performance hit compared to running a
non-paravirt kernel.

    J

Re: system call time increase when turning on CONFIG_PARAVIRT

[Zachary Amsden]

Jeremy Fitzhardinge wrote:
> Tim Chen wrote:
>   
>> I also hope that the performance can be recovered as this option could
>> enabled in distributions' kernels in future.
>>     
>
> Yes, the intent is that running a CONFIG_PARAVIRT kernel on native
> hardware will have negligible performance hit compared to running a
> non-paravirt kernel.
>   

We can validate that claim entirely.  The way we are proceeding, the 
native code will be inlined or direct called as much as possible.  With 
the VMI-Linux code we had earlier, this mostly created <3% overhead for 
microbenchmarks (and in some cases, we actually won over the unmodified 
native code).  For macro-benchmarks, with real-world workloads, this 
reduced to immeasurable noise, never off by more than +/- 0.5% IIRC.

I believe all of this is totally achievable.  We have the technology.  
We can rebuild it.

Zach

Re: system call time increase when turning on CONFIG_PARAVIRT

[Tim Chen]

On Fri, 2007-03-02 at 16:16 -0800, Jeremy Fitzhardinge wrote:

> 
> Yes, the intent is that running a CONFIG_PARAVIRT kernel on native
> hardware will have negligible performance hit compared to running a
> non-paravirt kernel.
> 
>     J

It turned out that VDSO was turned off by CONFIG_PARAVIRT option,
causing the system call to use inefficient int 0x80 which led to the
increase system_call time I was seeing.  I noted that Ingo has caught
this problem and proposed a patch to correct this issue in another mail
thread.

Tim

Re: system call time increase when turning on CONFIG_PARAVIRT

[Jeremy Fitzhardinge]

Tim Chen wrote:
> It turned out that VDSO was turned off by CONFIG_PARAVIRT option,
> causing the system call to use inefficient int 0x80 which led to the
> increase system_call time I was seeing.  I noted that Ingo has caught
> this problem and proposed a patch to correct this issue in another mail
> thread.

Thanks for identifying this.  We'll be posting a more general fix for
COMPAT_VDSO soon which will address this.

    J