RE: x86-64 Net Performance

RE: x86-64 Net Performance

[Ian Pratt]

 
> Update: I went ahead and compiled glibc without TLS / NPTL.  

For 64 bit Xen you don't need to worry about TLS/NPTL.

> Also, I realized that I wasn't allocating all my physical 
> memory to Domain0 for these tests, so I was giving Xen a 
> disadvantage.  It didn't change much, but those tests have 
> been re-done.
> 
> Here's the odd thing: disabling TLS for 64-bit improved 
> native performance by around 10%, but had no effect on Xen.

How did you disable TLS on 64 bit? It's odd that it would have any
effect on native or xen, let alone such a dramatic one. Are you sure
about this test?

>                 Native  Domain0 Penalty
> 64-bit TLS      50.2    65.0    29.5%
> 64-bit no TLS   44.8    65.0    45.1%
> 32-bit TLS      59.6    59.5    -0.2%
> 32-bit no TLS   59.6    60.2     1.0%

It looks like we need to investigate x86_64 networking performance. We
may well not be getting the pipelining that we do in 32bit, though I
can't immediately think why,

Ian 

Re: x86-64 Net Performance

[Andrew Theurer]

Ian Pratt wrote:

> 
>  
>
>>Update: I went ahead and compiled glibc without TLS / NPTL.  
>>    
>>
>
>For 64 bit Xen you don't need to worry about TLS/NPTL.
>
>  
>
>>Also, I realized that I wasn't allocating all my physical 
>>memory to Domain0 for these tests, so I was giving Xen a 
>>disadvantage.  It didn't change much, but those tests have 
>>been re-done.
>>
>>Here's the odd thing: disabling TLS for 64-bit improved 
>>native performance by around 10%, but had no effect on Xen.
>>    
>>
>
>How did you disable TLS on 64 bit? It's odd that it would have any
>effect on native or xen, let alone such a dramatic one. Are you sure
>about this test?
>
>  
>
>>                Native  Domain0 Penalty
>>64-bit TLS      50.2    65.0    29.5%
>>64-bit no TLS   44.8    65.0    45.1%
>>32-bit TLS      59.6    59.5    -0.2%
>>32-bit no TLS   59.6    60.2     1.0%
>>    
>>
>
>It looks like we need to investigate x86_64 networking performance. We
>may well not be getting the pipelining that we do in 32bit, though I
>can't immediately think why,
>  
>
This is a bit surprising as most newer cpus (really any amd_64) has 
plenty of processing power, and the Ethernet adapter is almost always 
the bottleneck.  This was with netperf, but I would not expect a http 
get of a single file to be much worse..  I have also seen more overhead 
in ia32 than amd_64.  Xenoprofile showed (for me) about a 13% cpu 
overhead (more than baremetal linux) in amd_64, 3% in xen and about 10% 
for the sw bridge.  On ia32 (intel xeon) this was about 23%, 13% in xen 
and 10% for the bridge.  I guess I should run these over again and 
check; maybe something have dramatically changed things.

You might want to try this test without starting xend, so the network 
bridge is not created and eth0 is put on it.  Perhaps that is an issue.  
Come to think of it, my tests were not with veth0/vif0.0 -that may 
indeed be an issue, going through the front/back-end virtual interfaces, 
then on the bridge, then on eth0.  I'll kick off some tests tomorrow and 
see what I get.

-Andrew

Re: x86-64 Net Performance

[Alexander Charbonnet]

> You might want to try this test without starting xend, so the network
> bridge is not created and eth0 is put on it.  Perhaps that is an issue.
> Come to think of it, my tests were not with veth0/vif0.0 -that may
> indeed be an issue, going through the front/back-end virtual interfaces,
> then on the bridge, then on eth0.  I'll kick off some tests tomorrow and
> see what I get.

xend was never started.  All I did was boot with the Domain0 kernel.  No 
bridges; I'm talking straight to eth0.

That answers Nicholas's question, too; I haven't done anything besides Domain0 
talking to itself.

Alex

Re: x86-64 Net Performance

[Alexander Charbonnet]

I did `apt-get source glibc`, edited debian/sysdeps/amd64.mk to comment out 
the couple of lines enabling NPTL, then used dpkg-buildpackage to generate 
new .debs.

Your skepticism was warranted; I double-checked my changes to that file, and I 
had also disabled a line that set -O3 for the compile.  This seems to account 
for the performance difference on native; recompiling with -O3 intact slows 
things down to normal.  So the conclusion is that NPTL / TLS doesn't make 
much difference.

However, it's kind of odd that disabling -O3 would make things faster on 
native, but have no effect on Xen.

Alex


On Sunday 09 October 2005 04:18 am, Ian Pratt wrote:
> > Update: I went ahead and compiled glibc without TLS / NPTL.
>
> For 64 bit Xen you don't need to worry about TLS/NPTL.
>
> > Also, I realized that I wasn't allocating all my physical
> > memory to Domain0 for these tests, so I was giving Xen a
> > disadvantage.  It didn't change much, but those tests have
> > been re-done.
> >
> > Here's the odd thing: disabling TLS for 64-bit improved
> > native performance by around 10%, but had no effect on Xen.
>
> How did you disable TLS on 64 bit? It's odd that it would have any
> effect on native or xen, let alone such a dramatic one. Are you sure
> about this test?
>
> >                 Native  Domain0 Penalty
> > 64-bit TLS      50.2    65.0    29.5%
> > 64-bit no TLS   44.8    65.0    45.1%
> > 32-bit TLS      59.6    59.5    -0.2%
> > 32-bit no TLS   59.6    60.2     1.0%
>
> It looks like we need to investigate x86_64 networking performance. We
> may well not be getting the pipelining that we do in 32bit, though I
> can't immediately think why,
>
> Ian

RE: Opteron server and NUMA

[Ian Pratt]

 > I used ab to benchmark grabbing a 100MB file 100 times from 
> localhost.  
> Natively it took 15.9 seconds.  Under Domain0 it took 41% 
> longer, 22.4 seconds.  That sure looks like it makes up a 
> huge chunk of the overall slowdown I'm noticing.
> 
> Any other tricks to try to get network performance closer to native?

Can you try running that test on the same machine running a 32bit
version of Xen. It's possible something weird is happening just on 64
bit.
Thanks,
Ian

x86-64 Net Performance [was: Opteron server and NUMA]

[Alexander Charbonnet]

> Can you try running that test on the same machine running a 32bit
> version of Xen. It's possible something weird is happening just on 64
> bit.

file.zip is approximately 100MB.  Ran `ab -n 300 -c 1 localhost/file.zip`.  
Disk I/O should not be a factor; I had Apache slurp the file into RAM before 
each test.  Table shows time in seconds and the percent penalty for Xen.

                Native  Domain0 Penalty
64-bit TLS      50.2    65.0    29.5%
32-bit TLS      59.6    64.8     8.7%
32-bit no TLS   59.6    62.6     5.0%

On 32-bit, disabling TLS is as easy as moving /lib/tls.  For 64-bit, it's 
compiled into Debian's glibc package.  I can recompile and run that test if 
you think it's worth doing.

Alex

Re: x86-64 Net Performance

[Xan Charbonnet]

Update: I went ahead and compiled glibc without TLS / NPTL.  Also, I realized 
that I wasn't allocating all my physical memory to Domain0 for these tests, 
so I was giving Xen a disadvantage.  It didn't change much, but those tests 
have been re-done.

Here's the odd thing: disabling TLS for 64-bit improved native performance by 
around 10%, but had no effect on Xen.

                Native  Domain0 Penalty
64-bit TLS      50.2    65.0    29.5%
64-bit no TLS   44.8    65.0    45.1%
32-bit TLS      59.6    59.5    -0.2%
32-bit no TLS   59.6    60.2     1.0%


Alex