All of lore.kernel.org
 help / color / mirror / Atom feed
* etherbridge bottleneck
@ 2004-11-10 21:48 David Becker
  2004-11-11 12:00 ` Bin Ren
  0 siblings, 1 reply; 6+ messages in thread
From: David Becker @ 2004-11-10 21:48 UTC (permalink / raw)
  To: xen-devel


I ran some iperf tests today and it looks like the etherbridge
is the limiting factor on throughput.  In the beforetime, I saw great
throughput to the VMs; over 800 Mbps.   With the bridge, the numbers
are in the 400s somewhere.
Is this the speed I can expect from the bridge?
Is there some tuning I should try, or another way to get more bandwidth
into the VMs?

This is with xen-2.0, 2.4.27-xen0 and 2.4.27-xenU.  


My iperf numbers:
 940 Mbps  stock linux -> stock linux
 470 Mbps  stock linux -> xenU
 533 Mbps  xenU -> stock linux 

ether bridge speed
 533 Mbps  xenU -> xen0  on the same host
 422 Mbps  xen0 -> xenU  on the same host

loopback speed
 4.4 Gbps  stock linux 
 3.2 Gbps  xenU
 3.2 Gbps  xen0

(stock linux is 2.4.25)



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

^ permalink raw reply	[flat|nested] 6+ messages in thread
* RE: etherbridge bottleneck
@ 2004-11-12  8:42 Ian Pratt
  0 siblings, 0 replies; 6+ messages in thread
From: Ian Pratt @ 2004-11-12  8:42 UTC (permalink / raw)
  To: David Becker, xen-devel

David,

You might want to try giving one of your domains a second VIF and then
manually assign an IP address to the vifX.Y interface in domain 0 rather
than using the bridge.  

I doubt the bridge is a serious overhead in xenU<->xenO communication,
but it would be useful to eliminate it from your results.

For high-performance xenU<->xenU communication it probably makes sense
to setup a direct netfront/back connection (to avoid going via the dom0
bridge). Although the netfront/back drivers are quite happy working
point-to-point, I doubt xend will let you set this up right now.
Shouldn't be too hard to fix, though...

Ian

> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net 
> [mailto:xen-devel-admin@lists.sourceforge.net] On Behalf Of 
> David Becker
> Sent: 11 November 2004 14:57
> To: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] etherbridge bottleneck
> 
> 
> " (1) Is Xen running on a UP, SMP or HT?
> 
> all of the above
> 
> " (2) If Xen is running on a SMP or HT, are xen0 and xenU 
> running " different processors or threads?
> 
> xm list doesn't break out HT contexts, so who's to say..
> 
> " (3) Which scheduler in Xen are you using? Have you changed 
> any parameters?
> 
> factory defaults.
> 
> 
> Here is a full account of configs and iperf speeds between 
> domains on an otherwise-idle host via the etherbridge, plus 
> iperf speed over the loopback interface for each domain.
> 
> Uni-processor (PIII)
>         426 Mbps xenU UNI -> xen0 UNI
>         267 Mbps xen0 UNI -> xenU UNI
>         3.34 Gbps xen0 loopback
>         3.14 Gbps xenU loopback
> 
>         xm list
>          Name              Id  Mem(MB)  CPU  State  Time(s)  Console
>          Domain-0           0       34    0  r----    297.0        
>          grant              2      499    0  -b---    161.2    9602
>         xm info
>          system                 : Linux
>          host                   : rack099-xen
>          release                : 2.4.27-xen0
>          version                : #3 Wed Nov 10 11:29:37 EST 2004
>          machine                : i686
>          cores                  : 1
>          hyperthreads_per_core  : 1
>          cpu_mhz                : 1396
>          memory                 : 2047
>          free_memory            : 1485
> 
> slow Uni HT processor (P4 2.0GHz)
>         498 Mbps xenU HT1 -> xen0 HT0
>         369 Mbps xen0 HT0 -> xenU HT1
>         2.49 Gbps xen0 loopback
>         2.54 Gbps xenU loopback
> 
>         xm list
>          Name              Id  Mem(MB)  CPU  State  Time(s)  Console
>          Domain-0           0       34    0  r----    561.7        
>          globus             1      799    1  -b---    225.5    9601
>         xm info
>          system                 : Linux
>          host                   : rack160-xen
>          release                : 2.4.27-xen0
>          version                : #2 Thu Nov 4 15:57:19 EST 2004
>          machine                : i686
>          cores                  : 1
>          hyperthreads_per_core  : 2
>          cpu_mhz                : 1993
>          memory                 : 1023
>          free_memory            : 167
> 
> faster Dual HT processors (P4 2.6GHz)
>         645 Mbps xenU ctxt-1 -> xen0 ctxt-0
>         470 Mbps xen0 ctxt-0 -> xenU ctxt-1
>         782 Mbps xenU ctxt-2 -> xen0 ctxt-0
>         563 Mbps xen0 ctxt-0 -> xenU ctxt-2
>         747 Mbps xenU ctxt-1 -> xenU ctxt-2
>         691 Mbps xenU ctxt-2 -> xenU ctxt-1
>         3.34 Gbps xen0 ctxt-0 loopback
>         3.50 Gbps xenU ctxt-1 loopback
>         3.41 Gbps xenU ctxt-2 loopback
> 
>         xm list
>          Name              Id  Mem(MB)  CPU  State  Time(s)  Console
>          Domain-0           0       34    0  r----   1032.7        
>          batch020           3     1999    1  -b---  108571.9    9603
>          batch040           2     1799    2  r----  154375.3    9602
>         xm info
>          system                 : Linux
>          host                   : rack276-xen
>          release                : 2.4.27-xen0
>          version                : #2 Thu Nov 4 15:57:19 EST 2004
>          machine                : i686
>          cores                  : 2
>          hyperthreads_per_core  : 2
>          cpu_mhz                : 2599
>          memory                 : 3967
>          free_memory            : 94
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE 
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
> 


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click

^ permalink raw reply	[flat|nested] 6+ messages in thread
* RE: etherbridge bottleneck
@ 2004-11-14 19:53 Ian Pratt
  2004-11-15 16:01 ` David Becker
  0 siblings, 1 reply; 6+ messages in thread
From: Ian Pratt @ 2004-11-14 19:53 UTC (permalink / raw)
  To: David Becker, xen-devel

 > I ran some iperf tests today and it looks like the 
> etherbridge is the limiting factor on throughput.  In the 
> beforetime, I saw great
> throughput to the VMs; over 800 Mbps.   With the bridge, the numbers
> are in the 400s somewhere.
> Is this the speed I can expect from the bridge?

We've had no particular problem with the bridge code causing a
performance bottleneck -- although there appears to be a lot of code in
the forwarding path, it mostly doesn't get executed. 

However, since its easy not to have the bridge involved in your tests it
should be fairly easy to eliminate -- just don't connect the vif to the
bridge.

xenU -> xenU performance isn't something we've particularly optimised,
and going via the bridge is going to be far from ideal. It would be much
better to set up a direct netfront->netback path between the two
domains.  Xend probably isn't geared up to do this right now, but it
should be possible to add. It'll probably be quite easy to add to x2d2
(new in the unstable tree) which is what we use for prototyping.

> My iperf numbers:
>  940 Mbps  stock linux -> stock linux
>  470 Mbps  stock linux -> xenU
>  533 Mbps  xenU -> stock linux 

We have no problem getting 900Mb/s in/out of xenU's and the wire on our
2.4GHz Xeon systems (regardless of whether using uniprocessor, SMP or
hyperthreading).

What do you get between the wire and domain 0? You should be getting
940Mb/s as per your stock-stock comparison. If not, sounds like there
may be an interrupt problem with Xen on your system.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2004-11-15 16:01 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-11-10 21:48 etherbridge bottleneck David Becker
2004-11-11 12:00 ` Bin Ren
2004-11-11 14:56   ` David Becker
  -- strict thread matches above, loose matches on Subject: below --
2004-11-12  8:42 Ian Pratt
2004-11-14 19:53 Ian Pratt
2004-11-15 16:01 ` David Becker

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.