Re: MPI benchmark performance gap between native linux anddomU

[xuehai zhang <hai@cs.uchicago.edu>]

Ian,

Thanks for the quick response! Explainations to your comments are inline below.

>>I did the following experiments to explore the MPI 
>>application execution performance on both native linux 
>>machines and inside of unpriviledged Xen user domains. I use 
>>8 machines with identical HW configurations (498.756 MHz dual 
>>CPU, 512MB memory, on a 10MB/sec LAN) and I use Pallas MPI 
>>Benchmarks (PMB).
> 
> 
>>The expreiment results show, running a same MPI benchmark in 
>>user domains usually results in a worse (sometimes very bad) 
>>performance comparing with on native linux machines. The 
>>following are the results for PMB SendRecv benchmark for both 
>>experiments (table1 and table2 report throughput and latency 
>>respectively). As you may notice, SendRecv can achieve a 
>>14.9MB/sec throughput on native linux machines but can get a 
>>maximum 7.07 MB/sec throughput if running inside of user 
>>domains. The latency results also have big gap.
> 
> 
>>I will appreciate your help if you had the similar experience 
>>and wanna share your insights.
> 
> 
> Xen (or any kind of virtualization) is not particularly well suited to
> MPI type applications, at least unless you're using Inifiniband or some
> other smart NIC that avoids having to use dom0 to do the IO
> virtualization.
> 
> However, the results you are seeing are lower than I'd expect.
> 
> Are you running dom0 and the domU on the same CPU or different CPUs. How
> does changing this effect the results?

I did not specify "cpu" option in Xen's configuration file, so I think both dom0 and domU run on the 
  same CPU (1st CPU). I will try to assign them to different CPUs later.

> Also, are you sure the MTU is the same in all cases?

The outputs of "ifconfig" show MTU is 1500 in all cases.

> Further, please can you repeat the experiements with just a dom0 running
> on each node.

I will do it and update you later.

Thanks again for the help.

Xuehai