From mboxrd@z Thu Jan 1 00:00:00 1970 From: xuehai zhang Subject: Re: MPI benchmark performance gap between native linux anddomU Date: Mon, 04 Apr 2005 18:43:08 -0500 Message-ID: <4251D10C.4040508@cs.uchicago.edu> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Ian Pratt Cc: Xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Ian, Thanks for the quick response! Explainations to your comments are inline below. >>I did the following experiments to explore the MPI >>application execution performance on both native linux >>machines and inside of unpriviledged Xen user domains. I use >>8 machines with identical HW configurations (498.756 MHz dual >>CPU, 512MB memory, on a 10MB/sec LAN) and I use Pallas MPI >>Benchmarks (PMB). > > >>The expreiment results show, running a same MPI benchmark in >>user domains usually results in a worse (sometimes very bad) >>performance comparing with on native linux machines. The >>following are the results for PMB SendRecv benchmark for both >>experiments (table1 and table2 report throughput and latency >>respectively). As you may notice, SendRecv can achieve a >>14.9MB/sec throughput on native linux machines but can get a >>maximum 7.07 MB/sec throughput if running inside of user >>domains. The latency results also have big gap. > > >>I will appreciate your help if you had the similar experience >>and wanna share your insights. > > > Xen (or any kind of virtualization) is not particularly well suited to > MPI type applications, at least unless you're using Inifiniband or some > other smart NIC that avoids having to use dom0 to do the IO > virtualization. > > However, the results you are seeing are lower than I'd expect. > > Are you running dom0 and the domU on the same CPU or different CPUs. How > does changing this effect the results? I did not specify "cpu" option in Xen's configuration file, so I think both dom0 and domU run on the same CPU (1st CPU). I will try to assign them to different CPUs later. > Also, are you sure the MTU is the same in all cases? The outputs of "ifconfig" show MTU is 1500 in all cases. > Further, please can you repeat the experiements with just a dom0 running > on each node. I will do it and update you later. Thanks again for the help. Xuehai