From mboxrd@z Thu Jan 1 00:00:00 1970 From: xuehai zhang Subject: Re: a question about popen() performance on domU Date: Thu, 24 Nov 2005 09:40:34 -0600 Message-ID: <4385DEF2.8030406@cs.uchicago.edu> References: <907625E08839C4409CE5768403633E0B0EAAB8@sefsexmb1.amd.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <907625E08839C4409CE5768403633E0B0EAAB8@sefsexmb1.amd.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Petersson, Mats" Cc: Kate Keahey , Tim Freeman , xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org > See comments below. Thanks Mats. I have more questions about your comments below. Xuehai >>-----Original Message----- >>From: xuehai zhang [mailto:hai@cs.uchicago.edu] >>Sent: 24 November 2005 14:02 >>To: Petersson, Mats >>Cc: xen-devel@lists.xensource.com; Tim Freeman; Kate Keahey >>Subject: Re: [Xen-devel] a question about popen() performance on domU >> >>Mats, >> >>Thanks a lot for the response. >> >> >>>I did have a look at popen, and essentially, it does the >> >>following [ >> >>>the real code is MUCH more complicated, doing lots of >> >>open/dup/close >> >>>on pipes and stuff]: >>>if (!fork()) >>> exec("/bin/sh", "sh", "-c", cmd, NULL); >> >>I took a look at the popen source code too yesterday and the >>above lines are the esstential part. A thread at gnu list >>(http://lists.gnu.org/archive/html/bug-global/2005-06/msg00001 >>.html) suggets >>popen() might depend on how fast /bin/sh is executed. On both >>my VM and the physical machine, the kernel version is 2.6.11, >>glibc version is 2.3.2.ds1-21, and /bin/sh is linked to >>/bin/bash. I also tried to see any difference of the shared >>libraries used by /bin/sh on both machines and found /bin/sh >>on the physical machine uses libraries from /lib/tls while >>for the VM this directory is disabled. >> >>VM$ ldd /bin/sh >> libncurses.so.5 => /lib/libncurses.so.5 (0xb7fa7000) >> libdl.so.2 => /lib/libdl.so.2 (0xb7fa3000) >> libc.so.6 => /lib/libc.so.6 (0xb7e70000) >> /lib/ld-.so.2 => /lib/ld-.so.2 (0xb7fea000) >> >>PHYSICAL$ ldd /bin/sh >> libncurses.so.5 => /lib/libncurses.so.5 (0xb7fa6000) >> libdl.so.2 => /lib/tls/libdl.so.2 (0xb7fa2000) >> libc.so.6 => /lib/tls/libc.so.6 (0xb7e6d000) >> /lib/ld-.so.2 => /lib/ld-.so.2 (0xb7fea000) > > > In this particular case, I would think that lib/tls is not a factor, but > it may be worth disabling the tls libraries on the pysical machine too, > just to make sure... [just "mv /lib/tls /lib/tls.disabled" should do > it]. I don't think /lib/tls is the factor too. I did rerun the tests with tls disabled on the physical machine and it gave even worse performance for the tests. So, I switched it back. > >>>The fork creates another process, which then executes the /bin/sh, >>>which again causes another fork/exec to take place in the effort of >>>executing the actual command given. >>> >>>So the major component of popen would be fork() and >> >>execl(), both of >> >>>which cause, amongst other things, a lot of page-table work and >>>task-switching. >>> >>>Note that popen is implemented in glibc [I took the 2.3.6 >> >>source code >> >>>from www.gnu.org for my look at this], so there's no >> >>difference in the >> >>>implementation of popen itself - the difference lies in how >> >>the Linux >> >>>kernel handles fork() and exec(), but maybe more importantly, how >>>task-switches and page-tables are handled in Linux native >> >>and Xen-Linux. >> >>>Because Xen keeps track of the page-tables on top of >> >>Linux's handling >> >>>of page-tables, you get some extra work here. So, it should >> >>really be >> >>>slower on Xen than on native Linux. >>>[In fact, the question came up not so long ago, why Xen was SLOWER >>>than native Linux on popen (and some others) in a particular >>>benchmark, and the result of that investigation was that >> >>it's down to, >> >>>mainly, task-switching takes longer in Xen.] >> >>I agree with your explanation about Xen was SLOWER than >>native Linux on popen because of the longer task-switching in >>Xen. The problem I met (popen runs faster on Xen VM than the >>physical machine) looks abnormal. I ran several home-made >>benchmarking programming and used the "strace" tool to trace >>the system call performance. The first program is to test the >>performance of both popen and pclose (a loop of popen call >>with a followup pclose call) and the source of the program >>and the strace results are available at >>http://people.cs.uchicago.edu/~hai/tmp/gt2gram/strace-popen/st >>race.txt. The results shows the waitpid syscall costs more >>time on physical machine than on the VM (see the usecs/call >>valuee in the following table). >> >> % time seconds usecs/call calls >>errors syscall >> ------ ----------- ----------- --------- >>--------- ---------------- >>VM: 63.43 0.127900 6395 20 >> waitpid >>PHYSICAL >>MACHINE: 93.87 0.532498 26625 20 >> waitpid >> >>waitpid is called by pclose as shown in the glibc source >>code. So, my original post questioning the performance of >>popen should take pclose into consideration too. A more >>accurate question I should post is, popen+pclose executes >>faster on my VM than my physical machine. The popen/pclose >>benchmark I did narrows the problem down to waitpid that >>waitpid somehow is suffering on the physical machine. >>So, I did a followup experiment to test the fork and waitpid >>performance on both machines. The program is a loop of fork >>call with a followup waitpid call. The source of the program >>and the strace results are available at >>http://people.cs.uchicago.edu/~hai/tmp/gt2gram/strace-fork/str >>ace.txt. The strace results confirm the waitpid costs more >>time on the physical machine (154 usec/call) than the VM (56 >>usec/call). >>However, the program runs faster on the physical machine (not >>like the popen/pclose program) and the results suggest the >>fork syscall used on the VM costs more time than the clone >>syscall on the physical machine. I have a question here, why >>the physical machine doesn't use fork syscall but the clone >>syscall for the same program? > > > Because it's using the same source for glibc! glibc says to use > _IO_fork(), which is calling the fork syscall. Clone would probably do > the same thing, but for whatever good or bad reason, the author(s) of > thise code chose to use fork. There may be good reasons, or no reason at > all to do it this way. I couldn't say. I don't think it makes a whole > lot of difference if the actual command executed by popen is actually > "doing something", rather than just an empty "return". Do you have any suggestion why the same code uses different syscalls on two machines which have the same kernel and glibc? >>>The reason it is not would probably have something to do with the >>>differences in hardware on Linux vs. Xen platforms, perhaps >> >>the fact >> >>>that your file-system is a virtual block-device and thus >> >>lives inside >> >>>a file that is perhaps better cached or otherwise handled in a >>>different way on the Xen-system. >> >>Let me describe the hardware context of my VM and physical >>machine. The host of my VM and the physical machine I tested >>against the VM, are two nodes of a physical cluster with the >>same hardware configuration (Dual Intel PIII 498.799 MHz CPU, >>512MB memory, a 4GB HD with same partitions). The physical >>machine is rebooted with "nosmp". The VM host is rebooted >>into Xen with "nosmp" (Xen version information is "Latest >>ChangeSet: 2005/05/03 17:30:40 1.1846 >>4277a730mvnFSFXrxJpVRNk8hjD4Vg"). Xen dom0 is assigned 96MB >>memory and the VM is the only user domain running on the VM >>host with 395MB memory. Both dom0 and the VM are pinned to CPU 0. >> >>Yes, the backends of the VM's VBDs are loopback files in >>dom0. Three loopback files are used to map to three >>partitions inside of the VM. I acutally thought about the >>possible caching effect of the VM's VBD backends, but not >>sure how to testify it and compare it with the physical >>machine. Is it possible the Xen has different assurance of >>writing back than the physical machine, that is, the data is >>kept in memory longer before is actually written to disk? > > > Xen itself doesn't know ANYTHING about the disk/file where the data for > the Dom0 or DomU comes from, so no, Xen would not do that. However, the > loopback file-system that is involved in VBD's would potentially do > things that are different from the actual hardware. So, there is possbility that the loopback file-system can do something tricky like caching and results in better performance for applications running inside of the VM? > I think you should be able to mount the virtual disk as a "device" on > your system. What does "your system" here refer to? Does it mean dom0 or inside of domU? > I don't know of the top of my head how to do that, but > essentially something like this: > mount myimage.hdd loop/ -t ext3 [additional parameters may be needed]. > > You could then do "chroot loop/", and perform your tests there. This > should execute the same thing from the same place on the native linux as > you would in DomU. > > Now, this may not run faster on native than your original setup, but I > wouldn't be surprised if it does... This is interesting. I will try to run the same tests if I canmount the virtual disk as "device" successfully. Thanks. Xuehai > >>>Now, I'm not saying that there isn't a possibility that >> >>something is >> >>>managed differently in Xen that makes this run faster - I >> >>just don't >> >>>really see how that would be likely, since everything that >> >>happens in >> >>>the system is going to be MORE complicated by the extra >> >>layer of Xen >> >>>involved. >> >>>If anyone else has some thoughts on this subject, it would be >>>interesting to hear. >> >>I agree. But given the VM having same hardware/software >>configuration as the physical machine, it runs faster still >>looks abnormal to me. I wonder if there is any other more >>efficient debugging strategies I can use to investigate it. I >>appreciate if any one has any more suggestions. >> >>Thanks again. >> >>Xuehai >> >> >>>>-----Original Message----- >>>>From: xen-devel-bounces@lists.xensource.com >>>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of xuehai >>>>zhang >>>>Sent: 23 November 2005 20:26 >>>>To: xen-devel@lists.xensource.com >>>>Cc: Tim Freeman; Kate Keahey >>>>Subject: [Xen-devel] a question about popen() performance on domU >>>> >>>>Dear all, >>>>When I compared the performance of some application on both >> >>a Xen domU >> >>>>and a standard linux machine (where domU runs on a similar physical >>>>mahine), I notice the application runs faster on the domU >> >>than on the >> >>>>physical machine. >>>>Instrumenting the application code shows the application >> >>spends more >> >>>>time on popen() calls on domU than on the physical machine. >> >>I wonder >> >>>>if xenlinux does some special modification of the popen code to >>>>improve its performance than the original Linux popen code? >>>>Thanks in advance for your help. >>>>Xuehai >>>> >>>>_______________________________________________ >>>>Xen-devel mailing list >>>>Xen-devel@lists.xensource.com >>>>http://lists.xensource.com/xen-devel >>>> >>>> >>> >>> >>> >> >> > >