open/stat64 syscalls run faster on Xen VM than standard Linux

[xuehai zhang <hai@cs.uchicago.edu>]

Dear all,
When I debugged the execution performance of an application using strace, I found there are some 
system calls like open and stat64 which run faster on XenLinux than the standard Linux. The 
following is the output of running "strace -c /bin/sh -c /bin/echo foo" on both systems. An open 
call runs averagely 109 usec on standard Linux but only 41 usecs on XenLinux. An stat64 call runs
75 usecs on standard Linux but only 19 usecs on XenLinux.
The Xen VM runs on the same physical machine as the standard Linux. It uses loopback files in dom0 
as the backends of VBDs.
Any insight is highly appreciated.
Thanks.
Xuehai

XenLinux:
# strace -c /bin/sh -c /bin/echo foo

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  39.05    0.001972        1972         1           write
  19.35    0.000977         977         1           execve
   7.74    0.000391          24        16           old_mmap
   7.23    0.000365          41         9         2 open
   6.06    0.000306          51         6           read
   5.17    0.000261          10        26           brk
   4.93    0.000249          83         3           munmap
   1.98    0.000100          14         7           close
   1.90    0.000096           9        11           rt_sigaction
   1.52    0.000077          11         7           fstat64
   1.01    0.000051           9         6           rt_sigprocmask
   0.95    0.000048          24         2           access
   0.81    0.000041          14         3           uname
   0.75    0.000038          19         2           stat64
   0.38    0.000019          19         1           ioctl
   0.18    0.000009           9         1           time
   0.18    0.000009           9         1           getppid
   0.16    0.000008           8         1           getpgrp
   0.16    0.000008           8         1           getuid32
   0.14    0.000007           7         1           getgid32
   0.12    0.000006           6         1           getpid
   0.12    0.000006           6         1           geteuid32
   0.12    0.000006           6         1           getegid32
------ ----------- ----------- --------- --------- ----------------
100.00    0.005050                   109         2 total


Standard Linux:
# strace -c /bin/sh -c /bin/echo foo

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  22.90    0.000982         109         9         2 open
  22.85    0.000980         980         1           execve
  10.87    0.000466          29        16           old_mmap
  10.45    0.000448         448         1           write
   7.06    0.000303          51         6           read
   6.67    0.000286          10        30           brk
   3.61    0.000155          78         2           access
   3.50    0.000150          75         2           stat64
   2.91    0.000125          42         3           munmap
   2.24    0.000096          14         7           close
   2.12    0.000091          13         7           fstat64
   1.84    0.000079           7        11           rt_sigaction
   1.03    0.000044           7         6           rt_sigprocmask
   0.72    0.000031          10         3           uname
   0.19    0.000008           8         1           geteuid32
   0.16    0.000007           7         1           time
   0.16    0.000007           7         1           getppid
   0.16    0.000007           7         1           getpgrp
   0.16    0.000007           7         1           getuid32
   0.14    0.000006           6         1           getpid
   0.14    0.000006           6         1           getgid32
   0.12    0.000005           5         1           getegid32
------ ----------- ----------- --------- --------- ----------------
100.00    0.004289                   112         2 total