Forking time in Xen

Forking time in Xen

[Sergey Zhukov]

Hi,

I repost this message from xen-users list following by others
subscribers suggestions:


I found an article about forking time for redis NoSQL database in
different systems:

http://redis.io/topics/latency
---------Quote------------------
Fork time in different systems
Modern hardware is pretty fast to copy the page table, but Xen is not.
The problem with Xen is not virtualization-specific, but Xen-specific.
For instance using VMware or Virutal Box does not result into slow fork
time. The following is a table that compares fork time for different
Redis instance size. Data is obtained performing a BGSAVE and looking at
the latest_fork_usec filed in the INFO command output.

      * Linux beefy VM on VMware 6.0GB RSS forked in 77 milliseconds
        (12.8 milliseconds per GB).
      * Linux running on physical machine (Unknown HW) 6.1GB RSS forked
        in 80 milliseconds (13.1 milliseconds per GB)
      * Linux running on physical machine (Xeon @ 2.27Ghz) 6.9GB RSS
        forked into 62 millisecodns (9 milliseconds per GB).
      * Linux VM on 6sync (KVM) 360 MB RSS forked in 8.2 milliseconds
        (23.3 millisecond per GB).
      * Linux VM on EC2 (Xen) 6.1GB RSS forked in 1460 milliseconds
        (239.3 milliseconds per GB).
      * Linux VM on Linode (Xen) 0.9GBRSS forked into 382 millisecodns
        (424 milliseconds per GB).

As you can see a VM running on Xen has a performance hit that is between
one order to two orders of magnitude. We believe this is a severe
problem with Xen and we hope it will be addressed ASAP.
----------End of quote-----------------

I made my own test with Xen 4.1 and Redis 2.4 with 7.04GB dataset. The
test was performed on Intel Core I5 2500 processor unit. Forking time
was about 1 sec or 151 ms/GB - it's faster then tests over Amazon
EC2/Linode were mentioned in the article, but still much slower then
VMWare or physical machines. Has anyone running with this issue? Or may
be there is a way to tune Xen for less forking times?

Sergey Zhukov

Re: Forking time in Xen

[George Dunlap]

On Mon, Jun 18, 2012 at 10:38 AM, Sergey Zhukov <svg@ngs.ru> wrote:
> Hi,
>
> I repost this message from xen-users list following by others
> subscribers suggestions:
>
>
> I found an article about forking time for redis NoSQL database in
> different systems:
>
> http://redis.io/topics/latency
> ---------Quote------------------
> Fork time in different systems
> Modern hardware is pretty fast to copy the page table, but Xen is not.
> The problem with Xen is not virtualization-specific, but Xen-specific.
> For instance using VMware or Virutal Box does not result into slow fork
> time. The following is a table that compares fork time for different
> Redis instance size. Data is obtained performing a BGSAVE and looking at
> the latest_fork_usec filed in the INFO command output.
>
>      * Linux beefy VM on VMware 6.0GB RSS forked in 77 milliseconds
>        (12.8 milliseconds per GB).
>      * Linux running on physical machine (Unknown HW) 6.1GB RSS forked
>        in 80 milliseconds (13.1 milliseconds per GB)
>      * Linux running on physical machine (Xeon @ 2.27Ghz) 6.9GB RSS
>        forked into 62 millisecodns (9 milliseconds per GB).
>      * Linux VM on 6sync (KVM) 360 MB RSS forked in 8.2 milliseconds
>        (23.3 millisecond per GB).
>      * Linux VM on EC2 (Xen) 6.1GB RSS forked in 1460 milliseconds
>        (239.3 milliseconds per GB).
>      * Linux VM on Linode (Xen) 0.9GBRSS forked into 382 millisecodns
>        (424 milliseconds per GB).
>
> As you can see a VM running on Xen has a performance hit that is between
> one order to two orders of magnitude. We believe this is a severe
> problem with Xen and we hope it will be addressed ASAP.
> ----------End of quote-----------------
>
> I made my own test with Xen 4.1 and Redis 2.4 with 7.04GB dataset. The
> test was performed on Intel Core I5 2500 processor unit. Forking time
> was about 1 sec or 151 ms/GB - it's faster then tests over Amazon
> EC2/Linode were mentioned in the article, but still much slower then
> VMWare or physical machines. Has anyone running with this issue? Or may
> be there is a way to tune Xen for less forking times?

[Sorry -- didn't see the post on this list before I responded to your
mail on xen-users.]

Which version of the kernel are you using for Xen?  And is it a pvops
kernel or a "classic xen" kernel?

Also, have you tried running Linux in HVM mode?

The original "classic Xen" kernel had a lot of performance tuning done
to it.  But the focus of pvops development for years has just been
*getting the necessary support into the kernel*, so these kinds of
microbenchmark performance things have suffered just because no one
has been looking at it.  We've got engineers looking more closely at
this now, so it's good to know what areas we need to look at.

If you use HVM mode, you'll most likely be using the same exact
codepaths that KVM will be using (hardware pagetable virtualization),
and so will hopefully get the same speed.

 -George

Re: Forking time in Xen

[Pasi Kärkkäinen]

On Mon, Jun 18, 2012 at 04:38:39PM +0700, Sergey Zhukov wrote:
> Hi,
> 
> I repost this message from xen-users list following by others
> subscribers suggestions:
> 
> 
> I found an article about forking time for redis NoSQL database in
> different systems:
> 
> http://redis.io/topics/latency
> ---------Quote------------------
> Fork time in different systems
> Modern hardware is pretty fast to copy the page table, but Xen is not.
> The problem with Xen is not virtualization-specific, but Xen-specific.
> For instance using VMware or Virutal Box does not result into slow fork
> time. The following is a table that compares fork time for different
> Redis instance size. Data is obtained performing a BGSAVE and looking at
> the latest_fork_usec filed in the INFO command output.
> 
>       * Linux beefy VM on VMware 6.0GB RSS forked in 77 milliseconds
>         (12.8 milliseconds per GB).
>       * Linux running on physical machine (Unknown HW) 6.1GB RSS forked
>         in 80 milliseconds (13.1 milliseconds per GB)
>       * Linux running on physical machine (Xeon @ 2.27Ghz) 6.9GB RSS
>         forked into 62 millisecodns (9 milliseconds per GB).
>       * Linux VM on 6sync (KVM) 360 MB RSS forked in 8.2 milliseconds
>         (23.3 millisecond per GB).
>       * Linux VM on EC2 (Xen) 6.1GB RSS forked in 1460 milliseconds
>         (239.3 milliseconds per GB).
>       * Linux VM on Linode (Xen) 0.9GBRSS forked into 382 millisecodns
>         (424 milliseconds per GB).
> 
> As you can see a VM running on Xen has a performance hit that is between
> one order to two orders of magnitude. We believe this is a severe
> problem with Xen and we hope it will be addressed ASAP.
> ----------End of quote-----------------
> 
> I made my own test with Xen 4.1 and Redis 2.4 with 7.04GB dataset. The
> test was performed on Intel Core I5 2500 processor unit. Forking time
> was about 1 sec or 151 ms/GB - it's faster then tests over Amazon
> EC2/Linode were mentioned in the article, but still much slower then
> VMWare or physical machines. Has anyone running with this issue? Or may
> be there is a way to tune Xen for less forking times?
> 

If you need good fork-performance you should use Xen PVHVM guests, not PV.

Xen PV model needs to validate the new process page tables in the hypervisor 
every time when a fork happens, so that will have some performance hit in fork-heavy workloads.

HVM does not need to do that.. so please try switching to Xen PVHVM and benchmark again.

-- Pasi