[Qemu-devel] [PATCH] cut large snapshot startup time (Vista runs in 5 secs)

[Qemu-devel] [PATCH] cut large snapshot startup time (Vista runs in 5 secs)

[Shahar Livne]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

This patch is a result of profiling research which I've done to shorten 
the startup time of a windows Vista snapshot on kvm, as part of 
http://support-matrix.com product.

The virtualization engine of SupportMatrix(tm) is based on kvm-35.
Background: The product starts for each network user (e.g. 30 users) a 
specific kvm/qemu snapshot based on details from the application 
database, and presents it on the user's browser using vnc java applet.

Everything played fine with less demanding guest OSs (like Windows XP / 
Windows 2000), but Vista which requires minimum 512MB took too long to 
start.
A customer's request to lower a the startup time of a vista snapshot 
from 10 seconds by at least 50% was the reason for the profiling effort.

It was found that loading the stored RAM within the qcow2 is responsible 
for a major part of the startup time. The function in vl.c, 
ram_load_static on snapshot loading (and ram_save_static on snapshot 
saving) run RAM/BDRV_HASH_BLOCK_SIZE times, which means that for 0.5GB 
RAM, and the default BDRV_HASH_BLOCK_SIZE=1k there are 500k calls of the 
ram_decompress_buf function (practically zlib decompression).
This is heavy, and it takes long time (around 9 seconds).
The patch uses a 16 times bigger buffer, and therefore reduces the 
amount of calls by 16 times (was found to be the best for my setup by 
trial and error). The time of the ram_load_static was reduced from 9 
seconds to around 4-5 seconds.
This change doesn't disturb also kvm (e.g. the video-addresses are still 
fully skipped within ram_load_static and ram_save_static).
Changing the compression (avoiding it or changing the default 
Z_DEFAULT_STRATEGY algorithm) did not improve the numbers.

Since I've seen that also in recent versions of qemu and kvm, the 
BDRV_HASH_BLOCK_SIZE is still 1k, I suggest the change. This change 
becomes more relevant when snapshots of guests with large amount of RAM 
are used.
I post it on qemu mailing list, since it is qemu issue and not kvm issue.

Details of the test setup:
Windows Vista was installed with 512MB.
Host OS is RedHat Enterprise 5.0
The server is Xeon E5310 1.60GHz, 8 cores, 16GB.
NO disk raids. Just simple IDE disk.
Final Vista snapshot startup time is around 5 seconds.

It is important to mention that old snapshots must be converted once 
this patch is applied.
Also, I would like to note that I've encountered no problems while 
working with the 16k based snapshots.
The patch was done against trunk r4497.

I'll be glad to get comments,
Shahar

[-- Attachment #2: snapshot_startup_time_cut.diff --]
[-- Type: text/x-patch, Size: 322 bytes --]

diff -Nuar qemu-4497/vl.c qemu-mod/vl.c
--- qemu-4497/vl.c	2008-05-19 12:26:46.000000000 +0300
+++ qemu-mod/vl.c	2008-05-19 12:30:51.000000000 +0300
@@ -6407,7 +6407,7 @@
     return 0;
 }
 
-#define BDRV_HASH_BLOCK_SIZE 1024
+#define BDRV_HASH_BLOCK_SIZE 16384
 #define IOBUF_SIZE 4096
 #define RAM_CBLOCK_MAGIC 0xfabe
 

Re: [Qemu-devel] [PATCH] cut large snapshot startup time (Vista runs in 5 secs)

[andrzej zaborowski]

On 19/05/2008, Shahar Livne <shahar@codefidence.com> wrote:
>  It is important to mention that old snapshots must be converted once this
> patch is applied.

Great finding, but if the snapshot format changes, you need to at
least attempt to detect old snapshots.  Qemu has an easy mechanism for
this (note the version_id parameter, just increase it).
-- 
Please do not print this email unless absolutely necessary. Spread
environmental awareness.

Re: [Qemu-devel] [PATCH] cut large snapshot startup time (Vista runs in 5 secs)

[Shahar Livne]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

Andrzej - thank you for the remark.

version_id increase is indeed a good way to track the change in the 
format, and I will be happy to implement it if the maintainers find 
interest in this improvement. It can also join the next version_id 
change, which may come from other reasons.


For my purposes, I didn't have to work with the 2 formats in parallel, 
as I just ran one time conversion.

The conversion was done by firing a modified qemu that read the 
static_ram in the old format, and wrote the static_ram back in the new 
format (patch attached).

The converter was started with -S, so the VM never ran. When the monitor 
prompt appeared, I just asked for loadvm, delvm, savevm, quit, and the 
image was converted. A script did this job on the 500 old images which 
required conversion, and that was it.

Another small remark: If the old image was saved while -usb flag in the 
qemu was present, it must be present also during the conversion, or else 
the usb related state is not saved. RAM saving depends on the current 
hardware.

Shahar


andrzej zaborowski wrote:
> On 19/05/2008, Shahar Livne <shahar@codefidence.com> wrote:
>   
>>  It is important to mention that old snapshots must be converted once this
>> patch is applied.
>>     
>
> Great finding, but if the snapshot format changes, you need to at
> least attempt to detect old snapshots.  Qemu has an easy mechanism for
> this (note the version_id parameter, just increase it).
>   


[-- Attachment #2: qcow2-static-ram-format-converter.patch --]
[-- Type: text/x-patch, Size: 1670 bytes --]

Index: vl.c
===================================================================
--- vl.c	(revision 688)
+++ vl.c	(revision 689)
@@ -5788,6 +5788,10 @@
     return 0;
 }
 
+/* The PREV_BDRV_HASH_BLOCK_SIZE is used for converter */
+/* It can be set to the default of 1024, and then it */
+/* reads old format and writes new format */
+#define PREV_BDRV_HASH_BLOCK_SIZE 16384
 #define BDRV_HASH_BLOCK_SIZE 16384
 #define IOBUF_SIZE 4096
 #define RAM_CBLOCK_MAGIC 0xfabe
@@ -6013,7 +6017,7 @@
         return -EINVAL;
     if (ram_decompress_open(s, f) < 0)
         return -EINVAL;
-    for(i = 0; i < phys_ram_size; i+= BDRV_HASH_BLOCK_SIZE) {
+    for(i = 0; i < phys_ram_size; i+= PREV_BDRV_HASH_BLOCK_SIZE) {
 #ifdef USE_KVM
         if (kvm_allowed && (i>=0xa0000) && (i<0xc0000)) /* do not access video-addresses */
             continue;
@@ -6023,7 +6027,7 @@
             goto error;
         }
         if (buf[0] == 0) {
-            if (ram_decompress_buf(s, phys_ram_base + i, BDRV_HASH_BLOCK_SIZE) < 0) {
+            if (ram_decompress_buf(s, phys_ram_base + i, PREV_BDRV_HASH_BLOCK_SIZE) < 0) {
                 fprintf(stderr, "Error while reading ram block address=0x%08x", i);
                 goto error;
             }
@@ -6041,7 +6045,7 @@
                 goto error;
             }
             if (bdrv_read(bs_table[bs_index], sector_num, phys_ram_base + i, 
-                          BDRV_HASH_BLOCK_SIZE / 512) < 0) {
+                          PREV_BDRV_HASH_BLOCK_SIZE / 512) < 0) {
                 fprintf(stderr, "Error while reading sector %d:%" PRId64 "\n", 
                         bs_index, sector_num);
                 goto error;