[Qemu-devel] [patch 0/4] [RFC] Add flash emulation to the Versatile PB platform

[Qemu-devel] [patch 0/4] [RFC] Add flash emulation to the Versatile PB platform

[Thomas Petazzoni]

Hi,

This small set of patches fixes a few problems in the CFI01 flash
emulation or make little improvements, and finally adds flash
emulation to the Versatile PB platform.

I am not completely sure about the correctness of all patches, so
don't hesitate to give your feedback and comments about them.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

[Qemu-devel] [patch 1/4] Increase write buffer size in pflash emulation

[Thomas Petazzoni]

[-- Attachment #1: pflash-cfi01-increase-write-buffer-size --]
[-- Type: text/plain, Size: 1484 bytes --]

The current flash emulation code advertises a write buffer size of 16
bytes (1 << 4, according to offset 0x2A of the CFI table). This is
very small compared to normal write buffer sizes, and makes the
process of writing to the flash very slow (at least from U-Boot).

This patch increases this size to 2048 bytes. Except the modification
of the CFI table, the only other required modification is to change
the type of the cmd local variable in pflash_write() from uint8_t to
uint16_t, because the length of the transfer doesn't fit on 8 bits
anymore.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 hw/pflash_cfi01.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Index: qemu/hw/pflash_cfi01.c
===================================================================
--- qemu.orig/hw/pflash_cfi01.c
+++ qemu/hw/pflash_cfi01.c
@@ -200,7 +200,7 @@
 {
     target_ulong boff;
     uint8_t *p;
-    uint8_t cmd;
+    uint16_t cmd;
 
     cmd = value;
     offset -= pfl->base;
@@ -583,7 +583,7 @@
     pfl->cfi_table[0x28] = 0x02;
     pfl->cfi_table[0x29] = 0x00;
     /* Max number of bytes in multi-bytes write */
-    pfl->cfi_table[0x2A] = 0x04;
+    pfl->cfi_table[0x2A] = 0x0B;
     pfl->cfi_table[0x2B] = 0x00;
     /* Number of erase block regions (uniform) */
     pfl->cfi_table[0x2C] = 0x01;

-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

[Qemu-devel] [patch 2/4] Reset wcycle after erase confirm

[Thomas Petazzoni]

[-- Attachment #1: pflash-cfi01-after-erase-confirm-reset-wcycle --]
[-- Type: text/plain, Size: 1134 bytes --]

pfl->wcycle was set to 1 when the erase confirm command was set, which
lead to the next command being misinterpreted by Qemu:

pflash_write: Unimplemented flash cmd sequence (offset 00000000,
wcycle 0x1 cmd 0x20 value 0x70)

This patch fixes this issue by resetting pfl->wcycle to 0 on erase
confirm so that the next command is considered as a new one.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 hw/pflash_cfi01.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: qemu/hw/pflash_cfi01.c
===================================================================
--- qemu.orig/hw/pflash_cfi01.c
+++ qemu/hw/pflash_cfi01.c
@@ -267,7 +267,7 @@
         case 0x20: /* Block erase */
         case 0x28:
             if (cmd == 0xd0) { /* confirm */
-                pfl->wcycle = 1;
+                pfl->wcycle = 0;
                 pfl->status |= 0x80;
             } else if (cmd == 0xff) { /* read array mode */
                 goto reset_flash;

-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

[Qemu-devel] [patch 3/4] Improve pflash cfi01 debug messages

[Thomas Petazzoni]

[-- Attachment #1: pflash-cfi01-improve-debug-msgs --]
[-- Type: text/plain, Size: 1239 bytes --]

This patches slightly improves the debugging messages in pflash_read()
and pflash_write().

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 hw/pflash_cfi01.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Index: qemu/hw/pflash_cfi01.c
===================================================================
--- qemu.orig/hw/pflash_cfi01.c
+++ qemu/hw/pflash_cfi01.c
@@ -111,8 +111,8 @@
     else if (pfl->width == 4)
         boff = boff >> 2;
 
-    DPRINTF("%s: reading offset " TARGET_FMT_lx " under cmd %02x\n",
-            __func__, boff, pfl->cmd);
+    DPRINTF("%s: reading offset " TARGET_FMT_lx " under cmd %02x width %d\n",
+            __func__, offset, pfl->cmd, width);
 
     switch (pfl->cmd) {
     case 0x00:
@@ -205,7 +205,7 @@
     cmd = value;
     offset -= pfl->base;
 
-    DPRINTF("%s: offset " TARGET_FMT_lx " %08x %d wcycle 0x%x\n",
+    DPRINTF("%s: writing offset " TARGET_FMT_lx " value %08x width %d wcycle 0x%x\n",
             __func__, offset, value, width, pfl->wcycle);
 
     /* Set the device in I/O access mode */

-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

[Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[Thomas Petazzoni]

[-- Attachment #1: versatilepb-add-flash-support --]
[-- Type: text/plain, Size: 3763 bytes --]

This patch adds the emulation of the 64 MB Intel Flash present at
address 0x34000000 on the ARM Versatile PB platform, with a 256 KB
sector size. This flash emulation is enabled using the -pflash
option. If not enabled, Qemu falls back to the traditionnal way of
loading the kernel.

The RAM size is also hardcoded to 128 MB, and the memory size provided
by Qemu's -m option is the total memory size, so it must be at least
192 MB to run the Versatile PB with flash emulation. The same method
is used on other platforms, such as Gumstix.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 hw/versatilepb.c |   44 +++++++++++++++++++++++++++++++++++++-------
 1 file changed, 37 insertions(+), 7 deletions(-)

Index: qemu/hw/versatilepb.c
===================================================================
--- qemu.orig/hw/versatilepb.c
+++ qemu/hw/versatilepb.c
@@ -15,6 +15,7 @@
 #include "sysemu.h"
 #include "pci.h"
 #include "boards.h"
+#include "flash.h"
 
 /* Primary interrupt controller.  */
 
@@ -159,6 +160,12 @@
 
 static struct arm_boot_info versatile_binfo;
 
+#define VERSATILE_RAM_ADDR        0
+#define VERSATILE_RAM_SIZE        (128 * 1024 * 1024)
+#define VERSATILE_FLASH_ADDR      0x34000000
+#define VERSATILE_FLASH_SIZE      (64 * 1024 * 1024)
+#define VERSATILE_FLASH_SECT_SIZE (256 * 1024)
+
 static void versatile_init(ram_addr_t ram_size, int vga_ram_size,
                      const char *boot_device, DisplayState *ds,
                      const char *kernel_filename, const char *kernel_cmdline,
@@ -174,6 +181,7 @@
     int n;
     int done_smc = 0;
     int index;
+    int hasflash = 0;
 
     if (!cpu_model)
         cpu_model = "arm926";
@@ -184,7 +192,9 @@
     }
     /* ??? RAM should repeat to fill physical memory space.  */
     /* SDRAM at address zero.  */
-    cpu_register_physical_memory(0, ram_size, IO_MEM_RAM);
+    cpu_register_physical_memory(VERSATILE_RAM_ADDR,
+				 VERSATILE_RAM_SIZE,
+				 qemu_ram_alloc(VERSATILE_RAM_SIZE) | IO_MEM_RAM);
 
     arm_sysctl_init(0x10000000, 0x41007004);
     pic = arm_pic_init_cpu(env);
@@ -249,6 +259,21 @@
     /* Add PL031 Real Time Clock. */
     pl031_init(0x101e8000,pic[10]);
 
+    index = drive_get_index(IF_PFLASH, 0, 0);
+    if (index != -1) {
+      if (!pflash_cfi01_register(VERSATILE_FLASH_ADDR,
+				 qemu_ram_alloc(VERSATILE_FLASH_SIZE),
+				 drives_table[index].bdrv,
+				 VERSATILE_FLASH_SECT_SIZE,
+				 VERSATILE_FLASH_SIZE / VERSATILE_FLASH_SECT_SIZE,
+				 4, 0, 0, 0, 0)) {
+	fprintf(stderr, "qemu: error registering flash memory.\n");
+	exit(1);
+      }
+
+      hasflash = 1;
+    }
+
     /* Memory map for Versatile/PB:  */
     /* 0x10000000 System registers.  */
     /* 0x10001000 PCI controller config registers.  */
@@ -285,12 +310,17 @@
     /*  0x101f3000 UART2.  */
     /* 0x101f4000 SSPI.  */
 
-    versatile_binfo.ram_size = ram_size;
-    versatile_binfo.kernel_filename = kernel_filename;
-    versatile_binfo.kernel_cmdline = kernel_cmdline;
-    versatile_binfo.initrd_filename = initrd_filename;
-    versatile_binfo.board_id = board_id;
-    arm_load_kernel(env, &versatile_binfo);
+    if (! hasflash) {
+      versatile_binfo.ram_size = VERSATILE_RAM_SIZE;
+      versatile_binfo.kernel_filename = kernel_filename;
+      versatile_binfo.kernel_cmdline = kernel_cmdline;
+      versatile_binfo.initrd_filename = initrd_filename;
+      versatile_binfo.board_id = board_id;
+      arm_load_kernel(env, &versatile_binfo);
+    }
+    else {
+      env->regs[15] = 0x34000000;
+    }
 }
 
 static void vpb_init(ram_addr_t ram_size, int vga_ram_size,

-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: [Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[Paul Brook]

> The RAM size is also hardcoded to 128 MB

This is wrong.

Paul

Re: [Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[Thomas Petazzoni]

Le Thu, 2 Oct 2008 17:33:10 +0100,
Paul Brook <paul@codesourcery.com> a écrit :

> > The RAM size is also hardcoded to 128 MB
> 
> This is wrong.

Ok. So, how should it be implemented ?

(For reference, as far as I understand, the RAM size is also hardcoded
in other platforms, such as Gumstix)

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: [Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[andrzej zaborowski]

2008/10/2 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>:
> This patch adds the emulation of the 64 MB Intel Flash present at
> address 0x34000000 on the ARM Versatile PB platform, with a 256 KB
> sector size. This flash emulation is enabled using the -pflash
> option. If not enabled, Qemu falls back to the traditionnal way of
> loading the kernel.
>
> The RAM size is also hardcoded to 128 MB, and the memory size provided
> by Qemu's -m option is the total memory size, so it must be at least
> 192 MB to run the Versatile PB with flash emulation. The same method
> is used on other platforms, such as Gumstix.

You would have to update the QEMUMachine ram_require field at the
bottom of the file to reflect that.  But I think Paul thinks this
board doesn't have a fixed amount of memory.

Cheers

Re: [Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[Thomas Petazzoni]

Le Thu, 2 Oct 2008 20:25:49 +0200,
"andrzej zaborowski" <balrogg@gmail.com> a écrit :

> You would have to update the QEMUMachine ram_require field at the
> bottom of the file to reflect that.  But I think Paul thinks this
> board doesn't have a fixed amount of memory.

Ok. But maybe Paul could be a little bit more descriptive than just
«This is wrong», it would help me in understanding what's wrong and how
it should be fixed.

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: [Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[Thomas Petazzoni]

Le Thu, 2 Oct 2008 17:33:10 +0100,
Paul Brook <paul@codesourcery.com> a écrit :

> > The RAM size is also hardcoded to 128 MB
> 
> This is wrong.

Another solution is to set

 .ram_require = 64 * 1024 * 1024

So that 64 MB will always be allocated for the Flash. To these 64 MB,
the user-specified amount of memory is added for the RAM.

Is this what you're thinking of ?

I don't mind changing my implementation, as I said, these patches are
for review. But just saying « This is wrong » is not helpful, because
it doesn't give any suggestion on how to improve the proposed patch.
Frankly, I do not understand what your answer was so harsh.

Anyway, thanks for your review,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: [Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[Thomas Petazzoni]

Le Fri, 3 Oct 2008 15:44:03 +0200,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a écrit :

> Another solution is to set
> 
>  .ram_require = 64 * 1024 * 1024
> 
> So that 64 MB will always be allocated for the Flash. To these 64 MB,
> the user-specified amount of memory is added for the RAM.

The following patch implements this solution. The drawback is that
these 64 MB of Flash are allocated even if the -pflash option is not
passed.

Thomas

---

Add Flash support to the Versatile PB platform

This patch adds the emulation of the 64 MB Intel Flash present at
address 0x34000000 on the ARM Versatile PB platform, with a 256 KB
sector size. This flash emulation is enabled using the -pflash
option. If not enabled, Qemu falls back to the traditionnal way of
loading the kernel.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 hw/versatilepb.c |   43 ++++++++++++++++++++++++++++++++++++-------
 1 file changed, 36 insertions(+), 7 deletions(-)

Index: qemu/hw/versatilepb.c
===================================================================
--- qemu.orig/hw/versatilepb.c
+++ qemu/hw/versatilepb.c
@@ -15,6 +15,7 @@
 #include "sysemu.h"
 #include "pci.h"
 #include "boards.h"
+#include "flash.h"
 
 /* Primary interrupt controller.  */
 
@@ -159,6 +160,10 @@
 
 static struct arm_boot_info versatile_binfo;
 
+#define VERSATILE_FLASH_ADDR      0x34000000
+#define VERSATILE_FLASH_SIZE      (64 * 1024 * 1024)
+#define VERSATILE_FLASH_SECT_SIZE (256 * 1024)
+
 static void versatile_init(ram_addr_t ram_size, int vga_ram_size,
                      const char *boot_device, DisplayState *ds,
                      const char *kernel_filename, const char *kernel_cmdline,
@@ -174,6 +179,7 @@
     int n;
     int done_smc = 0;
     int index;
+    int hasflash = 0;
 
     if (!cpu_model)
         cpu_model = "arm926";
@@ -184,7 +190,8 @@
     }
     /* ??? RAM should repeat to fill physical memory space.  */
     /* SDRAM at address zero.  */
-    cpu_register_physical_memory(0, ram_size, IO_MEM_RAM);
+    cpu_register_physical_memory(0, ram_size,
+				 qemu_ram_alloc(ram_size) | IO_MEM_RAM);
 
     arm_sysctl_init(0x10000000, 0x41007004);
     pic = arm_pic_init_cpu(env);
@@ -249,6 +256,21 @@
     /* Add PL031 Real Time Clock. */
     pl031_init(0x101e8000,pic[10]);
 
+    index = drive_get_index(IF_PFLASH, 0, 0);
+    if (index != -1) {
+      if (!pflash_cfi01_register(VERSATILE_FLASH_ADDR,
+				 qemu_ram_alloc(VERSATILE_FLASH_SIZE),
+				 drives_table[index].bdrv,
+				 VERSATILE_FLASH_SECT_SIZE,
+				 VERSATILE_FLASH_SIZE / VERSATILE_FLASH_SECT_SIZE,
+				 4, 0, 0, 0, 0)) {
+	fprintf(stderr, "qemu: error registering flash memory.\n");
+	exit(1);
+      }
+
+      hasflash = 1;
+    }
+
     /* Memory map for Versatile/PB:  */
     /* 0x10000000 System registers.  */
     /* 0x10001000 PCI controller config registers.  */
@@ -285,12 +307,17 @@
     /*  0x101f3000 UART2.  */
     /* 0x101f4000 SSPI.  */
 
-    versatile_binfo.ram_size = ram_size;
-    versatile_binfo.kernel_filename = kernel_filename;
-    versatile_binfo.kernel_cmdline = kernel_cmdline;
-    versatile_binfo.initrd_filename = initrd_filename;
-    versatile_binfo.board_id = board_id;
-    arm_load_kernel(env, &versatile_binfo);
+    if (! hasflash) {
+      versatile_binfo.ram_size = ram_size;
+      versatile_binfo.kernel_filename = kernel_filename;
+      versatile_binfo.kernel_cmdline = kernel_cmdline;
+      versatile_binfo.initrd_filename = initrd_filename;
+      versatile_binfo.board_id = board_id;
+      arm_load_kernel(env, &versatile_binfo);
+    }
+    else {
+      env->regs[15] = 0x34000000;
+    }
 }
 
 static void vpb_init(ram_addr_t ram_size, int vga_ram_size,
@@ -320,6 +347,7 @@
     .desc = "ARM Versatile/PB (ARM926EJ-S)",
     .init = vpb_init,
     .use_scsi = 1,
+    .ram_require = VERSATILE_FLASH_SIZE,
 };
 
 QEMUMachine versatileab_machine = {
@@ -327,4 +355,5 @@
     .desc = "ARM Versatile/AB (ARM926EJ-S)",
     .init = vab_init,
     .use_scsi = 1,
+    .ram_require = VERSATILE_FLASH_SIZE,
 };

-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: [Qemu-devel] [patch 1/4] Increase write buffer size in pflash emulation

[andrzej zaborowski]

2008/10/2 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>:
> --- qemu.orig/hw/pflash_cfi01.c
> +++ qemu/hw/pflash_cfi01.c
> @@ -200,7 +200,7 @@
>  {
>     target_ulong boff;
>     uint8_t *p;
> -    uint8_t cmd;
> +    uint16_t cmd;
>
>     cmd = value;

I think the function of this assignment might have been to clip the
command to one byte also.

How about chaging pfl->counter = cmd; to pfl->counter = value;?

Cheers

Re: [Qemu-devel] [patch 1/4] Increase write buffer size in pflash emulation

[Thomas Petazzoni]

Le Sat, 4 Oct 2008 01:05:55 +0200,
"andrzej zaborowski" <balrogg@gmail.com> a écrit :

> How about chaging pfl->counter = cmd; to pfl->counter = value;?

Makes sense, thanks for the idea! Here is an updated patch.

---

Increase write buffer size in pflash emulation

The current flash emulation code advertises a write buffer size of 16
bytes (1 << 4, according to offset 0x2A of the CFI table). This is
very small compared to normal write buffer sizes, and makes the
process of writing to the flash very slow (at least from U-Boot).

This patch increases this size to 2048 bytes. Except the modification
of the CFI table, the only other required modification is to use
"value" instead of "cmd" to set pfl->counter, because cmd is truncated
to the 8 lower bits of value, while the number of bytes for a write
can now be greater than 255 bytes.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 hw/pflash_cfi01.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Index: qemu/hw/pflash_cfi01.c
===================================================================
--- qemu.orig/hw/pflash_cfi01.c
+++ qemu/hw/pflash_cfi01.c
@@ -276,8 +276,8 @@
 
             break;
         case 0xe8:
-            DPRINTF("%s: block write of %x bytes\n", __func__, cmd);
-            pfl->counter = cmd;
+            DPRINTF("%s: block write of %x bytes\n", __func__, value);
+            pfl->counter = value;
             pfl->wcycle++;
             break;
         case 0x60:
@@ -583,7 +583,7 @@
     pfl->cfi_table[0x28] = 0x02;
     pfl->cfi_table[0x29] = 0x00;
     /* Max number of bytes in multi-bytes write */
-    pfl->cfi_table[0x2A] = 0x04;
+    pfl->cfi_table[0x2A] = 0x0B;
     pfl->cfi_table[0x2B] = 0x00;
     /* Number of erase block regions (uniform) */
     pfl->cfi_table[0x2C] = 0x01;


-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: [Qemu-devel] [patch 4/4] Add Flash support to the Versatile PB platform

[Thomas Petazzoni]

Le Fri, 3 Oct 2008 16:37:13 +0200,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a écrit :

> The following patch implements this solution. The drawback is that
> these 64 MB of Flash are allocated even if the -pflash option is not
> passed.

Paul, any opinion on this new version of the patch ?

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

[Qemu-devel] [Patch 0/4] [RFC] Zero Cluster Dedup, Offline dedup, qemu-img extentions

[Shahar Frank]

[-- Attachment #1: Type: text/plain, Size: 5562 bytes --]

Hi All,

This is a rewrite of the previous "zero dedup" patch I sent.

As it is described in the first patch the problem this patch attempt to solve is the "inflation" of the COW image. In a system that uses templates and COW images above (linked clones in VMWare terms) there is a problem that over time the COW layer deviates from the base image. Some of it is justified (data), but some of it is noise (undeleted files in NTFS, temporary files, swap, etc.).

The following patch introduces two mechanisms to handle the above problem:
1. A zero dedup - an extention to the qcow2 format that identifies zero cluster writes and dedup it to a shared zero cluster (similar to the shared zero cluster in the Linux kernel). Alternatively, when possible, the cluster is just de-allocated (was Laurent Vivier suggestion). In either case, the original cluster is de-allocated and its space can be reused for new clusters. To employ this feature, some kind of cleaning utility is expected to run in the OS context or offline. This utility should wipe unused, and/or temporary space by writing zeros on it. This will lead to space deallocation via the zero dedup mechanism.
2. A general dedup infrastructure to be used to compress the image. The current version supports only intra image deduping but future versions will be extended to support across images deduping. This can mitigate the effect of most OS updates, patches, virus updates, application installations and updates, etc. For example, an application is installed within an image that is part of a deduped repository, is going to use more space only if no other image contains this application. In mass deployment the benefit is huge. From my experience even the current intra-image version can save about %30 of the space.

In addition, a set of new verbs are implemented to extend qemu-img:
1. Info -r flag is added to show the image tree
2. Map [-r] [-s] shows the logical to physical mapping (optionally for the entire image tree, and/or with md5sum of the data). This can be used to check, verify and collect statistics about the internal image layout and its attributes (how fragmented it is, how long the cluster sequences are, etc.). It can also used to validate the data integrity for example, before and after image manipulations, etc.
3. Check performs an internal image check. Right now it is implemented only for qcow2. From my experience, this check can validate image integrity or identify image corruptions long before they corruption are user visible.
4. Dedup perform (simple, naïve) dedup of cluster X to cluster Y. Even in this early stage it can be used to compress the image and save space.

The changes from the first version are:
1. The zero dedup mechanism is improved to handle sequences of clusters
2. If no backing file is used, a "hole" is punched instead of deduping
3. A general dedup verb is added to the qemu-img.
4. An md5sum is added to the map verb.

The new dedup mechanism is a new fascinating way to mess up your images ;-) but is can also used to reduce the image storage size - I tested it on a pretty clean XP image and I got %30 dedup rate. Note that a dedup between images (not implemented yet) is expected to be much more efficient (%60 and higher).

Many of the issues covered here were raised by Kevin Wolf and Laurent Vivier, so I want to thanks them for their attention and comments.

Patch 1: Basic zero dedup optimization
Patch 2: New image checking verbs and extensionsh
Patch 3: Md5sum extension to map
Patch 4: offline dedup verb

The patches are not ready for integration, but they are very close to that. I send them for review and comments.

Still missing:
1. A flag to qemu main to control the zero dedup.
2. Cleanup (debugging, other).
3. A proper (efficient) dedup utility.
4. A method to compact/defrag the image after the dedup.
5. ?

The patches are tested as follows:

I created an image and export it using nbd:
./qemu-img create -b /tmp/data -f qcow2 /tmp/test.qcow2  && ./qemu-nbd -p 999 -vv /tmp/test.qcow2 # with backing file
# OR
./qemu-img create -f qcow2 /tmp/test.qcow2  && ./qemu-nbd -p 999 -vv /tmp/test.qcow2 # without a backing file

Then I mount it and run the check script

nbd-client localhost 999 /dev/nbd0

./checkzopt.sh /tmp/test.qcow2 /dev/nbd0

# data integrity check
./qemu-img map -s /tmp/test.qcow2

# image integrity check
./qemu-img check /tmp/test.qcow2

The general dedup verb is tested as follows:

# copy the original image 
cp winXp.qcow2 winXp-d.qcow2

# create a sorted cluster hashes list. The list is also secondary sorted using the physical offset to dedup all duplications into the lowest offset cluster.
./qemu-img map -s winXp-d.qcow2 | sort -k 4 -k 3 > winXp-d.sorted

# Create a script of dedup commands:
awk '{ if ($4 == lasthash) { print "./qemu-img dedup", $1,  $2, lastoffs; next } else { lasthash = $4; lastoffs = $2; }}' winXp-d.sorted > winXp-d.dedup

# check how many cluster we are going to dedup
wc -l winXp-d.dedup

# perform the dedup - takes a lot of time...
sh winXp-d.dedup

# check the resulting image consistency
./qemu-img check winXp-d.qcow2

# generate md5sums of the data clusters and compare it to the original image
./qemu-img map -s winXp-d.qcow2 > /tmp/1
./qemu-img map -s winXp.qcow2 > /tmp/2
cut -f 2,4 -d ' ' /tmp/1 > /tmp/1.cut
cut -f 2,4 -d ' ' /tmp/2 > /tmp/2.cut
diff /tmp/[12].cut

# check that it can run...
kvm winXp-d.qcow2

Signed-off-by: Shahar Frank <shaharf@qumranet.com>

[-- Attachment #2: checkzopt.sh --]
[-- Type: application/octet-stream, Size: 1714 bytes --]

#!/bin/bash

if [ -z "$1" -o -z "$2" -o ! -r "$1" -o ! -e "$2" ]; then
	echo "Usage: `basename $0` <file> <image>"
	exit 1
fi

f=$1
i=$2

function check(){
	sync
	du -a $f
	ls -l $f
	./qemu-img map $f
	./qemu-img check $f || exit 1
}

function title(){
	echo ""
	echo "#### " $*
}

function echodo() {
	echo "$*"
	eval "$*"
}

title "initial check"
check

title "normal two clusters"
echodo dd bs=4k count=2 seek=0 if=/boot/vmlinuz of=$i
check

title "first zero"
echodo dd bs=4k count=1 seek=2 if=/dev/zero of=$i
check

title "second zero"
echodo dd bs=4k count=1 seek=3 if=/dev/zero of=$i
check

title "non zero"
echodo dd bs=4k count=2 seek=4 if=/boot/vmlinuz of=$i
check

title "zero sequence"
echodo dd bs=4k count=5 seek=5 if=/dev/zero of=$i
check

title "two zero -> non zero"
echodo dd bs=4k count=2 seek=7 if=/boot/vmlinuz of=$i
check

title "restore to zero"
echodo dd bs=4k count=2 seek=7 if=/dev/zero of=$i
check

title "partial write1"
echodo dd bs=2k count=1 seek=15 if=/dev/zero of=$i
check

title "restore to zero 2"
echodo dd bs=4k count=1 seek=7 if=/dev/zero of=$i
#echodo dd bs=4k count=2 seek=7 if=/dev/zero of=$i
check

title "rewrite full to zero"
echodo dd bs=4k count=2 seek=0 if=/dev/zero of=$i
check

title "rewrite normal"
echodo dd bs=4k count=2 seek=2 if=/boot/vmlinuz of=$i
check

title "create long data section"
echodo dd bs=8k count=10 seek=1 if=/boot/vmlinuz of=$i
check

title "rewrite zero two clusters at a time"
echodo dd bs=8k count=1 seek=1 if=/dev/zero of=$i
check

title "rewrite zero three clusters at a time"
echodo dd bs=12k count=1 seek=1 if=/dev/zero of=$i
check

title "rewrite zero four clusters at a time"
echodo dd bs=16k count=1 seek=1 if=/dev/zero of=$i
check