[Qemu-devel] new block layer

[Qemu-devel] new block layer

[Fabrice Bellard]

Hi,

I commited the new disk image layer of QEMU which adds the following 
features:

- new disk image formats are easier to support

- new 'qcow' image format which supports:
     * empty sector compression even if the filesystem does not support 
holes (useful for Windows users)
     * zlib based compression to directly use compressed disk images 
(note: the written sectors are not compressed again, you must use 
qemu-img to compress a disk image).
     * AES based encryption

- native VMware 3 and 4 read-only disk image support (untested - I need 
such images to test it).

- new 'all-in-one' tool 'qemu-img' to create, convert, compress and 
encrypt disk images with all the supported formats.

Note: the qcow disk image format may change without notice until the 
next release.

Fabrice.

[Qemu-devel] Re: new block layer

[Ronald]

Le Tue, 03 Aug 2004 23:41:46 +0200, Fabrice Bellard a écrit :

> Hi,
> 
> I commited the new disk image layer of QEMU which adds the following
> features:
> 
> - new disk image formats are easier to support
> 
> - new 'qcow' image format which supports:
>      * empty sector compression even if the filesystem does not support
> holes (useful for Windows users)
>      * zlib based compression to directly use compressed disk images
> (note: the written sectors are not compressed again, you must use qemu-img
> to compress a disk image).
>      * AES based encryption
> 
> - native VMware 3 and 4 read-only disk image support (untested - I need
> such images to test it).
> 
> - new 'all-in-one' tool 'qemu-img' to create, convert, compress and
> encrypt disk images with all the supported formats.
> 
> Note: the qcow disk image format may change without notice until the next
> release.
> 
> Fabrice.

qemu.exe doesn't work for me, even if the build goes fine. I have tried it
under xp. Passing -d to log does produce only an empty file.
Good news: qemu-img works, I have created a raw and a qcow image.

qemu-img, BSD (was: Re: [Qemu-devel] new block layer)

[Juergen Lock]

On Tue, Aug 03, 2004 at 10:19:08PM +0000, Fabrice Bellard wrote:
> Hi,
> 
> I commited the new disk image layer of QEMU which adds the following 
> features:
> 
>...

First: How do you use qemu-img?  If i try
	qemu-img create -b /dev/ad0 -f qcow testcow 10M
I only get the usage back.

 Second question: Has anyone patched the new block layer for BSD already?
At least FreeBSD can't access cdroms in 512 byte blocks, everything
has to be done in the cd blocksize which is 2048.  (A patch for this
from the list for the old block layer is in the port; It also needs
to use DIOCGMEDIASIZE but that was quick to re-add.)

 Here is an update for the port, I send it here because I dont
think it should be committed until the cdrom issue is fixed...

 If you want to look at the patches in the port and don't have a
FreeBSD ports tree on your system: you can check out the port from
cvs by (for example) doing

	CVSROOT=:pserver:anoncvs@anoncvs.fr.FreeBSD.org:/home/ncvs cvs co ports/emulators/qemu

 (login with password anoncvs)

 the FreeBSD patches are in the files/ dir; patch-aa and -ab are from
the list, patch-ac, patch-bd and patch-bf are FreeBSD only

 (apply this on top of the port in cvs, then remove files/patch-ba,
files/patch-bb, files/patch-bg)

 Oh and if you try this after today 2340 (iirc) Jerusalem time
you need to adjust the date in DISTNAME (and maybe more depending
on what Fabrice commits...)

Index: Makefile
===================================================================
RCS file: /home/ncvs/ports/emulators/qemu/Makefile,v
retrieving revision 1.13
diff -u -r1.13 Makefile
--- Makefile	17 Jul 2004 12:24:33 -0000	1.13
+++ Makefile	4 Aug 2004 16:36:38 -0000
@@ -8,8 +8,9 @@
 PORTNAME=	qemu
 PORTVERSION=	0.6.0s
 CATEGORIES=	emulators
-MASTER_SITES=	http://dad-answers.com/qemu/FreeBSD/
-DISTNAME=	${PORTNAME}-snapshot-2004-07-15_23
+#MASTER_SITES=	http://dad-answers.com/qemu/FreeBSD/
+MASTER_SITES=	http://dad-answers.com/qemu/
+DISTNAME=	${PORTNAME}-snapshot-2004-08-03_23
 
 MAINTAINER=	nox@jelal.kn-bremen.de
 COMMENT=	QEMU CPU Emulator
@@ -26,7 +27,7 @@
 WRKSRC=		${WRKDIR}/${DISTNAME}
 PATCH_STRIP=	-p1
 CONFIGURE_ARGS+=	--prefix=${PREFIX} --cc=${CC}\ -I${PREFIX}/include
-MAN1=		qemu.1 qemu-mkcow.1
+MAN1=		qemu.1
 ONLY_FOR_ARCHS=	i386 amd64 powerpc
 
 post-install:
Index: distinfo
===================================================================
RCS file: /home/ncvs/ports/emulators/qemu/distinfo,v
retrieving revision 1.10
diff -u -r1.10 distinfo
--- distinfo	17 Jul 2004 12:24:33 -0000	1.10
+++ distinfo	4 Aug 2004 15:59:39 -0000
@@ -1,2 +1,2 @@
-MD5 (qemu-snapshot-2004-07-15_23.tar.bz2) = a45bec4f467f47401f745147091f1644
-SIZE (qemu-snapshot-2004-07-15_23.tar.bz2) = 747718
+MD5 (qemu-snapshot-2004-08-03_23.tar.bz2) = d93635596be9a94ec452f47505d73996
+SIZE (qemu-snapshot-2004-08-03_23.tar.bz2) = 770845
Index: pkg-plist
===================================================================
RCS file: /home/ncvs/ports/emulators/qemu/pkg-plist,v
retrieving revision 1.6
diff -u -r1.6 pkg-plist
--- pkg-plist	25 Jun 2004 07:37:31 -0000	1.6
+++ pkg-plist	4 Aug 2004 16:30:36 -0000
@@ -1,7 +1,6 @@
 bin/qemu
-bin/qemu-mkcow
+bin/qemu-img
 bin/qemu-system-ppc
-bin/vmdk2raw
 %%PORTDOCS%%%%DOCSDIR%%/qemu-doc.html
 %%PORTDOCS%%%%DOCSDIR%%/qemu-tech.html
 share/qemu/bios.bin
Index: files/patch-aa
===================================================================
RCS file: /home/ncvs/ports/emulators/qemu/files/patch-aa,v
retrieving revision 1.2
diff -u -r1.2 patch-aa
--- files/patch-aa	11 Jul 2004 11:41:06 -0000	1.2
+++ files/patch-aa	4 Aug 2004 16:16:24 -0000
@@ -10,106 +10,6 @@
  
  qemu.1: qemu-doc.texi
  	./texi2pod.pl $< qemu.pod
-diff -urd --exclude=CVS ../cvs/qemu/block.c qemu-0.5.5/block.c
---- ../cvs/qemu/block.c	Sat May  8 16:27:20 2004
-+++ qemu-0.5.5/block.c	Sun May 30 16:36:53 2004
-@@ -27,6 +27,13 @@
- #include <sys/mman.h>
- #endif
- 
-+#ifdef _BSD
-+#include <sys/types.h>
-+#include <sys/stat.h>
-+#include <sys/ioctl.h>
-+#include <sys/disk.h>
-+#endif
-+
- #include "cow.h"
- 
- struct BlockDriverState {
-@@ -81,7 +88,10 @@
- {
-     int fd;
-     int64_t size;
--    struct cow_header_v2 cow_header;
-+    union {
-+      struct cow_header_v2 cow_header;
-+      uint8_t cow_buffer[2048];
-+    } cow;
- #ifndef _WIN32
-     char template[] = "/tmp/vl.XXXXXX";
-     int cow_fd;
-@@ -117,15 +127,15 @@
-     bs->fd = fd;
- 
-     /* see if it is a cow image */
--    if (read(fd, &cow_header, sizeof(cow_header)) != sizeof(cow_header)) {
-+    if (read(fd, &cow.cow_header, sizeof(cow)) != sizeof(cow)) {
-         fprintf(stderr, "%s: could not read header\n", filename);
-         goto fail;
-     }
- #ifndef _WIN32
--    if (be32_to_cpu(cow_header.magic) == COW_MAGIC &&
--        be32_to_cpu(cow_header.version) == COW_VERSION) {
-+    if (be32_to_cpu(cow.cow_header.magic) == COW_MAGIC &&
-+        be32_to_cpu(cow.cow_header.version) == COW_VERSION) {
-         /* cow image found */
--        size = cow_header.size;
-+        size = cow.cow_header.size;
- #ifndef WORDS_BIGENDIAN
-         size = bswap64(size);
- #endif    
-@@ -133,34 +143,41 @@
- 
-         bs->cow_fd = fd;
-         bs->fd = -1;
--        if (cow_header.backing_file[0] != '\0') {
--            if (stat(cow_header.backing_file, &st) != 0) {
--                fprintf(stderr, "%s: could not find original disk image '%s'\n", filename, cow_header.backing_file);
-+        if (cow.cow_header.backing_file[0] != '\0') {
-+            if (stat(cow.cow_header.backing_file, &st) != 0) {
-+                fprintf(stderr, "%s: could not find original disk image '%s'\n", filename, cow.cow_header.backing_file);
-                 goto fail;
-             }
--            if (st.st_mtime != be32_to_cpu(cow_header.mtime)) {
--                fprintf(stderr, "%s: original raw disk image '%s' does not match saved timestamp\n", filename, cow_header.backing_file);
-+            if (st.st_mtime != be32_to_cpu(cow.cow_header.mtime)) {
-+                fprintf(stderr, "%s: original raw disk image '%s' does not match saved timestamp\n", filename, cow.cow_header.backing_file);
-                 goto fail;
-             }
--            fd = open(cow_header.backing_file, O_RDONLY | O_LARGEFILE);
-+            fd = open(cow.cow_header.backing_file, O_RDONLY | O_LARGEFILE);
-             if (fd < 0)
-                 goto fail;
-             bs->fd = fd;
-         }
-         /* mmap the bitmap */
--        bs->cow_bitmap_size = ((bs->total_sectors + 7) >> 3) + sizeof(cow_header);
-+        bs->cow_bitmap_size = ((bs->total_sectors + 7) >> 3) + sizeof(cow.cow_header);
-         bs->cow_bitmap_addr = mmap(get_mmap_addr(bs->cow_bitmap_size), 
-                                    bs->cow_bitmap_size, 
-                                    PROT_READ | PROT_WRITE,
-                                    MAP_SHARED, bs->cow_fd, 0);
-         if (bs->cow_bitmap_addr == MAP_FAILED)
-             goto fail;
--        bs->cow_bitmap = bs->cow_bitmap_addr + sizeof(cow_header);
-+        bs->cow_bitmap = bs->cow_bitmap_addr + sizeof(cow.cow_header);
-         bs->cow_sectors_offset = (bs->cow_bitmap_size + 511) & ~511;
-         snapshot = 0;
-     } else 
- #endif
-     {
-+#ifdef _BSD
-+        struct stat sb;
-+        if (!fstat(fd,&sb) && (S_IFCHR & sb.st_mode)) {
-+            if (ioctl(fd, DIOCGMEDIASIZE, (off_t *)&size))
-+                size = lseek(fd, 0LL, SEEK_END);
-+        } else
-+#endif                      
-         /* standard raw image */
-         size = lseek64(fd, 0, SEEK_END);
-         bs->total_sectors = size / 512;
-Only in qemu-0.5.5: block.c.bck
 Only in qemu-0.5.5: qemu.1
 diff -urd --exclude=CVS ../cvs/qemu/target-i386/cpu.h qemu-0.5.5/target-i386/cpu.h
 --- ../cvs/qemu/target-i386/cpu.h	Thu May 20 15:01:56 2004
--- /dev/null	Wed Aug  4 19:00:00 2004
+++ files/patch-bl	Wed Aug  4 18:42:32 2004
@@ -0,0 +1,35 @@
+Index: qemu/block.c
+@@ -21,6 +21,17 @@
+  * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+  * THE SOFTWARE.
+  */
++
++#include "config-host.h"
++
++#ifdef _BSD
++#include <sys/types.h>
++#include <sys/stat.h>
++#include <sys/ioctl.h>
++#include <sys/queue.h>
++#include <sys/disk.h>
++#endif
++
+ #include "vl.h"
+ #include "block_int.h"
+ 
+@@ -514,6 +525,15 @@
+             return -1;
+         bs->read_only = 1;
+     }
++#ifdef _BSD
++    struct stat sb;
++    if (!fstat(fd,&sb) && (S_IFCHR & sb.st_mode)) {
++#ifdef DIOCGMEDIASIZE
++        if (ioctl(fd, DIOCGMEDIASIZE, (off_t *)&size))
++#endif
++            size = lseek(fd, 0LL, SEEK_END);
++    } else
++#endif
+     size = lseek64(fd, 0, SEEK_END);
+     bs->total_sectors = size / 512;
+     s->fd = fd;

Re: qemu-img, BSD (was: Re: [Qemu-devel] new block layer)

[Antony T Curtis]

On Wed, 2004-08-04 at 19:11, Juergen Lock wrote:
> On Tue, Aug 03, 2004 at 10:19:08PM +0000, Fabrice Bellard wrote:
> > Hi,
> > 
> > I commited the new disk image layer of QEMU which adds the following 
> > features:
> > 
> >...
> 
> First: How do you use qemu-img?  If i try
> 	qemu-img create -b /dev/ad0 -f qcow testcow 10M
> I only get the usage back.
> 
>  Second question: Has anyone patched the new block layer for BSD already?
> At least FreeBSD can't access cdroms in 512 byte blocks, everything
> has to be done in the cd blocksize which is 2048.  (A patch for this
> from the list for the old block layer is in the port; It also needs
> to use DIOCGMEDIASIZE but that was quick to re-add.)

I have a new patch for the new block layer - it defines a new block type
(imaginately called 'dev') and the probe checks the DIOCGMEDIASIZE and
DIOCGSECTORSIZE ioctls... It also ensures that I/O is done in multiples
of the device sector size.

I'll have a new patch together for this soon...

On my machine (FreeBSD 5-NOT_SO_CURRENT), the slirp code causes a
SIGBUS... I might investigate - it looks like it occurs near
constructing the ethernet header... poor backtraces prevents me from
finding exactly where.

>  Here is an update for the port, I send it here because I dont
> think it should be committed until the cdrom issue is fixed...
> 
>  If you want to look at the patches in the port and don't have a
> FreeBSD ports tree on your system: you can check out the port from
> cvs by (for example) doing
> 
> 	CVSROOT=:pserver:anoncvs@anoncvs.fr.FreeBSD.org:/home/ncvs cvs co ports/emulators/qemu
> 
>  (login with password anoncvs)
> 
>  the FreeBSD patches are in the files/ dir; patch-aa and -ab are from
> the list, patch-ac, patch-bd and patch-bf are FreeBSD only
> 
>  (apply this on top of the port in cvs, then remove files/patch-ba,
> files/patch-bb, files/patch-bg)
> 
>  Oh and if you try this after today 2340 (iirc) Jerusalem time
> you need to adjust the date in DISTNAME (and maybe more depending
> on what Fabrice commits...)
> 
> Index: Makefile

<snip - variant of old block patch which I originally hacked >

-- 
Antony T Curtis, BSc.                   UNIX, Linux, *BSD, Networking
antony.t.curtis@ntlworld.com            C++, J2EE, Perl, MySQL, Apache
                                        IT Consultancy.

Re: qemu-img, BSD (was: Re: [Qemu-devel] new block layer)

[Juergen Lock]

In article <1091666342.61926.68.camel@pcgem.rdg.cyberkinetica.com> you write:
>On Wed, 2004-08-04 at 19:11, Juergen Lock wrote:
>> On Tue, Aug 03, 2004 at 10:19:08PM +0000, Fabrice Bellard wrote:
>> > Hi,
>> > 
>> > I commited the new disk image layer of QEMU which adds the following 
>> > features:
>> > 
>> >...
>> 
>> First: How do you use qemu-img?  If i try
>> 	qemu-img create -b /dev/ad0 -f qcow testcow 10M
>> I only get the usage back.
>> 
I just looked at the source and found out that it expects getopt
to support mixing args and options (it doesnt increment optind to skip
over the command before calling getopt), if I change this it works:

Index: qemu/qemu-img.c
@@ -303,7 +303,6 @@
             break;
         }
     }
-    optind++;
     if (optind >= argc) 
         help();
     filename = argv[optind++];
@@ -367,7 +366,6 @@
             break;
         }
     }
-    optind++;
     if (optind >= argc) 
         help();
     filename = argv[optind++];
@@ -505,7 +503,6 @@
             break;
         }
     }
-    optind++;
     if (optind >= argc) 
         help();
     filename = argv[optind++];
@@ -633,7 +630,6 @@
             break;
         }
     }
-    optind++;
     if (optind >= argc) 
         help();
     filename = argv[optind++];
@@ -680,6 +676,7 @@
     if (argc < 2)
         help();
     cmd = argv[1];
+    ++optind;
     if (!strcmp(cmd, "create")) {
         img_create(argc, argv);
     } else if (!strcmp(cmd, "commit")) {

>>  Second question: Has anyone patched the new block layer for BSD already?
>> At least FreeBSD can't access cdroms in 512 byte blocks, everything
>> has to be done in the cd blocksize which is 2048.  (A patch for this
>> from the list for the old block layer is in the port; It also needs
>> to use DIOCGMEDIASIZE but that was quick to re-add.)
>
>I have a new patch for the new block layer - it defines a new block type
>(imaginately called 'dev') and the probe checks the DIOCGMEDIASIZE and
>DIOCGSECTORSIZE ioctls... It also ensures that I/O is done in multiples
>of the device sector size.
>
>I'll have a new patch together for this soon...
>
 Thanks, that would be cool.

>On my machine (FreeBSD 5-NOT_SO_CURRENT), the slirp code causes a
>SIGBUS... I might investigate - it looks like it occurs near
>constructing the ethernet header... poor backtraces prevents me from
>finding exactly where.
>
 Hmm, never seen that here (RELENG_5_2)...

>>  Here is an update for the port, I send it here because I dont
>> think it should be committed until the cdrom issue is fixed...
>> ...

>>  Oh and if you try this after today 2340 (iirc) Jerusalem time
>> you need to adjust the date in DISTNAME (and maybe more depending
>> on what Fabrice commits...)

 You need to change lseek64 to lseek in files/patch-bl now btw.

	Juergen