* Re: [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files
2013-08-11 16:13 ` [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files Paolo Bonzini
@ 2013-08-12 1:31 ` Fam Zheng
2013-08-12 7:40 ` Paolo Bonzini
2013-08-12 8:39 ` Andreas Färber
2013-08-12 11:28 ` Stefan Hajnoczi
2 siblings, 1 reply; 9+ messages in thread
From: Fam Zheng @ 2013-08-12 1:31 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kwolf, qemu-devel
On Sun, 08/11 18:13, Paolo Bonzini wrote:
> VMware ESX hosts use a variant of the VMDK3 format, identified by the
> vmfsSparse create type ad the VMFSSPARSE extent type.
>
> It has 16 KB grain tables (L2) and a variable-size grain directory (L1).
> In addition, the grain size is always 512, but that is not a problem
> because it is included in the header.
>
> The format of the extents is documented in the VMDK spec. The format
> of the descriptor file is not documented precisely, but it can be
> found at http://kb.vmware.com/kb/10026353 (Recreating a missing virtual
> machine disk (VMDK) descriptor file for delta disks).
>
I don't have access to this link, could you include some documents to
this descriptor format in comment or commit message? IIRC, it's only the
type be "VMFSSPARSE", right?
What version of ESX has this format?
> With these patches, vmfsSparse files only work if opened through the
> descriptor file. Data files without descriptor files, as far as I
> could understand, are not supported by ESX.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> block/vmdk.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++-----
> 1 file changed, 46 insertions(+), 5 deletions(-)
>
> diff --git a/block/vmdk.c b/block/vmdk.c
> index b16d509..eaf484a 100644
> --- a/block/vmdk.c
> +++ b/block/vmdk.c
> @@ -505,6 +505,34 @@ static int vmdk_open_vmdk3(BlockDriverState *bs,
> return ret;
> }
>
> +static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
> + BlockDriverState *file,
> + int flags)
> +{
> + int ret;
> + uint32_t magic;
> + VMDK3Header header;
> + VmdkExtent *extent;
> +
> + ret = bdrv_pread(file, sizeof(magic), &header, sizeof(header));
> + if (ret < 0) {
> + return ret;
> + }
> + extent = vmdk_add_extent(bs, file, false,
> + le64_to_cpu(header.disk_sectors),
> + le64_to_cpu(header.l1dir_offset) << 9,
> + 0,
> + le64_to_cpu(header.l1dir_size) * 4,
> + 4096,
> + le64_to_cpu(header.granularity)); /* always 512 */
This needs to be rebased, vmdk_add_extent() signature has been changed
in:
commit 8aa1331c09a9b899f48d97f097bb49b7d458be1c
Author: Fam Zheng <famz@redhat.com>
Date: Tue Aug 6 15:44:51 2013 +0800
vmdk: check granularity field in opening
Granularity is used to calculate the cluster size and allocate r/w
buffer. Check the value from image before using it, so we don't abort()
for unbounded memory allocation.
Signed-off-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Since the new function is a variant of vmdk_open_vmdk3(), would you
consider doing a tiny refactor and reduce duplication? And l1dir_size
and granularity need to be checked, as in vmdk_open_vmdk4().
> + ret = vmdk_init_tables(bs, extent);
> + if (ret) {
> + /* free extent allocated by vmdk_add_extent */
> + vmdk_free_last_extent(bs);
> + }
> + return ret;
> +}
> +
> static int vmdk_open_desc_file(BlockDriverState *bs, int flags,
> uint64_t desc_offset);
>
> @@ -663,7 +691,7 @@ static int vmdk_parse_description(const char *desc, const char *opt_name,
> /* Open an extent file and append to bs array */
> static int vmdk_open_sparse(BlockDriverState *bs,
> BlockDriverState *file,
> - int flags)
> + int flags, bool vmfs_sparse)
> {
> uint32_t magic;
>
> @@ -674,7 +702,11 @@ static int vmdk_open_sparse(BlockDriverState *bs,
> magic = be32_to_cpu(magic);
> switch (magic) {
> case VMDK3_MAGIC:
> - return vmdk_open_vmdk3(bs, file, flags);
> + if (vmfs_sparse) {
> + return vmdk_open_vmfs_sparse(bs, file, flags);
> + } else {
> + return vmdk_open_vmdk3(bs, file, flags);
> + }
> break;
> case VMDK4_MAGIC:
> return vmdk_open_vmdk4(bs, file, flags);
> @@ -718,7 +750,8 @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
> }
>
> if (sectors <= 0 ||
> - (strcmp(type, "FLAT") && strcmp(type, "SPARSE")) ||
> + (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
> + strcmp(type, "VMFSSPARSE")) ||
> (strcmp(access, "RW"))) {
> goto next_line;
> }
> @@ -743,7 +776,14 @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
> extent->flat_start_offset = flat_offset << 9;
> } else if (!strcmp(type, "SPARSE")) {
> /* SPARSE extent */
> - ret = vmdk_open_sparse(bs, extent_file, bs->open_flags);
> + ret = vmdk_open_sparse(bs, extent_file, bs->open_flags, false);
> + if (ret) {
> + bdrv_delete(extent_file);
> + return ret;
> + }
> + } else if (!strcmp(type, "VMFSSPARSE")) {
> + /* VMFSSPARSE extent */
> + ret = vmdk_open_sparse(bs, extent_file, bs->open_flags, true);
> if (ret) {
> bdrv_delete(extent_file);
> return ret;
> @@ -789,6 +829,7 @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags,
> goto exit;
> }
> if (strcmp(ct, "monolithicFlat") &&
> + strcmp(ct, "vmfsSparse") &&
> strcmp(ct, "twoGbMaxExtentSparse") &&
> strcmp(ct, "twoGbMaxExtentFlat")) {
> fprintf(stderr,
> @@ -808,7 +849,7 @@ static int vmdk_open(BlockDriverState *bs, QDict *options, int flags)
> int ret;
> BDRVVmdkState *s = bs->opaque;
>
> - if (vmdk_open_sparse(bs, bs->file, flags) == 0) {
> + if (vmdk_open_sparse(bs, bs->file, flags, false) == 0) {
> s->desc_offset = 0x200;
> } else {
> ret = vmdk_open_desc_file(bs, flags, 0);
> --
> 1.8.3.1
>
>
--
Fam
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files
2013-08-12 1:31 ` Fam Zheng
@ 2013-08-12 7:40 ` Paolo Bonzini
2013-08-12 9:10 ` Fam Zheng
0 siblings, 1 reply; 9+ messages in thread
From: Paolo Bonzini @ 2013-08-12 7:40 UTC (permalink / raw)
To: famz; +Cc: kwolf, qemu-devel
----- Original Message -----
> From: "Fam Zheng" <famz@redhat.com>
> To: "Paolo Bonzini" <pbonzini@redhat.com>
> Cc: qemu-devel@nongnu.org, kwolf@redhat.com
> Sent: Monday, August 12, 2013 3:31:44 AM
> Subject: Re: [PATCH 1/2] vmdk: support vmfsSparse files
>
> On Sun, 08/11 18:13, Paolo Bonzini wrote:
> > VMware ESX hosts use a variant of the VMDK3 format, identified by the
> > vmfsSparse create type ad the VMFSSPARSE extent type.
> >
> > It has 16 KB grain tables (L2) and a variable-size grain directory (L1).
> > In addition, the grain size is always 512, but that is not a problem
> > because it is included in the header.
> >
> > The format of the extents is documented in the VMDK spec. The format
> > of the descriptor file is not documented precisely, but it can be
> > found at http://kb.vmware.com/kb/10026353 (Recreating a missing virtual
> > machine disk (VMDK) descriptor file for delta disks).
> >
> I don't have access to this link
It only works from Google. Try Googling for "vmfssparse delta".
> , could you include some documents to
> this descriptor format in comment or commit message? IIRC, it's only the
> type be "VMFSSPARSE", right?
Yes. And "VMFS" for the base (patch 2).
> What version of ESX has this format?
At least 4.0 and newer.
> This needs to be rebased, vmdk_add_extent() signature has been changed
> in:
>
> commit 8aa1331c09a9b899f48d97f097bb49b7d458be1c
> Author: Fam Zheng <famz@redhat.com>
> Date: Tue Aug 6 15:44:51 2013 +0800
>
> vmdk: check granularity field in opening
>
> Granularity is used to calculate the cluster size and allocate r/w
> buffer. Check the value from image before using it, so we don't
> abort()
> for unbounded memory allocation.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>
> Since the new function is a variant of vmdk_open_vmdk3(), would you
> consider doing a tiny refactor and reduce duplication? And l1dir_size
> and granularity need to be checked, as in vmdk_open_vmdk4().
I am not sure how to refactor it... in fact, since I'm on vacation I
wouldn't mind if somebody else fixes the patch. I can test it either
tomorrow or next week.
Paolo
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files
2013-08-12 7:40 ` Paolo Bonzini
@ 2013-08-12 9:10 ` Fam Zheng
0 siblings, 0 replies; 9+ messages in thread
From: Fam Zheng @ 2013-08-12 9:10 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kwolf, qemu-devel
On Mon, 08/12 03:40, Paolo Bonzini wrote:
>
>
> ----- Original Message -----
> > From: "Fam Zheng" <famz@redhat.com>
> > To: "Paolo Bonzini" <pbonzini@redhat.com>
> > Cc: qemu-devel@nongnu.org, kwolf@redhat.com
> > Sent: Monday, August 12, 2013 3:31:44 AM
> > Subject: Re: [PATCH 1/2] vmdk: support vmfsSparse files
> >
> > On Sun, 08/11 18:13, Paolo Bonzini wrote:
> > > VMware ESX hosts use a variant of the VMDK3 format, identified by the
> > > vmfsSparse create type ad the VMFSSPARSE extent type.
> > >
> > > It has 16 KB grain tables (L2) and a variable-size grain directory (L1).
> > > In addition, the grain size is always 512, but that is not a problem
> > > because it is included in the header.
> > >
> > > The format of the extents is documented in the VMDK spec. The format
> > > of the descriptor file is not documented precisely, but it can be
> > > found at http://kb.vmware.com/kb/10026353 (Recreating a missing virtual
> > > machine disk (VMDK) descriptor file for delta disks).
> > >
> > I don't have access to this link
>
> It only works from Google. Try Googling for "vmfssparse delta".
>
> > , could you include some documents to
> > this descriptor format in comment or commit message? IIRC, it's only the
> > type be "VMFSSPARSE", right?
>
> Yes. And "VMFS" for the base (patch 2).
>
> > What version of ESX has this format?
>
> At least 4.0 and newer.
>
> > This needs to be rebased, vmdk_add_extent() signature has been changed
> > in:
> >
> > commit 8aa1331c09a9b899f48d97f097bb49b7d458be1c
> > Author: Fam Zheng <famz@redhat.com>
> > Date: Tue Aug 6 15:44:51 2013 +0800
> >
> > vmdk: check granularity field in opening
> >
> > Granularity is used to calculate the cluster size and allocate r/w
> > buffer. Check the value from image before using it, so we don't
> > abort()
> > for unbounded memory allocation.
> >
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> >
> > Since the new function is a variant of vmdk_open_vmdk3(), would you
> > consider doing a tiny refactor and reduce duplication? And l1dir_size
> > and granularity need to be checked, as in vmdk_open_vmdk4().
>
> I am not sure how to refactor it... in fact, since I'm on vacation I
> wouldn't mind if somebody else fixes the patch. I can test it either
> tomorrow or next week.
>
OK, I'll take it for your happy vocation. :)
--
Fam
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files
2013-08-11 16:13 ` [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files Paolo Bonzini
2013-08-12 1:31 ` Fam Zheng
@ 2013-08-12 8:39 ` Andreas Färber
2013-08-12 11:28 ` Stefan Hajnoczi
2 siblings, 0 replies; 9+ messages in thread
From: Andreas Färber @ 2013-08-12 8:39 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kwolf, famz, qemu-devel
Am 11.08.2013 18:13, schrieb Paolo Bonzini:
> VMware ESX hosts use a variant of the VMDK3 format, identified by the
> vmfsSparse create type ad the VMFSSPARSE extent type.
"and"?
> It has 16 KB grain tables (L2) and a variable-size grain directory (L1).
> In addition, the grain size is always 512, but that is not a problem
> because it is included in the header.
>
> The format of the extents is documented in the VMDK spec. The format
> of the descriptor file is not documented precisely, but it can be
> found at http://kb.vmware.com/kb/10026353 (Recreating a missing virtual
> machine disk (VMDK) descriptor file for delta disks).
>
> With these patches, vmfsSparse files only work if opened through the
> descriptor file. Data files without descriptor files, as far as I
> could understand, are not supported by ESX.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> block/vmdk.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++-----
> 1 file changed, 46 insertions(+), 5 deletions(-)
>
> diff --git a/block/vmdk.c b/block/vmdk.c
> index b16d509..eaf484a 100644
> --- a/block/vmdk.c
> +++ b/block/vmdk.c
> @@ -505,6 +505,34 @@ static int vmdk_open_vmdk3(BlockDriverState *bs,
> return ret;
> }
>
> +static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
> + BlockDriverState *file,
> + int flags)
> +{
> + int ret;
> + uint32_t magic;
> + VMDK3Header header;
> + VmdkExtent *extent;
> +
> + ret = bdrv_pread(file, sizeof(magic), &header, sizeof(header));
> + if (ret < 0) {
> + return ret;
> + }
> + extent = vmdk_add_extent(bs, file, false,
> + le64_to_cpu(header.disk_sectors),
> + le64_to_cpu(header.l1dir_offset) << 9,
> + 0,
> + le64_to_cpu(header.l1dir_size) * 4,
> + 4096,
> + le64_to_cpu(header.granularity)); /* always 512 */
Indentation is 3 off (guessing the variable was called ext before ;)).
Andreas
> + ret = vmdk_init_tables(bs, extent);
> + if (ret) {
> + /* free extent allocated by vmdk_add_extent */
> + vmdk_free_last_extent(bs);
> + }
> + return ret;
> +}
> +
> static int vmdk_open_desc_file(BlockDriverState *bs, int flags,
> uint64_t desc_offset);
>
> @@ -663,7 +691,7 @@ static int vmdk_parse_description(const char *desc, const char *opt_name,
> /* Open an extent file and append to bs array */
> static int vmdk_open_sparse(BlockDriverState *bs,
> BlockDriverState *file,
> - int flags)
> + int flags, bool vmfs_sparse)
> {
> uint32_t magic;
>
> @@ -674,7 +702,11 @@ static int vmdk_open_sparse(BlockDriverState *bs,
> magic = be32_to_cpu(magic);
> switch (magic) {
> case VMDK3_MAGIC:
> - return vmdk_open_vmdk3(bs, file, flags);
> + if (vmfs_sparse) {
> + return vmdk_open_vmfs_sparse(bs, file, flags);
> + } else {
> + return vmdk_open_vmdk3(bs, file, flags);
> + }
> break;
> case VMDK4_MAGIC:
> return vmdk_open_vmdk4(bs, file, flags);
> @@ -718,7 +750,8 @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
> }
>
> if (sectors <= 0 ||
> - (strcmp(type, "FLAT") && strcmp(type, "SPARSE")) ||
> + (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
> + strcmp(type, "VMFSSPARSE")) ||
> (strcmp(access, "RW"))) {
> goto next_line;
> }
> @@ -743,7 +776,14 @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
> extent->flat_start_offset = flat_offset << 9;
> } else if (!strcmp(type, "SPARSE")) {
> /* SPARSE extent */
> - ret = vmdk_open_sparse(bs, extent_file, bs->open_flags);
> + ret = vmdk_open_sparse(bs, extent_file, bs->open_flags, false);
> + if (ret) {
> + bdrv_delete(extent_file);
> + return ret;
> + }
> + } else if (!strcmp(type, "VMFSSPARSE")) {
> + /* VMFSSPARSE extent */
> + ret = vmdk_open_sparse(bs, extent_file, bs->open_flags, true);
> if (ret) {
> bdrv_delete(extent_file);
> return ret;
> @@ -789,6 +829,7 @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags,
> goto exit;
> }
> if (strcmp(ct, "monolithicFlat") &&
> + strcmp(ct, "vmfsSparse") &&
> strcmp(ct, "twoGbMaxExtentSparse") &&
> strcmp(ct, "twoGbMaxExtentFlat")) {
> fprintf(stderr,
> @@ -808,7 +849,7 @@ static int vmdk_open(BlockDriverState *bs, QDict *options, int flags)
> int ret;
> BDRVVmdkState *s = bs->opaque;
>
> - if (vmdk_open_sparse(bs, bs->file, flags) == 0) {
> + if (vmdk_open_sparse(bs, bs->file, flags, false) == 0) {
> s->desc_offset = 0x200;
> } else {
> ret = vmdk_open_desc_file(bs, flags, 0);
>
--
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files
2013-08-11 16:13 ` [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files Paolo Bonzini
2013-08-12 1:31 ` Fam Zheng
2013-08-12 8:39 ` Andreas Färber
@ 2013-08-12 11:28 ` Stefan Hajnoczi
2013-08-12 11:45 ` Stefan Hajnoczi
2 siblings, 1 reply; 9+ messages in thread
From: Stefan Hajnoczi @ 2013-08-12 11:28 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kwolf, famz, qemu-devel
On Sun, Aug 11, 2013 at 06:13:57PM +0200, Paolo Bonzini wrote:
> @@ -505,6 +505,34 @@ static int vmdk_open_vmdk3(BlockDriverState *bs,
> return ret;
> }
>
> +static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
> + BlockDriverState *file,
> + int flags)
> +{
> + int ret;
> + uint32_t magic;
> + VMDK3Header header;
> + VmdkExtent *extent;
> +
> + ret = bdrv_pread(file, sizeof(magic), &header, sizeof(header));
> + if (ret < 0) {
> + return ret;
> + }
magic is unused. Did you forget to check it?
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [Qemu-devel] [PATCH 1/2] vmdk: support vmfsSparse files
2013-08-12 11:28 ` Stefan Hajnoczi
@ 2013-08-12 11:45 ` Stefan Hajnoczi
0 siblings, 0 replies; 9+ messages in thread
From: Stefan Hajnoczi @ 2013-08-12 11:45 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Kevin Wolf, Fam Zheng, qemu-devel
On Mon, Aug 12, 2013 at 1:28 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Sun, Aug 11, 2013 at 06:13:57PM +0200, Paolo Bonzini wrote:
>> @@ -505,6 +505,34 @@ static int vmdk_open_vmdk3(BlockDriverState *bs,
>> return ret;
>> }
>>
>> +static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
>> + BlockDriverState *file,
>> + int flags)
>> +{
>> + int ret;
>> + uint32_t magic;
>> + VMDK3Header header;
>> + VmdkExtent *extent;
>> +
>> + ret = bdrv_pread(file, sizeof(magic), &header, sizeof(header));
>> + if (ret < 0) {
>> + return ret;
>> + }
>
> magic is unused. Did you forget to check it?
Andreas Färber pointed out that we're only using the sizeof(magic),
not the actual variable.
Feel free to ignore my comment, at least the code documents that the
uint32_t offset belongs to the magic number before the header.
Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread