[Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Jun Li]

Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
the code as followings:
when create qcow2 format image:
qcow2_create2
  bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
  file(e.g: file1) on rbd server.
  ...
  ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
  qcow2 header into file1 due to the file1's length is 0. Seems
  qemu_rbd_aio_writev can not write beyond EOF.
  ...

As above analysis, there are two methods to solve the above bz as followings:
1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).
2, When write the qcow2 header into file1, just let qemu_rbd_aio_writev can
enlarge the file1. So should add qemu_rbd_truncate inside qemu_rbd_aio_writev.
qemu_rbd_truncate will call rbd_resize, but seems rbd_resize is
synchronous function. If so, when do bdrv_pwrite, guest will hang. This is not
our expected.

For method 1, maybe it's not corresponding to the original principle of qcow2.
Yes, it's very easy to solve the above bz. Nevertheless, I just want to use
method 2 to solve above issue.

For method 2, could anyone give some suggestions on howto realize a
asynchronous rbd_resize. Thanks.

Above analysis also based on stefan's hints. Thanks.

Best Regards,
Jun Li

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Max Reitz]

On 2014-12-05 at 16:32, Jun Li wrote:
> Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
> the code as followings:
> when create qcow2 format image:
> qcow2_create2
>    bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
>    file(e.g: file1) on rbd server.
>    ...
>    ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
>    qcow2 header into file1 due to the file1's length is 0. Seems
>    qemu_rbd_aio_writev can not write beyond EOF.
>    ...
>
> As above analysis, there are two methods to solve the above bz as followings:
> 1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).

Should be possible by using -o preallocation=falloc or -o 
preallocation=full.

I can't say a lot about making rbd growable because I know near to 
nothing about rbd; but there are protocols which really simply don't 
support writes beyond the end of file, and where that's intended (for 
instance, while nbd somehow does support it when using the qemu nbd 
server, normally (strictly according to the protocol) it does not); so 
for these protocols, you have to use a preallocated image file or an 
image format which does not grow on writes (such as raw).

Of course, while that may be a solution for nbd, it doesn't sound like a 
good solution for rbd, so writes beyond the EOF should probably be 
supported there (although once again, I don't know rbd well enough to 
judge that).

Max

> 2, When write the qcow2 header into file1, just let qemu_rbd_aio_writev can
> enlarge the file1. So should add qemu_rbd_truncate inside qemu_rbd_aio_writev.
> qemu_rbd_truncate will call rbd_resize, but seems rbd_resize is
> synchronous function. If so, when do bdrv_pwrite, guest will hang.This is not
> our expected.
>
> For method 1, maybe it's not corresponding to the original principle of qcow2.
> Yes, it's very easy to solve the above bz. Nevertheless, I just want to use
> method 2 to solve above issue.
>
> For method 2, could anyone give some suggestions on howto realize a
> asynchronous rbd_resize. Thanks.
>
> Above analysis also based on stefan's hints. Thanks.
>
> Best Regards,
> Jun Li

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Jun Li]

On Fri, 12/05 18:01, Max Reitz wrote:
> On 2014-12-05 at 16:32, Jun Li wrote:
> >Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
> >the code as followings:
> >when create qcow2 format image:
> >qcow2_create2
> >   bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
> >   file(e.g: file1) on rbd server.
> >   ...
> >   ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
> >   qcow2 header into file1 due to the file1's length is 0. Seems
> >   qemu_rbd_aio_writev can not write beyond EOF.
> >   ...
> >
> >As above analysis, there are two methods to solve the above bz as followings:
> >1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).
> 
> Should be possible by using -o preallocation=falloc or -o
> preallocation=full.

Sure. If bdrv_create_file(filename, opts, &local_err) create a fixed-size(not
0 size) just as using "preallocation=falloc or preallocation=full", it will
create a fixed-size file on rbd server. So it won't exist above issue.

> 
> I can't say a lot about making rbd growable because I know near to nothing
> about rbd; but there are protocols which really simply don't support writes
> beyond the end of file, and where that's intended (for instance, while nbd
> somehow does support it when using the qemu nbd server, normally (strictly
> according to the protocol) it does not); so for these protocols, you have to
> use a preallocated image file or an image format which does not grow on
> writes (such as raw).
> 

Here just want to use rbd_resize to realize rbd growable.

> Of course, while that may be a solution for nbd, it doesn't sound like a
> good solution for rbd, so writes beyond the EOF should probably be supported
> there (although once again, I don't know rbd well enough to judge that).
> 

Yes, you are right. Also talked with stefan. Here just want to ask Josh Durgin
whether it has other solutions or rbd can support asynchronous rbd_resize.

Regards,
Jun Li

> 
> >2, When write the qcow2 header into file1, just let qemu_rbd_aio_writev can
> >enlarge the file1. So should add qemu_rbd_truncate inside qemu_rbd_aio_writev.
> >qemu_rbd_truncate will call rbd_resize, but seems rbd_resize is
> >synchronous function. If so, when do bdrv_pwrite, guest will hang.This is not
> >our expected.
> >
> >For method 1, maybe it's not corresponding to the original principle of qcow2.
> >Yes, it's very easy to solve the above bz. Nevertheless, I just want to use
> >method 2 to solve above issue.
> >
> >For method 2, could anyone give some suggestions on howto realize a
> >asynchronous rbd_resize. Thanks.
> >
> >Above analysis also based on stefan's hints. Thanks.
> >
> >Best Regards,
> >Jun Li
> 

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Jun Li]

On Fri, 12/05 18:01, Max Reitz wrote:
> On 2014-12-05 at 16:32, Jun Li wrote:
> >Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
> >the code as followings:
> >when create qcow2 format image:
> >qcow2_create2
> >   bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
> >   file(e.g: file1) on rbd server.
> >   ...
> >   ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
> >   qcow2 header into file1 due to the file1's length is 0. Seems
> >   qemu_rbd_aio_writev can not write beyond EOF.
> >   ...
> >
> >As above analysis, there are two methods to solve the above bz as followings:
> >1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).
> 
> Should be possible by using -o preallocation=falloc or -o
> preallocation=full.

Although "-o preallocation=falloc or -o preallocation=full" can create a qcow2
format image successfully, but when perform "qemu-img resize file.qcow2
+500M", then use the extend 500M disk image still hit the same issue(as rbd
block driver does not support growable file).

So use "-o preallocation=falloc or -o preallocation=full", it not a good
solution.

Regards,
Jun Li

> 
> I can't say a lot about making rbd growable because I know near to nothing
> about rbd; but there are protocols which really simply don't support writes
> beyond the end of file, and where that's intended (for instance, while nbd
> somehow does support it when using the qemu nbd server, normally (strictly
> according to the protocol) it does not); so for these protocols, you have to
> use a preallocated image file or an image format which does not grow on
> writes (such as raw).
> 
> Of course, while that may be a solution for nbd, it doesn't sound like a
> good solution for rbd, so writes beyond the EOF should probably be supported
> there (although once again, I don't know rbd well enough to judge that).
> 
> Max
> 
> >2, When write the qcow2 header into file1, just let qemu_rbd_aio_writev can
> >enlarge the file1. So should add qemu_rbd_truncate inside qemu_rbd_aio_writev.
> >qemu_rbd_truncate will call rbd_resize, but seems rbd_resize is
> >synchronous function. If so, when do bdrv_pwrite, guest will hang.This is not
> >our expected.
> >
> >For method 1, maybe it's not corresponding to the original principle of qcow2.
> >Yes, it's very easy to solve the above bz. Nevertheless, I just want to use
> >method 2 to solve above issue.
> >
> >For method 2, could anyone give some suggestions on howto realize a
> >asynchronous rbd_resize. Thanks.
> >
> >Above analysis also based on stefan's hints. Thanks.
> >
> >Best Regards,
> >Jun Li
> 

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Josh Durgin]

On 12/08/2014 05:58 AM, Jun Li wrote:
> On Fri, 12/05 18:01, Max Reitz wrote:
>> On 2014-12-05 at 16:32, Jun Li wrote:
>>> Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
>>> the code as followings:
>>> when create qcow2 format image:
>>> qcow2_create2
>>>    bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
>>>    file(e.g: file1) on rbd server.
>>>    ...
>>>    ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
>>>    qcow2 header into file1 due to the file1's length is 0. Seems
>>>    qemu_rbd_aio_writev can not write beyond EOF.
>>>    ...
>>>
>>> As above analysis, there are two methods to solve the above bz as followings:
>>> 1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).
>>
>> Should be possible by using -o preallocation=falloc or -o
>> preallocation=full.
>
> Although "-o preallocation=falloc or -o preallocation=full" can create a qcow2
> format image successfully, but when perform "qemu-img resize file.qcow2
> +500M", then use the extend 500M disk image still hit the same issue(as rbd
> block driver does not support growable file).

Why not use 'rbd resize' and raw images instead?
rbd already supports snapshots, cloning, thin provisioning, and
differential backup natively, so putting qcow2 on top tends to just add
overhead.

Josh

> So use "-o preallocation=falloc or -o preallocation=full", it not a good
> solution.
>
> Regards,
> Jun Li
>
>>
>> I can't say a lot about making rbd growable because I know near to nothing
>> about rbd; but there are protocols which really simply don't support writes
>> beyond the end of file, and where that's intended (for instance, while nbd
>> somehow does support it when using the qemu nbd server, normally (strictly
>> according to the protocol) it does not); so for these protocols, you have to
>> use a preallocated image file or an image format which does not grow on
>> writes (such as raw).
>>
>> Of course, while that may be a solution for nbd, it doesn't sound like a
>> good solution for rbd, so writes beyond the EOF should probably be supported
>> there (although once again, I don't know rbd well enough to judge that).
>>
>> Max
>>
>>> 2, When write the qcow2 header into file1, just let qemu_rbd_aio_writev can
>>> enlarge the file1. So should add qemu_rbd_truncate inside qemu_rbd_aio_writev.
>>> qemu_rbd_truncate will call rbd_resize, but seems rbd_resize is
>>> synchronous function. If so, when do bdrv_pwrite, guest will hang.This is not
>>> our expected.
>>>
>>> For method 1, maybe it's not corresponding to the original principle of qcow2.
>>> Yes, it's very easy to solve the above bz. Nevertheless, I just want to use
>>> method 2 to solve above issue.
>>>
>>> For method 2, could anyone give some suggestions on howto realize a
>>> asynchronous rbd_resize. Thanks.
>>>
>>> Above analysis also based on stefan's hints. Thanks.
>>>
>>> Best Regards,
>>> Jun Li
>>
>

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Stefan Hajnoczi]

On Tue, Dec 9, 2014 at 3:52 AM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 12/08/2014 05:58 AM, Jun Li wrote:
>>
>> On Fri, 12/05 18:01, Max Reitz wrote:
>>>
>>> On 2014-12-05 at 16:32, Jun Li wrote:
>>>>
>>>> Currently, qemu-img can not create qcow2 image format on rbd server.
>>>> Analysis
>>>> the code as followings:
>>>> when create qcow2 format image:
>>>> qcow2_create2
>>>>    bdrv_create_file(filename, opts, &local_err);  --> Here will create a
>>>> 0 size
>>>>    file(e.g: file1) on rbd server.
>>>>    ...
>>>>    ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not
>>>> write
>>>>    qcow2 header into file1 due to the file1's length is 0. Seems
>>>>    qemu_rbd_aio_writev can not write beyond EOF.
>>>>    ...
>>>>
>>>> As above analysis, there are two methods to solve the above bz as
>>>> followings:
>>>> 1, When create file1, just create a fixed-size file1 on rbd server(not 0
>>>> size).
>>>
>>>
>>> Should be possible by using -o preallocation=falloc or -o
>>> preallocation=full.
>>
>>
>> Although "-o preallocation=falloc or -o preallocation=full" can create a
>> qcow2
>> format image successfully, but when perform "qemu-img resize file.qcow2
>> +500M", then use the extend 500M disk image still hit the same issue(as
>> rbd
>> block driver does not support growable file).
>
>
> Why not use 'rbd resize' and raw images instead?
> rbd already supports snapshots, cloning, thin provisioning, and
> differential backup natively, so putting qcow2 on top tends to just add
> overhead.

In general, I don't expect many people to use qcow2 on rbd either.

However, qcow2 does work on top of iSCSI, LVM, or host block devices
and some users actually use this feature (even though there is LVM
thin provisioning, for example).  So the failure on rbd is unexpected
and inconsistent.

The way it works on other protocols is that the user must first create
an adequately sized volume before running qemu-img create.  With rbd
this doesn't work because we truncate the volume to 0 bytes during
create.

Either we need to fix this (without losing the ability to qemu-img
create -f raw rbd:... 10G) or we should have a clear error message.

The simplest way would just be to detect rbd create with size 0 and
print a clear error message like "image formats that grow on demand
are not supported on rbd".

Stefan

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Josh Durgin]

On 12/09/2014 01:49 AM, Stefan Hajnoczi wrote:
> On Tue, Dec 9, 2014 at 3:52 AM, Josh Durgin <josh.durgin@inktank.com> wrote:
>> On 12/08/2014 05:58 AM, Jun Li wrote:
>>>
>>> On Fri, 12/05 18:01, Max Reitz wrote:
>>>>
>>>> On 2014-12-05 at 16:32, Jun Li wrote:
>>>>>
>>>>> Currently, qemu-img can not create qcow2 image format on rbd server.
>>>>> Analysis
>>>>> the code as followings:
>>>>> when create qcow2 format image:
>>>>> qcow2_create2
>>>>>     bdrv_create_file(filename, opts, &local_err);  --> Here will create a
>>>>> 0 size
>>>>>     file(e.g: file1) on rbd server.
>>>>>     ...
>>>>>     ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not
>>>>> write
>>>>>     qcow2 header into file1 due to the file1's length is 0. Seems
>>>>>     qemu_rbd_aio_writev can not write beyond EOF.
>>>>>     ...
>>>>>
>>>>> As above analysis, there are two methods to solve the above bz as
>>>>> followings:
>>>>> 1, When create file1, just create a fixed-size file1 on rbd server(not 0
>>>>> size).
>>>>
>>>>
>>>> Should be possible by using -o preallocation=falloc or -o
>>>> preallocation=full.
>>>
>>>
>>> Although "-o preallocation=falloc or -o preallocation=full" can create a
>>> qcow2
>>> format image successfully, but when perform "qemu-img resize file.qcow2
>>> +500M", then use the extend 500M disk image still hit the same issue(as
>>> rbd
>>> block driver does not support growable file).
>>
>>
>> Why not use 'rbd resize' and raw images instead?
>> rbd already supports snapshots, cloning, thin provisioning, and
>> differential backup natively, so putting qcow2 on top tends to just add
>> overhead.
>
> In general, I don't expect many people to use qcow2 on rbd either.
>
> However, qcow2 does work on top of iSCSI, LVM, or host block devices
> and some users actually use this feature (even though there is LVM
> thin provisioning, for example).  So the failure on rbd is unexpected
> and inconsistent.
>
> The way it works on other protocols is that the user must first create
> an adequately sized volume before running qemu-img create.  With rbd
> this doesn't work because we truncate the volume to 0 bytes during
> create.
>
> Either we need to fix this (without losing the ability to qemu-img
> create -f raw rbd:... 10G) or we should have a clear error message.
>
> The simplest way would just be to detect rbd create with size 0 and
> print a clear error message like "image formats that grow on demand
> are not supported on rbd".

Either way sounds fine to me. I don't think it's worth adding the
complexity of dynamic growth to the rbd driver when it can be solved
simply at higher levels.

Josh

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Jun Li]

On Tue, 12/09 09:49, Stefan Hajnoczi wrote:
> On Tue, Dec 9, 2014 at 3:52 AM, Josh Durgin <josh.durgin@inktank.com> wrote:
> > On 12/08/2014 05:58 AM, Jun Li wrote:
> >>
> >> On Fri, 12/05 18:01, Max Reitz wrote:
> >>>
> >>> On 2014-12-05 at 16:32, Jun Li wrote:
> >>>>
> >>>> Currently, qemu-img can not create qcow2 image format on rbd server.
> >>>> Analysis
> >>>> the code as followings:
> >>>> when create qcow2 format image:
> >>>> qcow2_create2
> >>>>    bdrv_create_file(filename, opts, &local_err);  --> Here will create a
> >>>> 0 size
> >>>>    file(e.g: file1) on rbd server.
> >>>>    ...
> >>>>    ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not
> >>>> write
> >>>>    qcow2 header into file1 due to the file1's length is 0. Seems
> >>>>    qemu_rbd_aio_writev can not write beyond EOF.
> >>>>    ...
> >>>>
> >>>> As above analysis, there are two methods to solve the above bz as
> >>>> followings:
> >>>> 1, When create file1, just create a fixed-size file1 on rbd server(not 0
> >>>> size).
> >>>
> >>>
> >>> Should be possible by using -o preallocation=falloc or -o
> >>> preallocation=full.
> >>
> >>
> >> Although "-o preallocation=falloc or -o preallocation=full" can create a
> >> qcow2
> >> format image successfully, but when perform "qemu-img resize file.qcow2
> >> +500M", then use the extend 500M disk image still hit the same issue(as
> >> rbd
> >> block driver does not support growable file).
> >
> >
> > Why not use 'rbd resize' and raw images instead?
> > rbd already supports snapshots, cloning, thin provisioning, and
> > differential backup natively, so putting qcow2 on top tends to just add
> > overhead.
> 
> In general, I don't expect many people to use qcow2 on rbd either.
> 
> However, qcow2 does work on top of iSCSI, LVM, or host block devices
> and some users actually use this feature (even though there is LVM
> thin provisioning, for example).  So the failure on rbd is unexpected
> and inconsistent.
> 
> The way it works on other protocols is that the user must first create
> an adequately sized volume before running qemu-img create.  With rbd
> this doesn't work because we truncate the volume to 0 bytes during
> create.
> 
> Either we need to fix this (without losing the ability to qemu-img
> create -f raw rbd:... 10G) or we should have a clear error message.

what you do mean by "fix this"? Realize a growable rbd block driver? If not,
seems we should have a clear error message currently.

I have attempted to realize a asynchronous rbd_resize at qemu level(create a
child-thread to perform rbd_resize in qemu_rbd_aio_writev), but it failed due
to the librbd. Gdb debug info just like followings:
(gdb) bt
#0  librbd::resize (ictx=0x48fffe948fe80574, size=120095921237475840,
				prog_ctx=...) at librbd/internal.cc:1492
#1  0x00007ffff7478bb2 in rbd_resize (image=<optimized out>, size=<optimized
				out>) at librbd/librbd.cc:726
#2  0x00005555555cfa9c in qemu_rbd_truncate (bs=0x7ffff7f9ca00,
				offset=120095921237475840) at block/rbd.c:798
#3  0x00005555555cf797 in thread_info (t_parameters=0x7ffff7f9c9d0) at
block/rbd.c:706
#4  0x00007ffff1c5bee5 in start_thread (arg=0x7fffd5ffb700) at
pthread_create.c:309
#5  0x00007ffff198ab8d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Based on above, seems rbd libraries are not thread-safe.

> 
> The simplest way would just be to detect rbd create with size 0 and
> print a clear error message like "image formats that grow on demand
> are not supported on rbd".

If we using this simplest way just give a clear error message, it means
following things:
1, Can create a qcow2 format image just using "-o preallocation=falloc or -o
preallocation=full".
2, Can not resize this qcow2 format image(created by above). In other words,
   even if we have resize this image successfully, can not write any data
   into the extend image.

I will submit a patch just using this simplest way.

Regards,
Jun Li

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Kevin Wolf]

Am 05.12.2014 um 16:32 hat Jun Li geschrieben:
> Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
> the code as followings:
> when create qcow2 format image:
> qcow2_create2
>   bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
>   file(e.g: file1) on rbd server.
>   ...
>   ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
>   qcow2 header into file1 due to the file1's length is 0. Seems
>   qemu_rbd_aio_writev can not write beyond EOF.
>   ...
> 
> As above analysis, there are two methods to solve the above bz as followings:
> 1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).

This is not a solution. Even if you might be able to create an image
successfully, using qcow2 without a backend that allows the image file
to grow is bound to fail sooner or later.

So yes, you'll want to extend the rbd block driver to grow the file
asynchronously when writing beyond EOF.

Kevin

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Jun Li]

On Mon, 12/08 11:07, Kevin Wolf wrote:
> Am 05.12.2014 um 16:32 hat Jun Li geschrieben:
> > Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
> > the code as followings:
> > when create qcow2 format image:
> > qcow2_create2
> >   bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
> >   file(e.g: file1) on rbd server.
> >   ...
> >   ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
> >   qcow2 header into file1 due to the file1's length is 0. Seems
> >   qemu_rbd_aio_writev can not write beyond EOF.
> >   ...
> > 
> > As above analysis, there are two methods to solve the above bz as followings:
> > 1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).
> 
> This is not a solution. Even if you might be able to create an image
> successfully, using qcow2 without a backend that allows the image file
> to grow is bound to fail sooner or later.
> 
> So yes, you'll want to extend the rbd block driver to grow the file
> asynchronously when writing beyond EOF.
> 

Hi Kevin,

  Although rbd_resize is synchronous, I just want to create a new child-thread
to realize asynchronous(Maybe just like userspace aio: libaio) rbd_resize.
Just like following:

qemu_rbd_aio_writev()
{
    if (BlockDriverState->file->growable == 1) {
        pthread_create(&thread_id, NULL, child_thread, NULL);
    }

    rbd_start_aio();
}

child_thread()
{
    ...
    rbd_resize();
    rbd_start_aio();
    ...
}

Currently, seems do not have original asynchronous rbd_resize. Besides, rbd block
driver do not support growable file. So I want to use above method to realize
asynchronous rbd_resize() in our qemu level.

What's your opinion?

Regards,
Jun Li

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Kevin Wolf]

Am 08.12.2014 um 14:50 hat Jun Li geschrieben:
> On Mon, 12/08 11:07, Kevin Wolf wrote:
> > Am 05.12.2014 um 16:32 hat Jun Li geschrieben:
> > > Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
> > > the code as followings:
> > > when create qcow2 format image:
> > > qcow2_create2
> > >   bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
> > >   file(e.g: file1) on rbd server.
> > >   ...
> > >   ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
> > >   qcow2 header into file1 due to the file1's length is 0. Seems
> > >   qemu_rbd_aio_writev can not write beyond EOF.
> > >   ...
> > > 
> > > As above analysis, there are two methods to solve the above bz as followings:
> > > 1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).
> > 
> > This is not a solution. Even if you might be able to create an image
> > successfully, using qcow2 without a backend that allows the image file
> > to grow is bound to fail sooner or later.
> > 
> > So yes, you'll want to extend the rbd block driver to grow the file
> > asynchronously when writing beyond EOF.
> > 
> 
> Hi Kevin,
> 
>   Although rbd_resize is synchronous, I just want to create a new child-thread
> to realize asynchronous(Maybe just like userspace aio: libaio) rbd_resize.
> Just like following:
> 
> qemu_rbd_aio_writev()
> {
>     if (BlockDriverState->file->growable == 1) {
>         pthread_create(&thread_id, NULL, child_thread, NULL);
>     }
> 
>     rbd_start_aio();
> }
> 
> child_thread()
> {
>     ...
>     rbd_resize();
>     rbd_start_aio();
>     ...
> }
> 
> Currently, seems do not have original asynchronous rbd_resize. Besides, rbd block
> driver do not support growable file. So I want to use above method to realize
> asynchronous rbd_resize() in our qemu level.
> 
> What's your opinion?

Are you sure that the rbd libraries are thread-safe and can be used in
this way?

Kevin

Re: [Qemu-devel] qcow2: Can create qcow2 image format on rbd server

[Jun Li]

On Mon, 12/08 15:49, Kevin Wolf wrote:
> Am 08.12.2014 um 14:50 hat Jun Li geschrieben:
> > On Mon, 12/08 11:07, Kevin Wolf wrote:
> > > Am 05.12.2014 um 16:32 hat Jun Li geschrieben:
> > > > Currently, qemu-img can not create qcow2 image format on rbd server. Analysis
> > > > the code as followings:
> > > > when create qcow2 format image:
> > > > qcow2_create2
> > > >   bdrv_create_file(filename, opts, &local_err);  --> Here will create a 0 size
> > > >   file(e.g: file1) on rbd server.
> > > >   ...
> > > >   ret = bdrv_pwrite(bs, 0, header, cluster_size); --> So here can not write
> > > >   qcow2 header into file1 due to the file1's length is 0. Seems
> > > >   qemu_rbd_aio_writev can not write beyond EOF.
> > > >   ...
> > > > 
> > > > As above analysis, there are two methods to solve the above bz as followings:
> > > > 1, When create file1, just create a fixed-size file1 on rbd server(not 0 size).
> > > 
> > > This is not a solution. Even if you might be able to create an image
> > > successfully, using qcow2 without a backend that allows the image file
> > > to grow is bound to fail sooner or later.
> > > 
> > > So yes, you'll want to extend the rbd block driver to grow the file
> > > asynchronously when writing beyond EOF.
> > > 
> > 
> > Hi Kevin,
> > 
> >   Although rbd_resize is synchronous, I just want to create a new child-thread
> > to realize asynchronous(Maybe just like userspace aio: libaio) rbd_resize.
> > Just like following:
> > 
> > qemu_rbd_aio_writev()
> > {
> >     if (BlockDriverState->file->growable == 1) {
> >         pthread_create(&thread_id, NULL, child_thread, NULL);
> >     }
> > 
> >     rbd_start_aio();
> > }
> > 
> > child_thread()
> > {
> >     ...
> >     rbd_resize();
> >     rbd_start_aio();
> >     ...
> > }
> > 
> > Currently, seems do not have original asynchronous rbd_resize. Besides, rbd block
> > driver do not support growable file. So I want to use above method to realize
> > asynchronous rbd_resize() in our qemu level.
> > 
> > What's your opinion?
> 
> Are you sure that the rbd libraries are thread-safe and can be used in
> this way?
> 

I have tested this method, seems it's not thread-safe. Gdb debug info just as
followings:
# gdb --args /opt/qemu-git-arm/bin/qemu-img create -f qcow2
  rbd:qemu-kvm-pool/juli-test44.qcow2:mon_host=$IP 10M
(gdb) bt
#0  librbd::resize (ictx=0x48fffe948fe80574, size=120095921237475840,
				prog_ctx=...) at librbd/internal.cc:1492
#1  0x00007ffff7478bb2 in rbd_resize (image=<optimized out>, size=<optimized
				out>) at librbd/librbd.cc:726
#2  0x00005555555cfa9c in qemu_rbd_truncate (bs=0x7ffff7f9ca00,
				offset=120095921237475840) at block/rbd.c:798
#3  0x00005555555cf797 in thread_info (t_parameters=0x7ffff7f9c9d0) at
block/rbd.c:706
#4  0x00007ffff1c5bee5 in start_thread (arg=0x7fffd5ffb700) at
pthread_create.c:309
#5  0x00007ffff198ab8d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111


Regards,
Jun Li