Offline manipulation of Dirty Bitmaps by qemu-img

Offline manipulation of Dirty Bitmaps by qemu-img

[John Snow]

This has come up in the past, and I believe we discussed this at KVM
Forum, too:

There have been requests from oVirt (via Nir Soffer) to add some offline
bitmap manipulation functionality. In the past, our stance has generally
been "Use QEMU without an accelerator, and use QMP to manipulate the
images."

We like this for a few reasons:

1. It keeps bitmap management code tightly centralized
2. It allows for the full suite of bitmap manipulations in either
offline or online mode with one tool
3. We wouldn't have to write new code.
4. Or design new CLIs and duplicate our existing work.
5. Or write even more tests.

However, tools like oVirt may or may not be fully equipped to launch
QEMU in this context, and there is always a desire for qemu-img to be
able to "do more", so existing management suites could extend
functionality more easily.

(Or so I am imagining.)

I am still leaning heavily against adding any more CLI commands or
options to qemu-img right now. Even if we do add some of the fundamental
ones like "add" or "remove", it seems only a matter of time before we
have to add "clear", "merge", etc. Is this just a race to code duplication?

On the other hand, one of the other suggestions is to have qemu-img
check --repair optionally delete corrupted bitmaps. I kind of like this
idea: it's a buyer beware operation that might make management layers
unhappy, but then again ... repair is always something that could make
things worse.

Plus, if you manage to corrupt bitmaps badly enough that they can't even
be parsed, you might need a heavyweight repair operation.

Nir, do you think that'd be sufficient for your needs for now, or would
you still like to see more granular offline management?

--js

Re: [libvirt] Offline manipulation of Dirty Bitmaps by qemu-img

[Peter Krempa]

On Thu, Dec 05, 2019 at 17:37:11 -0500, John Snow wrote:
> This has come up in the past, and I believe we discussed this at KVM
> Forum, too:
> 
> There have been requests from oVirt (via Nir Soffer) to add some offline
> bitmap manipulation functionality. In the past, our stance has generally
> been "Use QEMU without an accelerator, and use QMP to manipulate the
> images."

This is a thing I wanted to do for a long time but never had time for.
I'm not sure whether that will change though.

We have a workaround for this tough: you can start the VM with CPUs
stopped:

virsh start --pause $VMNAME 

(That translates to VIR_DOMAIN_START_PAUSED flag for
virDomainCreateWithFlags).

You can then use any libvirt API which requires a running VM including
blockjobs checkpoints etc.

The VM then can be destroyed. Since the CPUs didn't run the guest
visible image content was nott modified.

Alternatively to make this slightly more official we could introduce a
new flag for the VM starting API which will actually start the VM in the
no-machine mode, will interlock certain operations such as resuming of
the execution or migration perhaps  and the official purpose will be to
allow complex blockjobs without starting the actual VM.

Since starting an actual VM will be impossible anyways until such a VM
is gone it might be a sane thing to do here.

> We like this for a few reasons:
> 
> 1. It keeps bitmap management code tightly centralized
> 2. It allows for the full suite of bitmap manipulations in either
> offline or online mode with one tool
> 3. We wouldn't have to write new code.
> 4. Or design new CLIs and duplicate our existing work.
> 5. Or write even more tests.

In libvirt we'd like to use it because qemu-img has no reasonable
progress reporting and we could reuse the code we have for interacting
with the jobs when the VM is running.

> However, tools like oVirt may or may not be fully equipped to launch
> QEMU in this context, and there is always a desire for qemu-img to be
> able to "do more", so existing management suites could extend
> functionality more easily.
>
> (Or so I am imagining.)
> 
> I am still leaning heavily against adding any more CLI commands or
> options to qemu-img right now. Even if we do add some of the fundamental
> ones like "add" or "remove", it seems only a matter of time before we
> have to add "clear", "merge", etc. Is this just a race to code duplication?
> 
> On the other hand, one of the other suggestions is to have qemu-img
> check --repair optionally delete corrupted bitmaps. I kind of like this
> idea: it's a buyer beware operation that might make management layers
> unhappy, but then again ... repair is always something that could make
> things worse.

Well, dealing with corrupted bitmaps will be possible. I plan to expose
in the checkpoint XML whether a ckeckpoint is invalid (if it contains at
least one corrupted bitmap) and the user will have the option to delete
all previous checkpoints including the corrupted one to clear any
problem.

Note that deleting only the corrupted checkpoint will not be possible
until it's the oldest one as we attempt to merge them into the previous
ones. We could alternatively add a flag to skip merging of the invalid
checkpoint.

> Plus, if you manage to corrupt bitmaps badly enough that they can't even
> be parsed, you might need a heavyweight repair operation.
> 
> Nir, do you think that'd be sufficient for your needs for now, or would
> you still like to see more granular offline management?
> 
> --js
> 
> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list

Re: Offline manipulation of Dirty Bitmaps by qemu-img

[Vladimir Sementsov-Ogievskiy]

06.12.2019 1:37, John Snow wrote:
> This has come up in the past, and I believe we discussed this at KVM
> Forum, too:
> 
> There have been requests from oVirt (via Nir Soffer) to add some offline
> bitmap manipulation functionality. In the past, our stance has generally
> been "Use QEMU without an accelerator, and use QMP to manipulate the
> images."
> 
> We like this for a few reasons:
> 
> 1. It keeps bitmap management code tightly centralized
> 2. It allows for the full suite of bitmap manipulations in either
> offline or online mode with one tool
> 3. We wouldn't have to write new code.
> 4. Or design new CLIs and duplicate our existing work.
> 5. Or write even more tests.
> 
> However, tools like oVirt may or may not be fully equipped to launch
> QEMU in this context, and there is always a desire for qemu-img to be
> able to "do more", so existing management suites could extend
> functionality more easily.

I think, all guys, who don't want to use Qemu directly for image manipulations,
should hope for Kevin's "[RFC PATCH 00/18] Add qemu-storage-daemon", which is
the correct solution of their problem. Still, I'm not one of these guys.

> 
> (Or so I am imagining.)
> 
> I am still leaning heavily against adding any more CLI commands or
> options to qemu-img right now. Even if we do add some of the fundamental
> ones like "add" or "remove", it seems only a matter of time before we
> have to add "clear", "merge", etc. Is this just a race to code duplication?
> 
> On the other hand, one of the other suggestions is to have qemu-img
> check --repair optionally delete corrupted bitmaps. I kind of like this
> idea: it's a buyer beware operation that might make management layers
> unhappy, but then again ... repair is always something that could make
> things worse.
> 
> Plus, if you manage to corrupt bitmaps badly enough that they can't even
> be parsed, you might need a heavyweight repair operation.
> 

Improving "check" is a correct thing, because it's done inside qcow2 driver
itself. We just don't have corresponding qmp command or command line option
for Qemu to use this thing (or I missed it).

> Nir, do you think that'd be sufficient for your needs for now, or would
> you still like to see more granular offline management?
> 
> --js
> 


-- 
Best regards,
Vladimir

Re: Offline manipulation of Dirty Bitmaps by qemu-img

[John Snow]

On 12/6/19 5:37 AM, Vladimir Sementsov-Ogievskiy wrote:
> 06.12.2019 1:37, John Snow wrote:
>> This has come up in the past, and I believe we discussed this at KVM
>> Forum, too:
>>
>> There have been requests from oVirt (via Nir Soffer) to add some offline
>> bitmap manipulation functionality. In the past, our stance has generally
>> been "Use QEMU without an accelerator, and use QMP to manipulate the
>> images."
>>
>> We like this for a few reasons:
>>
>> 1. It keeps bitmap management code tightly centralized
>> 2. It allows for the full suite of bitmap manipulations in either
>> offline or online mode with one tool
>> 3. We wouldn't have to write new code.
>> 4. Or design new CLIs and duplicate our existing work.
>> 5. Or write even more tests.
>>
>> However, tools like oVirt may or may not be fully equipped to launch
>> QEMU in this context, and there is always a desire for qemu-img to be
>> able to "do more", so existing management suites could extend
>> functionality more easily.
> 
> I think, all guys, who don't want to use Qemu directly for image manipulations,
> should hope for Kevin's "[RFC PATCH 00/18] Add qemu-storage-daemon", which is
> the correct solution of their problem. Still, I'm not one of these guys.
> 
>>
>> (Or so I am imagining.)
>>
>> I am still leaning heavily against adding any more CLI commands or
>> options to qemu-img right now. Even if we do add some of the fundamental
>> ones like "add" or "remove", it seems only a matter of time before we
>> have to add "clear", "merge", etc. Is this just a race to code duplication?
>>
>> On the other hand, one of the other suggestions is to have qemu-img
>> check --repair optionally delete corrupted bitmaps. I kind of like this
>> idea: it's a buyer beware operation that might make management layers
>> unhappy, but then again ... repair is always something that could make
>> things worse.
>>
>> Plus, if you manage to corrupt bitmaps badly enough that they can't even
>> be parsed, you might need a heavyweight repair operation.
>>
> 
> Improving "check" is a correct thing, because it's done inside qcow2 driver
> itself. We just don't have corresponding qmp command or command line option
> for Qemu to use this thing (or I missed it).
> 

OK, I agree. I made a redhat BZ to track that we want this: 1780416 -
RFE: qemu-img check --repair should optionally remove any
corrupted bitmaps

I'll work on a patch and we can debate the details there.

--js