Questions about how block devices use snapshots

Questions about how block devices use snapshots

[Zhiyong Ye]

Dear all,

Qemu provides powerful snapshot capabilities for different file formats. 
But this is limited to the block backend being a file, and support is 
not good enough when it is a block device. When creating snapshots based 
on files, there is no need to specify the size of the snapshot image, 
which can grow dynamically as the virtual machine is used. But block 
devices are fixed in size at creation and cannot be dynamically grown at 
a later time.

So is there any way to support snapshots when the block backend is a 
block device?

Regards

Zhiyong

Re: Questions about how block devices use snapshots

[Kevin Wolf]

Am 09.01.2023 um 13:45 hat Zhiyong Ye geschrieben:
> Qemu provides powerful snapshot capabilities for different file
> formats. But this is limited to the block backend being a file, and
> support is not good enough when it is a block device. When creating
> snapshots based on files, there is no need to specify the size of the
> snapshot image, which can grow dynamically as the virtual machine is
> used. But block devices are fixed in size at creation and cannot be
> dynamically grown at a later time.
> 
> So is there any way to support snapshots when the block backend is a
> block device?

In order to have snapshots, you need to have an image format like qcow2.

A qcow2 file can have a raw block device as its backing file, so even if
you store the overlay image on a filesystem, you have technically
snapshotted a block device. This may or may not be enough for your use
case.

It is also possible to store qcow2 files on block devices, though
depending on your requirements, it can get very tricky because then
you're responsible for making sure that there is always enough free
space on the block device.

So a second, still very simple, approach could be taking a second block
device that is a little bit larger than the virtual disk (for the qcow2
metadata) and use that as the external snapshot. Obviously, you require
a lot of disk space this way, because each snapshots needs to be able to
store the full image.

You could also use internal snapshots. In this case, you just need to
make sure that the block device is a lot larger than the virtual disk,
so that there is enough space left for storing the snapshots. At some
point it will be full.

And finally, for example if your block devices are actually LVs, you
could start resizing the block device dynmically as needed. This becomes
very complex quickly and you're on your own, but it is possible and has
been done by oVirt.

Kevin

Re: Questions about how block devices use snapshots

[Zhiyong Ye]

Hi Kevin,

Thank you for your reply and detailed answers.

In my scenario is the iSCSI SAN environment. The network storage device 
is connected to the physical machine via iSCSI, and LVM is used as the 
middle layer between the storage device and the VM for storage 
management and metadata synchronization. Every VM uses both raw and 
qcow2 formats, with the system disk being qcow2 and the data disk being 
raw. Therefore block devices need to support snapshot capability in both 
raw and qcow2 store methods. In addition, snapshot images should also be 
stored in iSCSI storage, which is a block device.

Both internal and external snapshots can implement snapshots of block 
devices, but they both have their drawbacks when multiple snapshots are 
required.

Internal snapshots can only be used in qcow2 format and do not require 
additional creation of new block devices. As you said, the block device 
has much more space than the virtual disk. There is no telling when disk 
space will be full when creating multiple snapshots.

External snapshots require the creation of additional block devices to 
store the overlay images, but it is not clear how much space needs to be 
created. If the space is the same as the virtual disk, when there are 
multiple snapshots it will be a serious waste of disk space, because 
each time a new snapshot is created the previous one will become 
read-only. However, if the disk space created is too small, the snapshot 
data may not be stored when the disk space is full.

The problem with both is the uncertainty of the space size of the block 
device at the time of creation. Of course, we can rely on lvm's resize 
function to dynamically grow the space of the block device. But I think 
this is more of a workaround.

It is mentioned in the Qemu docs page under "QEMU disk image utility" 
that the qemu-img rebase can be used to perform a “diff” operation on 
two disk images.

Say that base.img has been cloned as modified.img by copying it, and 
that the modified.img guest has run so there are now some changes 
compared to base.img. To construct a thin image called diff.qcow2 that 
contains just the differences, do:

qemu-img create -f qcow2 -b modified.img diff.qcow2
qemu-img rebase -b base.img diff.qcow2

At this point, modified.img can be discarded, since base.img + 
diff.qcow2 contains the same information.

Can this “diff” operation be used on snapshots of block devices? The 
first snapshot is a copy of the original disk (to save space we can copy 
only the data that has already been used), while the subsequent 
snapshots are based on the diff of the previous snapshot, so that the 
space required for the created block device is known at the time of the 
snapshot.

Regards

Zhiyong

On 1/9/23 9:57 PM, Kevin Wolf wrote:
> Am 09.01.2023 um 13:45 hat Zhiyong Ye geschrieben:
>> Qemu provides powerful snapshot capabilities for different file
>> formats. But this is limited to the block backend being a file, and
>> support is not good enough when it is a block device. When creating
>> snapshots based on files, there is no need to specify the size of the
>> snapshot image, which can grow dynamically as the virtual machine is
>> used. But block devices are fixed in size at creation and cannot be
>> dynamically grown at a later time.
>>
>> So is there any way to support snapshots when the block backend is a
>> block device?
> 
> In order to have snapshots, you need to have an image format like qcow2.
> 
> A qcow2 file can have a raw block device as its backing file, so even if
> you store the overlay image on a filesystem, you have technically
> snapshotted a block device. This may or may not be enough for your use
> case.
> 
> It is also possible to store qcow2 files on block devices, though
> depending on your requirements, it can get very tricky because then
> you're responsible for making sure that there is always enough free
> space on the block device.
> 
> So a second, still very simple, approach could be taking a second block
> device that is a little bit larger than the virtual disk (for the qcow2
> metadata) and use that as the external snapshot. Obviously, you require
> a lot of disk space this way, because each snapshots needs to be able to
> store the full image.
> 
> You could also use internal snapshots. In this case, you just need to
> make sure that the block device is a lot larger than the virtual disk,
> so that there is enough space left for storing the snapshots. At some
> point it will be full.
> 
> And finally, for example if your block devices are actually LVs, you
> could start resizing the block device dynmically as needed. This becomes
> very complex quickly and you're on your own, but it is possible and has
> been done by oVirt.
> 
> Kevin
> 

Re: Questions about how block devices use snapshots

[Kevin Wolf]

Am 11.01.2023 um 08:55 hat Zhiyong Ye geschrieben:
> Hi Kevin,
> 
> Thank you for your reply and detailed answers.
> 
> In my scenario is the iSCSI SAN environment. The network storage device is
> connected to the physical machine via iSCSI, and LVM is used as the middle
> layer between the storage device and the VM for storage management and
> metadata synchronization. Every VM uses both raw and qcow2 formats, with the
> system disk being qcow2 and the data disk being raw. Therefore block devices
> need to support snapshot capability in both raw and qcow2 store methods. In
> addition, snapshot images should also be stored in iSCSI storage, which is a
> block device.
> 
> Both internal and external snapshots can implement snapshots of block
> devices, but they both have their drawbacks when multiple snapshots are
> required.
> 
> Internal snapshots can only be used in qcow2 format and do not require
> additional creation of new block devices. As you said, the block device has
> much more space than the virtual disk. There is no telling when disk space
> will be full when creating multiple snapshots.
> 
> External snapshots require the creation of additional block devices to store
> the overlay images, but it is not clear how much space needs to be created.
> If the space is the same as the virtual disk, when there are multiple
> snapshots it will be a serious waste of disk space, because each time a new
> snapshot is created the previous one will become read-only. However, if the
> disk space created is too small, the snapshot data may not be stored when
> the disk space is full.
> 
> The problem with both is the uncertainty of the space size of the block
> device at the time of creation. Of course, we can rely on lvm's resize
> function to dynamically grow the space of the block device. But I think this
> is more of a workaround.

Yes, this is why I said it's challenging. oVirt uses resizing of LVs to
achieve this, and it's still very complex. You need to decide yourself
if you think implementing the management software for this is worth it.

> It is mentioned in the Qemu docs page under "QEMU disk image utility" that
> the qemu-img rebase can be used to perform a “diff” operation on two disk
> images.
> 
> Say that base.img has been cloned as modified.img by copying it, and that
> the modified.img guest has run so there are now some changes compared to
> base.img. To construct a thin image called diff.qcow2 that contains just the
> differences, do:
> 
> qemu-img create -f qcow2 -b modified.img diff.qcow2
> qemu-img rebase -b base.img diff.qcow2
> 
> At this point, modified.img can be discarded, since base.img + diff.qcow2
> contains the same information.
> 
> Can this “diff” operation be used on snapshots of block devices? The first
> snapshot is a copy of the original disk (to save space we can copy only the
> data that has already been used), while the subsequent snapshots are based
> on the diff of the previous snapshot, so that the space required for the
> created block device is known at the time of the snapshot.

Yes, you can use raw block devices for both base.img and modified.img.
But of course, the result is still a qcow2 file that you need to store
somewhere.

Kevin

Re: Questions about how block devices use snapshots

[Zhiyong Ye]

Hi Kevin,

Can I ask again how base.img + diff.qcow2 can be re-merged into one 
image via qemu-img or hmp command when modified.img is discarded?

Regards

Zhiyong

On 1/11/23 10:32 PM, Kevin Wolf wrote:
> Am 11.01.2023 um 08:55 hat Zhiyong Ye geschrieben:
>> Hi Kevin,
>>
>> Thank you for your reply and detailed answers.
>>
>> In my scenario is the iSCSI SAN environment. The network storage device is
>> connected to the physical machine via iSCSI, and LVM is used as the middle
>> layer between the storage device and the VM for storage management and
>> metadata synchronization. Every VM uses both raw and qcow2 formats, with the
>> system disk being qcow2 and the data disk being raw. Therefore block devices
>> need to support snapshot capability in both raw and qcow2 store methods. In
>> addition, snapshot images should also be stored in iSCSI storage, which is a
>> block device.
>>
>> Both internal and external snapshots can implement snapshots of block
>> devices, but they both have their drawbacks when multiple snapshots are
>> required.
>>
>> Internal snapshots can only be used in qcow2 format and do not require
>> additional creation of new block devices. As you said, the block device has
>> much more space than the virtual disk. There is no telling when disk space
>> will be full when creating multiple snapshots.
>>
>> External snapshots require the creation of additional block devices to store
>> the overlay images, but it is not clear how much space needs to be created.
>> If the space is the same as the virtual disk, when there are multiple
>> snapshots it will be a serious waste of disk space, because each time a new
>> snapshot is created the previous one will become read-only. However, if the
>> disk space created is too small, the snapshot data may not be stored when
>> the disk space is full.
>>
>> The problem with both is the uncertainty of the space size of the block
>> device at the time of creation. Of course, we can rely on lvm's resize
>> function to dynamically grow the space of the block device. But I think this
>> is more of a workaround.
> 
> Yes, this is why I said it's challenging. oVirt uses resizing of LVs to
> achieve this, and it's still very complex. You need to decide yourself
> if you think implementing the management software for this is worth it.
> 
>> It is mentioned in the Qemu docs page under "QEMU disk image utility" that
>> the qemu-img rebase can be used to perform a “diff” operation on two disk
>> images.
>>
>> Say that base.img has been cloned as modified.img by copying it, and that
>> the modified.img guest has run so there are now some changes compared to
>> base.img. To construct a thin image called diff.qcow2 that contains just the
>> differences, do:
>>
>> qemu-img create -f qcow2 -b modified.img diff.qcow2
>> qemu-img rebase -b base.img diff.qcow2
>>
>> At this point, modified.img can be discarded, since base.img + diff.qcow2
>> contains the same information.
>>
>> Can this “diff” operation be used on snapshots of block devices? The first
>> snapshot is a copy of the original disk (to save space we can copy only the
>> data that has already been used), while the subsequent snapshots are based
>> on the diff of the previous snapshot, so that the space required for the
>> created block device is known at the time of the snapshot.
> 
> Yes, you can use raw block devices for both base.img and modified.img.
> But of course, the result is still a qcow2 file that you need to store
> somewhere.
> 
> Kevin
> 

Re: Questions about how block devices use snapshots

[Kevin Wolf]

Am 11.01.2023 um 17:21 hat Zhiyong Ye geschrieben:
> Hi Kevin,
> 
> Can I ask again how base.img + diff.qcow2 can be re-merged into one image
> via qemu-img or hmp command when modified.img is discarded?

You can either use 'qemu-img commit' to copy all of the data from
diff.qcow2 back into base.img (this is probably what you want), or
'qemu-img rebase' to copy all of the data from base.img into diff.qcow2.

Kevin

Re: Questions about how block devices use snapshots

[Zhiyong Ye]

Hi Kevin,

Thank you for your patience and explanations. I learned a lot from our 
discussions and thank you again for your help.

Regards

Zhiyong

On 1/12/23 7:47 PM, Kevin Wolf wrote:
> Am 11.01.2023 um 17:21 hat Zhiyong Ye geschrieben:
>> Hi Kevin,
>>
>> Can I ask again how base.img + diff.qcow2 can be re-merged into one image
>> via qemu-img or hmp command when modified.img is discarded?
> 
> You can either use 'qemu-img commit' to copy all of the data from
> diff.qcow2 back into base.img (this is probably what you want), or
> 'qemu-img rebase' to copy all of the data from base.img into diff.qcow2.
> 
> Kevin
> 

Re: Questions about how block devices use snapshots

[Zhiyong Ye]

Hi Kevin,

Sorry to bother you again.

I intend to use this approach for snapshots of block devices, which, as 
you say, requires a lot of disk space to store snapshot data. So, to 
save disk space, after each successful external snapshot creation, I 
want to shrink the block device that stores the backing_file image to 
the size that qcow2 data actually occupies, since it has become 
read-only. But there is no way to get the actual size of qcow2 when it 
is stored in a block device.

Qemu-img info can easily get the actual size of qcow2 when it is stored 
in a file using the fstat function, but this will fail and return 0 for 
block devices. Therefore, it is necessary to implement the method of 
getting data occupancy inside qcow2. I think there may be two possible 
ways to do this:

- Add a cluster count field @nb_clusters in the BDRVQcow2State for each 
new cluster allocated and the actual size occupied by qcow2 is: 
nb_clusters * cluster_size.
- Iterate through the refcount block to find the value with the largest 
host offset, and this is the actual size occupied by qcow2.

Since I'm not very familiar with qcow2, may I ask if you have any advice 
on getting the actual size when using qcow2?

Regards

Zhiyong

On 1/9/23 9:57 PM, Kevin Wolf wrote:
> 
> So a second, still very simple, approach could be taking a second block
> device that is a little bit larger than the virtual disk (for the qcow2
> metadata) and use that as the external snapshot. Obviously, you require
> a lot of disk space this way, because each snapshots needs to be able to
> store the full image.
> 

Re: Questions about how block devices use snapshots

[Kevin Wolf]

Am 21.02.2023 um 14:27 hat Zhiyong Ye geschrieben:
> 
> Hi Kevin,
> 
> Sorry to bother you again.
> 
> I intend to use this approach for snapshots of block devices, which, as you
> say, requires a lot of disk space to store snapshot data. So, to save disk
> space, after each successful external snapshot creation, I want to shrink
> the block device that stores the backing_file image to the size that qcow2
> data actually occupies, since it has become read-only. But there is no way
> to get the actual size of qcow2 when it is stored in a block device.
> 
> Qemu-img info can easily get the actual size of qcow2 when it is stored in a
> file using the fstat function, but this will fail and return 0 for block
> devices. Therefore, it is necessary to implement the method of getting data
> occupancy inside qcow2. I think there may be two possible ways to do this:
> 
> - Add a cluster count field @nb_clusters in the BDRVQcow2State for each new
> cluster allocated and the actual size occupied by qcow2 is: nb_clusters *
> cluster_size.
> - Iterate through the refcount block to find the value with the largest host
> offset, and this is the actual size occupied by qcow2.
> 
> Since I'm not very familiar with qcow2, may I ask if you have any advice on
> getting the actual size when using qcow2?

I think what you need is the 'image-end-offset' field from 'qemu-img
check --output=json'.

Kevin

Re: Questions about how block devices use snapshots

[Zhiyong Ye]

Hi Kevin,

Thank you for your reply and this method works.

May I ask if this 'image-end-offset' field can be shown in the qemu-img 
info too? Because it is also a very useful information whether qcow2 is 
placed on a file or a block device.

Regards

Zhiyong

On 2/21/23 11:58 PM, Kevin Wolf wrote:
> Am 21.02.2023 um 14:27 hat Zhiyong Ye geschrieben:
>>
>> Hi Kevin,
>>
>> Sorry to bother you again.
>>
>> I intend to use this approach for snapshots of block devices, which, as you
>> say, requires a lot of disk space to store snapshot data. So, to save disk
>> space, after each successful external snapshot creation, I want to shrink
>> the block device that stores the backing_file image to the size that qcow2
>> data actually occupies, since it has become read-only. But there is no way
>> to get the actual size of qcow2 when it is stored in a block device.
>>
>> Qemu-img info can easily get the actual size of qcow2 when it is stored in a
>> file using the fstat function, but this will fail and return 0 for block
>> devices. Therefore, it is necessary to implement the method of getting data
>> occupancy inside qcow2. I think there may be two possible ways to do this:
>>
>> - Add a cluster count field @nb_clusters in the BDRVQcow2State for each new
>> cluster allocated and the actual size occupied by qcow2 is: nb_clusters *
>> cluster_size.
>> - Iterate through the refcount block to find the value with the largest host
>> offset, and this is the actual size occupied by qcow2.
>>
>> Since I'm not very familiar with qcow2, may I ask if you have any advice on
>> getting the actual size when using qcow2?
> 
> I think what you need is the 'image-end-offset' field from 'qemu-img
> check --output=json'.
> 
> Kevin
> 

Re: Questions about how block devices use snapshots

[Kevin Wolf]

Am 23.02.2023 um 08:35 hat Zhiyong Ye geschrieben:
> Hi Kevin,
> 
> Thank you for your reply and this method works.
> 
> May I ask if this 'image-end-offset' field can be shown in the qemu-img info
> too? Because it is also a very useful information whether qcow2 is placed on
> a file or a block device.

The only way to know the largest offset is by looking at all the
metadata in qcow2. 'qemu-img info' is supposed to be very fast, so we
don't do that there. 'qemu-img check' already looks at all metadata, so
we have the number readily available there.

Kevin

> On 2/21/23 11:58 PM, Kevin Wolf wrote:
> > Am 21.02.2023 um 14:27 hat Zhiyong Ye geschrieben:
> > > 
> > > Hi Kevin,
> > > 
> > > Sorry to bother you again.
> > > 
> > > I intend to use this approach for snapshots of block devices, which, as you
> > > say, requires a lot of disk space to store snapshot data. So, to save disk
> > > space, after each successful external snapshot creation, I want to shrink
> > > the block device that stores the backing_file image to the size that qcow2
> > > data actually occupies, since it has become read-only. But there is no way
> > > to get the actual size of qcow2 when it is stored in a block device.
> > > 
> > > Qemu-img info can easily get the actual size of qcow2 when it is stored in a
> > > file using the fstat function, but this will fail and return 0 for block
> > > devices. Therefore, it is necessary to implement the method of getting data
> > > occupancy inside qcow2. I think there may be two possible ways to do this:
> > > 
> > > - Add a cluster count field @nb_clusters in the BDRVQcow2State for each new
> > > cluster allocated and the actual size occupied by qcow2 is: nb_clusters *
> > > cluster_size.
> > > - Iterate through the refcount block to find the value with the largest host
> > > offset, and this is the actual size occupied by qcow2.
> > > 
> > > Since I'm not very familiar with qcow2, may I ask if you have any advice on
> > > getting the actual size when using qcow2?
> > 
> > I think what you need is the 'image-end-offset' field from 'qemu-img
> > check --output=json'.
> > 
> > Kevin
> > 
> 

Re: Questions about how block devices use snapshots

[Zhiyong Ye]

Hi Kevin,

Thank you for your patience and explanations.

Thanks again!

Zhiyong

On 2/23/23 7:39 PM, Kevin Wolf wrote:
> Am 23.02.2023 um 08:35 hat Zhiyong Ye geschrieben:
>> Hi Kevin,
>>
>> Thank you for your reply and this method works.
>>
>> May I ask if this 'image-end-offset' field can be shown in the qemu-img info
>> too? Because it is also a very useful information whether qcow2 is placed on
>> a file or a block device.
> 
> The only way to know the largest offset is by looking at all the
> metadata in qcow2. 'qemu-img info' is supposed to be very fast, so we
> don't do that there. 'qemu-img check' already looks at all metadata, so
> we have the number readily available there.
> 
> Kevin
> 
>> On 2/21/23 11:58 PM, Kevin Wolf wrote:
>>> Am 21.02.2023 um 14:27 hat Zhiyong Ye geschrieben:
>>>>
>>>> Hi Kevin,
>>>>
>>>> Sorry to bother you again.
>>>>
>>>> I intend to use this approach for snapshots of block devices, which, as you
>>>> say, requires a lot of disk space to store snapshot data. So, to save disk
>>>> space, after each successful external snapshot creation, I want to shrink
>>>> the block device that stores the backing_file image to the size that qcow2
>>>> data actually occupies, since it has become read-only. But there is no way
>>>> to get the actual size of qcow2 when it is stored in a block device.
>>>>
>>>> Qemu-img info can easily get the actual size of qcow2 when it is stored in a
>>>> file using the fstat function, but this will fail and return 0 for block
>>>> devices. Therefore, it is necessary to implement the method of getting data
>>>> occupancy inside qcow2. I think there may be two possible ways to do this:
>>>>
>>>> - Add a cluster count field @nb_clusters in the BDRVQcow2State for each new
>>>> cluster allocated and the actual size occupied by qcow2 is: nb_clusters *
>>>> cluster_size.
>>>> - Iterate through the refcount block to find the value with the largest host
>>>> offset, and this is the actual size occupied by qcow2.
>>>>
>>>> Since I'm not very familiar with qcow2, may I ask if you have any advice on
>>>> getting the actual size when using qcow2?
>>>
>>> I think what you need is the 'image-end-offset' field from 'qemu-img
>>> check --output=json'.
>>>
>>> Kevin
>>>
>>
>