Re: Questions about how block devices use snapshots

[Zhiyong Ye <yezhiyong@bytedance.com>]

Hi Kevin,

Thank you for your reply and detailed answers.

In my scenario is the iSCSI SAN environment. The network storage device 
is connected to the physical machine via iSCSI, and LVM is used as the 
middle layer between the storage device and the VM for storage 
management and metadata synchronization. Every VM uses both raw and 
qcow2 formats, with the system disk being qcow2 and the data disk being 
raw. Therefore block devices need to support snapshot capability in both 
raw and qcow2 store methods. In addition, snapshot images should also be 
stored in iSCSI storage, which is a block device.

Both internal and external snapshots can implement snapshots of block 
devices, but they both have their drawbacks when multiple snapshots are 
required.

Internal snapshots can only be used in qcow2 format and do not require 
additional creation of new block devices. As you said, the block device 
has much more space than the virtual disk. There is no telling when disk 
space will be full when creating multiple snapshots.

External snapshots require the creation of additional block devices to 
store the overlay images, but it is not clear how much space needs to be 
created. If the space is the same as the virtual disk, when there are 
multiple snapshots it will be a serious waste of disk space, because 
each time a new snapshot is created the previous one will become 
read-only. However, if the disk space created is too small, the snapshot 
data may not be stored when the disk space is full.

The problem with both is the uncertainty of the space size of the block 
device at the time of creation. Of course, we can rely on lvm's resize 
function to dynamically grow the space of the block device. But I think 
this is more of a workaround.

It is mentioned in the Qemu docs page under "QEMU disk image utility" 
that the qemu-img rebase can be used to perform a “diff” operation on 
two disk images.

Say that base.img has been cloned as modified.img by copying it, and 
that the modified.img guest has run so there are now some changes 
compared to base.img. To construct a thin image called diff.qcow2 that 
contains just the differences, do:

qemu-img create -f qcow2 -b modified.img diff.qcow2
qemu-img rebase -b base.img diff.qcow2

At this point, modified.img can be discarded, since base.img + 
diff.qcow2 contains the same information.

Can this “diff” operation be used on snapshots of block devices? The 
first snapshot is a copy of the original disk (to save space we can copy 
only the data that has already been used), while the subsequent 
snapshots are based on the diff of the previous snapshot, so that the 
space required for the created block device is known at the time of the 
snapshot.

Regards

Zhiyong

On 1/9/23 9:57 PM, Kevin Wolf wrote:
> Am 09.01.2023 um 13:45 hat Zhiyong Ye geschrieben:
>> Qemu provides powerful snapshot capabilities for different file
>> formats. But this is limited to the block backend being a file, and
>> support is not good enough when it is a block device. When creating
>> snapshots based on files, there is no need to specify the size of the
>> snapshot image, which can grow dynamically as the virtual machine is
>> used. But block devices are fixed in size at creation and cannot be
>> dynamically grown at a later time.
>>
>> So is there any way to support snapshots when the block backend is a
>> block device?
> 
> In order to have snapshots, you need to have an image format like qcow2.
> 
> A qcow2 file can have a raw block device as its backing file, so even if
> you store the overlay image on a filesystem, you have technically
> snapshotted a block device. This may or may not be enough for your use
> case.
> 
> It is also possible to store qcow2 files on block devices, though
> depending on your requirements, it can get very tricky because then
> you're responsible for making sure that there is always enough free
> space on the block device.
> 
> So a second, still very simple, approach could be taking a second block
> device that is a little bit larger than the virtual disk (for the qcow2
> metadata) and use that as the external snapshot. Obviously, you require
> a lot of disk space this way, because each snapshots needs to be able to
> store the full image.
> 
> You could also use internal snapshots. In this case, you just need to
> make sure that the block device is a lot larger than the virtual disk,
> so that there is enough space left for storing the snapshots. At some
> point it will be full.
> 
> And finally, for example if your block devices are actually LVs, you
> could start resizing the block device dynmically as needed. This becomes
> very complex quickly and you're on your own, but it is possible and has
> been done by oVirt.
> 
> Kevin
>