Query on the dirty bitmap

Query on the dirty bitmap

[prashant patil]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

Hello All,
Hope this email finds you well.

I have been trying with qemu for a while now, and have come across a
problem specific to dirty bitmaps. I have enabled bitmap on the qcow2 disk
image using 'qemu-img bitmap' command, exposed the bitmap over a unix
socket using 'qemu-nbd' command. Now when I try to read the bitmap using
'qemu-img map' command with 'x-dirty-bitmap=qemu:dirty-bitmap:{bitmap}'
option, I get one single extent which shows that the entire disk is dirty.
Note that the disk size is 5 GB, and has only a few MB of data in it, and
had added very small data after the bitmap was enabled. Bitmap output has
been pasted below.

[{ "start": 0, "length": 5368709120, "depth": 0, "present": true, "zero":
false, "data": true, "compressed": false, "offset": 0}]

Can someone please help me understand why the bitmap content shows the
entire disk as dirty?

Regards
Prashant

[-- Attachment #2: Type: text/html, Size: 1138 bytes --]

Re: Query on the dirty bitmap

[Eric Blake]

On Wed, Feb 19, 2025 at 04:23:26PM +0530, prashant patil wrote:
> Hello All,
> Hope this email finds you well.
> 
> I have been trying with qemu for a while now, and have come across a
> problem specific to dirty bitmaps. I have enabled bitmap on the qcow2 disk
> image using 'qemu-img bitmap' command, exposed the bitmap over a unix
> socket using 'qemu-nbd' command. Now when I try to read the bitmap using
> 'qemu-img map' command with 'x-dirty-bitmap=qemu:dirty-bitmap:{bitmap}'
> option, I get one single extent which shows that the entire disk is dirty.
> Note that the disk size is 5 GB, and has only a few MB of data in it, and
> had added very small data after the bitmap was enabled. Bitmap output has
> been pasted below.

Can you show the exact sequence of command lines you used to create
the image, dirty a portion of it, then start up the qemu-nbd process
to inspect it?  As written, I can't reproduce your issue, but I know
it sounds similar to tests/qemu-iotests/tests/qemu-img-bitmaps which
does what you're talking about, so I know the code works and have to
suspect you may have missed a step or reordered things in such a way
that the entire bitmap is reading as dirty.

> 
> [{ "start": 0, "length": 5368709120, "depth": 0, "present": true, "zero":
> false, "data": true, "compressed": false, "offset": 0}]
> 
> Can someone please help me understand why the bitmap content shows the
> entire disk as dirty?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: Query on the dirty bitmap

[prashant patil]

[-- Attachment #1: Type: text/plain, Size: 2092 bytes --]

I was trying to read the bitmap of the running vm's disk. When I followed
below mentioned commands, then I was able to read the bitmap properly.
block-dirty-bitmap-add, block-dirty-bitmap-disable, nbd-server-start,
nbd-server-add, qemu-img
map with x-dirty-bitmap image-opts.

Please let me know if there is any other better alternative to do the same.

Thanks
Prashant

On Fri, Feb 28, 2025 at 3:53 AM Eric Blake <eblake@redhat.com> wrote:

> On Wed, Feb 19, 2025 at 04:23:26PM +0530, prashant patil wrote:
> > Hello All,
> > Hope this email finds you well.
> >
> > I have been trying with qemu for a while now, and have come across a
> > problem specific to dirty bitmaps. I have enabled bitmap on the qcow2
> disk
> > image using 'qemu-img bitmap' command, exposed the bitmap over a unix
> > socket using 'qemu-nbd' command. Now when I try to read the bitmap using
> > 'qemu-img map' command with 'x-dirty-bitmap=qemu:dirty-bitmap:{bitmap}'
> > option, I get one single extent which shows that the entire disk is
> dirty.
> > Note that the disk size is 5 GB, and has only a few MB of data in it, and
> > had added very small data after the bitmap was enabled. Bitmap output has
> > been pasted below.
>
> Can you show the exact sequence of command lines you used to create
> the image, dirty a portion of it, then start up the qemu-nbd process
> to inspect it?  As written, I can't reproduce your issue, but I know
> it sounds similar to tests/qemu-iotests/tests/qemu-img-bitmaps which
> does what you're talking about, so I know the code works and have to
> suspect you may have missed a step or reordered things in such a way
> that the entire bitmap is reading as dirty.
>
> >
> > [{ "start": 0, "length": 5368709120, "depth": 0, "present": true, "zero":
> > false, "data": true, "compressed": false, "offset": 0}]
> >
> > Can someone please help me understand why the bitmap content shows the
> > entire disk as dirty?
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.
> Virtualization:  qemu.org | libguestfs.org
>
>

[-- Attachment #2: Type: text/html, Size: 2829 bytes --]

Re: Query on the dirty bitmap

[Eric Blake]

On Wed, Mar 05, 2025 at 03:36:35PM +0530, prashant patil wrote:
> I was trying to read the bitmap of the running vm's disk. When I followed
> below mentioned commands, then I was able to read the bitmap properly.
> block-dirty-bitmap-add, block-dirty-bitmap-disable, nbd-server-start,
> nbd-server-add,

Up to here, everything you've done is indeed the ideal way to get at
the contents of a bitmap from a live qemu process over an NBD connection.

> qemu-img
> map with x-dirty-bitmap image-opts.

Here, this works, but feels like a hack, because it is relying on the
x-dirty-bitmap feature of qemu.  The libnbd project ships with an
application 'nbdinfo --map' that can read the same information as
'qemu-img map' but with a much nicer layout.  It's not going to
necessarily be faster, but because it is a fully-supported feature of
libnbd rather than a hack in qemu, it may prove more stable in the
long run, and certainly easier to understand.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: Query on the dirty bitmap

[prashant patil]

[-- Attachment #1: Type: text/plain, Size: 3118 bytes --]

Thanks Eric.
I have a few questions about the bitmap content shown by 'qemu-img map'.
From below sample bitmap data:
1. Why only some of the extents have start and offset values? And why are
they the same values?
2. What does the start value indicate? Is it logical offset or physical
offset of data into qcow2?

root@be-proxmox1:/# qemu-img map --output=json --image-opts
"$IMG,x-dirty-bitmap=qemu:dirty-bitmap:bitmap1"
  [{ "start": 0, "length": 196608, "depth": 0, "present": true, "zero":
false, "data": true, "compressed": false, "offset": 0},
{ "start": 196608, "length": 65536, "depth": 0, "present": false, "zero":
false, "data": false, "compressed": false},
{ "start": 262144, "length": 105840640, "depth": 0, "present": true,
"zero": false, "data": true, "compressed": false, "offset": 262144},
{ "start": 106102784, "length": 95485952, "depth": 0, "present": false,
"zero": false, "data": false, "compressed": false},
{ "start": 201588736, "length": 145227776, "depth": 0, "present": true,
"zero": false, "data": true, "compressed": false, "offset": 201588736},
{ "start": 346816512, "length": 131072, "depth": 0, "present": false,
"zero": false, "data": false, "compressed": false},
{ "start": 346947584, "length": 131072, "depth": 0, "present": true,
"zero": false, "data": true, "compressed": false, "offset": 346947584},
{ "start": 347078656, "length": 65536, "depth": 0, "present": false,
"zero": false, "data": false, "compressed": false},
{ "start": 347144192, "length": 9699328, "depth": 0, "present": true,
"zero": false, "data": true, "compressed": false, "offset": 347144192},
{ "start": 356843520, "length": 131072, "depth": 0, "present": false,
"zero": false, "data": false, "compressed": false},
{ "start": 356974592, "length": 716767232, "depth": 0, "present": true,
"zero": false, "data": true, "compressed": false, "offset": 356974592}]


Regards
Prashant

On Thu, Mar 6, 2025 at 3:44 AM Eric Blake <eblake@redhat.com> wrote:

> On Wed, Mar 05, 2025 at 03:36:35PM +0530, prashant patil wrote:
> > I was trying to read the bitmap of the running vm's disk. When I followed
> > below mentioned commands, then I was able to read the bitmap properly.
> > block-dirty-bitmap-add, block-dirty-bitmap-disable, nbd-server-start,
> > nbd-server-add,
>
> Up to here, everything you've done is indeed the ideal way to get at
> the contents of a bitmap from a live qemu process over an NBD connection.
>
> > qemu-img
> > map with x-dirty-bitmap image-opts.
>
> Here, this works, but feels like a hack, because it is relying on the
> x-dirty-bitmap feature of qemu.  The libnbd project ships with an
> application 'nbdinfo --map' that can read the same information as
> 'qemu-img map' but with a much nicer layout.  It's not going to
> necessarily be faster, but because it is a fully-supported feature of
> libnbd rather than a hack in qemu, it may prove more stable in the
> long run, and certainly easier to understand.
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.
> Virtualization:  qemu.org | libguestfs.org
>
>

[-- Attachment #2: Type: text/html, Size: 4644 bytes --]

Re: Query on the dirty bitmap

[Eric Blake]

On Mon, Apr 07, 2025 at 02:46:17PM +0530, prashant patil wrote:
> Thanks Eric.

[top-posting makes conversations harder to follow, so on this list we
typically reply inline]

> I have a few questions about the bitmap content shown by 'qemu-img map'.
> From below sample bitmap data:
> 1. Why only some of the extents have start and offset values? And why are
> they the same values?
> 2. What does the start value indicate? Is it logical offset or physical
> offset of data into qcow2?

Normally (when there is no x-dirty-bitmap in play), 'start' denotes
the logical offset being reported on (you'd expect a map to list every
logical offset; so the start of entry N+1 should be the sum of start +
length of entry N), while 'offset' is where that extent begins in the
underlying file.  For a raw source, offset and start will be
identical; for other sources, like qcow2, start is obviously logical,
while offset is physical.  Entries without 'offset' are places where
the logical contents are compressed, synthesized, or otherwise have no
1:1 correspondence to an offset in the physical file.  'present'
indicates whether the data is synthesized or not; 'offset' is going to
be absent if 'present' is false; although it can also be absent even
when 'present' is true such as in the case of compression.

When it comes to exposing a qcow2 file over NBD, you generally want to
have:

qemu-nbd using '-f qcow2' => raw view => qemu-img using '-f raw'

to expose only the logical contents over the wire.  It is also
possible to flip the responsibility:

qemu-nbd using '-f raw' => qcow2 view => qemu-img using '-f qcow2'

to expose the bare-metal qcow2 contents over the wire, but that gets
less testing, in part because if you make the image writable, it tends
to cause problems if the client writing to the qcow2 layer needs to
allocate (since NBD does not have a resize command).  Having the
server open the file as qcow2 and only serving raw contents means the
server can resize transparently.  What's more, block status commands
over NBD only work when the server is aware of the qcow2 nature of the
file it is serving (if you expose qcow2 bits over the wire, the server
treats the entire file as allocated, and there is is no dirty bitmap
context for the client to read over NBD).

Therefore, 'start' and 'offset' are going to be identical if you are
using qemu-img map to read bitmaps from a server, since bitmaps can
only be read when the wire has the raw view (not the qcow2 view).

But when you add x-dirty-bitmap into the mix, you are asking qemu to
DISREGARD the normal rules of backing file information and instead
report on dirty bitmap information as if it were backing file info.
This means that anywhere the dirty bitmap response differs from a
normal backing file response, the output of qemu-img is reporting
garbage data (for example, the 'present' bit is now bogus, which
explains why the rows where qemu-img claims 'present' is false omit an
'offset').  Only 'start', 'length', and 'data' matter when using
x-dirty-bitmap, with 'data=false' meaning the section was reported
dirty, and 'data=true' meaning the section was reported unchanged.

And if that is confusing, well yeah. Which is why I recommend using
libnbd's nbdinfo --map, where the output is more obvious.

> 
> root@be-proxmox1:/# qemu-img map --output=json --image-opts
> "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:bitmap1"
>   [{ "start": 0, "length": 196608, "depth": 0, "present": true, "zero":
> false, "data": true, "compressed": false, "offset": 0},

The extent starting at logical offset 0 and lasting 196608 bytes is
unchanged.

> { "start": 196608, "length": 65536, "depth": 0, "present": false, "zero":
> false, "data": false, "compressed": false},

The extent starting at 196608 and lasting 65536 bytes is dirty
(altered since the point in time when the bitmap was created).

> > > qemu-img
> > > map with x-dirty-bitmap image-opts.
> >
> > Here, this works, but feels like a hack, because it is relying on the
> > x-dirty-bitmap feature of qemu.  The libnbd project ships with an
> > application 'nbdinfo --map' that can read the same information as
> > 'qemu-img map' but with a much nicer layout.  It's not going to
> > necessarily be faster, but because it is a fully-supported feature of
> > libnbd rather than a hack in qemu, it may prove more stable in the
> > long run, and certainly easier to understand.


For a worked example:

# Create an image, dirty two different offsets, with a bitmap created in between
$ qemu-img create -f qcow2 foo.qcow2 3M
Formatting 'foo.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=3145728 lazy_refcounts=off refcount_bits=16
$ qemu-io -f qcow2 foo.qcow2 -c 'w 1M 512'
wrote 512/512 bytes at offset 1048576
512 bytes, 1 ops; 00.01 sec (42.906 KiB/sec and 85.8112 ops/sec)
$ qemu-img bitmap -f qcow2 foo.qcow2 --add b1
$ qemu-io -f qcow2 foo.qcow2 -c 'w 2M 512'
wrote 512/512 bytes at offset 2097152
512 bytes, 1 ops; 00.00 sec (107.735 KiB/sec and 215.4704 ops/sec)

# Now, expose the raw bytes of the image over NBD, and inspect with nbdinfo

$ qemu-nbd -f qcow2 foo.qcow2 -A -Bb1 -t
$ nbdinfo --map=qemu:dirty-bitmap:b1 nbd://localhost
         0     2097152    0  clean
   2097152       65536    1  dirty
   2162688      983040    0  clean
$ nbdinfo --map nbd://localhost
         0     1048576    3  hole,zero
   1048576        4096    0  data
   1052672       61440    2  zero
   1114112      983040    3  hole,zero
   2097152        4096    0  data
   2101248       61440    2  zero
   2162688      983040    3  hole,zero

Comparing those to qemu-img map --output=json should be instructive.


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: Query on the dirty bitmap

[prashant patil]

[-- Attachment #1: Type: text/plain, Size: 6944 bytes --]

 Thank you, Eric, for the thorough information—truly appreciate it.

Just to confirm what I understood, when we are reading a bitmap with
'x-dirty-bitmap' (for powered on vm of course), the 'start' is always a
logical offset no matter whether the record has 'offset' value or not. Is
this correct?

Also, I came across a case wherein we get the entire disk as allocated for
a raw format disk which is present on lvm or lvm-thin storage (the disk has
just a few MB data added, and the vm is in running state). Here is an
example of 1Gb data. Is this expected behaviour?
[{ "start": 0, "length": 1073741824, "depth": 0, "present": true, "zero":
false, "data": true, "compressed": false, "offset": 0}]

Regards,
Prashant

On Mon, Apr 7, 2025 at 8:24 PM Eric Blake <eblake@redhat.com> wrote:

> On Mon, Apr 07, 2025 at 02:46:17PM +0530, prashant patil wrote:
> > Thanks Eric.
>
> [top-posting makes conversations harder to follow, so on this list we
> typically reply inline]
>
> > I have a few questions about the bitmap content shown by 'qemu-img map'.
> > From below sample bitmap data:
> > 1. Why only some of the extents have start and offset values? And why are
> > they the same values?
> > 2. What does the start value indicate? Is it logical offset or physical
> > offset of data into qcow2?
>
> Normally (when there is no x-dirty-bitmap in play), 'start' denotes
> the logical offset being reported on (you'd expect a map to list every
> logical offset; so the start of entry N+1 should be the sum of start +
> length of entry N), while 'offset' is where that extent begins in the
> underlying file.  For a raw source, offset and start will be
> identical; for other sources, like qcow2, start is obviously logical,
> while offset is physical.  Entries without 'offset' are places where
> the logical contents are compressed, synthesized, or otherwise have no
> 1:1 correspondence to an offset in the physical file.  'present'
> indicates whether the data is synthesized or not; 'offset' is going to
> be absent if 'present' is false; although it can also be absent even
> when 'present' is true such as in the case of compression.
>
> When it comes to exposing a qcow2 file over NBD, you generally want to
> have:
>
> qemu-nbd using '-f qcow2' => raw view => qemu-img using '-f raw'
>
> to expose only the logical contents over the wire.  It is also
> possible to flip the responsibility:
>
> qemu-nbd using '-f raw' => qcow2 view => qemu-img using '-f qcow2'
>
> to expose the bare-metal qcow2 contents over the wire, but that gets
> less testing, in part because if you make the image writable, it tends
> to cause problems if the client writing to the qcow2 layer needs to
> allocate (since NBD does not have a resize command).  Having the
> server open the file as qcow2 and only serving raw contents means the
> server can resize transparently.  What's more, block status commands
> over NBD only work when the server is aware of the qcow2 nature of the
> file it is serving (if you expose qcow2 bits over the wire, the server
> treats the entire file as allocated, and there is is no dirty bitmap
> context for the client to read over NBD).
>
> Therefore, 'start' and 'offset' are going to be identical if you are
> using qemu-img map to read bitmaps from a server, since bitmaps can
> only be read when the wire has the raw view (not the qcow2 view).
>
> But when you add x-dirty-bitmap into the mix, you are asking qemu to
> DISREGARD the normal rules of backing file information and instead
> report on dirty bitmap information as if it were backing file info.
> This means that anywhere the dirty bitmap response differs from a
> normal backing file response, the output of qemu-img is reporting
> garbage data (for example, the 'present' bit is now bogus, which
> explains why the rows where qemu-img claims 'present' is false omit an
> 'offset').  Only 'start', 'length', and 'data' matter when using
> x-dirty-bitmap, with 'data=false' meaning the section was reported
> dirty, and 'data=true' meaning the section was reported unchanged.
>
> And if that is confusing, well yeah. Which is why I recommend using
> libnbd's nbdinfo --map, where the output is more obvious.
>
> >
> > root@be-proxmox1:/# qemu-img map --output=json --image-opts
> > "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:bitmap1"
> >   [{ "start": 0, "length": 196608, "depth": 0, "present": true, "zero":
> > false, "data": true, "compressed": false, "offset": 0},
>
> The extent starting at logical offset 0 and lasting 196608 bytes is
> unchanged.
>
> > { "start": 196608, "length": 65536, "depth": 0, "present": false, "zero":
> > false, "data": false, "compressed": false},
>
> The extent starting at 196608 and lasting 65536 bytes is dirty
> (altered since the point in time when the bitmap was created).
>
> > > > qemu-img
> > > > map with x-dirty-bitmap image-opts.
> > >
> > > Here, this works, but feels like a hack, because it is relying on the
> > > x-dirty-bitmap feature of qemu.  The libnbd project ships with an
> > > application 'nbdinfo --map' that can read the same information as
> > > 'qemu-img map' but with a much nicer layout.  It's not going to
> > > necessarily be faster, but because it is a fully-supported feature of
> > > libnbd rather than a hack in qemu, it may prove more stable in the
> > > long run, and certainly easier to understand.
>
>
> For a worked example:
>
> # Create an image, dirty two different offsets, with a bitmap created in
> between
> $ qemu-img create -f qcow2 foo.qcow2 3M
> Formatting 'foo.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off
> compression_type=zlib size=3145728 lazy_refcounts=off refcount_bits=16
> $ qemu-io -f qcow2 foo.qcow2 -c 'w 1M 512'
> wrote 512/512 bytes at offset 1048576
> 512 bytes, 1 ops; 00.01 sec (42.906 KiB/sec and 85.8112 ops/sec)
> $ qemu-img bitmap -f qcow2 foo.qcow2 --add b1
> $ qemu-io -f qcow2 foo.qcow2 -c 'w 2M 512'
> wrote 512/512 bytes at offset 2097152
> 512 bytes, 1 ops; 00.00 sec (107.735 KiB/sec and 215.4704 ops/sec)
>
> # Now, expose the raw bytes of the image over NBD, and inspect with nbdinfo
>
> $ qemu-nbd -f qcow2 foo.qcow2 -A -Bb1 -t
> $ nbdinfo --map=qemu:dirty-bitmap:b1 nbd://localhost
>          0     2097152    0  clean
>    2097152       65536    1  dirty
>    2162688      983040    0  clean
> $ nbdinfo --map nbd://localhost
>          0     1048576    3  hole,zero
>    1048576        4096    0  data
>    1052672       61440    2  zero
>    1114112      983040    3  hole,zero
>    2097152        4096    0  data
>    2101248       61440    2  zero
>    2162688      983040    3  hole,zero
>
> Comparing those to qemu-img map --output=json should be instructive.
>
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.
> Virtualization:  qemu.org | libguestfs.org
>
>

[-- Attachment #2: Type: text/html, Size: 8451 bytes --]

Re: Query on the dirty bitmap

[Eric Blake]

On Tue, Apr 08, 2025 at 04:30:38PM +0530, prashant patil wrote:
>  Thank you, Eric, for the thorough information—truly appreciate it.
> 
> Just to confirm what I understood, when we are reading a bitmap with
> 'x-dirty-bitmap' (for powered on vm of course), the 'start' is always a
> logical offset no matter whether the record has 'offset' value or not. Is
> this correct?

Whether you are querying dirty bitmaps (x-dirty-bitmap on command
line) or normal allocation (omitted), yes, the 'start' lists the
logical offset of each extent listed, where the extents correspond to
the offsets that a read over the same connection would access.

> 
> Also, I came across a case wherein we get the entire disk as allocated for
> a raw format disk which is present on lvm or lvm-thin storage (the disk has
> just a few MB data added, and the vm is in running state). Here is an
> example of 1Gb data. Is this expected behaviour?
> [{ "start": 0, "length": 1073741824, "depth": 0, "present": true, "zero":
> false, "data": true, "compressed": false, "offset": 0}]

For raw images, the ability to report holes depends on how well
lseek(SEEK_DATA) works; this is filesystem dependent (for example, for
the longest time, tmpfs had O(n) rather than O(1) performance for a
single call, making an lseek() map of the extents of the entire file
an untenable O(n^2) effort, so we purposefully avoid lseek when it is
not known to be efficient).  I would LOVE it if the kernel supported
lseek(SEEK_DATA) on block devices - in fact, here's a patch series
that Stefan started where we debated what that might look like,
although it never gained any traction at the time:
https://lore.kernel.org/lkml/20240328203910.2370087-1-stefanha@redhat.com/

It may be also possible for qemu to use ioctls to probe block device
extents when lseek() doesn't directly work, but patches would have to
be submitted, and that won't scale as well as having the kernel report
the information up front to all interested users, rather than patching
each client to learn the right ioctls to work around the kernel's lack
of a unified interface.

So in the short term, yes, it is reasonable to expect that qemu is not
able to report where the sparse regions of an lvm block device are.
Note that something reported as full data is always accurate, even if
inefficient.

One other side note - a few months back, I was working on a potential
project to write a CSI driver that used lvm devices, and was working
on what it would take for that CSI driver to expose the
'GetMetadataAllocated' and 'GetMetadataDelta' gRPC calls.  lvm code
did not, at the time, provide any convenient way to list which
portions of a thin volume were directly allocated or which were dirty
in relation to a prior snapshot.  There might be some hacks you can do
with device-mapper code to get at that, or newer versions of lvm code
might add something along those lines; but that was another place
where I would have loved to have a kernel interface for letting
seek(SEEK_DATA) expose where the allocations live.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org