[RFC] block: add an overlaybd image format driver

[RFC] block: add an overlaybd image format driver

[Huiba Li]

[-- Attachment #1: Type: text/plain, Size: 3594 bytes --]

Hi,

I would like to propose adding a QEMU block driver for the
overlaybd image format, and I am looking for early feedback on
scope and design before sending patches.

What overlaybd is
-----------------
Overlaybd is an open-source image format (
https://containerd.github.io/overlaybd/)
hosted as a sub-project of containerd (CNCF graduated). A disk
image is a stack of read-only snapshot blobs plus a writable top
file, in a layout compatible with OCI image registries: a base
image is stored once and shared by every derivative image, and
data is fetched on demand rather than pulled in full before boot.

Two properties distinguish it from qcow2/VHDX/VMDK-style chains:

  * Lookup cost is O(1) in chain depth. At open time the per-file
    indices are merged into a single in-memory index (an
    extent-based log-structured merge tree), so a read never walks
    backing files, and per-file index caches do not multiply with
    snapshots.

  * The merged index is small (about 300 KB average for 50+ GB
    images in our production data), and because extents are
    variable-length, writes land at 512-byte granularity with no
    cluster copy-on-write.

The format is openly specified:
  https://github.com/containerd/overlaybd/tree/main/docs/specs/lsmt.md
  https://github.com/containerd/overlaybd/tree/main/docs/specs/zfile.md

Reference implementation:
  https://github.com/containerd/overlaybd

Overlaybd has been in production at Alibaba for years (Taobao,
Tmall, Alibaba Cloud, Function Compute) and is used by Azure AKS
Artifact Streaming, Databricks serverless compute, and several
microVM-based sandbox platforms. The design is documented in two
USENIX ATC papers (DADI, ATC'20; FaaSNet, ATC'21). I am one of the
overlaybd/DADI authors.

Why a QEMU driver
-----------------
Today overlaybd images are served to VMs through a kernel blk dev
and a userspace daemon. That fits container workloads well, but
a native QEMU driver would let any QEMU-based stack (libvirt,
KubeVirt, OpenStack, plain qemu-system-*) boot overlaybd images
directly, with QEMU's usual amenities (backing files, block jobs,
throttling, migration) and no extra daemon on the host. This is
beneficial to agent sandboxes.

Proposed implementation
-----------------------
A new block format driver, block/overlaybd.c, written in C from the
open spec (like the existing vmdk/vhdx drivers) rather than binding
the C++ reference implementation:

  * v1: read-only support. Local layer files, index merge at open,
    raw and ZFile blobs (lz4/zstd are already optional QEMU
    dependencies). Writes can be served by a qcow2/raw file on top,
    exactly like booting a read-only backing image today.
  * v2: native writable layer, and lazy fetching of remote blobs —
    either composed over the existing curl driver or with a small
    built-in HTTP fetcher; guidance welcome.

Questions
---------
  1. Is a new in-tree format driver acceptable in principle, given
     the format is stable, openly specified, and deployed at scale?
  2. Is read-only-first a reasonable merge scope for the initial
     series (with iotests and a docs/interop/ spec page)?
  3. For remote blobs: compose over block/curl, or fetch in-driver
     with a local cache?
  4. CLI surface: point -drive at the OCI-style JSON image config
     (which lists the layer blobs), or expose per-layer options?

I will write the iotests and the docs/interop format page, and add
myself to MAINTAINERS in the first series.

Thanks for any comments,
Huiba Li

[-- Attachment #2: Type: text/html, Size: 4114 bytes --]

Re: [RFC] block: add an overlaybd image format driver

[Huiba Li]

Hi all,

Gentle ping on this RFC from Aug 18.

In short: overlaybd is an openly specified image format (CNCF-hosted;
read-only blob stack + writable top, O(1) lookup regardless of chain
depth) deployed at scale for container/VM image streaming. The RFC
proposes an in-tree C block driver, starting with read-only support
(index merge at open, raw + ZFile layers), writes handled by a
qcow2/raw file on top.

I'd especially appreciate thoughts on the four questions in the RFC:

  1. Is a new in-tree format driver acceptable in principle?
  2. Is read-only-first a reasonable initial scope (with iotests and a
     docs/interop spec page)?
  3. Remote blobs: compose over block/curl, or fetch in-driver?
  4. CLI surface: point -drive at the OCI-style JSON config, or
     per-layer options?

Question 1 is the main gate -- if in-principle feedback is easier to
give than a full design review, that alone would help me decide
whether to proceed with the first series.

Thanks,
Huiba


Huiba Li <lihuiba@gmail.com> 于2026年8月18日周二 17:38写道：
>
> Hi,
>
> I would like to propose adding a QEMU block driver for the
> overlaybd image format, and I am looking for early feedback on
> scope and design before sending patches.
>
> What overlaybd is
> -----------------
> Overlaybd is an open-source image format (https://containerd.github.io/overlaybd/)
> hosted as a sub-project of containerd (CNCF graduated). A disk
> image is a stack of read-only snapshot blobs plus a writable top
> file, in a layout compatible with OCI image registries: a base
> image is stored once and shared by every derivative image, and
> data is fetched on demand rather than pulled in full before boot.
>
> Two properties distinguish it from qcow2/VHDX/VMDK-style chains:
>
>   * Lookup cost is O(1) in chain depth. At open time the per-file
>     indices are merged into a single in-memory index (an
>     extent-based log-structured merge tree), so a read never walks
>     backing files, and per-file index caches do not multiply with
>     snapshots.
>
>   * The merged index is small (about 300 KB average for 50+ GB
>     images in our production data), and because extents are
>     variable-length, writes land at 512-byte granularity with no
>     cluster copy-on-write.
>
> The format is openly specified:
>   https://github.com/containerd/overlaybd/tree/main/docs/specs/lsmt.md
>   https://github.com/containerd/overlaybd/tree/main/docs/specs/zfile.md
>
> Reference implementation:
>   https://github.com/containerd/overlaybd
>
> Overlaybd has been in production at Alibaba for years (Taobao,
> Tmall, Alibaba Cloud, Function Compute) and is used by Azure AKS
> Artifact Streaming, Databricks serverless compute, and several
> microVM-based sandbox platforms. The design is documented in two
> USENIX ATC papers (DADI, ATC'20; FaaSNet, ATC'21). I am one of the
> overlaybd/DADI authors.
>
> Why a QEMU driver
> -----------------
> Today overlaybd images are served to VMs through a kernel blk dev
> and a userspace daemon. That fits container workloads well, but
> a native QEMU driver would let any QEMU-based stack (libvirt,
> KubeVirt, OpenStack, plain qemu-system-*) boot overlaybd images
> directly, with QEMU's usual amenities (backing files, block jobs,
> throttling, migration) and no extra daemon on the host. This is
> beneficial to agent sandboxes.
>
> Proposed implementation
> -----------------------
> A new block format driver, block/overlaybd.c, written in C from the
> open spec (like the existing vmdk/vhdx drivers) rather than binding
> the C++ reference implementation:
>
>   * v1: read-only support. Local layer files, index merge at open,
>     raw and ZFile blobs (lz4/zstd are already optional QEMU
>     dependencies). Writes can be served by a qcow2/raw file on top,
>     exactly like booting a read-only backing image today.
>   * v2: native writable layer, and lazy fetching of remote blobs —
>     either composed over the existing curl driver or with a small
>     built-in HTTP fetcher; guidance welcome.
>
> Questions
> ---------
>   1. Is a new in-tree format driver acceptable in principle, given
>      the format is stable, openly specified, and deployed at scale?
>   2. Is read-only-first a reasonable merge scope for the initial
>      series (with iotests and a docs/interop/ spec page)?
>   3. For remote blobs: compose over block/curl, or fetch in-driver
>      with a local cache?
>   4. CLI surface: point -drive at the OCI-style JSON image config
>      (which lists the layer blobs), or expose per-layer options?
>
> I will write the iotests and the docs/interop format page, and add
> myself to MAINTAINERS in the first series.
>
> Thanks for any comments,
> Huiba Li

Re: [RFC] block: add an overlaybd image format driver

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5321 bytes --]

On Tue, Sep 01, 2026 at 11:09:56AM +0800, Huiba Li wrote:
> Hi all,
> 
> Gentle ping on this RFC from Aug 18.

Hi Huiba,
Sorry your RFC was not seen. Kevin is on vacation and I will take a look
in the meantime. Please CC stefanha@redhat.com in future emails so it
comes to my attention.

Thanks!

Stefan

> 
> In short: overlaybd is an openly specified image format (CNCF-hosted;
> read-only blob stack + writable top, O(1) lookup regardless of chain
> depth) deployed at scale for container/VM image streaming. The RFC
> proposes an in-tree C block driver, starting with read-only support
> (index merge at open, raw + ZFile layers), writes handled by a
> qcow2/raw file on top.
> 
> I'd especially appreciate thoughts on the four questions in the RFC:
> 
>   1. Is a new in-tree format driver acceptable in principle?
>   2. Is read-only-first a reasonable initial scope (with iotests and a
>      docs/interop spec page)?
>   3. Remote blobs: compose over block/curl, or fetch in-driver?
>   4. CLI surface: point -drive at the OCI-style JSON config, or
>      per-layer options?
> 
> Question 1 is the main gate -- if in-principle feedback is easier to
> give than a full design review, that alone would help me decide
> whether to proceed with the first series.
> 
> Thanks,
> Huiba
> 
> 
> Huiba Li <lihuiba@gmail.com> 于2026年8月18日周二 17:38写道：
> >
> > Hi,
> >
> > I would like to propose adding a QEMU block driver for the
> > overlaybd image format, and I am looking for early feedback on
> > scope and design before sending patches.
> >
> > What overlaybd is
> > -----------------
> > Overlaybd is an open-source image format (https://containerd.github.io/overlaybd/)
> > hosted as a sub-project of containerd (CNCF graduated). A disk
> > image is a stack of read-only snapshot blobs plus a writable top
> > file, in a layout compatible with OCI image registries: a base
> > image is stored once and shared by every derivative image, and
> > data is fetched on demand rather than pulled in full before boot.
> >
> > Two properties distinguish it from qcow2/VHDX/VMDK-style chains:
> >
> >   * Lookup cost is O(1) in chain depth. At open time the per-file
> >     indices are merged into a single in-memory index (an
> >     extent-based log-structured merge tree), so a read never walks
> >     backing files, and per-file index caches do not multiply with
> >     snapshots.
> >
> >   * The merged index is small (about 300 KB average for 50+ GB
> >     images in our production data), and because extents are
> >     variable-length, writes land at 512-byte granularity with no
> >     cluster copy-on-write.
> >
> > The format is openly specified:
> >   https://github.com/containerd/overlaybd/tree/main/docs/specs/lsmt.md
> >   https://github.com/containerd/overlaybd/tree/main/docs/specs/zfile.md
> >
> > Reference implementation:
> >   https://github.com/containerd/overlaybd
> >
> > Overlaybd has been in production at Alibaba for years (Taobao,
> > Tmall, Alibaba Cloud, Function Compute) and is used by Azure AKS
> > Artifact Streaming, Databricks serverless compute, and several
> > microVM-based sandbox platforms. The design is documented in two
> > USENIX ATC papers (DADI, ATC'20; FaaSNet, ATC'21). I am one of the
> > overlaybd/DADI authors.
> >
> > Why a QEMU driver
> > -----------------
> > Today overlaybd images are served to VMs through a kernel blk dev
> > and a userspace daemon. That fits container workloads well, but
> > a native QEMU driver would let any QEMU-based stack (libvirt,
> > KubeVirt, OpenStack, plain qemu-system-*) boot overlaybd images
> > directly, with QEMU's usual amenities (backing files, block jobs,
> > throttling, migration) and no extra daemon on the host. This is
> > beneficial to agent sandboxes.
> >
> > Proposed implementation
> > -----------------------
> > A new block format driver, block/overlaybd.c, written in C from the
> > open spec (like the existing vmdk/vhdx drivers) rather than binding
> > the C++ reference implementation:
> >
> >   * v1: read-only support. Local layer files, index merge at open,
> >     raw and ZFile blobs (lz4/zstd are already optional QEMU
> >     dependencies). Writes can be served by a qcow2/raw file on top,
> >     exactly like booting a read-only backing image today.
> >   * v2: native writable layer, and lazy fetching of remote blobs —
> >     either composed over the existing curl driver or with a small
> >     built-in HTTP fetcher; guidance welcome.
> >
> > Questions
> > ---------
> >   1. Is a new in-tree format driver acceptable in principle, given
> >      the format is stable, openly specified, and deployed at scale?
> >   2. Is read-only-first a reasonable merge scope for the initial
> >      series (with iotests and a docs/interop/ spec page)?
> >   3. For remote blobs: compose over block/curl, or fetch in-driver
> >      with a local cache?
> >   4. CLI surface: point -drive at the OCI-style JSON image config
> >      (which lists the layer blobs), or expose per-layer options?
> >
> > I will write the iotests and the docs/interop format page, and add
> > myself to MAINTAINERS in the first series.
> >
> > Thanks for any comments,
> > Huiba Li
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC] block: add an overlaybd image format driver

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 284 bytes --]

Hi Huiba,
One thing I noticed is that the append-only format used for the upper
write layer is not documented. I guess it's different from the read-only
LSTM format since the index won't be kept in sorted order when
appending.

Do you want to document the append-only format?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC] block: add an overlaybd image format driver

[Huiba Li]

> One thing I noticed is that the append-only format used for the upper
> write layer is not documented.

I have added the spec for writable layers:
https://github.com/containerd/overlaybd/pull/451/changes

Stefan Hajnoczi <stefanha@redhat.com> 于2026年9月2日周三 19:54写道：
>
> Hi Huiba,
> One thing I noticed is that the append-only format used for the upper
> write layer is not documented. I guess it's different from the read-only
> LSTM format since the index won't be kept in sorted order when
> appending.
>
> Do you want to document the append-only format?
>
> Stefan

Re: [RFC] block: add an overlaybd image format driver

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

On Thu, Sep 03, 2026 at 05:27:28PM +0800, Huiba Li wrote:
> > One thing I noticed is that the append-only format used for the upper
> > write layer is not documented.
> 
> I have added the spec for writable layers:
> https://github.com/containerd/overlaybd/pull/451/changes

Thanks, that clarifies things.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC] block: add an overlaybd image format driver

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5610 bytes --]

On Tue, Aug 18, 2026 at 05:38:38PM +0800, Huiba Li wrote:
> Hi,
> 
> I would like to propose adding a QEMU block driver for the
> overlaybd image format, and I am looking for early feedback on
> scope and design before sending patches.
> 
> What overlaybd is
> -----------------
> Overlaybd is an open-source image format (
> https://containerd.github.io/overlaybd/)
> hosted as a sub-project of containerd (CNCF graduated). A disk
> image is a stack of read-only snapshot blobs plus a writable top
> file, in a layout compatible with OCI image registries: a base
> image is stored once and shared by every derivative image, and
> data is fetched on demand rather than pulled in full before boot.

This looks interesting. I've pondered whether the layered and immutable
image trends in recent years can be used effectively for virtual
machines. I will take a look at the specs you linked to understand the
internals better, but in the meantime some answers to your questions
below...

> 
> Two properties distinguish it from qcow2/VHDX/VMDK-style chains:
> 
>   * Lookup cost is O(1) in chain depth. At open time the per-file
>     indices are merged into a single in-memory index (an
>     extent-based log-structured merge tree), so a read never walks
>     backing files, and per-file index caches do not multiply with
>     snapshots.
> 
>   * The merged index is small (about 300 KB average for 50+ GB
>     images in our production data), and because extents are
>     variable-length, writes land at 512-byte granularity with no
>     cluster copy-on-write.
> 
> The format is openly specified:
>   https://github.com/containerd/overlaybd/tree/main/docs/specs/lsmt.md
>   https://github.com/containerd/overlaybd/tree/main/docs/specs/zfile.md
> 
> Reference implementation:
>   https://github.com/containerd/overlaybd
> 
> Overlaybd has been in production at Alibaba for years (Taobao,
> Tmall, Alibaba Cloud, Function Compute) and is used by Azure AKS
> Artifact Streaming, Databricks serverless compute, and several
> microVM-based sandbox platforms. The design is documented in two
> USENIX ATC papers (DADI, ATC'20; FaaSNet, ATC'21). I am one of the
> overlaybd/DADI authors.
> 
> Why a QEMU driver
> -----------------
> Today overlaybd images are served to VMs through a kernel blk dev
> and a userspace daemon. That fits container workloads well, but
> a native QEMU driver would let any QEMU-based stack (libvirt,
> KubeVirt, OpenStack, plain qemu-system-*) boot overlaybd images
> directly, with QEMU's usual amenities (backing files, block jobs,
> throttling, migration) and no extra daemon on the host. This is
> beneficial to agent sandboxes.
> 
> Proposed implementation
> -----------------------
> A new block format driver, block/overlaybd.c, written in C from the
> open spec (like the existing vmdk/vhdx drivers) rather than binding
> the C++ reference implementation:
> 
>   * v1: read-only support. Local layer files, index merge at open,
>     raw and ZFile blobs (lz4/zstd are already optional QEMU
>     dependencies). Writes can be served by a qcow2/raw file on top,
>     exactly like booting a read-only backing image today.
>   * v2: native writable layer, and lazy fetching of remote blobs —
>     either composed over the existing curl driver or with a small
>     built-in HTTP fetcher; guidance welcome.
> 
> Questions
> ---------
>   1. Is a new in-tree format driver acceptable in principle, given
>      the format is stable, openly specified, and deployed at scale?

Yes.

There are other options like writing a FUSE, iSCSI, or NBD daemon. Linux
ublk (https://www.kernel.org/doc/html/latest/block/ublk.html) is a newer
option with potentially better performance than other daemon approaches.

The advantage of a block driver is that it has better integration with
QEMU and may avoid the need for privileges. The downside is that it's
extra work to integrate with QEMU (and add libvirt support) that
non-QEMU use cases don't benefit from the QEMU block driver.

The choice is yours. There is no fundamental blocker from the QEMU side
if you're willing to write the QEMU code and the format is an open spec.

>   2. Is read-only-first a reasonable merge scope for the initial
>      series (with iotests and a docs/interop/ spec page)?

Yes.

>   3. For remote blobs: compose over block/curl, or fetch in-driver
>      with a local cache?

QEMU's block layer is designed around graphs of block driver nodes.
Reusing the curl block driver would be a natural choice that avoids code
duplication.

However, if you hit issues and it becomes clear that directly calling
libcurl is a better solution, then that's an option too.

>   4. CLI surface: point -drive at the OCI-style JSON image config
>      (which lists the layer blobs), or expose per-layer options?

Both :). Per-layer options allow for run-time (re)configuration or
hotplug using blockdev-add and similar monitor commands. Users may
prefer to point QEMU at a JSON file rather than building long
command-line options.

> 
> I will write the iotests and the docs/interop format page, and add
> myself to MAINTAINERS in the first series.

You do not need to document the on-disk format in docs/interop/ if you
already maintain the spec in
https://github.com/containerd/overlaybd.git. It would be fine to include
a comment in block/overlaybd.c with the spec URLs.

If it helps we could have a call with Kevin Wolf in about 2 weeks to
discuss any topics you have in more depth.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC] block: add an overlaybd image format driver

[Huiba Li]

Hi Stefan,

> This looks interesting. I've pondered whether the layered and immutable
> image trends in recent years can be used effectively for virtual
> machines.

There are already some examples:
https://www.databricks.com/blog/booting-databricks-vms-7x-faster-serverless-compute
https://community.fly.io/t/experimental-speedy-machine-creation-with-overlaybd/18958
And secure containers, agent sandboxes are special forms of VMs.


> The advantage of a block driver is that it has better integration with
> QEMU and may avoid the need for privileges.

> By the way, I see that overylaybd already supports tcmu-runner, so I
> guess you've decided that the benefits of integrating directly into QEMU
> are worth it.

Yes.


> >   4. CLI surface: point -drive at the OCI-style JSON image config
> >      (which lists the layer blobs), or expose per-layer options?
> Both :). Per-layer options allow for run-time (re)configuration or
> hotplug using blockdev-add and similar monitor commands. Users may
> prefer to point QEMU at a JSON file rather than building long
> command-line options.

OK


> One thing I noticed is that the append-only format used for the upper
> write layer is not documented. I guess it's different from the read-only
> LSTM format since the index won't be kept in sorted order when
> appending.
> Do you want to document the append-only format?

Yes, I'll document it later. It's only slightly different from the
read-only one.


Thanks

Huiba

Re: [RFC] block: add an overlaybd image format driver

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 164 bytes --]

By the way, I see that overylaybd already supports tcmu-runner, so I
guess you've decided that the benefits of integrating directly into QEMU
are worth it.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]