Re: [RFC] block: add an overlaybd image format driver

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5610 bytes --]

On Tue, Aug 18, 2026 at 05:38:38PM +0800, Huiba Li wrote:
> Hi,
> 
> I would like to propose adding a QEMU block driver for the
> overlaybd image format, and I am looking for early feedback on
> scope and design before sending patches.
> 
> What overlaybd is
> -----------------
> Overlaybd is an open-source image format (
> https://containerd.github.io/overlaybd/)
> hosted as a sub-project of containerd (CNCF graduated). A disk
> image is a stack of read-only snapshot blobs plus a writable top
> file, in a layout compatible with OCI image registries: a base
> image is stored once and shared by every derivative image, and
> data is fetched on demand rather than pulled in full before boot.

This looks interesting. I've pondered whether the layered and immutable
image trends in recent years can be used effectively for virtual
machines. I will take a look at the specs you linked to understand the
internals better, but in the meantime some answers to your questions
below...

> 
> Two properties distinguish it from qcow2/VHDX/VMDK-style chains:
> 
>   * Lookup cost is O(1) in chain depth. At open time the per-file
>     indices are merged into a single in-memory index (an
>     extent-based log-structured merge tree), so a read never walks
>     backing files, and per-file index caches do not multiply with
>     snapshots.
> 
>   * The merged index is small (about 300 KB average for 50+ GB
>     images in our production data), and because extents are
>     variable-length, writes land at 512-byte granularity with no
>     cluster copy-on-write.
> 
> The format is openly specified:
>   https://github.com/containerd/overlaybd/tree/main/docs/specs/lsmt.md
>   https://github.com/containerd/overlaybd/tree/main/docs/specs/zfile.md
> 
> Reference implementation:
>   https://github.com/containerd/overlaybd
> 
> Overlaybd has been in production at Alibaba for years (Taobao,
> Tmall, Alibaba Cloud, Function Compute) and is used by Azure AKS
> Artifact Streaming, Databricks serverless compute, and several
> microVM-based sandbox platforms. The design is documented in two
> USENIX ATC papers (DADI, ATC'20; FaaSNet, ATC'21). I am one of the
> overlaybd/DADI authors.
> 
> Why a QEMU driver
> -----------------
> Today overlaybd images are served to VMs through a kernel blk dev
> and a userspace daemon. That fits container workloads well, but
> a native QEMU driver would let any QEMU-based stack (libvirt,
> KubeVirt, OpenStack, plain qemu-system-*) boot overlaybd images
> directly, with QEMU's usual amenities (backing files, block jobs,
> throttling, migration) and no extra daemon on the host. This is
> beneficial to agent sandboxes.
> 
> Proposed implementation
> -----------------------
> A new block format driver, block/overlaybd.c, written in C from the
> open spec (like the existing vmdk/vhdx drivers) rather than binding
> the C++ reference implementation:
> 
>   * v1: read-only support. Local layer files, index merge at open,
>     raw and ZFile blobs (lz4/zstd are already optional QEMU
>     dependencies). Writes can be served by a qcow2/raw file on top,
>     exactly like booting a read-only backing image today.
>   * v2: native writable layer, and lazy fetching of remote blobs —
>     either composed over the existing curl driver or with a small
>     built-in HTTP fetcher; guidance welcome.
> 
> Questions
> ---------
>   1. Is a new in-tree format driver acceptable in principle, given
>      the format is stable, openly specified, and deployed at scale?

Yes.

There are other options like writing a FUSE, iSCSI, or NBD daemon. Linux
ublk (https://www.kernel.org/doc/html/latest/block/ublk.html) is a newer
option with potentially better performance than other daemon approaches.

The advantage of a block driver is that it has better integration with
QEMU and may avoid the need for privileges. The downside is that it's
extra work to integrate with QEMU (and add libvirt support) that
non-QEMU use cases don't benefit from the QEMU block driver.

The choice is yours. There is no fundamental blocker from the QEMU side
if you're willing to write the QEMU code and the format is an open spec.

>   2. Is read-only-first a reasonable merge scope for the initial
>      series (with iotests and a docs/interop/ spec page)?

Yes.

>   3. For remote blobs: compose over block/curl, or fetch in-driver
>      with a local cache?

QEMU's block layer is designed around graphs of block driver nodes.
Reusing the curl block driver would be a natural choice that avoids code
duplication.

However, if you hit issues and it becomes clear that directly calling
libcurl is a better solution, then that's an option too.

>   4. CLI surface: point -drive at the OCI-style JSON image config
>      (which lists the layer blobs), or expose per-layer options?

Both :). Per-layer options allow for run-time (re)configuration or
hotplug using blockdev-add and similar monitor commands. Users may
prefer to point QEMU at a JSON file rather than building long
command-line options.

> 
> I will write the iotests and the docs/interop format page, and add
> myself to MAINTAINERS in the first series.

You do not need to document the on-disk format in docs/interop/ if you
already maintain the spec in
https://github.com/containerd/overlaybd.git. It would be fine to include
a comment in block/overlaybd.c with the spec URLs.

If it helps we could have a call with Kevin Wolf in about 2 weeks to
discuss any topics you have in more depth.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]