[PATCH v2 0/2] FBE virtualization: inline encryption for virtio-blk guests

[PATCH v2 0/2] FBE virtualization: inline encryption for virtio-blk guests

[Linlin Zhang]

From: linlzhan <linlin.zhang@oss.qualcomm.com>

The virtio-blk driver currently does not preserve blk-crypto metadata when
dispatching encrypted bios to the virtio queue. As a result, inline
encryption cannot be used for virtio block devices.

This series enables inline encryption for guest VMs on platforms where the
Inline Crypto Engine (ICE) is owned by the host or another virtual machine.
It extends virtio-blk with a crypto control virtqueue and carries the
required encryption metadata with data requests.

The series consists of:

  1. Add control virtqueue support.
     This allows the guest to exchange key management requests with the
     virtio-blk backend without mixing them with regular I/O requests.

  2. Add inline encryption support.
     The driver advertises the device encryption capabilities through a
     struct blk_crypto_profile and carries encryption metadata, including
     the virtual keyslot and data unit number (DUN), in virtio requests.

On the backend (blk-crypto-proxy, will be commited in another patch as
suggested.), the key table mapping virtual slot to the block crypto
key is maintained, so that the request metadata can be used to resolve
the guest keyslot to the corresponding block crypto key and to reconstruct
the blk-crypto context before submitting the bio to the underlying block
device. 

This keeps the guest integrated with the existing blk-crypto and filesystem
encryption frameworks while preserving inline-encryption semantics across
the virtualization boundary.

This is compatible for the virtio SPEC update which is under review:
https://lore.kernel.org/all/20260913161628.368484-1-linlin.zhang@oss.qualcomm.com/

Known limitations:
  - Virtio block inline encryption depends on the new control virtqueue
  - Inline encryption is mutually exclusive with VIRTIO_BLK_F_ZONED.

Testing:
Compilation pass on Linux-next.
End-to-end FBE virtualization with wrapped key enabled was validated
on top of gunyah hypervisor.  wrapped_key_test is a local utility to
get wrapped key and ephemeral wrapped key via storage ioctl interfaces.
  - /data/wrapped_key_test /dev/block/userdata generate
  - /data/wrapped_key_test /dev/block/userdata prepare /data/lt_key.bin
  - /data/fscryptctl insert_wrapped_key < /data/eph_key.bin
  - /data/fscryptctl set_policy --identifier=20f553802e64e36b43469211266a5f1c /data/testing
  - echo "data" > /data/testing/file.txt
  - sync and reboot
  - /data/wrapped_key_test /dev/block/userdata prepare /data/lt_key.bin
  - /data/fscryptctl insert_wrapped_key < /data/eph_key_2.bin
  - /data/fscryptctl set_policy --identifier=d8ca51d6d2094b73b2dae5ee7e3a10b6 /data/testing
  - cat /data/testing/file.txt

---
Changes v1 => v2:
  - Use control virtqueue to perform key management requests
  - Extend virtio_blk_crypto_msg::dun from a single __virtio64 to a
    four-element __virtio64 array to support larger DUN sizes.
  - Remove data_unit_size_bit in virtio_blk_crypto_msg struct
v1: https://lore.kernel.org/all/20260827160806.1295313-1-linlin.zhang@oss.qualcomm.com/

linlzhan (2):
  virtio_blk: Add control virtqueue support
  virtio_blk: add inline encryption support

 drivers/block/Kconfig           |  12 +
 drivers/block/virtio_blk.c      | 753 +++++++++++++++++++++++++++++++-
 include/uapi/linux/virtio_blk.h | 116 +++++
 3 files changed, 865 insertions(+), 16 deletions(-)

-- 
2.34.1

[PATCH v2 1/2] virtio_blk: Add control virtqueue support

[Linlin Zhang]

From: linlzhan <linlin.zhang@oss.qualcomm.com>

Add support for the optional virtio-blk control virtqueue.

If control queue feature bit is negociated, this allows the driver
to manage control-queue requests independently from the data path
and to safely handle outstanding requests during device removal
and suspend.

No control command is submitted by this change. The control virtqueue
will be used by a subsequent inline encryption implementation.

Signed-off-by: linlzhan <linlin.zhang@oss.qualcomm.com>
---
 drivers/block/virtio_blk.c      | 179 +++++++++++++++++++++++++++++++-
 include/uapi/linux/virtio_blk.h |   1 +
 2 files changed, 176 insertions(+), 4 deletions(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 32bf3ba07a9d..252ec9366153 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -6,6 +6,7 @@
 #include <linux/hdreg.h>
 #include <linux/module.h>
 #include <linux/mutex.h>
+#include <linux/completion.h>
 #include <linux/interrupt.h>
 #include <linux/virtio.h>
 #include <linux/virtio_blk.h>
@@ -52,6 +53,15 @@ struct virtio_blk_vq {
 	char name[VQ_NAME_LEN];
 } ____cacheline_aligned_in_smp;
 
+struct virtio_blk_ctrl_vq {
+	struct virtqueue *vq;
+	struct mutex mutex;
+	spinlock_t lock;
+	unsigned int inflight;
+	bool dead;
+	struct completion drained;
+};
+
 struct virtio_blk {
 	/*
 	 * This mutex must be held by anything that may run after
@@ -83,6 +93,9 @@ struct virtio_blk {
 
 	/* For zoned device */
 	unsigned int zone_sectors;
+
+	/* Control virtqueue state. */
+	struct virtio_blk_ctrl_vq ctrl_vq;
 };
 
 struct virtblk_req {
@@ -110,6 +123,12 @@ struct virtblk_req {
 	struct scatterlist sg[];
 };
 
+struct virtblk_ctrl_request {
+	__virtio32 type;
+	u8 status;
+	struct completion compl;
+};
+
 static inline blk_status_t virtblk_result(u8 status)
 {
 	switch (status) {
@@ -863,11 +882,131 @@ static int virtblk_getgeo(struct gendisk *disk, struct hd_geometry *geo)
 	return ret;
 }
 
+#define VIRTBLK_CTRL_VQ_DRAIN_TIMEOUT (10 * HZ)
+
+/* Prevent new submissions and wait for in-flight requests to complete. */
+static void virtblk_ctrl_vq_quiesce(struct virtio_blk *vblk)
+{
+	unsigned long flags;
+	bool need_wait;
+
+	if (!vblk->ctrl_vq.vq)
+		return;
+
+	init_completion(&vblk->ctrl_vq.drained);
+
+	spin_lock_irqsave(&vblk->ctrl_vq.lock, flags);
+	vblk->ctrl_vq.dead = true;
+	need_wait = vblk->ctrl_vq.inflight != 0;
+	spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+
+	if (need_wait &&
+	    !wait_for_completion_timeout(&vblk->ctrl_vq.drained,
+					  VIRTBLK_CTRL_VQ_DRAIN_TIMEOUT))
+		dev_warn(&vblk->vdev->dev,
+			 "timed out waiting for control queue requests to complete\n");
+}
+
+/* Fail requests left in the control queue after reset. */
+static void virtblk_ctrl_vq_drain(struct virtio_blk *vblk)
+{
+	struct virtblk_ctrl_request *creq;
+	unsigned long flags;
+
+	if (!vblk->ctrl_vq.vq)
+		return;
+
+	spin_lock_irqsave(&vblk->ctrl_vq.lock, flags);
+	while ((creq = virtqueue_detach_unused_buf(vblk->ctrl_vq.vq)) != NULL) {
+		if (WARN_ON_ONCE(!vblk->ctrl_vq.inflight))
+			;
+		else
+			vblk->ctrl_vq.inflight--;
+		creq->status = VIRTIO_BLK_S_IOERR;
+		spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+		complete(&creq->compl);
+		spin_lock_irqsave(&vblk->ctrl_vq.lock, flags);
+	}
+	spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+}
+
+static void virtblk_ctrlq_callback(struct virtqueue *vq)
+{
+	struct virtio_blk *vblk = vq->vdev->priv;
+	struct virtblk_ctrl_request *creq;
+	unsigned long flags;
+	unsigned int len;
+
+	spin_lock_irqsave(&vblk->ctrl_vq.lock, flags);
+	do {
+		virtqueue_disable_cb(vq);
+		while ((creq = virtqueue_get_buf(vq, &len)) != NULL) {
+			bool drained = false;
+
+			if (WARN_ON_ONCE(!vblk->ctrl_vq.inflight)) {
+				/*
+				 * Still complete the request.  Never leave a
+				 * synchronous caller blocked because the accounting
+				 * state was already inconsistent.
+				 */
+				spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+				complete(&creq->compl);
+				spin_lock_irqsave(&vblk->ctrl_vq.lock, flags);
+				continue;
+			}
+
+			if (--vblk->ctrl_vq.inflight == 0 && vblk->ctrl_vq.dead)
+				drained = true;
+			spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+			if (drained)
+				complete(&vblk->ctrl_vq.drained);
+			complete(&creq->compl);
+			spin_lock_irqsave(&vblk->ctrl_vq.lock, flags);
+		}
+	} while (!virtqueue_enable_cb(vq));
+	spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+}
+
+/* Submit a control-queue request and wait for completion. */
+static int virtblk_ctrl_vq_request(struct virtio_blk *vblk,
+				    struct virtblk_ctrl_request *creq,
+				    struct scatterlist *sgs[],
+				    unsigned int out_sgs, unsigned int in_sgs)
+{
+	unsigned long flags;
+	int err;
+
+	mutex_lock(&vblk->ctrl_vq.mutex);
+	init_completion(&creq->compl);
+
+	spin_lock_irqsave(&vblk->ctrl_vq.lock, flags);
+	if (vblk->ctrl_vq.dead) {
+		spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+		mutex_unlock(&vblk->ctrl_vq.mutex);
+		return -ENODEV;
+	}
+	err = virtqueue_add_sgs(vblk->ctrl_vq.vq, sgs, out_sgs, in_sgs, creq, GFP_ATOMIC);
+	if (!err) {
+		vblk->ctrl_vq.inflight++;
+		virtqueue_kick(vblk->ctrl_vq.vq);
+	}
+	spin_unlock_irqrestore(&vblk->ctrl_vq.lock, flags);
+	if (err) {
+		mutex_unlock(&vblk->ctrl_vq.mutex);
+		return err;
+	}
+
+	wait_for_completion(&creq->compl);
+	mutex_unlock(&vblk->ctrl_vq.mutex);
+	return 0;
+}
+
 static void virtblk_free_disk(struct gendisk *disk)
 {
 	struct virtio_blk *vblk = disk->private_data;
 
 	ida_free(&vd_index_ida, vblk->index);
+	mutex_destroy(&vblk->ctrl_vq.mutex);
 	mutex_destroy(&vblk->vdev_mutex);
 	kfree(vblk);
 }
@@ -965,6 +1104,8 @@ static int init_vq(struct virtio_blk *vblk)
 	struct virtqueue **vqs;
 	unsigned short num_vqs;
 	unsigned short num_poll_vqs;
+	unsigned short total_vqs;
+	bool has_ctrl_vq;
 	struct virtio_device *vdev = vblk->vdev;
 	struct irq_affinity desc = { 0, };
 
@@ -993,12 +1134,19 @@ static int init_vq(struct virtio_blk *vblk)
 				vblk->io_queues[HCTX_TYPE_READ],
 				vblk->io_queues[HCTX_TYPE_POLL]);
 
+	/*
+	 * The control vq is appended after the data vqs whenever
+	 * F_CTRL_VQ is negotiated.
+	 */
+	has_ctrl_vq = virtio_has_feature(vdev, VIRTIO_BLK_F_CTRL_VQ);
+	total_vqs = num_vqs + (has_ctrl_vq ? 1 : 0);
+
 	vblk->vqs = kmalloc_objs(*vblk->vqs, num_vqs);
 	if (!vblk->vqs)
 		return -ENOMEM;
 
-	vqs_info = kzalloc_objs(*vqs_info, num_vqs);
-	vqs = kmalloc_objs(*vqs, num_vqs);
+	vqs_info = kzalloc_objs(*vqs_info, total_vqs);
+	vqs = kmalloc_objs(*vqs, total_vqs);
 	if (!vqs_info || !vqs) {
 		err = -ENOMEM;
 		goto out;
@@ -1015,8 +1163,13 @@ static int init_vq(struct virtio_blk *vblk)
 		vqs_info[i].name = vblk->vqs[i].name;
 	}
 
+	if (has_ctrl_vq) {
+		vqs_info[num_vqs].callback = virtblk_ctrlq_callback;
+		vqs_info[num_vqs].name = "control";
+	}
+
 	/* Discover virtqueues and write information to configuration.  */
-	err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info, &desc);
+	err = virtio_find_vqs(vdev, total_vqs, vqs, vqs_info, &desc);
 	if (err)
 		goto out;
 
@@ -1025,6 +1178,9 @@ static int init_vq(struct virtio_blk *vblk)
 		vblk->vqs[i].vq = vqs[i];
 	}
 	vblk->num_vqs = num_vqs;
+	vblk->ctrl_vq.vq = has_ctrl_vq ? vqs[num_vqs] : NULL;
+	vblk->ctrl_vq.dead = false;
+	vblk->ctrl_vq.inflight = 0;
 
 out:
 	kfree(vqs);
@@ -1464,14 +1620,18 @@ static int virtblk_probe(struct virtio_device *vdev)
 	}
 
 	mutex_init(&vblk->vdev_mutex);
+	mutex_init(&vblk->ctrl_vq.mutex);
+	spin_lock_init(&vblk->ctrl_vq.lock);
 
 	vblk->vdev = vdev;
 
 	INIT_WORK(&vblk->config_work, virtblk_config_changed_work);
 
 	err = init_vq(vblk);
-	if (err)
+	if (err) {
+		dev_err(&vdev->dev, "init virt queue failed: err = %d\n", err);
 		goto out_free_vblk;
+	}
 
 	/* Default queue sizing is to fill the ring. */
 	if (!virtblk_queue_depth) {
@@ -1553,6 +1713,7 @@ static int virtblk_probe(struct virtio_device *vdev)
 out_free_vq:
 	vdev->config->del_vqs(vdev);
 	kfree(vblk->vqs);
+	vblk->ctrl_vq.vq = NULL;
 out_free_vblk:
 	kfree(vblk);
 out_free_index:
@@ -1571,16 +1732,21 @@ static void virtblk_remove(struct virtio_device *vdev)
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
+	virtblk_ctrl_vq_quiesce(vblk);
+
 	mutex_lock(&vblk->vdev_mutex);
 
 	/* Stop all the virtqueues. */
 	virtio_reset_device(vdev);
+	virtblk_ctrl_vq_drain(vblk);
 
 	/* Virtqueues are stopped, nothing can use vblk->vdev anymore. */
 	vblk->vdev = NULL;
 
 	vdev->config->del_vqs(vdev);
 	kfree(vblk->vqs);
+	vblk->vqs = NULL;
+	vblk->ctrl_vq.vq = NULL;
 
 	mutex_unlock(&vblk->vdev_mutex);
 
@@ -1593,6 +1759,8 @@ static int virtblk_freeze_priv(struct virtio_device *vdev)
 	struct request_queue *q = vblk->disk->queue;
 	unsigned int memflags;
 
+	virtblk_ctrl_vq_quiesce(vblk);
+
 	/* Ensure no requests in virtqueues before deleting vqs. */
 	memflags = blk_mq_freeze_queue(q);
 	blk_mq_quiesce_queue_nowait(q);
@@ -1600,6 +1768,7 @@ static int virtblk_freeze_priv(struct virtio_device *vdev)
 
 	/* Ensure we don't receive any more interrupts */
 	virtio_reset_device(vdev);
+	virtblk_ctrl_vq_drain(vblk);
 
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
@@ -1612,6 +1781,7 @@ static int virtblk_freeze_priv(struct virtio_device *vdev)
 	 * pointers safely.
 	 */
 	vblk->vqs = NULL;
+	vblk->ctrl_vq.vq = NULL;
 
 	return 0;
 }
@@ -1672,6 +1842,7 @@ static unsigned int features[] = {
 	VIRTIO_BLK_F_FLUSH, VIRTIO_BLK_F_TOPOLOGY, VIRTIO_BLK_F_CONFIG_WCE,
 	VIRTIO_BLK_F_MQ, VIRTIO_BLK_F_DISCARD, VIRTIO_BLK_F_WRITE_ZEROES,
 	VIRTIO_BLK_F_SECURE_ERASE, VIRTIO_BLK_F_ZONED,
+	VIRTIO_BLK_F_CTRL_VQ,
 };
 
 static struct virtio_driver virtio_blk = {
diff --git a/include/uapi/linux/virtio_blk.h b/include/uapi/linux/virtio_blk.h
index 3744e4da1b2a..0a16972a1535 100644
--- a/include/uapi/linux/virtio_blk.h
+++ b/include/uapi/linux/virtio_blk.h
@@ -42,6 +42,7 @@
 #define VIRTIO_BLK_F_WRITE_ZEROES	14	/* WRITE ZEROES is supported */
 #define VIRTIO_BLK_F_SECURE_ERASE	16 /* Secure Erase is supported */
 #define VIRTIO_BLK_F_ZONED		17	/* Zoned block device */
+#define VIRTIO_BLK_F_CTRL_VQ			22	/* Control queue */
 
 /* Legacy feature bits */
 #ifndef VIRTIO_BLK_NO_LEGACY
-- 
2.34.1

[PATCH v2 2/2] virtio_blk: add inline encryption support

[Linlin Zhang]

From: linlzhan <linlin.zhang@oss.qualcomm.com>

Add support for the virtio-blk inline encryption feature
(VIRTIO_BLK_F_INLINE_ENCRYPTION), which lets the guest offload
per-I/O encryption to the host's inline crypto engine instead of
doing it in software in the guest.

Advertise the device's inline encryption characteristics (key
types, supported crypto modes, max keyslots, DUN size) and wire
them up to a struct blk_crypto_profile so upper layers can attach
encryption contexts to bios as usual. Key management (program,
evict, generate, import, prepare key, derive software secret) is
carried out as control commands over the control virtqueue, and
per-request keyslot and data unit number are carried in an
extended request header for read/write commands.

Inline encryption is only enabled when both the control virtqueue
and inline encryption feature bits are negotiated, and is gated
behind a new VIRTIO_BLK_INLINE_ENCRYPTION Kconfig option that
depends on BLK_INLINE_ENCRYPTION, so kernels that don't select it
are unaffected.

Signed-off-by: linlzhan <linlin.zhang@oss.qualcomm.com>
---
 drivers/block/Kconfig           |  12 +
 drivers/block/virtio_blk.c      | 576 +++++++++++++++++++++++++++++++-
 include/uapi/linux/virtio_blk.h | 115 +++++++
 3 files changed, 690 insertions(+), 13 deletions(-)

diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig
index 858320b6ebb7..58bb050d4617 100644
--- a/drivers/block/Kconfig
+++ b/drivers/block/Kconfig
@@ -372,4 +372,16 @@ config BLK_DEV_ZONED_LOOP
 
 	  If unsure, say N.
 
+config VIRTIO_BLK_INLINE_ENCRYPTION
+	tristate "Virtio block inline encryption support"
+	depends on VIRTIO_BLK && BLK_INLINE_ENCRYPTION
+	help
+	  Say 'Y or M' here will allow the virtio block driver to route crypto
+	  requests to a different operating system in a virtualized
+	  environment. This is useful to encrypt the data stored in the storage
+	  by using storage inline crypto engine. The control queue feature bit
+	  must be negotiated to enable this functionality.
+
+	  If unsure, say N.
+
 endif # BLK_DEV
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 252ec9366153..3fd318e5e1f1 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -17,6 +17,7 @@
 #include <linux/numa.h>
 #include <linux/vmalloc.h>
 #include <uapi/linux/virtio_ring.h>
+#include <linux/blk-crypto-profile.h>
 
 #define PART_BITS 4
 #define VQ_NAME_LEN 16
@@ -94,13 +95,24 @@ struct virtio_blk {
 	/* For zoned device */
 	unsigned int zone_sectors;
 
+	/* For inline encryption support */
+	struct blk_crypto_profile profile;
+	bool crypto_profile_initialized;
+
 	/* Control virtqueue state. */
 	struct virtio_blk_ctrl_vq ctrl_vq;
 };
 
 struct virtblk_req {
 	/* Out header */
-	struct virtio_blk_outhdr out_hdr;
+	union {
+		struct virtio_blk_outhdr base;
+		struct {
+			struct virtio_blk_outhdr base;
+			/* Crypto message (if VIRTIO_BLK_F_INLINE_ENCRYPTION) */
+			struct virtio_blk_crypto_msg msg;
+		} crypto_append;
+	} out_hdr;
 
 	/* In header */
 	union {
@@ -124,8 +136,23 @@ struct virtblk_req {
 };
 
 struct virtblk_ctrl_request {
+	/* Type byte, always its own out-sg for every command. */
 	__virtio32 type;
+	/* Out request, sent as a second, separate out-sg if any. */
+	union {
+		struct virtio_blk_crypto_key_desc key_desc;
+		struct virtio_blk_crypto_key_blob blob;
+	} out_req;
+
+	/* In response */
+	union {
+		struct virtio_blk_crypto_key_blob blob;
+		struct virtio_blk_crypto_sw_secret secret;
+		struct virtio_blk_crypto_modes modes;
+	} in_resp;
+	/* Status byte, always its own in-sg for every command. */
 	u8 status;
+
 	struct completion compl;
 };
 
@@ -159,12 +186,17 @@ static int virtblk_add_req(struct virtqueue *vq, struct virtblk_req *vbr)
 {
 	struct scatterlist out_hdr, in_hdr, *sgs[3];
 	unsigned int num_out = 0, num_in = 0;
+	size_t out_hdr_len = sizeof(vbr->out_hdr.base);
 
-	sg_init_one(&out_hdr, &vbr->out_hdr, sizeof(vbr->out_hdr));
+	if (vbr->out_hdr.base.type == cpu_to_virtio32(vq->vdev, VIRTIO_BLK_T_CRYPTO_IN) ||
+	    vbr->out_hdr.base.type == cpu_to_virtio32(vq->vdev, VIRTIO_BLK_T_CRYPTO_OUT))
+		out_hdr_len = sizeof(vbr->out_hdr.crypto_append);
+
+	sg_init_one(&out_hdr, &vbr->out_hdr, out_hdr_len);
 	sgs[num_out++] = &out_hdr;
 
 	if (vbr->sg_table.nents) {
-		if (vbr->out_hdr.type & cpu_to_virtio32(vq->vdev, VIRTIO_BLK_T_OUT))
+		if (vbr->out_hdr.base.type & cpu_to_virtio32(vq->vdev, VIRTIO_BLK_T_OUT))
 			sgs[num_out++] = vbr->sg_table.sgl;
 		else
 			sgs[num_out + num_in++] = vbr->sg_table.sgl;
@@ -254,6 +286,22 @@ static void virtblk_cleanup_cmd(struct request *req)
 		kfree(bvec_virt(&req->special_vec));
 }
 
+#if IS_ENABLED(CONFIG_VIRTIO_BLK_INLINE_ENCRYPTION)
+static bool is_crypto_request(struct request *req)
+{
+	struct request_queue *q = req->q;
+
+	return q->crypto_profile &&
+	       req->crypt_ctx &&
+	       req->crypt_keyslot;
+}
+#else
+static inline bool is_crypto_request(struct request *req)
+{
+	return false;
+}
+#endif
+
 static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
 				      struct request *req,
 				      struct virtblk_req *vbr)
@@ -262,20 +310,27 @@ static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
 	bool unmap = false;
 	u32 type;
 	u64 sector = 0;
+	int i;
 
 	if (!IS_ENABLED(CONFIG_BLK_DEV_ZONED) && op_is_zone_mgmt(req_op(req)))
 		return BLK_STS_NOTSUPP;
 
 	/* Set fields for all request types */
-	vbr->out_hdr.ioprio = cpu_to_virtio32(vdev, req_get_ioprio(req));
+	vbr->out_hdr.base.ioprio = cpu_to_virtio32(vdev, req_get_ioprio(req));
 
 	switch (req_op(req)) {
 	case REQ_OP_READ:
-		type = VIRTIO_BLK_T_IN;
+		if (is_crypto_request(req))
+			type = VIRTIO_BLK_T_CRYPTO_IN;
+		else
+			type = VIRTIO_BLK_T_IN;
 		sector = blk_rq_pos(req);
 		break;
 	case REQ_OP_WRITE:
-		type = VIRTIO_BLK_T_OUT;
+		if (is_crypto_request(req))
+			type = VIRTIO_BLK_T_CRYPTO_OUT;
+		else
+			type = VIRTIO_BLK_T_OUT;
 		sector = blk_rq_pos(req);
 		break;
 	case REQ_OP_FLUSH:
@@ -328,8 +383,8 @@ static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
 
 	/* Set fields for non-REQ_OP_DRV_IN request types */
 	vbr->in_hdr_len = in_hdr_len;
-	vbr->out_hdr.type = cpu_to_virtio32(vdev, type);
-	vbr->out_hdr.sector = cpu_to_virtio64(vdev, sector);
+	vbr->out_hdr.base.type = cpu_to_virtio32(vdev, type);
+	vbr->out_hdr.base.sector = cpu_to_virtio64(vdev, sector);
 
 	if (type == VIRTIO_BLK_T_DISCARD || type == VIRTIO_BLK_T_WRITE_ZEROES ||
 	    type == VIRTIO_BLK_T_SECURE_ERASE) {
@@ -337,6 +392,18 @@ static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
 			return BLK_STS_RESOURCE;
 	}
 
+	if (type == VIRTIO_BLK_T_CRYPTO_IN || type == VIRTIO_BLK_T_CRYPTO_OUT) {
+		memset(&vbr->out_hdr.crypto_append.msg, 0,
+			sizeof(vbr->out_hdr.crypto_append.msg));
+		vbr->out_hdr.crypto_append.msg.slot =
+			cpu_to_virtio32(vdev,
+				blk_crypto_keyslot_index(req->crypt_keyslot));
+		for (i = 0; i < ARRAY_SIZE(vbr->out_hdr.crypto_append.msg.dun); i++) {
+			vbr->out_hdr.crypto_append.msg.dun[i] =
+				cpu_to_virtio64(vdev, req->crypt_ctx->bc_dun[i]);
+		}
+	}
+
 	return 0;
 }
 
@@ -587,8 +654,8 @@ static int virtblk_submit_zone_report(struct virtio_blk *vblk,
 
 	vbr = blk_mq_rq_to_pdu(req);
 	vbr->in_hdr_len = sizeof(vbr->in_hdr.status);
-	vbr->out_hdr.type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_ZONE_REPORT);
-	vbr->out_hdr.sector = cpu_to_virtio64(vblk->vdev, sector);
+	vbr->out_hdr.base.type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_ZONE_REPORT);
+	vbr->out_hdr.base.sector = cpu_to_virtio64(vblk->vdev, sector);
 
 	err = blk_rq_map_kern(req, report_buf, report_len, GFP_KERNEL);
 	if (err)
@@ -836,8 +903,8 @@ static int virtblk_get_id(struct gendisk *disk, char *id_str)
 
 	vbr = blk_mq_rq_to_pdu(req);
 	vbr->in_hdr_len = sizeof(vbr->in_hdr.status);
-	vbr->out_hdr.type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_GET_ID);
-	vbr->out_hdr.sector = 0;
+	vbr->out_hdr.base.type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_GET_ID);
+	vbr->out_hdr.base.sector = 0;
 
 	err = blk_rq_map_kern(req, id_str, VIRTIO_BLK_ID_BYTES, GFP_KERNEL);
 	if (err)
@@ -1001,11 +1068,478 @@ static int virtblk_ctrl_vq_request(struct virtio_blk *vblk,
 	return 0;
 }
 
+#if IS_ENABLED(CONFIG_VIRTIO_BLK_INLINE_ENCRYPTION)
+// Maps VIRTIO_BLK_CRYPTO_MODE_* values to the kernel's internal enum.
+static const enum blk_crypto_mode_num
+	virtio_blk_crypto_mode_map[VIRTIO_BLK_CRYPTO_MODE_MAX + 1] = {
+	[VIRTIO_BLK_CRYPTO_MODE_INVALID]	= BLK_ENCRYPTION_MODE_INVALID,
+	[VIRTIO_BLK_CRYPTO_MODE_AES_256_XTS]	= BLK_ENCRYPTION_MODE_AES_256_XTS,
+};
+
+static int virtblk_get_crypto_modes(struct virtio_blk *vblk,
+				    unsigned int *crypto_modes_supported)
+{
+	unsigned int nr_modes = VIRTIO_BLK_CRYPTO_MODE_MAX + 1;
+	struct scatterlist type_sg, resp_sg, status_sg, *sgs[3];
+	struct virtblk_ctrl_request *creq;
+	unsigned int i;
+	int err;
+
+	creq = kzalloc_obj(*creq, GFP_KERNEL);
+	if (!creq)
+		return -ENOMEM;
+
+	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_GET_CRYPTO_MODES);
+
+	sg_init_one(&type_sg, &creq->type, sizeof(creq->type));
+	sg_init_one(&resp_sg, &creq->in_resp.modes, sizeof(creq->in_resp.modes));
+	sg_init_one(&status_sg, &creq->status, sizeof(creq->status));
+	sgs[0] = &type_sg;
+	sgs[1] = &resp_sg;
+	sgs[2] = &status_sg;
+
+	err = virtblk_ctrl_vq_request(vblk, creq, sgs, 1, 2);
+	if (err)
+		goto out_free;
+
+	err = blk_status_to_errno(virtblk_result(creq->status));
+	if (err)
+		goto out_free;
+
+	for (i = 1; i < nr_modes; i++) {
+		u32 mode_mask = virtio32_to_cpu(vblk->vdev,
+						 creq->in_resp.modes.modes[i]);
+		enum blk_crypto_mode_num mode = virtio_blk_crypto_mode_map[i];
+
+		if (!mode_mask)
+			continue;
+		if (!mode) {
+			dev_warn(&vblk->vdev->dev,
+				 "ignoring unknown crypto mode %u\n", i);
+			continue;
+		}
+		crypto_modes_supported[mode] = mode_mask;
+	}
+
+out_free:
+	kfree(creq);
+	return err;
+}
+
+static int set_virtblk_crypto_key_desc(struct virtio_device *vdev,
+				       struct virtblk_ctrl_request *creq,
+				       const struct blk_crypto_key *key,
+				       unsigned int slot)
+{
+	struct virtio_blk_crypto_key_desc *desc = &creq->out_req.key_desc;
+	enum blk_crypto_key_type key_type = key->crypto_cfg.key_type;
+
+	if (sizeof(desc->bytes) < key->size)
+		return -EOVERFLOW;
+
+	memset(desc, 0, sizeof(*desc));
+	desc->slot = cpu_to_virtio32(vdev, slot);
+	memcpy(desc->bytes, key->bytes, key->size);
+	desc->key_size = cpu_to_virtio32(vdev, key->size);
+	desc->crypto_mode = cpu_to_virtio32(vdev, key->crypto_cfg.crypto_mode);
+	switch (key->crypto_cfg.key_type) {
+	case BLK_CRYPTO_KEY_TYPE_RAW:
+		desc->key_type = cpu_to_virtio32(vdev,
+			VIRTIO_BLK_CRYPTO_KEY_TYPE_RAW);
+		break;
+	case BLK_CRYPTO_KEY_TYPE_HW_WRAPPED:
+		desc->key_type = cpu_to_virtio32(vdev,
+			VIRTIO_BLK_CRYPTO_KEY_TYPE_HW_WRAPPED);
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+	desc->data_unit_size_bits = cpu_to_virtio32(vdev, key->data_unit_size_bits);
+	desc->dun_bytes = cpu_to_virtio32(vdev, key->crypto_cfg.dun_bytes);
+
+	return 0;
+}
+
+static inline struct virtio_blk *virtblk_from_profile(struct blk_crypto_profile *profile)
+{
+	return container_of(profile, struct virtio_blk, profile);
+}
+
+static int virtblk_crypto_keyslot_program(struct blk_crypto_profile *profile,
+					   const struct blk_crypto_key *key,
+					   unsigned int slot)
+{
+	struct virtio_blk *vblk = virtblk_from_profile(profile);
+	struct scatterlist type_sg, out_req_sg, status_sg, *sgs[3];
+	struct virtblk_ctrl_request *creq;
+	int err;
+
+	creq = kzalloc_obj(*creq, GFP_KERNEL);
+	if (!creq)
+		return -ENOMEM;
+
+	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_KEYSLOT_PROGRAM);
+
+	err = set_virtblk_crypto_key_desc(vblk->vdev, creq, key, slot);
+	if (err)
+		goto out_free;
+
+	sg_init_one(&type_sg, &creq->type, sizeof(creq->type));
+	sg_init_one(&out_req_sg, &creq->out_req.key_desc, sizeof(creq->out_req.key_desc));
+	sg_init_one(&status_sg, &creq->status, sizeof(creq->status));
+	sgs[0] = &type_sg;
+	sgs[1] = &out_req_sg;
+	sgs[2] = &status_sg;
+
+	err = virtblk_ctrl_vq_request(vblk, creq, sgs, 2, 1);
+	if (err)
+		goto out_free;
+
+	err = blk_status_to_errno(virtblk_result(creq->status));
+out_free:
+	kfree(creq);
+	return err;
+}
+
+static int virtblk_crypto_keyslot_evict(struct blk_crypto_profile *profile,
+					 const struct blk_crypto_key *key,
+					 unsigned int slot)
+{
+	struct virtio_blk *vblk = virtblk_from_profile(profile);
+	struct scatterlist type_sg, out_req_sg, status_sg, *sgs[3];
+	struct virtblk_ctrl_request *creq;
+	int err;
+
+	creq = kzalloc_obj(*creq, GFP_KERNEL);
+	if (!creq)
+		return -ENOMEM;
+
+	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_KEYSLOT_EVICT);
+
+	err = set_virtblk_crypto_key_desc(vblk->vdev, creq, key, slot);
+	if (err)
+		goto out_free;
+
+	sg_init_one(&type_sg, &creq->type, sizeof(creq->type));
+	sg_init_one(&out_req_sg, &creq->out_req.key_desc, sizeof(creq->out_req.key_desc));
+	sg_init_one(&status_sg, &creq->status, sizeof(creq->status));
+	sgs[0] = &type_sg;
+	sgs[1] = &out_req_sg;
+	sgs[2] = &status_sg;
+
+	err = virtblk_ctrl_vq_request(vblk, creq, sgs, 2, 1);
+	if (err)
+		goto out_free;
+
+	err = blk_status_to_errno(virtblk_result(creq->status));
+out_free:
+	kfree(creq);
+	return err;
+}
+
+static int virtblk_crypto_derive_sw_secret(struct blk_crypto_profile *profile,
+					    const u8 *eph_key, size_t eph_key_size,
+					    u8 sw_secret[BLK_CRYPTO_SW_SECRET_SIZE])
+{
+	struct virtio_blk *vblk = virtblk_from_profile(profile);
+	struct scatterlist type_sg, out_req_sg, resp_sg, status_sg, *sgs[4];
+	struct virtblk_ctrl_request *creq;
+	int err;
+
+	if (eph_key_size > VIRTIO_BLK_CRYPTO_MAX_KEY_SIZE
+	    || sizeof(creq->in_resp.secret.secret) < BLK_CRYPTO_SW_SECRET_SIZE)
+		return -EOVERFLOW;
+
+	creq = kzalloc_obj(*creq, GFP_KERNEL);
+	if (!creq)
+		return -ENOMEM;
+
+	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_DERIVE_SW_SECRET);
+	memcpy(creq->out_req.blob.key, eph_key, eph_key_size);
+	creq->out_req.blob.key_size = cpu_to_virtio32(vblk->vdev, eph_key_size);
+
+	sg_init_one(&type_sg, &creq->type, sizeof(creq->type));
+	sg_init_one(&out_req_sg, &creq->out_req.blob, sizeof(creq->out_req.blob));
+	sg_init_one(&resp_sg, &creq->in_resp.secret, sizeof(creq->in_resp.secret));
+	sg_init_one(&status_sg, &creq->status, sizeof(creq->status));
+	sgs[0] = &type_sg;
+	sgs[1] = &out_req_sg;
+	sgs[2] = &resp_sg;
+	sgs[3] = &status_sg;
+
+	err = virtblk_ctrl_vq_request(vblk, creq, sgs, 2, 2);
+	if (err)
+		goto out_free;
+
+	err = blk_status_to_errno(virtblk_result(creq->status));
+	if (err)
+		goto out_free;
+
+	memcpy(sw_secret, creq->in_resp.secret.secret, BLK_CRYPTO_SW_SECRET_SIZE);
+out_free:
+	kfree(creq);
+	return err;
+}
+
+static int virtblk_crypto_generate_key(struct blk_crypto_profile *profile,
+					u8 lt_key[BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE])
+{
+	struct virtio_blk *vblk = virtblk_from_profile(profile);
+	struct scatterlist type_sg, resp_sg, status_sg, *sgs[3];
+	struct virtblk_ctrl_request *creq;
+	unsigned int key_size;
+	int err;
+
+	creq = kzalloc_obj(*creq, GFP_KERNEL);
+	if (!creq)
+		return -ENOMEM;
+
+	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_GENERATE_KEY);
+
+	sg_init_one(&type_sg, &creq->type, sizeof(creq->type));
+	sg_init_one(&resp_sg, &creq->in_resp.blob, sizeof(creq->in_resp.blob));
+	sg_init_one(&status_sg, &creq->status, sizeof(creq->status));
+	sgs[0] = &type_sg;
+	sgs[1] = &resp_sg;
+	sgs[2] = &status_sg;
+
+	err = virtblk_ctrl_vq_request(vblk, creq, sgs, 1, 2);
+	if (err)
+		goto out_free;
+
+	err = blk_status_to_errno(virtblk_result(creq->status));
+	if (err)
+		goto out_free;
+
+	key_size = virtio32_to_cpu(vblk->vdev, creq->in_resp.blob.key_size);
+	if (!key_size ||
+		key_size > BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE) {
+		dev_err(&vblk->vdev->dev,
+			"backend returned oversized generated key: %u\n", key_size);
+		err = -EOVERFLOW;
+		goto out_free;
+	}
+	memcpy(lt_key, creq->in_resp.blob.key, key_size);
+	err = key_size;
+out_free:
+	kfree(creq);
+	return err;
+}
+
+static int virtblk_crypto_prepare_key(struct blk_crypto_profile *profile,
+				       const u8 *lt_key, size_t lt_key_size,
+				       u8 eph_key[BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE])
+{
+	struct virtio_blk *vblk = virtblk_from_profile(profile);
+	struct scatterlist type_sg, out_req_sg, resp_sg, status_sg, *sgs[4];
+	struct virtblk_ctrl_request *creq;
+	unsigned int key_size;
+	int err;
+
+	if (lt_key_size > VIRTIO_BLK_CRYPTO_MAX_KEY_SIZE)
+		return -EOVERFLOW;
+
+	creq = kzalloc_obj(*creq, GFP_KERNEL);
+	if (!creq)
+		return -ENOMEM;
+
+	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_PREPARE_KEY);
+	memcpy(creq->out_req.blob.key, lt_key, lt_key_size);
+	creq->out_req.blob.key_size = cpu_to_virtio32(vblk->vdev, lt_key_size);
+
+	sg_init_one(&type_sg, &creq->type, sizeof(creq->type));
+	sg_init_one(&out_req_sg, &creq->out_req.blob, sizeof(creq->out_req.blob));
+	sg_init_one(&resp_sg, &creq->in_resp.blob, sizeof(creq->in_resp.blob));
+	sg_init_one(&status_sg, &creq->status, sizeof(creq->status));
+	sgs[0] = &type_sg;
+	sgs[1] = &out_req_sg;
+	sgs[2] = &resp_sg;
+	sgs[3] = &status_sg;
+
+	err = virtblk_ctrl_vq_request(vblk, creq, sgs, 2, 2);
+	if (err)
+		goto out_free;
+
+	err = blk_status_to_errno(virtblk_result(creq->status));
+	if (err)
+		goto out_free;
+
+	key_size = virtio32_to_cpu(vblk->vdev, creq->in_resp.blob.key_size);
+	if (!key_size ||
+		key_size > BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE) {
+		dev_err(&vblk->vdev->dev,
+			"backend returned oversized prepared key: %u\n", key_size);
+		err = -EOVERFLOW;
+		goto out_free;
+	}
+	memcpy(eph_key, creq->in_resp.blob.key, key_size);
+	err = key_size;
+out_free:
+	kfree(creq);
+	return err;
+}
+
+static int virtblk_crypto_import_key(struct blk_crypto_profile *profile,
+				      const u8 *raw_key, size_t raw_key_size,
+				      u8 lt_key[BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE])
+{
+	struct virtio_blk *vblk = virtblk_from_profile(profile);
+	struct scatterlist type_sg, out_req_sg, resp_sg, status_sg, *sgs[4];
+	struct virtblk_ctrl_request *creq;
+	unsigned int key_size;
+	int err;
+
+	if (raw_key_size > VIRTIO_BLK_CRYPTO_MAX_KEY_SIZE)
+		return -EOVERFLOW;
+
+	creq = kzalloc_obj(*creq, GFP_KERNEL);
+	if (!creq)
+		return -ENOMEM;
+
+	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_IMPORT_KEY);
+	memcpy(creq->out_req.blob.key, raw_key, raw_key_size);
+	creq->out_req.blob.key_size = cpu_to_virtio32(vblk->vdev, raw_key_size);
+
+	sg_init_one(&type_sg, &creq->type, sizeof(creq->type));
+	sg_init_one(&out_req_sg, &creq->out_req.blob, sizeof(creq->out_req.blob));
+	sg_init_one(&resp_sg, &creq->in_resp.blob, sizeof(creq->in_resp.blob));
+	sg_init_one(&status_sg, &creq->status, sizeof(creq->status));
+	sgs[0] = &type_sg;
+	sgs[1] = &out_req_sg;
+	sgs[2] = &resp_sg;
+	sgs[3] = &status_sg;
+
+	err = virtblk_ctrl_vq_request(vblk, creq, sgs, 2, 2);
+	if (err)
+		goto out_free;
+
+	err = blk_status_to_errno(virtblk_result(creq->status));
+	if (err)
+		goto out_free;
+
+	key_size = virtio32_to_cpu(vblk->vdev, creq->in_resp.blob.key_size);
+	if (!key_size ||
+		key_size > BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE) {
+		dev_err(&vblk->vdev->dev,
+			"backend returned oversized imported key: %u\n", key_size);
+		err = -EOVERFLOW;
+		goto out_free;
+	}
+	memcpy(lt_key, creq->in_resp.blob.key, key_size);
+	err = key_size;
+out_free:
+	kfree(creq);
+	return err;
+}
+
+static const struct blk_crypto_ll_ops virtblk_crypto_ops = {
+	.keyslot_program	= virtblk_crypto_keyslot_program,
+	.keyslot_evict		= virtblk_crypto_keyslot_evict,
+	.derive_sw_secret	= virtblk_crypto_derive_sw_secret,
+	.generate_key		= virtblk_crypto_generate_key,
+	.prepare_key		= virtblk_crypto_prepare_key,
+	.import_key		= virtblk_crypto_import_key,
+};
+
+static int virtblk_init_crypto(struct virtio_blk *vblk)
+{
+	struct virtio_device *vdev = vblk->vdev;
+	unsigned int crypto_modes_supported[BLK_ENCRYPTION_MODE_MAX] = { 0 };
+	unsigned int key_type_supported = 0;
+	u16 max_slots = 0;
+	/* virtio_cread() requires the variable size to match the config field exactly */
+	u8 max_dun_bytes = 0, key_types = 0;
+	int err;
+
+	virtio_cread(vdev, struct virtio_blk_config,
+		     enc_characteristics.max_slots, &max_slots);
+	virtio_cread(vdev, struct virtio_blk_config,
+		     enc_characteristics.max_dun_bytes, &max_dun_bytes);
+	virtio_cread(vdev, struct virtio_blk_config,
+		     enc_characteristics.key_types, &key_types);
+
+	dev_info_once(&vdev->dev,
+		 "max_slots = %u, max_dun_bytes = %u, key_types = 0x%x\n",
+		 max_slots, max_dun_bytes, key_types);
+
+	if (!max_slots)
+		return -EINVAL;
+
+	/*
+	 * struct virtio_blk_crypto_msg.dun is a fixed array of four __virtio64
+	 * values (32 bytes total), matching the size of
+	 * blk_crypto_ctx::bc_dun[4].  Refuse to advertise more than that as
+	 * supported, or blk-crypto could negotiate a larger dun_bytes with the
+	 * filesystem and have the high-order bytes of req->crypt_ctx->bc_dun
+	 * silently dropped in virtblk_setup_cmd().
+	 */
+	if (max_dun_bytes > sizeof_field(struct virtio_blk_crypto_msg, dun))
+		return -EINVAL;
+
+	if (key_types & VIRTIO_BLK_CRYPTO_KEY_TYPE_RAW)
+		key_type_supported |= BLK_CRYPTO_KEY_TYPE_RAW;
+	if (key_types & VIRTIO_BLK_CRYPTO_KEY_TYPE_HW_WRAPPED)
+		key_type_supported |= BLK_CRYPTO_KEY_TYPE_HW_WRAPPED;
+	if (!key_type_supported)
+		return -EINVAL;
+
+	err = virtblk_get_crypto_modes(vblk, crypto_modes_supported);
+	if (err) {
+		dev_err(&vdev->dev, "get crypto modes failed: %d\n", err);
+		return err;
+	}
+
+	/*
+	 * Use the plain (non-devm) initializer: vblk->profile is embedded in
+	 * struct virtio_blk, whose lifetime is tied to the gendisk, not to
+	 * &vdev->dev. Tying destruction to the vdev via devm would run the
+	 * destroy callback after virtblk_remove() has already freed vblk.
+	 * virtblk_free_disk() calls blk_crypto_profile_destroy() explicitly
+	 * instead, guarded by crypto_profile_initialized below.
+	 */
+	err = blk_crypto_profile_init(&vblk->profile, max_slots);
+	if (err) {
+		dev_err(&vdev->dev, "crypto profile initialization failed: %d\n", err);
+		return err;
+	}
+
+	vblk->profile.ll_ops = virtblk_crypto_ops;
+	vblk->profile.max_dun_bytes_supported = max_dun_bytes;
+	vblk->profile.key_types_supported = key_type_supported;
+	vblk->profile.dev = &vdev->dev;
+	memcpy(vblk->profile.modes_supported, crypto_modes_supported,
+	       BLK_ENCRYPTION_MODE_MAX * sizeof(unsigned int));
+
+	vblk->crypto_profile_initialized = true;
+
+	dev_info(&vdev->dev, "inline crypto profile initialized\n");
+
+	return 0;
+}
+
+static void virtblk_destroy_crypto(struct virtio_blk *vblk)
+{
+	if (vblk->crypto_profile_initialized)
+		blk_crypto_profile_destroy(&vblk->profile);
+}
+#else
+
+static inline int virtblk_init_crypto(struct virtio_blk *vblk)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void virtblk_destroy_crypto(struct virtio_blk *vblk)
+{
+}
+#endif /* CONFIG_VIRTIO_BLK_INLINE_ENCRYPTION */
+
 static void virtblk_free_disk(struct gendisk *disk)
 {
 	struct virtio_blk *vblk = disk->private_data;
 
 	ida_free(&vd_index_ida, vblk->index);
+	virtblk_destroy_crypto(vblk);
 	mutex_destroy(&vblk->ctrl_vq.mutex);
 	mutex_destroy(&vblk->vdev_mutex);
 	kfree(vblk);
@@ -1698,6 +2232,19 @@ static int virtblk_probe(struct virtio_device *vdev)
 		err = blk_revalidate_disk_zones(vblk->disk);
 		if (err)
 			goto out_cleanup_disk;
+	} else if (IS_ENABLED(CONFIG_VIRTIO_BLK_INLINE_ENCRYPTION) &&
+		   virtio_has_feature(vdev, VIRTIO_BLK_F_INLINE_ENCRYPTION) &&
+		   virtio_has_feature(vdev, VIRTIO_BLK_F_CTRL_VQ)) {
+		err = virtblk_init_crypto(vblk);
+		if (!err) {
+			if (!blk_crypto_register(&vblk->profile, vblk->disk->queue))
+				dev_warn(&vdev->dev,
+					 "failed to register inline crypto profile\n");
+		} else {
+			dev_warn(&vdev->dev,
+				 "inline crypto init failed: %d, continuing without inline crypto support\n",
+				 err);
+		}
 	}
 
 	err = device_add_disk(&vdev->dev, vblk->disk, virtblk_attr_groups);
@@ -1798,6 +2345,9 @@ static int virtblk_restore_priv(struct virtio_device *vdev)
 	virtio_device_ready(vdev);
 	blk_mq_unquiesce_queue(vblk->disk->queue);
 
+	if (vblk->profile.slots)
+		blk_crypto_reprogram_all_keys(&vblk->profile);
+
 	return 0;
 }
 
@@ -1842,7 +2392,7 @@ static unsigned int features[] = {
 	VIRTIO_BLK_F_FLUSH, VIRTIO_BLK_F_TOPOLOGY, VIRTIO_BLK_F_CONFIG_WCE,
 	VIRTIO_BLK_F_MQ, VIRTIO_BLK_F_DISCARD, VIRTIO_BLK_F_WRITE_ZEROES,
 	VIRTIO_BLK_F_SECURE_ERASE, VIRTIO_BLK_F_ZONED,
-	VIRTIO_BLK_F_CTRL_VQ,
+	VIRTIO_BLK_F_CTRL_VQ, VIRTIO_BLK_F_INLINE_ENCRYPTION,
 };
 
 static struct virtio_driver virtio_blk = {
diff --git a/include/uapi/linux/virtio_blk.h b/include/uapi/linux/virtio_blk.h
index 0a16972a1535..952534e2b489 100644
--- a/include/uapi/linux/virtio_blk.h
+++ b/include/uapi/linux/virtio_blk.h
@@ -43,6 +43,7 @@
 #define VIRTIO_BLK_F_SECURE_ERASE	16 /* Secure Erase is supported */
 #define VIRTIO_BLK_F_ZONED		17	/* Zoned block device */
 #define VIRTIO_BLK_F_CTRL_VQ			22	/* Control queue */
+#define VIRTIO_BLK_F_INLINE_ENCRYPTION		23	/* Inline encryption */
 
 /* Legacy feature bits */
 #ifndef VIRTIO_BLK_NO_LEGACY
@@ -58,6 +59,10 @@
 
 #define VIRTIO_BLK_ID_BYTES	20	/* ID string length */
 
+/* Key type bitmask for VIRTIO_BLK_F_INLINE_ENCRYPTION */
+#define VIRTIO_BLK_CRYPTO_KEY_TYPE_RAW	(1 << 0)
+#define VIRTIO_BLK_CRYPTO_KEY_TYPE_HW_WRAPPED	(1 << 1)
+
 struct virtio_blk_config {
 	/* The capacity (in 512-byte sectors). */
 	__virtio64 capacity;
@@ -149,6 +154,15 @@ struct virtio_blk_config {
 		__u8 model;
 		__u8 unused2[3];
 	} zoned;
+
+	/* Inline Encryption device characteristics (if VIRTIO_BLK_F_INLINE_ENCRYPTION) */
+	struct virtio_blk_enc_characteristics {
+		__virtio16 max_slots;
+		__u8 max_dun_bytes;
+		/* Bitmask of supported key types: VIRTIO_BLK_CRYPTO_KEY_TYPE_* */
+		__u8 key_types;
+		__virtio32 unused3;
+	} enc_characteristics;
 } __attribute__((packed));
 
 /*
@@ -207,6 +221,33 @@ struct virtio_blk_config {
 /* Reset All zones command */
 #define VIRTIO_BLK_T_ZONE_RESET_ALL 26
 
+/* Inline-encrypted write: crypto_msg set in outhdr */
+#define VIRTIO_BLK_T_CRYPTO_OUT		27
+
+/* Inline-encrypted read: crypto_msg set in outhdr */
+#define VIRTIO_BLK_T_CRYPTO_IN		28
+
+/* Get inline crypto modes */
+#define VIRTIO_BLK_T_GET_CRYPTO_MODES	29
+
+/* Program a key into the keyslot */
+#define VIRTIO_BLK_T_CRYPTO_KEYSLOT_PROGRAM	30
+
+/* Evict a key */
+#define VIRTIO_BLK_T_CRYPTO_KEYSLOT_EVICT	31
+
+/* Derive the software secret from a hardware-wrapped key */
+#define VIRTIO_BLK_T_CRYPTO_DERIVE_SW_SECRET	32
+
+/* Generate a new hardware-wrapped key */
+#define VIRTIO_BLK_T_CRYPTO_GENERATE_KEY	33
+
+/* Import a raw key as a hardware-wrapped key */
+#define VIRTIO_BLK_T_CRYPTO_IMPORT_KEY		34
+
+/* Convert a long-term wrapped key to its ephemerally-wrapped form */
+#define VIRTIO_BLK_T_CRYPTO_PREPARE_KEY	35
+
 #ifndef VIRTIO_BLK_NO_LEGACY
 /* Barrier before this op. */
 #define VIRTIO_BLK_T_BARRIER	0x80000000
@@ -226,6 +267,80 @@ struct virtio_blk_outhdr {
 	__virtio64 sector;
 };
 
+/*
+ * Crypto message descriptor, appended to the outhdr of a
+ * VIRTIO_BLK_T_CRYPTO_OUT or VIRTIO_BLK_T_CRYPTO_IN request.
+ */
+struct virtio_blk_crypto_msg {
+	/* virtual key slot index */
+	__virtio32 slot;
+	__u8 unused[4];
+	/* data unit number (DUN / IV) for this request */
+	__virtio64 dun[4];
+};
+
+/*
+ * Inline crypto key descriptor. Request part for
+ * VIRTIO_BLK_T_CRYPTO_KEYSLOT_PROGRAM/KEYSLOT_EVICT.
+ */
+/* Must be >= BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE in include/linux/blk-crypto.h */
+#define VIRTIO_BLK_CRYPTO_MAX_KEY_SIZE		128
+
+struct virtio_blk_crypto_key_desc {
+	__virtio32  slot;
+	__u8        bytes[VIRTIO_BLK_CRYPTO_MAX_KEY_SIZE];
+	__virtio32  key_size;
+	__virtio32  crypto_mode;
+	__virtio32  key_type;
+	__virtio32  data_unit_size_bits;
+	__virtio32  dun_bytes;
+};
+
+/*
+ * A raw or hardware-wrapped key blob, used as the request and/or reply part
+ * of the VIRTIO_BLK_T_CRYPTO_GENERATE_KEY/IMPORT_KEY/PREPARE_KEY/
+ * DERIVE_SW_SECRET commands.
+ */
+struct virtio_blk_crypto_key_blob {
+	__virtio32 key_size;
+	__u8 key[VIRTIO_BLK_CRYPTO_MAX_KEY_SIZE];
+};
+
+/* Must match BLK_CRYPTO_SW_SECRET_SIZE in include/linux/blk-crypto.h */
+#define VIRTIO_BLK_CRYPTO_SW_SECRET_SIZE	32
+
+/* Reply to a VIRTIO_BLK_T_CRYPTO_DERIVE_SW_SECRET request. */
+struct virtio_blk_crypto_sw_secret {
+	__u8 secret[VIRTIO_BLK_CRYPTO_SW_SECRET_SIZE];
+};
+
+/*
+ * Crypto mode numbers used in VIRTIO_BLK_T_GET_CRYPTO_MODES replies and in
+ * indexing struct virtio_blk_crypto_modes.modes[] below. These numbers are
+ * assigned by the virtio spec and are stable: a number is never reused for
+ * a different crypto mode, and additional crypto modes are assigned new,
+ * higher numbers.
+ */
+enum {
+	VIRTIO_BLK_CRYPTO_MODE_INVALID,
+	VIRTIO_BLK_CRYPTO_MODE_AES_256_XTS,
+	__VIRTIO_BLK_CRYPTO_MODE_MAX,	/* sentinel: always one past the last real mode */
+};
+
+/* Highest crypto mode number defined by this version of the header. */
+#define VIRTIO_BLK_CRYPTO_MODE_MAX (__VIRTIO_BLK_CRYPTO_MODE_MAX - 1)
+
+/* Reply to a VIRTIO_BLK_T_GET_CRYPTO_MODES request. */
+struct virtio_blk_crypto_modes {
+	/*
+	 * modes[N], for crypto mode number N <= VIRTIO_BLK_CRYPTO_MODE_MAX, is
+	 * a bitmask of the data unit sizes with which crypto mode N can be
+	 * used: bit i is set if a data unit size of (1 << i) bytes is
+	 * supported. modes[0] is reserved and always 0.
+	 */
+	__virtio32 modes[__VIRTIO_BLK_CRYPTO_MODE_MAX];
+};
+
 /*
  * Supported zoned device models.
  */
-- 
2.34.1

Re: [PATCH v2 1/2] virtio_blk: Add control virtqueue support

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

New issues:
- [High] DMA cacheline sharing causes memory corruption on non-coherent architectures.
- [High] Erroneous teardown ordering causes valid in-flight requests to fail with I/O errors during suspend.
- [High] Unbounded wait in `virtblk_ctrl_vq_request` deadlocks device teardown if the host is unresponsive.

Pre-existing issues:
- [High] Use-after-free and double-free of `vblk` in the `virtblk_probe` error path.
--

commit 36271d203fcaee50a88fd7b53aa73dbbba25cf7c
Author: linlzhan <linlin.zhang@oss.qualcomm.com>

virtio_blk: Add control virtqueue support

This commit adds support for the optional virtio-blk control virtqueue.
It allows the driver to manage control-queue requests independently from
the data path, preparing for subsequent inline encryption implementation.

> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
[ ... ]
> @@ -110,6 +123,12 @@ struct virtblk_req {
>  	struct scatterlist sg[];
>  };
>  
> +struct virtblk_ctrl_request {
> +	__virtio32 type;
> +	u8 status;
> +	struct completion compl;
> +};

[Severity: High]
Is it safe to place DMA targets (type, status) and CPU synchronization
primitives (struct completion compl) in the same struct without cacheline
alignment?

On non-coherent architectures, could the DMA API unmap operation invalidate
the cacheline and destroy the CPU's unwritten modifications to compl.wait,
thereby corrupting the waitqueue? Buffers that are targets of DMA typically
need to be cacheline-aligned or physically separated from CPU-modified
lock/sync primitives.

[ ... ]
> +static int virtblk_ctrl_vq_request(struct virtio_blk *vblk,
> +				    struct virtblk_ctrl_request *creq,
> +				    struct scatterlist *sgs[],
> +				    unsigned int out_sgs, unsigned int in_sgs)
> +{
[ ... ]
> +	wait_for_completion(&creq->compl);
> +	mutex_unlock(&vblk->ctrl_vq.mutex);
> +	return 0;
> +}

[Severity: High]
Could this unbounded wait deadlock device teardown if the host is
unresponsive?

When called from a block I/O context, this thread holds a reference to
q_usage_counter. If the host hangs, del_gendisk() in virtblk_remove() will
freeze the block queue and wait indefinitely for q_usage_counter to reach
zero.

Since virtblk_ctrl_vq_drain() is called after del_gendisk(), wouldn't the
system deadlock before the drain can recover the stuck requests?

[ ... ]
> @@ -1553,6 +1713,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>  out_free_vq:
>  	vdev->config->del_vqs(vdev);
>  	kfree(vblk->vqs);
> +	vblk->ctrl_vq.vq = NULL;
>  out_free_vblk:
>  	kfree(vblk);
>  out_free_index:

[Severity: High]
This is a pre-existing issue, but does the error path in virtblk_probe()
trigger a use-after-free and double-free?

If device_add_disk() fails, the path jumps to out_cleanup_disk which calls
put_disk(vblk->disk). Because vblk->disk->fops points to virtblk_fops,
dropping the last reference invokes virtblk_free_disk(), which frees vblk
and vblk->index.

By the time execution reaches out_free_vq, aren't accesses to vblk->vqs and
this new assignment to vblk->ctrl_vq.vq writing to freed memory, followed
by a double free of vblk in out_free_vblk?

[ ... ]
> @@ -1593,6 +1759,8 @@ static int virtblk_freeze_priv(struct virtio_device *vdev)
>  	struct request_queue *q = vblk->disk->queue;
>  	unsigned int memflags;
>  
> +	virtblk_ctrl_vq_quiesce(vblk);
> +
>  	/* Ensure no requests in virtqueues before deleting vqs. */
>  	memflags = blk_mq_freeze_queue(q);

[Severity: High]
By calling virtblk_ctrl_vq_quiesce() before blk_mq_freeze_queue(), could
valid in-flight requests fail with I/O errors?

virtblk_ctrl_vq_quiesce() sets ctrl_vq.dead = true. Since the block queue
is not yet frozen, in-flight block I/O requests that haven't yet submitted
their control requests (but have already incremented q_usage_counter) will
encounter dead == true when calling virtblk_ctrl_vq_request(), resulting in
-ENODEV.

Should the data plane be frozen before the control queue is marked dead?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260914133733.15429-1-linlin.zhang@oss.qualcomm.com?part=1

Re: [PATCH v2 2/2] virtio_blk: add inline encryption support

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:

New issues:
- [Critical] REQ_OP_ZONE_APPEND requests with an encryption context are silently written in plaintext.
- [High] Zoned block devices completely skip inline encryption initialization due to an 'else if' block.
- [High] Memory allocation on the block I/O submission path uses GFP_KERNEL without a mempool, causing deadlocks.
- [High] Hardware queues are unquiesced before encryption keys are reprogrammed during resume.
- [High] Cacheline sharing in `struct virtblk_ctrl_request` violates the DMA API, risking memory corruption.
- [High] Uninitialized vblk->crypto_profile_initialized leads to wild frees on probe error paths.
- [High] blk-crypto operations can race with device removal, leading to NULL pointer dereference.
- [Medium] The driver directly assigns Linux-internal crypto mode enums to the Virtio hardware descriptor without translation.
- [Low] VIRTIO_BLK_INLINE_ENCRYPTION is defined as a tristate but cannot be built as an independent module.

Pre-existing issues:
- [High] The error cleanup path in virtblk_probe contains a use-after-free and double free of vblk.
--

commit 787711d9770fce24131418d43dc33cb33a894de6
Author: linlzhan <linlin.zhang@oss.qualcomm.com>

virtio_blk: add inline encryption support

This commit adds support for the virtio-blk inline encryption feature,
allowing guest VMs to offload per-I/O encryption to the host's inline
crypto engine. It wires up device encryption characteristics to a
struct blk_crypto_profile and handles key management via control commands.

> diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig
> index 858320b6ebb7e..58bb050d4617c 100644
> --- a/drivers/block/Kconfig
> +++ b/drivers/block/Kconfig
> @@ -372,4 +372,16 @@ config BLK_DEV_ZONED_LOOP
>  
>  	  If unsure, say N.
>  
> +config VIRTIO_BLK_INLINE_ENCRYPTION
> +	tristate "Virtio block inline encryption support"
> +	depends on VIRTIO_BLK && BLK_INLINE_ENCRYPTION

[Severity: Low]
Could this lead to configuration ambiguity? VIRTIO_BLK_INLINE_ENCRYPTION is
defined as a tristate, but the code is conditionally included directly within
the virtio_blk module. Users might select 'M' thinking it builds a separate
crypto module, but it functions as an intra-file boolean.

[ ... ]

> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 252ec93661532..3fd318e5e1f17 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -124,8 +136,23 @@ struct virtblk_req {
>  };
>  
>  struct virtblk_ctrl_request {
> +	/* Type byte, always its own out-sg for every command. */
>  	__virtio32 type;
> +	/* Out request, sent as a second, separate out-sg if any. */
> +	union {
> +		struct virtio_blk_crypto_key_desc key_desc;
> +		struct virtio_blk_crypto_key_blob blob;
> +	} out_req;
> +
> +	/* In response */
> +	union {
> +		struct virtio_blk_crypto_key_blob blob;
> +		struct virtio_blk_crypto_sw_secret secret;
> +		struct virtio_blk_crypto_modes modes;
> +	} in_resp;
> +	/* Status byte, always its own in-sg for every command. */
>  	u8 status;
> +
>  	struct completion compl;
>  };

[Severity: High]
Does this struct packing risk memory corruption on non-cache-coherent
architectures?

The DMA-written buffers (in_resp, status) are packed directly adjacent to the
compl field. On non-coherent architectures, cache invalidation for the DMA
fields can destroy the CPU's ongoing modifications to the waitqueue inside
compl during wait_for_completion().

[ ... ]

> @@ -262,20 +310,27 @@ static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
>  				      struct request *req,
>  				      struct virtblk_req *vbr)
>  {
[ ... ]
>  	case REQ_OP_ZONE_FINISH:
>  		type = VIRTIO_BLK_T_ZONE_FINISH;
>  		sector = blk_rq_pos(req);
>  		break;
>  	case REQ_OP_ZONE_APPEND:
>  		type = VIRTIO_BLK_T_ZONE_APPEND;
>  		sector = blk_rq_pos(req);
>  		in_hdr_len = sizeof(vbr->in_hdr.zone_append);
>  		break;

[Severity: Critical]
Could this lead to plaintext data being written for encrypted zone append
requests?

When handling REQ_OP_ZONE_APPEND, the type is forced to
VIRTIO_BLK_T_ZONE_APPEND. Because this path does not check is_crypto_request(),
it bypasses the crypto setup...

[ ... ]

> @@ -337,6 +392,18 @@ static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
>  	if (type == VIRTIO_BLK_T_DISCARD || type == VIRTIO_BLK_T_WRITE_ZEROES ||
>  	    type == VIRTIO_BLK_T_SECURE_ERASE) {
>  		if (virtblk_setup_discard_write_zeroes_erase(req, unmap))
>  			return BLK_STS_RESOURCE;
>  	}
>  
> +	if (type == VIRTIO_BLK_T_CRYPTO_IN || type == VIRTIO_BLK_T_CRYPTO_OUT) {
> +		memset(&vbr->out_hdr.crypto_append.msg, 0,

[Severity: Critical]
...and subsequently fails this check since type is VIRTIO_BLK_T_ZONE_APPEND
instead of VIRTIO_BLK_T_CRYPTO_OUT.

This prevents the crypto message descriptor from being attached, causing the
host hypervisor to see a standard unencrypted write and store the plaintext
data to disk.

[ ... ]

> @@ -1001,11 +1068,478 @@ static int virtblk_ctrl_vq_request(struct virtio_blk *vblk,
[ ... ]
> +static int set_virtblk_crypto_key_desc(struct virtio_device *vdev,
> +				       struct virtblk_ctrl_request *creq,
> +				       const struct blk_crypto_key *key,
> +				       unsigned int slot)
> +{
> +	struct virtio_blk_crypto_key_desc *desc = &creq->out_req.key_desc;
> +	enum blk_crypto_key_type key_type = key->crypto_cfg.key_type;
> +
> +	if (sizeof(desc->bytes) < key->size)
> +		return -EOVERFLOW;
> +
> +	memset(desc, 0, sizeof(*desc));
> +	desc->slot = cpu_to_virtio32(vdev, slot);
> +	memcpy(desc->bytes, key->bytes, key->size);
> +	desc->key_size = cpu_to_virtio32(vdev, key->size);
> +	desc->crypto_mode = cpu_to_virtio32(vdev, key->crypto_cfg.crypto_mode);

[Severity: Medium]
Can this leak a Linux-internal enum value directly into the hardware protocol
descriptor?

The desc->crypto_mode is set directly from key->crypto_cfg.crypto_mode, which
is an internal enum blk_crypto_mode_num. If Linux adds a new mode or virtio
assigns different numbers in the future, this will silently send incorrect
crypto modes to the hypervisor.

[ ... ]

> +static int virtblk_crypto_keyslot_program(struct blk_crypto_profile *profile,
> +					   const struct blk_crypto_key *key,
> +					   unsigned int slot)
> +{
> +	struct virtio_blk *vblk = virtblk_from_profile(profile);
> +	struct scatterlist type_sg, out_req_sg, status_sg, *sgs[3];
> +	struct virtblk_ctrl_request *creq;
> +	int err;
> +
> +	creq = kzalloc_obj(*creq, GFP_KERNEL);
> +	if (!creq)
> +		return -ENOMEM;

[Severity: High]
Is it safe to use GFP_KERNEL here?

This allocation occurs on the block I/O submission path. Under memory pressure,
this could recurse into memory reclaim, which may attempt to flush dirty pages
to this same block device, leading to a deadlock.

[ ... ]

> +static int virtblk_crypto_keyslot_evict(struct blk_crypto_profile *profile,
> +					 const struct blk_crypto_key *key,
> +					 unsigned int slot)
> +{
> +	struct virtio_blk *vblk = virtblk_from_profile(profile);
> +	struct scatterlist type_sg, out_req_sg, status_sg, *sgs[3];
> +	struct virtblk_ctrl_request *creq;
> +	int err;
> +
> +	creq = kzalloc_obj(*creq, GFP_KERNEL);
> +	if (!creq)
> +		return -ENOMEM;

[Severity: High]
Similar to the programming path, is GFP_KERNEL appropriate here? The eviction
path often occurs during inode or dentry reclaim, and recursing into memory
reclaim could cause deadlocks.

> +
> +	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_KEYSLOT_EVICT);

[Severity: High]
Could this race with device removal?

In virtblk_crypto_keyslot_evict(), vblk->vdev is accessed unconditionally. If
the virtio device is hot-unplugged, virtblk_remove() sets vblk->vdev to NULL.
Since the block device may still be open by userspace, fscrypt might
subsequently tear down a key and call into this driver, causing a NULL pointer
dereference.

[ ... ]

> +static void virtblk_destroy_crypto(struct virtio_blk *vblk)
> +{
> +	if (vblk->crypto_profile_initialized)
> +		blk_crypto_profile_destroy(&vblk->profile);
> +}

[Severity: High]
Could this cause a wild free on probe error paths?

vblk is allocated using kmalloc_obj() which does not zero memory. If the probe
fails before the profile is initialized, virtblk_destroy_crypto() might read a
true value for vblk->crypto_profile_initialized and pass garbage to
blk_crypto_profile_destroy().

[ ... ]

> @@ -1698,6 +2232,19 @@ static int virtblk_probe(struct virtio_device *vdev)
[ ... ]
>  	if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) &&
>  	    (lim.features & BLK_FEAT_ZONED)) {
>  		err = blk_revalidate_disk_zones(vblk->disk);
>  		if (err)
>  			goto out_cleanup_disk;
> +	} else if (IS_ENABLED(CONFIG_VIRTIO_BLK_INLINE_ENCRYPTION) &&
> +		   virtio_has_feature(vdev, VIRTIO_BLK_F_INLINE_ENCRYPTION) &&
> +		   virtio_has_feature(vdev, VIRTIO_BLK_F_CTRL_VQ)) {
> +		err = virtblk_init_crypto(vblk);

[Severity: High]
Does this else if inadvertently disable inline encryption for zoned block
devices?

If a device advertises both ZONED and INLINE_ENCRYPTION features, the initial
if block executes, causing the entire crypto initialization block to be skipped
silently.

[ ... ]

>  	err = device_add_disk(&vdev->dev, vblk->disk, virtblk_attr_groups);
>  	if (err)
>  		goto out_cleanup_disk;
>  
>  	return 0;
>  
>  out_cleanup_disk:
>  	put_disk(vblk->disk);
>  out_free_tags:
>  	blk_mq_free_tag_set(&vblk->tag_set);
>  out_free_vq:
>  	vdev->config->del_vqs(vdev);
>  	kfree(vblk->vqs);
>  	vblk->ctrl_vq.vq = NULL;
>  out_free_vblk:
>  	kfree(vblk);

[Severity: High]
This is a pre-existing issue, but does this error cleanup path have a
use-after-free and double free?

The call to put_disk(vblk->disk) drops the reference count, invoking
virtblk_free_disk() which frees vblk. The subsequent labels continue to
dereference vblk->tag_set and vblk->vqs, and eventually kfree(vblk) is called
again.

[ ... ]

> @@ -1798,6 +2345,9 @@ static int virtblk_restore_priv(struct virtio_device *vdev)
>  	virtio_device_ready(vdev);
>  	blk_mq_unquiesce_queue(vblk->disk->queue);
>  
> +	if (vblk->profile.slots)
> +		blk_crypto_reprogram_all_keys(&vblk->profile);
> +
>  	return 0;
>  }

[Severity: High]
Could this expose unprovisioned keyslots during resume?

The hardware queues are unquiesced before blk_crypto_reprogram_all_keys() is
called. This instantly restarts I/O dispatch, meaning any pending encrypted
requests could be executed by the hypervisor with empty or invalid keyslots
before they are reprogrammed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260914133733.15429-1-linlin.zhang@oss.qualcomm.com?part=2

Re: [PATCH v2 1/2] virtio_blk: Add control virtqueue support

[Linlin Zhang]

On 9/14/2026 10:02 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
> 
> New issues:
> - [High] DMA cacheline sharing causes memory corruption on non-coherent architectures.
> - [High] Erroneous teardown ordering causes valid in-flight requests to fail with I/O errors during suspend.
> - [High] Unbounded wait in `virtblk_ctrl_vq_request` deadlocks device teardown if the host is unresponsive.
> 
> Pre-existing issues:
> - [High] Use-after-free and double-free of `vblk` in the `virtblk_probe` error path.
> --
> 
> commit 36271d203fcaee50a88fd7b53aa73dbbba25cf7c
> Author: linlzhan <linlin.zhang@oss.qualcomm.com>
> 
> virtio_blk: Add control virtqueue support
> 
> This commit adds support for the optional virtio-blk control virtqueue.
> It allows the driver to manage control-queue requests independently from
> the data path, preparing for subsequent inline encryption implementation.
> 
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
> [ ... ]
>> @@ -110,6 +123,12 @@ struct virtblk_req {
>>  	struct scatterlist sg[];
>>  };
>>  
>> +struct virtblk_ctrl_request {
>> +	__virtio32 type;
>> +	u8 status;
>> +	struct completion compl;
>> +};
> 
> [Severity: High]
> Is it safe to place DMA targets (type, status) and CPU synchronization
> primitives (struct completion compl) in the same struct without cacheline
> alignment?
> 
> On non-coherent architectures, could the DMA API unmap operation invalidate
> the cacheline and destroy the CPU's unwritten modifications to compl.wait,
> thereby corrupting the waitqueue? Buffers that are targets of DMA typically
> need to be cacheline-aligned or physically separated from CPU-modified
> lock/sync primitives.
> 
> [ ... ]

ACK

Update  struct completion compl to a pointer of struct completion compl.

>> +static int virtblk_ctrl_vq_request(struct virtio_blk *vblk,
>> +				    struct virtblk_ctrl_request *creq,
>> +				    struct scatterlist *sgs[],
>> +				    unsigned int out_sgs, unsigned int in_sgs)
>> +{
> [ ... ]
>> +	wait_for_completion(&creq->compl);
>> +	mutex_unlock(&vblk->ctrl_vq.mutex);
>> +	return 0;
>> +}
> 
> [Severity: High]
> Could this unbounded wait deadlock device teardown if the host is
> unresponsive?
> 
> When called from a block I/O context, this thread holds a reference to
> q_usage_counter. If the host hangs, del_gendisk() in virtblk_remove() will
> freeze the block queue and wait indefinitely for q_usage_counter to reach
> zero.
> 
> Since virtblk_ctrl_vq_drain() is called after del_gendisk(), wouldn't the
> system deadlock before the drain can recover the stuck requests?
> 
> [ ... ]

ACK

Add abandon flag in struct virtblk_ctrl_request. Set a 10 seconds timeout when
sending a request via the control queue. Set the control request as abandoned
once it doesn't return in 10 seconds to force finishing the caller in the control
queue, so that q_usage_counter in the data I/O thread can decrease to 0, unblock
del_gendisk() finally.

>> @@ -1553,6 +1713,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>>  out_free_vq:
>>  	vdev->config->del_vqs(vdev);
>>  	kfree(vblk->vqs);
>> +	vblk->ctrl_vq.vq = NULL;
>>  out_free_vblk:
>>  	kfree(vblk);
>>  out_free_index:
> 
> [Severity: High]
> This is a pre-existing issue, but does the error path in virtblk_probe()
> trigger a use-after-free and double-free?
> 
> If device_add_disk() fails, the path jumps to out_cleanup_disk which calls
> put_disk(vblk->disk). Because vblk->disk->fops points to virtblk_fops,
> dropping the last reference invokes virtblk_free_disk(), which frees vblk
> and vblk->index.
> 
> By the time execution reaches out_free_vq, aren't accesses to vblk->vqs and
> this new assignment to vblk->ctrl_vq.vq writing to freed memory, followed
> by a double free of vblk in out_free_vblk?
> 
> [ ... ]

It doesn't lead to use-after-free or double-free.

Because
  1. put_disk() calls disk_release();
  2. disk_release(）calls disk->fops->free_disk() only if GD_ADDED is set.
  3. GD_ADDED is only set in add_disk_final() after the successful call of
     device_add_disk().

GD_ADDED won't be set if device_add_disk() returns the failure, then
virtblk_free_disk() won't be called. vblk->ctrl_vq.vq is set to NULL before
free vblk.

>> @@ -1593,6 +1759,8 @@ static int virtblk_freeze_priv(struct virtio_device *vdev)
>>  	struct request_queue *q = vblk->disk->queue;
>>  	unsigned int memflags;
>>  
>> +	virtblk_ctrl_vq_quiesce(vblk);
>> +
>>  	/* Ensure no requests in virtqueues before deleting vqs. */
>>  	memflags = blk_mq_freeze_queue(q);
> 
> [Severity: High]
> By calling virtblk_ctrl_vq_quiesce() before blk_mq_freeze_queue(), could
> valid in-flight requests fail with I/O errors?
> 
> virtblk_ctrl_vq_quiesce() sets ctrl_vq.dead = true. Since the block queue
> is not yet frozen, in-flight block I/O requests that haven't yet submitted
> their control requests (but have already incremented q_usage_counter) will
> encounter dead == true when calling virtblk_ctrl_vq_request(), resulting in
> -ENODEV.
> 
> Should the data plane be frozen before the control queue is marked dead?
> 

ACK

Move virtblk_ctrl_vq_quiesce(vblk) after blk_mq_freeze_queue(q), before
virtio_reset_device().

Re: [PATCH v2 2/2] virtio_blk: add inline encryption support

[Linlin Zhang]

On 9/14/2026 10:19 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
> 
> New issues:
> - [Critical] REQ_OP_ZONE_APPEND requests with an encryption context are silently written in plaintext.
> - [High] Zoned block devices completely skip inline encryption initialization due to an 'else if' block.
> - [High] Memory allocation on the block I/O submission path uses GFP_KERNEL without a mempool, causing deadlocks.
> - [High] Hardware queues are unquiesced before encryption keys are reprogrammed during resume.
> - [High] Cacheline sharing in `struct virtblk_ctrl_request` violates the DMA API, risking memory corruption.
> - [High] Uninitialized vblk->crypto_profile_initialized leads to wild frees on probe error paths.
> - [High] blk-crypto operations can race with device removal, leading to NULL pointer dereference.
> - [Medium] The driver directly assigns Linux-internal crypto mode enums to the Virtio hardware descriptor without translation.
> - [Low] VIRTIO_BLK_INLINE_ENCRYPTION is defined as a tristate but cannot be built as an independent module.
> 
> Pre-existing issues:
> - [High] The error cleanup path in virtblk_probe contains a use-after-free and double free of vblk.
> --
> 
> commit 787711d9770fce24131418d43dc33cb33a894de6
> Author: linlzhan <linlin.zhang@oss.qualcomm.com>
> 
> virtio_blk: add inline encryption support
> 
> This commit adds support for the virtio-blk inline encryption feature,
> allowing guest VMs to offload per-I/O encryption to the host's inline
> crypto engine. It wires up device encryption characteristics to a
> struct blk_crypto_profile and handles key management via control commands.
> 
>> diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig
>> index 858320b6ebb7e..58bb050d4617c 100644
>> --- a/drivers/block/Kconfig
>> +++ b/drivers/block/Kconfig
>> @@ -372,4 +372,16 @@ config BLK_DEV_ZONED_LOOP
>>  
>>  	  If unsure, say N.
>>  
>> +config VIRTIO_BLK_INLINE_ENCRYPTION
>> +	tristate "Virtio block inline encryption support"
>> +	depends on VIRTIO_BLK && BLK_INLINE_ENCRYPTION
> 
> [Severity: Low]
> Could this lead to configuration ambiguity? VIRTIO_BLK_INLINE_ENCRYPTION is
> defined as a tristate, but the code is conditionally included directly within
> the virtio_blk module. Users might select 'M' thinking it builds a separate
> crypto module, but it functions as an intra-file boolean.
> 

Setting it as 'tristate' here to ensure that VIRTIO_BLK with inline encryption
supoort can still be built as a module. I.e. if CONFIG_VIRTIO_BLK=m,
CONFIG_VIRTIO_BLK_INLINE_ENCRYPTION must also be m.

> [ ... ]
> 
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index 252ec93661532..3fd318e5e1f17 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -124,8 +136,23 @@ struct virtblk_req {
>>  };
>>  
>>  struct virtblk_ctrl_request {
>> +	/* Type byte, always its own out-sg for every command. */
>>  	__virtio32 type;
>> +	/* Out request, sent as a second, separate out-sg if any. */
>> +	union {
>> +		struct virtio_blk_crypto_key_desc key_desc;
>> +		struct virtio_blk_crypto_key_blob blob;
>> +	} out_req;
>> +
>> +	/* In response */
>> +	union {
>> +		struct virtio_blk_crypto_key_blob blob;
>> +		struct virtio_blk_crypto_sw_secret secret;
>> +		struct virtio_blk_crypto_modes modes;
>> +	} in_resp;
>> +	/* Status byte, always its own in-sg for every command. */
>>  	u8 status;
>> +
>>  	struct completion compl;
>>  };
> 
> [Severity: High]
> Does this struct packing risk memory corruption on non-cache-coherent
> architectures?
> 
> The DMA-written buffers (in_resp, status) are packed directly adjacent to the
> compl field. On non-coherent architectures, cache invalidation for the DMA
> fields can destroy the CPU's ongoing modifications to the waitqueue inside
> compl during wait_for_completion().
> 

ACK

update struct completion compl to a pointer of struct completion.

> [ ... ]
> 
>> @@ -262,20 +310,27 @@ static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
>>  				      struct request *req,
>>  				      struct virtblk_req *vbr)
>>  {
> [ ... ]
>>  	case REQ_OP_ZONE_FINISH:
>>  		type = VIRTIO_BLK_T_ZONE_FINISH;
>>  		sector = blk_rq_pos(req);
>>  		break;
>>  	case REQ_OP_ZONE_APPEND:
>>  		type = VIRTIO_BLK_T_ZONE_APPEND;
>>  		sector = blk_rq_pos(req);
>>  		in_hdr_len = sizeof(vbr->in_hdr.zone_append);
>>  		break;
> 
> [Severity: Critical]
> Could this lead to plaintext data being written for encrypted zone append
> requests?
> 
> When handling REQ_OP_ZONE_APPEND, the type is forced to
> VIRTIO_BLK_T_ZONE_APPEND. Because this path does not check is_crypto_request(),
> it bypasses the crypto setup...
> 

This isn't a bug.

Because inline encryption is only added for the non-zoned device. REQ_OP_ZONE_FINISH
and REQ_OP_ZONE_APPEND are for zoned device, which doesn't have blk-crypto-profile
registered into its request queue.

> [ ... ]
> 
>> @@ -337,6 +392,18 @@ static blk_status_t virtblk_setup_cmd(struct virtio_device *vdev,
>>  	if (type == VIRTIO_BLK_T_DISCARD || type == VIRTIO_BLK_T_WRITE_ZEROES ||
>>  	    type == VIRTIO_BLK_T_SECURE_ERASE) {
>>  		if (virtblk_setup_discard_write_zeroes_erase(req, unmap))
>>  			return BLK_STS_RESOURCE;
>>  	}
>>  
>> +	if (type == VIRTIO_BLK_T_CRYPTO_IN || type == VIRTIO_BLK_T_CRYPTO_OUT) {
>> +		memset(&vbr->out_hdr.crypto_append.msg, 0,
> 
> [Severity: Critical]
> ...and subsequently fails this check since type is VIRTIO_BLK_T_ZONE_APPEND
> instead of VIRTIO_BLK_T_CRYPTO_OUT.
> 
> This prevents the crypto message descriptor from being attached, causing the
> host hypervisor to see a standard unencrypted write and store the plaintext
> data to disk.
> 

This is not a bug. It's excepted that the out header of virtio block request isn't
appened with crypto metadata for a zoned device.

> [ ... ]
> 
>> @@ -1001,11 +1068,478 @@ static int virtblk_ctrl_vq_request(struct virtio_blk *vblk,
> [ ... ]
>> +static int set_virtblk_crypto_key_desc(struct virtio_device *vdev,
>> +				       struct virtblk_ctrl_request *creq,
>> +				       const struct blk_crypto_key *key,
>> +				       unsigned int slot)
>> +{
>> +	struct virtio_blk_crypto_key_desc *desc = &creq->out_req.key_desc;
>> +	enum blk_crypto_key_type key_type = key->crypto_cfg.key_type;
>> +
>> +	if (sizeof(desc->bytes) < key->size)
>> +		return -EOVERFLOW;
>> +
>> +	memset(desc, 0, sizeof(*desc));
>> +	desc->slot = cpu_to_virtio32(vdev, slot);
>> +	memcpy(desc->bytes, key->bytes, key->size);
>> +	desc->key_size = cpu_to_virtio32(vdev, key->size);
>> +	desc->crypto_mode = cpu_to_virtio32(vdev, key->crypto_cfg.crypto_mode);
> 
> [Severity: Medium]
> Can this leak a Linux-internal enum value directly into the hardware protocol
> descriptor?
> 
> The desc->crypto_mode is set directly from key->crypto_cfg.crypto_mode, which
> is an internal enum blk_crypto_mode_num. If Linux adds a new mode or virtio
> assigns different numbers in the future, this will silently send incorrect
> crypto modes to the hypervisor.
> 

ACK

Transmit the kernel internal blk_crypto_mode_num to the virtio Spec defined
crypto mode num.

> [ ... ]
> 
>> +static int virtblk_crypto_keyslot_program(struct blk_crypto_profile *profile,
>> +					   const struct blk_crypto_key *key,
>> +					   unsigned int slot)
>> +{
>> +	struct virtio_blk *vblk = virtblk_from_profile(profile);
>> +	struct scatterlist type_sg, out_req_sg, status_sg, *sgs[3];
>> +	struct virtblk_ctrl_request *creq;
>> +	int err;
>> +
>> +	creq = kzalloc_obj(*creq, GFP_KERNEL);
>> +	if (!creq)
>> +		return -ENOMEM;
> 
> [Severity: High]
> Is it safe to use GFP_KERNEL here?
> 
> This allocation occurs on the block I/O submission path. Under memory pressure,
> this could recurse into memory reclaim, which may attempt to flush dirty pages
> to this same block device, leading to a deadlock.
> 

ACK

Replace GFP_KERNEL with GFP_NOIO

> [ ... ]
> 
>> +static int virtblk_crypto_keyslot_evict(struct blk_crypto_profile *profile,
>> +					 const struct blk_crypto_key *key,
>> +					 unsigned int slot)
>> +{
>> +	struct virtio_blk *vblk = virtblk_from_profile(profile);
>> +	struct scatterlist type_sg, out_req_sg, status_sg, *sgs[3];
>> +	struct virtblk_ctrl_request *creq;
>> +	int err;
>> +
>> +	creq = kzalloc_obj(*creq, GFP_KERNEL);
>> +	if (!creq)
>> +		return -ENOMEM;
> 
> [Severity: High]
> Similar to the programming path, is GFP_KERNEL appropriate here? The eviction
> path often occurs during inode or dentry reclaim, and recursing into memory
> reclaim could cause deadlocks.
> 

ACK

Replace GFP_KERNEL with GFP_NOIO

>> +
>> +	creq->type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_CRYPTO_KEYSLOT_EVICT);
> 
> [Severity: High]
> Could this race with device removal?
> 
> In virtblk_crypto_keyslot_evict(), vblk->vdev is accessed unconditionally. If
> the virtio device is hot-unplugged, virtblk_remove() sets vblk->vdev to NULL.
> Since the block device may still be open by userspace, fscrypt might
> subsequently tear down a key and call into this driver, causing a NULL pointer
> dereference.
> 

ACK

Protect key program/evict/derive_sw_secret/generate/prepare/import by acquiring
vdev_mutex lock.

> [ ... ]
> 
>> +static void virtblk_destroy_crypto(struct virtio_blk *vblk)
>> +{
>> +	if (vblk->crypto_profile_initialized)
>> +		blk_crypto_profile_destroy(&vblk->profile);
>> +}
> 
> [Severity: High]
> Could this cause a wild free on probe error paths?
> 
> vblk is allocated using kmalloc_obj() which does not zero memory. If the probe
> fails before the profile is initialized, virtblk_destroy_crypto() might read a
> true value for vblk->crypto_profile_initialized and pass garbage to
> blk_crypto_profile_destroy().
> 
> [ ... ]
> 
>> @@ -1698,6 +2232,19 @@ static int virtblk_probe(struct virtio_device *vdev)
> [ ... ]
>>  	if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) &&
>>  	    (lim.features & BLK_FEAT_ZONED)) {
>>  		err = blk_revalidate_disk_zones(vblk->disk);
>>  		if (err)
>>  			goto out_cleanup_disk;
>> +	} else if (IS_ENABLED(CONFIG_VIRTIO_BLK_INLINE_ENCRYPTION) &&
>> +		   virtio_has_feature(vdev, VIRTIO_BLK_F_INLINE_ENCRYPTION) &&
>> +		   virtio_has_feature(vdev, VIRTIO_BLK_F_CTRL_VQ)) {
>> +		err = virtblk_init_crypto(vblk);
> 
> [Severity: High]
> Does this else if inadvertently disable inline encryption for zoned block
> devices?
> 
> If a device advertises both ZONED and INLINE_ENCRYPTION features, the initial
> if block executes, causing the entire crypto initialization block to be skipped
> silently.
> 

ACK

Move the check of inline encryption support to a independent conditional branch.
Print the log in Zoned device case to show the mutual exclusion of zone device
and inline encryption support.

> [ ... ]
> 
>>  	err = device_add_disk(&vdev->dev, vblk->disk, virtblk_attr_groups);
>>  	if (err)
>>  		goto out_cleanup_disk;
>>  
>>  	return 0;
>>  
>>  out_cleanup_disk:
>>  	put_disk(vblk->disk);
>>  out_free_tags:
>>  	blk_mq_free_tag_set(&vblk->tag_set);
>>  out_free_vq:
>>  	vdev->config->del_vqs(vdev);
>>  	kfree(vblk->vqs);
>>  	vblk->ctrl_vq.vq = NULL;
>>  out_free_vblk:
>>  	kfree(vblk);
> 
> [Severity: High]
> This is a pre-existing issue, but does this error cleanup path have a
> use-after-free and double free?
> 
> The call to put_disk(vblk->disk) drops the reference count, invoking
> virtblk_free_disk() which frees vblk. The subsequent labels continue to
> dereference vblk->tag_set and vblk->vqs, and eventually kfree(vblk) is called
> again.
> 
> [ ... ]

This isn't a issue.

As GD_ADDED only be set in the successful case of device_add_disk(), and
disk->fops->free_disk() to virtblk_free_disk() is only called when GD_ADDED is set.

when device_add_disk() returns non-zero,  virtblk_free_disk() won't be called,
vblk isn't freed before vblk->ctrl_vq.vq = NULL.

> 
>> @@ -1798,6 +2345,9 @@ static int virtblk_restore_priv(struct virtio_device *vdev)
>>  	virtio_device_ready(vdev);
>>  	blk_mq_unquiesce_queue(vblk->disk->queue);
>>  
>> +	if (vblk->profile.slots)
>> +		blk_crypto_reprogram_all_keys(&vblk->profile);
>> +
>>  	return 0;
>>  }
> 
> [Severity: High]
> Could this expose unprovisioned keyslots during resume?
> 
> The hardware queues are unquiesced before blk_crypto_reprogram_all_keys() is
> called. This instantly restarts I/O dispatch, meaning any pending encrypted
> requests could be executed by the hypervisor with empty or invalid keyslots
> before they are reprogrammed.
>