* Re: RCU warning off ublk_buf_cleanup() -> mas_for_each()
From: Bernd Schubert @ 2026-04-23 21:51 UTC (permalink / raw)
To: Jens Axboe, Ming Lei
Cc: io-uring, linux-block@vger.kernel.org, Liam R. Howlett
In-Reply-To: <331c739c-d0fd-4c53-bbd5-cbb18f5dfedb@kernel.dk>
On 4/23/26 23:48, Jens Axboe wrote:
> On 4/23/26 3:45 PM, Bernd Schubert wrote:
>>
>>
>> On 4/21/26 19:47, Jens Axboe wrote:
>>> Hi Ming,
>>>
>>> Ran into the below running tests on the current tree:
>>>
>>> =============================
>>> WARNING: suspicious RCU usage
>>> 7.0.0+ #16 Tainted: G N
>>> -----------------------------
>>> lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!
>>>
>>> other info that might help us debug this:
>>>
>>>
>>> rcu_scheduler_active = 2, debug_locks = 1
>>> 1 lock held by iou-wrk-55535/55536:
>>> #0: ffff800085a451a0 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0xdc/0x2f8
>>>
>>> stack backtrace:
>>> CPU: 4 UID: 0 PID: 55536 Comm: iou-wrk-55535 Tainted: G N 7.0.0+ #16 PREEMPT
>>> Tainted: [N]=TEST
>>> Hardware name: linux,dummy-virt (DT)
>>> Call trace:
>>> show_stack+0x1c/0x30 (C)
>>> dump_stack_lvl+0x68/0x90
>>> dump_stack+0x18/0x20
>>> lockdep_rcu_suspicious+0x170/0x200
>>> mas_walk+0x3f0/0x6a0
>>> mas_find+0x1b4/0x6b0
>>> ublk_buf_cleanup+0xe0/0x240
>>> ublk_cdev_rel+0x34/0x1b0
>>> device_release+0xa4/0x350
>>> kobject_put+0x138/0x250
>>> put_device+0x18/0x30
>>> ublk_put_device+0x18/0x28
>>> ublk_ctrl_del_dev+0x120/0x2f8
>>> ublk_ctrl_uring_cmd+0x598/0x29b8
>>> io_uring_cmd+0x1e0/0x468
>>> __io_issue_sqe+0xa4/0x748
>>> io_issue_sqe+0x80/0xf68
>>> io_wq_submit_work+0x26c/0xdc8
>>> io_worker_handle_work+0x334/0xf20
>>> io_wq_worker+0x278/0x9e8
>>> ret_from_fork+0x10/0x20
>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>> ublkb0: unable to read partition table
>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>>
>>> and I briefly looked at it, but then just gave up as a) the maple tree
>>> documentation is not that detailed, and b) other in-tree users also just
>>> call mas_for_each() without either a lock held or RCU read side locked.
>>>
>>> Adding Liam for shedding some light on this...
>>>
>>
>> Hmm, that is another one, I run into
>>
>> Date: Thu Apr 16 15:16:04 2026 +0200
>>
>> ublk: Add rcu_read_lock/unlock in ublk_buf_cleanup
>>
>> [ 399.994025] =============================
>> [ 399.994694] WARNING: suspicious RCU usage
>> [ 399.995464] 7.0.0+ #52 Not tainted
>> [ 399.996034] -----------------------------
>> [ 399.996697] lib/maple_tree.c:780 suspicious rcu_dereference_check() usage!
>> [ 399.997748]
>> other info that might help us debug this:
>>
>> [ 399.998961]
>> rcu_scheduler_active = 2, debug_locks = 1
>> [ 399.999957] 1 lock held by iou-wrk-1495/1509:
>> [ 400.000596] #0: ffffffffa05e4590 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0x3d/0x190 [ublk_drv]
>> [ 400.001796]
>> stack backtrace:
>> [ 400.002492] CPU: 40 UID: 0 PID: 1509 Comm: iou-wrk-1495 Not tainted 7.0.0+ #52 PREEMPT
>> [ 400.002496] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.17.0-debian-1.17.0-1ubuntu1 04/01/2014
>> [ 400.002498] Call Trace:
>> [ 400.002500] <TASK>
>> [ 400.002503] dump_stack_lvl+0x54/0x70
>> [ 400.002511] lockdep_rcu_suspicious+0x119/0x160
>> [ 400.002521] mas_start+0xed/0x140
>> [ 400.002531] mas_find+0x1a6/0x270
>> [ 400.002538] ublk_cdev_rel+0xb8/0x320 [ublk_drv]
>>
>>
>> And have in my pbuf branch
>>
>> index eb9b5dd8122c..208f6b5ad892 100644
>> --- a/drivers/block/ublk_drv.c
>> +++ b/drivers/block/ublk_drv.c
>> @@ -5478,6 +5478,7 @@ static void ublk_buf_cleanup(struct ublk_device *ub)
>> struct ublk_buf_range *range;
>> struct page *pages[32];
>>
>> + rcu_read_lock();
>> mas_for_each(&mas, range, ULONG_MAX) {
>> unsigned long base = mas.index;
>> unsigned long nr = mas.last - base + 1;
>> @@ -5495,6 +5496,7 @@ static void ublk_buf_cleanup(struct ublk_device *ub)
>> }
>> kfree(range);
>> }
>> + rcu_read_unlock();
>> mtree_destroy(&ub->buf_tree);
>> ida_destroy(&ub->buf_ida);
>> }
>
> Fixed in patches going to Linus tomorrow, see HEAD~3..HEAD~1 in this
> branch:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/log/?h=block-7.1
>
Thank you, will check after getting some sleep.
^ permalink raw reply
* Re: RCU warning off ublk_buf_cleanup() -> mas_for_each()
From: Jens Axboe @ 2026-04-23 22:02 UTC (permalink / raw)
To: Bernd Schubert, Ming Lei
Cc: io-uring, linux-block@vger.kernel.org, Liam R. Howlett
In-Reply-To: <4ec7a5a2-2338-4440-aa7f-b193aac0217f@bsbernd.com>
On 4/23/26 3:51 PM, Bernd Schubert wrote:
>
>
> On 4/23/26 23:48, Jens Axboe wrote:
>> On 4/23/26 3:45 PM, Bernd Schubert wrote:
>>>
>>>
>>> On 4/21/26 19:47, Jens Axboe wrote:
>>>> Hi Ming,
>>>>
>>>> Ran into the below running tests on the current tree:
>>>>
>>>> =============================
>>>> WARNING: suspicious RCU usage
>>>> 7.0.0+ #16 Tainted: G N
>>>> -----------------------------
>>>> lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!
>>>>
>>>> other info that might help us debug this:
>>>>
>>>>
>>>> rcu_scheduler_active = 2, debug_locks = 1
>>>> 1 lock held by iou-wrk-55535/55536:
>>>> #0: ffff800085a451a0 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0xdc/0x2f8
>>>>
>>>> stack backtrace:
>>>> CPU: 4 UID: 0 PID: 55536 Comm: iou-wrk-55535 Tainted: G N 7.0.0+ #16 PREEMPT
>>>> Tainted: [N]=TEST
>>>> Hardware name: linux,dummy-virt (DT)
>>>> Call trace:
>>>> show_stack+0x1c/0x30 (C)
>>>> dump_stack_lvl+0x68/0x90
>>>> dump_stack+0x18/0x20
>>>> lockdep_rcu_suspicious+0x170/0x200
>>>> mas_walk+0x3f0/0x6a0
>>>> mas_find+0x1b4/0x6b0
>>>> ublk_buf_cleanup+0xe0/0x240
>>>> ublk_cdev_rel+0x34/0x1b0
>>>> device_release+0xa4/0x350
>>>> kobject_put+0x138/0x250
>>>> put_device+0x18/0x30
>>>> ublk_put_device+0x18/0x28
>>>> ublk_ctrl_del_dev+0x120/0x2f8
>>>> ublk_ctrl_uring_cmd+0x598/0x29b8
>>>> io_uring_cmd+0x1e0/0x468
>>>> __io_issue_sqe+0xa4/0x748
>>>> io_issue_sqe+0x80/0xf68
>>>> io_wq_submit_work+0x26c/0xdc8
>>>> io_worker_handle_work+0x334/0xf20
>>>> io_wq_worker+0x278/0x9e8
>>>> ret_from_fork+0x10/0x20
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> ublkb0: unable to read partition table
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>>>
>>>> and I briefly looked at it, but then just gave up as a) the maple tree
>>>> documentation is not that detailed, and b) other in-tree users also just
>>>> call mas_for_each() without either a lock held or RCU read side locked.
>>>>
>>>> Adding Liam for shedding some light on this...
>>>>
>>>
>>> Hmm, that is another one, I run into
>>>
>>> Date: Thu Apr 16 15:16:04 2026 +0200
>>>
>>> ublk: Add rcu_read_lock/unlock in ublk_buf_cleanup
>>>
>>> [ 399.994025] =============================
>>> [ 399.994694] WARNING: suspicious RCU usage
>>> [ 399.995464] 7.0.0+ #52 Not tainted
>>> [ 399.996034] -----------------------------
>>> [ 399.996697] lib/maple_tree.c:780 suspicious rcu_dereference_check() usage!
>>> [ 399.997748]
>>> other info that might help us debug this:
>>>
>>> [ 399.998961]
>>> rcu_scheduler_active = 2, debug_locks = 1
>>> [ 399.999957] 1 lock held by iou-wrk-1495/1509:
>>> [ 400.000596] #0: ffffffffa05e4590 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0x3d/0x190 [ublk_drv]
>>> [ 400.001796]
>>> stack backtrace:
>>> [ 400.002492] CPU: 40 UID: 0 PID: 1509 Comm: iou-wrk-1495 Not tainted 7.0.0+ #52 PREEMPT
>>> [ 400.002496] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.17.0-debian-1.17.0-1ubuntu1 04/01/2014
>>> [ 400.002498] Call Trace:
>>> [ 400.002500] <TASK>
>>> [ 400.002503] dump_stack_lvl+0x54/0x70
>>> [ 400.002511] lockdep_rcu_suspicious+0x119/0x160
>>> [ 400.002521] mas_start+0xed/0x140
>>> [ 400.002531] mas_find+0x1a6/0x270
>>> [ 400.002538] ublk_cdev_rel+0xb8/0x320 [ublk_drv]
>>>
>>>
>>> And have in my pbuf branch
>>>
>>> index eb9b5dd8122c..208f6b5ad892 100644
>>> --- a/drivers/block/ublk_drv.c
>>> +++ b/drivers/block/ublk_drv.c
>>> @@ -5478,6 +5478,7 @@ static void ublk_buf_cleanup(struct ublk_device *ub)
>>> struct ublk_buf_range *range;
>>> struct page *pages[32];
>>>
>>> + rcu_read_lock();
>>> mas_for_each(&mas, range, ULONG_MAX) {
>>> unsigned long base = mas.index;
>>> unsigned long nr = mas.last - base + 1;
>>> @@ -5495,6 +5496,7 @@ static void ublk_buf_cleanup(struct ublk_device *ub)
>>> }
>>> kfree(range);
>>> }
>>> + rcu_read_unlock();
>>> mtree_destroy(&ub->buf_tree);
>>> ida_destroy(&ub->buf_ida);
>>> }
>>
>> Fixed in patches going to Linus tomorrow, see HEAD~3..HEAD~1 in this
>> branch:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/log/?h=block-7.1
>>
>
> Thank you, will check after getting some sleep.
Sounds good - fixes the issue on my end.
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH v2] floppy: fix reference leak on platform_device_register() failure
From: Guangshuo Li @ 2026-04-24 3:14 UTC (permalink / raw)
To: Jens Axboe
Cc: Jiri Slaby, Denis Efremov, Greg Kroah-Hartman, linux-block,
linux-kernel, stable
In-Reply-To: <f661cf47-18bb-44f2-8764-c9f0b4fb68b1@kernel.dk>
Hi Jiri, Jens,
Thanks for the review and for catching this.
On Thu, 23 Apr 2026 at 19:06, Jens Axboe <axboe@kernel.dk> wrote:
>
> > The patch is likely wrong. Given the pdev is static, the struct device
> > has no ->release, so releasing it will trigger a warning. AFAIR, the
> > consensus was to fix platform_device_register() proper.
>
> Thanks for letting me know, I'll revert this change for now.
>
You are right that this fix is not appropriate for this case. Since the
platform device is static and the embedded struct device has no release
callback, calling platform_device_put() / releasing it here is wrong and
can trigger warnings.
Please disregard this patch. I will drop it and not pursue it further in
its current form.
Sorry for the noise, and thanks again for the clarification.
Best regards,
Guangshuo Li
^ permalink raw reply
* Re: [PATCH v2] floppy: fix reference leak on platform_device_register() failure
From: Greg KH @ 2026-04-24 4:15 UTC (permalink / raw)
To: Guangshuo Li
Cc: Jens Axboe, Jiri Slaby, Denis Efremov, Greg Kroah-Hartman,
linux-block, linux-kernel, stable
In-Reply-To: <CANUHTR8W7tz0me90GDci97ee6N+3MpB7uVYYFN0dtTf9u_Ui2g@mail.gmail.com>
On Fri, Apr 24, 2026 at 11:14:17AM +0800, Guangshuo Li wrote:
> Hi Jiri, Jens,
>
> Thanks for the review and for catching this.
>
> On Thu, 23 Apr 2026 at 19:06, Jens Axboe <axboe@kernel.dk> wrote:
> >
> > > The patch is likely wrong. Given the pdev is static, the struct device
> > > has no ->release, so releasing it will trigger a warning. AFAIR, the
> > > consensus was to fix platform_device_register() proper.
> >
> > Thanks for letting me know, I'll revert this change for now.
> >
>
> You are right that this fix is not appropriate for this case. Since the
> platform device is static and the embedded struct device has no release
> callback, calling platform_device_put() / releasing it here is wrong and
> can trigger warnings.
>
> Please disregard this patch. I will drop it and not pursue it further in
> its current form.
Can you go back and verify that all of the other patches you sent out
for this same pattern are also either ignored, or reverted?
thanks,
greg k-h
^ permalink raw reply
* Re: [PATCH v2] floppy: fix reference leak on platform_device_register() failure
From: Guangshuo Li @ 2026-04-24 7:40 UTC (permalink / raw)
To: Greg KH
Cc: Jens Axboe, Jiri Slaby, Denis Efremov, Greg Kroah-Hartman,
linux-block, linux-kernel, stable
In-Reply-To: <2026042411-repressed-manliness-4ee9@gregkh>
Hi Greg,
Thanks for reviewing and pointing this out.
On Fri, 24 Apr 2026 at 12:15, Greg KH <gregkh@linuxfoundation.org> wrote:
>
>
> Can you go back and verify that all of the other patches you sent out
> for this same pattern are also either ignored, or reverted?
>
> thanks,
>
> greg k-h
Yes, I will go back and check all the other patches I sent for this same
pattern. For any cases involving static platform devices or similar static
objects where this fix is not appropriate, I will follow up by email to
make sure they are either ignored or reverted as needed.
Thanks again for the guidance.
Best regards,
Guangshuo Li
^ permalink raw reply
* Re: [PATCH v3] Fix null-ptr-deref in bio_integrity_map_user()
From: Anuj gupta @ 2026-04-24 8:24 UTC (permalink / raw)
To: Sungwoo Kim
Cc: Jens Axboe, Keith Busch, Chao Shi, Weidong Zhu, Dave Tian,
linux-block, linux-kernel
In-Reply-To: <20260420020327.1667156-3-iam@sung-woo.kim>
Nit: the subject use the usual block prefix
^ permalink raw reply
* [PATCH] block: rbd: switch to dynamic root device
From: Johan Hovold @ 2026-04-24 10:39 UTC (permalink / raw)
To: Ilya Dryomov
Cc: Dongsheng Yang, Jens Axboe, ceph-devel, linux-block, linux-kernel,
Johan Hovold
Driver core expects devices to be dynamically allocated and will, for
example, complain loudly when no release function has been provided.
Use root_device_register() to allocate and register the root device
instead of open coding using a static device.
Signed-off-by: Johan Hovold <johan@kernel.org>
---
drivers/block/rbd.c | 23 +++++++----------------
1 file changed, 7 insertions(+), 16 deletions(-)
diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index e7da06200c1e..ffab1b4e7c25 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -580,14 +580,7 @@ static const struct bus_type rbd_bus_type = {
.bus_groups = rbd_bus_groups,
};
-static void rbd_root_dev_release(struct device *dev)
-{
-}
-
-static struct device rbd_root_dev = {
- .init_name = "rbd",
- .release = rbd_root_dev_release,
-};
+static struct device *rbd_root_dev;
static __printf(2, 3)
void rbd_warn(struct rbd_device *rbd_dev, const char *fmt, ...)
@@ -5390,7 +5383,7 @@ static struct rbd_device *__rbd_dev_create(struct rbd_spec *spec)
rbd_dev->dev.bus = &rbd_bus_type;
rbd_dev->dev.type = &rbd_device_type;
- rbd_dev->dev.parent = &rbd_root_dev;
+ rbd_dev->dev.parent = rbd_root_dev;
device_initialize(&rbd_dev->dev);
return rbd_dev;
@@ -7331,15 +7324,13 @@ static int __init rbd_sysfs_init(void)
{
int ret;
- ret = device_register(&rbd_root_dev);
- if (ret < 0) {
- put_device(&rbd_root_dev);
- return ret;
- }
+ rbd_root_dev = root_device_register("rbd");
+ if (IS_ERR(rbd_root_dev))
+ return PTR_ERR(rbd_root_dev);
ret = bus_register(&rbd_bus_type);
if (ret < 0)
- device_unregister(&rbd_root_dev);
+ root_device_unregister(rbd_root_dev);
return ret;
}
@@ -7347,7 +7338,7 @@ static int __init rbd_sysfs_init(void)
static void __exit rbd_sysfs_cleanup(void)
{
bus_unregister(&rbd_bus_type);
- device_unregister(&rbd_root_dev);
+ root_device_unregister(rbd_root_dev);
}
static int __init rbd_slab_init(void)
--
2.53.0
^ permalink raw reply related
* Re: [PATCH v2 3/3] scsi: use percpu counters for iorequest_cnt and iodone_cnt
From: Sumit Saxena @ 2026-04-24 11:17 UTC (permalink / raw)
To: John Garry
Cc: martin.petersen, axboe, linux-scsi, linux-block,
mpi3mr-linuxdrv.pdl, Bart Van Assche
In-Reply-To: <ab9e8e26-5522-4072-a2fe-07df5cb64441@oracle.com>
[-- Attachment #1: Type: text/plain, Size: 9277 bytes --]
On Tue, Apr 21, 2026 at 2:16 PM John Garry <john.g.garry@oracle.com> wrote:
>
> On 20/04/2026 12:38, Sumit Saxena wrote:
> > iorequest_cnt and iodone_cnt are updated on every command dispatch and
> > completion, often from different CPUs on high queue depth workloads.
> > Using adjacent atomic_t fields caused cache line contention between the
> > submission and completion paths.
> >
> > Represent these statistics with struct percpu_counter so increments are
> > mostly local to each CPU, avoiding false sharing without growing
> > struct scsi_device further for cache-line padding.
> >
> > Suggested-by: Bart Van Assche <bvanassche@acm.org>
> > Signed-off-by: Sumit Saxena <sumit.saxena@broadcom.com>
> > ---
> > drivers/scsi/scsi_error.c | 2 +-
> > drivers/scsi/scsi_lib.c | 8 ++++----
> > drivers/scsi/scsi_scan.c | 9 +++++++++
> > drivers/scsi/scsi_sysfs.c | 27 +++++++++++++++++++++++----
> > include/scsi/scsi_device.h | 5 +++--
> > 5 files changed, 40 insertions(+), 11 deletions(-)
> >
> > diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
> > index 147127fb4db9..c7424ce92f3e 100644
> > --- a/drivers/scsi/scsi_error.c
> > +++ b/drivers/scsi/scsi_error.c
> > @@ -370,7 +370,7 @@ enum blk_eh_timer_return scsi_timeout(struct request *req)
> > */
> > if (test_and_set_bit(SCMD_STATE_COMPLETE, &scmd->state))
> > return BLK_EH_DONE;
> > - atomic_inc(&scmd->device->iodone_cnt);
> > + percpu_counter_inc(&scmd->device->iodone_cnt);
> > if (scsi_abort_command(scmd) != SUCCESS) {
> > set_host_byte(scmd, DID_TIME_OUT);
> > scsi_eh_scmd_add(scmd);
> > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> > index 6e8c7a42603e..0b05cb63f630 100644
> > --- a/drivers/scsi/scsi_lib.c
> > +++ b/drivers/scsi/scsi_lib.c
> > @@ -1554,7 +1554,7 @@ static void scsi_complete(struct request *rq)
> >
> > INIT_LIST_HEAD(&cmd->eh_entry);
> >
> > - atomic_inc(&cmd->device->iodone_cnt);
> > + percpu_counter_inc(&cmd->device->iodone_cnt);
> > if (cmd->result)
> > atomic_inc(&cmd->device->ioerr_cnt);
> >
> > @@ -1592,7 +1592,7 @@ static enum scsi_qc_status scsi_dispatch_cmd(struct scsi_cmnd *cmd)
> > struct Scsi_Host *host = cmd->device->host;
> > int rtn = 0;
> >
> > - atomic_inc(&cmd->device->iorequest_cnt);
> > + percpu_counter_inc(&cmd->device->iorequest_cnt);
> >
> > /* check if the device is still usable */
> > if (unlikely(cmd->device->sdev_state == SDEV_DEL)) {
> > @@ -1614,7 +1614,7 @@ static enum scsi_qc_status scsi_dispatch_cmd(struct scsi_cmnd *cmd)
> > */
> > SCSI_LOG_MLQUEUE(3, scmd_printk(KERN_INFO, cmd,
> > "queuecommand : device blocked\n"));
> > - atomic_dec(&cmd->device->iorequest_cnt);
> > + percpu_counter_dec(&cmd->device->iorequest_cnt);
> > return SCSI_MLQUEUE_DEVICE_BUSY;
> > }
> >
> > @@ -1647,7 +1647,7 @@ static enum scsi_qc_status scsi_dispatch_cmd(struct scsi_cmnd *cmd)
> > trace_scsi_dispatch_cmd_start(cmd);
> > rtn = host->hostt->queuecommand(host, cmd);
> > if (rtn) {
> > - atomic_dec(&cmd->device->iorequest_cnt);
> > + percpu_counter_dec(&cmd->device->iorequest_cnt);
> > trace_scsi_dispatch_cmd_error(cmd, rtn);
> > if (rtn != SCSI_MLQUEUE_DEVICE_BUSY &&
> > rtn != SCSI_MLQUEUE_TARGET_BUSY)
> > diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
> > index 9749a8dbe964..0b4fa89149af 100644
> > --- a/drivers/scsi/scsi_scan.c
> > +++ b/drivers/scsi/scsi_scan.c
> > @@ -351,6 +351,15 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
> >
> > scsi_sysfs_device_initialize(sdev);
> >
> > + ret = percpu_counter_init(&sdev->iorequest_cnt, 0, GFP_KERNEL);
> > + if (ret)
> > + goto out_device_destroy;
> > + ret = percpu_counter_init(&sdev->iodone_cnt, 0, GFP_KERNEL);
> > + if (ret) {
> > + percpu_counter_destroy(&sdev->iorequest_cnt);
> > + goto out_device_destroy;
> > + }
>
> it could be neater to have:
> if (percpu_counter_init(&sdev->iorequest_cnt, 0, GFP_KERNEL) ||
> percpu_counter_init(&sdev->iodone_cnt, 0, GFP_KERNEL)) {
> err = some err;
> goto out_device_destroy;
> }
>
Ack
> > +
> > if (scsi_device_is_pseudo_dev(sdev))
> > return sdev;
> >
> > diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
> > index dfc3559e7e04..1f5b2dc156a8 100644
> > --- a/drivers/scsi/scsi_sysfs.c
> > +++ b/drivers/scsi/scsi_sysfs.c
> > @@ -516,6 +516,10 @@ static void scsi_device_dev_release(struct device *dev)
> > if (vpd_pgb7)
> > kfree_rcu(vpd_pgb7, rcu);
> > kfree(sdev->inquiry);
> > + if (percpu_counter_initialized(&sdev->iodone_cnt))
> > + percpu_counter_destroy(&sdev->iodone_cnt);
> > + if (percpu_counter_initialized(&sdev->iorequest_cnt))
> > + percpu_counter_destroy(&sdev->iorequest_cnt);
>
> Maybe I am wrong, but doesn't percpu_counter_destroy() handle the case
> of the percpu counter not being initialized? In other words, do we need
> the percpu_counter_initialized() checks?
percpu_counter_initialized() checks are not required; I will drop them.
>
> > kfree(sdev);
> >
> > if (parent)
> > @@ -936,11 +940,26 @@ static ssize_t
> > show_iostat_counterbits(struct device *dev, struct device_attribute *attr,
> > char *buf)
> > {
> > - return snprintf(buf, 20, "%d\n", (int)sizeof(atomic_t) * 8);
> > + /*
> > + * iorequest_cnt and iodone_cnt are per-CPU sums (s64); ioerr_cnt and
> > + * iotmo_cnt remain atomic_t. Report the widest counter for tools.
> > + */
> > + return snprintf(buf, 20, "%zu\n", sizeof(s64) * 8);
> > }
> >
> > static DEVICE_ATTR(iocounterbits, S_IRUGO, show_iostat_counterbits, NULL);
> >
> > +#define show_sdev_iostat_percpu(field) \
> > +static ssize_t \
> > +show_iostat_##field(struct device *dev, struct device_attribute *attr, \
> > + char *buf) \
> > +{ \
> > + struct scsi_device *sdev = to_scsi_device(dev); \
> > + unsigned long long count = percpu_counter_sum(&sdev->field); \
> > + return snprintf(buf, 20, "0x%llx\n", count); \
> > +} \
> > +static DEVICE_ATTR(field, 0444, show_iostat_##field, NULL)
> > +
> > #define show_sdev_iostat(field) \
> > static ssize_t \
> > show_iostat_##field(struct device *dev, struct device_attribute *attr, \
> > @@ -950,10 +969,10 @@ show_iostat_##field(struct device *dev, struct device_attribute *attr, \
> > unsigned long long count = atomic_read(&sdev->field); \
> > return snprintf(buf, 20, "0x%llx\n", count); \
> > } \
> > -static DEVICE_ATTR(field, S_IRUGO, show_iostat_##field, NULL)
> > +static DEVICE_ATTR(field, 0444, show_iostat_##field, NULL)
> >
> > -show_sdev_iostat(iorequest_cnt);
> > -show_sdev_iostat(iodone_cnt);
> > +show_sdev_iostat_percpu(iorequest_cnt);
> > +show_sdev_iostat_percpu(iodone_cnt);
> > show_sdev_iostat(ioerr_cnt);
> > show_sdev_iostat(iotmo_cnt);
> >
> > diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h
> > index 9c2a7bbe5891..ad80b500ced9 100644
> > --- a/include/scsi/scsi_device.h
> > +++ b/include/scsi/scsi_device.h
> > @@ -8,6 +8,7 @@
> > #include <linux/blk-mq.h>
> > #include <scsi/scsi.h>
> > #include <linux/atomic.h>
> > +#include <linux/percpu_counter.h>
> > #include <linux/sbitmap.h>
> >
> > struct bsg_device;
> > @@ -271,8 +272,8 @@ struct scsi_device {
> > unsigned int max_device_blocked; /* what device_blocked counts down from */
> > #define SCSI_DEFAULT_DEVICE_BLOCKED 3
> >
> > - atomic_t iorequest_cnt;
> > - atomic_t iodone_cnt;
> > + struct percpu_counter iorequest_cnt;
> > + struct percpu_counter iodone_cnt;
> > atomic_t ioerr_cnt;
> > atomic_t iotmo_cnt;
> >
>
> Would it be simpler to make ioerr_cnt and iotmo_cnt also as
> percpu_counter? We could then drop some sysfs code for handling atomic_t
> counter.
It looks trivial, I will handle it in next revision.
>
> I noticed that there is a percpu_counter_init_many() - maybe we could
> use that, and index into the array of counters. Note that I am not
> fimilar with that API, so it may not be a good idea.
I need to check this. If it works well, I will handle it in the next version.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5469 bytes --]
^ permalink raw reply
* Re: [PATCH v2 1/3] scsi: scan: allocate sdev and starget on the NUMA node of the host adapter
From: Sumit Saxena @ 2026-04-24 11:33 UTC (permalink / raw)
To: John Garry
Cc: martin.petersen, axboe, linux-scsi, linux-block,
mpi3mr-linuxdrv.pdl, James Rizzo
In-Reply-To: <a27a04be-6c41-4d7f-b697-b04c4fe9dc8c@oracle.com>
[-- Attachment #1: Type: text/plain, Size: 3330 bytes --]
On Mon, Apr 20, 2026 at 5:40 PM John Garry <john.g.garry@oracle.com> wrote:
>
> On 20/04/2026 12:38, Sumit Saxena wrote:
> > From: James Rizzo <james.rizzo@broadcom.com>
> >
> > When a host adapter is attached to a specific NUMA node, allocating
> > scsi_device and scsi_target via kzalloc() may place them on a remote
> > node. All hot-path I/O accesses to these structures then cross the NUMA
> > interconnect, adding latency and consuming inter-node bandwidth.
> >
> > Use kzalloc_node() with dev_to_node(shost->dma_dev) so allocations land
> > on the same node as the HBA, reducing cross-node traffic and improving
> > I/O performance on NUMA systems.
>
> I suppose that this makes sense. We already do this sort of thing in
> scsi_mq_setup_tags() (in setting numa node) and
> scsi_realloc_sdev_budget_map() ->
> sbitmap_init_node(sdev->request_queue->numa_node)
>
> For the actual shost allocation, we still use kzalloc() in
> scsi_host_alloc(). However, shost associated device is often a pci
> device, and we probe pci devices in the same NUMA node it exists, and we
> try NUMA local allocations by default, so nothing is needed to change
> for the shost allocation - is this right?
shost allocation is not an issue, I will drop changes related to it.
The tests indicate that at times sdev and starget are allocated to
remote NUMA node.
So, I will limit these changes to sdev and starget only.
>
> >
> > Signed-off-by: James Rizzo <james.rizzo@broadcom.com>
> > Signed-off-by: Sumit Saxena <sumit.saxena@broadcom.com>
> > ---
> > drivers/scsi/scsi_scan.c | 9 ++++++---
> > 1 file changed, 6 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
> > index ef22a4228b85..9749a8dbe964 100644
> > --- a/drivers/scsi/scsi_scan.c
> > +++ b/drivers/scsi/scsi_scan.c
> > @@ -34,6 +34,7 @@
> > #include <linux/kthread.h>
> > #include <linux/spinlock.h>
> > #include <linux/async.h>
> > +#include <linux/topology.h>f
> > #include <linux/slab.h>
> > #include <linux/unaligned.h>
> >
> > @@ -286,9 +287,10 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
> > int display_failure_msg = 1, ret;
> > struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);
> > struct queue_limits lim;
> > + int node = dev_to_node(shost->dma_dev);
>
> this variable is only used once, so we can use
> dev_to_node(shost->dma_dev) directly
Ack
>
> >
> > - sdev = kzalloc(sizeof(*sdev) + shost->transportt->device_size,
> > - GFP_KERNEL);
> > + sdev = kzalloc_node(sizeof(*sdev) + shost->transportt->device_size,
> > + GFP_KERNEL, node);
> > if (!sdev)
> > goto out;
> >
> > @@ -501,8 +503,9 @@ static struct scsi_target *scsi_alloc_target(struct device *parent,
> > struct scsi_target *starget;
> > struct scsi_target *found_target;
> > int error, ref_got;
> > + int node = dev_to_node(shost->dma_dev);
>
> same as above
Ack
>
> >
> > - starget = kzalloc(size, GFP_KERNEL);
> > + starget = kzalloc_node(size, GFP_KERNEL, node);
> > if (!starget) {
> > printk(KERN_ERR "%s: allocation failure\n", __func__);
> > return NULL;
>
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5469 bytes --]
^ permalink raw reply
* Re: [PATCH v2 3/3] dm: add documentation for dm-inlinecrypt target
From: Linlin Zhang @ 2026-04-24 13:53 UTC (permalink / raw)
To: Milan Broz, linux-block, ebiggers, mpatocka
Cc: linux-kernel, adrianvovk, dm-devel, quic_mdalam, israelr, hch,
axboe
In-Reply-To: <1a36c5e7-5fd6-4923-926e-65bb04c33b04@gmail.com>
On 4/11/2026 1:07 AM, Milan Broz wrote:
> On 4/10/26 3:40 PM, Linlin Zhang wrote:
>> This adds the admin-guide documentation for dm-inlinecrypt.
>>
>> dm-inlinecrypt.rst is the guide to using dm-inlinecrypt.
>>
>> Signed-off-by: Linlin Zhang <linlin.zhang@oss.qualcomm.com>
>> ---
>
> ...
>
>> +
>> +<cipher>
>> + Encryption cipher type.
>> +
>> + The cipher specifications format is::
>> +
>> + cipher
>> +
>> + Examples::
>> +
>> + aes-xts-plain64
>> +
>> + The cipher type is correspond one-to-one with encryption modes. For
>
> ... with encryption modes supported for inline crypto in block layer?
>
> In your patch only BLK_ENCRYPTION_MODE_AES_256_XTS.
Thanks for your insights!
Yes, here the encryption modes refer to the inline crypto modessupported
by the block layer. Currently, this patch only supports
BLK_ENCRYPTION_MODE_AES_256_XTS.
I will reword it as:
The cipher type corresponds to the encryption modes supported by
inline crypto in the block layer. Currently, only
BLK_ENCRYPTION_MODE_AES_256_XTS (i.e. aes-xts-plain64) is supported.
Could you please let me know if you expect more than that?
>
>> + instance, the corresponding crypto mode of aes-xts-plain64 is
>> + BLK_ENCRYPTION_MODE_AES_256_XTS.
>
> ...
>
>> +iv_large_sectors
>> + IV generators will use sector number counted in <sector_size> units
>> + instead of default 512 bytes sectors.
>> +
>> + For example, if <sector_size> is 4096 bytes, plain64 IV for the second
>> + sector will be 8 (without flag) and 1 if iv_large_sectors is present.
>> + The <iv_offset> must be multiple of <sector_size> (in 512 bytes units)
>> + if this flag is specified.
>
> Is it true? I see this comment in the code:
>
> /* dm-inlinecrypt doesn't implement iv_large_sectors=false. */
Thanks for your comment!
The example is describing the general IV generation semantics of
iv_large_sectors versus the legacy behavior, i.e. how plain64 IVs
would be computed conceptually with and without the flag.
However, for dm-inlinecrypt, the comment you quoted is correct:
iv_large_sectors=false is not implemented. When a sector size
larger than 512 bytes is used, iv_large_sectors is mandatory, and
the legacy 512-byte-based IV behavior is intentionally unsupported.
In the code this is enforced by rejecting configurations where
sector_size != 512 and iv_large_sectors is not specified, so in
practice the “without flag” case is not usable for dm-inlinecrypt.
I reword it as:
iv_large_sectors
Use <sector_size>-based sector numbers for IV generation instead of
512-byte sectors.
For dm-inlinecrypt, this flag must be specified when <sector_size>
is larger than 512 bytes. The legacy 512-byte-based IV behavior is
not supported.
When specified, if <sector_size> is 4096 bytes, plain64 IV for the
second sector will be 1, and <iv_offset> must be a multiple of
<sector_size> (in 512-byte units).
Do think it's enough?
>
> ...
>
>> +Example scripts
>> +===============
>> +LUKS (Linux Unified Key Setup) is now the preferred way to set up disk
>> +encryption with dm-inlinecrypt using the 'cryptsetup' utility, see
>> +https://gitlab.com/cryptsetup/cryptsetup
>
> Cryptsetup has no support for inlinecrypt and it is question if it should have.
> It would require additional options and maybe LUKS2 metadata flag to make it persistent.
>
> How did you test it? Please remove this cryptsetup example.
> It can be added later when userspace get this functionality.
You are right.
cryptsetup currently has no support for dm-inlinecrypt, and the example
would indeed create a dm-crypt device instead. Supporting dm-inlinecrypt
in cryptsetup would require explicit userspace changes and possibly
extensions to LUKS2 metadata to make it persistent.
I did the testing using dmsetup directly, not via cryptsetup/LUKS. And
I'll remove the LUKS/cryptsetup references and examples from the
documentation and leave LUKS integration to be documented once
userspace support exists.
I reword it as:
Currently, dm-inlinecrypt devices must be set up directly using dmsetup.
There is no userspace support yet to integrate dm-inlinecrypt with LUKS
or cryptsetup. In particular, cryptsetup currently only supports
dm-crypt, and cannot be used to create dm-inlinecrypt mappings.
The following examples demonstrate how to create dm-inlinecrypt devices
using dmsetup.
>
> ...> +
>> + #!/bin/sh
>> + # Create a inlinecrypt device using cryptsetup and LUKS header with default cipher
>> + cryptsetup luksFormat $1
>> + cryptsetup luksOpen $1 inlinecrypt1
>
> ditto. This example will use dm-crypt, not dm-inlinecrypt.
ACK
>
> Milan
>
^ permalink raw reply
* [PATCH blktests] blktrace/001: Skip test when kernel lockdown is enabled
From: Disha Goel @ 2026-04-24 14:11 UTC (permalink / raw)
To: linux-block; +Cc: shinichiro.kawasaki, Disha Goel
The blktrace/001 test fails on systems with Secure Boot enabled due to
kernel lockdown preventing access to debugfs. The test attempts to run
blktrace which requires access to /sys/kernel/debug/block/*/trace*
files, but kernel lockdown (enabled automatically with Secure Boot)
blocks this access, resulting in "Operation not permitted" errors.
Add _have_debugfs_access() helper function to detect kernel lockdown by
checking /sys/kernel/security/lockdown for [integrity] or
[confidentiality] modes. When detected, the test now skips gracefully
with a clear message: "kernel lockdown is enabled (Secure Boot may be
active)".
Also add _have_configfs check which was missing. The test uses
_configure_null_blk() which requires configfs to be mounted at
/sys/kernel/config.
Signed-off-by: Disha Goel <disgoel@linux.ibm.com>
---
common/rc | 14 ++++++++++++++
tests/blktrace/001 | 2 ++
2 files changed, 16 insertions(+)
diff --git a/common/rc b/common/rc
index 5350057..50d63e7 100644
--- a/common/rc
+++ b/common/rc
@@ -205,6 +205,20 @@ _have_blktrace() {
_have_program blktrace
}
+_have_debugfs_access() {
+ # Check if kernel lockdown is preventing debugfs access
+ # This happens when Secure Boot is enabled
+ if [[ -f /sys/kernel/security/lockdown ]]; then
+ local lockdown_state
+ lockdown_state=$(cat /sys/kernel/security/lockdown 2>/dev/null)
+ if [[ "$lockdown_state" =~ \[integrity\]|\[confidentiality\] ]]; then
+ SKIP_REASONS+=("kernel lockdown is enabled (Secure Boot may be active)")
+ return 1
+ fi
+ fi
+ return 0
+}
+
_have_configfs() {
if ! findmnt -t configfs /sys/kernel/config >/dev/null; then
SKIP_REASONS+=("configfs is not mounted at /sys/kernel/config")
diff --git a/tests/blktrace/001 b/tests/blktrace/001
index 2cdad02..4fe7807 100755
--- a/tests/blktrace/001
+++ b/tests/blktrace/001
@@ -22,6 +22,8 @@ requires() {
_have_program blkzone
_have_null_blk
_have_module_param null_blk zoned
+ _have_configfs
+ _have_debugfs_access
}
test() {
--
2.45.1
^ permalink raw reply related
* [GIT PULL] Block fixes for 7.1-rc1
From: Jens Axboe @ 2026-04-24 15:24 UTC (permalink / raw)
To: Linus Torvalds; +Cc: linux-block@vger.kernel.org
Hi Linus,
Set of fixes for block, that should go into the 7.1-rc1 kernel release.
Nothing major in here, mostly just some generic fixes or fixes for
issues introduced in this merge window. This pull request contains:
- Series for zloop, fixing a variety of issues.
- t10-pi code cleanup
- Fix for a merge window regression with the bio memory allocation mask.
- Fix for a merge window regression in ublk, caused by an issue with the
maple tree iteration code at teardown.
- ublk self tests additions.
- Zoned device pgmap fixes.
- Various little cleanups and fixes.
Please pull!
The following changes since commit 5bdb4078e1efba9650c03753616866192d680718:
Merge tag 'sched_ext-for-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext (2026-04-15 10:54:24 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git tags/block-7.1-20260424
for you to fetch changes up to 895a9b37917d2718ef2240a7ead7458c22f1f011:
Revert "floppy: fix reference leak on platform_device_register() failure" (2026-04-23 05:07:37 -0600)
----------------------------------------------------------------
block-7.1-20260424
----------------------------------------------------------------
Caleb Sander Mateos (4):
t10-pi: reduce ref tag code duplication
selftests: ublk: remove unused argument to _cleanup
selftests: ublk: enable test_integrity_02.sh on fio 3.42
selftests: ublk: add ublk auto integrity test
Christoph Hellwig (7):
zloop: fix write pointer calculation in zloop_forget_cache
zloop: use vfs_truncate
zloop: improve the unaligned write pointer warning
zloop: set RQF_QUIET when completing requests on deleted devices
zloop: factor out zloop_mark_{full,empty} helpers
zloop: remove irq-safe locking
block: only restrict bio allocation gfp mask asked to block
Guangshuo Li (1):
floppy: fix reference leak on platform_device_register() failure
Jens Axboe (1):
Revert "floppy: fix reference leak on platform_device_register() failure"
Marco Crivellari (2):
block: Add WQ_PERCPU to alloc_workqueue users
block/blk-throttle: Add WQ_PERCPU to alloc_workqueue users
Ming Lei (4):
ublk: use unchecked copy helpers for bio page data
ublk: fix maple tree lockdep warning in ublk_buf_cleanup
ublk: refactor common helper ublk_shmem_remove_ranges()
ublk: avoid unpinning pages under maple tree spinlock
Naman Jain (2):
block: add pgmap check to biovec_phys_mergeable
block: relax pgmap check in bio_add_page for compatible zone device pages
block/bio-integrity-auto.c | 2 +-
block/bio-integrity.c | 6 +-
block/bio.c | 11 +-
block/blk-core.c | 2 +-
block/blk-throttle.c | 2 +-
block/blk.h | 21 ++++
drivers/block/ublk_drv.c | 123 ++++++++++++++--------
drivers/block/zloop.c | 123 +++++++++++-----------
include/linux/t10-pi.h | 25 ++---
tools/testing/selftests/ublk/Makefile | 1 +
tools/testing/selftests/ublk/test_batch_01.sh | 4 +-
tools/testing/selftests/ublk/test_batch_02.sh | 2 +-
tools/testing/selftests/ublk/test_batch_03.sh | 2 +-
tools/testing/selftests/ublk/test_generic_02.sh | 4 +-
tools/testing/selftests/ublk/test_generic_03.sh | 2 +-
tools/testing/selftests/ublk/test_generic_06.sh | 2 +-
tools/testing/selftests/ublk/test_generic_07.sh | 2 +-
tools/testing/selftests/ublk/test_generic_08.sh | 4 +-
tools/testing/selftests/ublk/test_generic_09.sh | 2 +-
tools/testing/selftests/ublk/test_generic_10.sh | 2 +-
tools/testing/selftests/ublk/test_generic_12.sh | 4 +-
tools/testing/selftests/ublk/test_generic_13.sh | 2 +-
tools/testing/selftests/ublk/test_generic_16.sh | 4 +-
tools/testing/selftests/ublk/test_generic_17.sh | 2 +-
tools/testing/selftests/ublk/test_integrity_02.sh | 5 +-
tools/testing/selftests/ublk/test_integrity_03.sh | 103 ++++++++++++++++++
tools/testing/selftests/ublk/test_loop_01.sh | 2 +-
tools/testing/selftests/ublk/test_loop_02.sh | 2 +-
tools/testing/selftests/ublk/test_loop_03.sh | 2 +-
tools/testing/selftests/ublk/test_loop_04.sh | 2 +-
tools/testing/selftests/ublk/test_loop_05.sh | 2 +-
tools/testing/selftests/ublk/test_loop_06.sh | 2 +-
tools/testing/selftests/ublk/test_loop_07.sh | 2 +-
tools/testing/selftests/ublk/test_null_01.sh | 2 +-
tools/testing/selftests/ublk/test_null_02.sh | 2 +-
tools/testing/selftests/ublk/test_null_03.sh | 2 +-
tools/testing/selftests/ublk/test_part_01.sh | 4 +-
tools/testing/selftests/ublk/test_part_02.sh | 2 +-
tools/testing/selftests/ublk/test_recover_01.sh | 2 +-
tools/testing/selftests/ublk/test_recover_02.sh | 2 +-
tools/testing/selftests/ublk/test_recover_03.sh | 2 +-
tools/testing/selftests/ublk/test_recover_04.sh | 2 +-
tools/testing/selftests/ublk/test_shmemzc_01.sh | 2 +-
tools/testing/selftests/ublk/test_shmemzc_02.sh | 2 +-
tools/testing/selftests/ublk/test_shmemzc_03.sh | 2 +-
tools/testing/selftests/ublk/test_shmemzc_04.sh | 2 +-
tools/testing/selftests/ublk/test_stress_01.sh | 2 +-
tools/testing/selftests/ublk/test_stress_02.sh | 2 +-
tools/testing/selftests/ublk/test_stress_03.sh | 2 +-
tools/testing/selftests/ublk/test_stress_04.sh | 2 +-
tools/testing/selftests/ublk/test_stress_05.sh | 2 +-
tools/testing/selftests/ublk/test_stress_06.sh | 2 +-
tools/testing/selftests/ublk/test_stress_07.sh | 2 +-
tools/testing/selftests/ublk/test_stress_08.sh | 2 +-
tools/testing/selftests/ublk/test_stress_09.sh | 2 +-
tools/testing/selftests/ublk/test_stripe_01.sh | 2 +-
tools/testing/selftests/ublk/test_stripe_02.sh | 2 +-
tools/testing/selftests/ublk/test_stripe_03.sh | 2 +-
tools/testing/selftests/ublk/test_stripe_04.sh | 2 +-
tools/testing/selftests/ublk/test_stripe_05.sh | 2 +-
tools/testing/selftests/ublk/test_stripe_06.sh | 2 +-
61 files changed, 347 insertions(+), 187 deletions(-)
create mode 100755 tools/testing/selftests/ublk/test_integrity_03.sh
--
Jens Axboe
^ permalink raw reply
* [PATCH] blk-mq: recover from stale cached request in blk_mq_submit_bio
From: Keith Busch @ 2026-04-24 18:45 UTC (permalink / raw)
To: axboe, hch, linux-block; +Cc: Keith Busch
From: Keith Busch <kbusch@kernel.org>
When submitting a bio to blk-mq, if the task should sleep after peeking
a cached request, but before it pops it, the plug is flushed and calls
blk_mq_free_plug_rqs, freeing the cached_rqs.
The code had already warned of this possibility, and specifically popped
the request before other known blocking calls, but it didn't handle a
blocking GFP_NOIO alloc. Allocating the split bio or the integrity
payload are two such cases that can block under high memory pressure.
The blk-mq submit_bio function continues with the peeked request that
was just freed and re-initialized, so the driver receives that request
with a NULL'ed mq_hctx, and inevitably panics.
Relevant kernel messages if you should encounter this condition, where
the "WARNING" is the harbinger of the panic about to happen:
------------[ cut here ]------------
WARNING: CPU: 4 PID: 80820 at block/blk-mq.c:3071 blk_mq_submit_bio+0x2cf/0x5b0
...
BUG: kernel NULL pointer dereference, address: 0000000000000100
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 6b367b067 P4D 6b367b067 PUD 6bb5eb067 PMD 0
Oops: Oops: 0000 [#1] SMP
CPU: 4 UID: 36666 PID: 80820 Comm: IOUringThread0 Kdump: loaded Tainted: G S W 6.16.1-0_fbk3_0_gd6c130b80483 #1 NONE
Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN
Hardware name: Quanta Twin Lakes MP/Twin Lakes Passive MP, BIOS F09_3A23 12/08/2020
RIP: 0010:nvme_queue_rqs+0x93/0x180
Code: 44 24 48 00 00 00 00 4d 85 f6 74 19 49 8b 44 24 10 4c 3b b0 00 01 00 00 74 0b 4c 89 f7 4c 89 fe e8 c2 62 cd ff 49 8b 44 24 10 <4c> 8b b0 00 01 00 00 49 f7 46 78 01 00 00 00 74 1a 49 8b 3e 8b 8f
RSP: 0018:ffffc9004a8f34a8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff888c12b92920 RCX: 0000000000000020
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888c12b92920
RBP: 0000000000000000 R08: 000000000000000b R09: ffffffffffffffff
R10: 0000000000092800 R11: 000000000002e000 R12: ffff88828e325c00
R13: 0000000000000000 R14: 0000000000000000 R15: ffffc9004a8f34a8
FS: 00007f7cf30e6640(0000) GS:ffff8890fa908000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000100 CR3: 00000006bb387006 CR4: 00000000007726f0
PKRU: 55555554
Call Trace:
<TASK>
blk_mq_dispatch_queue_requests+0x46/0x120
blk_mq_flush_plug_list+0x38/0x130
blk_add_rq_to_plug+0xa2/0x160
blk_mq_submit_bio+0x3ab/0x5b0
__submit_bio+0x3a/0x260
submit_bio_noacct_nocheck+0xc6/0x2b0
btrfs_submit_bbio+0x14d/0x520
? btrfs_get_extent+0x43f/0x640
submit_extent_folio+0x31f/0x340
btrfs_do_readpage+0x2d7/0xac0
btrfs_readahead+0x142/0x200
? clear_state_bit+0x520/0x520
read_pages+0x57/0x200
? folio_alloc_noprof+0x10c/0x310
page_cache_ra_unbounded+0x28c/0x480
? asm_sysvec_call_function+0x16/0x20
? blk_cgroup_congested+0xa/0x50
? page_cache_sync_ra+0x41/0x2d0
filemap_get_pages+0x347/0xd50
filemap_read+0xd3/0x500
? 0xffffffff81000000
__io_read+0x111/0x440
io_read+0x23/0x90
__io_issue_sqe+0x40/0x120
io_issue_sqe+0x3f/0x3a0
io_submit_sqes+0x2bd/0x790
__se_sys_io_uring_enter+0x100/0xc10
? eventfd_read+0x100/0x1f0
? futex_wake+0x1b9/0x260
? syscall_trace_enter+0x34/0x1d0
do_syscall_64+0x6a/0x250
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x95d621e
Code: 8b 12 f6 c2 01 75 2a 84 c9 74 22 44 0f b6 d1 41 09 c2 8b bf cc 00 00 00 41 b9 08 00 00 00 b8 aa 01 00 00 31 d2 45 31 c0 0f 05 <48> 89 c6 89 f0 5d c3 83 c8 02 eb d5 cc cc cc cc cc cc 55 48 89 e5
RSP: 002b:00007f7cf30dc630 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
RAX: ffffffffffffffda RBX: 00007f7d18907000 RCX: 00000000095d621e
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 00000000000004cf
RBP: 00007f7cf30dc630 R08: 0000000000000000 R09: 0000000000000008
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
R13: 0000000000000001 R14: 00007f7d18907320 R15: 0000000000000009
</TASK>
Fixes: b0077e269f6c1 ("blk-mq: make sure active queue usage is held for bio_integrity_prep()")
Fixes: 7b4f36cd22a65 ("block: ensure we hold a queue reference when using queue limits")
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
block/blk-mq.c | 24 +++++++++++++++---------
1 file changed, 15 insertions(+), 9 deletions(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 4c5c16cce4f8f..ea44bd73a94b7 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -3096,22 +3096,23 @@ static struct request *blk_mq_peek_cached_request(struct blk_plug *plug,
return rq;
}
-static void blk_mq_use_cached_rq(struct request *rq, struct blk_plug *plug,
+static bool blk_mq_use_cached_rq(struct request *rq, struct blk_plug *plug,
struct bio *bio)
{
- if (rq_list_pop(&plug->cached_rqs) != rq)
- WARN_ON_ONCE(1);
-
/*
- * If any qos ->throttle() end up blocking, we will have flushed the
- * plug and hence killed the cached_rq list as well. Pop this entry
- * before we throttle.
+ * We will have flushed the plug and hence killed the cached_rq list as
+ * well if anything had blocked. Pop this entry before we throttle if
+ * the entry is still valid.
*/
+ if (rq_list_pop(&plug->cached_rqs) != rq)
+ return false;
+
rq_qos_throttle(rq->q, bio);
blk_mq_rq_time_init(rq, blk_time_get_ns());
rq->cmd_flags = bio->bi_opf;
INIT_LIST_HEAD(&rq->queuelist);
+ return true;
}
static bool bio_unaligned(const struct bio *bio, struct request_queue *q)
@@ -3211,8 +3212,13 @@ void blk_mq_submit_bio(struct bio *bio)
new_request:
if (rq) {
- blk_mq_use_cached_rq(rq, plug, bio);
- } else {
+ if (!blk_mq_use_cached_rq(rq, plug, bio)) {
+ rq = NULL;
+ if (unlikely(bio_queue_enter(bio)))
+ return;
+ }
+ }
+ if (!rq) {
rq = blk_mq_get_new_requests(q, plug, bio);
if (unlikely(!rq)) {
if (bio->bi_opf & REQ_NOWAIT)
--
2.52.0
^ permalink raw reply related
* Re: [PATCH] block: rbd: switch to dynamic root device
From: Viacheslav Dubeyko @ 2026-04-24 18:48 UTC (permalink / raw)
To: idryomov@gmail.com, johan@kernel.org
Cc: dongsheng.yang@linux.dev, ceph-devel@vger.kernel.org,
axboe@kernel.dk, linux-kernel@vger.kernel.org,
linux-block@vger.kernel.org
In-Reply-To: <20260424103931.2616903-1-johan@kernel.org>
On Fri, 2026-04-24 at 12:39 +0200, Johan Hovold wrote:
> Driver core expects devices to be dynamically allocated and will, for
> example, complain loudly when no release function has been provided.
>
> Use root_device_register() to allocate and register the root device
> instead of open coding using a static device.
I think the approach makes sense. Ilya, what do you think? Do we have any
objection?
>
> Signed-off-by: Johan Hovold <johan@kernel.org>
> ---
> drivers/block/rbd.c | 23 +++++++----------------
> 1 file changed, 7 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index e7da06200c1e..ffab1b4e7c25 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -580,14 +580,7 @@ static const struct bus_type rbd_bus_type = {
> .bus_groups = rbd_bus_groups,
> };
>
> -static void rbd_root_dev_release(struct device *dev)
> -{
> -}
> -
> -static struct device rbd_root_dev = {
> - .init_name = "rbd",
> - .release = rbd_root_dev_release,
> -};
> +static struct device *rbd_root_dev;
>
> static __printf(2, 3)
> void rbd_warn(struct rbd_device *rbd_dev, const char *fmt, ...)
> @@ -5390,7 +5383,7 @@ static struct rbd_device *__rbd_dev_create(struct rbd_spec *spec)
>
> rbd_dev->dev.bus = &rbd_bus_type;
> rbd_dev->dev.type = &rbd_device_type;
> - rbd_dev->dev.parent = &rbd_root_dev;
> + rbd_dev->dev.parent = rbd_root_dev;
> device_initialize(&rbd_dev->dev);
>
> return rbd_dev;
> @@ -7331,15 +7324,13 @@ static int __init rbd_sysfs_init(void)
> {
> int ret;
>
> - ret = device_register(&rbd_root_dev);
> - if (ret < 0) {
> - put_device(&rbd_root_dev);
> - return ret;
> - }
> + rbd_root_dev = root_device_register("rbd");
> + if (IS_ERR(rbd_root_dev))
> + return PTR_ERR(rbd_root_dev);
>
> ret = bus_register(&rbd_bus_type);
> if (ret < 0)
> - device_unregister(&rbd_root_dev);
> + root_device_unregister(rbd_root_dev);
I think we need to assign NULL here:
rbd_root_dev = NULL;
>
> return ret;
> }
> @@ -7347,7 +7338,7 @@ static int __init rbd_sysfs_init(void)
> static void __exit rbd_sysfs_cleanup(void)
> {
> bus_unregister(&rbd_bus_type);
> - device_unregister(&rbd_root_dev);
> + root_device_unregister(rbd_root_dev);
The same issue here:
rbd_root_dev = NULL;
Thanks,
Slava.
> }
>
> static int __init rbd_slab_init(void)
^ permalink raw reply
* Re: [GIT PULL] Block fixes for 7.1-rc1
From: pr-tracker-bot @ 2026-04-24 22:20 UTC (permalink / raw)
To: Jens Axboe; +Cc: Linus Torvalds, linux-block@vger.kernel.org
In-Reply-To: <f707ffcc-fb6e-4da1-a91f-5ed37e10cc0d@kernel.dk>
The pull request you sent on Fri, 24 Apr 2026 09:24:49 -0600:
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git tags/block-7.1-20260424
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/f3e3dbcea15e20f7413afd8c791a496f0b80e80b
Thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html
^ permalink raw reply
* [PATCH 00/12] Block storage copy offloading
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche
Hi Jens,
This patch series implements a new approach for copy offloading. Compared to
Nitesh' approach, the differences are as follows:
- Two new limits have been introduced representing the maximum number of source
and destination ranges. Support for multiple source ranges for the NVMe Copy
command has been added.
- The blkdev_copy_offload() function can now submit multiple copy offload
commands instead of only one.
- The implementation no longer depends on block layer plugging.
This patch series includes copy offloading support for the Linux kernel block
layer core, the device mapper core, the null_blk and the NVMe and nvmet drivers.
Support for the scsi_debug and SCSI core will follow later.
Test scripts are available here:
https://github.com/bvanassche/blktests/tree/copy-offloading
See also Bart Van Assche, [LSF/MM/BPF TOPIC] Block storage copy offloading,
January 2026
(https://lore.kernel.org/all/0cfe6fe2-3865-4dc2-92a7-74b1240f7b63@acm.org/).
See also Nitesh Shetty, Implement copy offload support, May 2024
(https://lore.kernel.org/linux-block/20240520102033.9361-1-nj.shetty@samsung.com/).
Anuj Gupta (1):
fs/read_write: Generalize generic_copy_file_checks()
Bart Van Assche (4):
block: Introduce blkdev_copy_offload()
block: Introduce accessor functions for copy offload bios
dm: Add support for copy offloading
dm-linear: Enable copy offloading
Nitesh Shetty (7):
block: Introduce queue limits for copy offloading
block: Add the REQ_OP_COPY_{SRC,DST} operations
block: Add an onloaded copy implementation
fs, block: Add copy_file_range() support for block devices
nvme: Add copy offloading support
nvmet: Support the Copy command
null_blk: Add support for REQ_OP_COPY_*
Documentation/ABI/stable/sysfs-block | 24 +
Documentation/block/null_blk.rst | 4 +
block/Makefile | 2 +-
block/bio.c | 1 +
block/blk-copy.c | 631 +++++++++++++++++++++++++++
block/blk-core.c | 38 ++
block/blk-merge.c | 13 +
block/blk-settings.c | 36 ++
block/blk-sysfs.c | 35 ++
block/blk.h | 5 +
block/fops.c | 54 +++
drivers/block/null_blk/main.c | 113 +++++
drivers/block/null_blk/null_blk.h | 1 +
drivers/md/dm-linear.c | 6 +
drivers/md/dm-table.c | 8 +
drivers/nvme/host/constants.c | 1 +
drivers/nvme/host/core.c | 106 +++++
drivers/nvme/host/trace.c | 21 +-
drivers/nvme/target/admin-cmd.c | 26 +-
drivers/nvme/target/io-cmd-bdev.c | 80 ++++
drivers/nvme/target/io-cmd-file.c | 59 ++-
drivers/nvme/target/trace.c | 19 +
fs/read_write.c | 8 +-
include/linux/blk-copy.h | 50 +++
include/linux/blk_types.h | 57 +++
include/linux/blkdev.h | 20 +-
include/linux/nvme.h | 47 +-
27 files changed, 1449 insertions(+), 16 deletions(-)
create mode 100644 block/blk-copy.c
create mode 100644 include/linux/blk-copy.h
^ permalink raw reply
* [PATCH 01/12] block: Introduce queue limits for copy offloading
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche, Kanchan Joshi, Anuj Gupta
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
From: Nitesh Shetty <nj.shetty@samsung.com>
Add the following request queue limits:
- max_copy_hw_sectors: the maximum number of sectors supported by the
block driver for a single offloaded copy operation.
- max_copy_src_segments: the maximum number of source segments
supported by the block driver for a single offloaded copy operation.
- max_copy_dst_segments: the maximum number of destination segments
supported by the block driver for a single offloaded copy operation.
- max_user_copy_sectors: the maximum number of sectors configured by the
user for a single offloaded copy operation.
- max_copy_sectors: the maximum number of sectors for a single
offloaded copy operation. This is the minimum of the above two
parameters.
The default value for all these new limits is zero which means that copy
offloading is not supported unless if these limits are set by the block
driver.
ake the following two limits available in sysfs:
- copy_max_bytes (RW)
- copy_max_hw_bytes (RO)
These limits will be used by the function that implements copy
offloading to decide the bio size.
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Kanchan Joshi <joshi.k@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
[ bvanassche: Added max_copy_{src,dst}_segments limits. Introduced
blk_validate_copy_limits(). Introduced BLK_FEAT_STACKING_COPY_OFFL.
Modified patch description. ]
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
Documentation/ABI/stable/sysfs-block | 24 +++++++++++++++++++
block/blk-settings.c | 36 ++++++++++++++++++++++++++++
block/blk-sysfs.c | 35 +++++++++++++++++++++++++++
include/linux/blkdev.h | 18 +++++++++++++-
4 files changed, 112 insertions(+), 1 deletion(-)
diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block
index 900b3fc4c72d..bec5e04085da 100644
--- a/Documentation/ABI/stable/sysfs-block
+++ b/Documentation/ABI/stable/sysfs-block
@@ -239,6 +239,30 @@ Description:
last zone of the device which may be smaller.
+What: /sys/block/<disk>/queue/copy_max_bytes
+Date: May 2026
+Contact: linux-block@vger.kernel.org
+Description:
+ [RW] This is the maximum number of bytes that the block layer
+ will allow for a copy request. This is always smaller or
+ equal to the maximum size allowed by the block driver.
+ Any value higher than 'copy_max_hw_bytes' will be reduced to
+ 'copy_max_hw_bytes'. Writing '0' to this attribute will disable
+ copy offloading for this block device. If copy offloading is
+ disabled, copy requests will be translated into read and write
+ requests.
+
+
+What: /sys/block/<disk>/queue/copy_max_hw_bytes
+Date: May 2026
+Contact: linux-block@vger.kernel.org
+Description:
+ [RO] This is the maximum number of bytes that is allowed for
+ a single data copy request. Set by the block driver. The value
+ zero indicates that the block device does not support copy
+ offloading.
+
+
What: /sys/block/<disk>/queue/crypto/
Date: February 2022
Contact: linux-block@vger.kernel.org
diff --git a/block/blk-settings.c b/block/blk-settings.c
index 78c83817b9d3..cb846ff2926e 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -57,6 +57,11 @@ void blk_set_stacking_limits(struct queue_limits *lim)
lim->max_hw_zone_append_sectors = UINT_MAX;
lim->max_user_discard_sectors = UINT_MAX;
lim->atomic_write_hw_max = UINT_MAX;
+
+ lim->max_user_copy_sectors = UINT_MAX;
+ lim->max_copy_hw_sectors = UINT_MAX;
+ lim->max_copy_src_segments = U16_MAX;
+ lim->max_copy_dst_segments = U16_MAX;
}
EXPORT_SYMBOL(blk_set_stacking_limits);
@@ -333,6 +338,21 @@ static void blk_validate_atomic_write_limits(struct queue_limits *lim)
lim->atomic_write_unit_max = 0;
}
+/*
+ * Check whether max_copy_hw_sectors and max_copy_{src,dst}_segments are
+ * either all nonzero or all zero.
+ */
+static int blk_validate_copy_limits(const struct queue_limits *lim)
+{
+ if (lim->max_copy_hw_sectors && lim->max_copy_src_segments &&
+ lim->max_copy_dst_segments)
+ return 0;
+ if (!lim->max_copy_hw_sectors && !lim->max_copy_src_segments &&
+ !lim->max_copy_dst_segments)
+ return 0;
+ return -EINVAL;
+}
+
/*
* Check that the limits in lim are valid, initialize defaults for unset
* values, and cap values based on others where needed.
@@ -510,6 +530,13 @@ int blk_validate_limits(struct queue_limits *lim)
err = blk_validate_integrity_limits(lim);
if (err)
return err;
+
+ err = blk_validate_copy_limits(lim);
+ if (err)
+ return err;
+ lim->max_copy_sectors =
+ min(lim->max_copy_hw_sectors, lim->max_user_copy_sectors);
+
return blk_validate_zoned_limits(lim);
}
EXPORT_SYMBOL_GPL(blk_validate_limits);
@@ -528,6 +555,7 @@ int blk_set_default_limits(struct queue_limits *lim)
*/
lim->max_user_discard_sectors = UINT_MAX;
lim->max_user_wzeroes_unmap_sectors = UINT_MAX;
+ lim->max_user_copy_sectors = UINT_MAX;
return blk_validate_limits(lim);
}
@@ -829,6 +857,14 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
t->max_segment_size = min_not_zero(t->max_segment_size,
b->max_segment_size);
+ t->max_copy_hw_sectors =
+ min(t->max_copy_hw_sectors, b->max_copy_hw_sectors);
+ t->max_copy_src_segments =
+ min(t->max_copy_src_segments, b->max_copy_src_segments);
+ t->max_copy_dst_segments =
+ min(t->max_copy_dst_segments, b->max_copy_dst_segments);
+ t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors);
+
alignment = queue_limit_alignment_offset(b, start);
/* Bottom device has different alignment. Check that it is
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index f22c1f253eb3..8e1e14d1682d 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -325,6 +325,36 @@ queue_max_sectors_store(struct gendisk *disk, const char *page, size_t count,
return 0;
}
+static ssize_t queue_copy_hw_max_show(struct gendisk *disk, char *page)
+{
+ return queue_var_show(
+ disk->queue->limits.max_copy_hw_sectors << SECTOR_SHIFT, page);
+}
+
+static ssize_t queue_copy_max_show(struct gendisk *disk, char *page)
+{
+ return queue_var_show(
+ disk->queue->limits.max_copy_sectors << SECTOR_SHIFT, page);
+}
+
+static int queue_copy_max_store(struct gendisk *disk, const char *page,
+ size_t count, struct queue_limits *lim)
+{
+ unsigned long max_copy_bytes;
+ ssize_t ret;
+
+ ret = queue_var_store(&max_copy_bytes, page, count);
+ if (ret < 0)
+ return ret;
+
+ if ((max_copy_bytes >> SECTOR_SHIFT) > UINT_MAX)
+ return -EINVAL;
+
+ lim->max_user_copy_sectors = max_copy_bytes >> SECTOR_SHIFT;
+
+ return 0;
+}
+
static ssize_t queue_feature_store(struct gendisk *disk, const char *page,
size_t count, struct queue_limits *lim, blk_features_t feature)
{
@@ -652,6 +682,9 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones");
QUEUE_LIM_RO_ENTRY(queue_max_open_zones, "max_open_zones");
QUEUE_LIM_RO_ENTRY(queue_max_active_zones, "max_active_zones");
+QUEUE_LIM_RO_ENTRY(queue_copy_hw_max, "copy_max_hw_bytes");
+QUEUE_LIM_RW_ENTRY(queue_copy_max, "copy_max_bytes");
+
QUEUE_RW_ENTRY(queue_nomerges, "nomerges");
QUEUE_LIM_RW_ENTRY(queue_iostats_passthrough, "iostats_passthrough");
QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity");
@@ -760,6 +793,8 @@ static const struct attribute *const queue_attrs[] = {
&queue_max_hw_wzeroes_unmap_sectors_entry.attr,
&queue_max_wzeroes_unmap_sectors_entry.attr,
&queue_max_zone_append_sectors_entry.attr,
+ &queue_copy_hw_max_entry.attr,
+ &queue_copy_max_entry.attr,
&queue_zone_write_granularity_entry.attr,
&queue_rotational_entry.attr,
&queue_zoned_entry.attr,
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 890128cdea1c..8ae64cc0546f 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -353,13 +353,17 @@ typedef unsigned int __bitwise blk_features_t;
#define BLK_FEAT_RAID_PARTIAL_STRIPES_EXPENSIVE \
((__force blk_features_t)(1u << 15))
+/* block driver is a stacking block driver that supports copy offloading */
+#define BLK_FEAT_STACKING_COPY_OFFL ((__force blk_features_t)(1u << 16))
+
/*
* Flags automatically inherited when stacking limits.
*/
#define BLK_FEAT_INHERIT_MASK \
(BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA | BLK_FEAT_ROTATIONAL | \
BLK_FEAT_STABLE_WRITES | BLK_FEAT_ZONED | \
- BLK_FEAT_RAID_PARTIAL_STRIPES_EXPENSIVE)
+ BLK_FEAT_RAID_PARTIAL_STRIPES_EXPENSIVE | \
+ BLK_FEAT_STACKING_COPY_OFFL)
/* internal flags in queue_limits.flags */
typedef unsigned int __bitwise blk_flags_t;
@@ -415,6 +419,13 @@ struct queue_limits {
unsigned int atomic_write_hw_unit_max;
unsigned int atomic_write_unit_max;
+ /* copy offloading limits */
+ unsigned int max_copy_hw_sectors; /* set by block driver*/
+ uint16_t max_copy_src_segments; /* set by block driver*/
+ uint16_t max_copy_dst_segments; /* set by block driver*/
+ unsigned int max_user_copy_sectors; /* set via sysfs */
+ unsigned int max_copy_sectors; /* min() of the above */
+
unsigned short max_segments;
unsigned short max_integrity_segments;
unsigned short max_discard_segments;
@@ -1454,6 +1465,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev)
return bdev_limits(bdev)->discard_granularity;
}
+static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev)
+{
+ return bdev_get_queue(bdev)->limits.max_copy_sectors;
+}
+
static inline unsigned int
bdev_max_secure_erase_sectors(struct block_device *bdev)
{
^ permalink raw reply related
* [PATCH 02/12] block: Add the REQ_OP_COPY_{SRC,DST} operations
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche, Anuj Gupta
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
From: Nitesh Shetty <nj.shetty@samsung.com>
Introduce the REQ_OP_COPY_SRC and REQ_OP_COPY_DST operations. The source
and destination LBA range information is in separate bios because any
other approach would require a rewrite of the device mapper. These bios
are associated with each other via the new bi_copy_ctx pointer. A new
pointer has been introduced because the copy offloading context
information must be preserved when cloning a bio and the bi_private bio
member must not be copied when cloning a bio.
This patch supports the following approach for copy offloading:
1. Allocate a struct bio_copy_offload_ctx instance and set phase to
BLKDEV_TRANSLATE_LBAS.
2. Allocate REQ_OP_COPY_SRC and REQ_OP_COPY_DST bios. Set the
bi_copy_ctx member of these bios.
3. Set the bio_count member of struct bio_copy_offload_ctx.
4. Submit all REQ_OP_COPY_* bios.
5. In submit_bio(), do the following for REQ_OP_COPY_* bios:
- If bio->bi_bdev is a stacking device, submit the bio. This will
send the bio to the device mapper. The device mapper will clone the
bio, translate the LBAs and will submit the cloned bio. That will
result in a recursive submit_bio() call.
- If bio->bi_bdev is not a stacking device, add the bio to the
copy_ctx->bios list and decrement copy_ctx->bio_count.
6. Once copy_ctx->bio_count == 0, call copy_ctx->translation_complete().
7. In the implementation of copy_ctx->translation_complete(), change
copy_ctx->phase from BLKDEV_TRANSLATE_LBAS into BLKDEV_COPY.
8. Submit the first REQ_OP_COPY_* bio of the copy_ctx->bios list.
9. Once this bio reaches the block driver associated with the bio,
retrieve the other bios involved in the copy operation from the copy
context data structure and convert all these bios into a copy offload
operation.
10. Once this bio completes, also complete all the other bios involved
in the copy offload operation.
This patch increases the size of struct bio from 104 to 112 bytes on 64-bit
systems.
To be discussed further: whether adding a new member in struct bio is
acceptable or whether the new pointer perhaps should be stored in front of
the bio. bioset_init() supports front padding.
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
[ bvanassche: changed the approach of this patch from combining the
COPY_SRC and COPY_DST operations immediately to translating the LBA
information first. ]
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
block/bio.c | 1 +
block/blk-core.c | 38 ++++++++++++++++++++++++++++++++
block/blk-merge.c | 13 +++++++++++
block/blk.h | 5 +++++
include/linux/blk-copy.h | 46 +++++++++++++++++++++++++++++++++++++++
include/linux/blk_types.h | 17 +++++++++++++++
6 files changed, 120 insertions(+)
create mode 100644 include/linux/blk-copy.h
diff --git a/block/bio.c b/block/bio.c
index b8972dba68a0..51480c9be27b 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -852,6 +852,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
bio->bi_write_hint = bio_src->bi_write_hint;
bio->bi_write_stream = bio_src->bi_write_stream;
bio->bi_iter = bio_src->bi_iter;
+ bio->bi_copy_ctx = bio_src->bi_copy_ctx;
if (bio->bi_bdev) {
if (bio->bi_bdev == bio_src->bi_bdev &&
diff --git a/block/blk-core.c b/block/blk-core.c
index 17450058ea6d..37c01e717202 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -16,6 +16,7 @@
#include <linux/module.h>
#include <linux/bio.h>
#include <linux/blkdev.h>
+#include <linux/blk-copy.h>
#include <linux/blk-pm.h>
#include <linux/blk-integrity.h>
#include <linux/highmem.h>
@@ -108,6 +109,8 @@ static const char *const blk_op_name[] = {
REQ_OP_NAME(ZONE_FINISH),
REQ_OP_NAME(ZONE_APPEND),
REQ_OP_NAME(WRITE_ZEROES),
+ REQ_OP_NAME(COPY_SRC),
+ REQ_OP_NAME(COPY_DST),
REQ_OP_NAME(DRV_IN),
REQ_OP_NAME(DRV_OUT),
};
@@ -782,6 +785,8 @@ void submit_bio_noacct(struct bio *bio)
struct block_device *bdev = bio->bi_bdev;
struct request_queue *q = bdev_get_queue(bdev);
blk_status_t status = BLK_STS_IOERR;
+ struct bio_copy_offload_ctx *copy_ctx;
+ u32 bio_count;
might_sleep();
@@ -875,6 +880,39 @@ void submit_bio_noacct(struct bio *bio)
* requests.
*/
fallthrough;
+ case REQ_OP_COPY_SRC:
+ case REQ_OP_COPY_DST:
+ copy_ctx = bio->bi_copy_ctx;
+ WARN_ON_ONCE(copy_ctx->phase == BLKDEV_COPY_DONE);
+ if (copy_ctx->phase == BLKDEV_COPY)
+ break;
+ /* If copy offloading is not supported, fail the bio. */
+ if (!q->limits.max_copy_sectors) {
+ scoped_guard(spinlock_irqsave, ©_ctx->lock)
+ copy_ctx->bio_count--;
+ goto not_supported;
+ }
+ /*
+ * If the block driver is a stacking driver that supports copy
+ * offloading, submit the bio.
+ */
+ if (q->limits.features & BLK_FEAT_STACKING_COPY_OFFL)
+ break;
+ /*
+ * Append the bio at the end of the bio->bi_copy_ctx->bios list.
+ */
+ scoped_guard(spinlock_irqsave, ©_ctx->lock) {
+ if (copy_ctx->biotail)
+ copy_ctx->biotail->bi_next = bio;
+ else
+ copy_ctx->bios = bio;
+ copy_ctx->biotail = bio;
+ bio_count = --copy_ctx->bio_count;
+ }
+ WARN_ON_ONCE(bio_count < 0);
+ if (bio_count == 0)
+ copy_ctx->translation_complete(copy_ctx);
+ return;
default:
goto not_supported;
}
diff --git a/block/blk-merge.c b/block/blk-merge.c
index fcf09325b22e..4678131650d2 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -207,6 +207,19 @@ struct bio *bio_split_discard(struct bio *bio, const struct queue_limits *lim,
return __bio_split_discard(bio, lim, nsegs, max_sectors);
}
+struct bio *bio_split_copy(struct bio *bio, const struct queue_limits *lim,
+ unsigned int *nsegs)
+{
+ *nsegs = 1;
+ if (bio_sectors(bio) <= lim->max_copy_sectors)
+ return bio;
+
+ /* Splitting a REQ_OP_COPY_* bio is not supported. */
+ bio->bi_status = BLK_STS_NOTSUPP;
+ bio_endio(bio);
+ return NULL;
+}
+
static inline unsigned int blk_boundary_sectors(const struct queue_limits *lim,
bool is_atomic)
{
diff --git a/block/blk.h b/block/blk.h
index b998a7761faf..274c226e87ee 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -379,6 +379,8 @@ struct bio *bio_split_rw(struct bio *bio, const struct queue_limits *lim,
unsigned *nr_segs);
struct bio *bio_split_zone_append(struct bio *bio,
const struct queue_limits *lim, unsigned *nr_segs);
+struct bio *bio_split_copy(struct bio *bio, const struct queue_limits *lim,
+ unsigned int *nsegs);
/*
* All drivers must accept single-segments bios that are smaller than PAGE_SIZE.
@@ -435,6 +437,9 @@ static inline struct bio *__bio_split_to_limits(struct bio *bio,
return bio_split_discard(bio, lim, nr_segs);
case REQ_OP_WRITE_ZEROES:
return bio_split_write_zeroes(bio, lim, nr_segs);
+ case REQ_OP_COPY_SRC:
+ case REQ_OP_COPY_DST:
+ return bio_split_copy(bio, lim, nr_segs);
default:
/* other operations can't be split */
*nr_segs = 0;
diff --git a/include/linux/blk-copy.h b/include/linux/blk-copy.h
new file mode 100644
index 000000000000..5e38cfc14a71
--- /dev/null
+++ b/include/linux/blk-copy.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __LINUX_BLK_COPY_H
+#define __LINUX_BLK_COPY_H
+
+#include <linux/blk_types.h>
+#include <linux/completion.h>
+#include <linux/list.h>
+#include <linux/spinlock_types.h>
+#include <linux/workqueue_types.h>
+
+struct blk_copy_params;
+struct request;
+
+enum blkdev_copy_phase {
+ BLKDEV_TRANSLATE_LBAS,
+ BLKDEV_COPY,
+ BLKDEV_COPY_DONE,
+};
+
+/*
+ * struct bio_copy_offload_ctx - context information for blkdev_copy_offload()
+ * @params: Input parameters passed to blkdev_copy_offload().
+ * @len: Number of bytes associated with this copy context.
+ * @phase: Copy offload phase: either translating LBAs or copying data.
+ * @lock: Protects @bios, @biotail and @bio_count.
+ * @bios: List with REQ_OP_COPY_* bios for which LBA translation completed.
+ * @biotail: Last element in the @bios list.
+ * @bio_count: Number bios for which LBA translation has not yet completed.
+ * @status: bio completion status.
+ * @translation_complete: Called after LBA translation has completed.
+ * LBA translation has completed once bio_count drops to zero.
+ */
+struct bio_copy_offload_ctx {
+ struct blk_copy_params *params;
+ loff_t len;
+ enum blkdev_copy_phase phase;
+ spinlock_t lock;
+ struct bio *bios __guarded_by(&lock);
+ struct bio *biotail __guarded_by(&lock);
+ u32 bio_count __guarded_by(&lock);
+ blk_status_t status __guarded_by(&lock);
+ void (*translation_complete)(struct bio_copy_offload_ctx *ctx);
+};
+
+#endif /* __LINUX_BLK_COPY_H */
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 8808ee76e73c..4e448e810b87 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -284,6 +284,8 @@ struct bio {
atomic_t __bi_cnt; /* pin count */
struct bio_set *bi_pool;
+
+ void *bi_copy_ctx;
};
#define BIO_RESET_BYTES offsetof(struct bio, bi_max_vecs)
@@ -370,6 +372,10 @@ enum req_op {
/** @REQ_OP_ZONE_RESET_ALL: reset all the zone present on the device */
REQ_OP_ZONE_RESET_ALL = (__force blk_opf_t)19,
+ /* copy offload source and destination operations */
+ REQ_OP_COPY_SRC = (__force blk_opf_t)20,
+ REQ_OP_COPY_DST = (__force blk_opf_t)21,
+
/* Driver private requests */
/* private: */
REQ_OP_DRV_IN = (__force blk_opf_t)34,
@@ -461,6 +467,17 @@ static inline bool op_is_write(blk_opf_t op)
return !!(op & (__force blk_opf_t)1);
}
+static inline bool op_is_copy(blk_opf_t op)
+{
+ switch (op & REQ_OP_MASK) {
+ case REQ_OP_COPY_DST:
+ case REQ_OP_COPY_SRC:
+ return true;
+ default:
+ return false;
+ }
+}
+
/*
* Check if the bio or request is one that needs special treatment in the
* flush state machine.
^ permalink raw reply related
* [PATCH 03/12] block: Introduce blkdev_copy_offload()
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
Introduce blkdev_copy_offload() for performing copy offloading. This
function implements the algorithm explained the description of the
previous patch. If the input parameters exceed what can be supported
with a single copy offload operation, multiple copy offload operations
are submitted.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
block/Makefile | 2 +-
block/blk-copy.c | 355 ++++++++++++++++++++++++++++++++++++++
include/linux/blk_types.h | 40 +++++
include/linux/blkdev.h | 1 +
4 files changed, 397 insertions(+), 1 deletion(-)
create mode 100644 block/blk-copy.c
diff --git a/block/Makefile b/block/Makefile
index 7dce2e44276c..d99e8d4fda7d 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -6,7 +6,7 @@
obj-y := bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \
blk-flush.o blk-settings.o blk-ioc.o blk-map.o \
blk-merge.o blk-timeout.o blk-lib.o blk-mq.o \
- blk-mq-tag.o blk-mq-dma.o blk-stat.o \
+ blk-mq-tag.o blk-mq-dma.o blk-stat.o blk-copy.o \
blk-mq-sysfs.o blk-mq-cpumap.o blk-mq-sched.o ioctl.o \
genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \
disk-events.o blk-ia-ranges.o early-lookup.o
diff --git a/block/blk-copy.c b/block/blk-copy.c
new file mode 100644
index 000000000000..8ac8879442f7
--- /dev/null
+++ b/block/blk-copy.c
@@ -0,0 +1,355 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Offloaded and onloaded data copying support.
+ */
+#include <linux/bio.h>
+#include <linux/blkdev.h>
+#include <linux/blk-copy.h>
+#include <linux/blk-mq.h>
+
+/* End all bios in the @ctx->bios list with status @ctx->status. */
+static void blkdev_end_bios(struct bio_copy_offload_ctx *ctx)
+{
+ struct bio *bio, *next;
+
+ bio = ctx->bios;
+ ctx->bios = NULL;
+ for (; bio; bio = next) {
+ next = bio->bi_next;
+ bio->bi_status = ctx->status;
+ bio_endio(bio);
+ }
+}
+
+/*
+ * Called after LBA translation finished for all bios associated with copy context
+ * @ctx.
+ */
+static void blkdev_translation_complete(struct bio_copy_offload_ctx *ctx)
+{
+ struct module *owner = NULL;
+ struct bio *bio;
+
+ WARN_ON_ONCE(ctx->phase != BLKDEV_TRANSLATE_LBAS);
+ ctx->phase = BLKDEV_COPY;
+
+ /* Check whether all bios are associated with the same block driver. */
+ for (bio = ctx->bios; bio; bio = bio->bi_next) {
+ if (!owner) {
+ owner = bio->bi_bdev->bd_disk->fops->owner;
+ } else if (owner != bio->bi_bdev->bd_disk->fops->owner) {
+ ctx->status = BLK_STS_INVAL;
+ break;
+ }
+ }
+
+ /* Remove the first bio from the bio list and submit it. */
+ bio = ctx->bios;
+ ctx->bios = bio->bi_next;
+ bio->bi_next = NULL;
+ if (ctx->biotail == bio)
+ ctx->biotail = NULL;
+ if (ctx->status == BLK_STS_OK)
+ submit_bio(bio);
+ else
+ bio_endio(bio);
+}
+
+/* REQ_OP_COPY_* completion handler. */
+static void blkdev_req_op_copy_done(struct bio *bio)
+{
+ struct bio_copy_offload_ctx *ctx = bio->bi_copy_ctx;
+ struct blk_copy_params *params = ctx->params;
+ blk_status_t status;
+
+ switch (ctx->phase) {
+ case BLKDEV_TRANSLATE_LBAS:
+ scoped_guard(spinlock_irqsave, &ctx->lock)
+ if (!ctx->status)
+ ctx->status = bio->bi_status;
+ break;
+ case BLKDEV_COPY:
+ status = ctx->status;
+ ctx->phase = BLKDEV_COPY_DONE;
+ blkdev_end_bios(ctx);
+ kfree(ctx);
+ scoped_guard(spinlock_irqsave, ¶ms->lock) {
+ if (!params->status)
+ params->status = status;
+ }
+ if (atomic_dec_and_test(¶ms->copy_ctx_count))
+ params->end_io(params);
+ break;
+ case BLKDEV_COPY_DONE:
+ break;
+ }
+}
+
+/*
+ * Check that all LBA offsets are aligned with both the source and the destination
+ * logical block sizes. Compare input and output length. Store the number of bytes
+ * to be transferred in *@len.
+ */
+static int blkdev_copy_check_params(const struct blk_copy_params *params,
+ loff_t *len)
+{
+ const unsigned int mask =
+ max(bdev_logical_block_size(params->in_bdev),
+ bdev_logical_block_size(params->out_bdev)) - 1;
+ loff_t in_len = 0, out_len = 0;
+ unsigned int i;
+
+ for (i = 0; i < params->in_nseg; i++) {
+ if ((params->in_segs[i].pos | params->in_segs[i].len) & mask)
+ return -EINVAL;
+ in_len += params->in_segs[i].len;
+ }
+
+ for (i = 0; i < params->out_nseg; i++) {
+ if ((params->out_segs[i].pos | params->out_segs[i].len) & mask)
+ return -EINVAL;
+ out_len += params->out_segs[i].len;
+ }
+
+ if (in_len != out_len)
+ return -EINVAL;
+
+ *len = in_len;
+
+ return 0;
+}
+
+/*
+ * Calculate the number of bytes in the max_copy_src_segments input segments
+ * starting from input segment @in_idx.
+ */
+static loff_t blk_max_src_len(const struct blk_copy_params *params,
+ unsigned int in_idx)
+{
+ uint16_t max_src_segments =
+ params->in_bdev->bd_queue->limits.max_copy_src_segments;
+ unsigned int max_i = min(params->in_nseg, in_idx + max_src_segments);
+ loff_t len = 0;
+
+ for (uint32_t i = in_idx; i < max_i; i++)
+ len += params->in_segs[i].len;
+
+ return len;
+}
+
+/*
+ * Calculate the number of bytes in the max_copy_dst_segments output segments
+ * starting from output segment @out_idx.
+ */
+static loff_t blk_max_dst_len(const struct blk_copy_params *params,
+ unsigned int out_idx)
+{
+ uint16_t max_dst_segments =
+ params->out_bdev->bd_queue->limits.max_copy_dst_segments;
+ unsigned int max_i = min(params->out_nseg, out_idx + max_dst_segments);
+ loff_t len = 0;
+
+ for (uint32_t i = out_idx; i < max_i; i++)
+ len += params->out_segs[i].len;
+
+ return len;
+}
+
+struct blkdev_copy_sync_ctx {
+ struct completion compl;
+ blk_status_t status;
+};
+
+static void blkdev_end_copy_sync(const struct blk_copy_params *params)
+{
+ struct blkdev_copy_sync_ctx *ctx = params->private;
+
+ complete(&ctx->compl);
+}
+
+static int blkdev_copy_sync(struct blk_copy_params *params)
+{
+ struct blkdev_copy_sync_ctx ctx = {
+ .compl = COMPLETION_INITIALIZER_ONSTACK(ctx.compl),
+ };
+ int ret;
+
+ WARN_ON_ONCE(params->end_io || params->private);
+ params->end_io = blkdev_end_copy_sync;
+ params->private = &ctx;
+
+ ret = blkdev_copy_offload(params);
+ if (ret && ret != -EIOCBQUEUED)
+ return ret;
+
+ wait_for_completion(&ctx.compl);
+ return blk_status_to_errno(ctx.status);
+}
+
+/**
+ * blkdev_copy_chunk() - submit a single copy offload operation
+ * @params: Copy offload input parameters.
+ * @in_idx: Index of the input segment from where to start copying.
+ * @out_idx: Index of the output segment to where to start copying.
+ * @in_offset: Offset in bytes from the start of input segment @in_idx.
+ * @out_offset: Offset in bytes from the start of output segment @out_idx.
+ * @chunk: Maximum number of bytes to copy.
+ *
+ * Returns: the number of bytes covered by the submitted copy operation or a
+ * negative error number.
+ */
+static loff_t blkdev_copy_chunk(struct blk_copy_params *params, u32 *in_idx,
+ u32 *out_idx, loff_t *in_offset,
+ loff_t *out_offset, loff_t chunk)
+{
+ struct bio_copy_offload_ctx *ctx;
+ u32 bio_count;
+
+ ctx = kzalloc_obj(*ctx);
+ if (!ctx)
+ return -ENOMEM;
+
+ spin_lock_init(&ctx->lock);
+ ctx->params = params;
+ ctx->phase = BLKDEV_TRANSLATE_LBAS;
+ ctx->translation_complete = blkdev_translation_complete;
+ /*
+ * Initialized to one to prevent that ctx->translation_complete() is
+ * called before bio submission has finished.
+ */
+ ctx->bio_count = 1;
+
+ WARN_ON_ONCE(chunk <= 0);
+ chunk = min(chunk, blk_max_src_len(params, *in_idx) - *in_offset);
+ WARN_ON_ONCE(chunk <= 0);
+ chunk = min(chunk, blk_max_dst_len(params, *out_idx) - *out_offset);
+ WARN_ON_ONCE(chunk <= 0);
+ ctx->len = chunk;
+ for (loff_t bytes, remaining_in = chunk; remaining_in > 0;
+ remaining_in -= bytes) {
+ struct bio *src_bio;
+
+ src_bio = bio_alloc(params->in_bdev, 0, REQ_OP_COPY_SRC,
+ GFP_NOIO);
+ if (!src_bio) {
+ if (remaining_in == chunk)
+ goto free_ctx;
+ else
+ goto enomem;
+ }
+ atomic_inc(¶ms->copy_ctx_count);
+ scoped_guard(spinlock_irqsave, &ctx->lock)
+ ctx->bio_count++;
+ bytes = min(remaining_in, params->in_segs[*in_idx].len -
+ *in_offset);
+ src_bio->bi_iter.bi_size = bytes;
+ src_bio->bi_iter.bi_sector = (params->in_segs[*in_idx].pos +
+ *in_offset) >> SECTOR_SHIFT;
+ src_bio->bi_copy_ctx = ctx;
+ src_bio->bi_end_io = blkdev_req_op_copy_done;
+ *in_offset += bytes;
+ if (*in_offset >= params->in_segs[*in_idx].len) {
+ *in_offset -= params->in_segs[*in_idx].len;
+ (*in_idx)++;
+ }
+ submit_bio(src_bio);
+ }
+ for (loff_t bytes, remaining_out = chunk; remaining_out;
+ remaining_out -= bytes) {
+ struct bio *dst_bio;
+
+ dst_bio = bio_alloc(params->out_bdev, 0, REQ_OP_COPY_DST,
+ GFP_NOIO);
+ if (!dst_bio)
+ goto enomem;
+ scoped_guard(spinlock_irqsave, &ctx->lock)
+ ctx->bio_count++;
+ bytes = min(remaining_out, params->out_segs[*out_idx].len -
+ *out_offset);
+ dst_bio->bi_iter.bi_size = bytes;
+ dst_bio->bi_iter.bi_sector = (params->out_segs[*out_idx].pos +
+ *out_offset) >> SECTOR_SHIFT;
+ dst_bio->bi_copy_ctx = ctx;
+ dst_bio->bi_end_io = blkdev_req_op_copy_done;
+ *out_offset += bytes;
+ if (*out_offset >= params->out_segs[*out_idx].len) {
+ *out_offset -= params->out_segs[*out_idx].len;
+ (*out_idx)++;
+ }
+ submit_bio(dst_bio);
+ }
+
+dec_bio_count:
+ scoped_guard(spinlock_irqsave, &ctx->lock)
+ bio_count = --ctx->bio_count;
+ if (bio_count == 0)
+ ctx->translation_complete(ctx);
+ return chunk;
+
+enomem:
+ scoped_guard(spinlock_irqsave, &ctx->lock)
+ if (!ctx->status)
+ ctx->status = BLK_STS_RESOURCE;
+ chunk = -ENOMEM;
+ goto dec_bio_count;
+
+free_ctx:
+ kfree(ctx);
+ return -ENOMEM;
+}
+
+/**
+ * blkdev_copy_offload() - copy data and offload copying if possible.
+ * @params: Source and destination block device, data ranges and completion
+ * callback.
+ *
+ * If @params->end_io != NULL, data is copied asynchronously. If @params->end_io
+ * == NULL, this function only returns after data copying finished.
+ *
+ * Return: 0 upon success; -EIOCBQUEUED if the completion callback function will
+ * be called or has already been called; -EOPNOTSUPP if copy offloading is
+ * not supported by the block device or if the source or destination
+ * address ranges span more than one dm device.
+ */
+int blkdev_copy_offload(struct blk_copy_params *params)
+{
+ loff_t in_offset = 0, out_offset = 0;
+ u32 in_idx = 0, out_idx = 0;
+ loff_t len, chunk, max_chunk;
+ int ret;
+
+ might_sleep();
+
+ if (!params->end_io)
+ return blkdev_copy_sync(params);
+
+ spin_lock_init(¶ms->lock);
+
+ if (!bdev_max_copy_sectors(params->in_bdev) ||
+ !bdev_max_copy_sectors(params->out_bdev))
+ return -EOPNOTSUPP;
+
+ ret = blkdev_copy_check_params(params, &len);
+ if (ret)
+ return ret;
+
+ params->len = len;
+
+ max_chunk = (u64)min(bdev_max_copy_sectors(params->in_bdev),
+ bdev_max_copy_sectors(params->out_bdev))
+ << SECTOR_SHIFT;
+
+ atomic_set(¶ms->copy_ctx_count, 1);
+
+ for (loff_t offset = 0; offset < len; offset += chunk) {
+ chunk = min(len - offset, max_chunk);
+ chunk = blkdev_copy_chunk(params, &in_idx, &out_idx, &in_offset,
+ &out_offset, chunk);
+ }
+
+ if (atomic_dec_and_test(¶ms->copy_ctx_count))
+ params->end_io(params);
+
+ return -EIOCBQUEUED;
+}
+EXPORT_SYMBOL_GPL(blkdev_copy_offload);
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 4e448e810b87..27a0f92fc2cb 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -535,4 +535,44 @@ struct blk_rq_stat {
u64 batch;
};
+/* A single input or output segment descriptor. */
+struct blk_copy_seg {
+ loff_t pos;
+ loff_t len;
+};
+
+/**
+ * struct blk_copy_params - input parameters and internal parameters for copy
+ * operations.
+ * @in_bdev: Input block device.
+ * @in_segs: Input LBA ranges.
+ * @in_nseg: Number of elements in @in_segs.
+ * @out_bdev: Output block device.
+ * @out_segs: Output LBA ranges.
+ * @out_nseg: Number of elements in @out_segs.
+ * @end_io: Called after copying data finished. If %NULL, copying data happens
+ * synchronously instead of asynchronously.
+ * @private: May be used by @end_io. Not used directly.
+ * @len: Total number of bytes to copy. Set by blkdev_copy_offload() or
+ * blkdev_copy_onload().
+ * @copy_ctxs: Number of in-flight copy contexts associated with copy offload
+ * operations.
+ * @lock: Protects @status updates.
+ * @status: I/O completion status.
+ */
+struct blk_copy_params {
+ struct block_device *in_bdev;
+ struct blk_copy_seg *in_segs;
+ unsigned int in_nseg;
+ struct block_device *out_bdev;
+ struct blk_copy_seg *out_segs;
+ unsigned int out_nseg;
+ void (*end_io)(const struct blk_copy_params *params);
+ void *private;
+ loff_t len;
+ atomic_t copy_ctx_count;
+ spinlock_t lock;
+ blk_status_t status;
+};
+
#endif /* __LINUX_BLK_TYPES_H */
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 8ae64cc0546f..fea296150cda 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1283,6 +1283,7 @@ void __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp_mask, struct bio **biop);
int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp);
+int blkdev_copy_offload(struct blk_copy_params *params);
#define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */
#define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */
^ permalink raw reply related
* [PATCH 05/12] block: Introduce accessor functions for copy offload bios
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
Make it easy for block drivers to iterate over the copy offload bios by
providing accessor functions for the copy offloading bios.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
block/blk-copy.c | 47 ++++++++++++++++++++++++++++++++++++++++
include/linux/blk-copy.h | 4 ++++
2 files changed, 51 insertions(+)
diff --git a/block/blk-copy.c b/block/blk-copy.c
index 459ed8581efc..f49a5f835b4a 100644
--- a/block/blk-copy.c
+++ b/block/blk-copy.c
@@ -7,6 +7,53 @@
#include <linux/blk-copy.h>
#include <linux/blk-mq.h>
+static struct bio *__blk_next_copy_bio(struct request *rq, struct bio *prev_bio,
+ enum req_op op)
+{
+ struct bio *bio;
+
+ if (prev_bio) {
+ bio = prev_bio->bi_next;
+ } else {
+ struct bio_copy_offload_ctx *copy_ctx = rq->bio->bi_copy_ctx;
+
+ bio = copy_ctx->bios;
+ }
+
+ for (; bio && bio_op(bio) != op; bio = bio->bi_next)
+ ;
+ return bio;
+}
+
+struct bio *blk_first_copy_bio(struct request *rq, enum req_op op)
+{
+ struct bio *bio = rq->bio;
+
+ if (bio_op(bio) == op)
+ return bio;
+
+ return __blk_next_copy_bio(rq, NULL, op);
+}
+EXPORT_SYMBOL_GPL(blk_first_copy_bio);
+
+struct bio *blk_next_copy_bio(struct bio *bio)
+{
+ return __blk_next_copy_bio(NULL, bio, bio_op(bio));
+}
+EXPORT_SYMBOL_GPL(blk_next_copy_bio);
+
+unsigned int blk_copy_bio_count(struct request *rq, enum req_op op)
+{
+ unsigned int count = 0;
+
+ for (struct bio *bio = blk_first_copy_bio(rq, op); bio;
+ bio = blk_next_copy_bio(bio))
+ count++;
+
+ return count;
+}
+EXPORT_SYMBOL_GPL(blk_copy_bio_count);
+
/**
* Tracks the state of a single onloaded copy operation.
* @params: Data copy parameters.
diff --git a/include/linux/blk-copy.h b/include/linux/blk-copy.h
index 5e38cfc14a71..4c8435312752 100644
--- a/include/linux/blk-copy.h
+++ b/include/linux/blk-copy.h
@@ -43,4 +43,8 @@ struct bio_copy_offload_ctx {
void (*translation_complete)(struct bio_copy_offload_ctx *ctx);
};
+struct bio *blk_first_copy_bio(struct request *rq, enum req_op op);
+struct bio *blk_next_copy_bio(struct bio *bio);
+unsigned int blk_copy_bio_count(struct request *rq, enum req_op op);
+
#endif /* __LINUX_BLK_COPY_H */
^ permalink raw reply related
* [PATCH 07/12] fs, block: Add copy_file_range() support for block devices
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche, Hannes Reinecke, Anuj Gupta
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
From: Nitesh Shetty <nj.shetty@samsung.com>
Add copy_file_range() support for block devices. If input and output block
devices have been opened with O_DIRECT and if copy offloading is supported
use blkdev_copy_offload(). Otherwise use splice_copy_file_range().
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
block/fops.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 54 insertions(+)
diff --git a/block/fops.c b/block/fops.c
index bb6642b45937..f438503f1b77 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -19,6 +19,7 @@
#include <linux/iomap.h>
#include <linux/module.h>
#include <linux/io_uring/cmd.h>
+#include <linux/splice.h>
#include "blk.h"
static inline struct inode *bdev_file_inode(struct file *file)
@@ -861,6 +862,58 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to)
return ret;
}
+static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in,
+ struct file *file_out, loff_t pos_out,
+ size_t len, unsigned int flags)
+{
+ struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in));
+ struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out));
+ loff_t in_end, out_end;
+ int err;
+
+ if (check_add_overflow(pos_in, len, &in_end) ||
+ PAGE_ALIGN(in_end) < in_end ||
+ check_add_overflow(pos_out, len, &out_end) ||
+ PAGE_ALIGN(out_end) < out_end)
+ return -EINVAL;
+
+ /*
+ * filemap_write_and_wait_range() and filemap_invalidate_inode() expect
+ * that the 'end' argument is rounded up to the next multiple of
+ * PAGE_SIZE.
+ */
+ in_end = PAGE_ALIGN(in_end);
+ out_end = PAGE_ALIGN(out_end);
+
+ if (bdev_max_copy_sectors(in_bdev) && bdev_max_copy_sectors(out_bdev) &&
+ file_in->f_iocb_flags & file_out->f_iocb_flags & IOCB_DIRECT) {
+ struct blk_copy_seg in_seg = { .pos = pos_in, .len = len };
+ struct blk_copy_seg out_seg = { .pos = pos_out, .len = len };
+ struct blk_copy_params params = {
+ .in_bdev = in_bdev,
+ .out_bdev = out_bdev,
+ .in_nseg = 1,
+ .in_segs = &in_seg,
+ .out_nseg = 1,
+ .out_segs = &out_seg,
+ };
+ err = filemap_write_and_wait_range(file_in->f_mapping, pos_in,
+ in_end);
+ if (err)
+ return err;
+ err = filemap_invalidate_inode(bdev_file_inode(file_out),
+ /*flush=*/false,
+ pos_out, out_end);
+ if (err)
+ return err;
+ if (blkdev_copy_offload(¶ms) == 0)
+ return len;
+ /* If copy offloading fails, fall back to onloading. */
+ }
+
+ return splice_copy_file_range(file_in, pos_in, file_out, pos_out, len);
+}
+
#define BLKDEV_FALLOC_FL_SUPPORTED \
(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \
FALLOC_FL_ZERO_RANGE | FALLOC_FL_WRITE_ZEROES)
@@ -967,6 +1020,7 @@ const struct file_operations def_blk_fops = {
.fallocate = blkdev_fallocate,
.uring_cmd = blkdev_uring_cmd,
.fop_flags = FOP_BUFFER_RASYNC,
+ .copy_file_range = blkdev_copy_file_range,
};
static __init int blkdev_init(void)
^ permalink raw reply related
* [PATCH 06/12] fs/read_write: Generalize generic_copy_file_checks()
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche, Anuj Gupta, Hannes Reinecke
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
From: Anuj Gupta <anuj20.g@samsung.com>
Prepare for adding copy_file_range() support for block devices by making
the following changes:
- Change file_inode(file) into file->f_mapping->host. Although only one
inode is associated with regular files, two inodes are associated
with block devices. file->f_mapping->host is the primary block device
inode.
- Change S_ISREG() into S_ISREG() || S_ISBLK().
- Add an inode->i_mode & S_IFMT check that verifies that source and
destination have the same type (block device or regular file).
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
[ bvanassche: rewrote patch description ]
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
fs/read_write.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/fs/read_write.c b/fs/read_write.c
index 50bff7edc91f..d6fba5afff94 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -1484,8 +1484,8 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
size_t *req_count, unsigned int flags)
{
- struct inode *inode_in = file_inode(file_in);
- struct inode *inode_out = file_inode(file_out);
+ struct inode *inode_in = file_in->f_mapping->host;
+ struct inode *inode_out = file_out->f_mapping->host;
uint64_t count = *req_count;
loff_t size_in;
int ret;
@@ -1791,7 +1791,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out)
/* Don't copy dirs, pipes, sockets... */
if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode))
return -EISDIR;
- if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode))
+ if (!S_ISREG(inode_in->i_mode) && !S_ISBLK(inode_in->i_mode))
+ return -EINVAL;
+ if ((inode_in->i_mode & S_IFMT) != (inode_out->i_mode & S_IFMT))
return -EINVAL;
if (!(file_in->f_mode & FMODE_READ) ||
^ permalink raw reply related
* [PATCH 10/12] dm: Add support for copy offloading
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
In dm_calculate_queue_limits(), clear the copy offload limits if the
device mapper driver does not support copy offloading. This is necessary
since blk_set_stacking_limits() sets the copy offload limits to their
maximum.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
drivers/md/dm-table.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index dc2eff6b739d..888c5bdca5f1 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -1816,6 +1816,14 @@ int dm_calculate_queue_limits(struct dm_table *t,
return -EINVAL;
combine_limits:
+ if (!(ti_limits.features & BLK_FEAT_STACKING_COPY_OFFL)) {
+ ti_limits.max_copy_hw_sectors = 0;
+ ti_limits.max_copy_src_segments = 0;
+ ti_limits.max_copy_dst_segments = 0;
+ ti_limits.max_user_copy_sectors = 0;
+ ti_limits.max_copy_sectors = 0;
+ }
+
/*
* Merge this target's queue limits into the overall limits
* for the table.
^ permalink raw reply related
* [PATCH 04/12] block: Add an onloaded copy implementation
From: Bart Van Assche @ 2026-04-24 22:41 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche, Vincent Fu, Anuj Gupta
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
From: Nitesh Shetty <nj.shetty@samsung.com>
For the devices which do not support copy offloading, add a function that
copies data by submitting READ and WRITE operations.
Onloaded copying is implemented by reading from the source block device
into memory and by writing this data to the destination block device.
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
block/blk-copy.c | 229 +++++++++++++++++++++++++++++++++++++++++
include/linux/blkdev.h | 1 +
2 files changed, 230 insertions(+)
diff --git a/block/blk-copy.c b/block/blk-copy.c
index 8ac8879442f7..459ed8581efc 100644
--- a/block/blk-copy.c
+++ b/block/blk-copy.c
@@ -7,6 +7,26 @@
#include <linux/blk-copy.h>
#include <linux/blk-mq.h>
+/**
+ * Tracks the state of a single onloaded copy operation.
+ * @params: Data copy parameters.
+ * @read_work: For scheduling read work.
+ * @write_work: For scheduling write work.
+ * @buf: Data buffer.
+ * @buf_len: Length in bytes of @buf.
+ * @offset: Current copying offset. Range: [0, @len[.
+ * @chunk: Size in bytes of the chunk of data that is being copied.
+ */
+struct blkdev_copy_onload_ctx {
+ struct blk_copy_params *params;
+ struct work_struct read_work;
+ struct work_struct write_work;
+ void *buf;
+ ssize_t buf_len;
+ loff_t offset;
+ loff_t chunk;
+};
+
/* End all bios in the @ctx->bios list with status @ctx->status. */
static void blkdev_end_bios(struct bio_copy_offload_ctx *ctx)
{
@@ -353,3 +373,212 @@ int blkdev_copy_offload(struct blk_copy_params *params)
return -EIOCBQUEUED;
}
EXPORT_SYMBOL_GPL(blkdev_copy_offload);
+
+static void *blkdev_copy_alloc_buf(size_t req_size, size_t *alloc_size)
+{
+ unsigned int min_size = PAGE_SIZE;
+ char *buf;
+
+ while (req_size >= min_size) {
+ buf = kmalloc(req_size, GFP_NOIO | __GFP_NOWARN);
+ if (buf) {
+ *alloc_size = req_size;
+ return buf;
+ }
+ req_size >>= 1;
+ }
+
+ return NULL;
+}
+
+static struct bio *bio_map_buf(void *buf, unsigned int len)
+{
+ struct page *page;
+ struct bio *bio;
+ static const uint16_t nr_vecs = 1;
+
+ bio = bio_kmalloc(nr_vecs, GFP_NOIO);
+ if (!bio)
+ return NULL;
+ bio_init_inline(bio, /*bdev=*/NULL, /*max_vecs=*/nr_vecs, /*opf=*/0);
+
+ page = virt_to_page(buf);
+ if (bio_add_page(bio, page, len, offset_in_page(buf)) < len) {
+ /* we don't support partial mappings */
+ bio_uninit(bio);
+ kfree(bio);
+ WARN_ON_ONCE(true);
+ return NULL;
+ }
+
+ return bio;
+}
+
+static void blkdev_write_done(struct bio *bio)
+{
+ struct blkdev_copy_onload_ctx *ctx = bio->bi_copy_ctx;
+ struct blk_copy_params *params = ctx->params;
+ blk_status_t sts = bio->bi_status;
+
+ kfree(bio);
+
+ if (sts) {
+ params->status = sts;
+ params->end_io(params);
+ return;
+ }
+
+ ctx->offset += ctx->chunk;
+
+ schedule_work(&ctx->read_work);
+}
+
+static sector_t blkdev_offset_to_out_pos(const struct blk_copy_params *params,
+ loff_t offset)
+{
+ for (int i = 0; i < params->out_nseg; i++) {
+ loff_t rem = params->out_segs[i].len - offset;
+
+ if (rem > 0)
+ return params->out_segs[i].pos + offset;
+ offset -= params->out_segs[i].len;
+ }
+ return 0;
+}
+
+static void blkdev_write_work(struct work_struct *work)
+{
+ struct blkdev_copy_onload_ctx *ctx =
+ container_of(work, typeof(*ctx), read_work);
+ struct blk_copy_params *params = ctx->params;
+ struct bio *bio;
+ loff_t out_pos;
+
+ out_pos = blkdev_offset_to_out_pos(params, ctx->offset);
+
+ bio = bio_map_buf(ctx->buf, ctx->buf_len);
+ if (!bio) {
+ params->status = BLK_STS_AGAIN;
+ params->end_io(params);
+ return;
+ }
+ bio->bi_opf = REQ_OP_WRITE;
+ bio_set_dev(bio, params->out_bdev);
+ bio->bi_iter.bi_sector = out_pos >> SECTOR_SHIFT;
+ bio->bi_iter.bi_size = ctx->chunk;
+ bio->bi_end_io = blkdev_write_done;
+ bio->bi_copy_ctx = ctx;
+ submit_bio(bio);
+}
+
+static void blkdev_read_done(struct bio *bio)
+{
+ struct blkdev_copy_onload_ctx *ctx = bio->bi_copy_ctx;
+ struct blk_copy_params *params = ctx->params;
+ blk_status_t sts = bio->bi_status;
+
+ kfree(bio);
+
+ if (sts) {
+ params->status = sts;
+ params->end_io(params);
+ return;
+ }
+
+ schedule_work(&ctx->write_work);
+}
+
+static sector_t blkdev_offset_to_in_pos(const struct blk_copy_params *params,
+ loff_t offset, loff_t *chunk)
+{
+ for (int i = 0; i < params->in_nseg; i++) {
+ loff_t rem = params->in_segs[i].len - offset;
+
+ if (rem > 0) {
+ if (*chunk > rem)
+ *chunk = rem;
+ return params->in_segs[i].pos + offset;
+ }
+ offset -= params->in_segs[i].len;
+ }
+ *chunk = 0;
+ return 0;
+}
+
+static void blkdev_read_work(struct work_struct *work)
+{
+ struct blkdev_copy_onload_ctx *ctx =
+ container_of(work, typeof(*ctx), read_work);
+ struct blk_copy_params *params = ctx->params;
+ loff_t offset = ctx->offset;
+ sector_t in_pos;
+ struct bio *bio;
+
+ ctx->chunk = min(ctx->buf_len, params->len - offset);
+ if (ctx->chunk)
+ in_pos = blkdev_offset_to_in_pos(params, offset, &ctx->chunk);
+ if (ctx->chunk == 0) {
+ params->end_io(params);
+ return;
+ }
+
+ bio = bio_map_buf(ctx->buf, ctx->buf_len);
+ if (!bio) {
+ params->status = BLK_STS_AGAIN;
+ params->end_io(params);
+ return;
+ }
+ bio->bi_opf = REQ_OP_READ;
+ bio_set_dev(bio, params->in_bdev);
+ bio->bi_iter.bi_sector = in_pos >> SECTOR_SHIFT;
+ bio->bi_iter.bi_size = ctx->chunk;
+ bio->bi_end_io = blkdev_read_done;
+ bio->bi_copy_ctx = ctx;
+ submit_bio(bio);
+}
+
+/**
+ * blkdev_copy_onload - asynchronously copy data between two block devices using
+ * read and write operations.
+ * @params: Input and output block devices, input and output ranges and
+ * completion callback pointer.
+ * Return: 0 upon success; -EIOCBQUEUED if the completion callback function will
+ * be called or has already been called.
+ */
+int blkdev_copy_onload(struct blk_copy_params *params)
+{
+ loff_t max_hw_bytes =
+ min(queue_max_hw_sectors(params->in_bdev->bd_queue),
+ queue_max_hw_sectors(params->out_bdev->bd_queue)) <<
+ SECTOR_SHIFT;
+ struct blkdev_copy_onload_ctx *ctx;
+ loff_t len;
+ int ret;
+
+ ret = blkdev_copy_check_params(params, &len);
+ if (ret)
+ return ret;
+
+ params->len = len;
+
+ ctx = kzalloc_obj(*ctx);
+ if (!ctx)
+ return -ENOMEM;
+
+ INIT_WORK(&ctx->read_work, blkdev_read_work);
+ INIT_WORK(&ctx->write_work, blkdev_write_work);
+ ctx->params = params;
+
+ ctx->buf = blkdev_copy_alloc_buf(min(max_hw_bytes, len), &ctx->buf_len);
+ if (!ctx->buf)
+ goto err;
+
+ blkdev_read_work(&ctx->read_work);
+
+ return -EIOCBQUEUED;
+
+err:
+ kfree(ctx);
+ return -ENOMEM;
+}
+EXPORT_SYMBOL_GPL(blkdev_copy_onload);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index fea296150cda..817eeba2f207 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1284,6 +1284,7 @@ void __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp);
int blkdev_copy_offload(struct blk_copy_params *params);
+int blkdev_copy_onload(struct blk_copy_params *params);
#define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */
#define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */
^ permalink raw reply related
* [PATCH 11/12] dm-linear: Enable copy offloading
From: Bart Van Assche @ 2026-04-24 22:42 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-block, linux-scsi, linux-nvme, Christoph Hellwig,
Nitesh Shetty, Bart Van Assche
In-Reply-To: <20260424224201.1949243-1-bvanassche@acm.org>
Set BLK_FEAT_STACKING_COPY_OFFL and max_copy_hw_sectors to enable copy
offloading.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
drivers/md/dm-linear.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
index 38c17846deb0..3de8bf5f11fb 100644
--- a/drivers/md/dm-linear.c
+++ b/drivers/md/dm-linear.c
@@ -119,6 +119,11 @@ static void linear_status(struct dm_target *ti, status_type_t type,
}
}
+static void linear_io_hints(struct dm_target *ti, struct queue_limits *limits)
+{
+ limits->features |= BLK_FEAT_STACKING_COPY_OFFL;
+}
+
static int linear_prepare_ioctl(struct dm_target *ti, struct block_device **bdev,
unsigned int cmd, unsigned long arg,
bool *forward)
@@ -211,6 +216,7 @@ static struct target_type linear_target = {
.dtr = linear_dtr,
.map = linear_map,
.status = linear_status,
+ .io_hints = linear_io_hints,
.prepare_ioctl = linear_prepare_ioctl,
.iterate_devices = linear_iterate_devices,
.direct_access = linear_dax_direct_access,
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox