[PATCH v3] bcache: fix cached_dev.sb_bio use-after-free and crash

[PATCH v3] bcache: fix cached_dev.sb_bio use-after-free and crash

[mingzhe.zou]

From: Mingzhe Zou <mingzhe.zou@easystack.cn>

In our production environment, we have received multiple crash reports
regarding libceph, which have caught our attention:

```
[6888366.280350] Call Trace:
[6888366.280452]  blk_update_request+0x14e/0x370
[6888366.280561]  blk_mq_end_request+0x1a/0x130
[6888366.280671]  rbd_img_handle_request+0x1a0/0x1b0 [rbd]
[6888366.280792]  rbd_obj_handle_request+0x32/0x40 [rbd]
[6888366.280903]  __complete_request+0x22/0x70 [libceph]
[6888366.281032]  osd_dispatch+0x15e/0xb40 [libceph]
[6888366.281164]  ? inet_recvmsg+0x5b/0xd0
[6888366.281272]  ? ceph_tcp_recvmsg+0x6f/0xa0 [libceph]
[6888366.281405]  ceph_con_process_message+0x79/0x140 [libceph]
[6888366.281534]  ceph_con_v1_try_read+0x5d7/0xf30 [libceph]
[6888366.281661]  ceph_con_workfn+0x329/0x680 [libceph]
```

After analyzing the coredump file, we found that the address of dc->sb_bio
has been freed. We know that cached_dev is only freed when it is stopped.

Since sb_bio is a part of struct cached_dev, rather than an alloc every time.
If the device is stopped while writing to the superblock, the released address
will be accessed at endio.

This patch hopes to wait for sb_write to complete in cached_dev_free.

Signed-off-by: Mingzhe Zou <mingzhe.zou@easystack.cn>

---
v2: fix the crash caused by not calling closure_init in v1
v3: pair the down and up of semaphore sb_write_mutex
---
 drivers/md/bcache/super.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
index 64bb38c95895..97d9adb0bf96 100644
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -1373,6 +1373,14 @@ static CLOSURE_CALLBACK(cached_dev_free)
 
 	mutex_unlock(&bch_register_lock);
 
+	/*
+	 * Wait for any pending sb_write to complete before free.
+	 * The sb_bio is embedded in struct cached_dev, so we must
+	 * ensure no I/O is in progress.
+	 */
+	down(&dc->sb_write_mutex);
+	up(&dc->sb_write_mutex);
+
 	if (dc->sb_disk)
 		folio_put(virt_to_folio(dc->sb_disk));
 
-- 
2.34.1

Re: [PATCH v3] bcache: fix cached_dev.sb_bio use-after-free and crash

[Coly Li]

On Mon, Mar 30, 2026 at 09:11:53PM +0800, mingzhe.zou@easystack.cn wrote:
> From: Mingzhe Zou <mingzhe.zou@easystack.cn>
> 
> In our production environment, we have received multiple crash reports
> regarding libceph, which have caught our attention:
> 
> ```
> [6888366.280350] Call Trace:
> [6888366.280452]  blk_update_request+0x14e/0x370
> [6888366.280561]  blk_mq_end_request+0x1a/0x130
> [6888366.280671]  rbd_img_handle_request+0x1a0/0x1b0 [rbd]
> [6888366.280792]  rbd_obj_handle_request+0x32/0x40 [rbd]
> [6888366.280903]  __complete_request+0x22/0x70 [libceph]
> [6888366.281032]  osd_dispatch+0x15e/0xb40 [libceph]
> [6888366.281164]  ? inet_recvmsg+0x5b/0xd0
> [6888366.281272]  ? ceph_tcp_recvmsg+0x6f/0xa0 [libceph]
> [6888366.281405]  ceph_con_process_message+0x79/0x140 [libceph]
> [6888366.281534]  ceph_con_v1_try_read+0x5d7/0xf30 [libceph]
> [6888366.281661]  ceph_con_workfn+0x329/0x680 [libceph]
> ```
> 
> After analyzing the coredump file, we found that the address of dc->sb_bio
> has been freed. We know that cached_dev is only freed when it is stopped.
> 
> Since sb_bio is a part of struct cached_dev, rather than an alloc every time.
> If the device is stopped while writing to the superblock, the released address
> will be accessed at endio.
> 
> This patch hopes to wait for sb_write to complete in cached_dev_free.
> 
> Signed-off-by: Mingzhe Zou <mingzhe.zou@easystack.cn>
> 

Hi Mingzhe,

Yeah, this patch is in better shape. Thank for the fix up again.

> ---
> v2: fix the crash caused by not calling closure_init in v1
> v3: pair the down and up of semaphore sb_write_mutex
> ---
>  drivers/md/bcache/super.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
> index 64bb38c95895..97d9adb0bf96 100644
> --- a/drivers/md/bcache/super.c
> +++ b/drivers/md/bcache/super.c
> @@ -1373,6 +1373,14 @@ static CLOSURE_CALLBACK(cached_dev_free)
>  
>  	mutex_unlock(&bch_register_lock);
>  
> +	/*
> +	 * Wait for any pending sb_write to complete before free.
> +	 * The sb_bio is embedded in struct cached_dev, so we must
> +	 * ensure no I/O is in progress.
> +	 */
> +	down(&dc->sb_write_mutex);
> +	up(&dc->sb_write_mutex);
> +
>  	if (dc->sb_disk)
>  		folio_put(virt_to_folio(dc->sb_disk));
>  

The patch itself is good. But the previous one (based on closre_sync()) is
in Jens block-tree. Let me ask.

Hi Jens,

Should Mingzhe send a incremental patch based on commit b36478a1fece in
block-7.0 branch of linux-block tree? Or Just use this patch to replace
the in-linux-block-tree one?

Thanks in advance for your hint.

Coly Li

Re: [PATCH v3] bcache: fix cached_dev.sb_bio use-after-free and crash

[Coly Li]

On Tue, Mar 31, 2026 at 09:56:18AM +0800, Coly Li wrote:
> On Mon, Mar 30, 2026 at 09:11:53PM +0800, mingzhe.zou@easystack.cn wrote:
> > From: Mingzhe Zou <mingzhe.zou@easystack.cn>
> > 
> > In our production environment, we have received multiple crash reports
> > regarding libceph, which have caught our attention:
> > 
> > ```
> > [6888366.280350] Call Trace:
> > [6888366.280452]  blk_update_request+0x14e/0x370
> > [6888366.280561]  blk_mq_end_request+0x1a/0x130
> > [6888366.280671]  rbd_img_handle_request+0x1a0/0x1b0 [rbd]
> > [6888366.280792]  rbd_obj_handle_request+0x32/0x40 [rbd]
> > [6888366.280903]  __complete_request+0x22/0x70 [libceph]
> > [6888366.281032]  osd_dispatch+0x15e/0xb40 [libceph]
> > [6888366.281164]  ? inet_recvmsg+0x5b/0xd0
> > [6888366.281272]  ? ceph_tcp_recvmsg+0x6f/0xa0 [libceph]
> > [6888366.281405]  ceph_con_process_message+0x79/0x140 [libceph]
> > [6888366.281534]  ceph_con_v1_try_read+0x5d7/0xf30 [libceph]
> > [6888366.281661]  ceph_con_workfn+0x329/0x680 [libceph]
> > ```
> > 
> > After analyzing the coredump file, we found that the address of dc->sb_bio
> > has been freed. We know that cached_dev is only freed when it is stopped.
> > 
> > Since sb_bio is a part of struct cached_dev, rather than an alloc every time.
> > If the device is stopped while writing to the superblock, the released address
> > will be accessed at endio.
> > 
> > This patch hopes to wait for sb_write to complete in cached_dev_free.
> > 
> > Signed-off-by: Mingzhe Zou <mingzhe.zou@easystack.cn>
> > 
> 
> Hi Mingzhe,
> 
> Yeah, this patch is in better shape. Thank for the fix up again.
> 
> > ---
> > v2: fix the crash caused by not calling closure_init in v1
> > v3: pair the down and up of semaphore sb_write_mutex
> > ---
> >  drivers/md/bcache/super.c | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> > 
> > diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
> > index 64bb38c95895..97d9adb0bf96 100644
> > --- a/drivers/md/bcache/super.c
> > +++ b/drivers/md/bcache/super.c
> > @@ -1373,6 +1373,14 @@ static CLOSURE_CALLBACK(cached_dev_free)
> >  
> >  	mutex_unlock(&bch_register_lock);
> >  
> > +	/*
> > +	 * Wait for any pending sb_write to complete before free.
> > +	 * The sb_bio is embedded in struct cached_dev, so we must
> > +	 * ensure no I/O is in progress.
> > +	 */
> > +	down(&dc->sb_write_mutex);
> > +	up(&dc->sb_write_mutex);
> > +
> >  	if (dc->sb_disk)
> >  		folio_put(virt_to_folio(dc->sb_disk));
> >  
> 
> The patch itself is good. But the previous one (based on closre_sync()) is
> in Jens block-tree. Let me ask.
> 
> Hi Jens,
> 
> Should Mingzhe send a incremental patch based on commit b36478a1fece in
> block-7.0 branch of linux-block tree? Or Just use this patch to replace
> the in-linux-block-tree one?
> 

Hi Mingzhe,

Considering no responds from Jens at this moment, I assume that an incremental
patch againt the already-in-linux-block patch might be more convenient for him.
Then he can simply apply the incremental patch without extra rebase stuffs.

Can you re-compose a patch against block-7.0 branch of linux-block tree?

Thanks.

Coly Li

Re: [PATCH v3] bcache: fix cached_dev.sb_bio use-after-free and crash

[Jens Axboe]

On 3/31/26 9:46 AM, Coly Li wrote:
> On Tue, Mar 31, 2026 at 09:56:18AM +0800, Coly Li wrote:
>> On Mon, Mar 30, 2026 at 09:11:53PM +0800, mingzhe.zou@easystack.cn wrote:
>>> From: Mingzhe Zou <mingzhe.zou@easystack.cn>
>>>
>>> In our production environment, we have received multiple crash reports
>>> regarding libceph, which have caught our attention:
>>>
>>> ```
>>> [6888366.280350] Call Trace:
>>> [6888366.280452]  blk_update_request+0x14e/0x370
>>> [6888366.280561]  blk_mq_end_request+0x1a/0x130
>>> [6888366.280671]  rbd_img_handle_request+0x1a0/0x1b0 [rbd]
>>> [6888366.280792]  rbd_obj_handle_request+0x32/0x40 [rbd]
>>> [6888366.280903]  __complete_request+0x22/0x70 [libceph]
>>> [6888366.281032]  osd_dispatch+0x15e/0xb40 [libceph]
>>> [6888366.281164]  ? inet_recvmsg+0x5b/0xd0
>>> [6888366.281272]  ? ceph_tcp_recvmsg+0x6f/0xa0 [libceph]
>>> [6888366.281405]  ceph_con_process_message+0x79/0x140 [libceph]
>>> [6888366.281534]  ceph_con_v1_try_read+0x5d7/0xf30 [libceph]
>>> [6888366.281661]  ceph_con_workfn+0x329/0x680 [libceph]
>>> ```
>>>
>>> After analyzing the coredump file, we found that the address of dc->sb_bio
>>> has been freed. We know that cached_dev is only freed when it is stopped.
>>>
>>> Since sb_bio is a part of struct cached_dev, rather than an alloc every time.
>>> If the device is stopped while writing to the superblock, the released address
>>> will be accessed at endio.
>>>
>>> This patch hopes to wait for sb_write to complete in cached_dev_free.
>>>
>>> Signed-off-by: Mingzhe Zou <mingzhe.zou@easystack.cn>
>>>
>>
>> Hi Mingzhe,
>>
>> Yeah, this patch is in better shape. Thank for the fix up again.
>>
>>> ---
>>> v2: fix the crash caused by not calling closure_init in v1
>>> v3: pair the down and up of semaphore sb_write_mutex
>>> ---
>>>  drivers/md/bcache/super.c | 8 ++++++++
>>>  1 file changed, 8 insertions(+)
>>>
>>> diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
>>> index 64bb38c95895..97d9adb0bf96 100644
>>> --- a/drivers/md/bcache/super.c
>>> +++ b/drivers/md/bcache/super.c
>>> @@ -1373,6 +1373,14 @@ static CLOSURE_CALLBACK(cached_dev_free)
>>>  
>>>  	mutex_unlock(&bch_register_lock);
>>>  
>>> +	/*
>>> +	 * Wait for any pending sb_write to complete before free.
>>> +	 * The sb_bio is embedded in struct cached_dev, so we must
>>> +	 * ensure no I/O is in progress.
>>> +	 */
>>> +	down(&dc->sb_write_mutex);
>>> +	up(&dc->sb_write_mutex);
>>> +
>>>  	if (dc->sb_disk)
>>>  		folio_put(virt_to_folio(dc->sb_disk));
>>>  
>>
>> The patch itself is good. But the previous one (based on closre_sync()) is
>> in Jens block-tree. Let me ask.
>>
>> Hi Jens,
>>
>> Should Mingzhe send a incremental patch based on commit b36478a1fece in
>> block-7.0 branch of linux-block tree? Or Just use this patch to replace
>> the in-linux-block-tree one?
>>
> 
> Hi Mingzhe,
> 
> Considering no responds from Jens at this moment, I assume that an incremental
> patch againt the already-in-linux-block patch might be more convenient for him.
> Then he can simply apply the incremental patch without extra rebase stuffs.
> 
> Can you re-compose a patch against block-7.0 branch of linux-block tree?

Yes send a new patch, I'm not going to rebase the tree just to fold in
a fix for a patch that's deep in the stack. The fixup doesn't even
mention which patch it's fixing either...

-- 
Jens Axboe