[PATCH 0/2] ubi: block: fix use-after-free and deadlock

[PATCH 0/2] ubi: block: fix use-after-free and deadlock

[Vincent Whitchurch]

This series fixes a use-after-free and a deadlock in ubiblock's removal
handling.

---
Vincent Whitchurch (2):
      ubi: block: Fix use-after-free of gendisk
      ubi: block: Fix deadlock on remove

 drivers/mtd/ubi/block.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
---
base-commit: 44c026a73be8038f03dbdeef028b642880cf1511
change-id: 20230523-ubiblock-remove-eab61cf683f0

Best regards,
-- 
Vincent Whitchurch <vincent.whitchurch@axis.com>

[PATCH 1/2] ubi: block: Fix use-after-free of gendisk

[Vincent Whitchurch]

Do not touch the gendisk after put_disk() to fix this use-after-free:

 ==================================================
 BUG: KASAN: slab-use-after-free in ubiblock_remove
 Read of size 4 by task ubiblock/361

 Call Trace:
 ubiblock_remove (drivers/mtd/ubi/block.c:459 drivers/mtd/ubi/block.c:483)
 vol_cdev_ioctl
 ...

 Allocated by task 358:
 __alloc_disk_node (block/genhd.c:1377)
 __blk_mq_alloc_disk (block/blk-mq.c:4093)
 ubiblock_create (drivers/mtd/ubi/block.c:397)
 vol_cdev_ioctl
 ...

 Freed by task 0:
 bdev_free_inode (block/bdev.c:337)
 i_callback
 rcu_core
 __do_softirq
 ...

Signed-off-by: Vincent Whitchurch <vincent.whitchurch@axis.com>
---
 drivers/mtd/ubi/block.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/mtd/ubi/block.c b/drivers/mtd/ubi/block.c
index 3711d7f74600..70caec4606cd 100644
--- a/drivers/mtd/ubi/block.c
+++ b/drivers/mtd/ubi/block.c
@@ -448,13 +448,15 @@ int ubiblock_create(struct ubi_volume_info *vi)
 
 static void ubiblock_cleanup(struct ubiblock *dev)
 {
+	int first_minor = dev->gd->first_minor;
+
 	/* Stop new requests to arrive */
 	del_gendisk(dev->gd);
 	/* Finally destroy the blk queue */
 	dev_info(disk_to_dev(dev->gd), "released");
 	put_disk(dev->gd);
 	blk_mq_free_tag_set(&dev->tag_set);
-	idr_remove(&ubiblock_minor_idr, dev->gd->first_minor);
+	idr_remove(&ubiblock_minor_idr, first_minor);
 }
 
 int ubiblock_remove(struct ubi_volume_info *vi)

-- 
2.34.1

Re: [PATCH 1/2] ubi: block: Fix use-after-free of gendisk

[Christoph Hellwig]

On Tue, May 23, 2023 at 03:12:16PM +0200, Vincent Whitchurch wrote:
>  static void ubiblock_cleanup(struct ubiblock *dev)
>  {
> +	int first_minor = dev->gd->first_minor;
> +
>  	/* Stop new requests to arrive */
>  	del_gendisk(dev->gd);
>  	/* Finally destroy the blk queue */
>  	dev_info(disk_to_dev(dev->gd), "released");
>  	put_disk(dev->gd);
>  	blk_mq_free_tag_set(&dev->tag_set);
> -	idr_remove(&ubiblock_minor_idr, dev->gd->first_minor);
> +	idr_remove(&ubiblock_minor_idr, first_minor);

I think the real fix here is to implement the free_disk method
and free the idr there.  That ensures the ID can't be reused until
the disk is entirely freed as well.

[PATCH 2/2] ubi: block: Fix deadlock on remove

[Vincent Whitchurch]

Lockdep warns about possible circular locking when the following
commands are run:

 ubiblock --create /dev/ubi0_0
 head -c1 /dev/ubiblock0_0 > /dev/null
 ubiblock --remove /dev/ubi0_0

 ======================================================
 WARNING: possible circular locking dependency detected

 ubiblock/364 is trying to acquire lock:
 (&disk->open_mutex){+.+.}-{3:3}, at: del_gendisk (block/genhd.c:616)

 but task is already holding lock:
 (&dev->dev_mutex){+.+.}-{3:3}, at: ubiblock_remove (drivers/mtd/ubi/block.c:476)

 which lock already depends on the new lock.

 the existing dependency chain (in reverse order) is:

 -> #1 (&dev->dev_mutex){+.+.}-{3:3}:
   ubiblock_open (drivers/mtd/ubi/block.c:236)
   blkdev_get_whole (block/bdev.c:607)
   blkdev_get_by_dev (block/bdev.c:756)
   blkdev_open (block/fops.c:493)
   ...
   do_sys_openat2 (fs/open.c:1356)

 -> #0 (&disk->open_mutex){+.+.}-{3:3}:
   del_gendisk (block/genhd.c:616)
   ubiblock_remove (drivers/mtd/ubi/block.c:456 drivers/mtd/ubi/block.c:483)
   vol_cdev_ioctl
   ...

 other info that might help us debug this:

  Possible unsafe locking scenario:

        CPU0                    CPU1
        ----                    ----
   lock(&dev->dev_mutex);
                                lock(&disk->open_mutex);
                                lock(&dev->dev_mutex);
   lock(&disk->open_mutex);

 *** DEADLOCK ***

 Call Trace:
 del_gendisk (block/genhd.c:616)
 ubiblock_remove (drivers/mtd/ubi/block.c:456 drivers/mtd/ubi/block.c:483)
 vol_cdev_ioctl
 ...

The actual deadlock is also easily reproducible by running the above
commands in parallel in a loop.

Fix this by marking the device as going away and releasing the dev mutex
before del_gendisk().  This is similar to other drivers such as
drivers/block/zram/zram_drv.c.

Signed-off-by: Vincent Whitchurch <vincent.whitchurch@axis.com>
---
 drivers/mtd/ubi/block.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/mtd/ubi/block.c b/drivers/mtd/ubi/block.c
index 70caec4606cd..fcfea7cfdb6b 100644
--- a/drivers/mtd/ubi/block.c
+++ b/drivers/mtd/ubi/block.c
@@ -83,6 +83,8 @@ struct ubiblock {
 	struct mutex dev_mutex;
 	struct list_head list;
 	struct blk_mq_tag_set tag_set;
+
+	bool removing;
 };
 
 /* Linked list of all ubiblock instances */
@@ -233,6 +235,11 @@ static int ubiblock_open(struct block_device *bdev, fmode_t mode)
 	int ret;
 
 	mutex_lock(&dev->dev_mutex);
+	if (dev->removing) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
 	if (dev->refcnt > 0) {
 		/*
 		 * The volume is already open, just increase the reference
@@ -480,8 +487,15 @@ int ubiblock_remove(struct ubi_volume_info *vi)
 
 	/* Remove from device list */
 	list_del(&dev->list);
-	ubiblock_cleanup(dev);
+
+	/*
+	 * Prevent further opens.  del_gendisk() will ensure that there are no
+	 * parallel openers.
+	 */
+	dev->removing = true;
 	mutex_unlock(&dev->dev_mutex);
+
+	ubiblock_cleanup(dev);
 	mutex_unlock(&devices_mutex);
 
 	kfree(dev);

-- 
2.34.1

Re: [PATCH 2/2] ubi: block: Fix deadlock on remove

[Christoph Hellwig]

If you imlement ->free_disk, the list_del and kfree can move into
that, and we don't really care if a new opener raced with the delete.

Re: [PATCH 2/2] ubi: block: Fix deadlock on remove

[Vincent Whitchurch]

On Tue, 2023-05-23 at 23:04 -0700, Christoph Hellwig wrote:
> If you imlement ->free_disk, the list_del and kfree can move into
> that, and we don't really care if a new opener raced with the delete.

Moving the kfree() to ->free_disk() works, but the list_del() still
needs to be in ubiblock_remove() since otherwise ubiblock_remove() could
attempt to remove the same device twice.

I assumed the current code really wanted to prevent new openers racing
with delete, but if that is not needed, yes, we don't need to add a
->removing flag if we move the kfree() to ->free_disk().  I'll re-spin
this based on your suggestions.  Thanks.

Re: [PATCH 2/2] ubi: block: Fix deadlock on remove

[hch]

On Wed, May 24, 2023 at 01:36:39PM +0000, Vincent Whitchurch wrote:
> On Tue, 2023-05-23 at 23:04 -0700, Christoph Hellwig wrote:
> > If you imlement ->free_disk, the list_del and kfree can move into
> > that, and we don't really care if a new opener raced with the delete.
> 
> Moving the kfree() to ->free_disk() works, but the list_del() still
> needs to be in ubiblock_remove() since otherwise ubiblock_remove() could
> attempt to remove the same device twice.

Or we'd still need your removed flag..

> I assumed the current code really wanted to prevent new openers racing
> with delete, but if that is not needed, yes, we don't need to add a
> ->removing flag if we move the kfree() to ->free_disk().  I'll re-spin
> this based on your suggestions.  Thanks.

I think in the past we always had to protect against removals of live
devices because handling of hot removes sucked so bad, both in drivers
and in the block layer itself.  With some newer infrastructure including
the ->free_disk method this can now be handled sanely.