Re: [PATCH v4] btrfs: fix a use-after-free bug when hitting errors inside btrfs_submit_chunk()

[David Sterba <dsterba@suse.cz>]

On Sat, Aug 24, 2024 at 07:28:23PM +0930, Qu Wenruo wrote:
> [BUG]
> There is an internal report that KASAN is reporting use-after-free, with
> the following backtrace:
> 
>  ==================================================================
>  BUG: KASAN: slab-use-after-free in btrfs_check_read_bio+0xa68/0xb70 [btrfs]
>  Read of size 4 at addr ffff8881117cec28 by task kworker/u16:2/45
>  CPU: 1 UID: 0 PID: 45 Comm: kworker/u16:2 Not tainted 6.11.0-rc2-next-20240805-default+ #76
>  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-3-gd478f380-rebuilt.opensuse.org 04/01/2014
>  Workqueue: btrfs-endio btrfs_end_bio_work [btrfs]
>  Call Trace:
>   <TASK>
>   dump_stack_lvl+0x61/0x80
>   print_address_description.constprop.0+0x5e/0x2f0
>   print_report+0x118/0x216
>   kasan_report+0x11d/0x1f0
>   btrfs_check_read_bio+0xa68/0xb70 [btrfs]
>   process_one_work+0xce0/0x12a0
>   worker_thread+0x717/0x1250
>   kthread+0x2e3/0x3c0
>   ret_from_fork+0x2d/0x70
>   ret_from_fork_asm+0x11/0x20
>   </TASK>
> 
>  Allocated by task 20917:
>   kasan_save_stack+0x37/0x60
>   kasan_save_track+0x10/0x30
>   __kasan_slab_alloc+0x7d/0x80
>   kmem_cache_alloc_noprof+0x16e/0x3e0
>   mempool_alloc_noprof+0x12e/0x310
>   bio_alloc_bioset+0x3f0/0x7a0
>   btrfs_bio_alloc+0x2e/0x50 [btrfs]
>   submit_extent_page+0x4d1/0xdb0 [btrfs]
>   btrfs_do_readpage+0x8b4/0x12a0 [btrfs]
>   btrfs_readahead+0x29a/0x430 [btrfs]
>   read_pages+0x1a7/0xc60
>   page_cache_ra_unbounded+0x2ad/0x560
>   filemap_get_pages+0x629/0xa20
>   filemap_read+0x335/0xbf0
>   vfs_read+0x790/0xcb0
>   ksys_read+0xfd/0x1d0
>   do_syscall_64+0x6d/0x140
>   entry_SYSCALL_64_after_hwframe+0x4b/0x53
> 
>  Freed by task 20917:
>   kasan_save_stack+0x37/0x60
>   kasan_save_track+0x10/0x30
>   kasan_save_free_info+0x37/0x50
>   __kasan_slab_free+0x4b/0x60
>   kmem_cache_free+0x214/0x5d0
>   bio_free+0xed/0x180
>   end_bbio_data_read+0x1cc/0x580 [btrfs]
>   btrfs_submit_chunk+0x98d/0x1880 [btrfs]
>   btrfs_submit_bio+0x33/0x70 [btrfs]
>   submit_one_bio+0xd4/0x130 [btrfs]
>   submit_extent_page+0x3ea/0xdb0 [btrfs]
>   btrfs_do_readpage+0x8b4/0x12a0 [btrfs]
>   btrfs_readahead+0x29a/0x430 [btrfs]
>   read_pages+0x1a7/0xc60
>   page_cache_ra_unbounded+0x2ad/0x560
>   filemap_get_pages+0x629/0xa20
>   filemap_read+0x335/0xbf0
>   vfs_read+0x790/0xcb0
>   ksys_read+0xfd/0x1d0
>   do_syscall_64+0x6d/0x140
>   entry_SYSCALL_64_after_hwframe+0x4b/0x53
> 
> [CAUSE]
> Although I can not reproduce the error, the report itself is good enough
> to pin down the cause.
> 
> The call trace is the regular endio workqueue context, but the
> free-by-task trace is showing that during btrfs_submit_chunk() we
> already hit a critical error, and is calling btrfs_bio_end_io() to error
> out.
> And the original endio function called bio_put() to free the whole bio.
> 
> This means a double freeing thus causing use-after-free, e.g:
> 
> 1. Enter btrfs_submit_bio() with a read bio
>    The read bio length is 128K, crossing two 64K stripes.
> 
> 2. The first run of btrfs_submit_chunk()
> 
> 2.1 Call btrfs_map_block(), which returns 64K
> 2.2 Call btrfs_split_bio()
>     Now there are two bios, one referring to the first 64K, the other
>     referring to the second 64K.
> 2.3 The first half is submitted.
> 
> 3. The second run of btrfs_submit_chunk()
> 
> 3.1 Call btrfs_map_block(), which by somehow failed
>     Now we call btrfs_bio_end_io() to handle the error
> 
> 3.2 btrfs_bio_end_io() calls the original endio function
>     Which is end_bbio_data_read(), and it calls bio_put() for the
>     original bio.
> 
>     Now the original bio is freed.
> 
> 4. The submitted first 64K bio finished
>    Now we call into btrfs_check_read_bio() and tries to advance the bio
>    iter.
>    But since the original bio (thus its iter) is already freed, we
>    trigger the above use-after free.
> 
>    And even if the memory is not poisoned/corrupted, we will later call
>    the original endio function, causing a double freeing.
> 
> [FIX]
> Instead of calling btrfs_bio_end_io(), call btrfs_orig_bbio_end_io(),
> which has the extra check on split bios and do the proper refcounting
> for cloned bios.
> 
> Furthermore there is already one extra btrfs_cleanup_bio() call, but
> that is duplicated to btrfs_orig_bbio_end_io() call, so remove that
> tag completely.
> 
> Reported-by: David Sterba <dsterba@suse.cz>
> Fixes: 852eee62d31a ("btrfs: allow btrfs_submit_bio to split bios")
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
> Changelog:
> v4:
> - Fix a case where the endio function is never called
>   If we have split the bbio and hit a critical error, we have to end
>   both the current and the remaining bbio.
>   As the remaining one will never be submitted, thus pending_ios will
>   never reach 0.

The tests now pass, thanks.

Reviewed-by: David Sterba <dsterba@suse.com>