From: Jens Axboe <axboe@kernel.dk>
To: Vishal Verma <vverma@digitalocean.com>, Song Liu <song@kernel.org>
Cc: linux-raid <linux-raid@vger.kernel.org>, rgoldwyn@suse.de
Subject: Re: [PATCH v5 3/4] md: raid10 add nowait support
Date: Tue, 21 Dec 2021 08:59:42 -0700 [thread overview]
Message-ID: <5d718039-52d4-33e7-b7be-c12c3100a1e5@kernel.dk> (raw)
In-Reply-To: <75ef7111-c291-51a9-5531-fa6965361b06@digitalocean.com>
On 12/21/21 8:29 AM, Vishal Verma wrote:
>
> On 12/21/21 1:13 AM, Song Liu wrote:
>> On Mon, Dec 20, 2021 at 6:22 AM Vishal Verma <vverma@digitalocean.com> wrote:
>>>
>>> On 12/16/21 4:50 PM, Song Liu wrote:
>>>
>>> On Thu, Dec 16, 2021 at 12:38 PM Vishal Verma <vverma@digitalocean.com> wrote:
>>>
>>> [...]
>>>
>>> [ 740.106431] invalid opcode: 0000 [#1] PREEMPT SMP KASAN NOPTI
>>>
>>> What's the exact command line that triggers this? I am not able to
>>> trigger it with
>>> either fio or t/io_uring.
>>>
>>> Song
>>>
>>> I only had 1 nvme so was creating 4 partitions on it and creating a
>>> raid10 and doing:
>>>
>>> mdadm -C /dev/md10 -l 10 -n 4 /dev/nvme4n1p1 /dev/nvme4n1p2
>>> /dev/nvme4n1p3 /dev/nvme4n1p4
>>> ./t/io_uring /dev/md10-d 256 -p 0 -a 0 -r 100
>>>
>>> on top of commit: c14704e1cb556 (md-next branch) + "md: add support for
>>> REQ_NOWAIT" patch
>>> Also, applied the commit (75feae73a28) Jens pointed earlier today.
>>>
>>> I am able to trigger the following error. I will look into it.
>>>
>>> Thanks,
>>> Song
>>>
>>> [ 1583.149004] ==================================================================
>>> [ 1583.150100] BUG: KASAN: use-after-free in raid10_end_read_request+0x91/0x310
>>> [ 1583.151042] Read of size 8 at addr ffff888160a1c928 by task io_uring/1165
>>> [ 1583.152016]
>>> [ 1583.152247] CPU: 0 PID: 1165 Comm: io_uring Not tainted 5.16.0-rc3+ #660
>>> [ 1583.153159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>>> BIOS 1.13.0-2.module_el8.4.0+547+a85d02ba 04/01/2014
>>> [ 1583.154572] Call Trace:
>>> [ 1583.155005] <IRQ>
>>> [ 1583.155338] dump_stack_lvl+0x44/0x57
>>> [ 1583.155950] print_address_description.constprop.8.cold.17+0x12/0x339
>>> [ 1583.156969] ? raid10_end_read_request+0x91/0x310
>>> [ 1583.157578] ? raid10_end_read_request+0x91/0x310
>>> [ 1583.158272] kasan_report.cold.18+0x83/0xdf
>>> [ 1583.158889] ? raid10_end_read_request+0x91/0x310
>>> [ 1583.159554] raid10_end_read_request+0x91/0x310
>>> [ 1583.160201] ? raid10_resize+0x270/0x270
>>> [ 1583.160724] ? bio_uninit+0xc7/0x1e0
>>> [ 1583.161274] blk_update_request+0x21f/0x810
>>> [ 1583.161893] blk_mq_end_request_batch+0x11c/0xa70
>>> [ 1583.162497] ? blk_mq_end_request+0x460/0x460
>>> [ 1583.163204] ? nvme_complete_batch_req+0x12/0x30
>>> [ 1583.163888] nvme_irq+0x6ad/0x6f0
>>> [ 1583.164354] ? io_queue_count_set+0xe0/0xe0
>>> [ 1583.164980] ? nvme_unmap_data+0x1e0/0x1e0
>>> [ 1583.165504] ? rcu_read_lock_bh_held+0xb0/0xb0
>>> [ 1583.166149] ? io_queue_count_set+0xe0/0xe0
>>> [ 1583.166721] __handle_irq_event_percpu+0x79/0x440
>>> [ 1583.167446] handle_irq_event_percpu+0x6f/0xe0
>>> [ 1583.168101] ? __handle_irq_event_percpu+0x440/0x440
>>> [ 1583.168734] ? lock_contended+0x6e0/0x6e0
>>> [ 1583.169349] ? do_raw_spin_unlock+0xa2/0x130
>>> [ 1583.169961] handle_irq_event+0x54/0x90
>>> [ 1583.170442] handle_edge_irq+0x121/0x300
>>> [ 1583.171012] __common_interrupt+0x7d/0x170
>>> [ 1583.171538] common_interrupt+0xa0/0xc0
>>> [ 1583.172103] </IRQ>
>>> [ 1583.172389] <TASK>
>>>
>>> When running t/io_uring on a raid1 array, I get following:
>>>
>>> [ 189.863726] RIP: 0010:__kmalloc+0xfa/0x430
>>> [ 189.867825] Code: 05 4b 9a 35 43 48 8b 50 08 48 83 78 10 00 4c 8b 20 0f 84 fa 02 00 00 4d 85 e4 0f 84 f1 02 00 00 41 8b 47 28 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f b8 00 00 00 4c 89 e0 48 0f c9 48 31 cb
>>> [ 189.886573] RSP: 0018:ffffaf09e28b7828 EFLAGS: 00010286
>>> [ 189.891799] RAX: a0fa1099d2b0fff3 RBX: 0000000000092900 RCX: 0000000000000000
>>> [ 189.898930] RDX: 00000002ba79600b RSI: 0000000000092900 RDI: 00000000000340e0
>>> [ 189.906062] RBP: ffffaf09e28b7860 R08: ffff90fb8b6ea560 R09: ffff90fba7205f60
>>> [ 189.913195] R10: ffffaf09e28b7c18 R11: 0000000000000000 R12: a0fa1099d2b0ffb3
>>> [ 189.920329] R13: 0000000000000000 R14: ffffffffc074c277 R15: ffff90bc00044700
>>> [ 189.927461] FS: 00007fd6209d7700(0000) GS:ffff913a6e140000(0000) knlGS:0000000000000000
>>> [ 189.935549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 189.941295] CR2: 00007f16998bebf0 CR3: 00000040be512005 CR4: 0000000000770ee0
>>> [ 189.948426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [ 189.955560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>> [ 189.962691] PKRU: 55555554
>>> [ 189.965403] Call Trace:
>>> [ 189.967857] <TASK>
>>> [ 189.969966] 0xffffffffc074c277
>>> [ 189.973110] mempool_alloc+0x61/0x180
>>> [ 189.976777] ? bio_associate_blkg_from_css+0xf5/0x2c0
>>> [ 189.981829] ? __bio_clone_fast+0xa9/0xf0
>>> [ 189.985842] ? __sbitmap_get_word+0x36/0x80
>>> [ 189.990027] 0xffffffffc074ac50
>>> [ 189.993174] ? __sbitmap_queue_get+0x9/0x10
>>> [ 189.997359] ? blk_mq_get_tag+0x241/0x270
>>> [ 190.001373] ? ktime_get+0x3b/0xa0
>>> [ 190.004776] ? blk_mq_rq_ctx_init.isra.0+0x1a5/0x1c0
>>> [ 190.009743] 0xffffffffc074efb3
>>> [ 190.012891] md_handle_request+0x134/0x1b0
>>> [ 190.016989] ? ktime_get+0x3b/0xa0
>>> [ 190.020395] md_submit_bio+0x6d/0xa0
>>> [ 190.023976] __submit_bio+0x94/0x140
>>> [ 190.027555] submit_bio_noacct+0xe1/0x2a0
>>> [ 190.031566] submit_bio+0x48/0x120
>>> [ 190.034972] blkdev_direct_IO+0x19b/0x540
>>> [ 190.038987] ? __fsnotify_parent+0xff/0x330
>>> [ 190.043172] ? __fsnotify_parent+0x10f/0x330
>>> [ 190.047445] generic_file_read_iter+0xa5/0x160
>>> [ 190.051889] blkdev_read_iter+0x38/0x70
>>> [ 190.055731] io_read+0x119/0x420
>>> [ 190.058963] ? blk_queue_exit+0x23/0x50
>>> [ 190.062801] ? __blk_mq_free_request+0x86/0xc0
>>> [ 190.067247] io_issue_sqe+0x7ec/0x19c0
>>> [ 190.071002] ? io_req_prep+0x6a9/0xe60
>>> [ 190.074754] io_submit_sqes+0x2a0/0x9f0
>>> [ 190.078594] ? __fget_files+0x6a/0x90
>>> [ 190.082259] __x64_sys_io_uring_enter+0x1da/0x8c0
>>> [ 190.086965] ? debug_smp_processor_id+0x17/0x20
>>> [ 190.091498] ? fpregs_assert_state_consistent+0x23/0x50
>>> [ 190.096723] ? exit_to_user_mode_prepare+0x4b/0x1e0
>>> [ 190.101602] do_syscall_64+0x38/0x90
>>> [ 190.105182] entry_SYSCALL_64_after_hwframe+0x44/0xae
>>> [ 190.110236] RIP: 0033:0x7fd620af589d
>>> [ 190.113815] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c3 f5 0c 00 f7 d8 64 89 01 48
>>> [ 190.132563] RSP: 002b:00007fd6209d6e98 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
>>> [ 190.140126] RAX: ffffffffffffffda RBX: 00007fd620d4bfc0 RCX: 00007fd620af589d
>>> [ 190.147261] RDX: 0000000000000000 RSI: 0000000000000020 RDI: 0000000000000004
>>> [ 190.154391] RBP: 0000000000000020 R08: 0000000000000000 R09: 0000000000000000
>>> [ 190.161524] R10: 0000000000000000 R11: 0000000000000246 R12: 0000561889c472a0
>>> [ 190.168657] R13: 0000000000000020 R14: 0000000000000000 R15: 0000000000000020
>>> [ 190.175793] </TASK>
>>>
>>> It seems this issue is getting triggered with the following commit:
>>>
>>> commit 5b13bc8a3fd519d86e5b1a0b1d1b996cace62f3f
>>> Author: Christoph Hellwig <hch@lst.de>
>>> Date: Wed Nov 24 07:28:56 2021 +0100
>>>
>>> blk-mq: cleanup request allocation
>> Good finding. I am not able to repro these issues after reverting this commit.
>>
>> Vishal, how does it work in your tests?
> Same. I haven't seen any issue in my tests (io_uring, aio) after
> reverting this commit.
> Should I go ahead and send v6 patchset after incororating your prev
> feedback?
Do you have this one:
commit a08ed9aae8a3d2321ef378d6581cc87a3fb75b44
Author: Jens Axboe <axboe@kernel.dk>
Date: Thu Dec 2 12:43:46 2021 -0700
block: fix double bio queue when merging in cached request path
--
Jens Axboe
next prev parent reply other threads:[~2021-12-21 15:59 UTC|newest]
Thread overview: 86+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-01 21:51 [PATCH] md: add support for REQ_NOWAIT Vishal Verma
2021-11-02 3:41 ` Li Feng
2021-11-02 5:01 ` Song Liu
2021-11-02 14:40 ` [PATCH v2] " Vishal Verma
2021-11-02 15:31 ` Jens Axboe
2021-11-02 18:35 ` Song Liu
2021-11-04 4:51 ` [PATCH v3 2/2] md: raid1 add nowait support Vishal Verma
2021-11-04 4:51 ` [PATCH v3 1/2] md: add support for REQ_NOWAIT Vishal Verma
2021-11-06 15:38 ` Guoqing Jiang
2021-11-07 0:16 ` Vishal Verma
2021-11-08 22:17 ` Song Liu
2021-11-08 22:36 ` Vishal Verma
2021-11-06 15:24 ` [PATCH v3 2/2] md: raid1 add nowait support Guoqing Jiang
2021-11-07 0:18 ` Vishal Verma
2021-11-08 22:32 ` Song Liu
2021-11-08 22:39 ` Vishal Verma
2021-11-09 20:59 ` Vishal Verma
2021-11-10 17:02 ` Song Liu
2021-11-10 17:04 ` Vishal Verma
2021-11-10 18:14 ` [RFC PATCH v4 1/4] md: add support for REQ_NOWAIT Vishal Verma
2021-11-10 18:14 ` [RFC PATCH v4 2/4] md: raid1 add nowait support Vishal Verma
2021-11-10 18:14 ` [RFC PATCH v4 3/4] md: raid10 " Vishal Verma
2021-12-14 0:32 ` Song Liu
2021-12-14 15:27 ` Vishal Verma
2021-11-10 18:14 ` [RFC PATCH v4 4/4] md: raid456 " Vishal Verma
2021-11-11 21:42 ` Song Liu
[not found] ` <f8c2a2bc-a885-8254-2b39-fc0c969ac70d@digitalocean.com>
2021-11-19 4:07 ` Song Liu
2021-11-19 4:20 ` Vishal Verma
2021-12-09 16:53 ` Vishal Verma
2021-12-09 16:59 ` Song Liu
2021-12-09 17:01 ` Vishal Verma
2021-12-10 2:16 ` Song Liu
2021-12-10 7:18 ` Song Liu
2021-12-10 18:26 ` Vishal Verma
2021-12-13 5:56 ` Song Liu
2021-12-13 22:43 ` Vishal Verma
2021-12-13 23:35 ` Jens Axboe
[not found] ` <78d5f029-791e-6d3f-4871-263ec6b5c09b@digitalocean.com>
2021-12-14 1:11 ` Song Liu
2021-12-14 1:12 ` Vishal Verma
2021-12-14 15:30 ` Vishal Verma
2021-12-14 17:08 ` Song Liu
2021-12-14 18:09 ` Vishal Verma
2021-12-15 6:09 ` [PATCH v5 1/4] md: add support for REQ_NOWAIT Vishal Verma
2021-12-15 6:09 ` [PATCH v5 2/4] md: raid1 add nowait support Vishal Verma
2021-12-15 20:33 ` Song Liu
2021-12-15 22:20 ` Vishal Verma
2021-12-21 20:06 ` [PATCH v6 1/4] md: add support for REQ_NOWAIT Vishal Verma
2021-12-21 20:06 ` [PATCH v6 2/4] md: raid1 add nowait support Vishal Verma
2021-12-21 20:06 ` [PATCH v6 3/4] md: raid10 " Vishal Verma
2021-12-22 23:58 ` Song Liu
2021-12-23 1:47 ` Song Liu
2021-12-21 20:06 ` [PATCH v6 4/4] md: raid456 " Vishal Verma
2021-12-21 22:02 ` John Stoffel
2021-12-25 2:14 ` Song Liu
[not found] ` <aadc6d52-bc6e-527a-3b9c-0be225f9b727@digitalocean.com>
2021-12-25 22:13 ` Vishal Verma
2021-12-26 0:07 ` Song Liu
2021-12-26 4:02 ` Vishal Verma
2021-12-26 21:20 ` Vishal Verma
2021-12-22 16:06 ` [PATCH v6 1/4] md: add support for REQ_NOWAIT Jens Axboe
2021-12-23 1:22 ` Song Liu
2021-12-23 2:57 ` Song Liu
2021-12-23 3:08 ` Vishal Verma
2022-01-02 0:11 ` Song Liu
2022-01-02 2:08 ` Vishal Verma
2021-12-23 8:36 ` Christoph Hellwig
2021-12-15 6:09 ` [PATCH v5 3/4] md: raid10 add nowait support Vishal Verma
2021-12-15 20:42 ` Song Liu
2021-12-15 22:20 ` Vishal Verma
2021-12-16 0:30 ` Vishal Verma
2021-12-16 16:40 ` Vishal Verma
2021-12-16 16:42 ` Jens Axboe
2021-12-16 16:45 ` Vishal Verma
2021-12-16 18:49 ` Jens Axboe
2021-12-16 19:40 ` Vishal Verma
2021-12-16 20:18 ` Song Liu
2021-12-16 20:37 ` Vishal Verma
2021-12-16 23:50 ` Song Liu
[not found] ` <bd90d6e6-adb4-2696-3110-fad0b1ee00dc@digitalocean.com>
2021-12-21 8:13 ` Song Liu
2021-12-21 15:29 ` Vishal Verma
2021-12-21 15:59 ` Jens Axboe [this message]
2021-12-21 16:26 ` Vishal Verma
2021-12-16 18:14 ` Vishal Verma
2021-12-15 6:09 ` [PATCH v5 4/4] md: raid456 " Vishal Verma
2021-12-15 20:02 ` [PATCH v5 1/4] md: add support for REQ_NOWAIT Song Liu
2021-12-14 0:36 ` [RFC PATCH v4 4/4] md: raid456 add nowait support Song Liu
2021-12-13 23:50 ` [RFC PATCH v4 1/4] md: add support for REQ_NOWAIT Song Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5d718039-52d4-33e7-b7be-c12c3100a1e5@kernel.dk \
--to=axboe@kernel.dk \
--cc=linux-raid@vger.kernel.org \
--cc=rgoldwyn@suse.de \
--cc=song@kernel.org \
--cc=vverma@digitalocean.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).