blktests: block/027 failing consistently

blktests: block/027 failing consistently

[Chaitanya Kulkarni]

Hi,

blktest:block/027 failing consistently on latest linux-block/for-next

HEAD :-

commit dac4337ef0ff38b723e81e2d15593e1de829c4d2 (origin/for-next)
Merge: c506c378c29b 8d283ee62b07
Author: Jens Axboe <axboe@kernel.dk>
Date:   Tue Nov 29 10:53:26 2022 -0700

     Merge branch 'for-6.2/block' into for-next

     * for-6.2/block:
       block: use bool as the return type of elv_iosched_allow_bio_merge
       block: replace "len+name" with "name+len" in elv_iosched_show
       block: always use 'e' when printing scheduler name
       block: replace continue with else-if in elv_iosched_show
       block: include 'none' for initial elv_iosched_show call


dev login: [  350.408229] BUG: kernel NULL pointer dereference, address: 
0000000000000008
[  350.410132] #PF: supervisor read access in kernel mode
[  350.411685] #PF: error_code(0x0000) - not-present page
[  350.412981] PGD 0 P4D 0
[  350.414258] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  350.415380] CPU: 38 PID: 13802 Comm: kworker/38:3 Tainted: G        W 
          6.1.0-rc6blk+ #70
[  350.417541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[  350.420313] Workqueue:  0x0 (cgroup_destroy)
[  350.421378] RIP: 0010:process_one_work+0x31/0x380
[  350.422570] Code: 41 56 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 
08 48 8b 06 4c 8b 67 48 49 89 c5 45 30 ed a8 04 b8 00 00 00 00 4c 0f 44 
e8 <49> 8b 45 08 44 8b b0 00 01 00 00 41 83 e6 20 41 f6 44 24 10 04 75
[  350.427162] RSP: 0018:ffffc900083c7ea0 EFLAGS: 00010046
[  350.428458] RAX: 0000000000000000 RBX: ffff8881054ef5c0 RCX: 
000000010000c6b4
[  350.430773] RDX: 000000010000c6b4 RSI: ffff88814843f468 RDI: 
ffff8881054ef5c0
[  350.432569] RBP: ffff88814843f468 R08: ffff88814843f468 R09: 
0000000000000000
[  350.434361] R10: ffff888100402b80 R11: ffffc900083c7e40 R12: 
ffff888fffbac740
[  350.435850] R13: 0000000000000000 R14: ffffffff810fb300 R15: 
ffff888fffbac740
[  350.437648] FS:  0000000000000000(0000) GS:ffff888fffb80000(0000) 
knlGS:0000000000000000
[  350.442354] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  350.443916] CR2: 0000000000000008 CR3: 000000000280a000 CR4: 
0000000000350ee0
[  350.446339] Call Trace:
[  350.446967]  <TASK>
[  350.447512]  ? rescuer_thread+0x390/0x390
[  350.448515]  worker_thread+0x50/0x3a0
[  350.449452]  ? rescuer_thread+0x390/0x390
[  350.450477]  kthread+0xe7/0x110
[  350.451319]  ? kthread_complete_and_exit+0x20/0x20
[  350.452569]  ret_from_fork+0x22/0x30
[  350.453530]  </TASK>
[  350.454121] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq 
snd_seq_device snd_timer snd soundcore ip6table_mangle ip6table_raw 
ip6table_security iptable_mangle iptable_raw iptable_security ip_set 
nf_tables rfkill nfnetlink ip6table_filter ip6_tables iptable_filter tun 
sunrpc intel_rapl_msr intel_rapl_common xfs kvm_amd ccp kvm ppdev 
parport_pc bfq irqbypass parport joydev pcspkr i2c_piix4 zram ip_tables 
bochs drm_vram_helper drm_kms_helper drm_ttm_helper ttm crct10dif_pclmul 
nvme crc32_pclmul drm crc32c_intel ghash_clmulni_intel virtio_net 
sha512_ssse3 sd_mod nvme_core net_failover failover ata_generic 
nvme_common virtio_blk serio_raw t10_pi qemu_fw_cfg pata_acpi fuse [last 
unloaded: scsi_debug]

Entering kdb (current=0xffff8881038f5080, pid 13802) on processor 38 
Oops: (null)
due to oops @ 0xffffffff810fac21
CPU: 38 PID: 13802 Comm: kworker/38:3 Tainted: G        W 
6.1.0-rc6blk+ #70
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Workqueue:  0x0 (cgroup_destroy)
RIP: 0010:process_one_work+0x31/0x380
Code: 41 56 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 06 4c 
8b 67 48 49 89 c5 45 30 ed a8 04 b8 00 00 00 00 4c 0f 44 e8 <49> 8b 45 
08 44 8b b0 00 01 00 00 41 83 e6 20 41 f6 44 24 10 04 75
RSP: 0018:ffffc900083c7ea0 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff8881054ef5c0 RCX: 000000010000c6b4
RDX: 000000010000c6b4 RSI: ffff88814843f468 RDI: ffff8881054ef5c0
RBP: ffff88814843f468 R08: ffff88814843f468 R09: 0000000000000000
R10: ffff888100402b80 R11: ffffc900083c7e40 R12: ffff888fffbac740
R13: 0000000000000000 R14: ffffffff810fb300 R15: ffff888fffbac740
FS:  0000000000000000(0000) GS:ffff888fffb80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000000280a000 CR4: 0000000000350ee0
Call Trace:
  <TASK>
  ? rescuer_thread+0x390/0x390
  worker_thread+0x50/0x3a0
  ? rescuer_thread+0x390/0x390
  kthread+0xe7/0x110
  ? kthread_complete_and_exit+0x20/0x20
  ret_from_fork+0x22/0x30
  </TASK>

[38]kdb> bt
Stack traceback for pid 13802
0xffff8881038f5080    13802        2  1   38   R  0xffff8881038f7440 
*kworker/38:3
CPU: 38 PID: 13802 Comm: kworker/38:3 Tainted: G        W 
6.1.0-rc6blk+ #70
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Workqueue:  0x0 (cgroup_destroy)
Call Trace:
  <TASK>
  dump_stack_lvl+0x45/0x5e
  kdb_show_stack+0x79/0x90
  kdb_bt1+0xbb/0x130
  kdb_bt+0x34f/0x3b0
  kdb_parse+0x2b7/0x700
  kdb_main_loop+0x4a8/0x990
  ? process_one_work+0x10/0x380
  kdb_stub+0x1ab/0x3f0
  kgdb_cpu_enter+0x331/0x5e0
  kgdb_handle_exception+0xb6/0x100
  __kgdb_notify+0x30/0x90
  kgdb_notify+0x1d/0x40
  notify_die+0x6f/0xb0
  __die_body+0x7d/0xa0
  page_fault_oops+0xae/0x270
  ? do_user_addr_fault+0x65/0x6a0
  exc_page_fault+0x71/0x170
  asm_exc_page_fault+0x22/0x30
RIP: 0010:process_one_work+0x31/0x380
Code: 41 56 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 06 4c 
8b 67 48 49 89 c5 45 30 ed a8 04 b8 00 00 00 00 4c 0f 44 e8 <49> 8b 45 
08 44 8b b0 00 01 00 00 41 83 e6 20 41 f6 44 24 10 04 75
RSP: 0018:ffffc900083c7ea0 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff8881054ef5c0 RCX: 000000010000c6b4
RDX: 000000010000c6b4 RSI: ffff88814843f468 RDI: ffff8881054ef5c0
RBP: ffff88814843f468 R08: ffff88814843f468 R09: 0000000000000000
R10: ffff888100402b80 R11: ffffc900083c7e40 R12: ffff888fffbac740
R13: 0000000000000000 R14: ffffffff810fb300 R15: ffff888fffbac740
  ? rescuer_thread+0x390/0x390
  ? rescuer_thread+0x390/0x390
  worker_thread+0x50/0x3a0
  ? rescuer_thread+0x390/0x390
  kthread+0xe7/0x110
  ? kthread_complete_and_exit+0x20/0x20
  ret_from_fork+0x22/0x30
  </TASK>
[38]kdb>

<4>[  118.772433] ------------[ cut here ]------------
<4>[  118.772438] list_add corruption. prev->next should be next 
(ffff888fff2ac768), but was ffff888fff2b3468. (prev=ffff88817b40cc70).
<4>[  118.772458] WARNING: CPU: 2 PID: 864 at lib/list_debug.c:30 
__list_add_valid+0x89/0xb0
more>
<4>[  118.772527] CPU: 2 PID: 864 Comm: kworker/2:2 Not tainted 
6.1.0-rc6blk+ #70
<4>[  118.772531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
<4>[  118.772534] Workqueue:  0x0 (cgwb_release)
<4>[  118.772539] RIP: 0010:__list_add_valid+0x89/0xb0

<4>[  118.772545] RSP: 0018:ffffc90000e83e70 EFLAGS: 00010082
<4>[  118.772548] RAX: 0000000000000000 RBX: ffff888105cb1ae0 RCX: 
0000000000000000
<4>[  118.772550] RDX: 0000000000000002 RSI: ffffffff825c7b01 RDI: 
00000000ffffffff
<4>[  118.772552] RBP: ffff888105cb1ae8 R08: ffff88903ff403a8 R09: 
00000000ffffbfff
<4>[  118.772553] R10: ffff888fff0a0000 R11: ffff88903fee03c0 R12: 
ffff888fff2ac768
<4>[  118.772555] R13: ffff888fff2b3e60 R14: 0000000000000000 R15: 
ffff88817b40cc70
<4>[  118.772559] FS:  0000000000000000(0000) GS:ffff888fff280000(0000) 
knlGS:0000000000000000
<4>[  118.772561] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[  118.772563] CR2: 00007f469e848224 CR3: 000000000280a000 CR4: 
0000000000350ee0
<4>[  118.772567] Call Trace:
<4>[  118.772581]  <TASK>
<4>[  118.772582]  move_linked_works+0x70/0xa0
<4>[  118.772589]  ? rescuer_thread+0x390/0x390
<4>[  118.772592]  pwq_activate_inactive_work+0x39/0xa0
<4>[  118.772595]  pwq_dec_nr_in_flight+0x65/0xc0
<4>[  118.772598]  worker_thread+0x50/0x3a0
<4>[  118.772602]  ? rescuer_thread+0x390/0x390
<4>[  118.772605]  kthread+0xe7/0x110
<4>[  118.772608]  ? kthread_complete_and_exit+0x20/0x20
<4>[  118.772611]  ret_from_fork+0x22/0x30
<4>[  118.772619]  </TASK>
<4>[  118.772620] ---[ end trace 0000000000000000 ]---
<1>[  118.772629] BUG: kernel NULL pointer dereference, address: 
0000000000000008
<1>[  118.774401] #PF: supervisor read access in kernel mode
<1>[  118.775688] #PF: error_code(0x0000) - not-present page
<6>[  118.777504] PGD 0 P4D 0
<4>[  118.778124] Oops: 0000 [#1] PREEMPT SMP NOPTI
<4>[  118.779227] CPU: 2 PID: 864 Comm: kworker/2:2 Tainted: G        W 
         6.1.0-rc6blk+ #70
<4>[  118.781326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
<4>[  118.784171] Workqueue:  0x0 (cgroup_destroy)
<4>[  118.785424] RIP: 0010:process_one_work+0x31/0x380

<4>[  118.790983] RSP: 0018:ffffc90000e83ea0 EFLAGS: 00010046
<4>[  118.792230] RAX: 0000000000000000 RBX: ffff888112d6a300 RCX: 
00000000fffd3aeb
<4>[  118.794563] RDX: 00000000fffd3aeb RSI: ffff88817b40cc68 RDI: 
ffff888112d6a300
<4>[  118.796447] RBP: ffff88817b40cc68 R08: ffff88817b40cc68 R09: 
0000000000000000
<4>[  118.798264] R10: ffff888100400028 R11: ffffc90000e83e40 R12: 
ffff888fff2ac740
<4>[  118.800002] R13: 0000000000000000 R14: ffffffff810fb300 R15: 
ffff888fff2ac740
<4>[  118.801743] FS:  0000000000000000(0000) GS:ffff888fff280000(0000) 
knlGS:0000000000000000
<4>[  118.803940] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[  118.806042] CR2: 0000000000000008 CR3: 000000000280a000 CR4: 
0000000000350ee0
<4>[  118.807975] Call Trace:
<4>[  118.808615]  <TASK>
<4>[  118.809657]  ? rescuer_thread+0x390/0x390
<4>[  118.810770]  worker_thread+0x50/0x3a0
<4>[  118.811725]  ? rescuer_thread+0x390/0x390
<4>[  118.812567]  kthread+0xe7/0x110
<4>[  118.813423]  ? kthread_complete_and_exit+0x20/0x20
<4>[  118.814760]  ret_from_fork+0x22/0x30
<4>[  118.815739]  </TASK>

[2]kdb>


-ck

Re: blktests: block/027 failing consistently

[Jens Axboe]

On 11/29/22 11:49 AM, Chaitanya Kulkarni wrote:
> Hi,
> 
> blktest:block/027 failing consistently on latest linux-block/for-next

https://lore.kernel.org/linux-block/20221128033057.1279383-1-longman@redhat.com/

been a few discussions on it. Waiman, how's the fix looking? We need to
get this sorted asap, or I'll just revert the offending patch.

-- 
Jens Axboe

Re: blktests: block/027 failing consistently

[Jens Axboe]

On 11/29/22 12:02 PM, Jens Axboe wrote:
> On 11/29/22 11:49 AM, Chaitanya Kulkarni wrote:
>> Hi,
>>
>> blktest:block/027 failing consistently on latest linux-block/for-next
> 
> https://lore.kernel.org/linux-block/20221128033057.1279383-1-longman@redhat.com/
> 
> been a few discussions on it. Waiman, how's the fix looking? We need to
> get this sorted asap, or I'll just revert the offending patch.

Let's actually make sure Waiman is CC'ed...

-- 
Jens Axboe

Re: blktests: block/027 failing consistently

[Bart Van Assche]

On 11/29/22 10:49, Chaitanya Kulkarni wrote:
> blktest:block/027 failing consistently on latest linux-block/for-next

Hi Chaitanya,

Is this report perhaps a duplicate of 
https://lore.kernel.org/linux-block/69af7ccb-6901-c84c-0e95-5682ccfb750c@acm.org/ 
?

Thanks,

Bart.

Re: blktests: block/027 failing consistently

[Chaitanya Kulkarni]

On 11/29/22 11:13, Bart Van Assche wrote:
> On 11/29/22 10:49, Chaitanya Kulkarni wrote:
>> blktest:block/027 failing consistently on latest linux-block/for-next
> 
> Hi Chaitanya,
> 
> Is this report perhaps a duplicate of 
> https://lore.kernel.org/linux-block/69af7ccb-6901-c84c-0e95-5682ccfb750c@acm.org/ 
> ?
> 
> Thanks,
> 
> Bart.

Indeed as Jens pointed out.

Waiman please run the blktests block/027 when you fix it so we all
know...

-ck

Re: blktests: block/027 failing consistently

[Waiman Long]

On 11/29/22 14:21, Chaitanya Kulkarni wrote:
> On 11/29/22 11:13, Bart Van Assche wrote:
>> On 11/29/22 10:49, Chaitanya Kulkarni wrote:
>>> blktest:block/027 failing consistently on latest linux-block/for-next
>> Hi Chaitanya,
>>
>> Is this report perhaps a duplicate of
>> https://lore.kernel.org/linux-block/69af7ccb-6901-c84c-0e95-5682ccfb750c@acm.org/
>> ?
>>
>> Thanks,
>>
>> Bart.
> Indeed as Jens pointed out.
>
> Waiman please run the blktests block/027 when you fix it so we all
> know...
>
> -ck

Sure. I will the run the test after applying my fix.

Cheers,
Longman