* Bug report - issue with bfq?
@ 2018-01-03 3:58 Guoqing Jiang
2018-01-03 7:44 ` Paolo Valente
0 siblings, 1 reply; 4+ messages in thread
From: Guoqing Jiang @ 2018-01-03 3:58 UTC (permalink / raw)
To: linux-block; +Cc: Paolo Valente
Hi,
In my test, I found some issues when try bfq with xfs.
The test basically just set the disk's scheduler to bfq,
create xfs on top of it, mount fs and write something,
then umount the fs. After several rounds of iteration,
I can see different calltraces appeared.
For example, the one which happened frequently:
Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 Filesystem
Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting Filesystem
Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 Filesystem
Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
Jan 03 11:35:19 linux-mainline kernel: BUG: unable to handle kernel
paging request at 0000000000029ec0
Jan 03 11:35:19 linux-mainline kernel: IP: __mod_node_page_state+0x5/0x50
Jan 03 11:35:19 linux-mainline kernel: PGD 0 P4D 0
Jan 03 11:35:19 linux-mainline kernel: Oops: 0000 [#1] SMP KASAN
Jan 03 11:35:19 linux-mainline kernel: Modules linked in: bfq(E)
joydev(E) uinput(E) fuse(E) af_packet(E) iscsi_ibft(E)
iscsi_boot_sysfs(E) snd_hda_codec_generic(E) crct10dif_pclmul(E)
crc32_pclmul(E) xfs(E) ghash_clmulni_intel(E) libcrc32c(E)
crc32c_intel(E) pcbc(E) snd_hda_intel(E) snd_hda_codec(E)
snd_hda_core(E) snd_hwdep(E) snd_pcm(E) ppdev(E) aesni_intel(E)
snd_timer(E) aes_x86_64(E) crypto_simd(E) snd(E) glue_helper(E)
cryptd(E) pcspkr(E) virtio_balloon(E) virtio_net(E) parport_pc(E)
parport(E) soundcore(E) i2c_piix4(E) ext4(E) crc16(E) mbcache(E) jbd2(E)
virtio_console(E) virtio_rng(E) virtio_blk(E) ata_generic(E) ata_piix(E)
ahci(E) libahci(E) floppy(E) ehci_pci(E) qxl(E) serio_raw(E)
drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E)
fb_sys_fops(E) sym53c8xx(E) scsi_transport_spi(E) button(E) libata(E)
Jan 03 11:35:19 linux-mainline kernel: ttm(E) drm(E) uhci_hcd(E)
ehci_hcd(E) usbcore(E) virtio_pci(E) virtio_ring(E) virtio(E) sg(E)
dm_multipath(E) dm_mod(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E)
scsi_mod(E) autofs4(E)
Jan 03 11:35:19 linux-mainline kernel: CPU: 0 PID: 3349 Comm: ps
Tainted: G E 4.15.0-rc1-69-default #1
Jan 03 11:35:19 linux-mainline kernel: Hardware name: QEMU Standard PC
(i440FX + PIIX, 1996), BIOS
rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
Jan 03 11:35:19 linux-mainline kernel: task: ffff880061efce80
task.stack: ffff880058bd0000
Jan 03 11:35:19 linux-mainline kernel: RIP:
0010:__mod_node_page_state+0x5/0x50
Jan 03 11:35:19 linux-mainline kernel: RSP: 0018:ffff880058bd7ce8
EFLAGS: 00010a07
Jan 03 11:35:19 linux-mainline kernel: RAX: 00000000000003ff RBX:
ffffea00011a3d80 RCX: 00000000011a3d80
Jan 03 11:35:19 linux-mainline kernel: RDX: ffffffffffffffff RSI:
000000000000000d RDI: 0000000000000000
Jan 03 11:35:19 linux-mainline kernel: RBP: ffffffffffffffff R08:
ffff88006378a630 R09: ffff880058bd7d98
Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11:
0000000000000000 R12: ffffea00011a3d80
Jan 03 11:35:19 linux-mainline kernel: R13: 00007f7f4f318000 R14:
00007f7f4f31c000 R15: ffff880058bd7e18
Jan 03 11:35:19 linux-mainline kernel: FS: 0000000000000000(0000)
GS:ffff880066c00000(0000) knlGS:0000000000000000
Jan 03 11:35:19 linux-mainline kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0 CR3:
0000000001c0d006 CR4: 00000000001606f0
Jan 03 11:35:19 linux-mainline kernel: Call Trace:
Jan 03 11:35:19 linux-mainline kernel: page_remove_rmap+0x11a/0x2b0
Jan 03 11:35:19 linux-mainline kernel: unmap_page_range+0x547/0xa30
Jan 03 11:35:19 linux-mainline kernel: unmap_vmas+0x42/0x90
Jan 03 11:35:19 linux-mainline kernel: exit_mmap+0x86/0x180
Jan 03 11:35:19 linux-mainline kernel: mmput+0x4a/0x110
Jan 03 11:35:19 linux-mainline kernel: do_exit+0x25d/0xae0
Jan 03 11:35:19 linux-mainline kernel: do_group_exit+0x39/0xa0
Jan 03 11:35:19 linux-mainline kernel: SyS_exit_group+0x10/0x10
Jan 03 11:35:19 linux-mainline kernel: entry_SYSCALL_64_fastpath+0x1a/0x7d
Jan 03 11:35:19 linux-mainline kernel: RIP: 0033:0x7f7f4eb8c338
Jan 03 11:35:19 linux-mainline kernel: RSP: 002b:00007ffca4400d48
EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Jan 03 11:35:19 linux-mainline kernel: RAX: ffffffffffffffda RBX:
0000000000000000 RCX: 00007f7f4eb8c338
Jan 03 11:35:19 linux-mainline kernel: RDX: 0000000000000000 RSI:
000000000000003c RDI: 0000000000000000
Jan 03 11:35:19 linux-mainline kernel: RBP: 00007ffca4400d40 R08:
00000000000000e7 R09: fffffffffffffef8
Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11:
0000000000000246 R12: 00007f7f4f756000
Jan 03 11:35:19 linux-mainline kernel: R13: 00007ffca4400cc8 R14:
00007f7f4f732b20 R15: 00007f7f4d5ebc70
Jan 03 11:35:19 linux-mainline kernel: Code: f7 d9 48 39 ca 7c 05 65 88
50 0f c3 f0 48 01 94 f7 00 05 00 00 f0 48 01 14 f5 c0 c4 c0 81 31 d2 eb
e5 0f 1f 40 00 0f 1f 44 00 00 <48> 8b 8f c0 9e 02 00 89 f6 48 8d 04 31
65 44 8a 40 01 4d 0f be
Jan 03 11:35:19 linux-mainline kernel: RIP:
__mod_node_page_state+0x5/0x50 RSP: ffff880058bd7ce8
Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0
Jan 03 11:35:19 linux-mainline kernel: ---[ end trace b5314eeef943a473 ]---
Jan 03 11:35:19 linux-mainline kernel: Fixing recursive fault but reboot
is needed!
Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting Filesystem
Occasionally mount process hangs forever.
linux-mainline:~ # cat /proc/19627/stack
[<ffffffff810a65f2>] io_schedule+0x12/0x40
[<ffffffff8119fbb7>] wait_on_page_bit+0xd7/0x100
[<ffffffff811b3713>] truncate_inode_pages_range+0x423/0x7c0
[<ffffffff81273768>] set_blocksize+0x98/0xb0
[<ffffffff81273798>] sb_set_blocksize+0x18/0x40
[<ffffffffa06a2e58>] xfs_fs_fill_super+0x1b8/0x590 [xfs]
[<ffffffff8123bd4d>] mount_bdev+0x17d/0x1b0
[<ffffffff8123c6d4>] mount_fs+0x34/0x150
[<ffffffff81259702>] vfs_kern_mount+0x62/0x110
[<ffffffff8125bd1a>] do_mount+0x1ca/0xc30
[<ffffffff8125ca6e>] SyS_mount+0x7e/0xd0
[<ffffffff8172fff3>] entry_SYSCALL_64_fastpath+0x1a/0x7d
[<ffffffffffffffff>] 0xffffffffffffffff
The test against bfq+ext4 have issues as well. However,
if I change the scheduler to mq-deadline or kyber, then
I can't see any calltrace, so I guess it is related with
bfq.
Thanks,
Guoqing
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Bug report - issue with bfq?
2018-01-03 3:58 Bug report - issue with bfq? Guoqing Jiang
@ 2018-01-03 7:44 ` Paolo Valente
2018-01-03 9:06 ` Guoqing Jiang
0 siblings, 1 reply; 4+ messages in thread
From: Paolo Valente @ 2018-01-03 7:44 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-block, DAVIDE FERRARI
> Il giorno 03 gen 2018, alle ore 04:58, Guoqing Jiang =
<gqjiang@suse.com> ha scritto:
>=20
> Hi,
>=20
Hi
> In my test, I found some issues when try bfq with xfs.
> The test basically just set the disk's scheduler to bfq,
> create xfs on top of it, mount fs and write something,
> then umount the fs. After several rounds of iteration,
> I can see different calltraces appeared.
>=20
> For example, the one which happened frequently:
>=20
> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 =
Filesystem
> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting =
Filesystem
> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 =
Filesystem
> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
> Jan 03 11:35:19 linux-mainline kernel: BUG: unable to handle kernel =
paging request at 0000000000029ec0
> Jan 03 11:35:19 linux-mainline kernel: IP: =
__mod_node_page_state+0x5/0x50
> Jan 03 11:35:19 linux-mainline kernel: PGD 0 P4D 0
> Jan 03 11:35:19 linux-mainline kernel: Oops: 0000 [#1] SMP KASAN
> Jan 03 11:35:19 linux-mainline kernel: Modules linked in: bfq(E) =
joydev(E) uinput(E) fuse(E) af_packet(E) iscsi_ibft(E) =
iscsi_boot_sysfs(E) snd_hda_codec_generic(E) crct10dif_pclmul(E) =
crc32_pclmul(E) xfs(E) ghash_clmulni_intel(E) libcrc32c(E) =
crc32c_intel(E) pcbc(E) snd_hda_intel(E) snd_hda_codec(E) =
snd_hda_core(E) snd_hwdep(E) snd_pcm(E) ppdev(E) aesni_intel(E) =
snd_timer(E) aes_x86_64(E) crypto_simd(E) snd(E) glue_helper(E) =
cryptd(E) pcspkr(E) virtio_balloon(E) virtio_net(E) parport_pc(E) =
parport(E) soundcore(E) i2c_piix4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) =
virtio_console(E) virtio_rng(E) virtio_blk(E) ata_generic(E) ata_piix(E) =
ahci(E) libahci(E) floppy(E) ehci_pci(E) qxl(E) serio_raw(E) =
drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) =
fb_sys_fops(E) sym53c8xx(E) scsi_transport_spi(E) button(E) libata(E)
> Jan 03 11:35:19 linux-mainline kernel: ttm(E) drm(E) uhci_hcd(E) =
ehci_hcd(E) usbcore(E) virtio_pci(E) virtio_ring(E) virtio(E) sg(E) =
dm_multipath(E) dm_mod(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) =
scsi_mod(E) autofs4(E)
> Jan 03 11:35:19 linux-mainline kernel: CPU: 0 PID: 3349 Comm: ps =
Tainted: G E 4.15.0-rc1-69-default #1
> Jan 03 11:35:19 linux-mainline kernel: Hardware name: QEMU Standard PC =
(i440FX + PIIX, 1996), BIOS =
rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
> Jan 03 11:35:19 linux-mainline kernel: task: ffff880061efce80 =
task.stack: ffff880058bd0000
> Jan 03 11:35:19 linux-mainline kernel: RIP: =
0010:__mod_node_page_state+0x5/0x50
> Jan 03 11:35:19 linux-mainline kernel: RSP: 0018:ffff880058bd7ce8 =
EFLAGS: 00010a07
> Jan 03 11:35:19 linux-mainline kernel: RAX: 00000000000003ff RBX: =
ffffea00011a3d80 RCX: 00000000011a3d80
> Jan 03 11:35:19 linux-mainline kernel: RDX: ffffffffffffffff RSI: =
000000000000000d RDI: 0000000000000000
> Jan 03 11:35:19 linux-mainline kernel: RBP: ffffffffffffffff R08: =
ffff88006378a630 R09: ffff880058bd7d98
> Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11: =
0000000000000000 R12: ffffea00011a3d80
> Jan 03 11:35:19 linux-mainline kernel: R13: 00007f7f4f318000 R14: =
00007f7f4f31c000 R15: ffff880058bd7e18
> Jan 03 11:35:19 linux-mainline kernel: FS: 0000000000000000(0000) =
GS:ffff880066c00000(0000) knlGS:0000000000000000
> Jan 03 11:35:19 linux-mainline kernel: CS: 0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
> Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0 CR3: =
0000000001c0d006 CR4: 00000000001606f0
> Jan 03 11:35:19 linux-mainline kernel: Call Trace:
> Jan 03 11:35:19 linux-mainline kernel: page_remove_rmap+0x11a/0x2b0
> Jan 03 11:35:19 linux-mainline kernel: unmap_page_range+0x547/0xa30
> Jan 03 11:35:19 linux-mainline kernel: unmap_vmas+0x42/0x90
> Jan 03 11:35:19 linux-mainline kernel: exit_mmap+0x86/0x180
> Jan 03 11:35:19 linux-mainline kernel: mmput+0x4a/0x110
> Jan 03 11:35:19 linux-mainline kernel: do_exit+0x25d/0xae0
> Jan 03 11:35:19 linux-mainline kernel: do_group_exit+0x39/0xa0
> Jan 03 11:35:19 linux-mainline kernel: SyS_exit_group+0x10/0x10
> Jan 03 11:35:19 linux-mainline kernel: =
entry_SYSCALL_64_fastpath+0x1a/0x7d
> Jan 03 11:35:19 linux-mainline kernel: RIP: 0033:0x7f7f4eb8c338
> Jan 03 11:35:19 linux-mainline kernel: RSP: 002b:00007ffca4400d48 =
EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> Jan 03 11:35:19 linux-mainline kernel: RAX: ffffffffffffffda RBX: =
0000000000000000 RCX: 00007f7f4eb8c338
> Jan 03 11:35:19 linux-mainline kernel: RDX: 0000000000000000 RSI: =
000000000000003c RDI: 0000000000000000
> Jan 03 11:35:19 linux-mainline kernel: RBP: 00007ffca4400d40 R08: =
00000000000000e7 R09: fffffffffffffef8
> Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11: =
0000000000000246 R12: 00007f7f4f756000
> Jan 03 11:35:19 linux-mainline kernel: R13: 00007ffca4400cc8 R14: =
00007f7f4f732b20 R15: 00007f7f4d5ebc70
> Jan 03 11:35:19 linux-mainline kernel: Code: f7 d9 48 39 ca 7c 05 65 =
88 50 0f c3 f0 48 01 94 f7 00 05 00 00 f0 48 01 14 f5 c0 c4 c0 81 31 d2 =
eb e5 0f 1f 40 00 0f 1f 44 00 00 <48> 8b 8f c0 9e 02 00 89 f6 48 8d 04 =
31 65 44 8a 40 01 4d 0f be
> Jan 03 11:35:19 linux-mainline kernel: RIP: =
__mod_node_page_state+0x5/0x50 RSP: ffff880058bd7ce8
> Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0
> Jan 03 11:35:19 linux-mainline kernel: ---[ end trace b5314eeef943a473 =
]---
> Jan 03 11:35:19 linux-mainline kernel: Fixing recursive fault but =
reboot is needed!
> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting =
Filesystem
>=20
>=20
Yes, this call trace may be related with bfq, because it concerns
(non-bfq) code that may get executed only when bfq is set as I/O
scheduler. In fact, the failure happens on an exit_group, and bfq
supports cgroups, while either mq-deadline or kyber don't.
Did you also check what __mod_node_page_state+0x5 corresponds to in
your sources? Maybe this piece of information could ring some bell, at
least for people more expert than me on the involved code.
> Occasionally mount process hangs forever.
> linux-mainline:~ # cat /proc/19627/stack
> [<ffffffff810a65f2>] io_schedule+0x12/0x40
> [<ffffffff8119fbb7>] wait_on_page_bit+0xd7/0x100
> [<ffffffff811b3713>] truncate_inode_pages_range+0x423/0x7c0
> [<ffffffff81273768>] set_blocksize+0x98/0xb0
> [<ffffffff81273798>] sb_set_blocksize+0x18/0x40
> [<ffffffffa06a2e58>] xfs_fs_fill_super+0x1b8/0x590 [xfs]
> [<ffffffff8123bd4d>] mount_bdev+0x17d/0x1b0
> [<ffffffff8123c6d4>] mount_fs+0x34/0x150
> [<ffffffff81259702>] vfs_kern_mount+0x62/0x110
> [<ffffffff8125bd1a>] do_mount+0x1ca/0xc30
> [<ffffffff8125ca6e>] SyS_mount+0x7e/0xd0
> [<ffffffff8172fff3>] entry_SYSCALL_64_fastpath+0x1a/0x7d
> [<ffffffffffffffff>] 0xffffffffffffffff
>=20
>=20
Maybe this hang has to do with the one already recently reported for
USB drives. We have already found the cause of that one, and are
finalizing our fix.
Thanks,
Paolo
> The test against bfq+ext4 have issues as well. However,
> if I change the scheduler to mq-deadline or kyber, then
> I can't see any calltrace, so I guess it is related with
> bfq.
>=20
>=20
> Thanks,
> Guoqing
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Bug report - issue with bfq?
2018-01-03 7:44 ` Paolo Valente
@ 2018-01-03 9:06 ` Guoqing Jiang
2018-01-03 18:10 ` Paolo Valente
0 siblings, 1 reply; 4+ messages in thread
From: Guoqing Jiang @ 2018-01-03 9:06 UTC (permalink / raw)
To: Paolo Valente; +Cc: linux-block, DAVIDE FERRARI
On 01/03/2018 03:44 PM, Paolo Valente wrote:
>
>> Il giorno 03 gen 2018, alle ore 04:58, Guoqing Jiang <gqjiang@suse.com> ha scritto:
>>
>> Hi,
>>
> Hi
>
>> In my test, I found some issues when try bfq with xfs.
>> The test basically just set the disk's scheduler to bfq,
>> create xfs on top of it, mount fs and write something,
>> then umount the fs. After several rounds of iteration,
>> I can see different calltraces appeared.
>>
>> For example, the one which happened frequently:
>>
>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 Filesystem
>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting Filesystem
>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 Filesystem
>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
>> Jan 03 11:35:19 linux-mainline kernel: BUG: unable to handle kernel paging request at 0000000000029ec0
>> Jan 03 11:35:19 linux-mainline kernel: IP: __mod_node_page_state+0x5/0x50
>> Jan 03 11:35:19 linux-mainline kernel: PGD 0 P4D 0
>> Jan 03 11:35:19 linux-mainline kernel: Oops: 0000 [#1] SMP KASAN
>> Jan 03 11:35:19 linux-mainline kernel: Modules linked in: bfq(E) joydev(E) uinput(E) fuse(E) af_packet(E) iscsi_ibft(E) iscsi_boot_sysfs(E) snd_hda_codec_generic(E) crct10dif_pclmul(E) crc32_pclmul(E) xfs(E) ghash_clmulni_intel(E) libcrc32c(E) crc32c_intel(E) pcbc(E) snd_hda_intel(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) snd_pcm(E) ppdev(E) aesni_intel(E) snd_timer(E) aes_x86_64(E) crypto_simd(E) snd(E) glue_helper(E) cryptd(E) pcspkr(E) virtio_balloon(E) virtio_net(E) parport_pc(E) parport(E) soundcore(E) i2c_piix4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) virtio_console(E) virtio_rng(E) virtio_blk(E) ata_generic(E) ata_piix(E) ahci(E) libahci(E) floppy(E) ehci_pci(E) qxl(E) serio_raw(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) sym53c8xx(E) scsi_transport_spi(E) button(E) libata(E)
>> Jan 03 11:35:19 linux-mainline kernel: ttm(E) drm(E) uhci_hcd(E) ehci_hcd(E) usbcore(E) virtio_pci(E) virtio_ring(E) virtio(E) sg(E) dm_multipath(E) dm_mod(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) scsi_mod(E) autofs4(E)
>> Jan 03 11:35:19 linux-mainline kernel: CPU: 0 PID: 3349 Comm: ps Tainted: G E 4.15.0-rc1-69-default #1
>> Jan 03 11:35:19 linux-mainline kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
>> Jan 03 11:35:19 linux-mainline kernel: task: ffff880061efce80 task.stack: ffff880058bd0000
>> Jan 03 11:35:19 linux-mainline kernel: RIP: 0010:__mod_node_page_state+0x5/0x50
>> Jan 03 11:35:19 linux-mainline kernel: RSP: 0018:ffff880058bd7ce8 EFLAGS: 00010a07
>> Jan 03 11:35:19 linux-mainline kernel: RAX: 00000000000003ff RBX: ffffea00011a3d80 RCX: 00000000011a3d80
>> Jan 03 11:35:19 linux-mainline kernel: RDX: ffffffffffffffff RSI: 000000000000000d RDI: 0000000000000000
>> Jan 03 11:35:19 linux-mainline kernel: RBP: ffffffffffffffff R08: ffff88006378a630 R09: ffff880058bd7d98
>> Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11: 0000000000000000 R12: ffffea00011a3d80
>> Jan 03 11:35:19 linux-mainline kernel: R13: 00007f7f4f318000 R14: 00007f7f4f31c000 R15: ffff880058bd7e18
>> Jan 03 11:35:19 linux-mainline kernel: FS: 0000000000000000(0000) GS:ffff880066c00000(0000) knlGS:0000000000000000
>> Jan 03 11:35:19 linux-mainline kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0 CR3: 0000000001c0d006 CR4: 00000000001606f0
>> Jan 03 11:35:19 linux-mainline kernel: Call Trace:
>> Jan 03 11:35:19 linux-mainline kernel: page_remove_rmap+0x11a/0x2b0
>> Jan 03 11:35:19 linux-mainline kernel: unmap_page_range+0x547/0xa30
>> Jan 03 11:35:19 linux-mainline kernel: unmap_vmas+0x42/0x90
>> Jan 03 11:35:19 linux-mainline kernel: exit_mmap+0x86/0x180
>> Jan 03 11:35:19 linux-mainline kernel: mmput+0x4a/0x110
>> Jan 03 11:35:19 linux-mainline kernel: do_exit+0x25d/0xae0
>> Jan 03 11:35:19 linux-mainline kernel: do_group_exit+0x39/0xa0
>> Jan 03 11:35:19 linux-mainline kernel: SyS_exit_group+0x10/0x10
>> Jan 03 11:35:19 linux-mainline kernel: entry_SYSCALL_64_fastpath+0x1a/0x7d
>> Jan 03 11:35:19 linux-mainline kernel: RIP: 0033:0x7f7f4eb8c338
>> Jan 03 11:35:19 linux-mainline kernel: RSP: 002b:00007ffca4400d48 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
>> Jan 03 11:35:19 linux-mainline kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7f4eb8c338
>> Jan 03 11:35:19 linux-mainline kernel: RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
>> Jan 03 11:35:19 linux-mainline kernel: RBP: 00007ffca4400d40 R08: 00000000000000e7 R09: fffffffffffffef8
>> Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11: 0000000000000246 R12: 00007f7f4f756000
>> Jan 03 11:35:19 linux-mainline kernel: R13: 00007ffca4400cc8 R14: 00007f7f4f732b20 R15: 00007f7f4d5ebc70
>> Jan 03 11:35:19 linux-mainline kernel: Code: f7 d9 48 39 ca 7c 05 65 88 50 0f c3 f0 48 01 94 f7 00 05 00 00 f0 48 01 14 f5 c0 c4 c0 81 31 d2 eb e5 0f 1f 40 00 0f 1f 44 00 00 <48> 8b 8f c0 9e 02 00 89 f6 48 8d 04 31 65 44 8a 40 01 4d 0f be
>> Jan 03 11:35:19 linux-mainline kernel: RIP: __mod_node_page_state+0x5/0x50 RSP: ffff880058bd7ce8
>> Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0
>> Jan 03 11:35:19 linux-mainline kernel: ---[ end trace b5314eeef943a473 ]---
>> Jan 03 11:35:19 linux-mainline kernel: Fixing recursive fault but reboot is needed!
>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting Filesystem
>>
>>
> Yes, this call trace may be related with bfq, because it concerns
> (non-bfq) code that may get executed only when bfq is set as I/O
> scheduler. In fact, the failure happens on an exit_group, and bfq
> supports cgroups, while either mq-deadline or kyber don't.
>
> Did you also check what __mod_node_page_state+0x5 corresponds to in
> your sources? Maybe this piece of information could ring some bell, at
> least for people more expert than me on the involved code.
Seems it is a null dereferenced since RDI is 0000000000000000.
(gdb) l *__mod_node_page_state+0x5
0x385 is in __mod_node_page_state
(/usr/src/kernels/4.15.0-rc1-69-default/mm/vmstat.c:338).
333 EXPORT_SYMBOL(__mod_zone_page_state);
334
335 void __mod_node_page_state(struct pglist_data *pgdat, enum
node_stat_item item,
336 long delta)
337 {
338 struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
339 s8 __percpu *p = pcp->vm_node_stat_diff + item;
340 long x;
341 long t;
342
Let me know if you need anything else.
>> Occasionally mount process hangs forever.
>> linux-mainline:~ # cat /proc/19627/stack
>> [<ffffffff810a65f2>] io_schedule+0x12/0x40
>> [<ffffffff8119fbb7>] wait_on_page_bit+0xd7/0x100
>> [<ffffffff811b3713>] truncate_inode_pages_range+0x423/0x7c0
>> [<ffffffff81273768>] set_blocksize+0x98/0xb0
>> [<ffffffff81273798>] sb_set_blocksize+0x18/0x40
>> [<ffffffffa06a2e58>] xfs_fs_fill_super+0x1b8/0x590 [xfs]
>> [<ffffffff8123bd4d>] mount_bdev+0x17d/0x1b0
>> [<ffffffff8123c6d4>] mount_fs+0x34/0x150
>> [<ffffffff81259702>] vfs_kern_mount+0x62/0x110
>> [<ffffffff8125bd1a>] do_mount+0x1ca/0xc30
>> [<ffffffff8125ca6e>] SyS_mount+0x7e/0xd0
>> [<ffffffff8172fff3>] entry_SYSCALL_64_fastpath+0x1a/0x7d
>> [<ffffffffffffffff>] 0xffffffffffffffff
> Maybe this hang has to do with the one already recently reported for
> USB drives. We have already found the cause of that one, and are
> finalizing our fix.
I am not sure it is the same one since the test is run against virtio disk
not usb, but anyway I can try the fix.
Thanks,
Guoqing
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Bug report - issue with bfq?
2018-01-03 9:06 ` Guoqing Jiang
@ 2018-01-03 18:10 ` Paolo Valente
0 siblings, 0 replies; 4+ messages in thread
From: Paolo Valente @ 2018-01-03 18:10 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-block, DAVIDE FERRARI
> Il giorno 03 gen 2018, alle ore 10:06, Guoqing Jiang =
<gqjiang@suse.com> ha scritto:
>=20
>=20
>=20
> On 01/03/2018 03:44 PM, Paolo Valente wrote:
>>=20
>>> Il giorno 03 gen 2018, alle ore 04:58, Guoqing Jiang =
<gqjiang@suse.com> ha scritto:
>>>=20
>>> Hi,
>>>=20
>> Hi
>>=20
>>> In my test, I found some issues when try bfq with xfs.
>>> The test basically just set the disk's scheduler to bfq,
>>> create xfs on top of it, mount fs and write something,
>>> then umount the fs. After several rounds of iteration,
>>> I can see different calltraces appeared.
>>>=20
>>> For example, the one which happened frequently:
>>>=20
>>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 =
Filesystem
>>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
>>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting =
Filesystem
>>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Mounting V5 =
Filesystem
>>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Ending clean mount
>>> Jan 03 11:35:19 linux-mainline kernel: BUG: unable to handle kernel =
paging request at 0000000000029ec0
>>> Jan 03 11:35:19 linux-mainline kernel: IP: =
__mod_node_page_state+0x5/0x50
>>> Jan 03 11:35:19 linux-mainline kernel: PGD 0 P4D 0
>>> Jan 03 11:35:19 linux-mainline kernel: Oops: 0000 [#1] SMP KASAN
>>> Jan 03 11:35:19 linux-mainline kernel: Modules linked in: bfq(E) =
joydev(E) uinput(E) fuse(E) af_packet(E) iscsi_ibft(E) =
iscsi_boot_sysfs(E) snd_hda_codec_generic(E) crct10dif_pclmul(E) =
crc32_pclmul(E) xfs(E) ghash_clmulni_intel(E) libcrc32c(E) =
crc32c_intel(E) pcbc(E) snd_hda_intel(E) snd_hda_codec(E) =
snd_hda_core(E) snd_hwdep(E) snd_pcm(E) ppdev(E) aesni_intel(E) =
snd_timer(E) aes_x86_64(E) crypto_simd(E) snd(E) glue_helper(E) =
cryptd(E) pcspkr(E) virtio_balloon(E) virtio_net(E) parport_pc(E) =
parport(E) soundcore(E) i2c_piix4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) =
virtio_console(E) virtio_rng(E) virtio_blk(E) ata_generic(E) ata_piix(E) =
ahci(E) libahci(E) floppy(E) ehci_pci(E) qxl(E) serio_raw(E) =
drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) =
fb_sys_fops(E) sym53c8xx(E) scsi_transport_spi(E) button(E) libata(E)
>>> Jan 03 11:35:19 linux-mainline kernel: ttm(E) drm(E) uhci_hcd(E) =
ehci_hcd(E) usbcore(E) virtio_pci(E) virtio_ring(E) virtio(E) sg(E) =
dm_multipath(E) dm_mod(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) =
scsi_mod(E) autofs4(E)
>>> Jan 03 11:35:19 linux-mainline kernel: CPU: 0 PID: 3349 Comm: ps =
Tainted: G E 4.15.0-rc1-69-default #1
>>> Jan 03 11:35:19 linux-mainline kernel: Hardware name: QEMU Standard =
PC (i440FX + PIIX, 1996), BIOS =
rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
>>> Jan 03 11:35:19 linux-mainline kernel: task: ffff880061efce80 =
task.stack: ffff880058bd0000
>>> Jan 03 11:35:19 linux-mainline kernel: RIP: =
0010:__mod_node_page_state+0x5/0x50
>>> Jan 03 11:35:19 linux-mainline kernel: RSP: 0018:ffff880058bd7ce8 =
EFLAGS: 00010a07
>>> Jan 03 11:35:19 linux-mainline kernel: RAX: 00000000000003ff RBX: =
ffffea00011a3d80 RCX: 00000000011a3d80
>>> Jan 03 11:35:19 linux-mainline kernel: RDX: ffffffffffffffff RSI: =
000000000000000d RDI: 0000000000000000
>>> Jan 03 11:35:19 linux-mainline kernel: RBP: ffffffffffffffff R08: =
ffff88006378a630 R09: ffff880058bd7d98
>>> Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11: =
0000000000000000 R12: ffffea00011a3d80
>>> Jan 03 11:35:19 linux-mainline kernel: R13: 00007f7f4f318000 R14: =
00007f7f4f31c000 R15: ffff880058bd7e18
>>> Jan 03 11:35:19 linux-mainline kernel: FS: 0000000000000000(0000) =
GS:ffff880066c00000(0000) knlGS:0000000000000000
>>> Jan 03 11:35:19 linux-mainline kernel: CS: 0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
>>> Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0 CR3: =
0000000001c0d006 CR4: 00000000001606f0
>>> Jan 03 11:35:19 linux-mainline kernel: Call Trace:
>>> Jan 03 11:35:19 linux-mainline kernel: page_remove_rmap+0x11a/0x2b0
>>> Jan 03 11:35:19 linux-mainline kernel: unmap_page_range+0x547/0xa30
>>> Jan 03 11:35:19 linux-mainline kernel: unmap_vmas+0x42/0x90
>>> Jan 03 11:35:19 linux-mainline kernel: exit_mmap+0x86/0x180
>>> Jan 03 11:35:19 linux-mainline kernel: mmput+0x4a/0x110
>>> Jan 03 11:35:19 linux-mainline kernel: do_exit+0x25d/0xae0
>>> Jan 03 11:35:19 linux-mainline kernel: do_group_exit+0x39/0xa0
>>> Jan 03 11:35:19 linux-mainline kernel: SyS_exit_group+0x10/0x10
>>> Jan 03 11:35:19 linux-mainline kernel: =
entry_SYSCALL_64_fastpath+0x1a/0x7d
>>> Jan 03 11:35:19 linux-mainline kernel: RIP: 0033:0x7f7f4eb8c338
>>> Jan 03 11:35:19 linux-mainline kernel: RSP: 002b:00007ffca4400d48 =
EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
>>> Jan 03 11:35:19 linux-mainline kernel: RAX: ffffffffffffffda RBX: =
0000000000000000 RCX: 00007f7f4eb8c338
>>> Jan 03 11:35:19 linux-mainline kernel: RDX: 0000000000000000 RSI: =
000000000000003c RDI: 0000000000000000
>>> Jan 03 11:35:19 linux-mainline kernel: RBP: 00007ffca4400d40 R08: =
00000000000000e7 R09: fffffffffffffef8
>>> Jan 03 11:35:19 linux-mainline kernel: R10: 00007f7f4d806280 R11: =
0000000000000246 R12: 00007f7f4f756000
>>> Jan 03 11:35:19 linux-mainline kernel: R13: 00007ffca4400cc8 R14: =
00007f7f4f732b20 R15: 00007f7f4d5ebc70
>>> Jan 03 11:35:19 linux-mainline kernel: Code: f7 d9 48 39 ca 7c 05 65 =
88 50 0f c3 f0 48 01 94 f7 00 05 00 00 f0 48 01 14 f5 c0 c4 c0 81 31 d2 =
eb e5 0f 1f 40 00 0f 1f 44 00 00 <48> 8b 8f c0 9e 02 00 89 f6 48 8d 04 =
31 65 44 8a 40 01 4d 0f be
>>> Jan 03 11:35:19 linux-mainline kernel: RIP: =
__mod_node_page_state+0x5/0x50 RSP: ffff880058bd7ce8
>>> Jan 03 11:35:19 linux-mainline kernel: CR2: 0000000000029ec0
>>> Jan 03 11:35:19 linux-mainline kernel: ---[ end trace =
b5314eeef943a473 ]---
>>> Jan 03 11:35:19 linux-mainline kernel: Fixing recursive fault but =
reboot is needed!
>>> Jan 03 11:35:19 linux-mainline kernel: XFS (vdd): Unmounting =
Filesystem
>>>=20
>>>=20
>> Yes, this call trace may be related with bfq, because it concerns
>> (non-bfq) code that may get executed only when bfq is set as I/O
>> scheduler. In fact, the failure happens on an exit_group, and bfq
>> supports cgroups, while either mq-deadline or kyber don't.
>>=20
>> Did you also check what __mod_node_page_state+0x5 corresponds to in
>> your sources? Maybe this piece of information could ring some bell, =
at
>> least for people more expert than me on the involved code.
>=20
>=20
> Seems it is a null dereferenced since RDI is 0000000000000000.
>=20
> (gdb) l *__mod_node_page_state+0x5
> 0x385 is in __mod_node_page_state =
(/usr/src/kernels/4.15.0-rc1-69-default/mm/vmstat.c:338).
> 333 EXPORT_SYMBOL(__mod_zone_page_state);
> 334
> 335 void __mod_node_page_state(struct pglist_data *pgdat, enum =
node_stat_item item,
> 336 long delta)
> 337 {
> 338 struct per_cpu_nodestat __percpu *pcp =3D =
pgdat->per_cpu_nodestats;
> 339 s8 __percpu *p =3D pcp->vm_node_stat_diff + item;
> 340 long x;
> 341 long t;
> 342
>=20
> Let me know if you need anything else.
>=20
This failure seems related to the mm data structures of the task.
Unfortunately, I have no idea of how a mistake in bfq could corrupt
such unrelated structures; unless bfq contains some serious error,
corrupting unrelated memory areas.
I hope someone can provide better insights,
Paolo
>>> Occasionally mount process hangs forever.
>>> linux-mainline:~ # cat /proc/19627/stack
>>> [<ffffffff810a65f2>] io_schedule+0x12/0x40
>>> [<ffffffff8119fbb7>] wait_on_page_bit+0xd7/0x100
>>> [<ffffffff811b3713>] truncate_inode_pages_range+0x423/0x7c0
>>> [<ffffffff81273768>] set_blocksize+0x98/0xb0
>>> [<ffffffff81273798>] sb_set_blocksize+0x18/0x40
>>> [<ffffffffa06a2e58>] xfs_fs_fill_super+0x1b8/0x590 [xfs]
>>> [<ffffffff8123bd4d>] mount_bdev+0x17d/0x1b0
>>> [<ffffffff8123c6d4>] mount_fs+0x34/0x150
>>> [<ffffffff81259702>] vfs_kern_mount+0x62/0x110
>>> [<ffffffff8125bd1a>] do_mount+0x1ca/0xc30
>>> [<ffffffff8125ca6e>] SyS_mount+0x7e/0xd0
>>> [<ffffffff8172fff3>] entry_SYSCALL_64_fastpath+0x1a/0x7d
>>> [<ffffffffffffffff>] 0xffffffffffffffff
>> Maybe this hang has to do with the one already recently reported for
>> USB drives. We have already found the cause of that one, and are
>> finalizing our fix.
>=20
> I am not sure it is the same one since the test is run against virtio =
disk
> not usb, but anyway I can try the fix.
>=20
> Thanks,
> Guoqing
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-01-03 18:11 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-03 3:58 Bug report - issue with bfq? Guoqing Jiang
2018-01-03 7:44 ` Paolo Valente
2018-01-03 9:06 ` Guoqing Jiang
2018-01-03 18:10 ` Paolo Valente
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox