* Re: Block IO issue in kernel-v4.17
[not found] <CAEemH2fbPv2wphVBHxSSS+KEK5S-rercXhvgLJ7YMOZnrTpexg@mail.gmail.com>
@ 2018-06-06 8:41 ` Kent Overstreet
2018-06-06 14:18 ` Jens Axboe
0 siblings, 1 reply; 9+ messages in thread
From: Kent Overstreet @ 2018-06-06 8:41 UTC (permalink / raw)
To: Li Wang; +Cc: Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel
On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote:
> Hi BIO experts,
>
> I catch this panic issue on some kind of arches(x86_64, ppc64,
> ppc64le..), it seems the root cause is very probably from BIO changes
> from kernel-4.17-rc7. Plz take a look.
That's the bioset changes, the fix is out and on its way in
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-06 8:41 ` Block IO issue in kernel-v4.17 Kent Overstreet
@ 2018-06-06 14:18 ` Jens Axboe
2018-06-07 3:22 ` Li Wang
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Jens Axboe @ 2018-06-06 14:18 UTC (permalink / raw)
To: Kent Overstreet, Li Wang
Cc: Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel
On 6/6/18 2:41 AM, Kent Overstreet wrote:
> On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote:
>> Hi BIO experts,
>>
>> I catch this panic issue on some kind of arches(x86_64, ppc64,
>> ppc64le..), it seems the root cause is very probably from BIO changes
>> from kernel-4.17-rc7. Plz take a look.
>
> That's the bioset changes, the fix is out and on its way in
It's already in mainline, since about lunch time yesterday.
--
Jens Axboe
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-06 14:18 ` Jens Axboe
@ 2018-06-07 3:22 ` Li Wang
2018-06-07 6:26 ` Li Wang
2018-06-07 6:33 ` Chunyu Hu
2 siblings, 0 replies; 9+ messages in thread
From: Li Wang @ 2018-06-07 3:22 UTC (permalink / raw)
To: Jens Axboe
Cc: Kent Overstreet, Coly Li, hch, darrick.wong, snitzer, linux-block,
linux-kernel
[-- Attachment #1: Type: text/plain, Size: 634 bytes --]
On Wed, Jun 6, 2018 at 10:18 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 6/6/18 2:41 AM, Kent Overstreet wrote:
> > On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote:
> >> Hi BIO experts,
> >>
> >> I catch this panic issue on some kind of arches(x86_64, ppc64,
> >> ppc64le..), it seems the root cause is very probably from BIO changes
> >> from kernel-4.17-rc7. Plz take a look.
> >
> > That's the bioset changes, the fix is out and on its way in
>
> It's already in mainline, since about lunch time yesterday.
>
Which commit fix that?
>
>
> --
> Jens Axboe
>
>
--
Regards,
Li Wang
[-- Attachment #2: Type: text/html, Size: 1509 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-06 14:18 ` Jens Axboe
2018-06-07 3:22 ` Li Wang
@ 2018-06-07 6:26 ` Li Wang
2018-06-07 6:33 ` Chunyu Hu
2 siblings, 0 replies; 9+ messages in thread
From: Li Wang @ 2018-06-07 6:26 UTC (permalink / raw)
To: Jens Axboe
Cc: Kent Overstreet, Coly Li, hch, darrick.wong, snitzer, linux-block,
linux-kernel
[-- Attachment #1: Type: text/plain, Size: 17933 bytes --]
On Wed, Jun 6, 2018 at 10:18 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 6/6/18 2:41 AM, Kent Overstreet wrote:
> > On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote:
> >> Hi BIO experts,
> >>
> >> I catch this panic issue on some kind of arches(x86_64, ppc64,
> >> ppc64le..), it seems the root cause is very probably from BIO changes
> >> from kernel-4.17-rc7. Plz take a look.
> >
> > That's the bioset changes, the fix is out and on its way in
>
> It's already in mainline, since about lunch time yesterday.
>
Do you mean this commit?
commit d377535405686f735b90a8ad4ba269484cd7c96e
Author: Kent Overstreet <kent.overstreet@gmail.com>
Date: Tue Jun 5 05:26:33 2018 -0400
dm: Use kzalloc for all structs with embedded biosets/mempools
Unfortunately, I pull the latest mainline kernel and test, still hit the
panic. :(
[ 21.832230] ------------[ cut here ]------------
[ 21.832244] list_add corruption. next->prev should be prev
(c00000045f5801b0), but was 0000000000000000. (next=c000000461b2d5b8).
[ 21.832265] WARNING: CPU: 12 PID: 2499 at lib/list_debug.c:25
.__list_add_valid+0xac/0xc0
[ 21.832270] Modules linked in: sunrpc nx_crypto sg pseries_rng xts
vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp
ibmveth dm_mirror dm_region_hash dm_log dm_mod
[ 21.832295] CPU: 12 PID: 2499 Comm: stress Not tainted 4.17.0 #1
[ 21.832300] NIP: c0000000005021dc LR: c0000000005021d8 CTR:
00000000013abb8c
[ 21.832304] REGS: c0000004623926d0 TRAP: 0700 Not tainted (4.17.0)
[ 21.832308] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI> CR:
42244822 XER: 00000006
[ 21.832318] CFAR: c0000000000f93f0 SOFTE: 1
[ 21.832318] GPR00: c0000000005021d8 c000000462392950
c0000000013ac100 0000000000000075
[ 21.832318] GPR04: 0000000000000001 000000005b2baea6
0000000000000000 0000000000000000
[ 21.832318] GPR08: 0000000000000001 0000000000000007
0000000000000006 000000000000017f
[ 21.832318] GPR12: 0000000028244824 c00000001ebddc00
0000000000000000 c000000461ccb800
[ 21.832318] GPR16: f00000000113d1e0 c0000004623931f0
0000000000000000 f00000000113d1c0
[ 21.832318] GPR20: 0000000000000000 0000000000000000
0000000000000000 c000000462393360
[ 21.832318] GPR24: 0000000000000000 c00000045f5801b0
c000000461b2d5b8 0000000000000001
[ 21.832318] GPR28: 0000000000000002 c000000462392af8
c00000045f5801a8 c000000462392ae0
[ 21.832372] NIP [c0000000005021dc] .__list_add_valid+0xac/0xc0
[ 21.832377] LR [c0000000005021d8] .__list_add_valid+0xa8/0xc0
[ 21.832381] Call Trace:
[ 21.832385] [c000000462392950] [c0000000005021d8]
.__list_add_valid+0xa8/0xc0 (unreliable)
[ 21.832393] [c0000004623929c0] [c000000000159b28] .prepare_to_wait+0xc8/0xf0
[ 21.832400] [c000000462392a70] [c000000000268720] .mempool_alloc+0x180/0x1d0
[ 21.832407] [c000000462392b50] [c000000000496ee4]
.bio_alloc_bioset+0x1c4/0x2b0
[ 21.832422] [c000000462392c20] [d0000000045f1578]
.alloc_io+0x38/0x170 [dm_mod]
[ 21.832435] [c000000462392cc0] [d0000000045f3824]
.__split_and_process_bio+0x34/0x1e0 [dm_mod]
[ 21.832449] [c000000462392d80] [d0000000045f17bc]
.__dm_make_request+0x6c/0x100 [dm_mod]
[ 21.832455] [c000000462392e20] [c0000000004a3368]
.generic_make_request+0x1e8/0x420
[ 21.832460] [c000000462392ee0] [c0000000004a3654] .submit_bio+0xb4/0x1c0
[ 21.832466] [c000000462392fa0] [c0000000002d74c4]
.__swap_writepage+0x394/0x540
[ 21.832472] [c000000462393090] [c000000000284ed8]
.pageout.isra.36+0x288/0x490
[ 21.832478] [c000000462393180] [c000000000287928]
.shrink_page_list+0x948/0x1090
[ 21.832484] [c0000004623932f0] [c000000000288b8c]
.shrink_inactive_list+0x2ec/0x720
[ 21.832490] [c000000462393400] [c0000000002897c8]
.shrink_node_memcg+0x238/0x7d0
[ 21.832495] [c000000462393550] [c000000000289eb8] .shrink_node+0x158/0x5e0
[ 21.832501] [c000000462393670] [c00000000028a838]
.do_try_to_free_pages+0x118/0x4e0
[ 21.832507] [c000000462393760] [c00000000028ad08]
.try_to_free_pages+0x108/0x290
[ 21.832512] [c000000462393860] [c000000000272508]
.__alloc_pages_nodemask+0x638/0xf90
[ 21.832518] [c000000462393a40] [c0000000002f15c8]
.alloc_pages_vma+0x98/0x240
[ 21.832524] [c000000462393b00] [c0000000002ba560]
.__handle_mm_fault+0xbc0/0x1900
[ 21.832530] [c000000462393c30] [c0000000002bb408]
.handle_mm_fault+0x168/0x280
[ 21.832535] [c000000462393cc0] [c000000000061cd4]
.__do_page_fault+0x1c4/0x9b0
[ 21.832539] [c000000462393da0] [c0000000000624e8] .do_page_fault+0x28/0xb0
[ 21.832544] [c000000462393e30] [c00000000000a634]
handle_page_fault+0x18/0x38
[ 21.832548] Instruction dump:
[ 21.832551] 38630948 7d264b78 4bbf71dd 60000000 0fe00000 38600000
4bffffac 3c62ff8c
[ 21.832558] 7d254b78 386308e0 4bbf71bd 60000000 <0fe00000> 38600000
4bffff8c 60000000
[ 21.832565] ---[ end trace c4d85974c8ca08fa ]---
[ 21.832569] ------------[ cut here ]------------
[ 21.832573] list_add corruption. next->prev should be prev
(c00000045f5801b0), but was 0000000000000000. (next=c000000461b2d5b8).
[ 21.832584] WARNING: CPU: 10 PID: 109 at lib/list_debug.c:25
.__list_add_valid+0xac/0xc0
[ 21.832589] Modules linked in: sunrpc nx_crypto sg pseries_rng xts
vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp
ibmveth dm_mirror dm_region_hash dm_log dm_mod
[ 21.832609] CPU: 10 PID: 109 Comm: kswapd0 Tainted: G W
4.17.0 #1
[ 21.832613] NIP: c0000000005021dc LR: c0000000005021d8 CTR:
00000000013abb8c
[ 21.832617] REGS: c000000472d2eb70 TRAP: 0700 Tainted: G W
(4.17.0)
[ 21.832621] MSR: 800000010282b032
<SF,VEC,VSX,EE,FP,ME,IR,DR,RI,TM[E]> CR: 42004022 XER: 00000006
[ 21.832632] CFAR: c0000000000f93f0 SOFTE: 1
[ 21.832632] GPR00: c0000000005021d8 c000000472d2edf0
c0000000013ac100 0000000000000075
[ 21.832632] GPR04: 0000000000000001 000000005b2bafba
0000000000000000 0000000000000000
[ 21.832632] GPR08: 0000000000000001 0000000000000007
0000000000000006 00000000000001a7
[ 21.832632] GPR12: 0000000028004024 c00000001ebf1200
0000000000000000 c000000461ccb800
[ 21.832632] GPR16: f000000001131de0 c000000472d2f690
0000000000000000 f000000001131dc0
[ 21.832632] GPR20: 0000000000000000 0000000000000000
0000000000000000 c000000472d2f800
[ 21.832632] GPR24: 0000000000000000 c00000045f5801b0
c000000461b2d5b8 0000000000000001
[ 21.832632] GPR28: 0000000000000002 c000000472d2ef98
c00000045f5801a8 c000000472d2ef80
[ 21.832675] NIP [c0000000005021dc] .__list_add_valid+0xac/0xc0
[ 21.832679] LR [c0000000005021d8] .__list_add_valid+0xa8/0xc0
[ 21.832682] Call Trace:
[ 21.832684] [c000000472d2edf0] [c0000000005021d8]
.__list_add_valid+0xa8/0xc0 (unreliable)
[ 21.832689] [c000000472d2ee60] [c000000000159b28] .prepare_to_wait+0xc8/0xf0
[ 21.832692] [c000000472d2ef10] [c000000000268720] .mempool_alloc+0x180/0x1d0
[ 21.832697] [c000000472d2eff0] [c000000000496ee4]
.bio_alloc_bioset+0x1c4/0x2b0
[ 21.832707] [c000000472d2f0c0] [d0000000045f1578]
.alloc_io+0x38/0x170 [dm_mod]
[ 21.832716] [c000000472d2f160] [d0000000045f3824]
.__split_and_process_bio+0x34/0x1e0 [dm_mod]
[ 21.832727] [c000000472d2f220] [d0000000045f17bc]
.__dm_make_request+0x6c/0x100 [dm_mod]
[ 21.832731] [c000000472d2f2c0] [c0000000004a3368]
.generic_make_request+0x1e8/0x420
[ 21.832735] [c000000472d2f380] [c0000000004a3654] .submit_bio+0xb4/0x1c0
[ 21.832739] [c000000472d2f440] [c0000000002d74c4]
.__swap_writepage+0x394/0x540
[ 21.832743] [c000000472d2f530] [c000000000284ed8]
.pageout.isra.36+0x288/0x490
[ 21.832747] [c000000472d2f620] [c000000000287928]
.shrink_page_list+0x948/0x1090
[ 21.832751] [c000000472d2f790] [c000000000288b8c]
.shrink_inactive_list+0x2ec/0x720
[ 21.832755] [c000000472d2f8a0] [c0000000002897c8]
.shrink_node_memcg+0x238/0x7d0
[ 21.832760] [c000000472d2f9f0] [c000000000289eb8] .shrink_node+0x158/0x5e0
[ 21.832763] [c000000472d2fb10] [c00000000028b590] .balance_pgdat+0x200/0x470
[ 21.832767] [c000000472d2fc50] [c00000000028b9bc] .kswapd+0x1bc/0x5f0
[ 21.832771] [c000000472d2fd70] [c000000000124d78] .kthread+0x158/0x1a0
[ 21.832776] [c000000472d2fe30] [c00000000000b654]
.ret_from_kernel_thread+0x58/0x84
[ 21.832779] Instruction dump:
[ 21.832781] 38630948 7d264b78 4bbf71dd 60000000 0fe00000 38600000
4bffffac 3c62ff8c
[ 21.832787] 7d254b78 386308e0 4bbf71bd 60000000 <0fe00000> 38600000
4bffff8c 60000000
[ 21.832792] ---[ end trace c4d85974c8ca08fb ]---
[ 21.833118] Unable to handle kernel paging request for instruction fetch
[ 21.833121] Faulting instruction address: 0x00000000
[ 21.833127] Oops: Kernel access of bad area, sig: 11 [#1]
[ 21.833129] BE SMP NR_CPUS=2048 NUMA pSeries
[ 21.833134] Modules linked in: sunrpc nx_crypto sg pseries_rng xts
vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp
ibmveth dm_mirror dm_region_hash dm_log dm_mod
[ 21.833148] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G W
4.17.0 #1
[ 21.833151] NIP: 0000000000000000 LR: c000000000159e8c CTR:
0000000000000000
[ 21.833155] REGS: c00000047ff93260 TRAP: 0400 Tainted: G W
(4.17.0)
[ 21.833158] MSR: 8000000040009032 <SF,EE,ME,IR,DR,RI> CR:
24028028 XER: 20000006
[ 21.833166] CFAR: c000000000008874 SOFTE: 1
[ 21.833166] GPR00: c00000000015a09c c00000047ff934e0
0000000000000000 c000000461b2d5a0
[ 21.833166] GPR04: 0000000000000003 0000000000000000
0000000000000000 0000000000000000
[ 21.833166] GPR08: c00000047ff93610 c000000479ae7880
0000000000000000 d000000004606758
[ 21.833166] GPR12: 0000000024022022 c00000001ebf1200
c000000474557f90 0000000000200042
[ 21.833166] GPR16: 00000000ffff9358 c00000047ff90000
0000000000000000 0000000000000001
[ 21.833166] GPR20: c000000000df7080 0000000000000005
c00000047ff93610 0000000000000001
[ 21.833166] GPR24: 0000000000000000 0000000000000000
0000000000000003 0000000000000000
[ 21.833166] GPR28: c00000045f5801b0 0000000000000000
c000000461b2d5a0 ffffffffffffffe8
[ 21.833208] NIP [0000000000000000] (null)
[ 21.833212] LR [c000000000159e8c] .__wake_up_common+0xdc/0x240
[ 21.833215] Call Trace:
[ 21.833217] [c00000047ff934e0] [c00000047ff935c0]
0xc00000047ff935c0 (unreliable)
[ 21.833222] [c00000047ff935a0] [c00000000015a09c]
.__wake_up_common_lock+0xac/0x100
[ 21.833227] [c00000047ff93680] [c000000000267dc8] .mempool_free+0xc8/0xe0
[ 21.833232] [c00000047ff93700] [c0000000004971ac] .bio_free+0x4c/0xa0
[ 21.833242] [c00000047ff93780] [d0000000045f1998]
.dec_pending+0x128/0x310 [dm_mod]
[ 21.833253] [c00000047ff93850] [d0000000045f1edc]
.clone_endio+0xcc/0x180 [dm_mod]
[ 21.833258] [c00000047ff938e0] [c000000000498c84] .bio_endio+0x184/0x290
[ 21.833262] [c00000047ff93990] [c0000000004a3d20]
.blk_update_request+0xf0/0x4a0
[ 21.833267] [c00000047ff93a40] [c000000000642ae8]
.scsi_end_request+0x48/0x270
[ 21.833272] [c00000047ff93af0] [c000000000642f44]
.scsi_io_completion+0x234/0x710
[ 21.833277] [c00000047ff93bd0] [c000000000637cc8]
.scsi_finish_command+0x148/0x1b0
[ 21.833282] [c00000047ff93c60] [c000000000641e18]
.scsi_softirq_done+0x198/0x1f0
[ 21.833286] [c00000047ff93d00] [c0000000004aee24]
.blk_done_softirq+0xc4/0xf0
[ 21.833292] [c00000047ff93d90] [c0000000009829a0] .__do_softirq+0x150/0x3a8
[ 21.833297] [c00000047ff93e90] [c000000000101d68] .irq_exit+0x198/0x1b0
[ 21.833302] [c00000047ff93f10] [c000000000016024] .__do_irq+0x94/0x1f0
[ 21.833306] [c00000047ff93f90] [c00000000002779c] .call_do_irq+0x14/0x24
[ 21.833311] [c0000004745577a0] [c00000000001620c] .do_IRQ+0x8c/0x100
[ 21.833315] [c000000474557840] [c000000000008c24]
hardware_interrupt_common+0x114/0x120
[ 21.833323] --- interrupt: 501 at .plpar_hcall_norets+0x14/0x20
[ 21.833323] LR = .check_and_cede_processor+0x24/0x40
[ 21.833328] [c000000474557b30] [7fffffffffffffff]
0x7fffffffffffffff (unreliable)
[ 21.833333] [c000000474557ba0] [c00000000076565c]
.dedicated_cede_loop+0x4c/0x130
[ 21.833339] [c000000474557c20] [c000000000762d9c]
.cpuidle_enter_state+0xac/0x390
[ 21.833343] [c000000474557cd0] [c00000000013b8cc] .call_cpuidle+0x3c/0x70
[ 21.833348] [c000000474557d40] [c00000000013be38] .do_idle+0x2e8/0x390
[ 21.833352] [c000000474557e20] [c00000000013c098]
.cpu_startup_entry+0x28/0x30
[ 21.833357] [c000000474557ea0] [c000000000046980]
.start_secondary+0x4d0/0x520
[ 21.833362] [c000000474557f90] [c00000000000ab70]
start_secondary_prolog+0x10/0x14
[ 21.833366] Instruction dump:
[ 21.833369] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
XXXXXXXX XXXXXXXX
[ 21.833374] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
XXXXXXXX XXXXXXXX
[ 21.833382] ---[ end trace c4d85974c8ca08fc ]---
[ 21.835408]
[ 22.835471] Kernel panic - not syncing: Fatal exception in interrupt
[ 23.884733] WARNING: CPU: 10 PID: 0 at drivers/tty/vt/vt.c:3885
.do_unblank_screen+0x268/0x270
[ 23.884740] Modules linked in: sunrpc nx_crypto sg pseries_rng xts
vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp
ibmveth dm_mirror dm_region_hash dm_log dm_mod
[ 23.884766] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G D W
4.17.0 #1
[ 23.884772] NIP: c0000000005bb758 LR: c0000000005bb524 CTR:
c00000000002ae00
[ 23.884778] REGS: c00000047ff92cc0 TRAP: 0700 Tainted: G D W
(4.17.0)
[ 23.884783] MSR: 8000000000021032 <SF,ME,IR,DR,RI> CR: 48022042
XER: 00000009
[ 23.884794] CFAR: c0000000005bb538 SOFTE: 1
[ 23.884794] GPR00: c0000000005bb714 c00000047ff92f40
c0000000013ac100 0000000000000000
[ 23.884794] GPR04: 0000000000000001 000000000000000b
0000000020646d5f 6d5f726567696f6e
[ 23.884794] GPR08: c00000000168c100 0000000000000000
0000000000000000 6e67207874732076
[ 23.884794] GPR12: 0000000088022044 c00000001ebf1200
c000000474557f90 0000000000200042
[ 23.884794] GPR16: 00000000ffff9358 c00000047ff90000
0000000000000000 0000000000000001
[ 23.884794] GPR20: c000000000df7080 0000000000000005
c00000047ff93610 0000000000000001
[ 23.884794] GPR24: 0000000000000000 0000000000000000
c0000000011da080 c0000000015767c0
[ 23.884794] GPR28: c000000001576798 0000000000000000
c000000001685340 c0000000015767c0
[ 23.884863] NIP [c0000000005bb758] .do_unblank_screen+0x268/0x270
[ 23.884868] LR [c0000000005bb524] .do_unblank_screen+0x34/0x270
[ 23.884873] Call Trace:
[ 23.884877] [c00000047ff92f40] [c0000000005bb714]
.do_unblank_screen+0x224/0x270 (unreliable)
[ 23.884887] [c00000047ff92fd0] [c0000000004e88a0] .bust_spinlocks+0x30/0x70
[ 23.884894] [c00000047ff93040] [c0000000000f9cf8] .panic+0x1b0/0x308
[ 23.884901] [c00000047ff930e0] [c000000000022e5c] .oops_end+0x1cc/0x1e0
[ 23.884908] [c00000047ff93170] [c000000000062648] .bad_page_fault+0xd8/0x144
[ 23.884916] [c00000047ff931f0] [c00000000000a650]
handle_page_fault+0x34/0x38
[ 23.884924] --- interrupt: 400 at (null)
[ 23.884924] LR = .__wake_up_common+0xdc/0x240
[ 23.884932] [c00000047ff934e0] [c00000047ff935c0]
0xc00000047ff935c0 (unreliable)
[ 23.884940] [c00000047ff935a0] [c00000000015a09c]
.__wake_up_common_lock+0xac/0x100
[ 23.884948] [c00000047ff93680] [c000000000267dc8] .mempool_free+0xc8/0xe0
[ 23.884955] [c00000047ff93700] [c0000000004971ac] .bio_free+0x4c/0xa0
[ 23.884973] [c00000047ff93780] [d0000000045f1998]
.dec_pending+0x128/0x310 [dm_mod]
[ 23.884991] [c00000047ff93850] [d0000000045f1edc]
.clone_endio+0xcc/0x180 [dm_mod]
[ 23.884998] [c00000047ff938e0] [c000000000498c84] .bio_endio+0x184/0x290
[ 23.885005] [c00000047ff93990] [c0000000004a3d20]
.blk_update_request+0xf0/0x4a0
[ 23.885013] [c00000047ff93a40] [c000000000642ae8]
.scsi_end_request+0x48/0x270
[ 23.885020] [c00000047ff93af0] [c000000000642f44]
.scsi_io_completion+0x234/0x710
[ 23.885028] [c00000047ff93bd0] [c000000000637cc8]
.scsi_finish_command+0x148/0x1b0
[ 23.885035] [c00000047ff93c60] [c000000000641e18]
.scsi_softirq_done+0x198/0x1f0
[ 23.885042] [c00000047ff93d00] [c0000000004aee24]
.blk_done_softirq+0xc4/0xf0
[ 23.885051] [c00000047ff93d90] [c0000000009829a0] .__do_softirq+0x150/0x3a8
[ 23.885058] [c00000047ff93e90] [c000000000101d68] .irq_exit+0x198/0x1b0
[ 23.885065] [c00000047ff93f10] [c000000000016024] .__do_irq+0x94/0x1f0
[ 23.885072] [c00000047ff93f90] [c00000000002779c] .call_do_irq+0x14/0x24
[ 23.885079] [c0000004745577a0] [c00000000001620c] .do_IRQ+0x8c/0x100
[ 23.885086] [c000000474557840] [c000000000008c24]
hardware_interrupt_common+0x114/0x120
[ 23.885096] --- interrupt: 501 at .plpar_hcall_norets+0x14/0x20
[ 23.885096] LR = .check_and_cede_processor+0x24/0x40
[ 23.885104] [c000000474557b30] [7fffffffffffffff]
0x7fffffffffffffff (unreliable)
[ 23.885112] [c000000474557ba0] [c00000000076565c]
.dedicated_cede_loop+0x4c/0x130
[ 23.885120] [c000000474557c20] [c000000000762d9c]
.cpuidle_enter_state+0xac/0x390
[ 23.885128] [c000000474557cd0] [c00000000013b8cc] .call_cpuidle+0x3c/0x70
[ 23.885134] [c000000474557d40] [c00000000013be38] .do_idle+0x2e8/0x390
[ 23.885141] [c000000474557e20] [c00000000013c098]
.cpu_startup_entry+0x28/0x30
[ 23.885148] [c000000474557ea0] [c000000000046980]
.start_secondary+0x4d0/0x520
[ 23.885156] [c000000474557f90] [c00000000000ab70]
start_secondary_prolog+0x10/0x14
[ 23.885162] Instruction dump:
[ 23.885167] 39487a00 3c62ffe7 3863aab0 e94a0000 7d2407b4 7c845214
4bbd6671 60000000
[ 23.885178] 39200001 3d42003f 912a2080 4bfffe60 <0fe00000> 4bfffde0
38600000 4bfffd8c
[ 23.885189] ---[ end trace c4d85974c8ca08fd ]---
>
>
> --
> Jens Axboe
>
>
--
Regards,
Li Wang
[-- Attachment #2: Type: text/html, Size: 19448 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-06 14:18 ` Jens Axboe
2018-06-07 3:22 ` Li Wang
2018-06-07 6:26 ` Li Wang
@ 2018-06-07 6:33 ` Chunyu Hu
2018-06-07 14:46 ` Jens Axboe
2 siblings, 1 reply; 9+ messages in thread
From: Chunyu Hu @ 2018-06-07 6:33 UTC (permalink / raw)
To: Jens Axboe
Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer,
linux-block, linux-kernel
kasan reported a user-after-free. I'm using a kvm machine, it panic
during boot. I'm using the latest linux tree. which contains below.
commit d377535405686f735b90a8ad4ba269484cd7c96e
Author: Kent Overstreet <kent.overstreet@gmail.com>
Date: Tue Jun 5 05:26:33 2018 -0400
dm: Use kzalloc for all structs with embedded biosets/mempools
[ 58.836774] ==================================================================
[ 58.839974] BUG: KASAN: use-after-free in __wake_up_common+0x7c7/0x880
[ 58.841988] Read of size 8 at addr ffff88025a47c3e8 by task kswapd0/66
[ 58.843986]
[ 58.845644]
[ 58.846127] Allocated by task 956:
[ 58.847249]
[ 58.847731] Freed by task 956:
[ 58.848856]
[ 58.849336] The buggy address belongs to the object at ffff88025a47c000
[ 58.849336] which belongs to the cache names_cache of size 4096
[ 58.853276] The buggy address is located 1000 bytes inside of
[ 58.853276] 4096-byte region [ffff88025a47c000, ffff88025a47d000)
[ 58.856924] The buggy address belongs to the page:
[ 58.858411] page:ffffea0009691f00 count:1 mapcount:0
mapping:0000000000000000 index:0x0 compound_mapcount: 0
[ 58.861443] flags: 0x6fffff80008100(slab|head)
[ 58.862820] raw: 006fffff80008100 0000000000000000 0000000000000000
0000000100010001
[ 58.865197] raw: dead000000000100 dead000000000200 ffff88012d940a00
0000000000000000
[ 58.867590] page dumped because: kasan: bad access detected
[ 58.869409]
[ 58.869933] Memory state around the buggy address:
[ 58.871116] ffff88025a47c280: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 58.872195] ffff88025a47c300: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 58.873295] >ffff88025a47c380: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 58.874358] ^
[ 58.875334] ffff88025a47c400: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 58.876395] ffff88025a47c480: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 58.877453] ==================================================================
[ 58.878547] BUG: unable to handle kernel NULL pointer dereference
at 0000000000000000
[ 58.879708] PGD 0 P4D 0
[ 58.880107] Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN PTI
[ 58.881013] CPU: 0 PID: 66 Comm: kswapd0 Tainted: G B W
4.17.0.fi+ #41
[ 58.881944] Hardware name: Red Hat KVM, BIOS 0.0.0 02/06/2015
[ 58.882646] RIP: 0010: (null)
[ 58.883106] Code: Bad RIP value.
[ 58.883520] RSP: 0018:ffff88012ec07818 EFLAGS: 00010086
[ 58.884158] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 58.885020] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88025a47c3d0
[ 58.886576] RBP: ffff880257960418 R08: ffff88025a47c3d0 R09: fffffbfff09587a4
[ 58.888370] R10: fffffbfff09587a4 R11: ffffffff84ac3d23 R12: ffffffffffffffe8
[ 58.890257] R13: dffffc0000000000 R14: 0000000000000000 R15: ffff88012ec07910
[ 58.892113] FS: 0000000000000000(0000) GS:ffff88012ec00000(0000)
knlGS:0000000000000000
[ 58.894194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 58.895648] CR2: ffffffffffffffd6 CR3: 0000000004214000 CR4: 00000000001406f0
[ 58.897468] Call Trace:
[ 58.898117] <IRQ>
[ 58.898716] ? __wake_up_common+0x18e/0x880
[ 58.899843] ? wait_woken+0x340/0x340
[ 58.900842] ? do_raw_spin_lock+0xcf/0x220
[ 58.901665] ? __wake_up_common_lock+0xe3/0x170
[ 58.902439] ? __wake_up_common+0x880/0x880
[ 58.903077] ? time_hardirqs_off+0x3e/0x4f0
[ 58.903742] ? _raw_spin_unlock_irqrestore+0x45/0xa0
[ 58.904486] ? mempool_free+0x270/0x3a0
[ 58.905185] ? bio_free+0x104/0x190
[ 58.905722] ? bio_put+0xb9/0x120
[ 58.906235] ? dec_pending+0x3cd/0xbe0 [dm_mod]
[ 58.906960] ? time_hardirqs_off+0x30/0x4f0
[ 58.907602] ? debug_check_no_locks_freed+0x260/0x260
[ 58.908389] ? alloc_io+0x820/0x820 [dm_mod]
[ 58.909130] ? linear_status+0x1b0/0x1b0 [dm_mod]
[ 58.909839] ? clone_endio+0x1ed/0x890 [dm_mod]
[ 58.910691] ? bio_disassociate_task+0x16e/0x450
[ 58.911453] ? dm_get_queue_limits+0x110/0x110 [dm_mod]
[ 58.912288] ? check_preemption_disabled+0x36/0x2a0
[ 58.913075] ? dm_get_queue_limits+0x110/0x110 [dm_mod]
[ 58.913919] ? bio_endio+0x423/0x8b0
[ 58.914513] ? blk_update_request+0x295/0xe40
[ 58.915215] ? virtqueue_get_buf_ctx+0x3b0/0xa60 [virtio_ring]
[ 58.916171] ? blk_mq_end_request+0x56/0x390
[ 58.916877] ? blk_mq_complete_request+0x36a/0x6e0
[ 58.917857] ? virtblk_done+0x1bc/0x450 [virtio_blk]
[ 58.918690] ? 0xffffffffa0158000
[ 58.919260] ? __lock_is_held+0xb6/0x170
[ 58.919917] ? check_preemption_disabled+0x36/0x2a0
[ 58.920730] ? 0xffffffffa0158000
[ 58.921300] ? vring_interrupt+0x170/0x280 [virtio_ring]
[ 58.922176] ? vring_alloc_queue+0x400/0x400 [virtio_ring]
[ 58.923080] ? __handle_irq_event_percpu+0x117/0x9a0
[ 58.923908] ? handle_irq_event_percpu+0x77/0x180
[ 58.924691] ? __handle_irq_event_percpu+0x9a0/0x9a0
[ 58.925519] ? do_raw_spin_unlock+0x156/0x250
[ 58.926257] ? handle_irq_event+0xc6/0x1a0
[ 58.926943] ? handle_edge_irq+0x229/0xd40
[ 58.927747] ? handle_irq+0x2e2/0x5fd
[ 58.928528] ? check_preemption_disabled+0x36/0x2a0
[ 58.929545] ? do_IRQ+0xa7/0x240
[ 58.930241] ? common_interrupt+0xf/0xf
[ 58.931072] </IRQ>
[ 58.931576] ? lock_acquire+0x184/0x470
[ 58.932395] ? list_lru_count_one+0xb8/0x3a0
[ 58.933330] ? list_lru_count_one+0x86/0x3a0
[ 58.934362] ? super_cache_count+0x152/0x2f0
[ 58.935268] ? shrink_slab.part.24+0x1fe/0xc60
[ 58.936205] ? mem_cgroup_from_task+0x180/0x180
[ 58.937162] ? prepare_kswapd_sleep+0x160/0x160
[ 58.938101] ? mem_cgroup_iter+0x165/0xc60
[ 58.938955] ? shrink_slab+0x9e/0xd0
[ 58.939707] ? shrink_node+0x3f1/0x17b0
[ 58.940675] ? shrink_node_memcg+0x1f10/0x1f10
[ 58.941794] ? mem_cgroup_iter+0x165/0xc60
[ 58.942668] ? mem_cgroup_nr_lru_pages+0xe0/0xe0
[ 58.943642] ? inactive_list_is_low+0x1f3/0x6e0
[ 58.944598] ? balance_pgdat+0x2c9/0x950
[ 58.945436] ? mem_cgroup_shrink_node+0x7d0/0x7d0
[ 58.946432] ? preempt_count_sub+0x101/0x190
[ 58.947340] ? check_preemption_disabled+0x36/0x2a0
[ 58.948373] ? kswapd+0x5c1/0x1060
[ 58.949109] ? balance_pgdat+0x950/0x950
[ 58.949948] ? __kthread_parkme+0x84/0x240
[ 58.950816] ? __kthread_parkme+0xff/0x240
[ 58.951685] ? finish_wait+0x3f0/0x3f0
[ 58.952484] ? schedule+0x92/0x230
[ 58.953202] ? balance_pgdat+0x950/0x950
[ 58.954030] ? balance_pgdat+0x950/0x950
[ 58.954861] ? kthread+0x37d/0x500
[ 58.955592] ? kthread_create_worker_on_cpu+0xe0/0xe0
[ 58.956654] ? ret_from_fork+0x3a/0x50
[ 58.957457] Modules linked in: sunrpc snd_hda_codec_generic
crct10dif_pclmul snd_hda_intel crc32_pclmul ghash_clmulni_intel
snd_hda_codec snd_hda_core snd_hwdep vfat fat iTCO_wdt
iTCO_vendor_support snd_seq snd_seq_device snd_pcm pcspkr
virtio_balloon snd_timer i2c_i801 sg lpc_ich snd soundcore shpchp
ip_tables xfs libcrc32c sr_mod cdrom virtio_net net_failover
virtio_scsi failover virtio_blk virtio_console bochs_drm
drm_kms_helper syscopyarea sysfillrect 8139too sysimgblt fb_sys_fops
ttm drm crc32c_intel ahci libahci 8139cp libata serio_raw mii i2c_core
virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
[ 58.968856] CR2: 0000000000000000
[ 58.969583] ---[ end trace 70ad259a1fd9713f ]---
[ 58.970624] RIP: 0010: (null)
[ 58.971417] Code: Bad RIP value.
[ 58.972095] RSP: 0018:ffff88012ec07818 EFLAGS: 00010086
[ 58.973181] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 58.974635] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88025a47c3d0
[ 58.976060] RBP: ffff880257960418 R08: ffff88025a47c3d0 R09: fffffbfff09587a4
[ 58.977495] R10: fffffbfff09587a4 R11: ffffffff84ac3d23 R12: ffffffffffffffe8
[ 58.978922] R13: dffffc0000000000 R14: 0000000000000000 R15: ffff88012ec07910
[ 58.980365] FS: 0000000000000000(0000) GS:ffff88012ec00000(0000)
knlGS:0000000000000000
[ 58.981984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 58.983139] CR2: ffffffffffffffd6 CR3: 0000000004214000 CR4: 00000000001406f0
[ 58.984568] Kernel panic - not syncing: Fatal exception in interrupt
[ 60.070024] Shutting down cpus with NMI
[ 60.070741] Kernel Offset: disabled
[ 60.071317] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
On 6 June 2018 at 22:18, Jens Axboe <axboe@kernel.dk> wrote:
> On 6/6/18 2:41 AM, Kent Overstreet wrote:
>> On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote:
>>> Hi BIO experts,
>>>
>>> I catch this panic issue on some kind of arches(x86_64, ppc64,
>>> ppc64le..), it seems the root cause is very probably from BIO changes
>>> from kernel-4.17-rc7. Plz take a look.
>>
>> That's the bioset changes, the fix is out and on its way in
>
> It's already in mainline, since about lunch time yesterday.
>
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-07 6:33 ` Chunyu Hu
@ 2018-06-07 14:46 ` Jens Axboe
2018-06-07 15:41 ` Jens Axboe
0 siblings, 1 reply; 9+ messages in thread
From: Jens Axboe @ 2018-06-07 14:46 UTC (permalink / raw)
To: Chunyu Hu
Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer,
linux-block, linux-kernel
On 6/7/18 12:33 AM, Chunyu Hu wrote:
> kasan reported a user-after-free. I'm using a kvm machine, it panic
> during boot. I'm using the latest linux tree. which contains below.
>
> commit d377535405686f735b90a8ad4ba269484cd7c96e
> Author: Kent Overstreet <kent.overstreet@gmail.com>
> Date: Tue Jun 5 05:26:33 2018 -0400
>
> dm: Use kzalloc for all structs with embedded biosets/mempools
Can you try with the below? Li Wang, would be great if you could too.
diff --git a/block/bio.c b/block/bio.c
index 595663e0281a..45bdee67d28b 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1967,6 +1967,27 @@ int bioset_init(struct bio_set *bs,
}
EXPORT_SYMBOL(bioset_init);
+void bioset_move(struct bio_set *dst, struct bio_set *src)
+{
+ dst->bio_slab = src->bio_slab;
+ dst->front_pad = src->front_pad;
+ mempool_move(&dst->bio_pool, &src->bio_pool);
+ mempool_move(&dst->bvec_pool, &src->bvec_pool);
+#if defined(CONFIG_BLK_DEV_INTEGRITY)
+ mempool_move(&dst->bio_integrity_pool, &src->bio_integrity_pool);
+ mempool_move(&dst->bvec_integrity_pool, &src->bvec_integrity_pool);
+#endif
+ BUG_ON(!bio_list_empty(&src->rescue_list));
+ BUG_ON(work_pending(&src->rescue_work));
+ spin_lock_init(&dst->rescue_lock);
+ bio_list_init(&dst->rescue_list);
+ INIT_WORK(&dst->rescue_work, bio_alloc_rescue);
+ dst->rescue_workqueue = src->rescue_workqueue;
+
+ memset(src, 0, sizeof(*src));
+}
+EXPORT_SYMBOL(bioset_move);
+
#ifdef CONFIG_BLK_CGROUP
/**
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 98dff36b89a3..87f636815baf 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1982,10 +1982,8 @@ static void __bind_mempools(struct mapped_device *md, struct dm_table *t)
bioset_initialized(&md->bs) ||
bioset_initialized(&md->io_bs));
- md->bs = p->bs;
- memset(&p->bs, 0, sizeof(p->bs));
- md->io_bs = p->io_bs;
- memset(&p->io_bs, 0, sizeof(p->io_bs));
+ bioset_move(&md->bs, &p->bs);
+ bioset_move(&md->io_bs, &p->io_bs);
out:
/* mempool bind completed, no longer need any mempools in the table */
dm_table_free_md_mempools(t);
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 810a8bee8f85..7581231dd0a3 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -417,6 +417,7 @@ enum {
extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags);
extern void bioset_exit(struct bio_set *);
extern int biovec_init_pool(mempool_t *pool, int pool_entries);
+extern void bioset_move(struct bio_set *dst, struct bio_set *src);
extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set *);
extern void bio_put(struct bio *);
diff --git a/include/linux/mempool.h b/include/linux/mempool.h
index 0c964ac107c2..20818919180c 100644
--- a/include/linux/mempool.h
+++ b/include/linux/mempool.h
@@ -47,6 +47,7 @@ extern int mempool_resize(mempool_t *pool, int new_min_nr);
extern void mempool_destroy(mempool_t *pool);
extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc;
extern void mempool_free(void *element, mempool_t *pool);
+extern void mempool_move(mempool_t *dst, mempool_t *src);
/*
* A mempool_alloc_t and mempool_free_t that get the memory from
diff --git a/mm/mempool.c b/mm/mempool.c
index b54f2c20e5e0..dd402653367b 100644
--- a/mm/mempool.c
+++ b/mm/mempool.c
@@ -181,6 +181,8 @@ int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn,
mempool_free_t *free_fn, void *pool_data,
gfp_t gfp_mask, int node_id)
{
+ memset(pool, 0, sizeof(*pool));
+
spin_lock_init(&pool->lock);
pool->min_nr = min_nr;
pool->pool_data = pool_data;
@@ -546,3 +548,19 @@ void mempool_free_pages(void *element, void *pool_data)
__free_pages(element, order);
}
EXPORT_SYMBOL(mempool_free_pages);
+
+void mempool_move(mempool_t *dst, mempool_t *src)
+{
+ BUG_ON(waitqueue_active(&src->wait));
+
+ spin_lock_init(&dst->lock);
+ dst->min_nr = src->min_nr;
+ dst->curr_nr = src->curr_nr;
+ memcpy(dst->elements, src->elements, sizeof(void *) * src->curr_nr);
+ dst->pool_data = src->pool_data;
+ dst->alloc = src->alloc;
+ dst->free = src->free;
+ init_waitqueue_head(&dst->wait);
+
+ memset(src, 0, sizeof(*src));
+}
--
Jens Axboe
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-07 14:46 ` Jens Axboe
@ 2018-06-07 15:41 ` Jens Axboe
2018-06-08 6:06 ` Li Wang
2018-06-08 6:12 ` Chunyu Hu
0 siblings, 2 replies; 9+ messages in thread
From: Jens Axboe @ 2018-06-07 15:41 UTC (permalink / raw)
To: Chunyu Hu
Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer,
linux-block, linux-kernel
On 6/7/18 8:46 AM, Jens Axboe wrote:
> On 6/7/18 12:33 AM, Chunyu Hu wrote:
>> kasan reported a user-after-free. I'm using a kvm machine, it panic
>> during boot. I'm using the latest linux tree. which contains below.
>>
>> commit d377535405686f735b90a8ad4ba269484cd7c96e
>> Author: Kent Overstreet <kent.overstreet@gmail.com>
>> Date: Tue Jun 5 05:26:33 2018 -0400
>>
>> dm: Use kzalloc for all structs with embedded biosets/mempools
>
> Can you try with the below? Li Wang, would be great if you could too.
Please try this one instead.
diff --git a/block/bio.c b/block/bio.c
index 595663e0281a..0616d86b15c6 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1967,6 +1967,21 @@ int bioset_init(struct bio_set *bs,
}
EXPORT_SYMBOL(bioset_init);
+int bioset_init_from_src(struct bio_set *new, struct bio_set *src)
+{
+ unsigned int pool_size = src->bio_pool.min_nr;
+ int flags;
+
+ flags = 0;
+ if (src->bvec_pool.min_nr)
+ flags |= BIOSET_NEED_BVECS;
+ if (src->rescue_workqueue)
+ flags |= BIOSET_NEED_RESCUER;
+
+ return bioset_init(new, pool_size, src->front_pad, flags);
+}
+EXPORT_SYMBOL(bioset_init_from_src);
+
#ifdef CONFIG_BLK_CGROUP
/**
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 98dff36b89a3..20a8d63754bf 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1953,9 +1953,10 @@ static void free_dev(struct mapped_device *md)
kvfree(md);
}
-static void __bind_mempools(struct mapped_device *md, struct dm_table *t)
+static int __bind_mempools(struct mapped_device *md, struct dm_table *t)
{
struct dm_md_mempools *p = dm_table_get_md_mempools(t);
+ int ret = 0;
if (dm_table_bio_based(t)) {
/*
@@ -1982,13 +1983,16 @@ static void __bind_mempools(struct mapped_device *md, struct dm_table *t)
bioset_initialized(&md->bs) ||
bioset_initialized(&md->io_bs));
- md->bs = p->bs;
- memset(&p->bs, 0, sizeof(p->bs));
- md->io_bs = p->io_bs;
- memset(&p->io_bs, 0, sizeof(p->io_bs));
+ ret = bioset_init_from_src(&md->bs, &p->bs);
+ if (ret)
+ goto out;
+ ret = bioset_init_from_src(&md->io_bs, &p->io_bs);
+ if (ret)
+ bioset_exit(&md->bs);
out:
/* mempool bind completed, no longer need any mempools in the table */
dm_table_free_md_mempools(t);
+ return ret;
}
/*
@@ -2033,6 +2037,7 @@ static struct dm_table *__bind(struct mapped_device *md, struct dm_table *t,
struct request_queue *q = md->queue;
bool request_based = dm_table_request_based(t);
sector_t size;
+ int ret;
lockdep_assert_held(&md->suspend_lock);
@@ -2068,7 +2073,11 @@ static struct dm_table *__bind(struct mapped_device *md, struct dm_table *t,
md->immutable_target = dm_table_get_immutable_target(t);
}
- __bind_mempools(md, t);
+ ret = __bind_mempools(md, t);
+ if (ret) {
+ old_map = ERR_PTR(ret);
+ goto out;
+ }
old_map = rcu_dereference_protected(md->map, lockdep_is_held(&md->suspend_lock));
rcu_assign_pointer(md->map, (void *)t);
@@ -2078,6 +2087,7 @@ static struct dm_table *__bind(struct mapped_device *md, struct dm_table *t,
if (old_map)
dm_sync_table(md);
+out:
return old_map;
}
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 810a8bee8f85..307682ac2f31 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -417,6 +417,7 @@ enum {
extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags);
extern void bioset_exit(struct bio_set *);
extern int biovec_init_pool(mempool_t *pool, int pool_entries);
+extern int bioset_init_from_src(struct bio_set *new, struct bio_set *src);
extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set *);
extern void bio_put(struct bio *);
--
Jens Axboe
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-07 15:41 ` Jens Axboe
@ 2018-06-08 6:06 ` Li Wang
2018-06-08 6:12 ` Chunyu Hu
1 sibling, 0 replies; 9+ messages in thread
From: Li Wang @ 2018-06-08 6:06 UTC (permalink / raw)
To: Jens Axboe
Cc: Chunyu Hu, Kent Overstreet, Coly Li, hch, darrick.wong, snitzer,
linux-block, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 4545 bytes --]
Hi Jens,
On Thu, Jun 7, 2018 at 11:41 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 6/7/18 8:46 AM, Jens Axboe wrote:
> > On 6/7/18 12:33 AM, Chunyu Hu wrote:
> >> kasan reported a user-after-free. I'm using a kvm machine, it panic
> >> during boot. I'm using the latest linux tree. which contains below.
> >>
> >> commit d377535405686f735b90a8ad4ba269484cd7c96e
> >> Author: Kent Overstreet <kent.overstreet@gmail.com>
> >> Date: Tue Jun 5 05:26:33 2018 -0400
> >>
> >> dm: Use kzalloc for all structs with embedded biosets/mempools
> >
> > Can you try with the below? Li Wang, would be great if you could too.
>
> Please try this one instead.
>
The problem was gone after applying this patch. Thanks!
>
>
> diff --git a/block/bio.c b/block/bio.c
> index 595663e0281a..0616d86b15c6 100644
> --- a/block/bio.c
> +++ b/block/bio.c
> @@ -1967,6 +1967,21 @@ int bioset_init(struct bio_set *bs,
> }
> EXPORT_SYMBOL(bioset_init);
>
> +int bioset_init_from_src(struct bio_set *new, struct bio_set *src)
> +{
> + unsigned int pool_size = src->bio_pool.min_nr;
> + int flags;
> +
> + flags = 0;
> + if (src->bvec_pool.min_nr)
> + flags |= BIOSET_NEED_BVECS;
> + if (src->rescue_workqueue)
> + flags |= BIOSET_NEED_RESCUER;
> +
> + return bioset_init(new, pool_size, src->front_pad, flags);
> +}
> +EXPORT_SYMBOL(bioset_init_from_src);
> +
> #ifdef CONFIG_BLK_CGROUP
>
> /**
> diff --git a/drivers/md/dm.c b/drivers/md/dm.c
> index 98dff36b89a3..20a8d63754bf 100644
> --- a/drivers/md/dm.c
> +++ b/drivers/md/dm.c
> @@ -1953,9 +1953,10 @@ static void free_dev(struct mapped_device *md)
> kvfree(md);
> }
>
> -static void __bind_mempools(struct mapped_device *md, struct dm_table *t)
> +static int __bind_mempools(struct mapped_device *md, struct dm_table *t)
> {
> struct dm_md_mempools *p = dm_table_get_md_mempools(t);
> + int ret = 0;
>
> if (dm_table_bio_based(t)) {
> /*
> @@ -1982,13 +1983,16 @@ static void __bind_mempools(struct mapped_device
> *md, struct dm_table *t)
> bioset_initialized(&md->bs) ||
> bioset_initialized(&md->io_bs));
>
> - md->bs = p->bs;
> - memset(&p->bs, 0, sizeof(p->bs));
> - md->io_bs = p->io_bs;
> - memset(&p->io_bs, 0, sizeof(p->io_bs));
> + ret = bioset_init_from_src(&md->bs, &p->bs);
> + if (ret)
> + goto out;
> + ret = bioset_init_from_src(&md->io_bs, &p->io_bs);
> + if (ret)
> + bioset_exit(&md->bs);
> out:
> /* mempool bind completed, no longer need any mempools in the
> table */
> dm_table_free_md_mempools(t);
> + return ret;
> }
>
> /*
> @@ -2033,6 +2037,7 @@ static struct dm_table *__bind(struct mapped_device
> *md, struct dm_table *t,
> struct request_queue *q = md->queue;
> bool request_based = dm_table_request_based(t);
> sector_t size;
> + int ret;
>
> lockdep_assert_held(&md->suspend_lock);
>
> @@ -2068,7 +2073,11 @@ static struct dm_table *__bind(struct mapped_device
> *md, struct dm_table *t,
> md->immutable_target = dm_table_get_immutable_target(t);
> }
>
> - __bind_mempools(md, t);
> + ret = __bind_mempools(md, t);
> + if (ret) {
> + old_map = ERR_PTR(ret);
> + goto out;
> + }
>
> old_map = rcu_dereference_protected(md->map,
> lockdep_is_held(&md->suspend_lock));
> rcu_assign_pointer(md->map, (void *)t);
> @@ -2078,6 +2087,7 @@ static struct dm_table *__bind(struct mapped_device
> *md, struct dm_table *t,
> if (old_map)
> dm_sync_table(md);
>
> +out:
> return old_map;
> }
>
> diff --git a/include/linux/bio.h b/include/linux/bio.h
> index 810a8bee8f85..307682ac2f31 100644
> --- a/include/linux/bio.h
> +++ b/include/linux/bio.h
> @@ -417,6 +417,7 @@ enum {
> extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int
> flags);
> extern void bioset_exit(struct bio_set *);
> extern int biovec_init_pool(mempool_t *pool, int pool_entries);
> +extern int bioset_init_from_src(struct bio_set *new, struct bio_set *src);
>
> extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set
> *);
> extern void bio_put(struct bio *);
>
> --
> Jens Axboe
>
>
--
Regards,
Li Wang
[-- Attachment #2: Type: text/html, Size: 6247 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17
2018-06-07 15:41 ` Jens Axboe
2018-06-08 6:06 ` Li Wang
@ 2018-06-08 6:12 ` Chunyu Hu
1 sibling, 0 replies; 9+ messages in thread
From: Chunyu Hu @ 2018-06-08 6:12 UTC (permalink / raw)
To: Jens Axboe
Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer,
linux-block, linux-kernel
On 7 June 2018 at 23:41, Jens Axboe <axboe@kernel.dk> wrote:
> On 6/7/18 8:46 AM, Jens Axboe wrote:
>> On 6/7/18 12:33 AM, Chunyu Hu wrote:
>>> kasan reported a user-after-free. I'm using a kvm machine, it panic
>>> during boot. I'm using the latest linux tree. which contains below.
>>>
>>> commit d377535405686f735b90a8ad4ba269484cd7c96e
>>> Author: Kent Overstreet <kent.overstreet@gmail.com>
>>> Date: Tue Jun 5 05:26:33 2018 -0400
>>>
>>> dm: Use kzalloc for all structs with embedded biosets/mempools
>>
>> Can you try with the below? Li Wang, would be great if you could too.
>
> Please try this one instead.
My kvm machine boot successfully with this and works fine under pressure.
Thanks for the fix.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-06-08 6:12 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAEemH2fbPv2wphVBHxSSS+KEK5S-rercXhvgLJ7YMOZnrTpexg@mail.gmail.com>
2018-06-06 8:41 ` Block IO issue in kernel-v4.17 Kent Overstreet
2018-06-06 14:18 ` Jens Axboe
2018-06-07 3:22 ` Li Wang
2018-06-07 6:26 ` Li Wang
2018-06-07 6:33 ` Chunyu Hu
2018-06-07 14:46 ` Jens Axboe
2018-06-07 15:41 ` Jens Axboe
2018-06-08 6:06 ` Li Wang
2018-06-08 6:12 ` Chunyu Hu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox