* Re: Block IO issue in kernel-v4.17 [not found] <CAEemH2fbPv2wphVBHxSSS+KEK5S-rercXhvgLJ7YMOZnrTpexg@mail.gmail.com> @ 2018-06-06 8:41 ` Kent Overstreet 2018-06-06 14:18 ` Jens Axboe 0 siblings, 1 reply; 9+ messages in thread From: Kent Overstreet @ 2018-06-06 8:41 UTC (permalink / raw) To: Li Wang; +Cc: Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote: > Hi BIO experts, > > I catch this panic issue on some kind of arches(x86_64, ppc64, > ppc64le..), it seems the root cause is very probably from BIO changes > from kernel-4.17-rc7. Plz take a look. That's the bioset changes, the fix is out and on its way in ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-06 8:41 ` Block IO issue in kernel-v4.17 Kent Overstreet @ 2018-06-06 14:18 ` Jens Axboe 2018-06-07 3:22 ` Li Wang ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Jens Axboe @ 2018-06-06 14:18 UTC (permalink / raw) To: Kent Overstreet, Li Wang Cc: Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel On 6/6/18 2:41 AM, Kent Overstreet wrote: > On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote: >> Hi BIO experts, >> >> I catch this panic issue on some kind of arches(x86_64, ppc64, >> ppc64le..), it seems the root cause is very probably from BIO changes >> from kernel-4.17-rc7. Plz take a look. > > That's the bioset changes, the fix is out and on its way in It's already in mainline, since about lunch time yesterday. -- Jens Axboe ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-06 14:18 ` Jens Axboe @ 2018-06-07 3:22 ` Li Wang 2018-06-07 6:26 ` Li Wang 2018-06-07 6:33 ` Chunyu Hu 2 siblings, 0 replies; 9+ messages in thread From: Li Wang @ 2018-06-07 3:22 UTC (permalink / raw) To: Jens Axboe Cc: Kent Overstreet, Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel [-- Attachment #1: Type: text/plain, Size: 634 bytes --] On Wed, Jun 6, 2018 at 10:18 PM, Jens Axboe <axboe@kernel.dk> wrote: > On 6/6/18 2:41 AM, Kent Overstreet wrote: > > On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote: > >> Hi BIO experts, > >> > >> I catch this panic issue on some kind of arches(x86_64, ppc64, > >> ppc64le..), it seems the root cause is very probably from BIO changes > >> from kernel-4.17-rc7. Plz take a look. > > > > That's the bioset changes, the fix is out and on its way in > > It's already in mainline, since about lunch time yesterday. > Which commit fix that? > > > -- > Jens Axboe > > -- Regards, Li Wang [-- Attachment #2: Type: text/html, Size: 1509 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-06 14:18 ` Jens Axboe 2018-06-07 3:22 ` Li Wang @ 2018-06-07 6:26 ` Li Wang 2018-06-07 6:33 ` Chunyu Hu 2 siblings, 0 replies; 9+ messages in thread From: Li Wang @ 2018-06-07 6:26 UTC (permalink / raw) To: Jens Axboe Cc: Kent Overstreet, Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel [-- Attachment #1: Type: text/plain, Size: 17933 bytes --] On Wed, Jun 6, 2018 at 10:18 PM, Jens Axboe <axboe@kernel.dk> wrote: > On 6/6/18 2:41 AM, Kent Overstreet wrote: > > On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote: > >> Hi BIO experts, > >> > >> I catch this panic issue on some kind of arches(x86_64, ppc64, > >> ppc64le..), it seems the root cause is very probably from BIO changes > >> from kernel-4.17-rc7. Plz take a look. > > > > That's the bioset changes, the fix is out and on its way in > > It's already in mainline, since about lunch time yesterday. > Do you mean this commit? commit d377535405686f735b90a8ad4ba269484cd7c96e Author: Kent Overstreet <kent.overstreet@gmail.com> Date: Tue Jun 5 05:26:33 2018 -0400 dm: Use kzalloc for all structs with embedded biosets/mempools Unfortunately, I pull the latest mainline kernel and test, still hit the panic. :( [ 21.832230] ------------[ cut here ]------------ [ 21.832244] list_add corruption. next->prev should be prev (c00000045f5801b0), but was 0000000000000000. (next=c000000461b2d5b8). [ 21.832265] WARNING: CPU: 12 PID: 2499 at lib/list_debug.c:25 .__list_add_valid+0xac/0xc0 [ 21.832270] Modules linked in: sunrpc nx_crypto sg pseries_rng xts vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp ibmveth dm_mirror dm_region_hash dm_log dm_mod [ 21.832295] CPU: 12 PID: 2499 Comm: stress Not tainted 4.17.0 #1 [ 21.832300] NIP: c0000000005021dc LR: c0000000005021d8 CTR: 00000000013abb8c [ 21.832304] REGS: c0000004623926d0 TRAP: 0700 Not tainted (4.17.0) [ 21.832308] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI> CR: 42244822 XER: 00000006 [ 21.832318] CFAR: c0000000000f93f0 SOFTE: 1 [ 21.832318] GPR00: c0000000005021d8 c000000462392950 c0000000013ac100 0000000000000075 [ 21.832318] GPR04: 0000000000000001 000000005b2baea6 0000000000000000 0000000000000000 [ 21.832318] GPR08: 0000000000000001 0000000000000007 0000000000000006 000000000000017f [ 21.832318] GPR12: 0000000028244824 c00000001ebddc00 0000000000000000 c000000461ccb800 [ 21.832318] GPR16: f00000000113d1e0 c0000004623931f0 0000000000000000 f00000000113d1c0 [ 21.832318] GPR20: 0000000000000000 0000000000000000 0000000000000000 c000000462393360 [ 21.832318] GPR24: 0000000000000000 c00000045f5801b0 c000000461b2d5b8 0000000000000001 [ 21.832318] GPR28: 0000000000000002 c000000462392af8 c00000045f5801a8 c000000462392ae0 [ 21.832372] NIP [c0000000005021dc] .__list_add_valid+0xac/0xc0 [ 21.832377] LR [c0000000005021d8] .__list_add_valid+0xa8/0xc0 [ 21.832381] Call Trace: [ 21.832385] [c000000462392950] [c0000000005021d8] .__list_add_valid+0xa8/0xc0 (unreliable) [ 21.832393] [c0000004623929c0] [c000000000159b28] .prepare_to_wait+0xc8/0xf0 [ 21.832400] [c000000462392a70] [c000000000268720] .mempool_alloc+0x180/0x1d0 [ 21.832407] [c000000462392b50] [c000000000496ee4] .bio_alloc_bioset+0x1c4/0x2b0 [ 21.832422] [c000000462392c20] [d0000000045f1578] .alloc_io+0x38/0x170 [dm_mod] [ 21.832435] [c000000462392cc0] [d0000000045f3824] .__split_and_process_bio+0x34/0x1e0 [dm_mod] [ 21.832449] [c000000462392d80] [d0000000045f17bc] .__dm_make_request+0x6c/0x100 [dm_mod] [ 21.832455] [c000000462392e20] [c0000000004a3368] .generic_make_request+0x1e8/0x420 [ 21.832460] [c000000462392ee0] [c0000000004a3654] .submit_bio+0xb4/0x1c0 [ 21.832466] [c000000462392fa0] [c0000000002d74c4] .__swap_writepage+0x394/0x540 [ 21.832472] [c000000462393090] [c000000000284ed8] .pageout.isra.36+0x288/0x490 [ 21.832478] [c000000462393180] [c000000000287928] .shrink_page_list+0x948/0x1090 [ 21.832484] [c0000004623932f0] [c000000000288b8c] .shrink_inactive_list+0x2ec/0x720 [ 21.832490] [c000000462393400] [c0000000002897c8] .shrink_node_memcg+0x238/0x7d0 [ 21.832495] [c000000462393550] [c000000000289eb8] .shrink_node+0x158/0x5e0 [ 21.832501] [c000000462393670] [c00000000028a838] .do_try_to_free_pages+0x118/0x4e0 [ 21.832507] [c000000462393760] [c00000000028ad08] .try_to_free_pages+0x108/0x290 [ 21.832512] [c000000462393860] [c000000000272508] .__alloc_pages_nodemask+0x638/0xf90 [ 21.832518] [c000000462393a40] [c0000000002f15c8] .alloc_pages_vma+0x98/0x240 [ 21.832524] [c000000462393b00] [c0000000002ba560] .__handle_mm_fault+0xbc0/0x1900 [ 21.832530] [c000000462393c30] [c0000000002bb408] .handle_mm_fault+0x168/0x280 [ 21.832535] [c000000462393cc0] [c000000000061cd4] .__do_page_fault+0x1c4/0x9b0 [ 21.832539] [c000000462393da0] [c0000000000624e8] .do_page_fault+0x28/0xb0 [ 21.832544] [c000000462393e30] [c00000000000a634] handle_page_fault+0x18/0x38 [ 21.832548] Instruction dump: [ 21.832551] 38630948 7d264b78 4bbf71dd 60000000 0fe00000 38600000 4bffffac 3c62ff8c [ 21.832558] 7d254b78 386308e0 4bbf71bd 60000000 <0fe00000> 38600000 4bffff8c 60000000 [ 21.832565] ---[ end trace c4d85974c8ca08fa ]--- [ 21.832569] ------------[ cut here ]------------ [ 21.832573] list_add corruption. next->prev should be prev (c00000045f5801b0), but was 0000000000000000. (next=c000000461b2d5b8). [ 21.832584] WARNING: CPU: 10 PID: 109 at lib/list_debug.c:25 .__list_add_valid+0xac/0xc0 [ 21.832589] Modules linked in: sunrpc nx_crypto sg pseries_rng xts vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp ibmveth dm_mirror dm_region_hash dm_log dm_mod [ 21.832609] CPU: 10 PID: 109 Comm: kswapd0 Tainted: G W 4.17.0 #1 [ 21.832613] NIP: c0000000005021dc LR: c0000000005021d8 CTR: 00000000013abb8c [ 21.832617] REGS: c000000472d2eb70 TRAP: 0700 Tainted: G W (4.17.0) [ 21.832621] MSR: 800000010282b032 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,TM[E]> CR: 42004022 XER: 00000006 [ 21.832632] CFAR: c0000000000f93f0 SOFTE: 1 [ 21.832632] GPR00: c0000000005021d8 c000000472d2edf0 c0000000013ac100 0000000000000075 [ 21.832632] GPR04: 0000000000000001 000000005b2bafba 0000000000000000 0000000000000000 [ 21.832632] GPR08: 0000000000000001 0000000000000007 0000000000000006 00000000000001a7 [ 21.832632] GPR12: 0000000028004024 c00000001ebf1200 0000000000000000 c000000461ccb800 [ 21.832632] GPR16: f000000001131de0 c000000472d2f690 0000000000000000 f000000001131dc0 [ 21.832632] GPR20: 0000000000000000 0000000000000000 0000000000000000 c000000472d2f800 [ 21.832632] GPR24: 0000000000000000 c00000045f5801b0 c000000461b2d5b8 0000000000000001 [ 21.832632] GPR28: 0000000000000002 c000000472d2ef98 c00000045f5801a8 c000000472d2ef80 [ 21.832675] NIP [c0000000005021dc] .__list_add_valid+0xac/0xc0 [ 21.832679] LR [c0000000005021d8] .__list_add_valid+0xa8/0xc0 [ 21.832682] Call Trace: [ 21.832684] [c000000472d2edf0] [c0000000005021d8] .__list_add_valid+0xa8/0xc0 (unreliable) [ 21.832689] [c000000472d2ee60] [c000000000159b28] .prepare_to_wait+0xc8/0xf0 [ 21.832692] [c000000472d2ef10] [c000000000268720] .mempool_alloc+0x180/0x1d0 [ 21.832697] [c000000472d2eff0] [c000000000496ee4] .bio_alloc_bioset+0x1c4/0x2b0 [ 21.832707] [c000000472d2f0c0] [d0000000045f1578] .alloc_io+0x38/0x170 [dm_mod] [ 21.832716] [c000000472d2f160] [d0000000045f3824] .__split_and_process_bio+0x34/0x1e0 [dm_mod] [ 21.832727] [c000000472d2f220] [d0000000045f17bc] .__dm_make_request+0x6c/0x100 [dm_mod] [ 21.832731] [c000000472d2f2c0] [c0000000004a3368] .generic_make_request+0x1e8/0x420 [ 21.832735] [c000000472d2f380] [c0000000004a3654] .submit_bio+0xb4/0x1c0 [ 21.832739] [c000000472d2f440] [c0000000002d74c4] .__swap_writepage+0x394/0x540 [ 21.832743] [c000000472d2f530] [c000000000284ed8] .pageout.isra.36+0x288/0x490 [ 21.832747] [c000000472d2f620] [c000000000287928] .shrink_page_list+0x948/0x1090 [ 21.832751] [c000000472d2f790] [c000000000288b8c] .shrink_inactive_list+0x2ec/0x720 [ 21.832755] [c000000472d2f8a0] [c0000000002897c8] .shrink_node_memcg+0x238/0x7d0 [ 21.832760] [c000000472d2f9f0] [c000000000289eb8] .shrink_node+0x158/0x5e0 [ 21.832763] [c000000472d2fb10] [c00000000028b590] .balance_pgdat+0x200/0x470 [ 21.832767] [c000000472d2fc50] [c00000000028b9bc] .kswapd+0x1bc/0x5f0 [ 21.832771] [c000000472d2fd70] [c000000000124d78] .kthread+0x158/0x1a0 [ 21.832776] [c000000472d2fe30] [c00000000000b654] .ret_from_kernel_thread+0x58/0x84 [ 21.832779] Instruction dump: [ 21.832781] 38630948 7d264b78 4bbf71dd 60000000 0fe00000 38600000 4bffffac 3c62ff8c [ 21.832787] 7d254b78 386308e0 4bbf71bd 60000000 <0fe00000> 38600000 4bffff8c 60000000 [ 21.832792] ---[ end trace c4d85974c8ca08fb ]--- [ 21.833118] Unable to handle kernel paging request for instruction fetch [ 21.833121] Faulting instruction address: 0x00000000 [ 21.833127] Oops: Kernel access of bad area, sig: 11 [#1] [ 21.833129] BE SMP NR_CPUS=2048 NUMA pSeries [ 21.833134] Modules linked in: sunrpc nx_crypto sg pseries_rng xts vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp ibmveth dm_mirror dm_region_hash dm_log dm_mod [ 21.833148] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G W 4.17.0 #1 [ 21.833151] NIP: 0000000000000000 LR: c000000000159e8c CTR: 0000000000000000 [ 21.833155] REGS: c00000047ff93260 TRAP: 0400 Tainted: G W (4.17.0) [ 21.833158] MSR: 8000000040009032 <SF,EE,ME,IR,DR,RI> CR: 24028028 XER: 20000006 [ 21.833166] CFAR: c000000000008874 SOFTE: 1 [ 21.833166] GPR00: c00000000015a09c c00000047ff934e0 0000000000000000 c000000461b2d5a0 [ 21.833166] GPR04: 0000000000000003 0000000000000000 0000000000000000 0000000000000000 [ 21.833166] GPR08: c00000047ff93610 c000000479ae7880 0000000000000000 d000000004606758 [ 21.833166] GPR12: 0000000024022022 c00000001ebf1200 c000000474557f90 0000000000200042 [ 21.833166] GPR16: 00000000ffff9358 c00000047ff90000 0000000000000000 0000000000000001 [ 21.833166] GPR20: c000000000df7080 0000000000000005 c00000047ff93610 0000000000000001 [ 21.833166] GPR24: 0000000000000000 0000000000000000 0000000000000003 0000000000000000 [ 21.833166] GPR28: c00000045f5801b0 0000000000000000 c000000461b2d5a0 ffffffffffffffe8 [ 21.833208] NIP [0000000000000000] (null) [ 21.833212] LR [c000000000159e8c] .__wake_up_common+0xdc/0x240 [ 21.833215] Call Trace: [ 21.833217] [c00000047ff934e0] [c00000047ff935c0] 0xc00000047ff935c0 (unreliable) [ 21.833222] [c00000047ff935a0] [c00000000015a09c] .__wake_up_common_lock+0xac/0x100 [ 21.833227] [c00000047ff93680] [c000000000267dc8] .mempool_free+0xc8/0xe0 [ 21.833232] [c00000047ff93700] [c0000000004971ac] .bio_free+0x4c/0xa0 [ 21.833242] [c00000047ff93780] [d0000000045f1998] .dec_pending+0x128/0x310 [dm_mod] [ 21.833253] [c00000047ff93850] [d0000000045f1edc] .clone_endio+0xcc/0x180 [dm_mod] [ 21.833258] [c00000047ff938e0] [c000000000498c84] .bio_endio+0x184/0x290 [ 21.833262] [c00000047ff93990] [c0000000004a3d20] .blk_update_request+0xf0/0x4a0 [ 21.833267] [c00000047ff93a40] [c000000000642ae8] .scsi_end_request+0x48/0x270 [ 21.833272] [c00000047ff93af0] [c000000000642f44] .scsi_io_completion+0x234/0x710 [ 21.833277] [c00000047ff93bd0] [c000000000637cc8] .scsi_finish_command+0x148/0x1b0 [ 21.833282] [c00000047ff93c60] [c000000000641e18] .scsi_softirq_done+0x198/0x1f0 [ 21.833286] [c00000047ff93d00] [c0000000004aee24] .blk_done_softirq+0xc4/0xf0 [ 21.833292] [c00000047ff93d90] [c0000000009829a0] .__do_softirq+0x150/0x3a8 [ 21.833297] [c00000047ff93e90] [c000000000101d68] .irq_exit+0x198/0x1b0 [ 21.833302] [c00000047ff93f10] [c000000000016024] .__do_irq+0x94/0x1f0 [ 21.833306] [c00000047ff93f90] [c00000000002779c] .call_do_irq+0x14/0x24 [ 21.833311] [c0000004745577a0] [c00000000001620c] .do_IRQ+0x8c/0x100 [ 21.833315] [c000000474557840] [c000000000008c24] hardware_interrupt_common+0x114/0x120 [ 21.833323] --- interrupt: 501 at .plpar_hcall_norets+0x14/0x20 [ 21.833323] LR = .check_and_cede_processor+0x24/0x40 [ 21.833328] [c000000474557b30] [7fffffffffffffff] 0x7fffffffffffffff (unreliable) [ 21.833333] [c000000474557ba0] [c00000000076565c] .dedicated_cede_loop+0x4c/0x130 [ 21.833339] [c000000474557c20] [c000000000762d9c] .cpuidle_enter_state+0xac/0x390 [ 21.833343] [c000000474557cd0] [c00000000013b8cc] .call_cpuidle+0x3c/0x70 [ 21.833348] [c000000474557d40] [c00000000013be38] .do_idle+0x2e8/0x390 [ 21.833352] [c000000474557e20] [c00000000013c098] .cpu_startup_entry+0x28/0x30 [ 21.833357] [c000000474557ea0] [c000000000046980] .start_secondary+0x4d0/0x520 [ 21.833362] [c000000474557f90] [c00000000000ab70] start_secondary_prolog+0x10/0x14 [ 21.833366] Instruction dump: [ 21.833369] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX [ 21.833374] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX [ 21.833382] ---[ end trace c4d85974c8ca08fc ]--- [ 21.835408] [ 22.835471] Kernel panic - not syncing: Fatal exception in interrupt [ 23.884733] WARNING: CPU: 10 PID: 0 at drivers/tty/vt/vt.c:3885 .do_unblank_screen+0x268/0x270 [ 23.884740] Modules linked in: sunrpc nx_crypto sg pseries_rng xts vmx_crypto ip_tables xfs libcrc32c sd_mod ibmvscsi scsi_transport_srp ibmveth dm_mirror dm_region_hash dm_log dm_mod [ 23.884766] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G D W 4.17.0 #1 [ 23.884772] NIP: c0000000005bb758 LR: c0000000005bb524 CTR: c00000000002ae00 [ 23.884778] REGS: c00000047ff92cc0 TRAP: 0700 Tainted: G D W (4.17.0) [ 23.884783] MSR: 8000000000021032 <SF,ME,IR,DR,RI> CR: 48022042 XER: 00000009 [ 23.884794] CFAR: c0000000005bb538 SOFTE: 1 [ 23.884794] GPR00: c0000000005bb714 c00000047ff92f40 c0000000013ac100 0000000000000000 [ 23.884794] GPR04: 0000000000000001 000000000000000b 0000000020646d5f 6d5f726567696f6e [ 23.884794] GPR08: c00000000168c100 0000000000000000 0000000000000000 6e67207874732076 [ 23.884794] GPR12: 0000000088022044 c00000001ebf1200 c000000474557f90 0000000000200042 [ 23.884794] GPR16: 00000000ffff9358 c00000047ff90000 0000000000000000 0000000000000001 [ 23.884794] GPR20: c000000000df7080 0000000000000005 c00000047ff93610 0000000000000001 [ 23.884794] GPR24: 0000000000000000 0000000000000000 c0000000011da080 c0000000015767c0 [ 23.884794] GPR28: c000000001576798 0000000000000000 c000000001685340 c0000000015767c0 [ 23.884863] NIP [c0000000005bb758] .do_unblank_screen+0x268/0x270 [ 23.884868] LR [c0000000005bb524] .do_unblank_screen+0x34/0x270 [ 23.884873] Call Trace: [ 23.884877] [c00000047ff92f40] [c0000000005bb714] .do_unblank_screen+0x224/0x270 (unreliable) [ 23.884887] [c00000047ff92fd0] [c0000000004e88a0] .bust_spinlocks+0x30/0x70 [ 23.884894] [c00000047ff93040] [c0000000000f9cf8] .panic+0x1b0/0x308 [ 23.884901] [c00000047ff930e0] [c000000000022e5c] .oops_end+0x1cc/0x1e0 [ 23.884908] [c00000047ff93170] [c000000000062648] .bad_page_fault+0xd8/0x144 [ 23.884916] [c00000047ff931f0] [c00000000000a650] handle_page_fault+0x34/0x38 [ 23.884924] --- interrupt: 400 at (null) [ 23.884924] LR = .__wake_up_common+0xdc/0x240 [ 23.884932] [c00000047ff934e0] [c00000047ff935c0] 0xc00000047ff935c0 (unreliable) [ 23.884940] [c00000047ff935a0] [c00000000015a09c] .__wake_up_common_lock+0xac/0x100 [ 23.884948] [c00000047ff93680] [c000000000267dc8] .mempool_free+0xc8/0xe0 [ 23.884955] [c00000047ff93700] [c0000000004971ac] .bio_free+0x4c/0xa0 [ 23.884973] [c00000047ff93780] [d0000000045f1998] .dec_pending+0x128/0x310 [dm_mod] [ 23.884991] [c00000047ff93850] [d0000000045f1edc] .clone_endio+0xcc/0x180 [dm_mod] [ 23.884998] [c00000047ff938e0] [c000000000498c84] .bio_endio+0x184/0x290 [ 23.885005] [c00000047ff93990] [c0000000004a3d20] .blk_update_request+0xf0/0x4a0 [ 23.885013] [c00000047ff93a40] [c000000000642ae8] .scsi_end_request+0x48/0x270 [ 23.885020] [c00000047ff93af0] [c000000000642f44] .scsi_io_completion+0x234/0x710 [ 23.885028] [c00000047ff93bd0] [c000000000637cc8] .scsi_finish_command+0x148/0x1b0 [ 23.885035] [c00000047ff93c60] [c000000000641e18] .scsi_softirq_done+0x198/0x1f0 [ 23.885042] [c00000047ff93d00] [c0000000004aee24] .blk_done_softirq+0xc4/0xf0 [ 23.885051] [c00000047ff93d90] [c0000000009829a0] .__do_softirq+0x150/0x3a8 [ 23.885058] [c00000047ff93e90] [c000000000101d68] .irq_exit+0x198/0x1b0 [ 23.885065] [c00000047ff93f10] [c000000000016024] .__do_irq+0x94/0x1f0 [ 23.885072] [c00000047ff93f90] [c00000000002779c] .call_do_irq+0x14/0x24 [ 23.885079] [c0000004745577a0] [c00000000001620c] .do_IRQ+0x8c/0x100 [ 23.885086] [c000000474557840] [c000000000008c24] hardware_interrupt_common+0x114/0x120 [ 23.885096] --- interrupt: 501 at .plpar_hcall_norets+0x14/0x20 [ 23.885096] LR = .check_and_cede_processor+0x24/0x40 [ 23.885104] [c000000474557b30] [7fffffffffffffff] 0x7fffffffffffffff (unreliable) [ 23.885112] [c000000474557ba0] [c00000000076565c] .dedicated_cede_loop+0x4c/0x130 [ 23.885120] [c000000474557c20] [c000000000762d9c] .cpuidle_enter_state+0xac/0x390 [ 23.885128] [c000000474557cd0] [c00000000013b8cc] .call_cpuidle+0x3c/0x70 [ 23.885134] [c000000474557d40] [c00000000013be38] .do_idle+0x2e8/0x390 [ 23.885141] [c000000474557e20] [c00000000013c098] .cpu_startup_entry+0x28/0x30 [ 23.885148] [c000000474557ea0] [c000000000046980] .start_secondary+0x4d0/0x520 [ 23.885156] [c000000474557f90] [c00000000000ab70] start_secondary_prolog+0x10/0x14 [ 23.885162] Instruction dump: [ 23.885167] 39487a00 3c62ffe7 3863aab0 e94a0000 7d2407b4 7c845214 4bbd6671 60000000 [ 23.885178] 39200001 3d42003f 912a2080 4bfffe60 <0fe00000> 4bfffde0 38600000 4bfffd8c [ 23.885189] ---[ end trace c4d85974c8ca08fd ]--- > > > -- > Jens Axboe > > -- Regards, Li Wang [-- Attachment #2: Type: text/html, Size: 19448 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-06 14:18 ` Jens Axboe 2018-06-07 3:22 ` Li Wang 2018-06-07 6:26 ` Li Wang @ 2018-06-07 6:33 ` Chunyu Hu 2018-06-07 14:46 ` Jens Axboe 2 siblings, 1 reply; 9+ messages in thread From: Chunyu Hu @ 2018-06-07 6:33 UTC (permalink / raw) To: Jens Axboe Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel kasan reported a user-after-free. I'm using a kvm machine, it panic during boot. I'm using the latest linux tree. which contains below. commit d377535405686f735b90a8ad4ba269484cd7c96e Author: Kent Overstreet <kent.overstreet@gmail.com> Date: Tue Jun 5 05:26:33 2018 -0400 dm: Use kzalloc for all structs with embedded biosets/mempools [ 58.836774] ================================================================== [ 58.839974] BUG: KASAN: use-after-free in __wake_up_common+0x7c7/0x880 [ 58.841988] Read of size 8 at addr ffff88025a47c3e8 by task kswapd0/66 [ 58.843986] [ 58.845644] [ 58.846127] Allocated by task 956: [ 58.847249] [ 58.847731] Freed by task 956: [ 58.848856] [ 58.849336] The buggy address belongs to the object at ffff88025a47c000 [ 58.849336] which belongs to the cache names_cache of size 4096 [ 58.853276] The buggy address is located 1000 bytes inside of [ 58.853276] 4096-byte region [ffff88025a47c000, ffff88025a47d000) [ 58.856924] The buggy address belongs to the page: [ 58.858411] page:ffffea0009691f00 count:1 mapcount:0 mapping:0000000000000000 index:0x0 compound_mapcount: 0 [ 58.861443] flags: 0x6fffff80008100(slab|head) [ 58.862820] raw: 006fffff80008100 0000000000000000 0000000000000000 0000000100010001 [ 58.865197] raw: dead000000000100 dead000000000200 ffff88012d940a00 0000000000000000 [ 58.867590] page dumped because: kasan: bad access detected [ 58.869409] [ 58.869933] Memory state around the buggy address: [ 58.871116] ffff88025a47c280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 58.872195] ffff88025a47c300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 58.873295] >ffff88025a47c380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 58.874358] ^ [ 58.875334] ffff88025a47c400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 58.876395] ffff88025a47c480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 58.877453] ================================================================== [ 58.878547] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 [ 58.879708] PGD 0 P4D 0 [ 58.880107] Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN PTI [ 58.881013] CPU: 0 PID: 66 Comm: kswapd0 Tainted: G B W 4.17.0.fi+ #41 [ 58.881944] Hardware name: Red Hat KVM, BIOS 0.0.0 02/06/2015 [ 58.882646] RIP: 0010: (null) [ 58.883106] Code: Bad RIP value. [ 58.883520] RSP: 0018:ffff88012ec07818 EFLAGS: 00010086 [ 58.884158] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 58.885020] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88025a47c3d0 [ 58.886576] RBP: ffff880257960418 R08: ffff88025a47c3d0 R09: fffffbfff09587a4 [ 58.888370] R10: fffffbfff09587a4 R11: ffffffff84ac3d23 R12: ffffffffffffffe8 [ 58.890257] R13: dffffc0000000000 R14: 0000000000000000 R15: ffff88012ec07910 [ 58.892113] FS: 0000000000000000(0000) GS:ffff88012ec00000(0000) knlGS:0000000000000000 [ 58.894194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 58.895648] CR2: ffffffffffffffd6 CR3: 0000000004214000 CR4: 00000000001406f0 [ 58.897468] Call Trace: [ 58.898117] <IRQ> [ 58.898716] ? __wake_up_common+0x18e/0x880 [ 58.899843] ? wait_woken+0x340/0x340 [ 58.900842] ? do_raw_spin_lock+0xcf/0x220 [ 58.901665] ? __wake_up_common_lock+0xe3/0x170 [ 58.902439] ? __wake_up_common+0x880/0x880 [ 58.903077] ? time_hardirqs_off+0x3e/0x4f0 [ 58.903742] ? _raw_spin_unlock_irqrestore+0x45/0xa0 [ 58.904486] ? mempool_free+0x270/0x3a0 [ 58.905185] ? bio_free+0x104/0x190 [ 58.905722] ? bio_put+0xb9/0x120 [ 58.906235] ? dec_pending+0x3cd/0xbe0 [dm_mod] [ 58.906960] ? time_hardirqs_off+0x30/0x4f0 [ 58.907602] ? debug_check_no_locks_freed+0x260/0x260 [ 58.908389] ? alloc_io+0x820/0x820 [dm_mod] [ 58.909130] ? linear_status+0x1b0/0x1b0 [dm_mod] [ 58.909839] ? clone_endio+0x1ed/0x890 [dm_mod] [ 58.910691] ? bio_disassociate_task+0x16e/0x450 [ 58.911453] ? dm_get_queue_limits+0x110/0x110 [dm_mod] [ 58.912288] ? check_preemption_disabled+0x36/0x2a0 [ 58.913075] ? dm_get_queue_limits+0x110/0x110 [dm_mod] [ 58.913919] ? bio_endio+0x423/0x8b0 [ 58.914513] ? blk_update_request+0x295/0xe40 [ 58.915215] ? virtqueue_get_buf_ctx+0x3b0/0xa60 [virtio_ring] [ 58.916171] ? blk_mq_end_request+0x56/0x390 [ 58.916877] ? blk_mq_complete_request+0x36a/0x6e0 [ 58.917857] ? virtblk_done+0x1bc/0x450 [virtio_blk] [ 58.918690] ? 0xffffffffa0158000 [ 58.919260] ? __lock_is_held+0xb6/0x170 [ 58.919917] ? check_preemption_disabled+0x36/0x2a0 [ 58.920730] ? 0xffffffffa0158000 [ 58.921300] ? vring_interrupt+0x170/0x280 [virtio_ring] [ 58.922176] ? vring_alloc_queue+0x400/0x400 [virtio_ring] [ 58.923080] ? __handle_irq_event_percpu+0x117/0x9a0 [ 58.923908] ? handle_irq_event_percpu+0x77/0x180 [ 58.924691] ? __handle_irq_event_percpu+0x9a0/0x9a0 [ 58.925519] ? do_raw_spin_unlock+0x156/0x250 [ 58.926257] ? handle_irq_event+0xc6/0x1a0 [ 58.926943] ? handle_edge_irq+0x229/0xd40 [ 58.927747] ? handle_irq+0x2e2/0x5fd [ 58.928528] ? check_preemption_disabled+0x36/0x2a0 [ 58.929545] ? do_IRQ+0xa7/0x240 [ 58.930241] ? common_interrupt+0xf/0xf [ 58.931072] </IRQ> [ 58.931576] ? lock_acquire+0x184/0x470 [ 58.932395] ? list_lru_count_one+0xb8/0x3a0 [ 58.933330] ? list_lru_count_one+0x86/0x3a0 [ 58.934362] ? super_cache_count+0x152/0x2f0 [ 58.935268] ? shrink_slab.part.24+0x1fe/0xc60 [ 58.936205] ? mem_cgroup_from_task+0x180/0x180 [ 58.937162] ? prepare_kswapd_sleep+0x160/0x160 [ 58.938101] ? mem_cgroup_iter+0x165/0xc60 [ 58.938955] ? shrink_slab+0x9e/0xd0 [ 58.939707] ? shrink_node+0x3f1/0x17b0 [ 58.940675] ? shrink_node_memcg+0x1f10/0x1f10 [ 58.941794] ? mem_cgroup_iter+0x165/0xc60 [ 58.942668] ? mem_cgroup_nr_lru_pages+0xe0/0xe0 [ 58.943642] ? inactive_list_is_low+0x1f3/0x6e0 [ 58.944598] ? balance_pgdat+0x2c9/0x950 [ 58.945436] ? mem_cgroup_shrink_node+0x7d0/0x7d0 [ 58.946432] ? preempt_count_sub+0x101/0x190 [ 58.947340] ? check_preemption_disabled+0x36/0x2a0 [ 58.948373] ? kswapd+0x5c1/0x1060 [ 58.949109] ? balance_pgdat+0x950/0x950 [ 58.949948] ? __kthread_parkme+0x84/0x240 [ 58.950816] ? __kthread_parkme+0xff/0x240 [ 58.951685] ? finish_wait+0x3f0/0x3f0 [ 58.952484] ? schedule+0x92/0x230 [ 58.953202] ? balance_pgdat+0x950/0x950 [ 58.954030] ? balance_pgdat+0x950/0x950 [ 58.954861] ? kthread+0x37d/0x500 [ 58.955592] ? kthread_create_worker_on_cpu+0xe0/0xe0 [ 58.956654] ? ret_from_fork+0x3a/0x50 [ 58.957457] Modules linked in: sunrpc snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul ghash_clmulni_intel snd_hda_codec snd_hda_core snd_hwdep vfat fat iTCO_wdt iTCO_vendor_support snd_seq snd_seq_device snd_pcm pcspkr virtio_balloon snd_timer i2c_i801 sg lpc_ich snd soundcore shpchp ip_tables xfs libcrc32c sr_mod cdrom virtio_net net_failover virtio_scsi failover virtio_blk virtio_console bochs_drm drm_kms_helper syscopyarea sysfillrect 8139too sysimgblt fb_sys_fops ttm drm crc32c_intel ahci libahci 8139cp libata serio_raw mii i2c_core virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod [ 58.968856] CR2: 0000000000000000 [ 58.969583] ---[ end trace 70ad259a1fd9713f ]--- [ 58.970624] RIP: 0010: (null) [ 58.971417] Code: Bad RIP value. [ 58.972095] RSP: 0018:ffff88012ec07818 EFLAGS: 00010086 [ 58.973181] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 58.974635] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88025a47c3d0 [ 58.976060] RBP: ffff880257960418 R08: ffff88025a47c3d0 R09: fffffbfff09587a4 [ 58.977495] R10: fffffbfff09587a4 R11: ffffffff84ac3d23 R12: ffffffffffffffe8 [ 58.978922] R13: dffffc0000000000 R14: 0000000000000000 R15: ffff88012ec07910 [ 58.980365] FS: 0000000000000000(0000) GS:ffff88012ec00000(0000) knlGS:0000000000000000 [ 58.981984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 58.983139] CR2: ffffffffffffffd6 CR3: 0000000004214000 CR4: 00000000001406f0 [ 58.984568] Kernel panic - not syncing: Fatal exception in interrupt [ 60.070024] Shutting down cpus with NMI [ 60.070741] Kernel Offset: disabled [ 60.071317] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- On 6 June 2018 at 22:18, Jens Axboe <axboe@kernel.dk> wrote: > On 6/6/18 2:41 AM, Kent Overstreet wrote: >> On Wed, Jun 06, 2018 at 04:37:25PM +0800, Li Wang wrote: >>> Hi BIO experts, >>> >>> I catch this panic issue on some kind of arches(x86_64, ppc64, >>> ppc64le..), it seems the root cause is very probably from BIO changes >>> from kernel-4.17-rc7. Plz take a look. >> >> That's the bioset changes, the fix is out and on its way in > > It's already in mainline, since about lunch time yesterday. > > > -- > Jens Axboe > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-07 6:33 ` Chunyu Hu @ 2018-06-07 14:46 ` Jens Axboe 2018-06-07 15:41 ` Jens Axboe 0 siblings, 1 reply; 9+ messages in thread From: Jens Axboe @ 2018-06-07 14:46 UTC (permalink / raw) To: Chunyu Hu Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel On 6/7/18 12:33 AM, Chunyu Hu wrote: > kasan reported a user-after-free. I'm using a kvm machine, it panic > during boot. I'm using the latest linux tree. which contains below. > > commit d377535405686f735b90a8ad4ba269484cd7c96e > Author: Kent Overstreet <kent.overstreet@gmail.com> > Date: Tue Jun 5 05:26:33 2018 -0400 > > dm: Use kzalloc for all structs with embedded biosets/mempools Can you try with the below? Li Wang, would be great if you could too. diff --git a/block/bio.c b/block/bio.c index 595663e0281a..45bdee67d28b 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1967,6 +1967,27 @@ int bioset_init(struct bio_set *bs, } EXPORT_SYMBOL(bioset_init); +void bioset_move(struct bio_set *dst, struct bio_set *src) +{ + dst->bio_slab = src->bio_slab; + dst->front_pad = src->front_pad; + mempool_move(&dst->bio_pool, &src->bio_pool); + mempool_move(&dst->bvec_pool, &src->bvec_pool); +#if defined(CONFIG_BLK_DEV_INTEGRITY) + mempool_move(&dst->bio_integrity_pool, &src->bio_integrity_pool); + mempool_move(&dst->bvec_integrity_pool, &src->bvec_integrity_pool); +#endif + BUG_ON(!bio_list_empty(&src->rescue_list)); + BUG_ON(work_pending(&src->rescue_work)); + spin_lock_init(&dst->rescue_lock); + bio_list_init(&dst->rescue_list); + INIT_WORK(&dst->rescue_work, bio_alloc_rescue); + dst->rescue_workqueue = src->rescue_workqueue; + + memset(src, 0, sizeof(*src)); +} +EXPORT_SYMBOL(bioset_move); + #ifdef CONFIG_BLK_CGROUP /** diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 98dff36b89a3..87f636815baf 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1982,10 +1982,8 @@ static void __bind_mempools(struct mapped_device *md, struct dm_table *t) bioset_initialized(&md->bs) || bioset_initialized(&md->io_bs)); - md->bs = p->bs; - memset(&p->bs, 0, sizeof(p->bs)); - md->io_bs = p->io_bs; - memset(&p->io_bs, 0, sizeof(p->io_bs)); + bioset_move(&md->bs, &p->bs); + bioset_move(&md->io_bs, &p->io_bs); out: /* mempool bind completed, no longer need any mempools in the table */ dm_table_free_md_mempools(t); diff --git a/include/linux/bio.h b/include/linux/bio.h index 810a8bee8f85..7581231dd0a3 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -417,6 +417,7 @@ enum { extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags); extern void bioset_exit(struct bio_set *); extern int biovec_init_pool(mempool_t *pool, int pool_entries); +extern void bioset_move(struct bio_set *dst, struct bio_set *src); extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set *); extern void bio_put(struct bio *); diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 0c964ac107c2..20818919180c 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -47,6 +47,7 @@ extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; extern void mempool_free(void *element, mempool_t *pool); +extern void mempool_move(mempool_t *dst, mempool_t *src); /* * A mempool_alloc_t and mempool_free_t that get the memory from diff --git a/mm/mempool.c b/mm/mempool.c index b54f2c20e5e0..dd402653367b 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -181,6 +181,8 @@ int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id) { + memset(pool, 0, sizeof(*pool)); + spin_lock_init(&pool->lock); pool->min_nr = min_nr; pool->pool_data = pool_data; @@ -546,3 +548,19 @@ void mempool_free_pages(void *element, void *pool_data) __free_pages(element, order); } EXPORT_SYMBOL(mempool_free_pages); + +void mempool_move(mempool_t *dst, mempool_t *src) +{ + BUG_ON(waitqueue_active(&src->wait)); + + spin_lock_init(&dst->lock); + dst->min_nr = src->min_nr; + dst->curr_nr = src->curr_nr; + memcpy(dst->elements, src->elements, sizeof(void *) * src->curr_nr); + dst->pool_data = src->pool_data; + dst->alloc = src->alloc; + dst->free = src->free; + init_waitqueue_head(&dst->wait); + + memset(src, 0, sizeof(*src)); +} -- Jens Axboe ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-07 14:46 ` Jens Axboe @ 2018-06-07 15:41 ` Jens Axboe 2018-06-08 6:06 ` Li Wang 2018-06-08 6:12 ` Chunyu Hu 0 siblings, 2 replies; 9+ messages in thread From: Jens Axboe @ 2018-06-07 15:41 UTC (permalink / raw) To: Chunyu Hu Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel On 6/7/18 8:46 AM, Jens Axboe wrote: > On 6/7/18 12:33 AM, Chunyu Hu wrote: >> kasan reported a user-after-free. I'm using a kvm machine, it panic >> during boot. I'm using the latest linux tree. which contains below. >> >> commit d377535405686f735b90a8ad4ba269484cd7c96e >> Author: Kent Overstreet <kent.overstreet@gmail.com> >> Date: Tue Jun 5 05:26:33 2018 -0400 >> >> dm: Use kzalloc for all structs with embedded biosets/mempools > > Can you try with the below? Li Wang, would be great if you could too. Please try this one instead. diff --git a/block/bio.c b/block/bio.c index 595663e0281a..0616d86b15c6 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1967,6 +1967,21 @@ int bioset_init(struct bio_set *bs, } EXPORT_SYMBOL(bioset_init); +int bioset_init_from_src(struct bio_set *new, struct bio_set *src) +{ + unsigned int pool_size = src->bio_pool.min_nr; + int flags; + + flags = 0; + if (src->bvec_pool.min_nr) + flags |= BIOSET_NEED_BVECS; + if (src->rescue_workqueue) + flags |= BIOSET_NEED_RESCUER; + + return bioset_init(new, pool_size, src->front_pad, flags); +} +EXPORT_SYMBOL(bioset_init_from_src); + #ifdef CONFIG_BLK_CGROUP /** diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 98dff36b89a3..20a8d63754bf 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1953,9 +1953,10 @@ static void free_dev(struct mapped_device *md) kvfree(md); } -static void __bind_mempools(struct mapped_device *md, struct dm_table *t) +static int __bind_mempools(struct mapped_device *md, struct dm_table *t) { struct dm_md_mempools *p = dm_table_get_md_mempools(t); + int ret = 0; if (dm_table_bio_based(t)) { /* @@ -1982,13 +1983,16 @@ static void __bind_mempools(struct mapped_device *md, struct dm_table *t) bioset_initialized(&md->bs) || bioset_initialized(&md->io_bs)); - md->bs = p->bs; - memset(&p->bs, 0, sizeof(p->bs)); - md->io_bs = p->io_bs; - memset(&p->io_bs, 0, sizeof(p->io_bs)); + ret = bioset_init_from_src(&md->bs, &p->bs); + if (ret) + goto out; + ret = bioset_init_from_src(&md->io_bs, &p->io_bs); + if (ret) + bioset_exit(&md->bs); out: /* mempool bind completed, no longer need any mempools in the table */ dm_table_free_md_mempools(t); + return ret; } /* @@ -2033,6 +2037,7 @@ static struct dm_table *__bind(struct mapped_device *md, struct dm_table *t, struct request_queue *q = md->queue; bool request_based = dm_table_request_based(t); sector_t size; + int ret; lockdep_assert_held(&md->suspend_lock); @@ -2068,7 +2073,11 @@ static struct dm_table *__bind(struct mapped_device *md, struct dm_table *t, md->immutable_target = dm_table_get_immutable_target(t); } - __bind_mempools(md, t); + ret = __bind_mempools(md, t); + if (ret) { + old_map = ERR_PTR(ret); + goto out; + } old_map = rcu_dereference_protected(md->map, lockdep_is_held(&md->suspend_lock)); rcu_assign_pointer(md->map, (void *)t); @@ -2078,6 +2087,7 @@ static struct dm_table *__bind(struct mapped_device *md, struct dm_table *t, if (old_map) dm_sync_table(md); +out: return old_map; } diff --git a/include/linux/bio.h b/include/linux/bio.h index 810a8bee8f85..307682ac2f31 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -417,6 +417,7 @@ enum { extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags); extern void bioset_exit(struct bio_set *); extern int biovec_init_pool(mempool_t *pool, int pool_entries); +extern int bioset_init_from_src(struct bio_set *new, struct bio_set *src); extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set *); extern void bio_put(struct bio *); -- Jens Axboe ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-07 15:41 ` Jens Axboe @ 2018-06-08 6:06 ` Li Wang 2018-06-08 6:12 ` Chunyu Hu 1 sibling, 0 replies; 9+ messages in thread From: Li Wang @ 2018-06-08 6:06 UTC (permalink / raw) To: Jens Axboe Cc: Chunyu Hu, Kent Overstreet, Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel [-- Attachment #1: Type: text/plain, Size: 4545 bytes --] Hi Jens, On Thu, Jun 7, 2018 at 11:41 PM, Jens Axboe <axboe@kernel.dk> wrote: > On 6/7/18 8:46 AM, Jens Axboe wrote: > > On 6/7/18 12:33 AM, Chunyu Hu wrote: > >> kasan reported a user-after-free. I'm using a kvm machine, it panic > >> during boot. I'm using the latest linux tree. which contains below. > >> > >> commit d377535405686f735b90a8ad4ba269484cd7c96e > >> Author: Kent Overstreet <kent.overstreet@gmail.com> > >> Date: Tue Jun 5 05:26:33 2018 -0400 > >> > >> dm: Use kzalloc for all structs with embedded biosets/mempools > > > > Can you try with the below? Li Wang, would be great if you could too. > > Please try this one instead. > The problem was gone after applying this patch. Thanks! > > > diff --git a/block/bio.c b/block/bio.c > index 595663e0281a..0616d86b15c6 100644 > --- a/block/bio.c > +++ b/block/bio.c > @@ -1967,6 +1967,21 @@ int bioset_init(struct bio_set *bs, > } > EXPORT_SYMBOL(bioset_init); > > +int bioset_init_from_src(struct bio_set *new, struct bio_set *src) > +{ > + unsigned int pool_size = src->bio_pool.min_nr; > + int flags; > + > + flags = 0; > + if (src->bvec_pool.min_nr) > + flags |= BIOSET_NEED_BVECS; > + if (src->rescue_workqueue) > + flags |= BIOSET_NEED_RESCUER; > + > + return bioset_init(new, pool_size, src->front_pad, flags); > +} > +EXPORT_SYMBOL(bioset_init_from_src); > + > #ifdef CONFIG_BLK_CGROUP > > /** > diff --git a/drivers/md/dm.c b/drivers/md/dm.c > index 98dff36b89a3..20a8d63754bf 100644 > --- a/drivers/md/dm.c > +++ b/drivers/md/dm.c > @@ -1953,9 +1953,10 @@ static void free_dev(struct mapped_device *md) > kvfree(md); > } > > -static void __bind_mempools(struct mapped_device *md, struct dm_table *t) > +static int __bind_mempools(struct mapped_device *md, struct dm_table *t) > { > struct dm_md_mempools *p = dm_table_get_md_mempools(t); > + int ret = 0; > > if (dm_table_bio_based(t)) { > /* > @@ -1982,13 +1983,16 @@ static void __bind_mempools(struct mapped_device > *md, struct dm_table *t) > bioset_initialized(&md->bs) || > bioset_initialized(&md->io_bs)); > > - md->bs = p->bs; > - memset(&p->bs, 0, sizeof(p->bs)); > - md->io_bs = p->io_bs; > - memset(&p->io_bs, 0, sizeof(p->io_bs)); > + ret = bioset_init_from_src(&md->bs, &p->bs); > + if (ret) > + goto out; > + ret = bioset_init_from_src(&md->io_bs, &p->io_bs); > + if (ret) > + bioset_exit(&md->bs); > out: > /* mempool bind completed, no longer need any mempools in the > table */ > dm_table_free_md_mempools(t); > + return ret; > } > > /* > @@ -2033,6 +2037,7 @@ static struct dm_table *__bind(struct mapped_device > *md, struct dm_table *t, > struct request_queue *q = md->queue; > bool request_based = dm_table_request_based(t); > sector_t size; > + int ret; > > lockdep_assert_held(&md->suspend_lock); > > @@ -2068,7 +2073,11 @@ static struct dm_table *__bind(struct mapped_device > *md, struct dm_table *t, > md->immutable_target = dm_table_get_immutable_target(t); > } > > - __bind_mempools(md, t); > + ret = __bind_mempools(md, t); > + if (ret) { > + old_map = ERR_PTR(ret); > + goto out; > + } > > old_map = rcu_dereference_protected(md->map, > lockdep_is_held(&md->suspend_lock)); > rcu_assign_pointer(md->map, (void *)t); > @@ -2078,6 +2087,7 @@ static struct dm_table *__bind(struct mapped_device > *md, struct dm_table *t, > if (old_map) > dm_sync_table(md); > > +out: > return old_map; > } > > diff --git a/include/linux/bio.h b/include/linux/bio.h > index 810a8bee8f85..307682ac2f31 100644 > --- a/include/linux/bio.h > +++ b/include/linux/bio.h > @@ -417,6 +417,7 @@ enum { > extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int > flags); > extern void bioset_exit(struct bio_set *); > extern int biovec_init_pool(mempool_t *pool, int pool_entries); > +extern int bioset_init_from_src(struct bio_set *new, struct bio_set *src); > > extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set > *); > extern void bio_put(struct bio *); > > -- > Jens Axboe > > -- Regards, Li Wang [-- Attachment #2: Type: text/html, Size: 6247 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Block IO issue in kernel-v4.17 2018-06-07 15:41 ` Jens Axboe 2018-06-08 6:06 ` Li Wang @ 2018-06-08 6:12 ` Chunyu Hu 1 sibling, 0 replies; 9+ messages in thread From: Chunyu Hu @ 2018-06-08 6:12 UTC (permalink / raw) To: Jens Axboe Cc: Kent Overstreet, Li Wang, Coly Li, hch, darrick.wong, snitzer, linux-block, linux-kernel On 7 June 2018 at 23:41, Jens Axboe <axboe@kernel.dk> wrote: > On 6/7/18 8:46 AM, Jens Axboe wrote: >> On 6/7/18 12:33 AM, Chunyu Hu wrote: >>> kasan reported a user-after-free. I'm using a kvm machine, it panic >>> during boot. I'm using the latest linux tree. which contains below. >>> >>> commit d377535405686f735b90a8ad4ba269484cd7c96e >>> Author: Kent Overstreet <kent.overstreet@gmail.com> >>> Date: Tue Jun 5 05:26:33 2018 -0400 >>> >>> dm: Use kzalloc for all structs with embedded biosets/mempools >> >> Can you try with the below? Li Wang, would be great if you could too. > > Please try this one instead. My kvm machine boot successfully with this and works fine under pressure. Thanks for the fix. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-06-08 6:12 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAEemH2fbPv2wphVBHxSSS+KEK5S-rercXhvgLJ7YMOZnrTpexg@mail.gmail.com>
2018-06-06 8:41 ` Block IO issue in kernel-v4.17 Kent Overstreet
2018-06-06 14:18 ` Jens Axboe
2018-06-07 3:22 ` Li Wang
2018-06-07 6:26 ` Li Wang
2018-06-07 6:33 ` Chunyu Hu
2018-06-07 14:46 ` Jens Axboe
2018-06-07 15:41 ` Jens Axboe
2018-06-08 6:06 ` Li Wang
2018-06-08 6:12 ` Chunyu Hu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox