* mdadm hang when one subdev error(raid1)
From: 席智勇 @ 2015-04-17 7:23 UTC (permalink / raw)
To: linux-raid
hi all:
I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
Can anyone give me a help？
thanks.
uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
mdadm --version:mdadm - v3.2.5 - 18th May 2012
kern.log:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
-------------
Zhiyong Xi
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Mike Snitzer @ 2015-04-17 7:52 UTC (permalink / raw)
To: Alex Elsayed; +Cc: dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <mgnv2g$if5$2@ger.gmane.org>
On Thu, Apr 16 2015 at 5:23am -0400,
Alex Elsayed <eternaleye@gmail.com> wrote:
> Mike Snitzer wrote:
>
> > On Thu, Apr 09 2015 at 9:28am -0400,
> > Pali Rohár <pali.rohar@gmail.com> wrote:
> >
> >> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
> >> > On Mon, Apr 06 2015 at 9:29am -0400,
> >> > Pali Rohár <pali.rohar@gmail.com> wrote:
> >> >
> >> > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> >> > > > On Sun, Apr 05 2015 at 1:20pm -0400,
> >> > > >
> >> > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> >> > > > > This patch series increase security of suspend and hibernate
> >> > > > > actions. It allows user to safely wipe crypto keys before
> >> > > > > suspend and hibernate actions starts without race
> >> > > > > conditions on userspace process with heavy I/O.
> >> > > > >
> >> > > > > To automatically wipe cryto key for <device> before
> >> > > > > hibernate action call: $ dmsetup message <device> 0 key
> >> > > > > wipe_on_hibernation 1
> >> > > > >
> >> > > > > To automatically wipe cryto key for <device> before suspend
> >> > > > > action call: $ dmsetup message <device> 0 key
> >> > > > > wipe_on_suspend 1
> >> > > > >
> >> > > > > (Value 0 after wipe_* string reverts original behaviour - to
> >> > > > > not wipe key)
> >> > > >
> >> > > > Can you elaborate on the attack vector your changes are meant
> >> > > > to protect against? The user already authorized access, why
> >> > > > is it inherently dangerous to _not_ wipe the associated key
> >> > > > across these events?
> >> > >
> >> > > Hi,
> >> > >
> >> > > yes, I will try to explain current problems with cryptsetup
> >> > > luksSuspend command and hibernation.
> >> > >
> >> > > First, sometimes it is needed to put machine into other hands.
> >> > > You can still watch other person what is doing with machine, but
> >> > > once if you let machine unlocked (e.g opened luks disk), she/he
> >> > > can access encrypted data.
> >> > >
> >> > > If you turn off machine, it could be safe, because luks disk
> >> > > devices are locked. But if you enter machine into suspend or
> >> > > hibernate state luks devices are still open. And my patches try
> >> > > to achieve similar security as when machine is off (= no crypto
> >> > > keys in RAM or on swap).
> >> > >
> >> > > When doing hibernate on unencrypted swap it is to prevent leaking
> >> > > crypto keys to hibernate image (which is stored in swap).
> >> > >
> >> > > When doing suspend action it is again to prevent leaking crypto
> >> > > keys. E.g when you suspend laptop and put it off (somebody can
> >> > > remove RAMs and do some cold boot attack).
> >> > >
> >> > > The most common situation is:
> >> > > You have mounted partition from dm-crypt device (e.g. /home/),
> >> > > some userspace processes access it (e.g opened firefox which
> >> > > still reads/writes to cache ~/.firefox/) and you want to drop
> >> > > crypto keys from kernel for some time.
> >> > >
> >> > > For that operation there is command cryptsetup luksSuspend, which
> >> > > suspend dm device and then tell kernel to wipe crypto keys. All
> >> > > I/O operations are then stopped and userspace processes which
> >> > > want to do some those I/O operations are stopped too (until you
> >> > > call cryptsetup luksResume and enter correct key).
> >> > >
> >> > > Now if you want to suspend/hiberate your machine (when some of dm
> >> > > devices are suspeneded and some processes are stopped due to
> >> > > pending I/O) it is not possible. Kernel freeze_processes function
> >> > > will fail because userspace processes are still stopped inside
> >> > > some I/O syscall (read/write, etc,...).
> >> > >
> >> > > My patches fixes this problem and do those operations (suspend dm
> >> > > device, wipe crypto keys, enter suspend/hiberate) in correct
> >> > > order and without race condition.
> >> > >
> >> > > dm device is suspended *after* userspace processes are freezed
> >> > > and after that are crypto keys wiped. And then computer/laptop
> >> > > enters into suspend/hibernate state.
> >> >
> >> > Wouldn't it be better to fix freeze_processes() to be tolerant of
> >> > processes that are hung as a side-effect of their backing storage being
> >> > suspended? A hibernate shouldn't fail simply because a user chose to
> >> > suspend a DM device.
> >> >
> >> > Then this entire problem goes away and the key can be wiped from
> >> > userspace (like you said above).
> >>
> >> Still there will be race condition. Before hibernation (and device
> >> poweroff) we should have synced disks and filesystems to prevent data
> >> lose (or other damage) as more as we can. And if there will be some
> >> application which using lot of I/O (e.g normal firefox) then there
> >> always will be race condtion.
> >
> > The DM suspend will take care of flushing any pending I/O. So I don't
> > see where the supposed race is...
> >
> > Anything else that is trapped in userspace memory will be there when the
> > machine resumes.
> >
> >> So proper way is to wipe luks crypto keys *after* userspace processes
> >> are freezed.
> >
> > I know you believe that I'm just not accepting that at face value.
>
> Um, pardon me if I'm being naive, but what about the case of hibernation
> where the swapdev and the root device are both LVs on the same dm_crypt
> device?
>
> The kernel is writing to swap _after_ userspace processes are all frozen;
> that seems to me like an ordering dependency entirely incompatible with
> userspace dropping the key...
Good point, definitely not compatible with the Pali's approach.
(but is swap really configured ontop of the same dm-crypt device like
this in practice? I've not heard of that being a common pattern but I
could just be sheltered)
^ permalink raw reply
* WARNING: at kernel/sched/core.c:7331 __might_sleep+0xa6/0xb0() during boot
From: Petr Janecek @ 2015-04-17 7:56 UTC (permalink / raw)
To: linux-raid; +Cc: linux-kernel
Hello,
I'm getting these warnings once during boot since 3.19.0. The fs
is on lvm on md raid1.
[ 57.217167] ------------[ cut here ]------------
[ 57.221800] WARNING: CPU: 4 PID: 1071 at kernel/sched/core.c:7331 __might_sleep+0xa6/0xb0()
[ 57.230184] do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff81085d1b>] prepare_to_wait+0x2b/0xa0
[ 57.240886] Modules linked in: xt_multiport iptable_filter ip_tables x_tables cn dummy bridge stp llc xfs libcrc32c coretemp kvm_intel kvm iTCO_wdt microcode evdev iTCO_vendor_support psmouse serio_raw tpm_infineon tpm_tis tpm i7core_edac edac_core i2c_i801 button lpc_ich mfd_core i2c_core acpi_cpufreq processor thermal_sys hwmon_vid loop ipmi_watchdog ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler fuse autofs4 btrfs xor raid6_pq dm_mod raid1 md_mod sg sd_mod hid_generic usbhid hid e1000e ptp crc32c_intel ahci libahci libata ehci_pci ehci_hcd scsi_mod pps_core usbcore usb_common
[ 57.274503] CPU: 4 PID: 1071 Comm: freshclam Not tainted 3.19.4 #72
[ 57.274503] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.1 05/27/2010
[ 57.274504] ffffffff8179723f ffff88022fedb2a8 ffffffff814f70f3 ffffffff81089bf9
[ 57.274507] ffff88022fedb2f8 ffff88022fedb2e8 ffffffff81049bc5 ffff88023466e540
[ 57.274509] ffffffff817af863 00000000000006f6 0000000000000000 ffff88023625d800
[ 57.274511] Call Trace:
[ 57.274512] [<ffffffff814f70f3>] dump_stack+0x4c/0x65
[ 57.274518] [<ffffffff81089bf9>] ? down_trylock+0x29/0x40
[ 57.274521] [<ffffffff81049bc5>] warn_slowpath_common+0x85/0xc0
[ 57.274524] [<ffffffff81049c41>] warn_slowpath_fmt+0x41/0x50
[ 57.274526] [<ffffffff81085d1b>] ? prepare_to_wait+0x2b/0xa0
[ 57.274527] [<ffffffff81085d1b>] ? prepare_to_wait+0x2b/0xa0
[ 57.274529] [<ffffffff8106df86>] __might_sleep+0xa6/0xb0
[ 57.274532] [<ffffffff81282beb>] generic_make_request_checks+0x2b/0x400
[ 57.274534] [<ffffffff8108eb6d>] ? trace_hardirqs_on_caller+0x15d/0x200
[ 57.274537] [<ffffffff8108ec1d>] ? trace_hardirqs_on+0xd/0x10
[ 57.274539] [<ffffffff81282fd3>] generic_make_request+0x13/0x120
[ 57.274541] [<ffffffffa02f95e8>] raid1_unplug+0x108/0x160 [raid1]
[ 57.274544] [<ffffffff812878d2>] blk_flush_plug_list+0xa2/0x240
[ 57.274546] [<ffffffff8108ec1d>] ? trace_hardirqs_on+0xd/0x10
[ 57.274548] [<ffffffff814f9510>] ? bit_wait_timeout+0x80/0x80
[ 57.274550] [<ffffffff814f8cad>] io_schedule+0x7d/0x150
[ 57.274552] [<ffffffff814f953f>] bit_wait_io+0x2f/0x50
[ 57.274553] [<ffffffff814f90cf>] __wait_on_bit+0x5f/0x90
[ 57.274555] [<ffffffff814f9510>] ? bit_wait_timeout+0x80/0x80
[ 57.274556] [<ffffffff814f916d>] out_of_line_wait_on_bit+0x6d/0x80
[ 57.274558] [<ffffffff810860e0>] ? autoremove_wake_function+0x40/0x40
[ 57.274559] [<ffffffff811b0b2f>] __wait_on_buffer+0x3f/0x50
[ 57.274562] [<ffffffff811f64e0>] ext4_wait_block_bitmap.part.10+0xc0/0xd0
[ 57.274566] [<ffffffff811f6509>] ext4_wait_block_bitmap+0x19/0x20
[ 57.274567] [<ffffffff81230ef3>] ext4_mb_init_cache+0x203/0x780
[ 57.274570] [<ffffffff811207e9>] ? lru_cache_add+0x9/0x10
[ 57.274574] [<ffffffff8123158e>] ext4_mb_init_group+0x11e/0x210
[ 57.274575] [<ffffffff812317ab>] ext4_mb_good_group+0x12b/0x1a0
[ 57.274577] [<ffffffff81234b38>] ext4_mb_regular_allocator+0x1b8/0x460
[ 57.274579] [<ffffffff8122ffe0>] ? ext4_mb_use_preallocated.constprop.27+0x40/0x350
[ 57.274580] [<ffffffff8122f6fd>] ? ext4_mb_initialize_context+0x6d/0x1b0
[ 57.274582] [<ffffffff81236830>] ext4_mb_new_blocks+0x480/0x600
[ 57.274584] [<ffffffff8122b27d>] ext4_ext_map_blocks+0x62d/0xeb0
[ 57.274586] [<ffffffff811feb53>] ext4_map_blocks+0x133/0x560
[ 57.274588] [<ffffffff81201ed0>] ext4_writepages+0x560/0xdb0
[ 57.274590] [<ffffffff8111e3ac>] do_writepages+0x1c/0x30
[ 57.274592] [<ffffffff811128d1>] __filemap_fdatawrite_range+0x51/0x60
[ 57.274595] [<ffffffff81112977>] filemap_flush+0x17/0x20
[ 57.274597] [<ffffffff811ff851>] ext4_alloc_da_blocks+0x41/0x90
[ 57.274598] [<ffffffff8120ce0c>] ext4_rename+0x62c/0x880
[ 57.274601] [<ffffffff8108e9e1>] ? mark_held_locks+0x71/0xa0
[ 57.274603] [<ffffffff814faa3e>] ? mutex_lock_nested+0x22e/0x3b0
[ 57.274605] [<ffffffff8120d078>] ext4_rename2+0x18/0x40
[ 57.274606] [<ffffffff811897c2>] vfs_rename+0x3a2/0x7e0
[ 57.274609] [<ffffffff8118c8b0>] SYSC_renameat2+0x5b0/0x620
[ 57.274610] [<ffffffff814fe3e1>] ? _raw_spin_unlock_irqrestore+0x31/0x50
[ 57.274613] [<ffffffff81085bd8>] ? remove_wait_queue+0x48/0x50
[ 57.274615] [<ffffffff8104cb98>] ? do_wait+0x198/0x2c0
[ 57.274617] [<ffffffff8108eb6d>] ? trace_hardirqs_on_caller+0x15d/0x200
[ 57.274618] [<ffffffff812b7b1b>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 57.274621] [<ffffffff8118dba9>] SyS_rename+0x19/0x20
[ 57.274623] [<ffffffff814fecb2>] system_call_fastpath+0x12/0x17
[ 57.274626] ---[ end trace b4ab91b16fa597a6 ]---
Thanks,
Petr Janecek
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Ondrej Kozina @ 2015-04-17 8:52 UTC (permalink / raw)
To: device-mapper development, Alex Elsayed, Mike Snitzer
Cc: linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150417075211.GC403@redhat.com>
On 04/17/2015 09:52 AM, Mike Snitzer wrote:
> On Thu, Apr 16 2015 at 5:23am -0400,
> Alex Elsayed <eternaleye@gmail.com> wrote:
>
>> Mike Snitzer wrote:
>>
>>> On Thu, Apr 09 2015 at 9:28am -0400,
>>> Pali Rohár <pali.rohar@gmail.com> wrote:
>>>
>>>> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
>>>>> On Mon, Apr 06 2015 at 9:29am -0400,
>>>>> Pali Rohár <pali.rohar@gmail.com> wrote:
>>>>>
>>>>>> On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
>>>>>>> On Sun, Apr 05 2015 at 1:20pm -0400,
>>>>>>>
>>>>>>> Pali Rohár <pali.rohar@gmail.com> wrote:
>>>>>>>> This patch series increase security of suspend and hibernate
>>>>>>>> actions. It allows user to safely wipe crypto keys before
>>>>>>>> suspend and hibernate actions starts without race
>>>>>>>> conditions on userspace process with heavy I/O.
>>>>>>>>
>>>>>>>> To automatically wipe cryto key for <device> before
>>>>>>>> hibernate action call: $ dmsetup message <device> 0 key
>>>>>>>> wipe_on_hibernation 1
>>>>>>>>
>>>>>>>> To automatically wipe cryto key for <device> before suspend
>>>>>>>> action call: $ dmsetup message <device> 0 key
>>>>>>>> wipe_on_suspend 1
>>>>>>>>
>>>>>>>> (Value 0 after wipe_* string reverts original behaviour - to
>>>>>>>> not wipe key)
>>>>>>>
>>>>>>> Can you elaborate on the attack vector your changes are meant
>>>>>>> to protect against? The user already authorized access, why
>>>>>>> is it inherently dangerous to _not_ wipe the associated key
>>>>>>> across these events?
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> yes, I will try to explain current problems with cryptsetup
>>>>>> luksSuspend command and hibernation.
>>>>>>
>>>>>> First, sometimes it is needed to put machine into other hands.
>>>>>> You can still watch other person what is doing with machine, but
>>>>>> once if you let machine unlocked (e.g opened luks disk), she/he
>>>>>> can access encrypted data.
>>>>>>
>>>>>> If you turn off machine, it could be safe, because luks disk
>>>>>> devices are locked. But if you enter machine into suspend or
>>>>>> hibernate state luks devices are still open. And my patches try
>>>>>> to achieve similar security as when machine is off (= no crypto
>>>>>> keys in RAM or on swap).
>>>>>>
>>>>>> When doing hibernate on unencrypted swap it is to prevent leaking
>>>>>> crypto keys to hibernate image (which is stored in swap).
>>>>>>
>>>>>> When doing suspend action it is again to prevent leaking crypto
>>>>>> keys. E.g when you suspend laptop and put it off (somebody can
>>>>>> remove RAMs and do some cold boot attack).
>>>>>>
>>>>>> The most common situation is:
>>>>>> You have mounted partition from dm-crypt device (e.g. /home/),
>>>>>> some userspace processes access it (e.g opened firefox which
>>>>>> still reads/writes to cache ~/.firefox/) and you want to drop
>>>>>> crypto keys from kernel for some time.
>>>>>>
>>>>>> For that operation there is command cryptsetup luksSuspend, which
>>>>>> suspend dm device and then tell kernel to wipe crypto keys. All
>>>>>> I/O operations are then stopped and userspace processes which
>>>>>> want to do some those I/O operations are stopped too (until you
>>>>>> call cryptsetup luksResume and enter correct key).
>>>>>>
>>>>>> Now if you want to suspend/hiberate your machine (when some of dm
>>>>>> devices are suspeneded and some processes are stopped due to
>>>>>> pending I/O) it is not possible. Kernel freeze_processes function
>>>>>> will fail because userspace processes are still stopped inside
>>>>>> some I/O syscall (read/write, etc,...).
>>>>>>
>>>>>> My patches fixes this problem and do those operations (suspend dm
>>>>>> device, wipe crypto keys, enter suspend/hiberate) in correct
>>>>>> order and without race condition.
>>>>>>
>>>>>> dm device is suspended *after* userspace processes are freezed
>>>>>> and after that are crypto keys wiped. And then computer/laptop
>>>>>> enters into suspend/hibernate state.
>>>>>
>>>>> Wouldn't it be better to fix freeze_processes() to be tolerant of
>>>>> processes that are hung as a side-effect of their backing storage being
>>>>> suspended? A hibernate shouldn't fail simply because a user chose to
>>>>> suspend a DM device.
>>>>>
>>>>> Then this entire problem goes away and the key can be wiped from
>>>>> userspace (like you said above).
>>>>
>>>> Still there will be race condition. Before hibernation (and device
>>>> poweroff) we should have synced disks and filesystems to prevent data
>>>> lose (or other damage) as more as we can. And if there will be some
>>>> application which using lot of I/O (e.g normal firefox) then there
>>>> always will be race condtion.
>>>
>>> The DM suspend will take care of flushing any pending I/O. So I don't
>>> see where the supposed race is...
>>>
>>> Anything else that is trapped in userspace memory will be there when the
>>> machine resumes.
>>>
>>>> So proper way is to wipe luks crypto keys *after* userspace processes
>>>> are freezed.
>>>
>>> I know you believe that I'm just not accepting that at face value.
>>
>> Um, pardon me if I'm being naive, but what about the case of hibernation
>> where the swapdev and the root device are both LVs on the same dm_crypt
>> device?
>>
>> The kernel is writing to swap _after_ userspace processes are all frozen;
>> that seems to me like an ordering dependency entirely incompatible with
>> userspace dropping the key...
>
> Good point, definitely not compatible with the Pali's approach.
Ouch! I'm afraid this effectively killed one of my experiments with
dm-crypt suspend. Good to get reminded sooner than later!
> (but is swap really configured ontop of the same dm-crypt device like
> this in practice? I've not heard of that being a common pattern but I
> could just be sheltered)
yes. It's one among many perfectly valid setups.
(For some the goal here would be to have whole disk encrypted including
boot partition unlocked during boot)
^ permalink raw reply
* Re: mdadm hang when one subdev error(raid1)
From: hui jiao @ 2015-04-17 9:56 UTC (permalink / raw)
To: 席智勇; +Cc: linux-raid
In-Reply-To: <55fb513d.5f1b.14cc6426c67.Coremail.xizhiyong18@163.com>
The md42 is trying to update on-disk superblock, but the operation
doesn't finish. During this peroid, the configuration lock is holded
by md42_raid1 thread, all other operations which need the lock will
wait, such as -D --fail --remove.
what's the status of the subdev now? is it suspended?
check and resume it:
[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
1
[root@node0 ~]# dmsetup resume /dev/dm-0
[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
0
On Fri, Apr 17, 2015 at 3:23 PM, 席智勇 <xizhiyong18@163.com> wrote:
> hi all:
>
> I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
> The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
> I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
> Can anyone give me a help？
> thanks.
>
> uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
> mdadm --version:mdadm - v3.2.5 - 18th May 2012
> kern.log:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>
>
> -------------
>
> Zhiyong Xi
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 0/3] dm-raid: Add support for the MD RAID0 personality
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patchset of 3 adds access to the MD RAID0 personality to dm-raid
(in addition to the already supported raid1/10/4/5/6 personalities)
to enable single zone striping.
The added functionality is the first step to allow full support
of MD kernel functionality via dm-raid and to eventually replace
the dm "striped" target altogether. Adding shrink/takeover and
reshape functionality to dm-raid is subject to future patches.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
Heinz Mauelshagen (3):
dm-raid: Add support for the MD RAID0 personality
dm core: export dm_disk to allow for dm-raid to pass it on to the MD
raid0 personality
raid0: conditional mddev->queue access to suit dm-raid
drivers/md/dm-raid.c | 115 ++++++++++++++++++++++++++++++---------------------
drivers/md/dm.c | 1 +
drivers/md/raid0.c | 48 +++++++++++----------
3 files changed, 96 insertions(+), 68 deletions(-)
--
2.1.0
^ permalink raw reply
* [PATCH 1/3] dm-raid: Add support for the MD RAID0 personality
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch adds access to the MD RAID0 personality to dm-raid
(in addition to the already supported raid1/10/4/5/6 personalities)
to enable single zone striping.
The following changes enable that access:
- add type definition to raid_types array
- make bitmap creation conditonal in super_validat(), because
it is not allowed in RAID0
- set rdev->sectors to the data image size in super_validate()
to allow the RAID0 personality to calculate the MD array
size properly
- set rs->md.gendisk, because it is required by RAID0
to set disk stack limits
- enhance raid_resume() to not load bitmap in case of "raid0"
- enhance raid_status() to always report full sync for "raid0"
so that userspace checks for 100% sync will succeed and allow
for resize (and takeover/reshape once added in future paches)
- use mdddev(un)lock() functions instead of direct mutex_(un)lock()
(wrapped in here because it's a trivial change)
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/dm-raid.c | 115 ++++++++++++++++++++++++++++++---------------------
1 file changed, 68 insertions(+), 47 deletions(-)
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index 88e4c7f..14890dd 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -18,6 +18,7 @@
#define DM_MSG_PREFIX "raid"
+
static bool devices_handle_discard_safely = false;
/*
@@ -81,6 +82,7 @@ static struct raid_type {
const unsigned level; /* RAID level. */
const unsigned algorithm; /* RAID algorithm. */
} raid_types[] = {
+ {"raid0", "RAID0 (striping)", 0, 2, 0, 0 /* NONE */},
{"raid1", "RAID1 (mirroring)", 0, 2, 1, 0 /* NONE */},
{"raid10", "RAID10 (striped mirrors)", 0, 2, 10, UINT_MAX /* Varies */},
{"raid4", "RAID4 (dedicated parity disk)", 1, 2, 5, ALGORITHM_PARITY_0},
@@ -720,7 +722,7 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
rs->md.layout = raid10_format_to_md_layout(raid10_format,
raid10_copies);
rs->md.new_layout = rs->md.layout;
- } else if ((rs->raid_type->level > 1) &&
+ } else if ((!rs->raid_type->level || rs->raid_type->level > 1) &&
sector_div(sectors_per_dev,
(rs->md.raid_disks - rs->raid_type->parity_devs))) {
rs->ti->error = "Target length not divisible by number of data devices";
@@ -1026,8 +1028,9 @@ static int super_init_validation(struct mddev *mddev, struct md_rdev *rdev)
return 0;
}
-static int super_validate(struct mddev *mddev, struct md_rdev *rdev)
+static int super_validate(struct raid_set *rs, struct md_rdev *rdev)
{
+ struct mddev *mddev = &rs->md;
struct dm_raid_superblock *sb = page_address(rdev->sb_page);
/*
@@ -1037,8 +1040,10 @@ static int super_validate(struct mddev *mddev, struct md_rdev *rdev)
if (!mddev->events && super_init_validation(mddev, rdev))
return -EINVAL;
- mddev->bitmap_info.offset = 4096 >> 9; /* Enable bitmap creation */
- rdev->mddev->bitmap_info.default_offset = 4096 >> 9;
+ /* Enable bitmap creation for RAID levels != 0 */
+ mddev->bitmap_info.offset = (rs->raid_type->level) ? 4096 >> 9 : 0;
+ rdev->mddev->bitmap_info.default_offset = mddev->bitmap_info.offset;
+
if (!test_bit(FirstUse, &rdev->flags)) {
rdev->recovery_offset = le64_to_cpu(sb->disk_recovery_offset);
if (rdev->recovery_offset != MaxSector)
@@ -1082,6 +1087,8 @@ static int analyse_superblocks(struct dm_target *ti, struct raid_set *rs)
* that the "sync" directive is disallowed during the
* reshape.
*/
+ rdev->sectors = to_sector(i_size_read(rdev->bdev->bd_inode));
+
if (rs->print_flags & DMPF_SYNC)
continue;
@@ -1140,11 +1147,11 @@ static int analyse_superblocks(struct dm_target *ti, struct raid_set *rs)
* validation for the remaining devices.
*/
ti->error = "Unable to assemble array: Invalid superblocks";
- if (super_validate(mddev, freshest))
+ if (super_validate(rs, freshest))
return -EINVAL;
rdev_for_each(rdev, mddev)
- if ((rdev != freshest) && super_validate(mddev, rdev))
+ if ((rdev != freshest) && super_validate(rs, rdev))
return -EINVAL;
return 0;
@@ -1282,10 +1289,14 @@ static int raid_ctr(struct dm_target *ti, unsigned argc, char **argv)
*/
configure_discard_support(ti, rs);
- mutex_lock(&rs->md.reconfig_mutex);
+ /* "raid0" calls disk_stack_limits() requiring gendisk access */
+ rs->md.gendisk = dm_disk(dm_table_get_md(ti->table));
+
+ /* Has to be held on running the array */
+ mddev_lock_nointr(&rs->md);
ret = md_run(&rs->md);
rs->md.in_sync = 0; /* Assume already marked dirty */
- mutex_unlock(&rs->md.reconfig_mutex);
+ mddev_unlock(&rs->md);
if (ret) {
ti->error = "Fail to run raid array";
@@ -1368,34 +1379,40 @@ static void raid_status(struct dm_target *ti, status_type_t type,
case STATUSTYPE_INFO:
DMEMIT("%s %d ", rs->raid_type->name, rs->md.raid_disks);
- if (test_bit(MD_RECOVERY_RUNNING, &rs->md.recovery))
- sync = rs->md.curr_resync_completed;
- else
- sync = rs->md.recovery_cp;
-
- if (sync >= rs->md.resync_max_sectors) {
- /*
- * Sync complete.
- */
+ if (rs->raid_type->level) {
+ if (test_bit(MD_RECOVERY_RUNNING, &rs->md.recovery))
+ sync = rs->md.curr_resync_completed;
+ else
+ sync = rs->md.recovery_cp;
+
+ if (sync >= rs->md.resync_max_sectors) {
+ /*
+ * Sync complete.
+ */
+ array_in_sync = 1;
+ sync = rs->md.resync_max_sectors;
+ } else if (test_bit(MD_RECOVERY_REQUESTED, &rs->md.recovery)) {
+ /*
+ * If "check" or "repair" is occurring, the array has
+ * undergone and initial sync and the health characters
+ * should not be 'a' anymore.
+ */
+ array_in_sync = 1;
+ } else {
+ /*
+ * The array may be doing an initial sync, or it may
+ * be rebuilding individual components. If all the
+ * devices are In_sync, then it is the array that is
+ * being initialized.
+ */
+ for (i = 0; i < rs->md.raid_disks; i++)
+ if (!test_bit(In_sync, &rs->dev[i].rdev.flags))
+ array_in_sync = 1;
+ }
+ } else {
+ /* RAID0 */
array_in_sync = 1;
sync = rs->md.resync_max_sectors;
- } else if (test_bit(MD_RECOVERY_REQUESTED, &rs->md.recovery)) {
- /*
- * If "check" or "repair" is occurring, the array has
- * undergone and initial sync and the health characters
- * should not be 'a' anymore.
- */
- array_in_sync = 1;
- } else {
- /*
- * The array may be doing an initial sync, or it may
- * be rebuilding individual components. If all the
- * devices are In_sync, then it is the array that is
- * being initialized.
- */
- for (i = 0; i < rs->md.raid_disks; i++)
- if (!test_bit(In_sync, &rs->dev[i].rdev.flags))
- array_in_sync = 1;
}
/*
@@ -1684,26 +1701,30 @@ static void raid_resume(struct dm_target *ti)
{
struct raid_set *rs = ti->private;
- set_bit(MD_CHANGE_DEVS, &rs->md.flags);
- if (!rs->bitmap_loaded) {
- bitmap_load(&rs->md);
- rs->bitmap_loaded = 1;
- } else {
- /*
- * A secondary resume while the device is active.
- * Take this opportunity to check whether any failed
- * devices are reachable again.
- */
- attempt_restore_of_faulty_devices(rs);
+ if (rs->raid_type->level) {
+ set_bit(MD_CHANGE_DEVS, &rs->md.flags);
+
+ if (!rs->bitmap_loaded) {
+ bitmap_load(&rs->md);
+ rs->bitmap_loaded = 1;
+ } else {
+ /*
+ * A secondary resume while the device is active.
+ * Take this opportunity to check whether any failed
+ * devices are reachable again.
+ */
+ attempt_restore_of_faulty_devices(rs);
+ }
+
+ clear_bit(MD_RECOVERY_FROZEN, &rs->md.recovery);
}
- clear_bit(MD_RECOVERY_FROZEN, &rs->md.recovery);
mddev_resume(&rs->md);
}
static struct target_type raid_target = {
.name = "raid",
- .version = {1, 6, 0},
+ .version = {1, 7, 0},
.module = THIS_MODULE,
.ctr = raid_ctr,
.dtr = raid_dtr,
--
2.1.0
^ permalink raw reply related
* [PATCH 2/3] dm core: export dm_disk to allow for dm-raid to pass it on to the MD raid0 personality
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch is a prerequisite for dm-raid "raid0" support to allow
dm-raid to set rs->md.gendisk, which is required by the MD RAID0
pesonality to set disk stacking limits.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/dm.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 8001fe9..f11e2f5 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -3206,6 +3206,7 @@ struct gendisk *dm_disk(struct mapped_device *md)
{
return md->disk;
}
+EXPORT_SYMBOL_GPL(dm_disk);
struct kobject *dm_kobject(struct mapped_device *md)
{
--
2.1.0
^ permalink raw reply related
* [PATCH 3/3] raid0: conditional mddev->queue access to suit dm-raid
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch is a prerequisite for dm-raid "raid0" support to allow
dm-raid to access the MD RAID0 personality doing unconditional
accesses to mddev->queue, which is NULL in case of dm-raid stacked
on top of MD.
This patch is here for completeness; it has been send to Neil Brown
and linux-raid, Neil said he applied it on Feb 23rd but it did not
show up in upstream as yet.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/raid0.c | 48 +++++++++++++++++++++++++++---------------------
1 file changed, 27 insertions(+), 21 deletions(-)
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 3ed9f42..e074813 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -271,14 +271,16 @@ static int create_strip_zones(struct mddev *mddev, struct r0conf **private_conf)
goto abort;
}
- blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
- blk_queue_io_opt(mddev->queue,
- (mddev->chunk_sectors << 9) * mddev->raid_disks);
-
- if (!discard_supported)
- queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
- else
- queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
+ if (mddev->queue) {
+ blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
+ blk_queue_io_opt(mddev->queue,
+ (mddev->chunk_sectors << 9) * mddev->raid_disks);
+
+ if (!discard_supported)
+ queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
+ else
+ queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
+ }
pr_debug("md/raid0:%s: done.\n", mdname(mddev));
*private_conf = conf;
@@ -429,9 +431,12 @@ static int raid0_run(struct mddev *mddev)
}
if (md_check_no_bitmap(mddev))
return -EINVAL;
- blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
- blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
- blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
+
+ if (mddev->queue) {
+ blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
+ blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
+ blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
+ }
/* if private is not null, we are here after takeover */
if (mddev->private == NULL) {
@@ -448,16 +453,17 @@ static int raid0_run(struct mddev *mddev)
printk(KERN_INFO "md/raid0:%s: md_size is %llu sectors.\n",
mdname(mddev),
(unsigned long long)mddev->array_sectors);
- /* calculate the max read-ahead size.
- * For read-ahead of large files to be effective, we need to
- * readahead at least twice a whole stripe. i.e. number of devices
- * multiplied by chunk size times 2.
- * If an individual device has an ra_pages greater than the
- * chunk size, then we will not drive that device as hard as it
- * wants. We consider this a configuration error: a larger
- * chunksize should be used in that case.
- */
- {
+
+ if (mddev->queue) {
+ /* calculate the max read-ahead size.
+ * For read-ahead of large files to be effective, we need to
+ * readahead at least twice a whole stripe. i.e. number of devices
+ * multiplied by chunk size times 2.
+ * If an individual device has an ra_pages greater than the
+ * chunk size, then we will not drive that device as hard as it
+ * wants. We consider this a configuration error: a larger
+ * chunksize should be used in that case.
+ */
int stripe = mddev->raid_disks *
(mddev->chunk_sectors << 9) / PAGE_SIZE;
if (mddev->queue->backing_dev_info.ra_pages < 2* stripe)
--
2.1.0
^ permalink raw reply related
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Alex Elsayed @ 2015-04-17 15:53 UTC (permalink / raw)
To: linux-kernel; +Cc: dm-devel, linux-raid, linux-pm
In-Reply-To: <20150417075211.GC403@redhat.com>
Mike Snitzer wrote:
> On Thu, Apr 16 2015 at 5:23am -0400,
> Alex Elsayed <eternaleye@gmail.com> wrote:
>
>> Mike Snitzer wrote:
>>
>> > On Thu, Apr 09 2015 at 9:28am -0400,
>> > Pali Rohár <pali.rohar@gmail.com> wrote:
>> >
>> >> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
>> >> > On Mon, Apr 06 2015 at 9:29am -0400,
>> >> > Pali Rohár <pali.rohar@gmail.com> wrote:
>> >> >
>> >> > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
>> >> > > > On Sun, Apr 05 2015 at 1:20pm -0400,
>> >> > > >
>> >> > > > Pali Rohár <pali.rohar@gmail.com> wrote:
>> >> > > > > This patch series increase security of suspend and hibernate
>> >> > > > > actions. It allows user to safely wipe crypto keys before
>> >> > > > > suspend and hibernate actions starts without race
>> >> > > > > conditions on userspace process with heavy I/O.
>> >> > > > >
>> >> > > > > To automatically wipe cryto key for <device> before
>> >> > > > > hibernate action call: $ dmsetup message <device> 0 key
>> >> > > > > wipe_on_hibernation 1
>> >> > > > >
>> >> > > > > To automatically wipe cryto key for <device> before suspend
>> >> > > > > action call: $ dmsetup message <device> 0 key
>> >> > > > > wipe_on_suspend 1
>> >> > > > >
>> >> > > > > (Value 0 after wipe_* string reverts original behaviour - to
>> >> > > > > not wipe key)
>> >> > > >
>> >> > > > Can you elaborate on the attack vector your changes are meant
>> >> > > > to protect against? The user already authorized access, why
>> >> > > > is it inherently dangerous to _not_ wipe the associated key
>> >> > > > across these events?
>> >> > >
>> >> > > Hi,
>> >> > >
>> >> > > yes, I will try to explain current problems with cryptsetup
>> >> > > luksSuspend command and hibernation.
>> >> > >
>> >> > > First, sometimes it is needed to put machine into other hands.
>> >> > > You can still watch other person what is doing with machine, but
>> >> > > once if you let machine unlocked (e.g opened luks disk), she/he
>> >> > > can access encrypted data.
>> >> > >
>> >> > > If you turn off machine, it could be safe, because luks disk
>> >> > > devices are locked. But if you enter machine into suspend or
>> >> > > hibernate state luks devices are still open. And my patches try
>> >> > > to achieve similar security as when machine is off (= no crypto
>> >> > > keys in RAM or on swap).
>> >> > >
>> >> > > When doing hibernate on unencrypted swap it is to prevent leaking
>> >> > > crypto keys to hibernate image (which is stored in swap).
>> >> > >
>> >> > > When doing suspend action it is again to prevent leaking crypto
>> >> > > keys. E.g when you suspend laptop and put it off (somebody can
>> >> > > remove RAMs and do some cold boot attack).
>> >> > >
>> >> > > The most common situation is:
>> >> > > You have mounted partition from dm-crypt device (e.g. /home/),
>> >> > > some userspace processes access it (e.g opened firefox which
>> >> > > still reads/writes to cache ~/.firefox/) and you want to drop
>> >> > > crypto keys from kernel for some time.
>> >> > >
>> >> > > For that operation there is command cryptsetup luksSuspend, which
>> >> > > suspend dm device and then tell kernel to wipe crypto keys. All
>> >> > > I/O operations are then stopped and userspace processes which
>> >> > > want to do some those I/O operations are stopped too (until you
>> >> > > call cryptsetup luksResume and enter correct key).
>> >> > >
>> >> > > Now if you want to suspend/hiberate your machine (when some of dm
>> >> > > devices are suspeneded and some processes are stopped due to
>> >> > > pending I/O) it is not possible. Kernel freeze_processes function
>> >> > > will fail because userspace processes are still stopped inside
>> >> > > some I/O syscall (read/write, etc,...).
>> >> > >
>> >> > > My patches fixes this problem and do those operations (suspend dm
>> >> > > device, wipe crypto keys, enter suspend/hiberate) in correct
>> >> > > order and without race condition.
>> >> > >
>> >> > > dm device is suspended *after* userspace processes are freezed
>> >> > > and after that are crypto keys wiped. And then computer/laptop
>> >> > > enters into suspend/hibernate state.
>> >> >
>> >> > Wouldn't it be better to fix freeze_processes() to be tolerant of
>> >> > processes that are hung as a side-effect of their backing storage
>> >> > being
>> >> > suspended? A hibernate shouldn't fail simply because a user chose
>> >> > to suspend a DM device.
>> >> >
>> >> > Then this entire problem goes away and the key can be wiped from
>> >> > userspace (like you said above).
>> >>
>> >> Still there will be race condition. Before hibernation (and device
>> >> poweroff) we should have synced disks and filesystems to prevent data
>> >> lose (or other damage) as more as we can. And if there will be some
>> >> application which using lot of I/O (e.g normal firefox) then there
>> >> always will be race condtion.
>> >
>> > The DM suspend will take care of flushing any pending I/O. So I don't
>> > see where the supposed race is...
>> >
>> > Anything else that is trapped in userspace memory will be there when
>> > the machine resumes.
>> >
>> >> So proper way is to wipe luks crypto keys *after* userspace processes
>> >> are freezed.
>> >
>> > I know you believe that I'm just not accepting that at face value.
>>
>> Um, pardon me if I'm being naive, but what about the case of hibernation
>> where the swapdev and the root device are both LVs on the same dm_crypt
>> device?
>>
>> The kernel is writing to swap _after_ userspace processes are all frozen;
>> that seems to me like an ordering dependency entirely incompatible with
>> userspace dropping the key...
>
> Good point, definitely not compatible with the Pali's approach.
>
> (but is swap really configured ontop of the same dm-crypt device like
> this in practice? I've not heard of that being a common pattern but I
> could just be sheltered)
Every laptop I've owned in the past five years has been set up as follows:
- GPT partition table (mostly for the redundant table at the end in case of
fuckups)
- 1GB ESP as /boot (first with grub2, then gummiboot) - It's there _anyway_
- 32MB BIOS Boot Partition (for a traditional BIOS bootloader, so I can pop
the drive in a non-efi machine if the laptop dies - this has happened)
- The rest of the drive is a single dm-crypt volume, with LVM on top. What
goes on top of LVM has varied, but these days it's just swap and btrfs.
The main reason for this is that I find dealing with crypttab / multiple
LUKS devices on boot (or resume from hibernate) to be an incredible hassle;
and it's vastly simpler to just have a single dm-crypt device and let
Dracut unlock it from a single boot prompt.
I haven't set up custom-key secure boot yet, so the evil maid attack is
still on the table, but I do this more out of "Eh, why not" (and originally,
"I should at least know _how_ to set it up") than actually having stuff I
need the security for anyway.
^ permalink raw reply
* Raid 5 rebuild and Raid 6 reshape question
From: Thomas Fjellstrom @ 2015-04-17 17:55 UTC (permalink / raw)
To: linux-raid
This is a potentially dumb question...
I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
seemed to work), and i decided to replace it. I actually bought two drives,
thinking I'd keep one as a cold spare, but when i got to replacing the drive
(after days of load testing to see if I'd gotten a dud), I figured maybe I'd
use the spare as a hot spare, or make a Raid6 array. In the end I decided on a
Raid6, as it is the recommended way to go these days for large disks.
When I did the add for the first disk, it started rebuilding the raid5, and
said it'd only take like 5 hours, then I added the second disk and it
skyrocketed to about 116 hours (7000s or so). I presume this is expected, due
to having to re-stripe all of the data, and re-calculate the parity at the
same time.
How much faster would this have finished if I had just waited for the first
drive to finish rebuilding and done the reshape as a separate step? I decided
to just let this reshape continue, its about 30% the way through, and this is
actually the backup array for my main NAS, so it isn't actually a problem if
it dies mid-reshape. I'm just curious about the alternatives to how I did it.
I know i could have just copied the old disk to one of the new ones, and
skipped the entire rebuild on one disk, but i didn't feel like it ;D
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: Raid 5 rebuild and Raid 6 reshape question
From: Mikael Abrahamsson @ 2015-04-18 13:13 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: linux-raid
In-Reply-To: <6744821.j7csV3RJVE@balsa>
On Fri, 17 Apr 2015, Thomas Fjellstrom wrote:
> I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
> seemed to work), and i decided to replace it. I actually bought two drives,
If you have a not-ancient kernel, you should have issued the replace
command. You should have added one new drive as spare, then (with a new
mdadm) issued the --replace command. This would have started copying the
drive you wanted replaced onto the spare. Then after all this was done,
you could have added another spare and told it to re-shape to raid6.
That's at least what I would have done.
http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid 5 rebuild and Raid 6 reshape question
From: Thomas Fjellstrom @ 2015-04-18 19:38 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1504181511310.16871@uplift.swm.pp.se>
On Sat 18 Apr 2015 03:13:52 PM Mikael Abrahamsson wrote:
> On Fri, 17 Apr 2015, Thomas Fjellstrom wrote:
> > I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
> > seemed to work), and i decided to replace it. I actually bought two
> > drives,
>
> If you have a not-ancient kernel, you should have issued the replace
> command. You should have added one new drive as spare, then (with a new
> mdadm) issued the --replace command. This would have started copying the
> drive you wanted replaced onto the spare. Then after all this was done,
> you could have added another spare and told it to re-shape to raid6.
> That's at least what I would have done.
>
> http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-ye
> t-failed-disk-in-a-linux-raid5-array
Yes, I could have. The machine's drive bays were full at the time, it wasnt
till I had inserted the new drives that I realized i could have left the old
one in and done a copy or replace. But then the machine was booting without it
and probably not a great time to put the old disk back in (though an assemble
force could have brought it back?). It's almost 60% done, so it's not a huge
problem. I was just curious as to how long an actual raid5->raid6 reshape
would normally be or this setup. I expect it normally can take a while as it
has to rewrite a lot of data (11TiB worth).
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* [PATCH] dm log: Use ULL suffix for 64-bit constants
From: Geert Uytterhoeven @ 2015-04-18 22:07 UTC (permalink / raw)
To: Josef Bacik, Alasdair Kergon, Mike Snitzer
Cc: dm-devel, Neil Brown, linux-raid, linux-kernel,
Geert Uytterhoeven
On 32-bit:
drivers/md/dm-log-writes.c: In function ‘log_super’:
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
Add a ULL suffix to WRITE_LOG_MAGIC to fix this.
Also add a ULL suffix to WRITE_LOG_VERSION as it's stored in a __le64
field.
Signed-off-by: Geert Uytterhoeven <geert@linux-m68k.org>
---
drivers/md/dm-log-writes.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
index 93e08446a87d06ff..ad1b049ae2ab80a9 100644
--- a/drivers/md/dm-log-writes.c
+++ b/drivers/md/dm-log-writes.c
@@ -55,8 +55,8 @@
#define LOG_DISCARD_FLAG (1 << 2)
#define LOG_MARK_FLAG (1 << 3)
-#define WRITE_LOG_VERSION 1
-#define WRITE_LOG_MAGIC 0x6a736677736872
+#define WRITE_LOG_VERSION 1ULL
+#define WRITE_LOG_MAGIC 0x6a736677736872ULL
/*
* The disk format for this is braindead simple.
--
1.9.1
^ permalink raw reply related
* dm log: struct log_write_super and padding (was: Re: dm: add log writes target)
From: Geert Uytterhoeven @ 2015-04-18 22:16 UTC (permalink / raw)
To: Josef Bacik, Alasdair Kergon, Mike Snitzer
Cc: dm-devel, Neil Brown, linux-raid, linux-kernel@vger.kernel.org
On Sat, Apr 18, 2015 at 2:34 PM, Linux Kernel Mailing List
<linux-kernel@vger.kernel.org> wrote:
> Gitweb: http://git.kernel.org/linus/;a=commit;h=0e9cebe724597a76ab1b0ebc0a21e16f7db11b47
> Commit: 0e9cebe724597a76ab1b0ebc0a21e16f7db11b47
> --- /dev/null
> +++ b/drivers/md/dm-log-writes.c
> @@ -0,0 +1,825 @@
> +/*
> + * Basic info about the log for userspace.
> + */
> +struct log_write_super {
> + __le64 magic;
> + __le64 version;
> + __le64 nr_entries;
> + __le32 sectorsize;
Should there be an explicit padding of 4 bytes?
Does anything assume that sizeof(struct log_write_super) is always 4 * 8?
On architectures where the natural alignment of 64-bit quantities is less
than 8 bytes, sizeof(struct log_write_super) is smaller than 32 bytes
(e.g. on m68k it's 28).
> +};
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Reply:Re: mdadm hang when one subdev error(raid1)
From: 席智勇 @ 2015-04-19 0:06 UTC (permalink / raw)
To: hui jiao; +Cc: linux-raid
In-Reply-To: <CAEuG+XJbMqyQ4_nS=ZKEoDW0shQgLgEqNirxx=xNBa6yH_Y=_w@mail.gmail.com>
thanks for your reply.
in my system, the subdev was a device exported by iscsi through net. as you said the device was suspended by unknown resion.(in fact, the exported disk was errorred.)
what i do not understand is why the io did not timeout, if it was writing info on superblock while the device cannot response it.
now i think the io should have timeout at iscsi level.but it did not, in this situation，mdadm can did nothing except waiting?
At 2015-04-17 17:56:55, "hui jiao" <simonjiaoh@gmail.com> wrote:
>The md42 is trying to update on-disk superblock, but the operation
>doesn't finish. During this peroid, the configuration lock is holded
>by md42_raid1 thread, all other operations which need the lock will
>wait, such as -D --fail --remove.
>
>what's the status of the subdev now? is it suspended?
>check and resume it:
>[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
>1
>[root@node0 ~]# dmsetup resume /dev/dm-0
>[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
>0
>
>On Fri, Apr 17, 2015 at 3:23 PM, 席智勇 <xizhiyong18@163.com> wrote:
>> hi all:
>>
>> I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
>> The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
>> I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
>> Can anyone give me a help？
>> thanks.
>>
>> uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
>> mdadm --version:mdadm - v3.2.5 - 18th May 2012
>> kern.log:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>>
>>
>> -------------
>>
>> Zhiyong Xi
^ permalink raw reply
* Re: Raid 5 rebuild and Raid 6 reshape question
From: Mikael Abrahamsson @ 2015-04-19 8:43 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: linux-raid
In-Reply-To: <1454025.2pEa89kuvv@balsa>
On Sat, 18 Apr 2015, Thomas Fjellstrom wrote:
> problem. I was just curious as to how long an actual raid5->raid6
> reshape would normally be or this setup. I expect it normally can take a
> while as it has to rewrite a lot of data (11TiB worth).
In my experience it takes 5-15 times longer than a normal resync takes, to
reshape from raid5->raid6. So for your 3TB disk size case, I'd say expect
4-8 days or so.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid 5 rebuild and Raid 6 reshape question
From: Thomas Fjellstrom @ 2015-04-19 19:42 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1504191041250.16871@uplift.swm.pp.se>
On Sun 19 Apr 2015 10:43:32 AM Mikael Abrahamsson wrote:
> On Sat, 18 Apr 2015, Thomas Fjellstrom wrote:
> > problem. I was just curious as to how long an actual raid5->raid6
> > reshape would normally be or this setup. I expect it normally can take a
> > while as it has to rewrite a lot of data (11TiB worth).
>
> In my experience it takes 5-15 times longer than a normal resync takes, to
> reshape from raid5->raid6. So for your 3TB disk size case, I'd say expect
> 4-8 days or so.
Ah. Good to know. mdstat was saying about 4-5 days. I kindof thought that was
about normal given the amount of work involved to restripe all drives and
calculate parity.
Thanks :)
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: [PATCH 1/6] md-cluster: correct the num for comparison
From: NeilBrown @ 2015-04-20 1:48 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: GQJiang, linux-raid
In-Reply-To: <20150414154232.GA4079@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]
On Tue, 14 Apr 2015 10:42:32 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> Author: Guoqing Jiang <gqjiang@suse.com>
>
> Since the node num of md-cluster is from zero, and
> cinfo->slot_number represents the slot num of dlm,
> no need to check for equality.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md-cluster.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index 96679b2..6bdc45e 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -612,7 +612,7 @@ static int join(struct mddev *mddev, int nodes)
> if (ret)
> goto err;
> wait_for_completion(&cinfo->completion);
> - if (nodes <= cinfo->slot_number) {
> + if (nodes < cinfo->slot_number) {
> pr_err("md-cluster: Slot allotted(%d) greater than available slots(%d)", cinfo->slot_number - 1,
> nodes);
> ret = -ERANGE;
I'll apply this, but it would be nice to fix the error message too.
If 'nodes' is 3 and 'slot_number' is 4, then the message would be
md-cluster: Slot allotted(3) greater than available slots(3)
which looks weird.
(it should be "is greater", and there should be a "." at the end of the
sentence).
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 5/6] md: re-add a failed disk
From: NeilBrown @ 2015-04-20 1:56 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: GQJiang, linux-raid
In-Reply-To: <20150414154522.GA4105@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 4570 bytes --]
On Tue, 14 Apr 2015 10:45:22 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> This adds the capability of re-adding a failed disk by
> writing "re-add" to /sys/block/mdXX/md/dev-YYY/state.
>
> This facilitates adding disks which have encountered a temporary
> error such as a network disconnection/hiccup in an iSCSI device,
> or a SAN cable disconnection which has been restored. In such
> a situation, you do not need to remove and re-add the device.
> Writing re-add to the failed device's state would add it again
> to the array and perform the recovery of only the blocks which
> were written after the device failed.
>
> This works for generic md, and is not related to clustering. However,
> this patch is to ease re-add operations listed above in clustering
> environments.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md.c | 56 +++++++++++++++++++++++++++++++++++---------------------
> 1 file changed, 35 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 9127d11..ba01605 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -2379,6 +2379,36 @@ repeat:
> }
> EXPORT_SYMBOL(md_update_sb);
>
> +static int add_bound_rdev(struct md_rdev *rdev)
> +{
> + struct mddev *mddev = rdev->mddev;
> + int err = 0;
> +
> + if (!mddev->pers->hot_remove_disk) {
> + /* If there is hot_add_disk but no hot_remove_disk
> + * then added disks for geometry changes,
> + * and should be added immediately.
> + */
> + super_types[mddev->major_version].
> + validate_super(mddev, rdev);
> + err = mddev->pers->hot_add_disk(mddev, rdev);
> + if (err) {
> + unbind_rdev_from_array(rdev);
> + export_rdev(rdev);
> + return err;
> + }
> + }
> + sysfs_notify_dirent_safe(rdev->sysfs_state);
> +
> + set_bit(MD_CHANGE_DEVS, &mddev->flags);
> + if (mddev->degraded)
> + set_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
> + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> + md_new_event(mddev);
> + md_wakeup_thread(mddev->thread);
> + return 0;
> +}
> +
> /* words written to sysfs files may, or may not, be \n terminated.
> * We want to accept with case. For this we use cmd_match.
> */
> @@ -2568,7 +2598,10 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
> clear_bit(Replacement, &rdev->flags);
> err = 0;
> }
> - }
> + } else if (cmd_match(buf, "re-add") && (test_bit(Faulty, &rdev->flags) || (rdev->raid_disk == -1))) {
> + clear_bit(Faulty, &rdev->flags);
> + err = add_bound_rdev(rdev);
> + }
I changed this to:
} else if (cmd_match(buf, "re-add")) {
if (test_bit(Faulty, &rdev->flags) && (rdev->raid_disk == -1)) {
clear_bit(Faulty, &rdev->flags);
err = add_bound_rdev(rdev);
} else
err = -EBUSY;
}
because:
1/ I want all branches of the main if/else to be just "cmd_match...",
2/ I want to return EBUSY, if 're-add' was recognised as a command, but
the default wasn't available for a re-add, and
3/ re-add can only be allowed if the device is faulty AND raid_disk is -1.
If not faulty, re-add makes no sense.
If raid_disk is not -1, then the device still has outstanding IO and
we need to keep waiting for that to complete.
Otherwise, patch accepted - thanks.
NeilBrown
> if (!err)
> sysfs_notify_dirent_safe(rdev->sysfs_state);
> return err ? err : len;
> @@ -5882,29 +5915,10 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
>
> rdev->raid_disk = -1;
> err = bind_rdev_to_array(rdev, mddev);
> - if (!err && !mddev->pers->hot_remove_disk) {
> - /* If there is hot_add_disk but no hot_remove_disk
> - * then added disks for geometry changes,
> - * and should be added immediately.
> - */
> - super_types[mddev->major_version].
> - validate_super(mddev, rdev);
> - err = mddev->pers->hot_add_disk(mddev, rdev);
> - if (err)
> - unbind_rdev_from_array(rdev);
> - }
> if (err)
> export_rdev(rdev);
> else
> - sysfs_notify_dirent_safe(rdev->sysfs_state);
> -
> - set_bit(MD_CHANGE_DEVS, &mddev->flags);
> - if (mddev->degraded)
> - set_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
> - set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> - if (!err)
> - md_new_event(mddev);
> - md_wakeup_thread(mddev->thread);
> + err = add_bound_rdev(rdev);
> if (mddev_is_clustered(mddev) &&
> (info->state & (1 << MD_DISK_CLUSTER_ADD)))
> md_cluster_ops->add_new_disk_finish(mddev);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 6/6] md-cluster: re-add capabilities
From: NeilBrown @ 2015-04-20 2:01 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: GQJiang, linux-raid
In-Reply-To: <20150414154541.GA4138@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 7679 bytes --]
On Tue, 14 Apr 2015 10:45:42 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> When "re-add" is writted to /sys/block/mdXX/md/dev-YYY/state,
> the clustered md:
>
> 1. Sends RE_ADD message with the desc_nr. Nodes receiving the message
> clear the Faulty bit in their respective rdev->flags.
> 2. The node initiating re-add, gathers the bitmaps of all nodes
> and copies them into the local bitmap. It does not clear the bitmap
> from which it is copying.
> 3. Initiating node schedules a md recovery to sync the devices.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/bitmap.c | 20 +++++++++++---------
> drivers/md/bitmap.h | 2 +-
> drivers/md/md-cluster.c | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
> drivers/md/md-cluster.h | 1 +
> drivers/md/md.c | 12 ++++++++++++
> 5 files changed, 72 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index 5ff67c3..956cfb9 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -1852,7 +1852,7 @@ EXPORT_SYMBOL_GPL(bitmap_load);
> * to our bitmap
> */
> int bitmap_copy_from_slot(struct mddev *mddev, int slot,
> - sector_t *low, sector_t *high)
> + sector_t *low, sector_t *high, bool clear_bits)
> {
> int rv = 0, i, j;
> sector_t block, lo = 0, hi = 0;
> @@ -1879,14 +1879,16 @@ int bitmap_copy_from_slot(struct mddev *mddev, int slot,
> }
> }
>
> - bitmap_update_sb(bitmap);
> - /* Setting this for the ev_page should be enough.
> - * And we do not require both write_all and PAGE_DIRT either
> - */
> - for (i = 0; i < bitmap->storage.file_pages; i++)
> - set_page_attr(bitmap, i, BITMAP_PAGE_DIRTY);
> - bitmap_write_all(bitmap);
> - bitmap_unplug(bitmap);
> + if (clear_bits) {
> + bitmap_update_sb(bitmap);
> + /* Setting this for the ev_page should be enough.
> + * And we do not require both write_all and PAGE_DIRT either
> + */
> + for (i = 0; i < bitmap->storage.file_pages; i++)
> + set_page_attr(bitmap, i, BITMAP_PAGE_DIRTY);
> + bitmap_write_all(bitmap);
> + bitmap_unplug(bitmap);
> + }
> *low = lo;
> *high = hi;
> err:
> diff --git a/drivers/md/bitmap.h b/drivers/md/bitmap.h
> index 4aabc74..f1f4dd0 100644
> --- a/drivers/md/bitmap.h
> +++ b/drivers/md/bitmap.h
> @@ -263,7 +263,7 @@ void bitmap_daemon_work(struct mddev *mddev);
> int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
> int chunksize, int init);
> int bitmap_copy_from_slot(struct mddev *mddev, int slot,
> - sector_t *lo, sector_t *hi);
> + sector_t *lo, sector_t *hi, bool clear_bits);
> #endif
>
> #endif
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index 82f1b7b..ad2b5b7 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -73,6 +73,7 @@ enum msg_type {
> RESYNCING,
> NEWDISK,
> REMOVE,
> + RE_ADD,
> };
>
> struct cluster_msg {
> @@ -253,7 +254,7 @@ void recover_bitmaps(struct md_thread *thread)
> str, ret);
> goto clear_bit;
> }
> - ret = bitmap_copy_from_slot(mddev, slot, &lo, &hi);
> + ret = bitmap_copy_from_slot(mddev, slot, &lo, &hi, true);
> if (ret) {
> pr_err("md-cluster: Could not copy data from bitmap %d\n", slot);
> goto dlm_unlock;
> @@ -412,6 +413,16 @@ static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
> pr_warn("%s: %d Could not find disk(%d) to REMOVE\n", __func__, __LINE__, msg->raid_slot);
> }
>
> +static void process_readd_disk(struct mddev *mddev, struct cluster_msg *msg)
> +{
> + struct md_rdev *rdev = md_find_rdev_nr_rcu(mddev, msg->raid_slot);
> +
> + if (rdev && test_bit(Faulty, &rdev->flags))
> + clear_bit(Faulty, &rdev->flags);
> + else
> + pr_warn("%s: %d Could not find disk(%d) which is faulty", __func__, __LINE__, msg->raid_slot);
> +}
> +
> static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
> {
> switch (msg->type) {
> @@ -436,6 +447,11 @@ static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
> __func__, __LINE__, msg->slot);
> process_remove_disk(mddev, msg);
> break;
> + case RE_ADD:
> + pr_info("%s: %d Received RE_ADD from %d\n",
> + __func__, __LINE__, msg->slot);
> + process_readd_disk(mddev, msg);
> + break;
> default:
> pr_warn("%s:%d Received unknown message from %d\n",
> __func__, __LINE__, msg->slot);
> @@ -883,6 +899,35 @@ static int remove_disk(struct mddev *mddev, struct md_rdev *rdev)
> return __sendmsg(cinfo, &cmsg);
> }
>
> +static int gather_bitmaps(struct md_rdev *rdev)
> +{
> + int sn, err;
> + sector_t lo, hi;
> + struct cluster_msg cmsg;
> + struct mddev *mddev = rdev->mddev;
> + struct md_cluster_info *cinfo = mddev->cluster_info;
> +
> + cmsg.type = RE_ADD;
> + cmsg.raid_slot = rdev->desc_nr;
> + err = sendmsg(cinfo, &cmsg);
> + if (err)
> + goto out;
> +
> + for (sn = 0; sn < mddev->bitmap_info.nodes; sn++) {
> + if (sn == (cinfo->slot_number - 1))
> + continue;
> + err = bitmap_copy_from_slot(mddev, sn, &lo, &hi, false);
> + if (err) {
> + pr_warn("md-cluster: Could not gather bitmaps from slot %d", sn);
> + goto out;
> + }
> + if ((hi > 0) && (lo < mddev->recovery_cp))
> + mddev->recovery_cp = lo;
> + }
> +out:
> + return err;
> +}
> +
> static struct md_cluster_operations cluster_ops = {
> .join = join,
> .leave = leave,
> @@ -898,6 +943,7 @@ static struct md_cluster_operations cluster_ops = {
> .add_new_disk_finish = add_new_disk_finish,
> .new_disk_ack = new_disk_ack,
> .remove_disk = remove_disk,
> + .gather_bitmaps = gather_bitmaps,
> };
>
> static int __init cluster_init(void)
> diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
> index 71e5143..6817ee0 100644
> --- a/drivers/md/md-cluster.h
> +++ b/drivers/md/md-cluster.h
> @@ -23,6 +23,7 @@ struct md_cluster_operations {
> int (*add_new_disk_finish)(struct mddev *mddev);
> int (*new_disk_ack)(struct mddev *mddev, bool ack);
> int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
> + int (*gather_bitmaps)(struct md_rdev *rdev);
> };
>
> #endif /* _MD_CLUSTER_H */
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index ba01605..8c37bbf 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -2599,11 +2599,23 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
> err = 0;
> }
> } else if (cmd_match(buf, "re-add") && (test_bit(Faulty, &rdev->flags) || (rdev->raid_disk == -1))) {
> + /* clear_bit is performed _after_ all the devices
> + * have their local Faulty bit cleared. If any writes
> + * happen in the meantime in the local node, they
> + * will land in the local bitmap, which will be synced
> + * by this node eventually
> + */
> + if (mddev_is_clustered(rdev->mddev)) {
> + err = md_cluster_ops->gather_bitmaps(rdev);
> + if (err)
> + goto out;
> + }
> clear_bit(Faulty, &rdev->flags);
> err = add_bound_rdev(rdev);
> }
> if (!err)
> sysfs_notify_dirent_safe(rdev->sysfs_state);
> +out:
> return err ? err : len;
> }
> static struct rdev_sysfs_entry rdev_state =
I changed this to:
if (!mddev_is_clustered(rdev->mddev) ||
(err = md_cluster_ops->gather_bitmaps(rdev)) == 0) {
clear_bit(Faulty, &rdev->flags);
err = add_bound_rdev(rdev);
}
because I think it makes the code a bit clearer.
otherwise, applied.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: NeilBrown @ 2015-04-20 6:20 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <550B265C.7070907@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 5874 bytes --]
On Thu, 19 Mar 2015 13:41:16 -0600 Eric Mei <meijia@gmail.com> wrote:
> On 2015-03-19 12:02 AM, NeilBrown wrote:
> > On Wed, 18 Mar 2015 23:39:11 -0600 Eric Mei <meijia@gmail.com> wrote:
> >
> >> From: Eric Mei <eric.mei@seagate.com>
> >>
> >> When array is degraded, read data landed on failed drives will result in
> >> reading rest of data in a stripe. So a single sequential read would
> >> result in same data being read twice.
> >>
> >> This patch is to avoid chunk aligned read for degraded array. The
> >> downside is to involve stripe cache which means associated CPU overhead
> >> and extra memory copy.
> >>
> >> Signed-off-by: Eric Mei <eric.mei@seagate.com>
> >> ---
> >> drivers/md/raid5.c | 15 ++++++++++++---
> >> 1 files changed, 12 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> >> index cd2f96b..763c64a 100644
> >> --- a/drivers/md/raid5.c
> >> +++ b/drivers/md/raid5.c
> >> @@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
> >> unsigned int chunk_sectors = mddev->chunk_sectors;
> >> unsigned int bio_sectors = bvm->bi_size >> 9;
> >>
> >> - if ((bvm->bi_rw & 1) == WRITE)
> >> - return biovec->bv_len; /* always allow writes to be
> >> mergeable */
> >> + /*
> >> + * always allow writes to be mergeable, read as well if array
> >> + * is degraded as we'll go through stripe cache anyway.
> >> + */
> >> + if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
> >> + return biovec->bv_len;
> >>
> >> if (mddev->new_chunk_sectors < mddev->chunk_sectors)
> >> chunk_sectors = mddev->new_chunk_sectors;
> >> @@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
> >> struct bio * bi)
> >>
> >> md_write_start(mddev, bi);
> >>
> >> - if (rw == READ &&
> >> + /*
> >> + * If array is degraded, better not do chunk aligned read because
> >> + * later we might have to read it again in order to reconstruct
> >> + * data on failed drives.
> >> + */
> >> + if (rw == READ && mddev->degraded == 0 &&
> >> mddev->reshape_position == MaxSector &&
> >> chunk_aligned_read(mddev,bi))
> >> return;
> >
> > Thanks for the patch.
> >
> > However this sort of patch really needs to come with some concrete
> > performance numbers. Preferably both sequential reads and random reads.
> >
> > I agree that sequential reads are likely to be faster, but how much faster
> > are they?
> > I imagine that this might make random reads a little slower. Does it? By
> > how much?
> >
> > Thanks,
> > NeilBrown
> >
>
> Hi Neil,
>
> Sorry I should have done the test in first place.
>
> Following test are done on a enterprise storage node with Seagate 6T SAS
> drives and Xeon E5-2648L CPU (10 cores, 1.9Ghz), 10 disks MD RAID6 8+2,
> chunk size 128 KiB.
>
> I use FIO, using direct-io with various bs size, enough queue depth,
> tested sequential and 100% random read against 3 array config: 1)
> optimal, as baseline; 2) degraded; 3) degraded with this patch. Kernel
> version is 4.0-rc3.
>
> Each individual test I only did once so there might be some variations,
> but we just focus on big trend.
>
> Sequential Read:
> bs=(KiB) optimal(MiB/s) degraded(MiB/s) degraded-with-patch (MiB/s)
> 1024 1608 656 995
> 512 1624 710 956
> 256 1635 728 980
> 128 1636 771 983
> 64 1612 1119 1000
> 32 1580 1420 1004
> 16 1368 688 986
> 8 768 647 953
> 4 411 413 850
>
> Random Read:
> bs=(KiB) optimal(IOPS) degraded(IOPS) degraded-with-patch (IOPS)
> 1024 163 160 156
> 512 274 273 272
> 256 426 428 424
> 128 576 592 591
> 64 726 724 726
> 32 849 848 837
> 16 900 970 971
> 8 927 940 929
> 4 948 940 955
>
> Some notes:
> * In sequential + optimal, as bs size getting smaller, the FIO thread
> become CPU bound.
> * In sequential + degraded, there's big increase when bs is 64K and
> 32K, I don't have explanation.
> * In sequential + degraded-with-patch, the MD thread mostly become CPU
> bound.
>
> If you want to we can discuss specific data point in those data. But in
> general it seems with this patch, we have more predictable and in most
> cases significant better sequential read performance when array is
> degraded, and almost no noticeable impact on random read.
>
> Performance is a complicated thing, the patch works well for this
> particular configuration, but may not be universal. For example I
> imagine testing on all SSD array may have very different result. But I
> personally think in most cases IO bandwidth is more scarce resource than
> CPU.
>
> Eric
Thanks.
That is reasonably convincing.
I've added that text to the commit message, fixed up all the white-space
damage in the patch (tabs were converted to spaces etc ... if you are going
to be sending more patches, please find a way to convince your mailer that
spaces are important), and applied it.
It should be included in my pull request for 4.1
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Breaks LSI RAID on C600 chipset
From: David F. @ 2015-04-21 5:39 UTC (permalink / raw)
To: NeilBrown; +Cc: John Stoffel, linux-raid@vger.kernel.org
In-Reply-To: <CAGRSmLvoj-UhV+8hSESQ+RRyzMHZ68QuBdRZzVU0PWEztZWeOw@mail.gmail.com>
If you need access to the system, a network kvm is available - or if
you have a C600 based system, you should see the same problem there.
On Fri, Apr 10, 2015 at 1:07 PM, David F. <df7729@gmail.com> wrote:
> Okay, and to confirm, this is happening on other C600 based systems.
> Boot to Linux with MDADM raid support and the raid is gone after
> reboot.
>
> On Thu, Apr 9, 2015 at 5:50 PM, NeilBrown <neilb@suse.de> wrote:
>> On Thu, 9 Apr 2015 14:13:05 -0700 "David F." <df7729@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I built a new system, installed new drives (no partitioning, just raw
>>> new drives), configured RAID 1, boot to linux, created reports
>>> attached. Rebooted and the system doesn't see any configured raid
>>> drives (just the two raw drives).
>>
>> Thanks. I might take a look, but I would greatly prefer it if you kept the
>> linux-raid list on the Cc.....
>>
>> NeilBrown
>>
^ permalink raw reply
* [RFC] A caching layer for RAID5/6
From: Shaohua Li @ 2015-04-21 5:54 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, songliubraving, Kernel-team
Hi,
This is the new design/code to implement caching layer for RAID5/6. It's not
for merge yet, but basic code works fine here. I'd like to hear some
comments/suggestions early before I continue.
Thanks,
Shaohua
Main goal of the caching layer is to aggregate IO write to hopefully
make full stripe IO write and fix write hole issue. This might speed up
read too, but it's not optimized for read, eg, we don't proactively
cache data for read. The aggregation makes a lot of sense for workloads
which sequentially write to several files with/without fsync. Such
workloads are popular in today's datacenter.
Write IO data will write to a cache disk (SSD) first, then later the
data will be flushed to raid disks.
The cache disk will be organized as a simple ring buffer log. For IO
data, a tuple (raid_sector, io length, checksum, data) will be appended
to the log; for raid 5/6 parity, a tuple (stripe_sector, parity length,
checksum, parity data) will be appended to the log. We don't have
on-disk index for the data appended to the log. So either we can rebuild
an in-memory index at startup with scanning the whole log, or we can
flush all data from cache disk to raid disks at shutdown so cache disk
has no valid data. Current code chooses the first option, but this can
be easily changed.
We have a simple meta data for the above tuples. It's essentially a
tuple (sequence, metadata length). Crash recovery or startup will scan
the log to read the metadata and rebuild in-memory index. If metadata is
broken at the head of the log, even metadata afterward is ok, the
scanning will not work well. So we take some steps to mitigate the
issue:
-meta data is written with WRITE_FUA. write IO return only happens after both
data and metadata IO finish
-IO return happens in the order they are added in the log.
These will make sure write IO return happens after its meta and all
previous meta IO finish, eg all related meta are persistent.
The in-memory index is a simple list of io_range (sequence, metadata
sector, data sector, length). The list is orded by sequence. The first
io_range entry's metadata sector is the tail of the log. There is also a
struct to track io ranges within a stripe. All stripes will be organized
as a radix tree.
All IO data will be copied to a memory pool for caching too until the
data is flushed to raid disks. This is just to simplify the
implementation, it's not mandated. In the future flush can do a data
copy from cache disk to raid disks, so the memory pool can be discarded.
If a stripe is flushed to raid disks, memory of the stripe can be
reused.
We have two limited resources here, memory pool and cache disk space. If
resource is tight, we will do reclaim. In either case, we will flush
some data from cache disk to raid disks. However, we implement different
strategies. For memory reclaim, we prefer reclaiming full stripe. For
cache disk space reclaim, we prefer reclaiming io_range entry at the
head of index list.
We always do reclaim in stripe unit. Reclaim could create holes in the
log, eg, some io_range in the middle is reclaimed, but io_range at the
head remains. So the index list entries don't always have continuous
sequence. But this doesn't matter, the first io_range is always the log
tail. Superblock has a field pointing to the position of log tail. The
hole can waste a lot of disk space though. In the future, we can
implement a garbage collection to mitigrate the issue, eg, copy data
from the index tail to head.
In the process reclaim flush data to raid disk, stripe parity will be
append to cache log. Parity is always after its corresponding data. Data
writing to raid disks only happens after all data and parity are already
in cache disk. This will fix the write whole issue. After a stripe is
flushed to raid disks, we will add a checkpoint. The checkpoint, unlike
classic checkpoint, indicates a stripe is flushed to raid disks only if
the stripe has both data and parity in the log ahead of the chekcpoint.
There are still valid data of other stripes before checkpoint.
Recovery relies on if stripe data and parity matches. If recovery finds
data and parity of a stripe, it will calculate how many parity data the
stripe should have according to its data. If not all parity exist in the
log, the stripe hasn't started to be flushed to raid disks yet, so the
parity can be ignored. Otherwise, recovery might write both data and
parity of a stripe to raid disks depending on if checkpoint is found.
IO write code path:
1. copy bio data to stripe memory pages
2. append metadata and data to cache log
3. IO write endio
reclaim code path:
1. select stripe to reclaim
2. write all stripe data to raid disk
3. in raid5 ops_run_io, append metadata and parity data to cache log.
ops_run_io doesn't write data/parity to raid disks at this time
4. flush cache disk cache
5. ops_run_io continues. data/parity will be written to raid disks
6. flush all raid disks cache
7. add a checkpoint
8. delete in-memory index of the stripe, and advance superblock log checkpoint
Recovery:
Crash in IO write code path doesn't need recovery. If data and checksum
don't match, the data will be ignored so read will return old data. In
reclaim code path, crash before step 4 doesn't need recovery as
data/parity don't touch raid disk yet. Parity can be ignored too. crash
after 7 doesn't need recovery too, as the stripe is fully flushed to
raid disks. Crash between 4 and 7 need recovery. Data and parity in the
log will be written to raid disks.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/Makefile | 2 +-
drivers/md/raid5-cache.c | 2412 ++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 51 +-
drivers/md/raid5.h | 10 +
include/uapi/linux/raid/md_p.h | 70 ++
5 files changed, 2540 insertions(+), 5 deletions(-)
create mode 100644 drivers/md/raid5-cache.c
diff --git a/drivers/md/Makefile b/drivers/md/Makefile
index a2da532..7d85784 100644
--- a/drivers/md/Makefile
+++ b/drivers/md/Makefile
@@ -16,7 +16,7 @@ dm-cache-mq-y += dm-cache-policy-mq.o
dm-cache-cleaner-y += dm-cache-policy-cleaner.o
dm-era-y += dm-era-target.o
md-mod-y += md.o bitmap.o
-raid456-y += raid5.o
+raid456-y += raid5.o raid5-cache.o
# Note: link order is important. All raid personalities
# and must come before md.o, as they each initialise
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
new file mode 100644
index 0000000..cfb8bed
--- /dev/null
+++ b/drivers/md/raid5-cache.c
@@ -0,0 +1,2412 @@
+#include <linux/kernel.h>
+#include <linux/wait.h>
+#include <linux/blkdev.h>
+#include <linux/raid/md_p.h>
+#include <linux/crc32.h>
+#include "md.h"
+#include "raid5.h"
+
+#define MAX_MEM (256 * 1024 * 1024)
+#define RECLAIM_BATCH 16
+#define RECLAIM_TIMEOUT (5 * HZ)
+#define FULL_STRIPE_MAX_AGE (10 * HZ)
+
+struct r5l_log {
+ struct r5c_cache *cache;
+ struct block_device *bdev;
+ struct md_rdev *rdev;
+
+ struct page *super_page;
+ u8 uuid[16];
+ u32 uuid_checksum_data;
+ u32 uuid_checksum_meta;
+
+ int block_size; /* bytes */
+ int block_sector_shift;
+ int page_block_shift;
+ int stripe_data_size; /* sector */
+ int chunk_size; /* sector */
+ int stripe_size; /* sector */
+ int parity_disks;
+
+ u64 total_blocks;
+ u64 first_block;
+ u64 last_block;
+
+ u64 low_watermark; /* For disk space, block */
+ u64 high_watermark;
+
+ u64 last_checkpoint; /* block */
+ u64 last_cp_seq;
+
+ int do_discard;
+
+ u64 seq; /* get after read log */
+ u64 log_start; /* get after read log */
+
+ u8 data_checksum_type;
+ u8 meta_checksum_type;
+
+ ssize_t reserved_blocks;
+ wait_queue_head_t space_waitq;
+
+ struct mutex io_mutex;
+ struct r5l_ioctl *current_io;
+
+ spinlock_t io_list_lock;
+ struct list_head running_ios; /* order is important */
+ wait_queue_head_t io_waitq;
+};
+
+struct r5l_task;
+/* end function must free task */
+typedef void (r5l_task_end_fn)(struct r5l_task *task);
+struct r5l_task {
+ struct list_head list;
+ int type;
+ struct bio *bio;
+ union {
+ struct {
+ bio_end_io_t *saved_endio;
+ void *saved_private;
+ };
+ struct {
+ sector_t stripe_sector;
+ struct page *page_p;
+ struct page *page_q;
+ };
+ };
+ /* tasks in a single r5l_ioctl will have the same seq and meta_start */
+ sector_t meta_start;
+ sector_t data_start;
+ u64 seq;
+ r5l_task_end_fn *fn;
+ void *private;
+ u32 checksum[];
+};
+
+/*
+ * meta page should write with FUA, and meta page should return IO in order,
+ * otherwise recovery will have trouble. The reason is we are using a simple
+ * log. If metadata corrupts in the middle, recovery can't work even metadata
+ * at the end is good.
+ */
+struct r5l_ioctl {
+ struct r5l_log *log;
+ struct list_head log_sibling;
+
+ struct page *meta_page;
+ sector_t meta_sector;
+ int meta_offset;
+ u64 seq;
+ struct bio *meta_bio;
+
+ struct list_head tasks;
+ struct bio *current_bio;
+ atomic_t refcnt;
+};
+
+struct r5l_rsv {
+ ssize_t reserved;
+ ssize_t used;
+};
+
+struct r5c_io_range {
+ struct list_head log_sibling;
+ struct list_head stripe_sibling;
+
+ u64 seq;
+
+ sector_t meta_start; /* cache position */
+ sector_t data_start; /* cache position */
+ sector_t raid_start;
+ ssize_t data_sectors;
+
+ struct r5c_stripe *stripe;
+ union {
+ struct bio *bio;
+ u32 *checksum; /* only for recovery */
+ };
+};
+
+struct r5c_stripe {
+ u64 raid_index;
+ struct r5c_cache *cache;
+ atomic_t ref;
+ int state;
+
+ struct list_head io_ranges; /* order list */
+ union {
+ struct list_head stripes;
+ struct list_head parity_list; /* just for recovery */
+ };
+
+ struct list_head lru;
+
+ int existing_pages;
+ atomic_t dirty_stripes;
+ atomic_t pending_bios;
+ struct page **parity_pages; /* just for recovery */
+ struct page *data_pages[];
+};
+
+enum {
+ STRIPE_RUNNING = 0,
+ STRIPE_FROZEN = 1, /* Doesn't accept new IO */
+ STRIPE_PARITY_DONE = 2,
+ STRIPE_INRAID = 3,
+ STRIPE_DEAD = 4,
+};
+
+#define STRIPE_LOCK_BITS 8
+struct r5c_cache {
+ struct mddev *mddev;
+ struct md_rdev *rdev;
+
+ struct r5l_log log;
+
+ spinlock_t tree_lock; /* protect stripe_tree, log_list, full_stripes */
+ struct radix_tree_root stripe_tree;
+ struct list_head log_list; /* sorted list of io_range */
+ struct list_head full_stripes;
+
+ struct list_head page_pool;
+ spinlock_t pool_lock;
+ u64 free_pages;
+ u64 total_pages;
+ u64 max_pages;
+ u64 low_watermark; /* for memory, pages */
+ u64 high_watermark;
+ unsigned long full_stripe_ready_time;
+
+ int stripe_data_size; /* stripe size excluding parity, sector */
+ int chunk_size; /* one disk chunk size including parity, sector */
+ int stripe_size; /* stripe size including parity, sector */
+ int parity_disks;
+
+ int reserved_space; /* log reserved size, sector */
+
+ unsigned long reclaim_reason;
+ wait_queue_head_t reclaim_wait;
+ struct md_thread *reclaim_thread;
+
+ int in_recovery;
+
+ spinlock_t stripe_locks[1 << STRIPE_LOCK_BITS];
+ wait_queue_head_t stripe_waitq[1 << STRIPE_LOCK_BITS];
+};
+
+enum {
+ RECLAIM_MEM = 0, /* work hard to reclaim memory */
+ RECLAIM_MEM_BACKGROUND = 1, /* try to reclaim memory */
+ RECLAIM_MEM_FULL = 2, /* only reclaim full stripe */
+ RECLAIM_DISK = 8, /* work hard to reclaim disk */
+ RECLAIM_DISK_BACKGROUND = 9, /* try to reclaim disk */
+};
+
+#define STRIPE_INDEX_OFFSET(c, sect, index, offset) \
+({ \
+ sector_t tmp = sect; \
+ offset = sector_div(tmp, c->stripe_data_size); \
+ index = tmp; \
+})
+
+#define STRIPE_RAID_SECTOR(cache, stripe) \
+ (stripe->raid_index * cache->stripe_data_size)
+
+#define PAGE_SECTOR_SHIFT (PAGE_SHIFT - 9)
+
+#define STRIPE_DATA_PAGES(c) (c->stripe_data_size >> PAGE_SECTOR_SHIFT)
+#define STRIPE_PARITY_PAGES(c) \
+ ((c->stripe_size - c->stripe_data_size) >> PAGE_SECTOR_SHIFT)
+#define BLOCK_SECTOR(log, b) ((b) << log->block_sector_shift)
+#define PAGE_BLOCKS(log, p) ((p) >> log->page_block_shift)
+
+#define UUID_CHECKSUM(log, data) \
+ (data ? log->uuid_checksum_data : log->uuid_checksum_meta)
+
+static u32 r5l_calculate_checksum(struct r5l_log *log, u32 crc,
+ void *buf, ssize_t size, bool data)
+{
+ if (log->data_checksum_type != R5LOG_CHECKSUM_CRC32)
+ BUG();
+ if (log->meta_checksum_type != R5LOG_CHECKSUM_CRC32)
+ BUG();
+ return crc32_le(crc, buf, size);
+}
+
+static u64 r5l_ring_add(struct r5l_log *log, u64 block, int inc)
+{
+ block += inc;
+ if (block >= log->last_block)
+ block = block - log->last_block + log->first_block;
+ return block;
+}
+
+static void r5c_wake_wait_reclaimer(struct r5c_cache *cache, int reason);
+static void r5c_wake_reclaimer(struct r5c_cache *cache, int reason);
+static void r5l_reclaim_space(struct r5l_log *log)
+{
+ r5c_wake_wait_reclaimer(log->cache, RECLAIM_DISK);
+ wake_up_all(&log->space_waitq);
+}
+
+static u64 r5l_free_space(struct r5l_log *log)
+{
+ u64 used_size = (log->total_blocks + log->log_start -
+ log->last_checkpoint) % log->total_blocks;
+ return log->total_blocks - used_size - log->reserved_blocks;
+}
+
+static int r5l_get_reserve(struct r5l_log *log,
+ struct r5l_rsv *rsv, ssize_t size, bool parity)
+{
+ u64 free;
+ BUG_ON(!mutex_is_locked(&log->io_mutex));
+
+ if (parity)
+ size = 0;
+ rsv->reserved = size;
+ rsv->used = 0;
+
+ free = r5l_free_space(log);
+ if (free >= size) {
+ log->reserved_blocks += size;
+
+ if (free - size < log->low_watermark)
+ r5c_wake_reclaimer(log->cache,
+ RECLAIM_DISK_BACKGROUND);
+ return 0;
+ }
+
+ mutex_unlock(&log->io_mutex);
+
+ r5l_reclaim_space(log);
+
+ mutex_lock(&log->io_mutex);
+ wait_event_cmd(log->space_waitq, r5l_free_space(log) >= size,
+ mutex_unlock(&log->io_mutex), mutex_lock(&log->io_mutex));
+ log->reserved_blocks += size;
+
+ return 0;
+}
+
+static void r5l_put_reserve(struct r5l_log *log,
+ struct r5l_rsv *rsv)
+{
+ ssize_t remaining;
+
+ BUG_ON(!mutex_is_locked(&log->io_mutex));
+ remaining = rsv->reserved - rsv->used;
+
+ if (remaining < 0)
+ remaining = 0;
+ log->reserved_blocks -= remaining;
+ if (remaining > 0)
+ wake_up_all(&log->space_waitq);
+}
+
+static void r5l_submit_bio(struct r5l_log *log, int rw, struct bio *bio)
+{
+ /* all IO must start from rdev->data_offset */
+ struct bio *next = bio->bi_next;
+
+ /* queue_task split bio into bio->bi_next */
+ if (next) {
+ next->bi_iter.bi_sector += log->rdev->data_offset;
+ submit_bio(rw, next);
+ bio->bi_next = NULL;
+ }
+
+ bio->bi_iter.bi_sector += log->rdev->data_offset;
+ submit_bio(rw, bio);
+}
+
+static void r5l_put_ioctl(struct r5l_ioctl *io)
+{
+ struct r5l_log *log = io->log;
+ LIST_HEAD(finished_io);
+ unsigned long flags;
+ bool do_wakeup;
+
+ if (!atomic_dec_and_test(&io->refcnt))
+ return;
+
+ /*
+ * finish IO in order. Note this guarantees later task runs ->fn after
+ * previous task's IO and this task's IO are finished, but doesn't mean
+ * previous task already runs its ->fn
+ */
+ spin_lock_irqsave(&log->io_list_lock, flags);
+ while (!list_empty(&log->running_ios)) {
+ io = list_first_entry(&log->running_ios, struct r5l_ioctl,
+ log_sibling);
+ if (atomic_read(&io->refcnt))
+ break;
+ list_move_tail(&io->log_sibling, &finished_io);
+
+ }
+ do_wakeup = list_empty(&log->running_ios);
+ spin_unlock_irqrestore(&log->io_list_lock, flags);
+
+ while (!list_empty(&finished_io)) {
+ struct r5l_task *task;
+
+ io = list_first_entry(&finished_io, struct r5l_ioctl,
+ log_sibling);
+ list_del(&io->log_sibling);
+
+ while (!list_empty(&io->tasks)) {
+ task = list_first_entry(&io->tasks, struct r5l_task,
+ list);
+ list_del(&task->list);
+ if (task->type == R5LOG_PAYLOAD_DATA) {
+ task->bio->bi_end_io = task->saved_endio;
+ task->bio->bi_private = task->saved_private;
+ }
+ task->fn(task);
+ }
+ __free_page(io->meta_page);
+ kfree(io);
+ }
+ if (do_wakeup)
+ wake_up(&log->io_waitq);
+}
+
+static void r5l_log_endio(struct bio *bio, int error)
+{
+ struct r5l_ioctl *io = bio->bi_private;
+
+ bio_put(bio);
+ r5l_put_ioctl(io);
+}
+
+static int r5l_submit_io(struct r5l_log *log)
+{
+ struct r5l_ioctl *io = log->current_io;
+ struct r5l_task *task;
+ struct r5l_meta_header *header;
+ u32 crc;
+
+ header = kmap_atomic(io->meta_page);
+ header->meta_size = cpu_to_le32(io->meta_offset);
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false),
+ header, log->block_size, false);
+ header->checksum = cpu_to_le32(crc);
+ kunmap_atomic(header);
+
+ r5l_submit_bio(log, WRITE_FUA, io->meta_bio);
+
+ list_for_each_entry(task, &io->tasks, list) {
+ if (task->bio)
+ r5l_submit_bio(log, task->bio->bi_rw, task->bio);
+ }
+ log->current_io = NULL;
+ r5l_put_ioctl(io);
+ return 0;
+}
+
+static struct r5l_ioctl *r5l_new_meta(struct r5l_log *log, struct r5l_rsv *rsv)
+{
+ struct r5l_ioctl *io;
+
+ io = kmalloc(sizeof(*io), GFP_NOIO);
+ io->log = log;
+ io->meta_page = alloc_page(GFP_NOIO|__GFP_ZERO);
+ io->meta_sector = BLOCK_SECTOR(log, log->log_start);
+ io->meta_offset = sizeof(struct r5l_meta_header);
+ atomic_set(&io->refcnt, 1);
+ INIT_LIST_HEAD(&io->tasks);
+ io->seq = log->seq;
+
+ io->meta_bio = bio_kmalloc(GFP_NOIO, bio_get_nr_vecs(log->bdev));
+ io->meta_bio->bi_bdev = log->bdev;
+ io->meta_bio->bi_iter.bi_sector = io->meta_sector;
+ bio_add_page(io->meta_bio, io->meta_page, log->block_size, 0);
+ io->meta_bio->bi_end_io = r5l_log_endio;
+ io->meta_bio->bi_private = io;
+
+ atomic_inc(&io->refcnt);
+
+ log->seq++;
+ log->log_start = r5l_ring_add(log, log->log_start, 1);
+ io->current_bio = io->meta_bio;
+
+ spin_lock_irq(&log->io_list_lock);
+ list_add_tail(&io->log_sibling, &log->running_ios);
+ spin_unlock_irq(&log->io_list_lock);
+
+ rsv->used++;
+ return io;
+}
+
+static int r5l_get_meta(struct r5l_log *log, struct r5l_rsv *rsv,
+ ssize_t pages, bool is_bio)
+{
+ struct r5l_ioctl *io;
+ struct r5l_meta_header *header;
+ ssize_t meta_size;
+
+ meta_size = sizeof(struct r5l_meta_payload) +
+ sizeof(u32) * pages;
+ io = log->current_io;
+ if (io && io->meta_offset + meta_size > log->block_size)
+ r5l_submit_io(log);
+ io = log->current_io;
+ if (io)
+ return 0;
+
+ io = r5l_new_meta(log, rsv);
+ log->current_io = io;
+
+ if (is_bio)
+ io->current_bio = NULL;
+
+ header = kmap_atomic(io->meta_page);
+ header->magic = cpu_to_le32(R5LOG_MAGIC);
+ header->type = cpu_to_le32(R5LOG_TYPE_META);
+ header->seq = cpu_to_le64(log->seq - 1);
+ if (log->log_start == log->first_block)
+ header->position = cpu_to_le64(log->last_block - 1);
+ else
+ header->position = cpu_to_le64(log->log_start - 1);
+ kunmap_atomic(header);
+
+ return 0;
+}
+
+static int r5l_log_add_parity_pages(struct r5l_log *log,
+ struct r5l_task *task)
+{
+ struct r5l_ioctl *io = log->current_io;
+ struct bio *bio;
+ struct page *pages[] = {task->page_p, task->page_q};
+ int i;
+
+ bio = io->current_bio;
+
+alloc_bio:
+ if (!bio) {
+ /* bio can only contain one page ? */
+ BUG_ON(task->bio);
+ bio = bio_kmalloc(GFP_NOIO, bio_get_nr_vecs(log->bdev));
+ bio->bi_rw = WRITE;
+ task->bio = bio;
+ io->current_bio = bio;
+ bio->bi_bdev = log->bdev;
+ bio->bi_iter.bi_sector = BLOCK_SECTOR(log, log->log_start);
+ }
+
+ for (i = 0; i < ARRAY_SIZE(pages); i++) {
+ if (!pages[i])
+ continue;
+ if (!bio_add_page(bio, pages[i], PAGE_SIZE, 0)) {
+ bio = NULL;
+ goto alloc_bio;
+ }
+ pages[i] = NULL;
+ }
+ return 0;
+}
+
+static int r5l_queue_task(struct r5l_log *log, struct r5l_task *task)
+{
+ ssize_t pages;
+ struct r5l_ioctl *io;
+ struct r5l_rsv rsv;
+ struct r5l_meta_payload *payload;
+ struct bio *bio;
+ void *meta;
+ int i;
+
+ if (task->type == R5LOG_PAYLOAD_DATA)
+ pages = bio_sectors(task->bio) >> PAGE_SECTOR_SHIFT;
+ else
+ pages = !!task->page_p + !!task->page_q;
+
+ /* FIXME: handle tasks in batch to be more efficient */
+ mutex_lock(&log->io_mutex);
+
+ r5l_get_reserve(log, &rsv, 1 + (pages << log->page_block_shift),
+ task->type != R5LOG_PAYLOAD_DATA);
+
+ r5l_get_meta(log, &rsv, pages, task->type == R5LOG_PAYLOAD_DATA);
+
+ io = log->current_io;
+
+ meta = kmap_atomic(io->meta_page);
+ payload = meta + io->meta_offset;
+ payload->payload_type = cpu_to_le16(task->type);
+ payload->blocks = cpu_to_le32(PAGE_BLOCKS(log, pages));
+ if (task->type == R5LOG_PAYLOAD_DATA) {
+ bio = task->bio;
+ payload->location = cpu_to_le64(bio->bi_iter.bi_sector);
+ } else
+ payload->location = cpu_to_le64(task->stripe_sector);
+ for (i = 0; i < pages; i++)
+ payload->data_checksum[i] = cpu_to_le32(task->checksum[i]);
+ kunmap_atomic(meta);
+
+ io->meta_offset += sizeof(struct r5l_meta_payload) +
+ sizeof(u32) * pages;
+ task->seq = io->seq;
+ task->meta_start = io->meta_sector;
+ task->data_start = BLOCK_SECTOR(log, log->log_start);
+
+ if (task->type == R5LOG_PAYLOAD_PARITY)
+ r5l_log_add_parity_pages(log, task);
+
+ if (task->bio) {
+ bio = task->bio;
+ bio->bi_iter.bi_sector = task->data_start;
+ bio->bi_end_io = r5l_log_endio;
+ bio->bi_private = io;
+ bio->bi_bdev = log->bdev;
+ atomic_inc(&io->refcnt);
+ }
+
+ if (task->type == R5LOG_PAYLOAD_PARITY)
+ bio = io->current_bio;
+ else {
+ bio = task->bio;
+ /* endio will bio_put, but this bio isn't allocated by us */
+ bio_get(bio);
+ }
+
+ if (bio_end_sector(bio) > (BLOCK_SECTOR(log, log->last_block))) {
+ struct bio *split;
+ split = bio_split(bio, BLOCK_SECTOR(log, log->last_block) -
+ bio->bi_iter.bi_sector, GFP_NOIO, NULL);
+ bio->bi_next = split;
+ bio_chain(split, bio);
+ bio->bi_iter.bi_sector = BLOCK_SECTOR(log, log->first_block);
+ }
+
+ log->log_start = r5l_ring_add(log, log->log_start,
+ PAGE_BLOCKS(log, pages));
+ rsv.used += PAGE_BLOCKS(log, pages);
+
+ list_add_tail(&task->list, &io->tasks);
+
+ r5l_submit_io(log);
+
+ r5l_put_reserve(log, &rsv);
+ mutex_unlock(&log->io_mutex);
+ return 0;
+}
+
+/*
+ * Since we finish IO in order and meta is written with FUA, so meta is always
+ * consistent. For FUA, just make sure both meta and data are written with FUA
+ * and finish IO after both parts are finished. For FLUSH, previous data and
+ * meta are already finished, just need flush the disk. In either case, bio
+ * already has the flag, so just handle them like normal bio
+ */
+static void r5l_queue_empty_flush_bio(struct r5l_log *log, struct bio *bio)
+{
+ bio->bi_bdev = log->bdev;
+ generic_make_request(bio);
+}
+
+static int r5l_queue_bio(struct r5l_log *log, struct bio *bio,
+ r5l_task_end_fn fn, void *private)
+{
+ struct r5l_task *task;
+ int pages = bio_sectors(bio) >> PAGE_SECTOR_SHIFT;
+ u32 *checksum;
+ struct bio *src = bio;
+ struct bvec_iter iter;
+ struct bio_vec bv;
+ void *src_p;
+ unsigned int page_index = 0, page_offset = 0;
+ unsigned int bytes;
+
+ task = kmalloc(sizeof(*task) + sizeof(u32) * pages, GFP_NOIO);
+
+ INIT_LIST_HEAD(&task->list);
+ task->type = R5LOG_PAYLOAD_DATA;
+ task->bio = bio;
+ task->saved_endio = bio->bi_end_io;
+ task->saved_private = bio->bi_private;
+ task->fn = fn;
+ task->private = private;
+
+ checksum = (u32 *)(task + 1);
+
+ iter = src->bi_iter;
+
+ checksum[0] = UUID_CHECKSUM(log, true);
+ while (1) {
+ if (!iter.bi_size) {
+ src = src->bi_next;
+ if (!src)
+ break;
+
+ iter = src->bi_iter;
+ }
+
+ if (page_offset == PAGE_SIZE) {
+ page_index++;
+ page_offset = 0;
+ checksum[page_index] = UUID_CHECKSUM(log, true);
+ }
+
+ bv = bio_iter_iovec(src, iter);
+
+ bytes = min_t(unsigned int, bv.bv_len, PAGE_SIZE - page_offset);
+
+ src_p = kmap_atomic(bv.bv_page);
+
+ checksum[page_index] = r5l_calculate_checksum(log,
+ checksum[page_index], src_p + bv.bv_offset,
+ bytes, true);
+
+ kunmap_atomic(src_p);
+
+ bio_advance_iter(src, &iter, bytes);
+ page_offset += bytes;
+ }
+ return r5l_queue_task(log, task);
+}
+
+static int r5l_queue_parity(struct r5l_log *log,
+ sector_t stripe_sector, struct page *page_p,
+ struct page *page_q, r5l_task_end_fn fn, void *private)
+{
+ struct r5l_task *task;
+ void *addr;
+ u32 *checksum;
+
+ task = kmalloc(sizeof(*task) + sizeof(u32) * 2, GFP_NOIO);
+
+ INIT_LIST_HEAD(&task->list);
+ task->type = R5LOG_PAYLOAD_PARITY;
+ task->bio = NULL;
+ task->stripe_sector = stripe_sector;
+ task->page_p = page_p;
+ task->page_q = page_q;
+ task->fn = fn;
+ task->private = private;
+
+ checksum = (u32 *)(task + 1);
+ addr = kmap_atomic(page_p);
+ checksum[0] = r5l_calculate_checksum(log,
+ UUID_CHECKSUM(log, true), addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+
+ if (page_q) {
+ addr = kmap_atomic(page_q);
+ checksum[1] = r5l_calculate_checksum(log,
+ UUID_CHECKSUM(log, true), addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+ } else
+ checksum[1] = 0;
+
+ return r5l_queue_task(log, task);
+}
+
+/* all previous queued data are settled down */
+static void r5l_commit(struct r5l_log *log)
+{
+ blkdev_issue_flush(log->bdev, GFP_NOIO, NULL);
+}
+
+static void r5l_cp_end(struct r5l_task *task)
+{
+ struct completion *comp = task->private;
+ complete(comp);
+}
+
+static int r5l_checkpoint(struct r5l_log *log, u64 *next_seq, sector_t *next_sec)
+{
+ struct r5l_ioctl *io;
+ struct r5l_rsv rsv;
+ struct r5l_checkpoint_block *cp;
+ struct timespec now = current_kernel_time();
+ struct r5l_task task;
+ DECLARE_COMPLETION_ONSTACK(cp_complete);
+ u32 crc;
+
+ mutex_lock(&log->io_mutex);
+ r5l_get_reserve(log, &rsv, 1, true);
+
+ if (log->current_io)
+ r5l_submit_io(log);
+
+ /* FIXME: we already wait in reclaim thread, don't need wait again */
+ spin_lock_irq(&log->io_list_lock);
+ wait_event_lock_irq(log->io_waitq, list_empty(&log->running_ios),
+ log->io_list_lock);
+ spin_unlock_irq(&log->io_list_lock);
+
+ io = r5l_new_meta(log, &rsv);
+ task.fn = r5l_cp_end;
+ task.private = &cp_complete;
+ /* pretend we are parity, so r5l_put_ioctl knows how to handle it */
+ task.type = R5LOG_PAYLOAD_PARITY;
+
+ cp = kmap_atomic(io->meta_page);
+ cp->header.magic = cpu_to_le32(R5LOG_MAGIC);
+ cp->header.type = cpu_to_le32(R5LOG_TYPE_CHECKPOINT);
+ cp->header.seq = cpu_to_le64(log->seq - 1);
+ if (log->log_start == log->first_block)
+ cp->header.position = cpu_to_le64(log->last_block - 1);
+ else
+ cp->header.position = cpu_to_le64(log->log_start - 1);
+ cp->header.meta_size = cpu_to_le32(sizeof(struct r5l_checkpoint_block));
+ cp->cp_sec = cpu_to_le64(now.tv_sec);
+ cp->cp_nsec = cpu_to_le64(now.tv_nsec);
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false), cp,
+ log->block_size, false);
+ cp->header.checksum = cpu_to_le32(crc);
+ kunmap_atomic(cp);
+
+ list_add_tail(&task.list, &io->tasks);
+
+ r5l_submit_bio(log, WRITE_FLUSH_FUA, io->meta_bio);
+ r5l_put_ioctl(io);
+
+ r5l_put_reserve(log, &rsv);
+ *next_seq = log->seq;
+ *next_sec = BLOCK_SECTOR(log, log->log_start);
+
+ mutex_unlock(&log->io_mutex);
+
+ wait_for_completion_io(&cp_complete);
+ return 0;
+}
+
+static void r5l_discard_blocks(struct r5l_log *log, u64 start, u64 end)
+{
+ if (!log->do_discard)
+ return;
+ if (start < end) {
+ blkdev_issue_discard(log->bdev,
+ BLOCK_SECTOR(log, start) + log->rdev->data_offset,
+ BLOCK_SECTOR(log, end - start), GFP_NOIO, 0);
+ } else {
+ blkdev_issue_discard(log->bdev,
+ BLOCK_SECTOR(log, start) + log->rdev->data_offset,
+ BLOCK_SECTOR(log, log->last_block - start),
+ GFP_NOIO, 0);
+ blkdev_issue_discard(log->bdev,
+ BLOCK_SECTOR(log, log->first_block) +
+ log->rdev->data_offset,
+ BLOCK_SECTOR(log, end - log->first_block),
+ GFP_NOIO, 0);
+ }
+}
+
+static int r5l_write_super(struct r5l_log *log, u64 seq, sector_t cp)
+{
+ struct r5c_cache *cache = log->cache;
+ struct r5l_super_block *sb_blk;
+ u32 crc;
+
+ if (seq == log->last_cp_seq)
+ return 0;
+
+ mutex_lock(&log->io_mutex);
+ r5l_discard_blocks(log, log->last_checkpoint, cp);
+
+ log->last_cp_seq = seq;
+ log->last_checkpoint = cp >> log->block_sector_shift;
+
+ clear_bit(RECLAIM_DISK, &cache->reclaim_reason);
+ if (r5l_free_space(log) > log->high_watermark)
+ clear_bit(RECLAIM_DISK_BACKGROUND, &cache->reclaim_reason);
+
+ mutex_unlock(&log->io_mutex);
+
+ sb_blk = kmap_atomic(log->super_page);
+ sb_blk->header.seq = cpu_to_le64(seq);
+ sb_blk->last_checkpoint = cpu_to_le64(cp);
+ sb_blk->header.checksum = 0;
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false),
+ sb_blk, log->block_size, false);
+ sb_blk->header.checksum = cpu_to_le32(crc);
+ kunmap_atomic(sb_blk);
+
+ if (!sync_page_io(log->rdev, 0, log->block_size, log->super_page,
+ WRITE_FUA, false))
+ return -EIO;
+
+ return 0;
+}
+
+static void r5c_lock_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ unsigned long *flags)
+{
+ spinlock_t *lock;
+
+ lock = &cache->stripe_locks[hash_ptr(stripe, STRIPE_LOCK_BITS)];
+ spin_lock_irqsave(lock, *flags);
+}
+
+static void r5c_unlock_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ unsigned long *flags)
+{
+ spinlock_t *lock;
+
+ lock = &cache->stripe_locks[hash_ptr(stripe, STRIPE_LOCK_BITS)];
+ spin_unlock_irqrestore(lock, *flags);
+}
+
+static wait_queue_head_t *r5c_stripe_waitq(struct r5c_cache *cache,
+ struct r5c_stripe *stripe)
+{
+ return &cache->stripe_waitq[hash_ptr(stripe, STRIPE_LOCK_BITS)];
+}
+
+static void r5c_stripe_wait_state(struct r5c_cache *cache,
+ struct r5c_stripe *stripe, int state)
+{
+ wait_event(*r5c_stripe_waitq(cache, stripe), stripe->state >= state);
+}
+
+static struct page *r5c_get_page(struct r5c_cache *cache)
+{
+ struct page *page;
+again:
+ spin_lock_irq(&cache->pool_lock);
+ if (!list_empty(&cache->page_pool)) {
+ page = list_first_entry(&cache->page_pool,
+ struct page, lru);
+ list_del_init(&page->lru);
+ cache->free_pages--;
+ if (cache->free_pages < cache->low_watermark)
+ r5c_wake_reclaimer(cache,
+ RECLAIM_MEM_BACKGROUND);
+ }
+ spin_unlock_irq(&cache->pool_lock);
+ if (page)
+ return page;
+ r5c_wake_wait_reclaimer(cache, RECLAIM_MEM);
+ goto again;
+}
+
+static void r5c_put_pages(struct r5c_cache *cache,
+ struct page *pages[], int size)
+{
+ unsigned long flags;
+ int i;
+
+ spin_lock_irqsave(&cache->pool_lock, flags);
+ for (i = 0; i < size; i++) {
+ if (pages[i])
+ list_add(&pages[i]->lru, &cache->page_pool);
+ }
+ cache->free_pages += size;
+ if (cache->free_pages >= cache->high_watermark)
+ clear_bit(RECLAIM_MEM_BACKGROUND, &cache->reclaim_reason);
+ clear_bit(RECLAIM_MEM, &cache->reclaim_reason);
+ spin_unlock_irqrestore(&cache->pool_lock, flags);
+
+ wake_up_all(&cache->reclaim_wait);
+}
+
+static struct r5c_stripe *
+r5c_search_stripe(struct r5c_cache *cache, u64 stripe_index)
+{
+ struct r5c_stripe *stripe;
+
+ spin_lock_irq(&cache->tree_lock);
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ spin_unlock_irq(&cache->tree_lock);
+ return stripe;
+}
+
+static void r5c_put_stripe(struct r5c_stripe *stripe);
+static struct r5c_stripe *
+r5c_get_stripe(struct r5c_cache *cache, u64 stripe_index)
+{
+ struct r5c_stripe *stripe;
+
+again:
+ spin_lock_irq(&cache->tree_lock);
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ if (stripe)
+ atomic_inc(&stripe->ref);
+ spin_unlock_irq(&cache->tree_lock);
+
+ if (!stripe)
+ return NULL;
+ /* The stripe is being reclaimed, wait reclaim finish */
+ if (stripe->state >= STRIPE_FROZEN) {
+ r5c_stripe_wait_state(cache, stripe, STRIPE_DEAD);
+ r5c_put_stripe(stripe);
+ goto again;
+ }
+ return stripe;
+}
+
+static struct r5c_stripe *
+r5c_create_get_stripe(struct r5c_cache *cache, u64 stripe_index)
+{
+ struct r5c_stripe *stripe, *new;
+ int error;
+
+again:
+ spin_lock_irq(&cache->tree_lock);
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ if (stripe)
+ atomic_inc(&stripe->ref);
+ spin_unlock_irq(&cache->tree_lock);
+
+ if (stripe)
+ goto has_stripe;
+
+ new = kzalloc(sizeof(struct r5c_stripe) + sizeof(struct page *) *
+ STRIPE_DATA_PAGES(cache), GFP_NOIO);
+ new->raid_index = stripe_index;
+ atomic_set(&new->ref, 2);
+ new->state = STRIPE_RUNNING;
+ new->cache = cache;
+ INIT_LIST_HEAD(&new->io_ranges);
+ INIT_LIST_HEAD(&new->stripes);
+ INIT_LIST_HEAD(&new->lru);
+
+ error = radix_tree_preload(GFP_NOIO);
+ if (error) {
+ kfree(new);
+ return NULL;
+ }
+
+ spin_lock_irq(&cache->tree_lock);
+ if (radix_tree_insert(&cache->stripe_tree, stripe_index, new)) {
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ atomic_inc(&stripe->ref);
+ kfree(new);
+ } else
+ stripe = new;
+ spin_unlock_irq(&cache->tree_lock);
+
+ radix_tree_preload_end();
+
+has_stripe:
+ /* The stripe is being reclaimed, wait reclaim finish */
+ if (stripe->state >= STRIPE_FROZEN) {
+ r5c_stripe_wait_state(cache, stripe, STRIPE_DEAD);
+ r5c_put_stripe(stripe);
+ goto again;
+ }
+
+ return stripe;
+}
+
+static void r5c_put_stripe(struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+ int ref = atomic_dec_return(&stripe->ref);
+ unsigned long flags, flags2;
+
+ /* the stripe is freezing, might wait for ref */
+ if (ref == 1)
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+
+ if (ref)
+ return;
+
+ spin_lock_irqsave(&cache->tree_lock, flags);
+ radix_tree_delete(&cache->stripe_tree, stripe->raid_index);
+
+ r5c_lock_stripe(cache, stripe, &flags2);
+ while (!list_empty(&stripe->io_ranges)) {
+ range = list_first_entry(&stripe->io_ranges,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ list_del(&range->log_sibling);
+
+ kfree(range);
+ }
+ r5c_put_pages(cache, stripe->data_pages, STRIPE_DATA_PAGES(cache));
+ BUG_ON(stripe->parity_pages);
+
+ r5c_unlock_stripe(cache, stripe, &flags2);
+
+ spin_unlock_irqrestore(&cache->tree_lock, flags);
+
+ kfree(stripe);
+}
+
+/* must hold cache->tree_lock */
+static bool r5c_freeze_stripe(struct r5c_cache *cache,
+ struct r5c_stripe *stripe, bool blocking)
+{
+ if (atomic_read(&stripe->ref) == 1) {
+ stripe->state = STRIPE_FROZEN;
+ return true;
+ }
+ if (!blocking)
+ return false;
+ /* Make sure no IO running in stripe */
+ wait_event_lock_irq(*r5c_stripe_waitq(cache, stripe),
+ atomic_read(&stripe->ref) == 1, cache->tree_lock);
+ stripe->state = STRIPE_FROZEN;
+ return true;
+}
+
+static void r5c_bio_task_end(struct r5l_task *task)
+{
+ struct r5c_io_range *range = task->private;
+ struct r5c_stripe *stripe = range->stripe;
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *tmp;
+ unsigned long flags, flags2;
+
+ range->seq = task->seq;
+ range->meta_start = task->meta_start;
+ range->data_start = task->data_start;
+ kfree(task);
+
+ spin_lock_irqsave(&cache->tree_lock, flags);
+
+ if (list_empty(&cache->log_list)) {
+ list_add_tail(&range->log_sibling, &cache->log_list);
+ goto out;
+ }
+ /* generally tmp will be the last one */
+ list_for_each_entry_reverse(tmp, &cache->log_list, log_sibling) {
+ /* later range has bigger seq and meta_start than previous range */
+ if (range->seq >= tmp->seq)
+ break;
+ }
+ list_add_tail(&range->log_sibling, &tmp->log_sibling);
+out:
+ r5c_lock_stripe(cache, stripe, &flags2);
+ if (stripe->existing_pages == STRIPE_DATA_PAGES(cache) &&
+ list_empty(&stripe->lru)) {
+ list_add_tail(&stripe->lru, &cache->full_stripes);
+ if (!cache->full_stripe_ready_time)
+ cache->full_stripe_ready_time = jiffies;
+ }
+ r5c_unlock_stripe(cache, stripe, &flags2);
+
+ spin_unlock_irqrestore(&cache->tree_lock, flags);
+
+ r5c_put_stripe(stripe);
+
+ bio_endio_nodec(range->bio, 0);
+}
+
+static void r5c_copy_bio(struct bio *bio, struct page *pages[], bool tobio)
+{
+ struct bio *src = bio;
+ struct bvec_iter iter;
+ struct bio_vec bv;
+ void *src_p, *dst_p;
+ unsigned page_index = 0, page_offset = 0;
+ unsigned bytes;
+
+ iter = src->bi_iter;
+
+ while (1) {
+ if (!iter.bi_size) {
+ src = src->bi_next;
+ if (!src)
+ break;
+
+ iter = src->bi_iter;
+ }
+
+ if (page_offset == PAGE_SIZE) {
+ page_index++;
+ page_offset = 0;
+ }
+
+ bv = bio_iter_iovec(src, iter);
+
+ bytes = min_t(unsigned int, bv.bv_len, PAGE_SIZE - page_offset);
+
+ src_p = kmap_atomic(bv.bv_page);
+ dst_p = kmap_atomic(pages[page_index]);
+
+ if (tobio) {
+ memcpy(src_p + bv.bv_offset,
+ dst_p + page_offset, bytes);
+ } else {
+ memcpy(dst_p + page_offset,
+ src_p + bv.bv_offset, bytes);
+ }
+
+ kunmap_atomic(dst_p);
+ kunmap_atomic(src_p);
+
+ bio_advance_iter(src, &iter, bytes);
+ page_offset += bytes;
+ }
+}
+
+static void r5c_write_bio(struct r5c_cache *cache, struct bio *bio)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *io_range;
+ unsigned long index;
+ unsigned int offset;
+ int i, new_pages = 0;
+ unsigned long flags;
+
+ /* Doesn't support discard */
+ if (bio->bi_rw & REQ_DISCARD) {
+ bio_endio(bio, 0);
+ return;
+ }
+ if (bio->bi_iter.bi_size == 0) {
+ BUG_ON(!(bio->bi_rw & REQ_FLUSH));
+ r5l_queue_empty_flush_bio(&cache->log, bio);
+ return;
+ }
+
+ STRIPE_INDEX_OFFSET(cache, bio->bi_iter.bi_sector, index, offset);
+
+ stripe = r5c_create_get_stripe(cache, index);
+
+ io_range = kmalloc(sizeof(struct r5c_io_range), GFP_NOIO);
+ io_range->bio = bio;
+ io_range->raid_start = bio->bi_iter.bi_sector;
+ io_range->data_sectors = bio_sectors(bio);
+ io_range->stripe = stripe;
+
+ /* FIXME: read can get garbage data here */
+ offset >>= PAGE_SECTOR_SHIFT;
+ for (i = offset; i < offset + (bio_sectors(bio) >> PAGE_SECTOR_SHIFT);
+ i++) {
+ if (stripe->data_pages[i])
+ continue;
+ stripe->data_pages[i] = r5c_get_page(cache);
+ new_pages++;
+ }
+
+ r5c_lock_stripe(cache, stripe, &flags);
+ list_add_tail(&io_range->stripe_sibling, &stripe->io_ranges);
+ stripe->existing_pages += new_pages;
+ r5c_unlock_stripe(cache, stripe, &flags);
+
+ r5c_copy_bio(bio, &stripe->data_pages[offset], false);
+
+ r5l_queue_bio(&cache->log, bio, r5c_bio_task_end, io_range);
+}
+
+static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
+{
+ struct r5c_stripe *stripe;
+ u64 stripe_index;
+ int offset;
+ u64 start, end, tmp;
+ struct bio *split;
+
+ STRIPE_INDEX_OFFSET(cache, bio->bi_iter.bi_sector, stripe_index, offset);
+
+ stripe = r5c_get_stripe(cache, stripe_index);
+ if (!stripe) {
+ raid5_make_request(cache->mddev, bio);
+ return;
+ }
+
+ start = offset >> PAGE_SECTOR_SHIFT;
+ end = start + (bio_sectors(bio) >> PAGE_SECTOR_SHIFT);
+
+ while (start < end) {
+ if (stripe->data_pages[start]) {
+ tmp = start;
+ while (tmp < end && stripe->data_pages[tmp])
+ tmp++;
+ if (tmp < end) {
+ split = bio_split(bio,
+ (tmp - start) << PAGE_SECTOR_SHIFT,
+ GFP_NOIO, NULL);
+ bio_chain(split, bio);
+ } else /* all in cache */
+ split = bio;
+
+ r5c_copy_bio(split, &stripe->data_pages[start], true);
+
+ bio_endio(split, 0);
+
+ start = tmp;
+ } else {
+ tmp = start;
+ while (tmp < end && !stripe->data_pages[tmp])
+ tmp++;
+ if (tmp < end) {
+ split = bio_split(bio,
+ (tmp - start) << PAGE_SECTOR_SHIFT,
+ GFP_NOIO, NULL);
+ bio_chain(split, bio);
+ } else
+ split = bio;
+
+ raid5_make_request(cache->mddev, split);
+
+ start = tmp;
+ }
+ }
+ r5c_put_stripe(stripe);
+}
+
+void r5c_handle_bio(struct r5c_cache *cache, struct bio *bio)
+{
+ if (bio_data_dir(bio) == READ)
+ r5c_read_bio(cache, bio);
+ else
+ r5c_write_bio(cache, bio);
+}
+
+static void r5c_flush_endio(struct bio *bio, int err)
+{
+ struct r5c_stripe *stripe = bio->bi_private;
+ struct r5c_cache *cache = stripe->cache;
+
+ if (atomic_dec_and_test(&stripe->pending_bios)) {
+ stripe->state = STRIPE_INRAID;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+ }
+
+ bio_put(bio);
+}
+
+static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ struct block_device *bdev, int start, int pages)
+{
+ sector_t base;
+ struct bio *bio;
+ int i;
+
+ i = 0;
+ base = STRIPE_RAID_SECTOR(cache, stripe) + (start << PAGE_SECTOR_SHIFT);
+ while (i < pages) {
+ bio = bio_kmalloc(GFP_NOIO, min_t(int, pages - i,
+ bio_get_nr_vecs(bdev)));
+ bio->bi_iter.bi_sector = base + (i << PAGE_SECTOR_SHIFT);
+ bio->bi_bdev = bdev;
+ bio->bi_private = stripe;
+ while (i < pages) {
+ if (!bio_add_page(bio, stripe->data_pages[i + start],
+ PAGE_SIZE, 0))
+ break;
+ i++;
+ }
+ bio->bi_end_io = r5c_flush_endio;
+ bio->bi_rw = WRITE;
+ atomic_inc(&stripe->pending_bios);
+ raid5_make_request(cache->mddev, bio);
+ }
+}
+
+static void r5c_put_stripe_dirty(struct r5c_cache *cache, struct r5c_stripe *stripe)
+{
+ if (!atomic_dec_return(&stripe->dirty_stripes)) {
+ stripe->state = STRIPE_PARITY_DONE;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+ }
+}
+
+static void r5c_flush_stripe(struct r5c_cache *cache, struct block_device *bdev,
+ struct r5c_stripe *stripe)
+{
+ unsigned long *stripe_bits;
+ int chunk_stripes;
+ int start;
+ int end = 0;
+ int i;
+
+ chunk_stripes = cache->chunk_size >> PAGE_SECTOR_SHIFT;
+ stripe_bits = kzalloc(BITS_TO_LONGS(chunk_stripes) * sizeof(long),
+ GFP_NOIO);
+ for (i = 0; i < STRIPE_DATA_PAGES(cache); i++) {
+ if (stripe->data_pages[i])
+ __set_bit(i % chunk_stripes, stripe_bits);
+ }
+ atomic_set(&stripe->dirty_stripes, bitmap_weight(stripe_bits,
+ chunk_stripes) + 1);
+ kfree(stripe_bits);
+
+ while (end < STRIPE_DATA_PAGES(cache)) {
+ while (end < STRIPE_DATA_PAGES(cache) && !stripe->data_pages[end])
+ end++;
+ if (end >= STRIPE_DATA_PAGES(cache))
+ break;
+ start = end;
+ while (end < STRIPE_DATA_PAGES(cache) && stripe->data_pages[end])
+ end++;
+ r5c_flush_one(cache, stripe, bdev, start, end - start);
+ }
+ r5c_put_stripe_dirty(cache, stripe);
+}
+
+static int r5c_select_front_stripes(struct r5c_cache *cache,
+ struct list_head *list, int count, bool blocking)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ int stripes = 0;
+
+ list_for_each_entry(range, &cache->log_list, log_sibling) {
+ stripe = range->stripe;
+ if (stripe->state >= STRIPE_FROZEN)
+ continue;
+
+ if (!r5c_freeze_stripe(cache, stripe, blocking))
+ continue;
+
+ list_move_tail(&stripe->lru, list);
+ stripes++;
+ if (stripes >= count)
+ break;
+ }
+ return stripes;
+}
+
+static int r5c_select_full_stripes(struct r5c_cache *cache,
+ struct list_head *list, int count, bool blocking)
+{
+ struct r5c_stripe *stripe, *tmp;
+ int stripes = 0;
+
+ list_for_each_entry_safe(stripe, tmp, &cache->full_stripes, lru) {
+ if (stripe->state >= STRIPE_FROZEN)
+ continue;
+ if (!r5c_freeze_stripe(cache, stripe, blocking))
+ continue;
+
+ list_move_tail(&stripe->lru, list);
+ stripes++;
+ if (stripes >= count)
+ break;
+ }
+ if (list_empty(&cache->full_stripes)) {
+ cache->full_stripe_ready_time = 0;
+ clear_bit(RECLAIM_MEM_FULL, &cache->reclaim_reason);
+ }
+ return stripes;
+}
+
+static void r5c_select_stripes(struct r5c_cache *cache, struct list_head *list)
+{
+ int stripes;
+ bool blocking;
+
+ /*
+ * generally select full stripe, if no disk space, select first stripe
+ */
+ spin_lock_irq(&cache->tree_lock);
+ /* Don't need stripe lock, as nobody is operating on the stripe */
+ if (test_bit(RECLAIM_DISK, &cache->reclaim_reason) ||
+ test_bit(RECLAIM_DISK_BACKGROUND, &cache->reclaim_reason)) {
+ blocking = test_bit(RECLAIM_DISK, &cache->reclaim_reason);
+ r5c_select_front_stripes(cache, list, RECLAIM_BATCH, blocking);
+ } else if (test_bit(RECLAIM_MEM, &cache->reclaim_reason) ||
+ test_bit(RECLAIM_MEM_BACKGROUND, &cache->reclaim_reason)) {
+ stripes = r5c_select_full_stripes(cache, list, RECLAIM_BATCH, false);
+ if (stripes < RECLAIM_BATCH)
+ r5c_select_front_stripes(cache, list, RECLAIM_BATCH, false);
+ } else if (test_bit(RECLAIM_MEM_FULL, &cache->reclaim_reason)){
+ r5c_select_full_stripes(cache, list, -1, false);
+ }
+
+ spin_unlock_irq(&cache->tree_lock);
+}
+
+static void r5c_disks_flush_end(struct bio *bio, int err)
+{
+ struct completion *io_complete = bio->bi_private;
+
+ complete(io_complete);
+ bio_put(bio);
+}
+
+static void r5c_flush_all_disks(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ struct bio *bi;
+ DECLARE_COMPLETION_ONSTACK(io_complete);
+
+ bi = bio_alloc_mddev(GFP_NOIO, 0, mddev);
+ bi->bi_end_io = r5c_disks_flush_end;
+ bi->bi_private = &io_complete;
+
+ /* If bio hasn't payload, this function will just flush all disks */
+ md_flush_request(mddev, bi);
+
+ wait_for_completion_io(&io_complete);
+}
+
+static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
+ struct list_head *stripe_list)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ u64 seq;
+ sector_t meta;
+ struct blk_plug plug;
+ struct block_device *md_bdev;
+
+ if (list_empty(stripe_list))
+ return;
+ md_bdev = blkdev_get_by_dev(cache->mddev->unit,
+ FMODE_READ|FMODE_WRITE, NULL);
+
+ blk_start_plug(&plug);
+ /* step 1: start write to raid */
+ list_for_each_entry(stripe, stripe_list, lru)
+ r5c_flush_stripe(cache, md_bdev, stripe);
+ blk_finish_plug(&plug);
+
+ /* step 2: wait parity write to cache */
+ list_for_each_entry_reverse(stripe, stripe_list, lru)
+ r5c_stripe_wait_state(cache, stripe, STRIPE_PARITY_DONE);
+
+ /* step 3: make sure data and parity settle down */
+ r5l_commit(&cache->log);
+
+ /* step 4: continue write to raid */
+ list_for_each_entry(stripe, stripe_list, lru) {
+ atomic_set(&stripe->dirty_stripes, 1);
+ while (!list_empty(&stripe->stripes)) {
+ struct stripe_head *sh;
+
+ sh = list_first_entry(&stripe->stripes,
+ struct stripe_head, stripe_list);
+ list_del(&sh->stripe_list);
+ set_bit(STRIPE_HANDLE, &sh->state);
+ if (!list_empty(&stripe->stripes))
+ atomic_inc(&stripe->dirty_stripes);
+ release_stripe(sh);
+ }
+ }
+
+ /* step 5: wait to make sure stripe data is in raid */
+ list_for_each_entry_reverse(stripe, stripe_list, lru)
+ r5c_stripe_wait_state(cache, stripe, STRIPE_INRAID);
+
+ blkdev_put(md_bdev, FMODE_READ|FMODE_WRITE);
+
+ /* step 6: flush raid disks */
+ r5c_flush_all_disks(cache);
+
+ /* step 7: add a checkpoint */
+ r5l_checkpoint(&cache->log, &seq, &meta);
+
+ /* step 8: mark stripe as dead */
+ while (!list_empty(stripe_list)) {
+ stripe = list_first_entry(stripe_list, struct r5c_stripe,
+ lru);
+ list_del_init(&stripe->lru);
+
+ stripe->state = STRIPE_DEAD;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+
+ r5c_put_stripe(stripe);
+ }
+ /* step 9: advance superblock checkpoint */
+ spin_lock_irq(&cache->tree_lock);
+ /* if no data, superblock records the next position of checkpoint */
+ if (!list_empty(&cache->log_list)) {
+ range = list_first_entry(&cache->log_list,
+ struct r5c_io_range, log_sibling);
+ /* can't cross checkpoint */
+ if (range->seq < seq) {
+ seq = range->seq;
+ meta = range->meta_start;
+ }
+ }
+ spin_unlock_irq(&cache->tree_lock);
+
+ r5l_write_super(&cache->log, seq, meta);
+}
+
+static void r5c_reclaim_thread(struct md_thread *thread)
+{
+ struct mddev *mddev = thread->mddev;
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ LIST_HEAD(stripe_list);
+
+ if (cache->full_stripe_ready_time && cache->full_stripe_ready_time +
+ FULL_STRIPE_MAX_AGE < jiffies)
+ set_bit(RECLAIM_MEM_FULL, &cache->reclaim_reason);
+ while (cache->reclaim_reason != 0) {
+ r5c_select_stripes(cache, &stripe_list);
+
+ r5c_reclaim_stripe_list(cache, &stripe_list);
+
+ wake_up_all(&cache->reclaim_wait);
+ }
+}
+
+static void r5c_wake_reclaimer(struct r5c_cache *cache, int reason)
+{
+ set_bit(reason, &cache->reclaim_reason);
+ md_wakeup_thread(cache->reclaim_thread);
+}
+
+static void r5c_wake_wait_reclaimer(struct r5c_cache *cache, int reason)
+{
+ r5c_wake_reclaimer(cache, reason);
+ wait_event(cache->reclaim_wait, !test_bit(reason,
+ &cache->reclaim_reason));
+}
+
+static void r5c_parity_task_end(struct r5l_task *task)
+{
+ struct stripe_head *sh = task->private;
+ struct r5conf *conf = sh->raid_conf;
+ struct r5c_cache *cache = conf->cache;
+ struct r5c_stripe *stripe = sh->stripe;
+
+ r5c_put_stripe_dirty(cache, stripe);
+ kfree(task);
+}
+
+/*
+ * we don't record parity range in cache->log_list, because after a success
+ * reclaim, parity always is discarded
+ */
+int r5c_write_parity(struct r5c_cache *cache, struct stripe_head *sh)
+{
+ struct r5c_stripe *stripe;
+ u64 stripe_index;
+ int stripe_offset;
+ int parity_cnt;
+ unsigned long flags;
+
+ /* parity is already written */
+ if (sh->stripe) {
+ sh->stripe = NULL;
+ return -EAGAIN;
+ }
+ if (!test_bit(R5_Wantwrite, &sh->dev[sh->pd_idx].flags))
+ return -EAGAIN;
+
+ stripe_index = sh->sector;
+ stripe_offset = sector_div(stripe_index, cache->chunk_size);
+ stripe_offset >>= PAGE_SECTOR_SHIFT;
+
+ stripe = r5c_search_stripe(cache, stripe_index);
+ r5c_lock_stripe(cache, stripe, &flags);
+ list_add_tail(&sh->stripe_list, &stripe->stripes);
+ r5c_unlock_stripe(cache, stripe, &flags);
+ sh->stripe = stripe;
+
+ atomic_inc(&sh->count);
+
+ parity_cnt = !!(sh->pd_idx >= 0) + !!(sh->qd_idx >= 0);
+ BUG_ON(parity_cnt != cache->parity_disks);
+
+ if (cache->in_recovery) {
+ stripe_offset *= cache->parity_disks;
+
+ copy_highpage(sh->dev[sh->pd_idx].page,
+ stripe->parity_pages[stripe_offset]);
+ if (sh->qd_idx >= 0)
+ copy_highpage(sh->dev[sh->qd_idx].page,
+ stripe->parity_pages[stripe_offset + 1]);
+
+ if (!atomic_dec_return(&stripe->dirty_stripes)) {
+ stripe->state = STRIPE_PARITY_DONE;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+ }
+ return 0;
+ }
+
+ r5l_queue_parity(&cache->log, sh->sector,
+ sh->dev[sh->pd_idx].page,
+ sh->qd_idx >= 0 ? sh->dev[sh->qd_idx].page : NULL,
+ r5c_parity_task_end, sh);
+ return 0;
+}
+
+struct r5c_load_ctx {
+ struct r5c_cache *cache;
+ struct list_head io_ranges;
+ struct list_head stripes_with_parity;
+ struct list_head stripes_without_parity;
+ unsigned long *stripe_bits;
+ int chunk_stripes;
+ atomic_t io_cnt;
+ wait_queue_head_t io_wq;
+};
+
+static int r5c_add_checkpoint(struct r5c_load_ctx *ctx, u64 seq)
+{
+ struct r5c_cache *cache = ctx->cache;
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+
+ while (!list_empty(&ctx->stripes_with_parity)) {
+ stripe = list_first_entry(&ctx->stripes_with_parity,
+ struct r5c_stripe, lru);
+ list_del(&stripe->lru);
+ /* essentially r5c_put_stripe */
+ radix_tree_delete(&cache->stripe_tree, stripe->raid_index);
+
+ while (!list_empty(&stripe->io_ranges)) {
+ range = list_first_entry(&stripe->io_ranges,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ list_del(&range->log_sibling);
+
+ kfree(range->checksum);
+ kfree(range);
+ }
+ while (!list_empty(&stripe->parity_list)) {
+ range = list_first_entry(&stripe->parity_list,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ kfree(range->checksum);
+ kfree(range);
+ }
+
+ kfree(stripe);
+ }
+ return 0;
+}
+
+static int r5c_add_data(struct r5c_load_ctx *ctx, u64 seq,
+ sector_t meta_start, sector_t data_start, sector_t raid_start,
+ ssize_t sectors, u32 *checksum)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ u64 index;
+ int offset;
+
+ STRIPE_INDEX_OFFSET(ctx->cache, raid_start, index, offset);
+
+ stripe = r5c_create_get_stripe(ctx->cache, index);
+
+ range = kmalloc(sizeof(struct r5c_io_range), GFP_NOIO);
+
+ range->seq = seq;
+ range->meta_start = meta_start;
+ range->data_start = data_start;
+ range->raid_start = raid_start;
+ range->data_sectors = sectors;
+ range->stripe = stripe;
+ range->checksum = checksum;
+
+ list_add_tail(&range->stripe_sibling, &stripe->io_ranges);
+ list_add_tail(&range->log_sibling, &ctx->io_ranges);
+
+ if (list_empty(&stripe->lru))
+ list_add_tail(&stripe->lru, &ctx->stripes_without_parity);
+
+ r5c_put_stripe(stripe);
+ return 0;
+}
+
+static int r5c_add_parity(struct r5c_load_ctx *ctx, u64 seq,
+ sector_t meta_start, sector_t data_start, sector_t stripe_sect,
+ ssize_t sectors, u32 *checksum)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ sector_t index = stripe_sect;
+
+ sector_div(index, ctx->cache->log.chunk_size);
+
+ stripe = r5c_get_stripe(ctx->cache, index);
+ BUG_ON(!stripe);
+
+ range = kmalloc(sizeof(struct r5c_io_range), GFP_NOIO);
+
+ range->seq = seq;
+ range->meta_start = meta_start;
+ range->data_start = data_start;
+ range->raid_start = stripe_sect;
+ range->data_sectors = sectors;
+ range->stripe = stripe;
+ range->checksum = checksum;
+
+ if (list_empty(&stripe->parity_list))
+ list_move_tail(&stripe->lru, &ctx->stripes_with_parity);
+ list_add_tail(&range->stripe_sibling, &stripe->parity_list);
+
+ r5c_put_stripe(stripe);
+ return 0;
+}
+
+static bool r5c_check_parity_full(struct r5c_load_ctx *ctx,
+ struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ int chunk_stripes = ctx->chunk_stripes;
+ unsigned long *stripe_bits = ctx->stripe_bits;
+ struct r5c_io_range *range;
+ sector_t index;
+ int offset, max;
+
+ memset(stripe_bits, 0, BITS_TO_LONGS(chunk_stripes) * sizeof(long));
+ list_for_each_entry(range, &stripe->io_ranges, stripe_sibling) {
+ STRIPE_INDEX_OFFSET(cache, range->raid_start, index, offset);
+ offset >>= PAGE_SECTOR_SHIFT;
+ max = (range->data_sectors >> PAGE_SECTOR_SHIFT) + offset;
+ for (; offset < max; offset++)
+ __set_bit(offset % chunk_stripes, stripe_bits);
+ }
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+
+ if ((range->data_sectors >> PAGE_SECTOR_SHIFT) ==
+ cache->parity_disks)
+ __clear_bit(offset, stripe_bits);
+ }
+
+ return bitmap_weight(stripe_bits, chunk_stripes) == 0;
+}
+
+static void r5c_free_parity_ranges(struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+
+ while (!list_empty(&stripe->parity_list)) {
+ range = list_first_entry(&stripe->parity_list,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ kfree(range->checksum);
+ kfree(range);
+ }
+ if (stripe->parity_pages) {
+ r5c_put_pages(cache, stripe->parity_pages,
+ STRIPE_PARITY_PAGES(cache));
+ kfree(stripe->parity_pages);
+ stripe->parity_pages = NULL;
+ }
+}
+
+/* If there is commit block after parity, all stripes with parity are already in
+ * raid. All data and parity can be ignored. Otherwise, if all stripes with
+ * parity have full parity, those stripes are in the process writting to raid,
+ * such stripes must be recovered. Otherwise, ignore all parity
+ */
+static int r5c_analyse_log(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe;
+ bool full = true;
+
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ if (r5c_check_parity_full(ctx, stripe))
+ continue;
+ full = false;
+ break;
+ }
+ if (!full) {
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ r5c_free_parity_ranges(stripe);
+ }
+ list_splice_tail_init(&ctx->stripes_with_parity,
+ &ctx->stripes_without_parity);
+ }
+ return 0;
+}
+
+static void r5l_fetch_endio(struct bio *bio, int err)
+{
+ struct r5c_load_ctx *ctx = bio->bi_private;
+
+ bio_put(bio);
+ if (atomic_dec_and_test(&ctx->io_cnt))
+ wake_up(&ctx->io_wq);
+}
+
+static int r5l_fetch_one_range(struct r5c_load_ctx *ctx,
+ struct r5c_stripe *stripe, struct page **pages,
+ ssize_t page_cnt, sector_t sec, bool data)
+{
+ struct r5l_log *log = &ctx->cache->log;
+ struct bio *bio, *split;
+ int page_index = 0;
+ int nvec;
+ int i;
+
+again:
+ nvec = min_t(int, page_cnt - page_index, bio_get_nr_vecs(log->bdev));
+ bio = bio_kmalloc(GFP_NOIO, nvec);
+ bio->bi_iter.bi_sector = sec;
+ bio->bi_end_io = r5l_fetch_endio;
+ bio->bi_private = ctx;
+ bio->bi_bdev = log->bdev;
+ atomic_inc(&ctx->io_cnt);
+
+ /* FIXME: we might use too many memory */
+ for (i = 0; i < nvec; i++) {
+ pages[i + page_index] = alloc_page(GFP_NOIO);
+ if (data)
+ stripe->existing_pages++;
+ ctx->cache->total_pages++;
+ bio_add_page(bio, pages[i + page_index], PAGE_SIZE, 0);
+ }
+
+ if (bio_end_sector(bio) > (BLOCK_SECTOR(log, log->last_block))) {
+ split = bio_split(bio, BLOCK_SECTOR(log, log->last_block) -
+ bio->bi_iter.bi_sector, GFP_NOIO, NULL);
+ bio_chain(split, bio);
+ bio->bi_iter.bi_sector = BLOCK_SECTOR(log, log->first_block);
+ r5l_submit_bio(log, READ, split);
+ }
+ sec = BLOCK_SECTOR(log, r5l_ring_add(log, sec >> log->block_sector_shift,
+ PAGE_BLOCKS(log, nvec)));
+ page_index += nvec;
+ r5l_submit_bio(log, READ, bio);
+
+ if (page_index < page_cnt)
+ goto again;
+ return 0;
+}
+
+static int r5l_fetch_one_stripe(struct r5c_load_ctx *ctx, struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+ u64 index;
+ int offset, max, start;
+
+ /* new data is at the tail */
+ list_for_each_entry_reverse(range, &stripe->io_ranges, stripe_sibling) {
+ STRIPE_INDEX_OFFSET(cache, range->raid_start, index, offset);
+ offset >>= PAGE_SECTOR_SHIFT;
+ max = (range->data_sectors >> PAGE_SECTOR_SHIFT) + offset;
+again:
+ while (offset < max && stripe->data_pages[offset])
+ offset++;
+ start = offset;
+ while (offset < max && !stripe->data_pages[offset])
+ offset++;
+ r5l_fetch_one_range(ctx, stripe, &stripe->data_pages[start],
+ offset - start,
+ range->data_start + (start << PAGE_SECTOR_SHIFT), true);
+ if (offset < max)
+ goto again;
+ }
+
+ if (list_empty(&stripe->parity_list))
+ return 0;
+
+ stripe->parity_pages = kmalloc(STRIPE_PARITY_PAGES(cache) *
+ sizeof(struct page *), GFP_NOIO | __GFP_ZERO);
+
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+ offset *= cache->parity_disks;
+
+ r5l_fetch_one_range(ctx, stripe, &stripe->parity_pages[offset],
+ cache->parity_disks, range->data_start, false);
+ }
+ return 0;
+}
+
+static int r5l_fetch_stripes(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe;
+ struct blk_plug plug;
+
+ blk_start_plug(&plug);
+ list_for_each_entry(stripe, &ctx->stripes_without_parity, lru) {
+ r5l_fetch_one_stripe(ctx, stripe);
+ }
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ r5l_fetch_one_stripe(ctx, stripe);
+ }
+ blk_finish_plug(&plug);
+
+ atomic_dec(&ctx->io_cnt);
+ wait_event(ctx->io_wq, atomic_read(&ctx->io_cnt) == 0);
+
+ return 0;
+}
+
+static u32 r5l_calculate_page_checksum(struct r5l_log *log, struct page *page)
+{
+ u32 csum;
+ void *addr = kmap_atomic(page);
+ csum = r5l_calculate_checksum(log, UUID_CHECKSUM(log, true),
+ addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+ return csum;
+}
+
+static int r5l_check_one_stripe_checksum(struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5l_log *log = &cache->log;
+ struct r5c_io_range *range;
+ struct page *page;
+ u64 index;
+ int offset;
+ int ret = 0;
+
+ /* FIXME: we currently ignore all data parity check */
+ list_for_each_entry(range, &stripe->io_ranges, stripe_sibling) {
+ kfree(range->checksum);
+ range->checksum = NULL;
+ }
+
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+ offset *= cache->parity_disks;
+
+ page = stripe->parity_pages[offset];
+
+ if (le32_to_cpu(range->checksum[0]) !=
+ r5l_calculate_page_checksum(log, page)) {
+ ret = -EINVAL;
+ break;
+ }
+ if (cache->parity_disks > 1) {
+ page = stripe->parity_pages[offset + 1];
+ if (le32_to_cpu(range->checksum[1]) !=
+ r5l_calculate_page_checksum(log, page)) {
+ ret = -EINVAL;
+ break;
+ }
+ }
+ }
+ r5c_free_parity_ranges(stripe);
+ return ret;
+}
+
+static int r5l_check_stripes_checksum(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe;
+ bool skip = false;
+
+ while (!list_empty(&ctx->stripes_without_parity)) {
+ stripe = list_first_entry(&ctx->stripes_without_parity,
+ struct r5c_stripe, lru);
+ list_del_init(&stripe->lru);
+
+ r5l_check_one_stripe_checksum(stripe);
+ }
+
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ if (skip)
+ r5c_free_parity_ranges(stripe);
+ else
+ skip = !!r5l_check_one_stripe_checksum(stripe);
+ }
+
+ /* If any parity checksum is wrong, we think the stripes are not hiting to raid */
+ if (skip) {
+ while (!list_empty(&ctx->stripes_with_parity)) {
+ stripe = list_first_entry(&ctx->stripes_with_parity,
+ struct r5c_stripe, lru);
+ list_del_init(&stripe->lru);
+ }
+ }
+ return 0;
+}
+
+static int r5c_recover_stripes(struct r5c_load_ctx *ctx)
+{
+ struct r5c_cache *cache = ctx->cache;
+
+ r5l_check_stripes_checksum(ctx);
+
+ list_splice_tail(&ctx->io_ranges, &ctx->cache->log_list);
+
+ if (list_empty(&ctx->stripes_with_parity))
+ return 0;
+
+ cache->in_recovery = 1;
+
+ r5c_reclaim_stripe_list(cache, &ctx->stripes_with_parity);
+
+ cache->in_recovery = 0;
+ return 0;
+}
+
+static void *r5l_read_meta_block(struct r5l_log *log, u64 block,
+ u64 expected_seq, struct page **retpage)
+{
+ struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
+ struct r5l_meta_header *header;
+ u32 crc, stored_crc;
+
+ if (!sync_page_io(log->rdev, BLOCK_SECTOR(log, block),
+ log->block_size, page, READ, false))
+ return NULL;
+
+ header = kmap(page);
+ stored_crc = le32_to_cpu(header->checksum);
+ header->checksum = 0;
+
+ if (le32_to_cpu(header->magic) != R5LOG_MAGIC ||
+ le64_to_cpu(header->seq) != expected_seq ||
+ le64_to_cpu(header->position) != block)
+ goto error;
+ if (le32_to_cpu(header->type) != R5LOG_TYPE_META &&
+ le32_to_cpu(header->type) != R5LOG_TYPE_CHECKPOINT)
+ goto error;
+
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false),
+ header, log->block_size, false);
+ if (stored_crc != crc)
+ goto error;
+
+ if (le32_to_cpu(header->meta_size) > log->block_size)
+ goto error;
+ *retpage = page;
+ return header;
+error:
+ kunmap(page);
+ __free_page(page);
+ return NULL;
+}
+
+static int r5l_load_log(struct r5l_log *log)
+{
+ u64 last_cp = log->last_checkpoint;
+ u64 last_seq = log->last_cp_seq;
+ u64 payload_block;
+ struct page *page = NULL;
+ struct r5l_meta_header *header;
+ void *meta;
+ struct r5l_meta_payload *payload;
+ struct r5c_load_ctx ctx;
+ int offset;
+
+ ctx.cache = log->cache;
+ INIT_LIST_HEAD(&ctx.io_ranges);
+ INIT_LIST_HEAD(&ctx.stripes_with_parity);
+ INIT_LIST_HEAD(&ctx.stripes_without_parity);
+ ctx.chunk_stripes = log->chunk_size >> PAGE_SECTOR_SHIFT;
+ ctx.stripe_bits = kmalloc(BITS_TO_LONGS(ctx.chunk_stripes) * sizeof(long),
+ GFP_KERNEL);
+ atomic_set(&ctx.io_cnt, 1);
+ init_waitqueue_head(&ctx.io_wq);
+
+again:
+ if (page) {
+ kunmap(page);
+ __free_page(page);
+ page = NULL;
+ }
+ header = r5l_read_meta_block(log, last_cp, last_seq, &page);
+ if (!header)
+ goto finish;
+ offset = sizeof(struct r5l_meta_header);
+
+ if (le32_to_cpu(header->type) == R5LOG_TYPE_CHECKPOINT) {
+ r5c_add_checkpoint(&ctx, last_seq);
+ last_cp = r5l_ring_add(log, last_cp, 1);
+ last_seq++;
+ goto again;
+ }
+
+ meta = header;
+ payload = meta + offset;
+ payload_block = r5l_ring_add(log, last_cp, 1);
+
+ while (offset < cpu_to_le32(header->meta_size)) {
+ u16 type = le16_to_cpu(payload->payload_type);
+ u16 entries = le32_to_cpu(payload->blocks) >>log->page_block_shift;
+ u32 *checksum;
+
+ checksum = kmalloc(sizeof(u32) * entries, GFP_KERNEL);
+ memcpy(checksum, payload->data_checksum, sizeof(u32) * entries);
+ if (type == R5LOG_PAYLOAD_DATA) {
+ r5c_add_data(&ctx, last_seq,
+ BLOCK_SECTOR(log, last_cp),
+ BLOCK_SECTOR(log, payload_block),
+ le64_to_cpu(payload->location),
+ entries << PAGE_SECTOR_SHIFT, checksum);
+ } else {
+ r5c_add_parity(&ctx, last_seq,
+ BLOCK_SECTOR(log, last_cp),
+ BLOCK_SECTOR(log, payload_block),
+ le64_to_cpu(payload->location),
+ entries << PAGE_SECTOR_SHIFT, checksum);
+ }
+ payload_block = r5l_ring_add(log, payload_block,
+ entries << log->page_block_shift);
+ offset += sizeof(struct r5l_meta_payload) +
+ entries * sizeof(u32);
+ }
+
+ last_seq++;
+ last_cp = payload_block;
+ goto again;
+finish:
+ if (page) {
+ kunmap(page);
+ __free_page(page);
+ }
+
+ r5c_analyse_log(&ctx);
+
+ r5l_fetch_stripes(&ctx);
+
+ log->seq = last_seq;
+ log->log_start = last_cp;
+ r5c_recover_stripes(&ctx);
+
+ kfree(ctx.stripe_bits);
+ return 0;
+}
+
+void r5l_fake_super(struct r5c_cache *cache, struct md_rdev *rdev)
+{
+ struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
+ struct r5l_super_block *sb_blk;
+ u32 crc;
+
+#define BLKSIZE 4096
+ sb_blk = kmap_atomic(page);
+ sb_blk->header.magic = cpu_to_le32(R5LOG_MAGIC);
+ sb_blk->header.type = cpu_to_le32(R5LOG_TYPE_SUPER);
+ sb_blk->header.seq = cpu_to_le32(0x111);
+ sb_blk->header.meta_size = cpu_to_le32(sizeof(*sb_blk));
+ sb_blk->version = cpu_to_le32(R5LOG_VERSION);
+ sb_blk->stripe_cache_size = cpu_to_le32(PAGE_SIZE);
+ sb_blk->block_size = cpu_to_le32(BLKSIZE);
+ sb_blk->total_blocks = (rdev->sectors * 512 / BLKSIZE) - 1;
+ sb_blk->stripe_data_size = cpu_to_le32(cache->stripe_data_size << 9);
+ sb_blk->chunk_size = cpu_to_le32(cache->chunk_size << 9);
+ sb_blk->stripe_size = cpu_to_le32(cache->stripe_size << 9);
+ sb_blk->parity_disks = cpu_to_le32(cache->parity_disks);
+
+ sb_blk->first_block = cpu_to_le64(1);
+ sb_blk->last_checkpoint = cpu_to_le64(1);
+ sb_blk->meta_checksum_type = R5LOG_CHECKSUM_CRC32;
+ sb_blk->data_checksum_type = R5LOG_CHECKSUM_CRC32;
+ memcpy(sb_blk->uuid, rdev->mddev->uuid, sizeof(sb_blk->uuid));
+
+ crc = crc32_le(~0, sb_blk, BLKSIZE);
+ crc = crc32_le(crc, sb_blk->uuid, sizeof(sb_blk->uuid));
+ sb_blk->header.checksum = crc;
+ kunmap_atomic(sb_blk);
+
+ sync_page_io(rdev, 0, BLKSIZE, page, WRITE, false);
+ __free_page(page);
+}
+
+static int r5l_read_super(struct r5l_log *log)
+{
+ struct md_rdev *rdev = log->rdev;
+ struct r5l_super_block *sb_blk;
+ struct page *page = log->super_page;
+ u32 crc, stored_crc;
+
+ if (!sync_page_io(rdev, 0, PAGE_SIZE, page, READ, false))
+ return -EIO;
+
+ sb_blk = kmap_atomic(page);
+
+ if (le32_to_cpu(sb_blk->version) != R5LOG_VERSION ||
+ le32_to_cpu(sb_blk->header.magic) != R5LOG_MAGIC ||
+ le32_to_cpu(sb_blk->header.type) != R5LOG_TYPE_SUPER ||
+ le64_to_cpu(sb_blk->header.position) != 0 ||
+ le32_to_cpu(sb_blk->header.meta_size) !=
+ sizeof(struct r5l_super_block))
+ goto error;
+
+ log->last_cp_seq = le64_to_cpu(sb_blk->header.seq);
+
+ log->block_size = le32_to_cpu(sb_blk->block_size);
+ log->block_sector_shift = ilog2(log->block_size >> 9);
+ log->page_block_shift = PAGE_SHIFT - ilog2(log->block_size);
+
+ /* Only support this stripe size right now */
+ if (le32_to_cpu(sb_blk->stripe_cache_size) != PAGE_SIZE)
+ goto error;
+ if (log->block_size > PAGE_SIZE)
+ goto error;
+
+ log->stripe_data_size = le32_to_cpu(sb_blk->stripe_data_size) >> 9;
+ log->chunk_size = le32_to_cpu(sb_blk->chunk_size) >> 9;
+ log->stripe_size = le32_to_cpu(sb_blk->stripe_size) >> 9;
+ log->parity_disks = le32_to_cpu(sb_blk->parity_disks);
+
+ if (sb_blk->meta_checksum_type >= R5LOG_CHECKSUM_NR ||
+ sb_blk->data_checksum_type >= R5LOG_CHECKSUM_NR)
+ goto error;
+ log->meta_checksum_type = sb_blk->meta_checksum_type;
+ log->data_checksum_type = sb_blk->data_checksum_type;
+
+ stored_crc = le32_to_cpu(sb_blk->header.checksum);
+ sb_blk->header.checksum = 0;
+ crc = r5l_calculate_checksum(log, ~0,
+ sb_blk, log->block_size, false);
+ crc = r5l_calculate_checksum(log, crc,
+ sb_blk->uuid, sizeof(sb_blk->uuid), false);
+ if (crc != stored_crc)
+ goto error;
+
+ if (memcmp(log->uuid, sb_blk->uuid, sizeof(log->uuid)))
+ goto error;
+
+ log->first_block = le64_to_cpu(sb_blk->first_block);
+ if (log->first_block != 1)
+ goto error;
+ log->total_blocks = le64_to_cpu(sb_blk->total_blocks);
+ log->last_block = log->first_block + log->total_blocks;
+ log->last_checkpoint = le64_to_cpu(sb_blk->last_checkpoint);
+ kunmap_atomic(sb_blk);
+
+ return 0;
+error:
+ kunmap_atomic(sb_blk);
+ return -EINVAL;
+}
+
+static int r5l_init_log(struct r5c_cache *cache)
+{
+ struct r5l_log *log;
+
+ log = &cache->log;
+
+ log->cache = cache;
+ log->bdev = cache->rdev->bdev;
+ log->rdev = cache->rdev;
+
+ log->do_discard = blk_queue_discard(bdev_get_queue(log->bdev));
+
+ log->super_page = alloc_page(GFP_KERNEL);
+ memcpy(log->uuid, cache->mddev->uuid, sizeof(log->uuid));
+
+ init_waitqueue_head(&log->space_waitq);
+ mutex_init(&log->io_mutex);
+
+ spin_lock_init(&log->io_list_lock);
+ INIT_LIST_HEAD(&log->running_ios);
+ init_waitqueue_head(&log->io_waitq);
+
+ if (r5l_read_super(log)) {
+ r5l_fake_super(cache, log->rdev);
+ if (r5l_read_super(log))
+ goto error;
+ }
+ log->uuid_checksum_data = r5l_calculate_checksum(log, ~0, log->uuid,
+ sizeof(log->uuid), true);
+ log->uuid_checksum_meta = r5l_calculate_checksum(log, ~0, log->uuid,
+ sizeof(log->uuid), false);
+
+ log->reserved_blocks = (cache->reserved_space >>
+ log->block_sector_shift) + 1;
+
+ if (log->stripe_data_size != cache->stripe_data_size ||
+ log->chunk_size != cache->chunk_size ||
+ log->stripe_size != cache->stripe_size ||
+ log->parity_disks != cache->parity_disks)
+ goto error;
+
+ r5l_load_log(log);
+
+ if (log->total_blocks * log->block_size / 10 < 1024 * 1024 * 1024)
+ log->low_watermark = log->total_blocks / 10;
+ else
+ log->low_watermark = (1024 * 1024 * 1024 / log->block_size);
+ log->high_watermark = log->low_watermark * 3 / 2;
+
+ return 0;
+error:
+ __free_page(log->super_page);
+ kfree(log);
+ return -EINVAL;
+}
+
+static void r5l_exit_log(struct r5l_log *log)
+{
+ r5l_commit(log);
+
+ __free_page(log->super_page);
+ kfree(log);
+}
+
+struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
+{
+ struct mddev *mddev = rdev->mddev;
+ struct r5c_cache *cache;
+ int i;
+
+ cache = kzalloc(sizeof(*cache), GFP_KERNEL);
+ cache->mddev = mddev;
+ cache->rdev = rdev;
+
+ spin_lock_init(&cache->tree_lock);
+ INIT_RADIX_TREE(&cache->stripe_tree, GFP_ATOMIC);
+ INIT_LIST_HEAD(&cache->log_list);
+ INIT_LIST_HEAD(&cache->full_stripes);
+
+ INIT_LIST_HEAD(&cache->page_pool);
+ spin_lock_init(&cache->pool_lock);
+ cache->max_pages = MAX_MEM >> PAGE_SHIFT;
+
+ cache->stripe_data_size = conf->chunk_sectors * (conf->raid_disks -
+ conf->max_degraded);
+ cache->chunk_size = conf->chunk_sectors;
+ cache->stripe_size = conf->chunk_sectors * conf->raid_disks;
+ cache->parity_disks = conf->max_degraded;
+
+ /* make sure we can add checkpoint */
+ cache->reserved_space = (STRIPE_PARITY_PAGES(cache) <<
+ PAGE_SECTOR_SHIFT) * RECLAIM_BATCH;
+
+ init_waitqueue_head(&cache->reclaim_wait);
+
+ for (i = 0; i < (1 << STRIPE_LOCK_BITS); i++) {
+ spin_lock_init(&cache->stripe_locks[i]);
+ init_waitqueue_head(&cache->stripe_waitq[i]);
+ }
+
+ r5l_init_log(cache);
+
+ while (cache->total_pages < cache->max_pages) {
+ struct page *page = alloc_page(GFP_KERNEL);
+
+ list_add(&page->lru, &cache->page_pool);
+ cache->free_pages++;
+ cache->total_pages++;
+ }
+
+ if (cache->max_pages / 10 < 1024)
+ cache->low_watermark = 1024;
+ else
+ cache->low_watermark = cache->max_pages / 10;
+ cache->high_watermark = cache->low_watermark * 3 /2;
+
+ cache->reclaim_thread = md_register_thread(r5c_reclaim_thread,
+ mddev, "reclaim");
+ cache->reclaim_thread->timeout = RECLAIM_TIMEOUT;
+ //FIXME: make sure total_pages < max_pages
+ return cache;
+}
+
+void r5c_exit_cache(struct r5c_cache *cache)
+{
+ struct r5c_stripe *stripe;
+ struct page *page, *tmp;
+ struct radix_tree_iter iter;
+ void **slot;
+
+ md_unregister_thread(&cache->reclaim_thread);
+ r5l_exit_log(&cache->log);
+
+ radix_tree_for_each_slot(slot, &cache->stripe_tree, &iter, 0) {
+ stripe = radix_tree_deref_slot(slot);
+ r5c_put_stripe(stripe);
+ }
+
+ BUG_ON(!list_empty(&cache->log_list));
+
+ list_for_each_entry_safe(page, tmp, &cache->page_pool, lru) {
+ list_del_init(&page->lru);
+ __free_page(page);
+ }
+}
+
+//sysfs max memory
+//background reclaim time
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cd2f96b..fbef599 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -409,7 +409,7 @@ static int release_stripe_list(struct r5conf *conf,
return count;
}
-static void release_stripe(struct stripe_head *sh)
+void release_stripe(struct stripe_head *sh)
{
struct r5conf *conf = sh->raid_conf;
unsigned long flags;
@@ -741,6 +741,11 @@ static void ops_run_io(struct stripe_head *sh, struct stripe_head_state *s)
might_sleep();
+ if (conf->cache) {
+ if (!r5c_write_parity(conf->cache, sh))
+ return;
+ }
+
for (i = disks; i--; ) {
int rw;
int replace_only = 0;
@@ -3125,6 +3130,7 @@ static void handle_stripe_clean_event(struct r5conf *conf,
WARN_ON(test_bit(R5_SkipCopy, &dev->flags));
WARN_ON(dev->page != dev->orig_page);
}
+
if (!discard_pending &&
test_bit(R5_Discard, &sh->dev[sh->pd_idx].flags)) {
clear_bit(R5_Discard, &sh->dev[sh->pd_idx].flags);
@@ -4179,13 +4185,20 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
int max;
unsigned int chunk_sectors = mddev->chunk_sectors;
unsigned int bio_sectors = bvm->bi_size >> 9;
+ struct r5conf *conf = mddev->private;
- if ((bvm->bi_rw & 1) == WRITE)
+ /* with cache, write must align within stripe */
+ if (((bvm->bi_rw & 1) == WRITE) && !conf->cache)
return biovec->bv_len; /* always allow writes to be mergeable */
if (mddev->new_chunk_sectors < mddev->chunk_sectors)
chunk_sectors = mddev->new_chunk_sectors;
max = (chunk_sectors - ((sector & (chunk_sectors - 1)) + bio_sectors)) << 9;
+ if (((bvm->bi_rw & 1) == WRITE) && conf->cache) {
+ chunk_sectors *= conf->raid_disks - conf->max_degraded;
+ max = (chunk_sectors - (sector_div(sector, chunk_sectors) +
+ bio_sectors)) << 9;
+ }
if (max < 0) max = 0;
if (max <= biovec->bv_len && bio_sectors == 0)
return biovec->bv_len;
@@ -4637,7 +4650,7 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
}
}
-static void make_request(struct mddev *mddev, struct bio * bi)
+void raid5_make_request(struct mddev *mddev, struct bio * bi)
{
struct r5conf *conf = mddev->private;
int dd_idx;
@@ -4785,7 +4798,8 @@ static void make_request(struct mddev *mddev, struct bio * bi)
}
set_bit(STRIPE_HANDLE, &sh->state);
clear_bit(STRIPE_DELAYED, &sh->state);
- if ((bi->bi_rw & REQ_SYNC) &&
+ /* there's no point to delay stripe with cache */
+ if (((bi->bi_rw & REQ_SYNC) || conf->cache) &&
!test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
atomic_inc(&conf->preread_active_stripes);
release_stripe_plug(mddev, sh);
@@ -4809,6 +4823,16 @@ static void make_request(struct mddev *mddev, struct bio * bi)
}
}
+static void make_request(struct mddev *mddev, struct bio *bi)
+{
+ struct r5conf *conf = mddev->private;
+
+ if (conf->cache)
+ r5c_handle_bio(conf->cache, bi);
+ else
+ raid5_make_request(mddev, bi);
+}
+
static sector_t raid5_size(struct mddev *mddev, sector_t sectors, int raid_disks);
static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr, int *skipped)
@@ -5740,6 +5764,9 @@ static void raid5_free_percpu(struct r5conf *conf)
static void free_conf(struct r5conf *conf)
{
+ if (conf->cache)
+ r5c_exit_cache(conf->cache);
+
free_thread_groups(conf);
shrink_stripes(conf);
raid5_free_percpu(conf);
@@ -6339,6 +6366,22 @@ static int run(struct mddev *mddev)
mddev->queue);
}
+ rdev_for_each(rdev, mddev) {
+ if (rdev->raid_disk < 0) {
+ char b[BDEVNAME_SIZE];
+ printk(KERN_INFO "using device %s as cache\n",
+ bdevname(rdev->bdev, b));
+ conf->cache = r5c_init_cache(conf, rdev);
+ }
+ }
+
+ if (conf->cache) {
+ /* Make sure all write do overwite */
+ if (mddev->queue)
+ blk_queue_logical_block_size(mddev->queue, STRIPE_SIZE);
+ conf->skip_copy = 1;
+ }
+
return 0;
abort:
md_unregister_thread(&mddev->thread);
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 983e18a..4c0d573 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -215,6 +215,8 @@ struct stripe_head {
spinlock_t stripe_lock;
int cpu;
struct r5worker_group *group;
+ struct r5c_stripe *stripe;
+ struct list_head stripe_list;
/**
* struct stripe_operations
* @target - STRIPE_OP_COMPUTE_BLK target
@@ -495,6 +497,7 @@ struct r5conf {
struct r5worker_group *worker_groups;
int group_cnt;
int worker_cnt_per_group;
+ struct r5c_cache *cache;
};
/*
@@ -560,4 +563,11 @@ static inline int algorithm_is_DDF(int layout)
extern void md_raid5_kick_device(struct r5conf *conf);
extern int raid5_set_cache_size(struct mddev *mddev, int size);
+
+void release_stripe(struct stripe_head *sh);
+void raid5_make_request(struct mddev *mddev, struct bio *bi);
+void r5c_handle_bio(struct r5c_cache *cache, struct bio *bi);
+int r5c_write_parity(struct r5c_cache *cache, struct stripe_head *sh);
+struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev);
+void r5c_exit_cache(struct r5c_cache *cache);
#endif
diff --git a/include/uapi/linux/raid/md_p.h b/include/uapi/linux/raid/md_p.h
index 49f4210..ed4e504 100644
--- a/include/uapi/linux/raid/md_p.h
+++ b/include/uapi/linux/raid/md_p.h
@@ -305,4 +305,74 @@ struct mdp_superblock_1 {
|MD_FEATURE_RECOVERY_BITMAP \
)
+/* all disk position of below struct start from rdev->start_offset */
+struct r5l_meta_header {
+ __le32 magic;
+ __le32 type;
+ __le32 checksum; /* checksum(metadata block + uuid) */
+ __le32 meta_size;
+ __le64 seq;
+ __le64 position; /* block number the meta is written */
+} __attribute__ ((__packed__));
+
+#define R5LOG_VERSION 0x1
+#define R5LOG_MAGIC 0x6433c509
+
+enum {
+ R5LOG_TYPE_META = 0,
+ R5LOG_TYPE_CHECKPOINT = 1,
+ R5LOG_TYPE_SUPER = 2,
+};
+
+struct r5l_super_block {
+ struct r5l_meta_header header;
+ __le32 version;
+ __le32 stripe_cache_size; /* bytes */
+ __le32 block_size; /* bytes */
+ __le32 stripe_data_size; /* bytes */
+ __le32 chunk_size; /* bytes */
+ __le32 stripe_size; /* bytes */
+ __le32 parity_disks;
+ __le64 total_blocks;
+ __le64 first_block;
+ __le64 last_checkpoint; /* block */
+ __le64 update_time_sec;
+ __le64 update_time_nsec;
+ __u8 meta_checksum_type;
+ __u8 data_checksum_type;
+ __u8 uuid[16];
+} __attribute__ ((__packed__));
+
+enum {
+ R5LOG_CHECKSUM_CRC32 = 0,
+ R5LOG_CHECKSUM_NR = 1,
+};
+
+struct r5l_meta_payload {
+ __le16 payload_type;
+ __le16 payload_flags;
+ __le32 blocks; /* for parity, blocks should be 1 or 2 pages */
+ /* For data, it's raid sector. For stripe, it's stripe sector */
+ __le64 location; /* sector */
+ __le32 data_checksum[]; /* checksum(data + uuid) */
+} __attribute__ ((__packed__));
+
+enum {
+ /* type */
+ R5LOG_PAYLOAD_DATA = 0,
+ R5LOG_PAYLOAD_PARITY = 1,
+ /* flags */
+ R5LOG_PAYLOAD_DISCARD = 1,
+};
+
+struct r5l_meta_block {
+ struct r5l_meta_header header;
+ struct r5l_meta_payload payloads[];
+} __attribute__ ((__packed__));
+
+struct r5l_checkpoint_block {
+ struct r5l_meta_header header;
+ __le64 cp_sec;
+ __le64 cp_nsec;
+} __attribute__ ((__packed__));
#endif
--
1.8.1
^ permalink raw reply related
* [PATCH RESEND] md-cluster: correct the num for comparison
From: Goldwyn Rodrigues @ 2015-04-21 16:25 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: GQJiang
Author: Guoqing Jiang <gqjiang@suse.com>
Since the node num of md-cluster is from zero, and
cinfo->slot_number represents the slot num of dlm,
no need to check for equality.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 96679b2..8ef9e3d 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -612,9 +612,9 @@ static int join(struct mddev *mddev, int nodes)
if (ret)
goto err;
wait_for_completion(&cinfo->completion);
- if (nodes <= cinfo->slot_number) {
- pr_err("md-cluster: Slot allotted(%d) greater than available slots(%d)", cinfo->slot_number - 1,
- nodes);
+ if (nodes < cinfo->slot_number) {
+ pr_err("md-cluster: Slot allotted(%d) is greater than available slots(%d).",
+ cinfo->slot_number, nodes);
ret = -ERANGE;
goto err;
}
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox