* Re: Raid 5 rebuild and Raid 6 reshape question
From: Mikael Abrahamsson @ 2015-04-18 13:13 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: linux-raid
In-Reply-To: <6744821.j7csV3RJVE@balsa>
On Fri, 17 Apr 2015, Thomas Fjellstrom wrote:
> I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
> seemed to work), and i decided to replace it. I actually bought two drives,
If you have a not-ancient kernel, you should have issued the replace
command. You should have added one new drive as spare, then (with a new
mdadm) issued the --replace command. This would have started copying the
drive you wanted replaced onto the spare. Then after all this was done,
you could have added another spare and told it to re-shape to raid6.
That's at least what I would have done.
http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid 5 rebuild and Raid 6 reshape question
From: Thomas Fjellstrom @ 2015-04-18 19:38 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1504181511310.16871@uplift.swm.pp.se>
On Sat 18 Apr 2015 03:13:52 PM Mikael Abrahamsson wrote:
> On Fri, 17 Apr 2015, Thomas Fjellstrom wrote:
> > I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
> > seemed to work), and i decided to replace it. I actually bought two
> > drives,
>
> If you have a not-ancient kernel, you should have issued the replace
> command. You should have added one new drive as spare, then (with a new
> mdadm) issued the --replace command. This would have started copying the
> drive you wanted replaced onto the spare. Then after all this was done,
> you could have added another spare and told it to re-shape to raid6.
> That's at least what I would have done.
>
> http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-ye
> t-failed-disk-in-a-linux-raid5-array
Yes, I could have. The machine's drive bays were full at the time, it wasnt
till I had inserted the new drives that I realized i could have left the old
one in and done a copy or replace. But then the machine was booting without it
and probably not a great time to put the old disk back in (though an assemble
force could have brought it back?). It's almost 60% done, so it's not a huge
problem. I was just curious as to how long an actual raid5->raid6 reshape
would normally be or this setup. I expect it normally can take a while as it
has to rewrite a lot of data (11TiB worth).
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* [PATCH] dm log: Use ULL suffix for 64-bit constants
From: Geert Uytterhoeven @ 2015-04-18 22:07 UTC (permalink / raw)
To: Josef Bacik, Alasdair Kergon, Mike Snitzer
Cc: dm-devel, Neil Brown, linux-raid, linux-kernel,
Geert Uytterhoeven
On 32-bit:
drivers/md/dm-log-writes.c: In function ‘log_super’:
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
drivers/md/dm-log-writes.c:323: warning: integer constant is too large for ‘long’ type
Add a ULL suffix to WRITE_LOG_MAGIC to fix this.
Also add a ULL suffix to WRITE_LOG_VERSION as it's stored in a __le64
field.
Signed-off-by: Geert Uytterhoeven <geert@linux-m68k.org>
---
drivers/md/dm-log-writes.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
index 93e08446a87d06ff..ad1b049ae2ab80a9 100644
--- a/drivers/md/dm-log-writes.c
+++ b/drivers/md/dm-log-writes.c
@@ -55,8 +55,8 @@
#define LOG_DISCARD_FLAG (1 << 2)
#define LOG_MARK_FLAG (1 << 3)
-#define WRITE_LOG_VERSION 1
-#define WRITE_LOG_MAGIC 0x6a736677736872
+#define WRITE_LOG_VERSION 1ULL
+#define WRITE_LOG_MAGIC 0x6a736677736872ULL
/*
* The disk format for this is braindead simple.
--
1.9.1
^ permalink raw reply related
* dm log: struct log_write_super and padding (was: Re: dm: add log writes target)
From: Geert Uytterhoeven @ 2015-04-18 22:16 UTC (permalink / raw)
To: Josef Bacik, Alasdair Kergon, Mike Snitzer
Cc: dm-devel, Neil Brown, linux-raid, linux-kernel@vger.kernel.org
On Sat, Apr 18, 2015 at 2:34 PM, Linux Kernel Mailing List
<linux-kernel@vger.kernel.org> wrote:
> Gitweb: http://git.kernel.org/linus/;a=commit;h=0e9cebe724597a76ab1b0ebc0a21e16f7db11b47
> Commit: 0e9cebe724597a76ab1b0ebc0a21e16f7db11b47
> --- /dev/null
> +++ b/drivers/md/dm-log-writes.c
> @@ -0,0 +1,825 @@
> +/*
> + * Basic info about the log for userspace.
> + */
> +struct log_write_super {
> + __le64 magic;
> + __le64 version;
> + __le64 nr_entries;
> + __le32 sectorsize;
Should there be an explicit padding of 4 bytes?
Does anything assume that sizeof(struct log_write_super) is always 4 * 8?
On architectures where the natural alignment of 64-bit quantities is less
than 8 bytes, sizeof(struct log_write_super) is smaller than 32 bytes
(e.g. on m68k it's 28).
> +};
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Reply:Re: mdadm hang when one subdev error(raid1)
From: 席智勇 @ 2015-04-19 0:06 UTC (permalink / raw)
To: hui jiao; +Cc: linux-raid
In-Reply-To: <CAEuG+XJbMqyQ4_nS=ZKEoDW0shQgLgEqNirxx=xNBa6yH_Y=_w@mail.gmail.com>
thanks for your reply.
in my system, the subdev was a device exported by iscsi through net. as you said the device was suspended by unknown resion.(in fact, the exported disk was errorred.)
what i do not understand is why the io did not timeout, if it was writing info on superblock while the device cannot response it.
now i think the io should have timeout at iscsi level.but it did not, in this situation，mdadm can did nothing except waiting?
At 2015-04-17 17:56:55, "hui jiao" <simonjiaoh@gmail.com> wrote:
>The md42 is trying to update on-disk superblock, but the operation
>doesn't finish. During this peroid, the configuration lock is holded
>by md42_raid1 thread, all other operations which need the lock will
>wait, such as -D --fail --remove.
>
>what's the status of the subdev now? is it suspended?
>check and resume it:
>[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
>1
>[root@node0 ~]# dmsetup resume /dev/dm-0
>[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
>0
>
>On Fri, Apr 17, 2015 at 3:23 PM, 席智勇 <xizhiyong18@163.com> wrote:
>> hi all:
>>
>> I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
>> The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
>> I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
>> Can anyone give me a help？
>> thanks.
>>
>> uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
>> mdadm --version:mdadm - v3.2.5 - 18th May 2012
>> kern.log:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>>
>>
>> -------------
>>
>> Zhiyong Xi
^ permalink raw reply
* Re: Raid 5 rebuild and Raid 6 reshape question
From: Mikael Abrahamsson @ 2015-04-19 8:43 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: linux-raid
In-Reply-To: <1454025.2pEa89kuvv@balsa>
On Sat, 18 Apr 2015, Thomas Fjellstrom wrote:
> problem. I was just curious as to how long an actual raid5->raid6
> reshape would normally be or this setup. I expect it normally can take a
> while as it has to rewrite a lot of data (11TiB worth).
In my experience it takes 5-15 times longer than a normal resync takes, to
reshape from raid5->raid6. So for your 3TB disk size case, I'd say expect
4-8 days or so.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid 5 rebuild and Raid 6 reshape question
From: Thomas Fjellstrom @ 2015-04-19 19:42 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1504191041250.16871@uplift.swm.pp.se>
On Sun 19 Apr 2015 10:43:32 AM Mikael Abrahamsson wrote:
> On Sat, 18 Apr 2015, Thomas Fjellstrom wrote:
> > problem. I was just curious as to how long an actual raid5->raid6
> > reshape would normally be or this setup. I expect it normally can take a
> > while as it has to rewrite a lot of data (11TiB worth).
>
> In my experience it takes 5-15 times longer than a normal resync takes, to
> reshape from raid5->raid6. So for your 3TB disk size case, I'd say expect
> 4-8 days or so.
Ah. Good to know. mdstat was saying about 4-5 days. I kindof thought that was
about normal given the amount of work involved to restripe all drives and
calculate parity.
Thanks :)
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: [PATCH 1/6] md-cluster: correct the num for comparison
From: NeilBrown @ 2015-04-20 1:48 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: GQJiang, linux-raid
In-Reply-To: <20150414154232.GA4079@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]
On Tue, 14 Apr 2015 10:42:32 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> Author: Guoqing Jiang <gqjiang@suse.com>
>
> Since the node num of md-cluster is from zero, and
> cinfo->slot_number represents the slot num of dlm,
> no need to check for equality.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md-cluster.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index 96679b2..6bdc45e 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -612,7 +612,7 @@ static int join(struct mddev *mddev, int nodes)
> if (ret)
> goto err;
> wait_for_completion(&cinfo->completion);
> - if (nodes <= cinfo->slot_number) {
> + if (nodes < cinfo->slot_number) {
> pr_err("md-cluster: Slot allotted(%d) greater than available slots(%d)", cinfo->slot_number - 1,
> nodes);
> ret = -ERANGE;
I'll apply this, but it would be nice to fix the error message too.
If 'nodes' is 3 and 'slot_number' is 4, then the message would be
md-cluster: Slot allotted(3) greater than available slots(3)
which looks weird.
(it should be "is greater", and there should be a "." at the end of the
sentence).
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 5/6] md: re-add a failed disk
From: NeilBrown @ 2015-04-20 1:56 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: GQJiang, linux-raid
In-Reply-To: <20150414154522.GA4105@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 4570 bytes --]
On Tue, 14 Apr 2015 10:45:22 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> This adds the capability of re-adding a failed disk by
> writing "re-add" to /sys/block/mdXX/md/dev-YYY/state.
>
> This facilitates adding disks which have encountered a temporary
> error such as a network disconnection/hiccup in an iSCSI device,
> or a SAN cable disconnection which has been restored. In such
> a situation, you do not need to remove and re-add the device.
> Writing re-add to the failed device's state would add it again
> to the array and perform the recovery of only the blocks which
> were written after the device failed.
>
> This works for generic md, and is not related to clustering. However,
> this patch is to ease re-add operations listed above in clustering
> environments.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md.c | 56 +++++++++++++++++++++++++++++++++++---------------------
> 1 file changed, 35 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 9127d11..ba01605 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -2379,6 +2379,36 @@ repeat:
> }
> EXPORT_SYMBOL(md_update_sb);
>
> +static int add_bound_rdev(struct md_rdev *rdev)
> +{
> + struct mddev *mddev = rdev->mddev;
> + int err = 0;
> +
> + if (!mddev->pers->hot_remove_disk) {
> + /* If there is hot_add_disk but no hot_remove_disk
> + * then added disks for geometry changes,
> + * and should be added immediately.
> + */
> + super_types[mddev->major_version].
> + validate_super(mddev, rdev);
> + err = mddev->pers->hot_add_disk(mddev, rdev);
> + if (err) {
> + unbind_rdev_from_array(rdev);
> + export_rdev(rdev);
> + return err;
> + }
> + }
> + sysfs_notify_dirent_safe(rdev->sysfs_state);
> +
> + set_bit(MD_CHANGE_DEVS, &mddev->flags);
> + if (mddev->degraded)
> + set_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
> + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> + md_new_event(mddev);
> + md_wakeup_thread(mddev->thread);
> + return 0;
> +}
> +
> /* words written to sysfs files may, or may not, be \n terminated.
> * We want to accept with case. For this we use cmd_match.
> */
> @@ -2568,7 +2598,10 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
> clear_bit(Replacement, &rdev->flags);
> err = 0;
> }
> - }
> + } else if (cmd_match(buf, "re-add") && (test_bit(Faulty, &rdev->flags) || (rdev->raid_disk == -1))) {
> + clear_bit(Faulty, &rdev->flags);
> + err = add_bound_rdev(rdev);
> + }
I changed this to:
} else if (cmd_match(buf, "re-add")) {
if (test_bit(Faulty, &rdev->flags) && (rdev->raid_disk == -1)) {
clear_bit(Faulty, &rdev->flags);
err = add_bound_rdev(rdev);
} else
err = -EBUSY;
}
because:
1/ I want all branches of the main if/else to be just "cmd_match...",
2/ I want to return EBUSY, if 're-add' was recognised as a command, but
the default wasn't available for a re-add, and
3/ re-add can only be allowed if the device is faulty AND raid_disk is -1.
If not faulty, re-add makes no sense.
If raid_disk is not -1, then the device still has outstanding IO and
we need to keep waiting for that to complete.
Otherwise, patch accepted - thanks.
NeilBrown
> if (!err)
> sysfs_notify_dirent_safe(rdev->sysfs_state);
> return err ? err : len;
> @@ -5882,29 +5915,10 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
>
> rdev->raid_disk = -1;
> err = bind_rdev_to_array(rdev, mddev);
> - if (!err && !mddev->pers->hot_remove_disk) {
> - /* If there is hot_add_disk but no hot_remove_disk
> - * then added disks for geometry changes,
> - * and should be added immediately.
> - */
> - super_types[mddev->major_version].
> - validate_super(mddev, rdev);
> - err = mddev->pers->hot_add_disk(mddev, rdev);
> - if (err)
> - unbind_rdev_from_array(rdev);
> - }
> if (err)
> export_rdev(rdev);
> else
> - sysfs_notify_dirent_safe(rdev->sysfs_state);
> -
> - set_bit(MD_CHANGE_DEVS, &mddev->flags);
> - if (mddev->degraded)
> - set_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
> - set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> - if (!err)
> - md_new_event(mddev);
> - md_wakeup_thread(mddev->thread);
> + err = add_bound_rdev(rdev);
> if (mddev_is_clustered(mddev) &&
> (info->state & (1 << MD_DISK_CLUSTER_ADD)))
> md_cluster_ops->add_new_disk_finish(mddev);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 6/6] md-cluster: re-add capabilities
From: NeilBrown @ 2015-04-20 2:01 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: GQJiang, linux-raid
In-Reply-To: <20150414154541.GA4138@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 7679 bytes --]
On Tue, 14 Apr 2015 10:45:42 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> When "re-add" is writted to /sys/block/mdXX/md/dev-YYY/state,
> the clustered md:
>
> 1. Sends RE_ADD message with the desc_nr. Nodes receiving the message
> clear the Faulty bit in their respective rdev->flags.
> 2. The node initiating re-add, gathers the bitmaps of all nodes
> and copies them into the local bitmap. It does not clear the bitmap
> from which it is copying.
> 3. Initiating node schedules a md recovery to sync the devices.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/bitmap.c | 20 +++++++++++---------
> drivers/md/bitmap.h | 2 +-
> drivers/md/md-cluster.c | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
> drivers/md/md-cluster.h | 1 +
> drivers/md/md.c | 12 ++++++++++++
> 5 files changed, 72 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index 5ff67c3..956cfb9 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -1852,7 +1852,7 @@ EXPORT_SYMBOL_GPL(bitmap_load);
> * to our bitmap
> */
> int bitmap_copy_from_slot(struct mddev *mddev, int slot,
> - sector_t *low, sector_t *high)
> + sector_t *low, sector_t *high, bool clear_bits)
> {
> int rv = 0, i, j;
> sector_t block, lo = 0, hi = 0;
> @@ -1879,14 +1879,16 @@ int bitmap_copy_from_slot(struct mddev *mddev, int slot,
> }
> }
>
> - bitmap_update_sb(bitmap);
> - /* Setting this for the ev_page should be enough.
> - * And we do not require both write_all and PAGE_DIRT either
> - */
> - for (i = 0; i < bitmap->storage.file_pages; i++)
> - set_page_attr(bitmap, i, BITMAP_PAGE_DIRTY);
> - bitmap_write_all(bitmap);
> - bitmap_unplug(bitmap);
> + if (clear_bits) {
> + bitmap_update_sb(bitmap);
> + /* Setting this for the ev_page should be enough.
> + * And we do not require both write_all and PAGE_DIRT either
> + */
> + for (i = 0; i < bitmap->storage.file_pages; i++)
> + set_page_attr(bitmap, i, BITMAP_PAGE_DIRTY);
> + bitmap_write_all(bitmap);
> + bitmap_unplug(bitmap);
> + }
> *low = lo;
> *high = hi;
> err:
> diff --git a/drivers/md/bitmap.h b/drivers/md/bitmap.h
> index 4aabc74..f1f4dd0 100644
> --- a/drivers/md/bitmap.h
> +++ b/drivers/md/bitmap.h
> @@ -263,7 +263,7 @@ void bitmap_daemon_work(struct mddev *mddev);
> int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
> int chunksize, int init);
> int bitmap_copy_from_slot(struct mddev *mddev, int slot,
> - sector_t *lo, sector_t *hi);
> + sector_t *lo, sector_t *hi, bool clear_bits);
> #endif
>
> #endif
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index 82f1b7b..ad2b5b7 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -73,6 +73,7 @@ enum msg_type {
> RESYNCING,
> NEWDISK,
> REMOVE,
> + RE_ADD,
> };
>
> struct cluster_msg {
> @@ -253,7 +254,7 @@ void recover_bitmaps(struct md_thread *thread)
> str, ret);
> goto clear_bit;
> }
> - ret = bitmap_copy_from_slot(mddev, slot, &lo, &hi);
> + ret = bitmap_copy_from_slot(mddev, slot, &lo, &hi, true);
> if (ret) {
> pr_err("md-cluster: Could not copy data from bitmap %d\n", slot);
> goto dlm_unlock;
> @@ -412,6 +413,16 @@ static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
> pr_warn("%s: %d Could not find disk(%d) to REMOVE\n", __func__, __LINE__, msg->raid_slot);
> }
>
> +static void process_readd_disk(struct mddev *mddev, struct cluster_msg *msg)
> +{
> + struct md_rdev *rdev = md_find_rdev_nr_rcu(mddev, msg->raid_slot);
> +
> + if (rdev && test_bit(Faulty, &rdev->flags))
> + clear_bit(Faulty, &rdev->flags);
> + else
> + pr_warn("%s: %d Could not find disk(%d) which is faulty", __func__, __LINE__, msg->raid_slot);
> +}
> +
> static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
> {
> switch (msg->type) {
> @@ -436,6 +447,11 @@ static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
> __func__, __LINE__, msg->slot);
> process_remove_disk(mddev, msg);
> break;
> + case RE_ADD:
> + pr_info("%s: %d Received RE_ADD from %d\n",
> + __func__, __LINE__, msg->slot);
> + process_readd_disk(mddev, msg);
> + break;
> default:
> pr_warn("%s:%d Received unknown message from %d\n",
> __func__, __LINE__, msg->slot);
> @@ -883,6 +899,35 @@ static int remove_disk(struct mddev *mddev, struct md_rdev *rdev)
> return __sendmsg(cinfo, &cmsg);
> }
>
> +static int gather_bitmaps(struct md_rdev *rdev)
> +{
> + int sn, err;
> + sector_t lo, hi;
> + struct cluster_msg cmsg;
> + struct mddev *mddev = rdev->mddev;
> + struct md_cluster_info *cinfo = mddev->cluster_info;
> +
> + cmsg.type = RE_ADD;
> + cmsg.raid_slot = rdev->desc_nr;
> + err = sendmsg(cinfo, &cmsg);
> + if (err)
> + goto out;
> +
> + for (sn = 0; sn < mddev->bitmap_info.nodes; sn++) {
> + if (sn == (cinfo->slot_number - 1))
> + continue;
> + err = bitmap_copy_from_slot(mddev, sn, &lo, &hi, false);
> + if (err) {
> + pr_warn("md-cluster: Could not gather bitmaps from slot %d", sn);
> + goto out;
> + }
> + if ((hi > 0) && (lo < mddev->recovery_cp))
> + mddev->recovery_cp = lo;
> + }
> +out:
> + return err;
> +}
> +
> static struct md_cluster_operations cluster_ops = {
> .join = join,
> .leave = leave,
> @@ -898,6 +943,7 @@ static struct md_cluster_operations cluster_ops = {
> .add_new_disk_finish = add_new_disk_finish,
> .new_disk_ack = new_disk_ack,
> .remove_disk = remove_disk,
> + .gather_bitmaps = gather_bitmaps,
> };
>
> static int __init cluster_init(void)
> diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
> index 71e5143..6817ee0 100644
> --- a/drivers/md/md-cluster.h
> +++ b/drivers/md/md-cluster.h
> @@ -23,6 +23,7 @@ struct md_cluster_operations {
> int (*add_new_disk_finish)(struct mddev *mddev);
> int (*new_disk_ack)(struct mddev *mddev, bool ack);
> int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
> + int (*gather_bitmaps)(struct md_rdev *rdev);
> };
>
> #endif /* _MD_CLUSTER_H */
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index ba01605..8c37bbf 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -2599,11 +2599,23 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
> err = 0;
> }
> } else if (cmd_match(buf, "re-add") && (test_bit(Faulty, &rdev->flags) || (rdev->raid_disk == -1))) {
> + /* clear_bit is performed _after_ all the devices
> + * have their local Faulty bit cleared. If any writes
> + * happen in the meantime in the local node, they
> + * will land in the local bitmap, which will be synced
> + * by this node eventually
> + */
> + if (mddev_is_clustered(rdev->mddev)) {
> + err = md_cluster_ops->gather_bitmaps(rdev);
> + if (err)
> + goto out;
> + }
> clear_bit(Faulty, &rdev->flags);
> err = add_bound_rdev(rdev);
> }
> if (!err)
> sysfs_notify_dirent_safe(rdev->sysfs_state);
> +out:
> return err ? err : len;
> }
> static struct rdev_sysfs_entry rdev_state =
I changed this to:
if (!mddev_is_clustered(rdev->mddev) ||
(err = md_cluster_ops->gather_bitmaps(rdev)) == 0) {
clear_bit(Faulty, &rdev->flags);
err = add_bound_rdev(rdev);
}
because I think it makes the code a bit clearer.
otherwise, applied.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: NeilBrown @ 2015-04-20 6:20 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <550B265C.7070907@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 5874 bytes --]
On Thu, 19 Mar 2015 13:41:16 -0600 Eric Mei <meijia@gmail.com> wrote:
> On 2015-03-19 12:02 AM, NeilBrown wrote:
> > On Wed, 18 Mar 2015 23:39:11 -0600 Eric Mei <meijia@gmail.com> wrote:
> >
> >> From: Eric Mei <eric.mei@seagate.com>
> >>
> >> When array is degraded, read data landed on failed drives will result in
> >> reading rest of data in a stripe. So a single sequential read would
> >> result in same data being read twice.
> >>
> >> This patch is to avoid chunk aligned read for degraded array. The
> >> downside is to involve stripe cache which means associated CPU overhead
> >> and extra memory copy.
> >>
> >> Signed-off-by: Eric Mei <eric.mei@seagate.com>
> >> ---
> >> drivers/md/raid5.c | 15 ++++++++++++---
> >> 1 files changed, 12 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> >> index cd2f96b..763c64a 100644
> >> --- a/drivers/md/raid5.c
> >> +++ b/drivers/md/raid5.c
> >> @@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
> >> unsigned int chunk_sectors = mddev->chunk_sectors;
> >> unsigned int bio_sectors = bvm->bi_size >> 9;
> >>
> >> - if ((bvm->bi_rw & 1) == WRITE)
> >> - return biovec->bv_len; /* always allow writes to be
> >> mergeable */
> >> + /*
> >> + * always allow writes to be mergeable, read as well if array
> >> + * is degraded as we'll go through stripe cache anyway.
> >> + */
> >> + if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
> >> + return biovec->bv_len;
> >>
> >> if (mddev->new_chunk_sectors < mddev->chunk_sectors)
> >> chunk_sectors = mddev->new_chunk_sectors;
> >> @@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
> >> struct bio * bi)
> >>
> >> md_write_start(mddev, bi);
> >>
> >> - if (rw == READ &&
> >> + /*
> >> + * If array is degraded, better not do chunk aligned read because
> >> + * later we might have to read it again in order to reconstruct
> >> + * data on failed drives.
> >> + */
> >> + if (rw == READ && mddev->degraded == 0 &&
> >> mddev->reshape_position == MaxSector &&
> >> chunk_aligned_read(mddev,bi))
> >> return;
> >
> > Thanks for the patch.
> >
> > However this sort of patch really needs to come with some concrete
> > performance numbers. Preferably both sequential reads and random reads.
> >
> > I agree that sequential reads are likely to be faster, but how much faster
> > are they?
> > I imagine that this might make random reads a little slower. Does it? By
> > how much?
> >
> > Thanks,
> > NeilBrown
> >
>
> Hi Neil,
>
> Sorry I should have done the test in first place.
>
> Following test are done on a enterprise storage node with Seagate 6T SAS
> drives and Xeon E5-2648L CPU (10 cores, 1.9Ghz), 10 disks MD RAID6 8+2,
> chunk size 128 KiB.
>
> I use FIO, using direct-io with various bs size, enough queue depth,
> tested sequential and 100% random read against 3 array config: 1)
> optimal, as baseline; 2) degraded; 3) degraded with this patch. Kernel
> version is 4.0-rc3.
>
> Each individual test I only did once so there might be some variations,
> but we just focus on big trend.
>
> Sequential Read:
> bs=(KiB) optimal(MiB/s) degraded(MiB/s) degraded-with-patch (MiB/s)
> 1024 1608 656 995
> 512 1624 710 956
> 256 1635 728 980
> 128 1636 771 983
> 64 1612 1119 1000
> 32 1580 1420 1004
> 16 1368 688 986
> 8 768 647 953
> 4 411 413 850
>
> Random Read:
> bs=(KiB) optimal(IOPS) degraded(IOPS) degraded-with-patch (IOPS)
> 1024 163 160 156
> 512 274 273 272
> 256 426 428 424
> 128 576 592 591
> 64 726 724 726
> 32 849 848 837
> 16 900 970 971
> 8 927 940 929
> 4 948 940 955
>
> Some notes:
> * In sequential + optimal, as bs size getting smaller, the FIO thread
> become CPU bound.
> * In sequential + degraded, there's big increase when bs is 64K and
> 32K, I don't have explanation.
> * In sequential + degraded-with-patch, the MD thread mostly become CPU
> bound.
>
> If you want to we can discuss specific data point in those data. But in
> general it seems with this patch, we have more predictable and in most
> cases significant better sequential read performance when array is
> degraded, and almost no noticeable impact on random read.
>
> Performance is a complicated thing, the patch works well for this
> particular configuration, but may not be universal. For example I
> imagine testing on all SSD array may have very different result. But I
> personally think in most cases IO bandwidth is more scarce resource than
> CPU.
>
> Eric
Thanks.
That is reasonably convincing.
I've added that text to the commit message, fixed up all the white-space
damage in the patch (tabs were converted to spaces etc ... if you are going
to be sending more patches, please find a way to convince your mailer that
spaces are important), and applied it.
It should be included in my pull request for 4.1
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Breaks LSI RAID on C600 chipset
From: David F. @ 2015-04-21 5:39 UTC (permalink / raw)
To: NeilBrown; +Cc: John Stoffel, linux-raid@vger.kernel.org
In-Reply-To: <CAGRSmLvoj-UhV+8hSESQ+RRyzMHZ68QuBdRZzVU0PWEztZWeOw@mail.gmail.com>
If you need access to the system, a network kvm is available - or if
you have a C600 based system, you should see the same problem there.
On Fri, Apr 10, 2015 at 1:07 PM, David F. <df7729@gmail.com> wrote:
> Okay, and to confirm, this is happening on other C600 based systems.
> Boot to Linux with MDADM raid support and the raid is gone after
> reboot.
>
> On Thu, Apr 9, 2015 at 5:50 PM, NeilBrown <neilb@suse.de> wrote:
>> On Thu, 9 Apr 2015 14:13:05 -0700 "David F." <df7729@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I built a new system, installed new drives (no partitioning, just raw
>>> new drives), configured RAID 1, boot to linux, created reports
>>> attached. Rebooted and the system doesn't see any configured raid
>>> drives (just the two raw drives).
>>
>> Thanks. I might take a look, but I would greatly prefer it if you kept the
>> linux-raid list on the Cc.....
>>
>> NeilBrown
>>
^ permalink raw reply
* [RFC] A caching layer for RAID5/6
From: Shaohua Li @ 2015-04-21 5:54 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, songliubraving, Kernel-team
Hi,
This is the new design/code to implement caching layer for RAID5/6. It's not
for merge yet, but basic code works fine here. I'd like to hear some
comments/suggestions early before I continue.
Thanks,
Shaohua
Main goal of the caching layer is to aggregate IO write to hopefully
make full stripe IO write and fix write hole issue. This might speed up
read too, but it's not optimized for read, eg, we don't proactively
cache data for read. The aggregation makes a lot of sense for workloads
which sequentially write to several files with/without fsync. Such
workloads are popular in today's datacenter.
Write IO data will write to a cache disk (SSD) first, then later the
data will be flushed to raid disks.
The cache disk will be organized as a simple ring buffer log. For IO
data, a tuple (raid_sector, io length, checksum, data) will be appended
to the log; for raid 5/6 parity, a tuple (stripe_sector, parity length,
checksum, parity data) will be appended to the log. We don't have
on-disk index for the data appended to the log. So either we can rebuild
an in-memory index at startup with scanning the whole log, or we can
flush all data from cache disk to raid disks at shutdown so cache disk
has no valid data. Current code chooses the first option, but this can
be easily changed.
We have a simple meta data for the above tuples. It's essentially a
tuple (sequence, metadata length). Crash recovery or startup will scan
the log to read the metadata and rebuild in-memory index. If metadata is
broken at the head of the log, even metadata afterward is ok, the
scanning will not work well. So we take some steps to mitigate the
issue:
-meta data is written with WRITE_FUA. write IO return only happens after both
data and metadata IO finish
-IO return happens in the order they are added in the log.
These will make sure write IO return happens after its meta and all
previous meta IO finish, eg all related meta are persistent.
The in-memory index is a simple list of io_range (sequence, metadata
sector, data sector, length). The list is orded by sequence. The first
io_range entry's metadata sector is the tail of the log. There is also a
struct to track io ranges within a stripe. All stripes will be organized
as a radix tree.
All IO data will be copied to a memory pool for caching too until the
data is flushed to raid disks. This is just to simplify the
implementation, it's not mandated. In the future flush can do a data
copy from cache disk to raid disks, so the memory pool can be discarded.
If a stripe is flushed to raid disks, memory of the stripe can be
reused.
We have two limited resources here, memory pool and cache disk space. If
resource is tight, we will do reclaim. In either case, we will flush
some data from cache disk to raid disks. However, we implement different
strategies. For memory reclaim, we prefer reclaiming full stripe. For
cache disk space reclaim, we prefer reclaiming io_range entry at the
head of index list.
We always do reclaim in stripe unit. Reclaim could create holes in the
log, eg, some io_range in the middle is reclaimed, but io_range at the
head remains. So the index list entries don't always have continuous
sequence. But this doesn't matter, the first io_range is always the log
tail. Superblock has a field pointing to the position of log tail. The
hole can waste a lot of disk space though. In the future, we can
implement a garbage collection to mitigrate the issue, eg, copy data
from the index tail to head.
In the process reclaim flush data to raid disk, stripe parity will be
append to cache log. Parity is always after its corresponding data. Data
writing to raid disks only happens after all data and parity are already
in cache disk. This will fix the write whole issue. After a stripe is
flushed to raid disks, we will add a checkpoint. The checkpoint, unlike
classic checkpoint, indicates a stripe is flushed to raid disks only if
the stripe has both data and parity in the log ahead of the chekcpoint.
There are still valid data of other stripes before checkpoint.
Recovery relies on if stripe data and parity matches. If recovery finds
data and parity of a stripe, it will calculate how many parity data the
stripe should have according to its data. If not all parity exist in the
log, the stripe hasn't started to be flushed to raid disks yet, so the
parity can be ignored. Otherwise, recovery might write both data and
parity of a stripe to raid disks depending on if checkpoint is found.
IO write code path:
1. copy bio data to stripe memory pages
2. append metadata and data to cache log
3. IO write endio
reclaim code path:
1. select stripe to reclaim
2. write all stripe data to raid disk
3. in raid5 ops_run_io, append metadata and parity data to cache log.
ops_run_io doesn't write data/parity to raid disks at this time
4. flush cache disk cache
5. ops_run_io continues. data/parity will be written to raid disks
6. flush all raid disks cache
7. add a checkpoint
8. delete in-memory index of the stripe, and advance superblock log checkpoint
Recovery:
Crash in IO write code path doesn't need recovery. If data and checksum
don't match, the data will be ignored so read will return old data. In
reclaim code path, crash before step 4 doesn't need recovery as
data/parity don't touch raid disk yet. Parity can be ignored too. crash
after 7 doesn't need recovery too, as the stripe is fully flushed to
raid disks. Crash between 4 and 7 need recovery. Data and parity in the
log will be written to raid disks.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/Makefile | 2 +-
drivers/md/raid5-cache.c | 2412 ++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 51 +-
drivers/md/raid5.h | 10 +
include/uapi/linux/raid/md_p.h | 70 ++
5 files changed, 2540 insertions(+), 5 deletions(-)
create mode 100644 drivers/md/raid5-cache.c
diff --git a/drivers/md/Makefile b/drivers/md/Makefile
index a2da532..7d85784 100644
--- a/drivers/md/Makefile
+++ b/drivers/md/Makefile
@@ -16,7 +16,7 @@ dm-cache-mq-y += dm-cache-policy-mq.o
dm-cache-cleaner-y += dm-cache-policy-cleaner.o
dm-era-y += dm-era-target.o
md-mod-y += md.o bitmap.o
-raid456-y += raid5.o
+raid456-y += raid5.o raid5-cache.o
# Note: link order is important. All raid personalities
# and must come before md.o, as they each initialise
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
new file mode 100644
index 0000000..cfb8bed
--- /dev/null
+++ b/drivers/md/raid5-cache.c
@@ -0,0 +1,2412 @@
+#include <linux/kernel.h>
+#include <linux/wait.h>
+#include <linux/blkdev.h>
+#include <linux/raid/md_p.h>
+#include <linux/crc32.h>
+#include "md.h"
+#include "raid5.h"
+
+#define MAX_MEM (256 * 1024 * 1024)
+#define RECLAIM_BATCH 16
+#define RECLAIM_TIMEOUT (5 * HZ)
+#define FULL_STRIPE_MAX_AGE (10 * HZ)
+
+struct r5l_log {
+ struct r5c_cache *cache;
+ struct block_device *bdev;
+ struct md_rdev *rdev;
+
+ struct page *super_page;
+ u8 uuid[16];
+ u32 uuid_checksum_data;
+ u32 uuid_checksum_meta;
+
+ int block_size; /* bytes */
+ int block_sector_shift;
+ int page_block_shift;
+ int stripe_data_size; /* sector */
+ int chunk_size; /* sector */
+ int stripe_size; /* sector */
+ int parity_disks;
+
+ u64 total_blocks;
+ u64 first_block;
+ u64 last_block;
+
+ u64 low_watermark; /* For disk space, block */
+ u64 high_watermark;
+
+ u64 last_checkpoint; /* block */
+ u64 last_cp_seq;
+
+ int do_discard;
+
+ u64 seq; /* get after read log */
+ u64 log_start; /* get after read log */
+
+ u8 data_checksum_type;
+ u8 meta_checksum_type;
+
+ ssize_t reserved_blocks;
+ wait_queue_head_t space_waitq;
+
+ struct mutex io_mutex;
+ struct r5l_ioctl *current_io;
+
+ spinlock_t io_list_lock;
+ struct list_head running_ios; /* order is important */
+ wait_queue_head_t io_waitq;
+};
+
+struct r5l_task;
+/* end function must free task */
+typedef void (r5l_task_end_fn)(struct r5l_task *task);
+struct r5l_task {
+ struct list_head list;
+ int type;
+ struct bio *bio;
+ union {
+ struct {
+ bio_end_io_t *saved_endio;
+ void *saved_private;
+ };
+ struct {
+ sector_t stripe_sector;
+ struct page *page_p;
+ struct page *page_q;
+ };
+ };
+ /* tasks in a single r5l_ioctl will have the same seq and meta_start */
+ sector_t meta_start;
+ sector_t data_start;
+ u64 seq;
+ r5l_task_end_fn *fn;
+ void *private;
+ u32 checksum[];
+};
+
+/*
+ * meta page should write with FUA, and meta page should return IO in order,
+ * otherwise recovery will have trouble. The reason is we are using a simple
+ * log. If metadata corrupts in the middle, recovery can't work even metadata
+ * at the end is good.
+ */
+struct r5l_ioctl {
+ struct r5l_log *log;
+ struct list_head log_sibling;
+
+ struct page *meta_page;
+ sector_t meta_sector;
+ int meta_offset;
+ u64 seq;
+ struct bio *meta_bio;
+
+ struct list_head tasks;
+ struct bio *current_bio;
+ atomic_t refcnt;
+};
+
+struct r5l_rsv {
+ ssize_t reserved;
+ ssize_t used;
+};
+
+struct r5c_io_range {
+ struct list_head log_sibling;
+ struct list_head stripe_sibling;
+
+ u64 seq;
+
+ sector_t meta_start; /* cache position */
+ sector_t data_start; /* cache position */
+ sector_t raid_start;
+ ssize_t data_sectors;
+
+ struct r5c_stripe *stripe;
+ union {
+ struct bio *bio;
+ u32 *checksum; /* only for recovery */
+ };
+};
+
+struct r5c_stripe {
+ u64 raid_index;
+ struct r5c_cache *cache;
+ atomic_t ref;
+ int state;
+
+ struct list_head io_ranges; /* order list */
+ union {
+ struct list_head stripes;
+ struct list_head parity_list; /* just for recovery */
+ };
+
+ struct list_head lru;
+
+ int existing_pages;
+ atomic_t dirty_stripes;
+ atomic_t pending_bios;
+ struct page **parity_pages; /* just for recovery */
+ struct page *data_pages[];
+};
+
+enum {
+ STRIPE_RUNNING = 0,
+ STRIPE_FROZEN = 1, /* Doesn't accept new IO */
+ STRIPE_PARITY_DONE = 2,
+ STRIPE_INRAID = 3,
+ STRIPE_DEAD = 4,
+};
+
+#define STRIPE_LOCK_BITS 8
+struct r5c_cache {
+ struct mddev *mddev;
+ struct md_rdev *rdev;
+
+ struct r5l_log log;
+
+ spinlock_t tree_lock; /* protect stripe_tree, log_list, full_stripes */
+ struct radix_tree_root stripe_tree;
+ struct list_head log_list; /* sorted list of io_range */
+ struct list_head full_stripes;
+
+ struct list_head page_pool;
+ spinlock_t pool_lock;
+ u64 free_pages;
+ u64 total_pages;
+ u64 max_pages;
+ u64 low_watermark; /* for memory, pages */
+ u64 high_watermark;
+ unsigned long full_stripe_ready_time;
+
+ int stripe_data_size; /* stripe size excluding parity, sector */
+ int chunk_size; /* one disk chunk size including parity, sector */
+ int stripe_size; /* stripe size including parity, sector */
+ int parity_disks;
+
+ int reserved_space; /* log reserved size, sector */
+
+ unsigned long reclaim_reason;
+ wait_queue_head_t reclaim_wait;
+ struct md_thread *reclaim_thread;
+
+ int in_recovery;
+
+ spinlock_t stripe_locks[1 << STRIPE_LOCK_BITS];
+ wait_queue_head_t stripe_waitq[1 << STRIPE_LOCK_BITS];
+};
+
+enum {
+ RECLAIM_MEM = 0, /* work hard to reclaim memory */
+ RECLAIM_MEM_BACKGROUND = 1, /* try to reclaim memory */
+ RECLAIM_MEM_FULL = 2, /* only reclaim full stripe */
+ RECLAIM_DISK = 8, /* work hard to reclaim disk */
+ RECLAIM_DISK_BACKGROUND = 9, /* try to reclaim disk */
+};
+
+#define STRIPE_INDEX_OFFSET(c, sect, index, offset) \
+({ \
+ sector_t tmp = sect; \
+ offset = sector_div(tmp, c->stripe_data_size); \
+ index = tmp; \
+})
+
+#define STRIPE_RAID_SECTOR(cache, stripe) \
+ (stripe->raid_index * cache->stripe_data_size)
+
+#define PAGE_SECTOR_SHIFT (PAGE_SHIFT - 9)
+
+#define STRIPE_DATA_PAGES(c) (c->stripe_data_size >> PAGE_SECTOR_SHIFT)
+#define STRIPE_PARITY_PAGES(c) \
+ ((c->stripe_size - c->stripe_data_size) >> PAGE_SECTOR_SHIFT)
+#define BLOCK_SECTOR(log, b) ((b) << log->block_sector_shift)
+#define PAGE_BLOCKS(log, p) ((p) >> log->page_block_shift)
+
+#define UUID_CHECKSUM(log, data) \
+ (data ? log->uuid_checksum_data : log->uuid_checksum_meta)
+
+static u32 r5l_calculate_checksum(struct r5l_log *log, u32 crc,
+ void *buf, ssize_t size, bool data)
+{
+ if (log->data_checksum_type != R5LOG_CHECKSUM_CRC32)
+ BUG();
+ if (log->meta_checksum_type != R5LOG_CHECKSUM_CRC32)
+ BUG();
+ return crc32_le(crc, buf, size);
+}
+
+static u64 r5l_ring_add(struct r5l_log *log, u64 block, int inc)
+{
+ block += inc;
+ if (block >= log->last_block)
+ block = block - log->last_block + log->first_block;
+ return block;
+}
+
+static void r5c_wake_wait_reclaimer(struct r5c_cache *cache, int reason);
+static void r5c_wake_reclaimer(struct r5c_cache *cache, int reason);
+static void r5l_reclaim_space(struct r5l_log *log)
+{
+ r5c_wake_wait_reclaimer(log->cache, RECLAIM_DISK);
+ wake_up_all(&log->space_waitq);
+}
+
+static u64 r5l_free_space(struct r5l_log *log)
+{
+ u64 used_size = (log->total_blocks + log->log_start -
+ log->last_checkpoint) % log->total_blocks;
+ return log->total_blocks - used_size - log->reserved_blocks;
+}
+
+static int r5l_get_reserve(struct r5l_log *log,
+ struct r5l_rsv *rsv, ssize_t size, bool parity)
+{
+ u64 free;
+ BUG_ON(!mutex_is_locked(&log->io_mutex));
+
+ if (parity)
+ size = 0;
+ rsv->reserved = size;
+ rsv->used = 0;
+
+ free = r5l_free_space(log);
+ if (free >= size) {
+ log->reserved_blocks += size;
+
+ if (free - size < log->low_watermark)
+ r5c_wake_reclaimer(log->cache,
+ RECLAIM_DISK_BACKGROUND);
+ return 0;
+ }
+
+ mutex_unlock(&log->io_mutex);
+
+ r5l_reclaim_space(log);
+
+ mutex_lock(&log->io_mutex);
+ wait_event_cmd(log->space_waitq, r5l_free_space(log) >= size,
+ mutex_unlock(&log->io_mutex), mutex_lock(&log->io_mutex));
+ log->reserved_blocks += size;
+
+ return 0;
+}
+
+static void r5l_put_reserve(struct r5l_log *log,
+ struct r5l_rsv *rsv)
+{
+ ssize_t remaining;
+
+ BUG_ON(!mutex_is_locked(&log->io_mutex));
+ remaining = rsv->reserved - rsv->used;
+
+ if (remaining < 0)
+ remaining = 0;
+ log->reserved_blocks -= remaining;
+ if (remaining > 0)
+ wake_up_all(&log->space_waitq);
+}
+
+static void r5l_submit_bio(struct r5l_log *log, int rw, struct bio *bio)
+{
+ /* all IO must start from rdev->data_offset */
+ struct bio *next = bio->bi_next;
+
+ /* queue_task split bio into bio->bi_next */
+ if (next) {
+ next->bi_iter.bi_sector += log->rdev->data_offset;
+ submit_bio(rw, next);
+ bio->bi_next = NULL;
+ }
+
+ bio->bi_iter.bi_sector += log->rdev->data_offset;
+ submit_bio(rw, bio);
+}
+
+static void r5l_put_ioctl(struct r5l_ioctl *io)
+{
+ struct r5l_log *log = io->log;
+ LIST_HEAD(finished_io);
+ unsigned long flags;
+ bool do_wakeup;
+
+ if (!atomic_dec_and_test(&io->refcnt))
+ return;
+
+ /*
+ * finish IO in order. Note this guarantees later task runs ->fn after
+ * previous task's IO and this task's IO are finished, but doesn't mean
+ * previous task already runs its ->fn
+ */
+ spin_lock_irqsave(&log->io_list_lock, flags);
+ while (!list_empty(&log->running_ios)) {
+ io = list_first_entry(&log->running_ios, struct r5l_ioctl,
+ log_sibling);
+ if (atomic_read(&io->refcnt))
+ break;
+ list_move_tail(&io->log_sibling, &finished_io);
+
+ }
+ do_wakeup = list_empty(&log->running_ios);
+ spin_unlock_irqrestore(&log->io_list_lock, flags);
+
+ while (!list_empty(&finished_io)) {
+ struct r5l_task *task;
+
+ io = list_first_entry(&finished_io, struct r5l_ioctl,
+ log_sibling);
+ list_del(&io->log_sibling);
+
+ while (!list_empty(&io->tasks)) {
+ task = list_first_entry(&io->tasks, struct r5l_task,
+ list);
+ list_del(&task->list);
+ if (task->type == R5LOG_PAYLOAD_DATA) {
+ task->bio->bi_end_io = task->saved_endio;
+ task->bio->bi_private = task->saved_private;
+ }
+ task->fn(task);
+ }
+ __free_page(io->meta_page);
+ kfree(io);
+ }
+ if (do_wakeup)
+ wake_up(&log->io_waitq);
+}
+
+static void r5l_log_endio(struct bio *bio, int error)
+{
+ struct r5l_ioctl *io = bio->bi_private;
+
+ bio_put(bio);
+ r5l_put_ioctl(io);
+}
+
+static int r5l_submit_io(struct r5l_log *log)
+{
+ struct r5l_ioctl *io = log->current_io;
+ struct r5l_task *task;
+ struct r5l_meta_header *header;
+ u32 crc;
+
+ header = kmap_atomic(io->meta_page);
+ header->meta_size = cpu_to_le32(io->meta_offset);
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false),
+ header, log->block_size, false);
+ header->checksum = cpu_to_le32(crc);
+ kunmap_atomic(header);
+
+ r5l_submit_bio(log, WRITE_FUA, io->meta_bio);
+
+ list_for_each_entry(task, &io->tasks, list) {
+ if (task->bio)
+ r5l_submit_bio(log, task->bio->bi_rw, task->bio);
+ }
+ log->current_io = NULL;
+ r5l_put_ioctl(io);
+ return 0;
+}
+
+static struct r5l_ioctl *r5l_new_meta(struct r5l_log *log, struct r5l_rsv *rsv)
+{
+ struct r5l_ioctl *io;
+
+ io = kmalloc(sizeof(*io), GFP_NOIO);
+ io->log = log;
+ io->meta_page = alloc_page(GFP_NOIO|__GFP_ZERO);
+ io->meta_sector = BLOCK_SECTOR(log, log->log_start);
+ io->meta_offset = sizeof(struct r5l_meta_header);
+ atomic_set(&io->refcnt, 1);
+ INIT_LIST_HEAD(&io->tasks);
+ io->seq = log->seq;
+
+ io->meta_bio = bio_kmalloc(GFP_NOIO, bio_get_nr_vecs(log->bdev));
+ io->meta_bio->bi_bdev = log->bdev;
+ io->meta_bio->bi_iter.bi_sector = io->meta_sector;
+ bio_add_page(io->meta_bio, io->meta_page, log->block_size, 0);
+ io->meta_bio->bi_end_io = r5l_log_endio;
+ io->meta_bio->bi_private = io;
+
+ atomic_inc(&io->refcnt);
+
+ log->seq++;
+ log->log_start = r5l_ring_add(log, log->log_start, 1);
+ io->current_bio = io->meta_bio;
+
+ spin_lock_irq(&log->io_list_lock);
+ list_add_tail(&io->log_sibling, &log->running_ios);
+ spin_unlock_irq(&log->io_list_lock);
+
+ rsv->used++;
+ return io;
+}
+
+static int r5l_get_meta(struct r5l_log *log, struct r5l_rsv *rsv,
+ ssize_t pages, bool is_bio)
+{
+ struct r5l_ioctl *io;
+ struct r5l_meta_header *header;
+ ssize_t meta_size;
+
+ meta_size = sizeof(struct r5l_meta_payload) +
+ sizeof(u32) * pages;
+ io = log->current_io;
+ if (io && io->meta_offset + meta_size > log->block_size)
+ r5l_submit_io(log);
+ io = log->current_io;
+ if (io)
+ return 0;
+
+ io = r5l_new_meta(log, rsv);
+ log->current_io = io;
+
+ if (is_bio)
+ io->current_bio = NULL;
+
+ header = kmap_atomic(io->meta_page);
+ header->magic = cpu_to_le32(R5LOG_MAGIC);
+ header->type = cpu_to_le32(R5LOG_TYPE_META);
+ header->seq = cpu_to_le64(log->seq - 1);
+ if (log->log_start == log->first_block)
+ header->position = cpu_to_le64(log->last_block - 1);
+ else
+ header->position = cpu_to_le64(log->log_start - 1);
+ kunmap_atomic(header);
+
+ return 0;
+}
+
+static int r5l_log_add_parity_pages(struct r5l_log *log,
+ struct r5l_task *task)
+{
+ struct r5l_ioctl *io = log->current_io;
+ struct bio *bio;
+ struct page *pages[] = {task->page_p, task->page_q};
+ int i;
+
+ bio = io->current_bio;
+
+alloc_bio:
+ if (!bio) {
+ /* bio can only contain one page ? */
+ BUG_ON(task->bio);
+ bio = bio_kmalloc(GFP_NOIO, bio_get_nr_vecs(log->bdev));
+ bio->bi_rw = WRITE;
+ task->bio = bio;
+ io->current_bio = bio;
+ bio->bi_bdev = log->bdev;
+ bio->bi_iter.bi_sector = BLOCK_SECTOR(log, log->log_start);
+ }
+
+ for (i = 0; i < ARRAY_SIZE(pages); i++) {
+ if (!pages[i])
+ continue;
+ if (!bio_add_page(bio, pages[i], PAGE_SIZE, 0)) {
+ bio = NULL;
+ goto alloc_bio;
+ }
+ pages[i] = NULL;
+ }
+ return 0;
+}
+
+static int r5l_queue_task(struct r5l_log *log, struct r5l_task *task)
+{
+ ssize_t pages;
+ struct r5l_ioctl *io;
+ struct r5l_rsv rsv;
+ struct r5l_meta_payload *payload;
+ struct bio *bio;
+ void *meta;
+ int i;
+
+ if (task->type == R5LOG_PAYLOAD_DATA)
+ pages = bio_sectors(task->bio) >> PAGE_SECTOR_SHIFT;
+ else
+ pages = !!task->page_p + !!task->page_q;
+
+ /* FIXME: handle tasks in batch to be more efficient */
+ mutex_lock(&log->io_mutex);
+
+ r5l_get_reserve(log, &rsv, 1 + (pages << log->page_block_shift),
+ task->type != R5LOG_PAYLOAD_DATA);
+
+ r5l_get_meta(log, &rsv, pages, task->type == R5LOG_PAYLOAD_DATA);
+
+ io = log->current_io;
+
+ meta = kmap_atomic(io->meta_page);
+ payload = meta + io->meta_offset;
+ payload->payload_type = cpu_to_le16(task->type);
+ payload->blocks = cpu_to_le32(PAGE_BLOCKS(log, pages));
+ if (task->type == R5LOG_PAYLOAD_DATA) {
+ bio = task->bio;
+ payload->location = cpu_to_le64(bio->bi_iter.bi_sector);
+ } else
+ payload->location = cpu_to_le64(task->stripe_sector);
+ for (i = 0; i < pages; i++)
+ payload->data_checksum[i] = cpu_to_le32(task->checksum[i]);
+ kunmap_atomic(meta);
+
+ io->meta_offset += sizeof(struct r5l_meta_payload) +
+ sizeof(u32) * pages;
+ task->seq = io->seq;
+ task->meta_start = io->meta_sector;
+ task->data_start = BLOCK_SECTOR(log, log->log_start);
+
+ if (task->type == R5LOG_PAYLOAD_PARITY)
+ r5l_log_add_parity_pages(log, task);
+
+ if (task->bio) {
+ bio = task->bio;
+ bio->bi_iter.bi_sector = task->data_start;
+ bio->bi_end_io = r5l_log_endio;
+ bio->bi_private = io;
+ bio->bi_bdev = log->bdev;
+ atomic_inc(&io->refcnt);
+ }
+
+ if (task->type == R5LOG_PAYLOAD_PARITY)
+ bio = io->current_bio;
+ else {
+ bio = task->bio;
+ /* endio will bio_put, but this bio isn't allocated by us */
+ bio_get(bio);
+ }
+
+ if (bio_end_sector(bio) > (BLOCK_SECTOR(log, log->last_block))) {
+ struct bio *split;
+ split = bio_split(bio, BLOCK_SECTOR(log, log->last_block) -
+ bio->bi_iter.bi_sector, GFP_NOIO, NULL);
+ bio->bi_next = split;
+ bio_chain(split, bio);
+ bio->bi_iter.bi_sector = BLOCK_SECTOR(log, log->first_block);
+ }
+
+ log->log_start = r5l_ring_add(log, log->log_start,
+ PAGE_BLOCKS(log, pages));
+ rsv.used += PAGE_BLOCKS(log, pages);
+
+ list_add_tail(&task->list, &io->tasks);
+
+ r5l_submit_io(log);
+
+ r5l_put_reserve(log, &rsv);
+ mutex_unlock(&log->io_mutex);
+ return 0;
+}
+
+/*
+ * Since we finish IO in order and meta is written with FUA, so meta is always
+ * consistent. For FUA, just make sure both meta and data are written with FUA
+ * and finish IO after both parts are finished. For FLUSH, previous data and
+ * meta are already finished, just need flush the disk. In either case, bio
+ * already has the flag, so just handle them like normal bio
+ */
+static void r5l_queue_empty_flush_bio(struct r5l_log *log, struct bio *bio)
+{
+ bio->bi_bdev = log->bdev;
+ generic_make_request(bio);
+}
+
+static int r5l_queue_bio(struct r5l_log *log, struct bio *bio,
+ r5l_task_end_fn fn, void *private)
+{
+ struct r5l_task *task;
+ int pages = bio_sectors(bio) >> PAGE_SECTOR_SHIFT;
+ u32 *checksum;
+ struct bio *src = bio;
+ struct bvec_iter iter;
+ struct bio_vec bv;
+ void *src_p;
+ unsigned int page_index = 0, page_offset = 0;
+ unsigned int bytes;
+
+ task = kmalloc(sizeof(*task) + sizeof(u32) * pages, GFP_NOIO);
+
+ INIT_LIST_HEAD(&task->list);
+ task->type = R5LOG_PAYLOAD_DATA;
+ task->bio = bio;
+ task->saved_endio = bio->bi_end_io;
+ task->saved_private = bio->bi_private;
+ task->fn = fn;
+ task->private = private;
+
+ checksum = (u32 *)(task + 1);
+
+ iter = src->bi_iter;
+
+ checksum[0] = UUID_CHECKSUM(log, true);
+ while (1) {
+ if (!iter.bi_size) {
+ src = src->bi_next;
+ if (!src)
+ break;
+
+ iter = src->bi_iter;
+ }
+
+ if (page_offset == PAGE_SIZE) {
+ page_index++;
+ page_offset = 0;
+ checksum[page_index] = UUID_CHECKSUM(log, true);
+ }
+
+ bv = bio_iter_iovec(src, iter);
+
+ bytes = min_t(unsigned int, bv.bv_len, PAGE_SIZE - page_offset);
+
+ src_p = kmap_atomic(bv.bv_page);
+
+ checksum[page_index] = r5l_calculate_checksum(log,
+ checksum[page_index], src_p + bv.bv_offset,
+ bytes, true);
+
+ kunmap_atomic(src_p);
+
+ bio_advance_iter(src, &iter, bytes);
+ page_offset += bytes;
+ }
+ return r5l_queue_task(log, task);
+}
+
+static int r5l_queue_parity(struct r5l_log *log,
+ sector_t stripe_sector, struct page *page_p,
+ struct page *page_q, r5l_task_end_fn fn, void *private)
+{
+ struct r5l_task *task;
+ void *addr;
+ u32 *checksum;
+
+ task = kmalloc(sizeof(*task) + sizeof(u32) * 2, GFP_NOIO);
+
+ INIT_LIST_HEAD(&task->list);
+ task->type = R5LOG_PAYLOAD_PARITY;
+ task->bio = NULL;
+ task->stripe_sector = stripe_sector;
+ task->page_p = page_p;
+ task->page_q = page_q;
+ task->fn = fn;
+ task->private = private;
+
+ checksum = (u32 *)(task + 1);
+ addr = kmap_atomic(page_p);
+ checksum[0] = r5l_calculate_checksum(log,
+ UUID_CHECKSUM(log, true), addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+
+ if (page_q) {
+ addr = kmap_atomic(page_q);
+ checksum[1] = r5l_calculate_checksum(log,
+ UUID_CHECKSUM(log, true), addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+ } else
+ checksum[1] = 0;
+
+ return r5l_queue_task(log, task);
+}
+
+/* all previous queued data are settled down */
+static void r5l_commit(struct r5l_log *log)
+{
+ blkdev_issue_flush(log->bdev, GFP_NOIO, NULL);
+}
+
+static void r5l_cp_end(struct r5l_task *task)
+{
+ struct completion *comp = task->private;
+ complete(comp);
+}
+
+static int r5l_checkpoint(struct r5l_log *log, u64 *next_seq, sector_t *next_sec)
+{
+ struct r5l_ioctl *io;
+ struct r5l_rsv rsv;
+ struct r5l_checkpoint_block *cp;
+ struct timespec now = current_kernel_time();
+ struct r5l_task task;
+ DECLARE_COMPLETION_ONSTACK(cp_complete);
+ u32 crc;
+
+ mutex_lock(&log->io_mutex);
+ r5l_get_reserve(log, &rsv, 1, true);
+
+ if (log->current_io)
+ r5l_submit_io(log);
+
+ /* FIXME: we already wait in reclaim thread, don't need wait again */
+ spin_lock_irq(&log->io_list_lock);
+ wait_event_lock_irq(log->io_waitq, list_empty(&log->running_ios),
+ log->io_list_lock);
+ spin_unlock_irq(&log->io_list_lock);
+
+ io = r5l_new_meta(log, &rsv);
+ task.fn = r5l_cp_end;
+ task.private = &cp_complete;
+ /* pretend we are parity, so r5l_put_ioctl knows how to handle it */
+ task.type = R5LOG_PAYLOAD_PARITY;
+
+ cp = kmap_atomic(io->meta_page);
+ cp->header.magic = cpu_to_le32(R5LOG_MAGIC);
+ cp->header.type = cpu_to_le32(R5LOG_TYPE_CHECKPOINT);
+ cp->header.seq = cpu_to_le64(log->seq - 1);
+ if (log->log_start == log->first_block)
+ cp->header.position = cpu_to_le64(log->last_block - 1);
+ else
+ cp->header.position = cpu_to_le64(log->log_start - 1);
+ cp->header.meta_size = cpu_to_le32(sizeof(struct r5l_checkpoint_block));
+ cp->cp_sec = cpu_to_le64(now.tv_sec);
+ cp->cp_nsec = cpu_to_le64(now.tv_nsec);
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false), cp,
+ log->block_size, false);
+ cp->header.checksum = cpu_to_le32(crc);
+ kunmap_atomic(cp);
+
+ list_add_tail(&task.list, &io->tasks);
+
+ r5l_submit_bio(log, WRITE_FLUSH_FUA, io->meta_bio);
+ r5l_put_ioctl(io);
+
+ r5l_put_reserve(log, &rsv);
+ *next_seq = log->seq;
+ *next_sec = BLOCK_SECTOR(log, log->log_start);
+
+ mutex_unlock(&log->io_mutex);
+
+ wait_for_completion_io(&cp_complete);
+ return 0;
+}
+
+static void r5l_discard_blocks(struct r5l_log *log, u64 start, u64 end)
+{
+ if (!log->do_discard)
+ return;
+ if (start < end) {
+ blkdev_issue_discard(log->bdev,
+ BLOCK_SECTOR(log, start) + log->rdev->data_offset,
+ BLOCK_SECTOR(log, end - start), GFP_NOIO, 0);
+ } else {
+ blkdev_issue_discard(log->bdev,
+ BLOCK_SECTOR(log, start) + log->rdev->data_offset,
+ BLOCK_SECTOR(log, log->last_block - start),
+ GFP_NOIO, 0);
+ blkdev_issue_discard(log->bdev,
+ BLOCK_SECTOR(log, log->first_block) +
+ log->rdev->data_offset,
+ BLOCK_SECTOR(log, end - log->first_block),
+ GFP_NOIO, 0);
+ }
+}
+
+static int r5l_write_super(struct r5l_log *log, u64 seq, sector_t cp)
+{
+ struct r5c_cache *cache = log->cache;
+ struct r5l_super_block *sb_blk;
+ u32 crc;
+
+ if (seq == log->last_cp_seq)
+ return 0;
+
+ mutex_lock(&log->io_mutex);
+ r5l_discard_blocks(log, log->last_checkpoint, cp);
+
+ log->last_cp_seq = seq;
+ log->last_checkpoint = cp >> log->block_sector_shift;
+
+ clear_bit(RECLAIM_DISK, &cache->reclaim_reason);
+ if (r5l_free_space(log) > log->high_watermark)
+ clear_bit(RECLAIM_DISK_BACKGROUND, &cache->reclaim_reason);
+
+ mutex_unlock(&log->io_mutex);
+
+ sb_blk = kmap_atomic(log->super_page);
+ sb_blk->header.seq = cpu_to_le64(seq);
+ sb_blk->last_checkpoint = cpu_to_le64(cp);
+ sb_blk->header.checksum = 0;
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false),
+ sb_blk, log->block_size, false);
+ sb_blk->header.checksum = cpu_to_le32(crc);
+ kunmap_atomic(sb_blk);
+
+ if (!sync_page_io(log->rdev, 0, log->block_size, log->super_page,
+ WRITE_FUA, false))
+ return -EIO;
+
+ return 0;
+}
+
+static void r5c_lock_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ unsigned long *flags)
+{
+ spinlock_t *lock;
+
+ lock = &cache->stripe_locks[hash_ptr(stripe, STRIPE_LOCK_BITS)];
+ spin_lock_irqsave(lock, *flags);
+}
+
+static void r5c_unlock_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ unsigned long *flags)
+{
+ spinlock_t *lock;
+
+ lock = &cache->stripe_locks[hash_ptr(stripe, STRIPE_LOCK_BITS)];
+ spin_unlock_irqrestore(lock, *flags);
+}
+
+static wait_queue_head_t *r5c_stripe_waitq(struct r5c_cache *cache,
+ struct r5c_stripe *stripe)
+{
+ return &cache->stripe_waitq[hash_ptr(stripe, STRIPE_LOCK_BITS)];
+}
+
+static void r5c_stripe_wait_state(struct r5c_cache *cache,
+ struct r5c_stripe *stripe, int state)
+{
+ wait_event(*r5c_stripe_waitq(cache, stripe), stripe->state >= state);
+}
+
+static struct page *r5c_get_page(struct r5c_cache *cache)
+{
+ struct page *page;
+again:
+ spin_lock_irq(&cache->pool_lock);
+ if (!list_empty(&cache->page_pool)) {
+ page = list_first_entry(&cache->page_pool,
+ struct page, lru);
+ list_del_init(&page->lru);
+ cache->free_pages--;
+ if (cache->free_pages < cache->low_watermark)
+ r5c_wake_reclaimer(cache,
+ RECLAIM_MEM_BACKGROUND);
+ }
+ spin_unlock_irq(&cache->pool_lock);
+ if (page)
+ return page;
+ r5c_wake_wait_reclaimer(cache, RECLAIM_MEM);
+ goto again;
+}
+
+static void r5c_put_pages(struct r5c_cache *cache,
+ struct page *pages[], int size)
+{
+ unsigned long flags;
+ int i;
+
+ spin_lock_irqsave(&cache->pool_lock, flags);
+ for (i = 0; i < size; i++) {
+ if (pages[i])
+ list_add(&pages[i]->lru, &cache->page_pool);
+ }
+ cache->free_pages += size;
+ if (cache->free_pages >= cache->high_watermark)
+ clear_bit(RECLAIM_MEM_BACKGROUND, &cache->reclaim_reason);
+ clear_bit(RECLAIM_MEM, &cache->reclaim_reason);
+ spin_unlock_irqrestore(&cache->pool_lock, flags);
+
+ wake_up_all(&cache->reclaim_wait);
+}
+
+static struct r5c_stripe *
+r5c_search_stripe(struct r5c_cache *cache, u64 stripe_index)
+{
+ struct r5c_stripe *stripe;
+
+ spin_lock_irq(&cache->tree_lock);
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ spin_unlock_irq(&cache->tree_lock);
+ return stripe;
+}
+
+static void r5c_put_stripe(struct r5c_stripe *stripe);
+static struct r5c_stripe *
+r5c_get_stripe(struct r5c_cache *cache, u64 stripe_index)
+{
+ struct r5c_stripe *stripe;
+
+again:
+ spin_lock_irq(&cache->tree_lock);
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ if (stripe)
+ atomic_inc(&stripe->ref);
+ spin_unlock_irq(&cache->tree_lock);
+
+ if (!stripe)
+ return NULL;
+ /* The stripe is being reclaimed, wait reclaim finish */
+ if (stripe->state >= STRIPE_FROZEN) {
+ r5c_stripe_wait_state(cache, stripe, STRIPE_DEAD);
+ r5c_put_stripe(stripe);
+ goto again;
+ }
+ return stripe;
+}
+
+static struct r5c_stripe *
+r5c_create_get_stripe(struct r5c_cache *cache, u64 stripe_index)
+{
+ struct r5c_stripe *stripe, *new;
+ int error;
+
+again:
+ spin_lock_irq(&cache->tree_lock);
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ if (stripe)
+ atomic_inc(&stripe->ref);
+ spin_unlock_irq(&cache->tree_lock);
+
+ if (stripe)
+ goto has_stripe;
+
+ new = kzalloc(sizeof(struct r5c_stripe) + sizeof(struct page *) *
+ STRIPE_DATA_PAGES(cache), GFP_NOIO);
+ new->raid_index = stripe_index;
+ atomic_set(&new->ref, 2);
+ new->state = STRIPE_RUNNING;
+ new->cache = cache;
+ INIT_LIST_HEAD(&new->io_ranges);
+ INIT_LIST_HEAD(&new->stripes);
+ INIT_LIST_HEAD(&new->lru);
+
+ error = radix_tree_preload(GFP_NOIO);
+ if (error) {
+ kfree(new);
+ return NULL;
+ }
+
+ spin_lock_irq(&cache->tree_lock);
+ if (radix_tree_insert(&cache->stripe_tree, stripe_index, new)) {
+ stripe = radix_tree_lookup(&cache->stripe_tree, stripe_index);
+ atomic_inc(&stripe->ref);
+ kfree(new);
+ } else
+ stripe = new;
+ spin_unlock_irq(&cache->tree_lock);
+
+ radix_tree_preload_end();
+
+has_stripe:
+ /* The stripe is being reclaimed, wait reclaim finish */
+ if (stripe->state >= STRIPE_FROZEN) {
+ r5c_stripe_wait_state(cache, stripe, STRIPE_DEAD);
+ r5c_put_stripe(stripe);
+ goto again;
+ }
+
+ return stripe;
+}
+
+static void r5c_put_stripe(struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+ int ref = atomic_dec_return(&stripe->ref);
+ unsigned long flags, flags2;
+
+ /* the stripe is freezing, might wait for ref */
+ if (ref == 1)
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+
+ if (ref)
+ return;
+
+ spin_lock_irqsave(&cache->tree_lock, flags);
+ radix_tree_delete(&cache->stripe_tree, stripe->raid_index);
+
+ r5c_lock_stripe(cache, stripe, &flags2);
+ while (!list_empty(&stripe->io_ranges)) {
+ range = list_first_entry(&stripe->io_ranges,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ list_del(&range->log_sibling);
+
+ kfree(range);
+ }
+ r5c_put_pages(cache, stripe->data_pages, STRIPE_DATA_PAGES(cache));
+ BUG_ON(stripe->parity_pages);
+
+ r5c_unlock_stripe(cache, stripe, &flags2);
+
+ spin_unlock_irqrestore(&cache->tree_lock, flags);
+
+ kfree(stripe);
+}
+
+/* must hold cache->tree_lock */
+static bool r5c_freeze_stripe(struct r5c_cache *cache,
+ struct r5c_stripe *stripe, bool blocking)
+{
+ if (atomic_read(&stripe->ref) == 1) {
+ stripe->state = STRIPE_FROZEN;
+ return true;
+ }
+ if (!blocking)
+ return false;
+ /* Make sure no IO running in stripe */
+ wait_event_lock_irq(*r5c_stripe_waitq(cache, stripe),
+ atomic_read(&stripe->ref) == 1, cache->tree_lock);
+ stripe->state = STRIPE_FROZEN;
+ return true;
+}
+
+static void r5c_bio_task_end(struct r5l_task *task)
+{
+ struct r5c_io_range *range = task->private;
+ struct r5c_stripe *stripe = range->stripe;
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *tmp;
+ unsigned long flags, flags2;
+
+ range->seq = task->seq;
+ range->meta_start = task->meta_start;
+ range->data_start = task->data_start;
+ kfree(task);
+
+ spin_lock_irqsave(&cache->tree_lock, flags);
+
+ if (list_empty(&cache->log_list)) {
+ list_add_tail(&range->log_sibling, &cache->log_list);
+ goto out;
+ }
+ /* generally tmp will be the last one */
+ list_for_each_entry_reverse(tmp, &cache->log_list, log_sibling) {
+ /* later range has bigger seq and meta_start than previous range */
+ if (range->seq >= tmp->seq)
+ break;
+ }
+ list_add_tail(&range->log_sibling, &tmp->log_sibling);
+out:
+ r5c_lock_stripe(cache, stripe, &flags2);
+ if (stripe->existing_pages == STRIPE_DATA_PAGES(cache) &&
+ list_empty(&stripe->lru)) {
+ list_add_tail(&stripe->lru, &cache->full_stripes);
+ if (!cache->full_stripe_ready_time)
+ cache->full_stripe_ready_time = jiffies;
+ }
+ r5c_unlock_stripe(cache, stripe, &flags2);
+
+ spin_unlock_irqrestore(&cache->tree_lock, flags);
+
+ r5c_put_stripe(stripe);
+
+ bio_endio_nodec(range->bio, 0);
+}
+
+static void r5c_copy_bio(struct bio *bio, struct page *pages[], bool tobio)
+{
+ struct bio *src = bio;
+ struct bvec_iter iter;
+ struct bio_vec bv;
+ void *src_p, *dst_p;
+ unsigned page_index = 0, page_offset = 0;
+ unsigned bytes;
+
+ iter = src->bi_iter;
+
+ while (1) {
+ if (!iter.bi_size) {
+ src = src->bi_next;
+ if (!src)
+ break;
+
+ iter = src->bi_iter;
+ }
+
+ if (page_offset == PAGE_SIZE) {
+ page_index++;
+ page_offset = 0;
+ }
+
+ bv = bio_iter_iovec(src, iter);
+
+ bytes = min_t(unsigned int, bv.bv_len, PAGE_SIZE - page_offset);
+
+ src_p = kmap_atomic(bv.bv_page);
+ dst_p = kmap_atomic(pages[page_index]);
+
+ if (tobio) {
+ memcpy(src_p + bv.bv_offset,
+ dst_p + page_offset, bytes);
+ } else {
+ memcpy(dst_p + page_offset,
+ src_p + bv.bv_offset, bytes);
+ }
+
+ kunmap_atomic(dst_p);
+ kunmap_atomic(src_p);
+
+ bio_advance_iter(src, &iter, bytes);
+ page_offset += bytes;
+ }
+}
+
+static void r5c_write_bio(struct r5c_cache *cache, struct bio *bio)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *io_range;
+ unsigned long index;
+ unsigned int offset;
+ int i, new_pages = 0;
+ unsigned long flags;
+
+ /* Doesn't support discard */
+ if (bio->bi_rw & REQ_DISCARD) {
+ bio_endio(bio, 0);
+ return;
+ }
+ if (bio->bi_iter.bi_size == 0) {
+ BUG_ON(!(bio->bi_rw & REQ_FLUSH));
+ r5l_queue_empty_flush_bio(&cache->log, bio);
+ return;
+ }
+
+ STRIPE_INDEX_OFFSET(cache, bio->bi_iter.bi_sector, index, offset);
+
+ stripe = r5c_create_get_stripe(cache, index);
+
+ io_range = kmalloc(sizeof(struct r5c_io_range), GFP_NOIO);
+ io_range->bio = bio;
+ io_range->raid_start = bio->bi_iter.bi_sector;
+ io_range->data_sectors = bio_sectors(bio);
+ io_range->stripe = stripe;
+
+ /* FIXME: read can get garbage data here */
+ offset >>= PAGE_SECTOR_SHIFT;
+ for (i = offset; i < offset + (bio_sectors(bio) >> PAGE_SECTOR_SHIFT);
+ i++) {
+ if (stripe->data_pages[i])
+ continue;
+ stripe->data_pages[i] = r5c_get_page(cache);
+ new_pages++;
+ }
+
+ r5c_lock_stripe(cache, stripe, &flags);
+ list_add_tail(&io_range->stripe_sibling, &stripe->io_ranges);
+ stripe->existing_pages += new_pages;
+ r5c_unlock_stripe(cache, stripe, &flags);
+
+ r5c_copy_bio(bio, &stripe->data_pages[offset], false);
+
+ r5l_queue_bio(&cache->log, bio, r5c_bio_task_end, io_range);
+}
+
+static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
+{
+ struct r5c_stripe *stripe;
+ u64 stripe_index;
+ int offset;
+ u64 start, end, tmp;
+ struct bio *split;
+
+ STRIPE_INDEX_OFFSET(cache, bio->bi_iter.bi_sector, stripe_index, offset);
+
+ stripe = r5c_get_stripe(cache, stripe_index);
+ if (!stripe) {
+ raid5_make_request(cache->mddev, bio);
+ return;
+ }
+
+ start = offset >> PAGE_SECTOR_SHIFT;
+ end = start + (bio_sectors(bio) >> PAGE_SECTOR_SHIFT);
+
+ while (start < end) {
+ if (stripe->data_pages[start]) {
+ tmp = start;
+ while (tmp < end && stripe->data_pages[tmp])
+ tmp++;
+ if (tmp < end) {
+ split = bio_split(bio,
+ (tmp - start) << PAGE_SECTOR_SHIFT,
+ GFP_NOIO, NULL);
+ bio_chain(split, bio);
+ } else /* all in cache */
+ split = bio;
+
+ r5c_copy_bio(split, &stripe->data_pages[start], true);
+
+ bio_endio(split, 0);
+
+ start = tmp;
+ } else {
+ tmp = start;
+ while (tmp < end && !stripe->data_pages[tmp])
+ tmp++;
+ if (tmp < end) {
+ split = bio_split(bio,
+ (tmp - start) << PAGE_SECTOR_SHIFT,
+ GFP_NOIO, NULL);
+ bio_chain(split, bio);
+ } else
+ split = bio;
+
+ raid5_make_request(cache->mddev, split);
+
+ start = tmp;
+ }
+ }
+ r5c_put_stripe(stripe);
+}
+
+void r5c_handle_bio(struct r5c_cache *cache, struct bio *bio)
+{
+ if (bio_data_dir(bio) == READ)
+ r5c_read_bio(cache, bio);
+ else
+ r5c_write_bio(cache, bio);
+}
+
+static void r5c_flush_endio(struct bio *bio, int err)
+{
+ struct r5c_stripe *stripe = bio->bi_private;
+ struct r5c_cache *cache = stripe->cache;
+
+ if (atomic_dec_and_test(&stripe->pending_bios)) {
+ stripe->state = STRIPE_INRAID;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+ }
+
+ bio_put(bio);
+}
+
+static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ struct block_device *bdev, int start, int pages)
+{
+ sector_t base;
+ struct bio *bio;
+ int i;
+
+ i = 0;
+ base = STRIPE_RAID_SECTOR(cache, stripe) + (start << PAGE_SECTOR_SHIFT);
+ while (i < pages) {
+ bio = bio_kmalloc(GFP_NOIO, min_t(int, pages - i,
+ bio_get_nr_vecs(bdev)));
+ bio->bi_iter.bi_sector = base + (i << PAGE_SECTOR_SHIFT);
+ bio->bi_bdev = bdev;
+ bio->bi_private = stripe;
+ while (i < pages) {
+ if (!bio_add_page(bio, stripe->data_pages[i + start],
+ PAGE_SIZE, 0))
+ break;
+ i++;
+ }
+ bio->bi_end_io = r5c_flush_endio;
+ bio->bi_rw = WRITE;
+ atomic_inc(&stripe->pending_bios);
+ raid5_make_request(cache->mddev, bio);
+ }
+}
+
+static void r5c_put_stripe_dirty(struct r5c_cache *cache, struct r5c_stripe *stripe)
+{
+ if (!atomic_dec_return(&stripe->dirty_stripes)) {
+ stripe->state = STRIPE_PARITY_DONE;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+ }
+}
+
+static void r5c_flush_stripe(struct r5c_cache *cache, struct block_device *bdev,
+ struct r5c_stripe *stripe)
+{
+ unsigned long *stripe_bits;
+ int chunk_stripes;
+ int start;
+ int end = 0;
+ int i;
+
+ chunk_stripes = cache->chunk_size >> PAGE_SECTOR_SHIFT;
+ stripe_bits = kzalloc(BITS_TO_LONGS(chunk_stripes) * sizeof(long),
+ GFP_NOIO);
+ for (i = 0; i < STRIPE_DATA_PAGES(cache); i++) {
+ if (stripe->data_pages[i])
+ __set_bit(i % chunk_stripes, stripe_bits);
+ }
+ atomic_set(&stripe->dirty_stripes, bitmap_weight(stripe_bits,
+ chunk_stripes) + 1);
+ kfree(stripe_bits);
+
+ while (end < STRIPE_DATA_PAGES(cache)) {
+ while (end < STRIPE_DATA_PAGES(cache) && !stripe->data_pages[end])
+ end++;
+ if (end >= STRIPE_DATA_PAGES(cache))
+ break;
+ start = end;
+ while (end < STRIPE_DATA_PAGES(cache) && stripe->data_pages[end])
+ end++;
+ r5c_flush_one(cache, stripe, bdev, start, end - start);
+ }
+ r5c_put_stripe_dirty(cache, stripe);
+}
+
+static int r5c_select_front_stripes(struct r5c_cache *cache,
+ struct list_head *list, int count, bool blocking)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ int stripes = 0;
+
+ list_for_each_entry(range, &cache->log_list, log_sibling) {
+ stripe = range->stripe;
+ if (stripe->state >= STRIPE_FROZEN)
+ continue;
+
+ if (!r5c_freeze_stripe(cache, stripe, blocking))
+ continue;
+
+ list_move_tail(&stripe->lru, list);
+ stripes++;
+ if (stripes >= count)
+ break;
+ }
+ return stripes;
+}
+
+static int r5c_select_full_stripes(struct r5c_cache *cache,
+ struct list_head *list, int count, bool blocking)
+{
+ struct r5c_stripe *stripe, *tmp;
+ int stripes = 0;
+
+ list_for_each_entry_safe(stripe, tmp, &cache->full_stripes, lru) {
+ if (stripe->state >= STRIPE_FROZEN)
+ continue;
+ if (!r5c_freeze_stripe(cache, stripe, blocking))
+ continue;
+
+ list_move_tail(&stripe->lru, list);
+ stripes++;
+ if (stripes >= count)
+ break;
+ }
+ if (list_empty(&cache->full_stripes)) {
+ cache->full_stripe_ready_time = 0;
+ clear_bit(RECLAIM_MEM_FULL, &cache->reclaim_reason);
+ }
+ return stripes;
+}
+
+static void r5c_select_stripes(struct r5c_cache *cache, struct list_head *list)
+{
+ int stripes;
+ bool blocking;
+
+ /*
+ * generally select full stripe, if no disk space, select first stripe
+ */
+ spin_lock_irq(&cache->tree_lock);
+ /* Don't need stripe lock, as nobody is operating on the stripe */
+ if (test_bit(RECLAIM_DISK, &cache->reclaim_reason) ||
+ test_bit(RECLAIM_DISK_BACKGROUND, &cache->reclaim_reason)) {
+ blocking = test_bit(RECLAIM_DISK, &cache->reclaim_reason);
+ r5c_select_front_stripes(cache, list, RECLAIM_BATCH, blocking);
+ } else if (test_bit(RECLAIM_MEM, &cache->reclaim_reason) ||
+ test_bit(RECLAIM_MEM_BACKGROUND, &cache->reclaim_reason)) {
+ stripes = r5c_select_full_stripes(cache, list, RECLAIM_BATCH, false);
+ if (stripes < RECLAIM_BATCH)
+ r5c_select_front_stripes(cache, list, RECLAIM_BATCH, false);
+ } else if (test_bit(RECLAIM_MEM_FULL, &cache->reclaim_reason)){
+ r5c_select_full_stripes(cache, list, -1, false);
+ }
+
+ spin_unlock_irq(&cache->tree_lock);
+}
+
+static void r5c_disks_flush_end(struct bio *bio, int err)
+{
+ struct completion *io_complete = bio->bi_private;
+
+ complete(io_complete);
+ bio_put(bio);
+}
+
+static void r5c_flush_all_disks(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ struct bio *bi;
+ DECLARE_COMPLETION_ONSTACK(io_complete);
+
+ bi = bio_alloc_mddev(GFP_NOIO, 0, mddev);
+ bi->bi_end_io = r5c_disks_flush_end;
+ bi->bi_private = &io_complete;
+
+ /* If bio hasn't payload, this function will just flush all disks */
+ md_flush_request(mddev, bi);
+
+ wait_for_completion_io(&io_complete);
+}
+
+static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
+ struct list_head *stripe_list)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ u64 seq;
+ sector_t meta;
+ struct blk_plug plug;
+ struct block_device *md_bdev;
+
+ if (list_empty(stripe_list))
+ return;
+ md_bdev = blkdev_get_by_dev(cache->mddev->unit,
+ FMODE_READ|FMODE_WRITE, NULL);
+
+ blk_start_plug(&plug);
+ /* step 1: start write to raid */
+ list_for_each_entry(stripe, stripe_list, lru)
+ r5c_flush_stripe(cache, md_bdev, stripe);
+ blk_finish_plug(&plug);
+
+ /* step 2: wait parity write to cache */
+ list_for_each_entry_reverse(stripe, stripe_list, lru)
+ r5c_stripe_wait_state(cache, stripe, STRIPE_PARITY_DONE);
+
+ /* step 3: make sure data and parity settle down */
+ r5l_commit(&cache->log);
+
+ /* step 4: continue write to raid */
+ list_for_each_entry(stripe, stripe_list, lru) {
+ atomic_set(&stripe->dirty_stripes, 1);
+ while (!list_empty(&stripe->stripes)) {
+ struct stripe_head *sh;
+
+ sh = list_first_entry(&stripe->stripes,
+ struct stripe_head, stripe_list);
+ list_del(&sh->stripe_list);
+ set_bit(STRIPE_HANDLE, &sh->state);
+ if (!list_empty(&stripe->stripes))
+ atomic_inc(&stripe->dirty_stripes);
+ release_stripe(sh);
+ }
+ }
+
+ /* step 5: wait to make sure stripe data is in raid */
+ list_for_each_entry_reverse(stripe, stripe_list, lru)
+ r5c_stripe_wait_state(cache, stripe, STRIPE_INRAID);
+
+ blkdev_put(md_bdev, FMODE_READ|FMODE_WRITE);
+
+ /* step 6: flush raid disks */
+ r5c_flush_all_disks(cache);
+
+ /* step 7: add a checkpoint */
+ r5l_checkpoint(&cache->log, &seq, &meta);
+
+ /* step 8: mark stripe as dead */
+ while (!list_empty(stripe_list)) {
+ stripe = list_first_entry(stripe_list, struct r5c_stripe,
+ lru);
+ list_del_init(&stripe->lru);
+
+ stripe->state = STRIPE_DEAD;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+
+ r5c_put_stripe(stripe);
+ }
+ /* step 9: advance superblock checkpoint */
+ spin_lock_irq(&cache->tree_lock);
+ /* if no data, superblock records the next position of checkpoint */
+ if (!list_empty(&cache->log_list)) {
+ range = list_first_entry(&cache->log_list,
+ struct r5c_io_range, log_sibling);
+ /* can't cross checkpoint */
+ if (range->seq < seq) {
+ seq = range->seq;
+ meta = range->meta_start;
+ }
+ }
+ spin_unlock_irq(&cache->tree_lock);
+
+ r5l_write_super(&cache->log, seq, meta);
+}
+
+static void r5c_reclaim_thread(struct md_thread *thread)
+{
+ struct mddev *mddev = thread->mddev;
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ LIST_HEAD(stripe_list);
+
+ if (cache->full_stripe_ready_time && cache->full_stripe_ready_time +
+ FULL_STRIPE_MAX_AGE < jiffies)
+ set_bit(RECLAIM_MEM_FULL, &cache->reclaim_reason);
+ while (cache->reclaim_reason != 0) {
+ r5c_select_stripes(cache, &stripe_list);
+
+ r5c_reclaim_stripe_list(cache, &stripe_list);
+
+ wake_up_all(&cache->reclaim_wait);
+ }
+}
+
+static void r5c_wake_reclaimer(struct r5c_cache *cache, int reason)
+{
+ set_bit(reason, &cache->reclaim_reason);
+ md_wakeup_thread(cache->reclaim_thread);
+}
+
+static void r5c_wake_wait_reclaimer(struct r5c_cache *cache, int reason)
+{
+ r5c_wake_reclaimer(cache, reason);
+ wait_event(cache->reclaim_wait, !test_bit(reason,
+ &cache->reclaim_reason));
+}
+
+static void r5c_parity_task_end(struct r5l_task *task)
+{
+ struct stripe_head *sh = task->private;
+ struct r5conf *conf = sh->raid_conf;
+ struct r5c_cache *cache = conf->cache;
+ struct r5c_stripe *stripe = sh->stripe;
+
+ r5c_put_stripe_dirty(cache, stripe);
+ kfree(task);
+}
+
+/*
+ * we don't record parity range in cache->log_list, because after a success
+ * reclaim, parity always is discarded
+ */
+int r5c_write_parity(struct r5c_cache *cache, struct stripe_head *sh)
+{
+ struct r5c_stripe *stripe;
+ u64 stripe_index;
+ int stripe_offset;
+ int parity_cnt;
+ unsigned long flags;
+
+ /* parity is already written */
+ if (sh->stripe) {
+ sh->stripe = NULL;
+ return -EAGAIN;
+ }
+ if (!test_bit(R5_Wantwrite, &sh->dev[sh->pd_idx].flags))
+ return -EAGAIN;
+
+ stripe_index = sh->sector;
+ stripe_offset = sector_div(stripe_index, cache->chunk_size);
+ stripe_offset >>= PAGE_SECTOR_SHIFT;
+
+ stripe = r5c_search_stripe(cache, stripe_index);
+ r5c_lock_stripe(cache, stripe, &flags);
+ list_add_tail(&sh->stripe_list, &stripe->stripes);
+ r5c_unlock_stripe(cache, stripe, &flags);
+ sh->stripe = stripe;
+
+ atomic_inc(&sh->count);
+
+ parity_cnt = !!(sh->pd_idx >= 0) + !!(sh->qd_idx >= 0);
+ BUG_ON(parity_cnt != cache->parity_disks);
+
+ if (cache->in_recovery) {
+ stripe_offset *= cache->parity_disks;
+
+ copy_highpage(sh->dev[sh->pd_idx].page,
+ stripe->parity_pages[stripe_offset]);
+ if (sh->qd_idx >= 0)
+ copy_highpage(sh->dev[sh->qd_idx].page,
+ stripe->parity_pages[stripe_offset + 1]);
+
+ if (!atomic_dec_return(&stripe->dirty_stripes)) {
+ stripe->state = STRIPE_PARITY_DONE;
+ wake_up_all(r5c_stripe_waitq(cache, stripe));
+ }
+ return 0;
+ }
+
+ r5l_queue_parity(&cache->log, sh->sector,
+ sh->dev[sh->pd_idx].page,
+ sh->qd_idx >= 0 ? sh->dev[sh->qd_idx].page : NULL,
+ r5c_parity_task_end, sh);
+ return 0;
+}
+
+struct r5c_load_ctx {
+ struct r5c_cache *cache;
+ struct list_head io_ranges;
+ struct list_head stripes_with_parity;
+ struct list_head stripes_without_parity;
+ unsigned long *stripe_bits;
+ int chunk_stripes;
+ atomic_t io_cnt;
+ wait_queue_head_t io_wq;
+};
+
+static int r5c_add_checkpoint(struct r5c_load_ctx *ctx, u64 seq)
+{
+ struct r5c_cache *cache = ctx->cache;
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+
+ while (!list_empty(&ctx->stripes_with_parity)) {
+ stripe = list_first_entry(&ctx->stripes_with_parity,
+ struct r5c_stripe, lru);
+ list_del(&stripe->lru);
+ /* essentially r5c_put_stripe */
+ radix_tree_delete(&cache->stripe_tree, stripe->raid_index);
+
+ while (!list_empty(&stripe->io_ranges)) {
+ range = list_first_entry(&stripe->io_ranges,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ list_del(&range->log_sibling);
+
+ kfree(range->checksum);
+ kfree(range);
+ }
+ while (!list_empty(&stripe->parity_list)) {
+ range = list_first_entry(&stripe->parity_list,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ kfree(range->checksum);
+ kfree(range);
+ }
+
+ kfree(stripe);
+ }
+ return 0;
+}
+
+static int r5c_add_data(struct r5c_load_ctx *ctx, u64 seq,
+ sector_t meta_start, sector_t data_start, sector_t raid_start,
+ ssize_t sectors, u32 *checksum)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ u64 index;
+ int offset;
+
+ STRIPE_INDEX_OFFSET(ctx->cache, raid_start, index, offset);
+
+ stripe = r5c_create_get_stripe(ctx->cache, index);
+
+ range = kmalloc(sizeof(struct r5c_io_range), GFP_NOIO);
+
+ range->seq = seq;
+ range->meta_start = meta_start;
+ range->data_start = data_start;
+ range->raid_start = raid_start;
+ range->data_sectors = sectors;
+ range->stripe = stripe;
+ range->checksum = checksum;
+
+ list_add_tail(&range->stripe_sibling, &stripe->io_ranges);
+ list_add_tail(&range->log_sibling, &ctx->io_ranges);
+
+ if (list_empty(&stripe->lru))
+ list_add_tail(&stripe->lru, &ctx->stripes_without_parity);
+
+ r5c_put_stripe(stripe);
+ return 0;
+}
+
+static int r5c_add_parity(struct r5c_load_ctx *ctx, u64 seq,
+ sector_t meta_start, sector_t data_start, sector_t stripe_sect,
+ ssize_t sectors, u32 *checksum)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ sector_t index = stripe_sect;
+
+ sector_div(index, ctx->cache->log.chunk_size);
+
+ stripe = r5c_get_stripe(ctx->cache, index);
+ BUG_ON(!stripe);
+
+ range = kmalloc(sizeof(struct r5c_io_range), GFP_NOIO);
+
+ range->seq = seq;
+ range->meta_start = meta_start;
+ range->data_start = data_start;
+ range->raid_start = stripe_sect;
+ range->data_sectors = sectors;
+ range->stripe = stripe;
+ range->checksum = checksum;
+
+ if (list_empty(&stripe->parity_list))
+ list_move_tail(&stripe->lru, &ctx->stripes_with_parity);
+ list_add_tail(&range->stripe_sibling, &stripe->parity_list);
+
+ r5c_put_stripe(stripe);
+ return 0;
+}
+
+static bool r5c_check_parity_full(struct r5c_load_ctx *ctx,
+ struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ int chunk_stripes = ctx->chunk_stripes;
+ unsigned long *stripe_bits = ctx->stripe_bits;
+ struct r5c_io_range *range;
+ sector_t index;
+ int offset, max;
+
+ memset(stripe_bits, 0, BITS_TO_LONGS(chunk_stripes) * sizeof(long));
+ list_for_each_entry(range, &stripe->io_ranges, stripe_sibling) {
+ STRIPE_INDEX_OFFSET(cache, range->raid_start, index, offset);
+ offset >>= PAGE_SECTOR_SHIFT;
+ max = (range->data_sectors >> PAGE_SECTOR_SHIFT) + offset;
+ for (; offset < max; offset++)
+ __set_bit(offset % chunk_stripes, stripe_bits);
+ }
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+
+ if ((range->data_sectors >> PAGE_SECTOR_SHIFT) ==
+ cache->parity_disks)
+ __clear_bit(offset, stripe_bits);
+ }
+
+ return bitmap_weight(stripe_bits, chunk_stripes) == 0;
+}
+
+static void r5c_free_parity_ranges(struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+
+ while (!list_empty(&stripe->parity_list)) {
+ range = list_first_entry(&stripe->parity_list,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ kfree(range->checksum);
+ kfree(range);
+ }
+ if (stripe->parity_pages) {
+ r5c_put_pages(cache, stripe->parity_pages,
+ STRIPE_PARITY_PAGES(cache));
+ kfree(stripe->parity_pages);
+ stripe->parity_pages = NULL;
+ }
+}
+
+/* If there is commit block after parity, all stripes with parity are already in
+ * raid. All data and parity can be ignored. Otherwise, if all stripes with
+ * parity have full parity, those stripes are in the process writting to raid,
+ * such stripes must be recovered. Otherwise, ignore all parity
+ */
+static int r5c_analyse_log(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe;
+ bool full = true;
+
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ if (r5c_check_parity_full(ctx, stripe))
+ continue;
+ full = false;
+ break;
+ }
+ if (!full) {
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ r5c_free_parity_ranges(stripe);
+ }
+ list_splice_tail_init(&ctx->stripes_with_parity,
+ &ctx->stripes_without_parity);
+ }
+ return 0;
+}
+
+static void r5l_fetch_endio(struct bio *bio, int err)
+{
+ struct r5c_load_ctx *ctx = bio->bi_private;
+
+ bio_put(bio);
+ if (atomic_dec_and_test(&ctx->io_cnt))
+ wake_up(&ctx->io_wq);
+}
+
+static int r5l_fetch_one_range(struct r5c_load_ctx *ctx,
+ struct r5c_stripe *stripe, struct page **pages,
+ ssize_t page_cnt, sector_t sec, bool data)
+{
+ struct r5l_log *log = &ctx->cache->log;
+ struct bio *bio, *split;
+ int page_index = 0;
+ int nvec;
+ int i;
+
+again:
+ nvec = min_t(int, page_cnt - page_index, bio_get_nr_vecs(log->bdev));
+ bio = bio_kmalloc(GFP_NOIO, nvec);
+ bio->bi_iter.bi_sector = sec;
+ bio->bi_end_io = r5l_fetch_endio;
+ bio->bi_private = ctx;
+ bio->bi_bdev = log->bdev;
+ atomic_inc(&ctx->io_cnt);
+
+ /* FIXME: we might use too many memory */
+ for (i = 0; i < nvec; i++) {
+ pages[i + page_index] = alloc_page(GFP_NOIO);
+ if (data)
+ stripe->existing_pages++;
+ ctx->cache->total_pages++;
+ bio_add_page(bio, pages[i + page_index], PAGE_SIZE, 0);
+ }
+
+ if (bio_end_sector(bio) > (BLOCK_SECTOR(log, log->last_block))) {
+ split = bio_split(bio, BLOCK_SECTOR(log, log->last_block) -
+ bio->bi_iter.bi_sector, GFP_NOIO, NULL);
+ bio_chain(split, bio);
+ bio->bi_iter.bi_sector = BLOCK_SECTOR(log, log->first_block);
+ r5l_submit_bio(log, READ, split);
+ }
+ sec = BLOCK_SECTOR(log, r5l_ring_add(log, sec >> log->block_sector_shift,
+ PAGE_BLOCKS(log, nvec)));
+ page_index += nvec;
+ r5l_submit_bio(log, READ, bio);
+
+ if (page_index < page_cnt)
+ goto again;
+ return 0;
+}
+
+static int r5l_fetch_one_stripe(struct r5c_load_ctx *ctx, struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+ u64 index;
+ int offset, max, start;
+
+ /* new data is at the tail */
+ list_for_each_entry_reverse(range, &stripe->io_ranges, stripe_sibling) {
+ STRIPE_INDEX_OFFSET(cache, range->raid_start, index, offset);
+ offset >>= PAGE_SECTOR_SHIFT;
+ max = (range->data_sectors >> PAGE_SECTOR_SHIFT) + offset;
+again:
+ while (offset < max && stripe->data_pages[offset])
+ offset++;
+ start = offset;
+ while (offset < max && !stripe->data_pages[offset])
+ offset++;
+ r5l_fetch_one_range(ctx, stripe, &stripe->data_pages[start],
+ offset - start,
+ range->data_start + (start << PAGE_SECTOR_SHIFT), true);
+ if (offset < max)
+ goto again;
+ }
+
+ if (list_empty(&stripe->parity_list))
+ return 0;
+
+ stripe->parity_pages = kmalloc(STRIPE_PARITY_PAGES(cache) *
+ sizeof(struct page *), GFP_NOIO | __GFP_ZERO);
+
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+ offset *= cache->parity_disks;
+
+ r5l_fetch_one_range(ctx, stripe, &stripe->parity_pages[offset],
+ cache->parity_disks, range->data_start, false);
+ }
+ return 0;
+}
+
+static int r5l_fetch_stripes(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe;
+ struct blk_plug plug;
+
+ blk_start_plug(&plug);
+ list_for_each_entry(stripe, &ctx->stripes_without_parity, lru) {
+ r5l_fetch_one_stripe(ctx, stripe);
+ }
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ r5l_fetch_one_stripe(ctx, stripe);
+ }
+ blk_finish_plug(&plug);
+
+ atomic_dec(&ctx->io_cnt);
+ wait_event(ctx->io_wq, atomic_read(&ctx->io_cnt) == 0);
+
+ return 0;
+}
+
+static u32 r5l_calculate_page_checksum(struct r5l_log *log, struct page *page)
+{
+ u32 csum;
+ void *addr = kmap_atomic(page);
+ csum = r5l_calculate_checksum(log, UUID_CHECKSUM(log, true),
+ addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+ return csum;
+}
+
+static int r5l_check_one_stripe_checksum(struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5l_log *log = &cache->log;
+ struct r5c_io_range *range;
+ struct page *page;
+ u64 index;
+ int offset;
+ int ret = 0;
+
+ /* FIXME: we currently ignore all data parity check */
+ list_for_each_entry(range, &stripe->io_ranges, stripe_sibling) {
+ kfree(range->checksum);
+ range->checksum = NULL;
+ }
+
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+ offset *= cache->parity_disks;
+
+ page = stripe->parity_pages[offset];
+
+ if (le32_to_cpu(range->checksum[0]) !=
+ r5l_calculate_page_checksum(log, page)) {
+ ret = -EINVAL;
+ break;
+ }
+ if (cache->parity_disks > 1) {
+ page = stripe->parity_pages[offset + 1];
+ if (le32_to_cpu(range->checksum[1]) !=
+ r5l_calculate_page_checksum(log, page)) {
+ ret = -EINVAL;
+ break;
+ }
+ }
+ }
+ r5c_free_parity_ranges(stripe);
+ return ret;
+}
+
+static int r5l_check_stripes_checksum(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe;
+ bool skip = false;
+
+ while (!list_empty(&ctx->stripes_without_parity)) {
+ stripe = list_first_entry(&ctx->stripes_without_parity,
+ struct r5c_stripe, lru);
+ list_del_init(&stripe->lru);
+
+ r5l_check_one_stripe_checksum(stripe);
+ }
+
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ if (skip)
+ r5c_free_parity_ranges(stripe);
+ else
+ skip = !!r5l_check_one_stripe_checksum(stripe);
+ }
+
+ /* If any parity checksum is wrong, we think the stripes are not hiting to raid */
+ if (skip) {
+ while (!list_empty(&ctx->stripes_with_parity)) {
+ stripe = list_first_entry(&ctx->stripes_with_parity,
+ struct r5c_stripe, lru);
+ list_del_init(&stripe->lru);
+ }
+ }
+ return 0;
+}
+
+static int r5c_recover_stripes(struct r5c_load_ctx *ctx)
+{
+ struct r5c_cache *cache = ctx->cache;
+
+ r5l_check_stripes_checksum(ctx);
+
+ list_splice_tail(&ctx->io_ranges, &ctx->cache->log_list);
+
+ if (list_empty(&ctx->stripes_with_parity))
+ return 0;
+
+ cache->in_recovery = 1;
+
+ r5c_reclaim_stripe_list(cache, &ctx->stripes_with_parity);
+
+ cache->in_recovery = 0;
+ return 0;
+}
+
+static void *r5l_read_meta_block(struct r5l_log *log, u64 block,
+ u64 expected_seq, struct page **retpage)
+{
+ struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
+ struct r5l_meta_header *header;
+ u32 crc, stored_crc;
+
+ if (!sync_page_io(log->rdev, BLOCK_SECTOR(log, block),
+ log->block_size, page, READ, false))
+ return NULL;
+
+ header = kmap(page);
+ stored_crc = le32_to_cpu(header->checksum);
+ header->checksum = 0;
+
+ if (le32_to_cpu(header->magic) != R5LOG_MAGIC ||
+ le64_to_cpu(header->seq) != expected_seq ||
+ le64_to_cpu(header->position) != block)
+ goto error;
+ if (le32_to_cpu(header->type) != R5LOG_TYPE_META &&
+ le32_to_cpu(header->type) != R5LOG_TYPE_CHECKPOINT)
+ goto error;
+
+ crc = r5l_calculate_checksum(log, UUID_CHECKSUM(log, false),
+ header, log->block_size, false);
+ if (stored_crc != crc)
+ goto error;
+
+ if (le32_to_cpu(header->meta_size) > log->block_size)
+ goto error;
+ *retpage = page;
+ return header;
+error:
+ kunmap(page);
+ __free_page(page);
+ return NULL;
+}
+
+static int r5l_load_log(struct r5l_log *log)
+{
+ u64 last_cp = log->last_checkpoint;
+ u64 last_seq = log->last_cp_seq;
+ u64 payload_block;
+ struct page *page = NULL;
+ struct r5l_meta_header *header;
+ void *meta;
+ struct r5l_meta_payload *payload;
+ struct r5c_load_ctx ctx;
+ int offset;
+
+ ctx.cache = log->cache;
+ INIT_LIST_HEAD(&ctx.io_ranges);
+ INIT_LIST_HEAD(&ctx.stripes_with_parity);
+ INIT_LIST_HEAD(&ctx.stripes_without_parity);
+ ctx.chunk_stripes = log->chunk_size >> PAGE_SECTOR_SHIFT;
+ ctx.stripe_bits = kmalloc(BITS_TO_LONGS(ctx.chunk_stripes) * sizeof(long),
+ GFP_KERNEL);
+ atomic_set(&ctx.io_cnt, 1);
+ init_waitqueue_head(&ctx.io_wq);
+
+again:
+ if (page) {
+ kunmap(page);
+ __free_page(page);
+ page = NULL;
+ }
+ header = r5l_read_meta_block(log, last_cp, last_seq, &page);
+ if (!header)
+ goto finish;
+ offset = sizeof(struct r5l_meta_header);
+
+ if (le32_to_cpu(header->type) == R5LOG_TYPE_CHECKPOINT) {
+ r5c_add_checkpoint(&ctx, last_seq);
+ last_cp = r5l_ring_add(log, last_cp, 1);
+ last_seq++;
+ goto again;
+ }
+
+ meta = header;
+ payload = meta + offset;
+ payload_block = r5l_ring_add(log, last_cp, 1);
+
+ while (offset < cpu_to_le32(header->meta_size)) {
+ u16 type = le16_to_cpu(payload->payload_type);
+ u16 entries = le32_to_cpu(payload->blocks) >>log->page_block_shift;
+ u32 *checksum;
+
+ checksum = kmalloc(sizeof(u32) * entries, GFP_KERNEL);
+ memcpy(checksum, payload->data_checksum, sizeof(u32) * entries);
+ if (type == R5LOG_PAYLOAD_DATA) {
+ r5c_add_data(&ctx, last_seq,
+ BLOCK_SECTOR(log, last_cp),
+ BLOCK_SECTOR(log, payload_block),
+ le64_to_cpu(payload->location),
+ entries << PAGE_SECTOR_SHIFT, checksum);
+ } else {
+ r5c_add_parity(&ctx, last_seq,
+ BLOCK_SECTOR(log, last_cp),
+ BLOCK_SECTOR(log, payload_block),
+ le64_to_cpu(payload->location),
+ entries << PAGE_SECTOR_SHIFT, checksum);
+ }
+ payload_block = r5l_ring_add(log, payload_block,
+ entries << log->page_block_shift);
+ offset += sizeof(struct r5l_meta_payload) +
+ entries * sizeof(u32);
+ }
+
+ last_seq++;
+ last_cp = payload_block;
+ goto again;
+finish:
+ if (page) {
+ kunmap(page);
+ __free_page(page);
+ }
+
+ r5c_analyse_log(&ctx);
+
+ r5l_fetch_stripes(&ctx);
+
+ log->seq = last_seq;
+ log->log_start = last_cp;
+ r5c_recover_stripes(&ctx);
+
+ kfree(ctx.stripe_bits);
+ return 0;
+}
+
+void r5l_fake_super(struct r5c_cache *cache, struct md_rdev *rdev)
+{
+ struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
+ struct r5l_super_block *sb_blk;
+ u32 crc;
+
+#define BLKSIZE 4096
+ sb_blk = kmap_atomic(page);
+ sb_blk->header.magic = cpu_to_le32(R5LOG_MAGIC);
+ sb_blk->header.type = cpu_to_le32(R5LOG_TYPE_SUPER);
+ sb_blk->header.seq = cpu_to_le32(0x111);
+ sb_blk->header.meta_size = cpu_to_le32(sizeof(*sb_blk));
+ sb_blk->version = cpu_to_le32(R5LOG_VERSION);
+ sb_blk->stripe_cache_size = cpu_to_le32(PAGE_SIZE);
+ sb_blk->block_size = cpu_to_le32(BLKSIZE);
+ sb_blk->total_blocks = (rdev->sectors * 512 / BLKSIZE) - 1;
+ sb_blk->stripe_data_size = cpu_to_le32(cache->stripe_data_size << 9);
+ sb_blk->chunk_size = cpu_to_le32(cache->chunk_size << 9);
+ sb_blk->stripe_size = cpu_to_le32(cache->stripe_size << 9);
+ sb_blk->parity_disks = cpu_to_le32(cache->parity_disks);
+
+ sb_blk->first_block = cpu_to_le64(1);
+ sb_blk->last_checkpoint = cpu_to_le64(1);
+ sb_blk->meta_checksum_type = R5LOG_CHECKSUM_CRC32;
+ sb_blk->data_checksum_type = R5LOG_CHECKSUM_CRC32;
+ memcpy(sb_blk->uuid, rdev->mddev->uuid, sizeof(sb_blk->uuid));
+
+ crc = crc32_le(~0, sb_blk, BLKSIZE);
+ crc = crc32_le(crc, sb_blk->uuid, sizeof(sb_blk->uuid));
+ sb_blk->header.checksum = crc;
+ kunmap_atomic(sb_blk);
+
+ sync_page_io(rdev, 0, BLKSIZE, page, WRITE, false);
+ __free_page(page);
+}
+
+static int r5l_read_super(struct r5l_log *log)
+{
+ struct md_rdev *rdev = log->rdev;
+ struct r5l_super_block *sb_blk;
+ struct page *page = log->super_page;
+ u32 crc, stored_crc;
+
+ if (!sync_page_io(rdev, 0, PAGE_SIZE, page, READ, false))
+ return -EIO;
+
+ sb_blk = kmap_atomic(page);
+
+ if (le32_to_cpu(sb_blk->version) != R5LOG_VERSION ||
+ le32_to_cpu(sb_blk->header.magic) != R5LOG_MAGIC ||
+ le32_to_cpu(sb_blk->header.type) != R5LOG_TYPE_SUPER ||
+ le64_to_cpu(sb_blk->header.position) != 0 ||
+ le32_to_cpu(sb_blk->header.meta_size) !=
+ sizeof(struct r5l_super_block))
+ goto error;
+
+ log->last_cp_seq = le64_to_cpu(sb_blk->header.seq);
+
+ log->block_size = le32_to_cpu(sb_blk->block_size);
+ log->block_sector_shift = ilog2(log->block_size >> 9);
+ log->page_block_shift = PAGE_SHIFT - ilog2(log->block_size);
+
+ /* Only support this stripe size right now */
+ if (le32_to_cpu(sb_blk->stripe_cache_size) != PAGE_SIZE)
+ goto error;
+ if (log->block_size > PAGE_SIZE)
+ goto error;
+
+ log->stripe_data_size = le32_to_cpu(sb_blk->stripe_data_size) >> 9;
+ log->chunk_size = le32_to_cpu(sb_blk->chunk_size) >> 9;
+ log->stripe_size = le32_to_cpu(sb_blk->stripe_size) >> 9;
+ log->parity_disks = le32_to_cpu(sb_blk->parity_disks);
+
+ if (sb_blk->meta_checksum_type >= R5LOG_CHECKSUM_NR ||
+ sb_blk->data_checksum_type >= R5LOG_CHECKSUM_NR)
+ goto error;
+ log->meta_checksum_type = sb_blk->meta_checksum_type;
+ log->data_checksum_type = sb_blk->data_checksum_type;
+
+ stored_crc = le32_to_cpu(sb_blk->header.checksum);
+ sb_blk->header.checksum = 0;
+ crc = r5l_calculate_checksum(log, ~0,
+ sb_blk, log->block_size, false);
+ crc = r5l_calculate_checksum(log, crc,
+ sb_blk->uuid, sizeof(sb_blk->uuid), false);
+ if (crc != stored_crc)
+ goto error;
+
+ if (memcmp(log->uuid, sb_blk->uuid, sizeof(log->uuid)))
+ goto error;
+
+ log->first_block = le64_to_cpu(sb_blk->first_block);
+ if (log->first_block != 1)
+ goto error;
+ log->total_blocks = le64_to_cpu(sb_blk->total_blocks);
+ log->last_block = log->first_block + log->total_blocks;
+ log->last_checkpoint = le64_to_cpu(sb_blk->last_checkpoint);
+ kunmap_atomic(sb_blk);
+
+ return 0;
+error:
+ kunmap_atomic(sb_blk);
+ return -EINVAL;
+}
+
+static int r5l_init_log(struct r5c_cache *cache)
+{
+ struct r5l_log *log;
+
+ log = &cache->log;
+
+ log->cache = cache;
+ log->bdev = cache->rdev->bdev;
+ log->rdev = cache->rdev;
+
+ log->do_discard = blk_queue_discard(bdev_get_queue(log->bdev));
+
+ log->super_page = alloc_page(GFP_KERNEL);
+ memcpy(log->uuid, cache->mddev->uuid, sizeof(log->uuid));
+
+ init_waitqueue_head(&log->space_waitq);
+ mutex_init(&log->io_mutex);
+
+ spin_lock_init(&log->io_list_lock);
+ INIT_LIST_HEAD(&log->running_ios);
+ init_waitqueue_head(&log->io_waitq);
+
+ if (r5l_read_super(log)) {
+ r5l_fake_super(cache, log->rdev);
+ if (r5l_read_super(log))
+ goto error;
+ }
+ log->uuid_checksum_data = r5l_calculate_checksum(log, ~0, log->uuid,
+ sizeof(log->uuid), true);
+ log->uuid_checksum_meta = r5l_calculate_checksum(log, ~0, log->uuid,
+ sizeof(log->uuid), false);
+
+ log->reserved_blocks = (cache->reserved_space >>
+ log->block_sector_shift) + 1;
+
+ if (log->stripe_data_size != cache->stripe_data_size ||
+ log->chunk_size != cache->chunk_size ||
+ log->stripe_size != cache->stripe_size ||
+ log->parity_disks != cache->parity_disks)
+ goto error;
+
+ r5l_load_log(log);
+
+ if (log->total_blocks * log->block_size / 10 < 1024 * 1024 * 1024)
+ log->low_watermark = log->total_blocks / 10;
+ else
+ log->low_watermark = (1024 * 1024 * 1024 / log->block_size);
+ log->high_watermark = log->low_watermark * 3 / 2;
+
+ return 0;
+error:
+ __free_page(log->super_page);
+ kfree(log);
+ return -EINVAL;
+}
+
+static void r5l_exit_log(struct r5l_log *log)
+{
+ r5l_commit(log);
+
+ __free_page(log->super_page);
+ kfree(log);
+}
+
+struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
+{
+ struct mddev *mddev = rdev->mddev;
+ struct r5c_cache *cache;
+ int i;
+
+ cache = kzalloc(sizeof(*cache), GFP_KERNEL);
+ cache->mddev = mddev;
+ cache->rdev = rdev;
+
+ spin_lock_init(&cache->tree_lock);
+ INIT_RADIX_TREE(&cache->stripe_tree, GFP_ATOMIC);
+ INIT_LIST_HEAD(&cache->log_list);
+ INIT_LIST_HEAD(&cache->full_stripes);
+
+ INIT_LIST_HEAD(&cache->page_pool);
+ spin_lock_init(&cache->pool_lock);
+ cache->max_pages = MAX_MEM >> PAGE_SHIFT;
+
+ cache->stripe_data_size = conf->chunk_sectors * (conf->raid_disks -
+ conf->max_degraded);
+ cache->chunk_size = conf->chunk_sectors;
+ cache->stripe_size = conf->chunk_sectors * conf->raid_disks;
+ cache->parity_disks = conf->max_degraded;
+
+ /* make sure we can add checkpoint */
+ cache->reserved_space = (STRIPE_PARITY_PAGES(cache) <<
+ PAGE_SECTOR_SHIFT) * RECLAIM_BATCH;
+
+ init_waitqueue_head(&cache->reclaim_wait);
+
+ for (i = 0; i < (1 << STRIPE_LOCK_BITS); i++) {
+ spin_lock_init(&cache->stripe_locks[i]);
+ init_waitqueue_head(&cache->stripe_waitq[i]);
+ }
+
+ r5l_init_log(cache);
+
+ while (cache->total_pages < cache->max_pages) {
+ struct page *page = alloc_page(GFP_KERNEL);
+
+ list_add(&page->lru, &cache->page_pool);
+ cache->free_pages++;
+ cache->total_pages++;
+ }
+
+ if (cache->max_pages / 10 < 1024)
+ cache->low_watermark = 1024;
+ else
+ cache->low_watermark = cache->max_pages / 10;
+ cache->high_watermark = cache->low_watermark * 3 /2;
+
+ cache->reclaim_thread = md_register_thread(r5c_reclaim_thread,
+ mddev, "reclaim");
+ cache->reclaim_thread->timeout = RECLAIM_TIMEOUT;
+ //FIXME: make sure total_pages < max_pages
+ return cache;
+}
+
+void r5c_exit_cache(struct r5c_cache *cache)
+{
+ struct r5c_stripe *stripe;
+ struct page *page, *tmp;
+ struct radix_tree_iter iter;
+ void **slot;
+
+ md_unregister_thread(&cache->reclaim_thread);
+ r5l_exit_log(&cache->log);
+
+ radix_tree_for_each_slot(slot, &cache->stripe_tree, &iter, 0) {
+ stripe = radix_tree_deref_slot(slot);
+ r5c_put_stripe(stripe);
+ }
+
+ BUG_ON(!list_empty(&cache->log_list));
+
+ list_for_each_entry_safe(page, tmp, &cache->page_pool, lru) {
+ list_del_init(&page->lru);
+ __free_page(page);
+ }
+}
+
+//sysfs max memory
+//background reclaim time
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cd2f96b..fbef599 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -409,7 +409,7 @@ static int release_stripe_list(struct r5conf *conf,
return count;
}
-static void release_stripe(struct stripe_head *sh)
+void release_stripe(struct stripe_head *sh)
{
struct r5conf *conf = sh->raid_conf;
unsigned long flags;
@@ -741,6 +741,11 @@ static void ops_run_io(struct stripe_head *sh, struct stripe_head_state *s)
might_sleep();
+ if (conf->cache) {
+ if (!r5c_write_parity(conf->cache, sh))
+ return;
+ }
+
for (i = disks; i--; ) {
int rw;
int replace_only = 0;
@@ -3125,6 +3130,7 @@ static void handle_stripe_clean_event(struct r5conf *conf,
WARN_ON(test_bit(R5_SkipCopy, &dev->flags));
WARN_ON(dev->page != dev->orig_page);
}
+
if (!discard_pending &&
test_bit(R5_Discard, &sh->dev[sh->pd_idx].flags)) {
clear_bit(R5_Discard, &sh->dev[sh->pd_idx].flags);
@@ -4179,13 +4185,20 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
int max;
unsigned int chunk_sectors = mddev->chunk_sectors;
unsigned int bio_sectors = bvm->bi_size >> 9;
+ struct r5conf *conf = mddev->private;
- if ((bvm->bi_rw & 1) == WRITE)
+ /* with cache, write must align within stripe */
+ if (((bvm->bi_rw & 1) == WRITE) && !conf->cache)
return biovec->bv_len; /* always allow writes to be mergeable */
if (mddev->new_chunk_sectors < mddev->chunk_sectors)
chunk_sectors = mddev->new_chunk_sectors;
max = (chunk_sectors - ((sector & (chunk_sectors - 1)) + bio_sectors)) << 9;
+ if (((bvm->bi_rw & 1) == WRITE) && conf->cache) {
+ chunk_sectors *= conf->raid_disks - conf->max_degraded;
+ max = (chunk_sectors - (sector_div(sector, chunk_sectors) +
+ bio_sectors)) << 9;
+ }
if (max < 0) max = 0;
if (max <= biovec->bv_len && bio_sectors == 0)
return biovec->bv_len;
@@ -4637,7 +4650,7 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
}
}
-static void make_request(struct mddev *mddev, struct bio * bi)
+void raid5_make_request(struct mddev *mddev, struct bio * bi)
{
struct r5conf *conf = mddev->private;
int dd_idx;
@@ -4785,7 +4798,8 @@ static void make_request(struct mddev *mddev, struct bio * bi)
}
set_bit(STRIPE_HANDLE, &sh->state);
clear_bit(STRIPE_DELAYED, &sh->state);
- if ((bi->bi_rw & REQ_SYNC) &&
+ /* there's no point to delay stripe with cache */
+ if (((bi->bi_rw & REQ_SYNC) || conf->cache) &&
!test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
atomic_inc(&conf->preread_active_stripes);
release_stripe_plug(mddev, sh);
@@ -4809,6 +4823,16 @@ static void make_request(struct mddev *mddev, struct bio * bi)
}
}
+static void make_request(struct mddev *mddev, struct bio *bi)
+{
+ struct r5conf *conf = mddev->private;
+
+ if (conf->cache)
+ r5c_handle_bio(conf->cache, bi);
+ else
+ raid5_make_request(mddev, bi);
+}
+
static sector_t raid5_size(struct mddev *mddev, sector_t sectors, int raid_disks);
static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr, int *skipped)
@@ -5740,6 +5764,9 @@ static void raid5_free_percpu(struct r5conf *conf)
static void free_conf(struct r5conf *conf)
{
+ if (conf->cache)
+ r5c_exit_cache(conf->cache);
+
free_thread_groups(conf);
shrink_stripes(conf);
raid5_free_percpu(conf);
@@ -6339,6 +6366,22 @@ static int run(struct mddev *mddev)
mddev->queue);
}
+ rdev_for_each(rdev, mddev) {
+ if (rdev->raid_disk < 0) {
+ char b[BDEVNAME_SIZE];
+ printk(KERN_INFO "using device %s as cache\n",
+ bdevname(rdev->bdev, b));
+ conf->cache = r5c_init_cache(conf, rdev);
+ }
+ }
+
+ if (conf->cache) {
+ /* Make sure all write do overwite */
+ if (mddev->queue)
+ blk_queue_logical_block_size(mddev->queue, STRIPE_SIZE);
+ conf->skip_copy = 1;
+ }
+
return 0;
abort:
md_unregister_thread(&mddev->thread);
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 983e18a..4c0d573 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -215,6 +215,8 @@ struct stripe_head {
spinlock_t stripe_lock;
int cpu;
struct r5worker_group *group;
+ struct r5c_stripe *stripe;
+ struct list_head stripe_list;
/**
* struct stripe_operations
* @target - STRIPE_OP_COMPUTE_BLK target
@@ -495,6 +497,7 @@ struct r5conf {
struct r5worker_group *worker_groups;
int group_cnt;
int worker_cnt_per_group;
+ struct r5c_cache *cache;
};
/*
@@ -560,4 +563,11 @@ static inline int algorithm_is_DDF(int layout)
extern void md_raid5_kick_device(struct r5conf *conf);
extern int raid5_set_cache_size(struct mddev *mddev, int size);
+
+void release_stripe(struct stripe_head *sh);
+void raid5_make_request(struct mddev *mddev, struct bio *bi);
+void r5c_handle_bio(struct r5c_cache *cache, struct bio *bi);
+int r5c_write_parity(struct r5c_cache *cache, struct stripe_head *sh);
+struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev);
+void r5c_exit_cache(struct r5c_cache *cache);
#endif
diff --git a/include/uapi/linux/raid/md_p.h b/include/uapi/linux/raid/md_p.h
index 49f4210..ed4e504 100644
--- a/include/uapi/linux/raid/md_p.h
+++ b/include/uapi/linux/raid/md_p.h
@@ -305,4 +305,74 @@ struct mdp_superblock_1 {
|MD_FEATURE_RECOVERY_BITMAP \
)
+/* all disk position of below struct start from rdev->start_offset */
+struct r5l_meta_header {
+ __le32 magic;
+ __le32 type;
+ __le32 checksum; /* checksum(metadata block + uuid) */
+ __le32 meta_size;
+ __le64 seq;
+ __le64 position; /* block number the meta is written */
+} __attribute__ ((__packed__));
+
+#define R5LOG_VERSION 0x1
+#define R5LOG_MAGIC 0x6433c509
+
+enum {
+ R5LOG_TYPE_META = 0,
+ R5LOG_TYPE_CHECKPOINT = 1,
+ R5LOG_TYPE_SUPER = 2,
+};
+
+struct r5l_super_block {
+ struct r5l_meta_header header;
+ __le32 version;
+ __le32 stripe_cache_size; /* bytes */
+ __le32 block_size; /* bytes */
+ __le32 stripe_data_size; /* bytes */
+ __le32 chunk_size; /* bytes */
+ __le32 stripe_size; /* bytes */
+ __le32 parity_disks;
+ __le64 total_blocks;
+ __le64 first_block;
+ __le64 last_checkpoint; /* block */
+ __le64 update_time_sec;
+ __le64 update_time_nsec;
+ __u8 meta_checksum_type;
+ __u8 data_checksum_type;
+ __u8 uuid[16];
+} __attribute__ ((__packed__));
+
+enum {
+ R5LOG_CHECKSUM_CRC32 = 0,
+ R5LOG_CHECKSUM_NR = 1,
+};
+
+struct r5l_meta_payload {
+ __le16 payload_type;
+ __le16 payload_flags;
+ __le32 blocks; /* for parity, blocks should be 1 or 2 pages */
+ /* For data, it's raid sector. For stripe, it's stripe sector */
+ __le64 location; /* sector */
+ __le32 data_checksum[]; /* checksum(data + uuid) */
+} __attribute__ ((__packed__));
+
+enum {
+ /* type */
+ R5LOG_PAYLOAD_DATA = 0,
+ R5LOG_PAYLOAD_PARITY = 1,
+ /* flags */
+ R5LOG_PAYLOAD_DISCARD = 1,
+};
+
+struct r5l_meta_block {
+ struct r5l_meta_header header;
+ struct r5l_meta_payload payloads[];
+} __attribute__ ((__packed__));
+
+struct r5l_checkpoint_block {
+ struct r5l_meta_header header;
+ __le64 cp_sec;
+ __le64 cp_nsec;
+} __attribute__ ((__packed__));
#endif
--
1.8.1
^ permalink raw reply related
* [PATCH RESEND] md-cluster: correct the num for comparison
From: Goldwyn Rodrigues @ 2015-04-21 16:25 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: GQJiang
Author: Guoqing Jiang <gqjiang@suse.com>
Since the node num of md-cluster is from zero, and
cinfo->slot_number represents the slot num of dlm,
no need to check for equality.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 96679b2..8ef9e3d 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -612,9 +612,9 @@ static int join(struct mddev *mddev, int nodes)
if (ret)
goto err;
wait_for_completion(&cinfo->completion);
- if (nodes <= cinfo->slot_number) {
- pr_err("md-cluster: Slot allotted(%d) greater than available slots(%d)", cinfo->slot_number - 1,
- nodes);
+ if (nodes < cinfo->slot_number) {
+ pr_err("md-cluster: Slot allotted(%d) is greater than available slots(%d).",
+ cinfo->slot_number, nodes);
ret = -ERANGE;
goto err;
}
^ permalink raw reply related
* Re: [PATCH 1/6] md-cluster: correct the num for comparison
From: Goldwyn Rodrigues @ 2015-04-21 16:31 UTC (permalink / raw)
To: NeilBrown; +Cc: GQJiang, linux-raid
In-Reply-To: <20150420114804.6e570389@notabene.brown>
Hi Neil,
On 04/19/2015 08:48 PM, NeilBrown wrote:
> On Tue, 14 Apr 2015 10:42:32 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>
>> Author: Guoqing Jiang <gqjiang@suse.com>
>>
>> Since the node num of md-cluster is from zero, and
>> cinfo->slot_number represents the slot num of dlm,
>> no need to check for equality.
>>
>> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> ---
>> drivers/md/md-cluster.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
>> index 96679b2..6bdc45e 100644
>> --- a/drivers/md/md-cluster.c
>> +++ b/drivers/md/md-cluster.c
>> @@ -612,7 +612,7 @@ static int join(struct mddev *mddev, int nodes)
>> if (ret)
>> goto err;
>> wait_for_completion(&cinfo->completion);
>> - if (nodes <= cinfo->slot_number) {
>> + if (nodes < cinfo->slot_number) {
>> pr_err("md-cluster: Slot allotted(%d) greater than available slots(%d)", cinfo->slot_number - 1,
>> nodes);
>> ret = -ERANGE;
>
> I'll apply this, but it would be nice to fix the error message too.
> If 'nodes' is 3 and 'slot_number' is 4, then the message would be
> md-cluster: Slot allotted(3) greater than available slots(3)
> which looks weird.
> (it should be "is greater", and there should be a "." at the end of the
> sentence).
>
Yes, I have corrected this and re-sent the patch.
Thanks,
--
Goldwyn
^ permalink raw reply
* mdadm.conf issue after updating system
From: Stefan Lamby @ 2015-04-22 9:56 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hello list.
I just updated my ubuntu system. The update provided a new kernel image also.
Here is what I got back:
update-initramfs: Generating /boot/initrd.img-3.13.0-49-generic
W: mdadm: the array /dev/md/kvm15:10 with UUID
c4540426:9c668fe2:479513f2:42d233b4
W: mdadm: is currently active, but it is not listed in mdadm.conf. if
W: mdadm: it is needed for boot, then YOUR SYSTEM IS NOW UNBOOTABLE!
W: mdadm: please inspect the output of /usr/share/mdadm/mkconf, compare
W: mdadm: it to /etc/mdadm/mdadm.conf, and make the necessary changes.
Is ist OK to just edit mdadm.conf and replace
ARRAY /dev/md/0 metadata=1.2 UUID=75079a2f:acb8c475:85f8ca43:0ad85c4c
name=kvm15:0
with:
ARRAY /dev/md/127 metadata=1.2 UUID=c4540426:9c668fe2:479513f2:42d233b4
name=kvm15:10
Or is there any other action recommended?
Thank you for your help.
Stefan
Here is some additonal information about the system:
root@kvm15:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4]
md127 : active raid10 sdc1[1] sdb1[5] sdd1[3] sda1[4]
3808330752 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
unused devices: <none>
root@kvm15:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1,8T 0 disk
└─sda1 8:1 0 1,8T 0 part
└─md127 9:127 0 3,6T 0 raid10
├─vg_raid10-home (dm-0) 252:0 0 1,2T 0 lvm /home
├─vg_raid10-root (dm-1) 252:1 0 93,1G 0 lvm /
├─vg_raid10-var (dm-2) 252:2 0 393,1G 0 lvm /var
├─vg_raid10-tmp (dm-3) 252:3 0 46,6G 0 lvm /tmp
└─vg_raid10-swap (dm-4) 252:4 0 23,3G 0 lvm [SWAP]
sdb 8:16 0 1,8T 0 disk
└─sdb1 8:17 0 1,8T 0 part
└─md127 9:127 0 3,6T 0 raid10
├─vg_raid10-home (dm-0) 252:0 0 1,2T 0 lvm /home
├─vg_raid10-root (dm-1) 252:1 0 93,1G 0 lvm /
├─vg_raid10-var (dm-2) 252:2 0 393,1G 0 lvm /var
├─vg_raid10-tmp (dm-3) 252:3 0 46,6G 0 lvm /tmp
└─vg_raid10-swap (dm-4) 252:4 0 23,3G 0 lvm [SWAP]
sdc 8:32 0 1,8T 0 disk
└─sdc1 8:33 0 1,8T 0 part
└─md127 9:127 0 3,6T 0 raid10
├─vg_raid10-home (dm-0) 252:0 0 1,2T 0 lvm /home
├─vg_raid10-root (dm-1) 252:1 0 93,1G 0 lvm /
├─vg_raid10-var (dm-2) 252:2 0 393,1G 0 lvm /var
├─vg_raid10-tmp (dm-3) 252:3 0 46,6G 0 lvm /tmp
└─vg_raid10-swap (dm-4) 252:4 0 23,3G 0 lvm [SWAP]
sdd 8:48 0 1,8T 0 disk
└─sdd1 8:49 0 1,8T 0 part
└─md127 9:127 0 3,6T 0 raid10
├─vg_raid10-home (dm-0) 252:0 0 1,2T 0 lvm /home
├─vg_raid10-root (dm-1) 252:1 0 93,1G 0 lvm /
├─vg_raid10-var (dm-2) 252:2 0 393,1G 0 lvm /var
├─vg_raid10-tmp (dm-3) 252:3 0 46,6G 0 lvm /tmp
└─vg_raid10-swap (dm-4) 252:4 0 23,3G 0 lvm [SWAP]
sr0 11:0 1 3,7G 0 rom
Gerät boot. Anfang Ende Blöcke Id System
/dev/sda1 * 98435072 3907028991 1904296960 fd Linux raid autodetect
root@kvm15:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=75079a2f:acb8c475:85f8ca43:0ad85c4c
name=kvm15:0
# This file was auto-generated on Tue, 17 Feb 2015 15:57:16 +0100
# by mkconf $Id$
root@kvm15:~# mdadm --detail /dev/md/kvm15\:10
/dev/md/kvm15:10:
Version : 1.2
Creation Time : Fri Mar 6 10:18:15 2015
Raid Level : raid10
Array Size : 3808330752 (3631.91 GiB 3899.73 GB)
Used Dev Size : 1904165376 (1815.95 GiB 1949.87 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Wed Apr 22 11:42:28 2015
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Name : kvm15:10 (local to host kvm15)
UUID : c4540426:9c668fe2:479513f2:42d233b4
Events : 14870
Number Major Minor RaidDevice State
5 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
4 8 1 2 active sync /dev/sda1
3 8 49 3 active sync /dev/sdd1
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Bisected, with rfc/patch - was Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: NeilBrown @ 2015-04-23 6:05 UTC (permalink / raw)
To: Azat Khuzhin, Christoph Hellwig
Cc: Kernel.org-Linux-RAID, Guoqing Jiang, Tejun Heo, Jan Kara,
Jens Axboe
In-Reply-To: <20150414171537.GH25394@azat>
[-- Attachment #1: Type: text/plain, Size: 15917 bytes --]
Hi Christoph et-al,
As Azat reports (see mail below with stack traces etc), it is now fairly easy
to trigger a WARNING followed by a BUG when you stop an 'md' array.
Bisection shows that this was introduced by:
commit c4db59d31e39ea067c32163ac961e9c80198fd37
Author: Christoph Hellwig <hch@lst.de>
fs: don't reassign dirty inodes to default_backing_dev_info
When an md array is stopped, a udev event causes udev to run blkid which
opens the md device. Due to the "interesting" semantics of md devices, this
creates a new device, at least temporarily. So we have device removal
immediately followed by creation of the same device.
And particularly: bdi removal immediately before bdi creation with same name.
Prior to the above commit, the bdi entry would be removed from sysfs when
bdi_unregister is called by del_gendisk() (bdi_unregister called
device_unregister(bdi->dev)).
Importantly del_gendisk() calls this *before* calling blk_unregister_region().
As soon as the latter is called, a new md device can be created simply by
opening the device node.
After the identified commit, the device_unregister call is delayed until
bdi_destroy(). I'm not sure how long this delay is, but it happens *after*
the blk_unregister_region() call.
This means there is a window during which the block device has been
unregistered, but the bdi still appears in sysfs.
If the md device is opened during this window (by blkid from udev), the
WARNING shown below results and the new bdi does not get registered properly.
The following RFC patch moves the device_unregister() call (and related
bdi_debug_unregister()) back into bdi_unregister().
This by itself is not sufficient as bdi_writeback_workfn() uses bdi->dev, and
that can run later (which was half the point of the patch I believe).
bdi_writeback_workfn() *only* uses bdi->dev to get a name for the worker
process, so I changed that to only use the bdi->dev name if bdi->dev was not
NULL.
Finally, to ensure no races, I flush the dwork if it is pending between
clearing bdi->dev and unregistering it. This ensure bdi_writeback_workfn()
doesn't try to find the name of a device which is just being freed.
I left the unregister code in bdi_destroy() as I'm not certain that
bdi_unregister() is always called (though I suspect it is) and the code cannot
hurt as it only runs if bdi->dev is not NULL. I'll remove that if that
would be more correct.
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 32a8bbd7a9ad..93f872ec434b 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -1093,9 +1093,13 @@ void bdi_writeback_workfn(struct work_struct *work)
struct bdi_writeback *wb = container_of(to_delayed_work(work),
struct bdi_writeback, dwork);
struct backing_dev_info *bdi = wb->bdi;
+ struct device *dev = bdi->dev;
long pages_written;
- set_worker_desc("flush-%s", dev_name(bdi->dev));
+ if (dev)
+ set_worker_desc("flush-%s", dev_name(dev));
+ else
+ set_worker_desc("flush-final");
current->flags |= PF_SWAPWRITE;
if (likely(!current_is_workqueue_rescuer() ||
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 6dc4580df2af..110af4534905 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -369,9 +369,17 @@ static void bdi_wb_shutdown(struct backing_dev_info *bdi)
*/
void bdi_unregister(struct backing_dev_info *bdi)
{
- if (WARN_ON_ONCE(!bdi->dev))
+ struct device *dev = bdi->dev;
+ if (WARN_ON_ONCE(!dev))
return;
+ bdi->dev = NULL;
+ if (delayed_work_pending(&bdi->wb.dwork))
+ /* The worker can access bdi->dev */
+ flush_work(&bdi->wb.dwork.work);
+ bdi_debug_unregister(bdi);
+ device_unregister(dev);
+
bdi_set_min_ratio(bdi, 0);
}
EXPORT_SYMBOL(bdi_unregister);
As mentioned the WARNING is followed shortly by a BUG.
This is because add_disk() doesn't check if bdi_register_dev() failed, and
proceeds to use bdi->dev. This is probably best fixed by the following patch:
diff --git a/block/genhd.c b/block/genhd.c
index 0a536dc05f3b..e351fc521053 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -612,6 +612,10 @@ void add_disk(struct gendisk *disk)
/* Register BDI before referencing it from bdev */
bdi = &disk->queue->backing_dev_info;
bdi_register_dev(bdi, disk_devt(disk));
+ if (!bdi->dev) {
+ WARN_ON(1);
+ return;
+ }
blk_register_region(disk_devt(disk), disk->minors, NULL,
exact_match, exact_lock, disk);
which Jens or Tejun might like to comment on.
If I can get an 'ack' or a proposal for a different approach I will happy
create a properly formatted patch for submission to -stable and to Linus.
And Azat: thanks for reporting!! sorry it took over a week to get to this.
NeilBrown
On Tue, 14 Apr 2015 20:15:37 +0300 Azat Khuzhin <a3at.mail@gmail.com> wrote:
> $ git describe
> v4.0-2620-gb79013b
>
> During setting up partitions with mdadm, mdadm hung, after attaching to mdadm with strace I got next:
>
> # pgrep mdadm | xargs strace -fp
> Process 27389 attached - interrupt to quit
> unlink("/dev/.tmp.md.27389:9:127") = 0
> mknod("/tmp/.tmp.md.27389:9:127", S_IFBLK|0600, makedev(9, 127)) = 0
> open("/tmp/.tmp.md.27389:9:127", O_RDWR|O_EXCL|O_DIRECT) <-- *hung*
>
> After, I looked into dmesg, and found this:
> [ 9627.630018] ------------[ cut here ]------------
> [ 9627.630029] WARNING: CPU: 18 PID: 3330 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x5a/0x70()
> [ 9627.630032] sysfs: cannot create duplicate filename '/devices/virtual/bdi/9:127'
> [ 9627.630033] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables nfsd nfs lockd grace sunrpc ipmi_devintf netconsole configfs loop hid_generic usbhid hid x86_pkg_temp_thermal coretemp ghash_clmulni_intel aesni_intel ioatdma ehci_pci aes_x86_64 iTCO_wdt iTCO_ve
> [ 9627.630074] CPU: 18 PID: 3330 Comm: mdadm Not tainted 4.0.0bl-azat-v6+ #1
> [ 9627.630076] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
> [ 9627.630077] 0000000000000000 ffffffff814e3fcc ffffffff813e590f ffff885f9bcd3808
> [ 9627.630079] ffffffff8104575c ffff885f96acb000 ffff885fa4b3e3c0 ffff885fa2fec780
> [ 9627.630081] ffff885fa4bc4000 0000000000000000 ffffffff810457d5 ffffffff814e5d78
> [ 9627.630083] Call Trace:
> [ 9627.630091] [<ffffffff813e590f>] ? dump_stack+0x40/0x50
> [ 9627.630096] [<ffffffff8104575c>] ? warn_slowpath_common+0x7c/0xb0
> [ 9627.630098] [<ffffffff810457d5>] ? warn_slowpath_fmt+0x45/0x50
> [ 9627.630100] [<ffffffff81185092>] ? kernfs_path+0x42/0x50
> [ 9627.630102] [<ffffffff811883da>] ? sysfs_warn_dup+0x5a/0x70
> [ 9627.630104] [<ffffffff8118846e>] ? sysfs_create_dir_ns+0x7e/0x90
> [ 9627.630108] [<ffffffff811d94ab>] ? kobject_add_internal+0x9b/0x2f0
> [ 9627.630109] [<ffffffff811d9af6>] ? kobject_add+0x66/0xb0
> [ 9627.630114] [<ffffffff812bb2e3>] ? device_add+0x263/0x620
> [ 9627.630116] [<ffffffff812bb8a8>] ? device_create_groups_vargs+0xe8/0x100
> [ 9627.630118] [<ffffffff812bb8d3>] ? device_create_vargs+0x13/0x20
> [ 9627.630124] [<ffffffff810ed128>] ? bdi_register+0x68/0x150
> [ 9627.630129] [<ffffffff811c535d>] ? add_disk+0x14d/0x4a0
> [ 9627.630132] [<ffffffff811c585f>] ? alloc_disk_node+0xaf/0x100
> [ 9627.630137] [<ffffffffa0252269>] ? md_alloc+0x1e9/0x350 [md_mod]
> [ 9627.630141] [<ffffffffa02523db>] ? md_probe+0xb/0x20 [md_mod]
> [ 9627.630143] [<ffffffff812c0654>] ? kobj_lookup+0x104/0x170
> [ 9627.630147] [<ffffffffa02523d0>] ? md_alloc+0x350/0x350 [md_mod]
> [ 9627.630149] [<ffffffff811c4da8>] ? get_gendisk+0x28/0xf0
> [ 9627.630153] [<ffffffff8115fb74>] ? __blkdev_get+0x114/0x3c0
> [ 9627.630156] [<ffffffff8115e590>] ? bdev_direct_access+0xa0/0xa0
> [ 9627.630158] [<ffffffff8115e5a0>] ? bdev_test+0x10/0x10
> [ 9627.630160] [<ffffffff8115fe58>] ? blkdev_get+0x38/0x310
> [ 9627.630162] [<ffffffff81160170>] ? blkdev_get_by_dev+0x40/0x40
> [ 9627.630167] [<ffffffff8112b3d3>] ? do_dentry_open.isra.16+0x153/0x320
> [ 9627.630170] [<ffffffff811380f3>] ? do_last.isra.51+0x323/0xd50
> [ 9627.630172] [<ffffffff8111f5b3>] ? kmem_cache_alloc+0x123/0x130
> [ 9627.630174] [<ffffffff8113a97f>] ? path_openat+0x7f/0x610
> [ 9627.630177] [<ffffffff810f7480>] ? tlb_flush_mmu_free+0x30/0x50
> [ 9627.630180] [<ffffffff810fe800>] ? unmap_region+0xb0/0xf0
> [ 9627.630182] [<ffffffff8113bb3b>] ? do_filp_open+0x2b/0x90
> [ 9627.630187] [<ffffffff811472ec>] ? __alloc_fd+0x7c/0x120
> [ 9627.630189] [<ffffffff8112c531>] ? do_sys_open+0x121/0x210
> [ 9627.630193] [<ffffffff813ea097>] ? system_call_fastpath+0x12/0x6a
> [ 9627.630195] ---[ end trace b7a3e9c6f05c2666 ]---
> [ 9627.630196] ------------[ cut here ]------------
> [ 9627.630198] WARNING: CPU: 18 PID: 3330 at lib/kobject.c:240 kobject_add_internal+0x274/0x2f0()
> [ 9627.630200] kobject_add_internal failed for 9:127 with -EEXIST, don't try to register things with the same name in the same directory.
> [ 9627.630201] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables nfsd nfs lockd grace sunrpc ipmi_devintf netconsole configfs loop hid_generic usbhid hid x86_pkg_temp_thermal coretemp ghash_clmulni_intel aesni_intel ioatdma ehci_pci aes_x86_64 iTCO_wdt iTCO_ve
> [ 9627.630223] CPU: 18 PID: 3330 Comm: mdadm Tainted: G W 4.0.0bl-azat-v6+ #1
> [ 9627.630224] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
> [ 9627.630225] 0000000000000000 ffffffff814f2c88 ffffffff813e590f ffff885f9bcd3858
> [ 9627.630227] ffffffff8104575c ffff885fa4bc4010 00000000ffffffef ffff885fa473f420
> [ 9627.630229] ffff885fa4bc4000 0000000000000000 ffffffff810457d5 ffffffff814f2e48
> [ 9627.630230] Call Trace:
> [ 9627.630233] [<ffffffff813e590f>] ? dump_stack+0x40/0x50
> [ 9627.630235] [<ffffffff8104575c>] ? warn_slowpath_common+0x7c/0xb0
> [ 9627.630236] [<ffffffff810457d5>] ? warn_slowpath_fmt+0x45/0x50
> [ 9627.630238] [<ffffffff8118846e>] ? sysfs_create_dir_ns+0x7e/0x90
> [ 9627.630240] [<ffffffff811d9684>] ? kobject_add_internal+0x274/0x2f0
> [ 9627.630242] [<ffffffff811d9af6>] ? kobject_add+0x66/0xb0
> [ 9627.630244] [<ffffffff812bb2e3>] ? device_add+0x263/0x620
> [ 9627.630245] [<ffffffff812bb8a8>] ? device_create_groups_vargs+0xe8/0x100
> [ 9627.630247] [<ffffffff812bb8d3>] ? device_create_vargs+0x13/0x20
> [ 9627.630250] [<ffffffff810ed128>] ? bdi_register+0x68/0x150
> [ 9627.630252] [<ffffffff811c535d>] ? add_disk+0x14d/0x4a0
> [ 9627.630255] [<ffffffff811c585f>] ? alloc_disk_node+0xaf/0x100
> [ 9627.630258] [<ffffffffa0252269>] ? md_alloc+0x1e9/0x350 [md_mod]
> [ 9627.630261] [<ffffffffa02523db>] ? md_probe+0xb/0x20 [md_mod]
> [ 9627.630262] [<ffffffff812c0654>] ? kobj_lookup+0x104/0x170
> [ 9627.630266] [<ffffffffa02523d0>] ? md_alloc+0x350/0x350 [md_mod]
> [ 9627.630268] [<ffffffff811c4da8>] ? get_gendisk+0x28/0xf0
> [ 9627.630270] [<ffffffff8115fb74>] ? __blkdev_get+0x114/0x3c0
> [ 9627.630272] [<ffffffff8115e590>] ? bdev_direct_access+0xa0/0xa0
> [ 9627.630274] [<ffffffff8115e5a0>] ? bdev_test+0x10/0x10
> [ 9627.630276] [<ffffffff8115fe58>] ? blkdev_get+0x38/0x310
> [ 9627.630278] [<ffffffff81160170>] ? blkdev_get_by_dev+0x40/0x40
> [ 9627.630280] [<ffffffff8112b3d3>] ? do_dentry_open.isra.16+0x153/0x320
> [ 9627.630282] [<ffffffff811380f3>] ? do_last.isra.51+0x323/0xd50
> [ 9627.630283] [<ffffffff8111f5b3>] ? kmem_cache_alloc+0x123/0x130
> [ 9627.630285] [<ffffffff8113a97f>] ? path_openat+0x7f/0x610
> [ 9627.630287] [<ffffffff810f7480>] ? tlb_flush_mmu_free+0x30/0x50
> [ 9627.630289] [<ffffffff810fe800>] ? unmap_region+0xb0/0xf0
> [ 9627.630291] [<ffffffff8113bb3b>] ? do_filp_open+0x2b/0x90
> [ 9627.630293] [<ffffffff811472ec>] ? __alloc_fd+0x7c/0x120
> [ 9627.630295] [<ffffffff8112c531>] ? do_sys_open+0x121/0x210
> [ 9627.630297] [<ffffffff813ea097>] ? system_call_fastpath+0x12/0x6a
> [ 9627.630298] ---[ end trace b7a3e9c6f05c2667 ]---
> [ 9627.630395] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040
> [ 9627.630430] IP: [<ffffffff8118869a>] sysfs_do_create_link_sd.isra.2+0x2a/0xb0
> [ 9627.630524] PGD 5fa2d03067 PUD 5f9d679067 PMD 0
> [ 9627.630550] Oops: 0000 [#1] SMP
> [ 9627.630624] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables nfsd nfs lockd grace sunrpc ipmi_devintf netconsole configfs loop hid_generic usbhid hid x86_pkg_temp_thermal coretemp ghash_clmulni_intel aesni_intel ioatdma ehci_pci aes_x86_64 iTCO_wdt iTCO_ve
> [ 9627.631073] CPU: 18 PID: 3330 Comm: mdadm Tainted: G W 4.0.0bl-azat-v6+ #1
> [ 9627.631090] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
> [ 9627.631109] task: ffff885fa4659f00 ti: ffff885f9bcd0000 task.ti: ffff885f9bcd0000
> [ 9627.631124] RIP: 0010:[<ffffffff8118869a>] [<ffffffff8118869a>] sysfs_do_create_link_sd.isra.2+0x2a/0xb0
> [ 9627.631162] RSP: 0018:ffff885f9bcd3a78 EFLAGS: 00010246
> [ 9627.631189] RAX: 000000000000e6e6 RBX: 0000000000000040 RCX: 00000000000000e6
> [ 9627.631219] RDX: ffffffff814e19d0 RSI: 0000000000000040 RDI: ffffffff81740d88
> [ 9627.631249] RBP: ffffffff814e19d0 R08: 0000000000017d60 R09: ffff88607fcd7d60
> [ 9627.631278] R10: ffff882fbf802400 R11: ffffea017e830e00 R12: 0000000000000001
> [ 9627.631308] R13: ffff885fa5a61ca8 R14: ffff885fa4bc3c70 R15: ffff885fa4bc3c00
> [ 9627.631338] FS: 00007f439e991700(0000) GS:ffff88607fcc0000(0000) knlGS:0000000000000000
> [ 9627.631383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 9627.631411] CR2: 0000000000000040 CR3: 0000005f9fe22000 CR4: 00000000001407e0
> [ 9627.631440] Stack:
> [ 9627.631460] ffff885fa4bc3c00 ffff885f9bf41428 ffff885fa4bc3c0c ffff885fa4bc3c80
> [ 9627.631515] ffff885fa4bc3c70 ffffffff811c53f3 ffff885fa4bc3c00 ffffffff00000018
> [ 9627.631569] 0090007f9bcd3b08 ffff885fa4bc3c00 0000000000000000 0000000000000001
> [ 9627.631680] Call Trace:
> [ 9627.631703] [<ffffffff811c53f3>] ? add_disk+0x1e3/0x4a0
> [ 9627.631733] [<ffffffffa0252269>] ? md_alloc+0x1e9/0x350 [md_mod]
> [ 9627.631763] [<ffffffffa02523db>] ? md_probe+0xb/0x20 [md_mod]
> [ 9627.631791] [<ffffffff812c0654>] ? kobj_lookup+0x104/0x170
> [ 9627.631820] [<ffffffffa02523d0>] ? md_alloc+0x350/0x350 [md_mod]
> [ 9627.631849] [<ffffffff811c4da8>] ? get_gendisk+0x28/0xf0
> [ 9627.631877] [<ffffffff8115fb74>] ? __blkdev_get+0x114/0x3c0
> [ 9627.631905] [<ffffffff8115e590>] ? bdev_direct_access+0xa0/0xa0
> [ 9627.631933] [<ffffffff8115e5a0>] ? bdev_test+0x10/0x10
> [ 9627.631961] [<ffffffff8115fe58>] ? blkdev_get+0x38/0x310
> [ 9627.631988] [<ffffffff81160170>] ? blkdev_get_by_dev+0x40/0x40
> [ 9627.632017] [<ffffffff8112b3d3>] ? do_dentry_open.isra.16+0x153/0x320
> [ 9627.632046] [<ffffffff811380f3>] ? do_last.isra.51+0x323/0xd50
> [ 9627.632075] [<ffffffff8111f5b3>] ? kmem_cache_alloc+0x123/0x130
> [ 9627.632103] [<ffffffff8113a97f>] ? path_openat+0x7f/0x610
> [ 9627.632131] [<ffffffff810f7480>] ? tlb_flush_mmu_free+0x30/0x50
> [ 9627.632159] [<ffffffff810fe800>] ? unmap_region+0xb0/0xf0
> [ 9627.632186] [<ffffffff8113bb3b>] ? do_filp_open+0x2b/0x90
> [ 9627.632215] [<ffffffff811472ec>] ? __alloc_fd+0x7c/0x120
> [ 9627.632242] [<ffffffff8112c531>] ? do_sys_open+0x121/0x210
> [ 9627.632270] [<ffffffff813ea097>] ? system_call_fastpath+0x12/0x6a
> [ 9627.632298] Code: 00 48 85 d2 74 73 48 85 ff 74 6e 41 56 41 55 49 89 fd 41 54 55 48 c7 c7 88 0d 74 81 53 48 89 f3 41 89 cc 48 89 d5 e8 76 15 26 00 <48> 8b 1b 48 85 db 74 08 48 89 df e8 f6 c9 ff ff 80 05 d7 86 5b
> [ 9627.632557] RIP [<ffffffff8118869a>] sysfs_do_create_link_sd.isra.2+0x2a/0xb0
> [ 9627.632604] RSP <ffff885f9bcd3a78>
> [ 9627.632627] CR2: 0000000000000040
> [ 9627.633014] ---[ end trace b7a3e9c6f05c2668 ]---
>
> Any assumptions?
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* [GIT PULL REQUEST] md updates for 4.1
From: NeilBrown @ 2015-04-23 6:14 UTC (permalink / raw)
To: Linus Torvalds
Cc: lkml, linux RAID, Eric Mei, Fengguang Wu, Goldwyn Rodrigues,
Guoqing Jiang, Heinz Mauelshagen, Lidong Zhong,
Markus Stockhausen, Shaohua Li, Stephen Rothwell
[-- Attachment #1: Type: text/plain, Size: 6467 bytes --]
Hi Linus,
more updates that usual this time. A few have performance impacts which
should mostly be positive, but RAID5 (in particular) can be very work-load
sensitive .... we'll have to wait and see.
More detailed highlights below.
Thanks,
NeilBrown
The following changes since commit 47d68979cc968535cb87f3e5f2e6a3533ea48fbd:
md/raid0: fix bug with chunksize not a power of 2. (2015-04-10 15:36:31 +1000)
are available in the git repository at:
git://neil.brown.name/md/ tags/md/4.1
for you to fetch changes up to 9ffc8f7cb9647b13dfe4d1ad0d5e1427bb8b46d6:
md/raid5: don't do chunk aligned read on degraded array. (2015-04-22 08:00:43 +1000)
----------------------------------------------------------------
md updates for 4.1
Highlights:
- "experimental" code for managing md/raid1 across a cluster using
DLM. Code is not ready for general use and triggers a WARNING if used.
However it is looking good and mostly done and having in mainline
will help co-ordinate development.
- RAID5/6 can now batch multiple (4K wide) stripe_heads so as to
handle a full (chunk wide) stripe as a single unit.
- RAID6 can now perform read-modify-write cycles which should
help performance on larger arrays: 6 or more devices.
- RAID5/6 stripe cache now grows and shrinks dynamically. The value
set is used as a minimum.
- Resync is now allowed to go a little faster than the 'mininum' when
there is competing IO. How much faster depends on the speed of the
devices, so the effective minimum should scale with device speed to
some extent.
----------------------------------------------------------------
Eric Mei (1):
md/raid5: don't do chunk aligned read on degraded array.
Goldwyn Rodrigues (31):
md-cluster: Design Documentation
Add number of nodes to bitmap structure for clustering
Create a separate module for clustering support
DLM lock and unlock functions
Introduce md_cluster_operations to handle cluster functions
Introduce md_cluster_info
Return MD_SB_CLUSTERED if mddev is clustered
Add node recovery callbacks
Use separate bitmaps for each nodes in the cluster
Lock bitmap while joining the cluster
Gather on-going resync information of other nodes
bitmap_create returns bitmap pointer
Copy set bits from another slot
Initiate recovery on node failure
Perform resync for cluster node failure
Communication Framework: Receiving
Communication Framework: Sending functions
metadata_update sends message to other nodes
Reload superblock if METADATA_UPDATED is received
Send RESYNCING while performing resync start/stop
Resync start/Finish actions
Suspend writes in RAID1 if within range
Read from the first device when an area is resyncing
Add new disk to clustered array
md: Fix stray --cluster-confirm crash
md: Fix bitmap offset calculations
md: Export and rename kick_rdev_from_array
md: Export and rename find_rdev_nr_rcu
md-cluster: remove capabilities
md: re-add a failed disk
md-cluster: re-add capabilities
Guoqing Jiang (1):
md-cluster: correct the num for comparison
Heinz Mauelshagen (1):
md raid0: access mddev->queue (request queue member) conditionally because it is not set when accessed from dm-raid
Markus Stockhausen (6):
md/raid6 algorithms: delta syndrome functions
md/raid6 algorithms: improve test program
md/raid6 algorithms: xor_syndrome() for generic int
md/raid6 algorithms: xor_syndrome() for SSE2
md/raid5: activate raid6 rmw feature
md/raid5: introduce configuration option rmw_level
NeilBrown (10):
md: fix error paths from bitmap_create.
md/bitmap: fix incorrect DIV_ROUND_UP usage.
Merge branch 'cluster' into for-next
md: don't require sync_min to be a multiple of chunk_size.
md: remove 'go_faster' option from ->sync_request()
md: allow resync to go faster when there is competing IO.
md/raid5: pass gfp_t arg to grow_one_stripe()
md/raid5: move max_nr_stripes management into grow_one_stripe and drop_one_stripe
md/raid5: change ->inactive_blocked to a bit-flag.
md/raid5: allow the stripe_cache to grow and shrink.
Stephen Rothwell (1):
md/bitmap: use sector_div for sector_t divisions
kbuild test robot (2):
md: recover_bitmaps() can be static
md/cluster: Communication Framework: fix semicolon.cocci warnings
shli@kernel.org (6):
raid5: use flex_array for scribble data
raid5: add a new flag to track if a stripe can be batched
raid5: track overwrite disk count
RAID5: batch adjacent full stripe write
raid5: handle io error of batch list
raid5: handle expansion/resync case with stripe batching
Documentation/md-cluster.txt | 176 ++++++++
crypto/async_tx/async_pq.c | 19 +-
drivers/md/Kconfig | 16 +
drivers/md/Makefile | 1 +
drivers/md/bitmap.c | 189 +++++++-
drivers/md/bitmap.h | 10 +-
drivers/md/md-cluster.c | 965 +++++++++++++++++++++++++++++++++++++++++
drivers/md/md-cluster.h | 29 ++
drivers/md/md.c | 382 +++++++++++++---
drivers/md/md.h | 26 +-
drivers/md/raid0.c | 48 +-
drivers/md/raid1.c | 29 +-
drivers/md/raid10.c | 8 +-
drivers/md/raid5.c | 826 +++++++++++++++++++++++++++++------
drivers/md/raid5.h | 59 ++-
include/linux/async_tx.h | 3 +
include/linux/raid/pq.h | 1 +
include/uapi/linux/raid/md_p.h | 7 +
include/uapi/linux/raid/md_u.h | 1 +
lib/raid6/algos.c | 41 +-
lib/raid6/altivec.uc | 1 +
lib/raid6/avx2.c | 3 +
lib/raid6/int.uc | 41 +-
lib/raid6/mmx.c | 2 +
lib/raid6/neon.c | 1 +
lib/raid6/sse1.c | 2 +
lib/raid6/sse2.c | 227 ++++++++++
lib/raid6/test/test.c | 51 ++-
lib/raid6/tilegx.uc | 1 +
29 files changed, 2860 insertions(+), 305 deletions(-)
create mode 100644 Documentation/md-cluster.txt
create mode 100644 drivers/md/md-cluster.c
create mode 100644 drivers/md/md-cluster.h
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Bisected, with rfc/patch - was Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: Christoph Hellwig @ 2015-04-23 7:37 UTC (permalink / raw)
To: NeilBrown
Cc: Azat Khuzhin, Christoph Hellwig, Kernel.org-Linux-RAID,
Guoqing Jiang, Tejun Heo, Jan Kara, Jens Axboe
In-Reply-To: <20150423160551.45345f96@notabene.brown>
Plase fix your device name lifetimes.
See the DM commit
63a4f0 ("dm: fix add_disk() NULL pointer due to race with free_dev()")
for a template.
Unregistering the device too early means we'll have half constructed
bdis hanging around, which caused all kinds of problems for filesystems
and the writeback code.
^ permalink raw reply
* Re: Bisected, with rfc/patch - was Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: NeilBrown @ 2015-04-23 8:03 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Azat Khuzhin, Kernel.org-Linux-RAID, Guoqing Jiang, Tejun Heo,
Jan Kara, Jens Axboe
In-Reply-To: <20150423073724.GA8139@lst.de>
[-- Attachment #1: Type: text/plain, Size: 798 bytes --]
On Thu, 23 Apr 2015 09:37:24 +0200 Christoph Hellwig <hch@lst.de> wrote:
> Plase fix your device name lifetimes.
Any chance you could be more explicit?
The commit you identified doesn't seem to help much - md and dm are quite
different in this area.
It seems that it is no longer safe to call 'add_disk' between calling
'del_gendisk' and bdi_destroy being called. How can I find out if I am in
that window, or wait for bdi_destroy to be called?
Thanks,
NeilBrown
>
> See the DM commit
>
> 63a4f0 ("dm: fix add_disk() NULL pointer due to race with free_dev()")
>
> for a template.
>
> Unregistering the device too early means we'll have half constructed
> bdis hanging around, which caused all kinds of problems for filesystems
> and the writeback code.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Bisected, with rfc/patch - was Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: Christoph Hellwig @ 2015-04-23 16:10 UTC (permalink / raw)
To: NeilBrown
Cc: Christoph Hellwig, Azat Khuzhin, Kernel.org-Linux-RAID,
Guoqing Jiang, Tejun Heo, Jan Kara, Jens Axboe, dm-devel
In-Reply-To: <20150423180314.367c0876@notabene.brown>
On Thu, Apr 23, 2015 at 06:03:14PM +1000, NeilBrown wrote:
> On Thu, 23 Apr 2015 09:37:24 +0200 Christoph Hellwig <hch@lst.de> wrote:
>
> > Plase fix your device name lifetimes.
>
> Any chance you could be more explicit?
>
> The commit you identified doesn't seem to help much - md and dm are quite
> different in this area.
>
> It seems that it is no longer safe to call 'add_disk' between calling
> 'del_gendisk' and bdi_destroy being called. How can I find out if I am in
> that window, or wait for bdi_destroy to be called?
The bdi is only around if the device is open, either through a device
node, or through a blkdev_get from a file system. If you get duplicate
names that means you're trying to allocate a new gendisk while the old
one is still around.
In theory you're fine once the device gets ->release called.
Except that we can hold sysfs reference to the qeue, eww. So for now
try to follow the dm model, but I'll need to add a callback to the
queue called once the request_queue actually is released for this.
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pali Rohár @ 2015-04-23 17:02 UTC (permalink / raw)
To: Pavel Machek
Cc: Mike Snitzer, Alasdair Kergon, Neil Brown, Rafael J. Wysocki,
Len Brown, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150414065016.GB20597@amd>
On Tuesday 14 April 2015 08:50:16 Pavel Machek wrote:
> Hi!
>
> > > > > > So proper way is to wipe luks crypto keys *after* userspace processes
> > > > > > are freezed.
> > > > >
> > > > > I know you believe that I'm just not accepting that at face value.
> > > >
> > > > If disks are synced before any DM suspend operation then we have higher
> > > > chance of preventing data corruption.
> > >
> > > disks are already synced as part of the DM suspend operation!
> > >
> >
> > Yes, but part of hibernate operation is also sync call.
>
> Yes. Maybe that was a mistake.
>
I think not. I do not see any reason why system should not sync disks...
> > > > I still think that correct order is only:
> > > >
> > > > * freeze processes (which doing continous I/O)
> > > > * fs & disk sync
> > > > * DM suspend
> > > > * wipe crypto keys
> > > > * enter hibernate
> > >
> > > I just don't think that extreme is _required_ to have a hibernate/resume
> > > that incorporates dm-crypt key wiping.
> >
> > Ok, and what other developers think?
>
> If someone can fix freezer to work with LUKS stopped, that would be a
> good thing. Can you do it, Mike? Then we can look if it works well
> enough for Pali.
>
Its not only LUKS devices. Its for all dm devices when are suspended.
> But that might be too hard / impossible. And at that point, I think
> Pali's patch is right thing to do.
> Pavel
Yes I think it is hard too...
--
Pali Rohár
pali.rohar@gmail.com
^ permalink raw reply
* Degraded Array event on /dev/md1:thelma
From: Robert Steinmetz AIA @ 2015-04-23 17:38 UTC (permalink / raw)
To: linux-raid
I have a small Ubuntu 14.04 LTS server with two Type 1 arrays, primarily
for redundancy. Recently I started getting email with the above title.
I checked the array and one drive had failed. I replaced the drive, ran
fdisk to create a new Linux Raid Array partition (type fd) of exactly
the same size as the existing disk.
Then using gnome-disk-utility added the new drive to the array.
Apparently that added the entire disk, not the partition I created
however, everything seems to be working properly.
BUT I am still regularly getting the email below indicating a problem
with /dev/md1, even though I think I fixed the problem.
> This is an automatically generated mail message from mdadm
> running on thelma
>
> A DegradedArray event had been detected on md device /dev/md1.
>
> Faithfully yours, etc.
>
> P.S. The /proc/mdstat file currently contains the following:
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md1 : active raid1 sdc1[0]
> 312567552 blocks [2/1] [U_]
>
> md0 : active raid1 sda1[0] sdb1[1]
> 78148096 blocks [2/2] [UU]
>
> unused devices: <none>
However mdadm reports:
> root@thelma:~# mdadm --detail /dev/md1
> /dev/md1:
> Version : 0.90
> Creation Time : Sun Mar 11 19:06:41 2007
> Raid Level : raid1
> Array Size : 312567552 (298.09 GiB 320.07 GB)
> Used Dev Size : 312567552 (298.09 GiB 320.07 GB)
> Raid Devices : 2
> Total Devices : 2
> Preferred Minor : 1
> Persistence : Superblock is persistent
>
> Update Time : Thu Apr 23 12:36:16 2015
> State : clean
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> UUID : a5865e80:27a899df:bfaac05b:eff3fc62
> Events : 0.16313034
>
> Number Major Minor RaidDevice State
> 0 8 33 0 active sync /dev/sdc1
> 1 8 48 1 active sync /dev/sdd
mdadm.conf contains the following:
> root@thelma:/etc/mdadm# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : active raid1 sdc1[0] sdd[1]
> 312567552 blocks [2/2] [UU]
>
> md0 : active raid1 sda1[0] sdb1[1]
> 78148096 blocks [2/2] [UU]
cat /proc/mdstat produces.
> root@thelma:/etc/mdadm# cat /proc/mdstat
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : active raid1 sdc1[0] sdd[1]
> 312567552 blocks [2/2] [UU]
>
> md0 : active raid1 sda1[0] sdb1[1]
> 78148096 blocks [2/2] [UU]
>
> unused devices: <none>
I'm not sure whether there is a real problem or how to fix it. It seems
some inaccurate configuration information is stored somewhere.
^ permalink raw reply
* [PATCH v3 2/4] md/raid10: make sync_request_write() call bio_copy_data()
From: Ming Lin @ 2015-04-23 23:04 UTC (permalink / raw)
To: linux-kernel
Cc: Christoph Hellwig, Kent Overstreet, Jens Axboe, Dongsu Park,
Neil Brown, linux-raid, Ming Lin
In-Reply-To: <1429830275-6792-1-git-send-email-mlin@kernel.org>
From: Kent Overstreet <kent.overstreet@gmail.com>
Refactor sync_request_write() of md/raid10 to use bio_copy_data()
instead of open coding bio_vec iterations.
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Neil Brown <neilb@suse.de>
Cc: linux-raid@vger.kernel.org
Signed-off-by: Kent Overstreet <kent.overstreet@gmail.com>
[dpark: add more description in commit message]
Signed-off-by: Dongsu Park <dongsu.park@profitbricks.com>
Signed-off-by: Ming Lin <mlin@kernel.org>
---
drivers/md/raid10.c | 20 +++++---------------
1 file changed, 5 insertions(+), 15 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index a7196c4..02e33f1 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2097,18 +2097,11 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
tbio->bi_vcnt = vcnt;
tbio->bi_iter.bi_size = r10_bio->sectors << 9;
tbio->bi_rw = WRITE;
- tbio->bi_private = r10_bio;
tbio->bi_iter.bi_sector = r10_bio->devs[i].addr;
-
- for (j=0; j < vcnt ; j++) {
- tbio->bi_io_vec[j].bv_offset = 0;
- tbio->bi_io_vec[j].bv_len = PAGE_SIZE;
-
- memcpy(page_address(tbio->bi_io_vec[j].bv_page),
- page_address(fbio->bi_io_vec[j].bv_page),
- PAGE_SIZE);
- }
tbio->bi_end_io = end_sync_write;
+ tbio->bi_private = r10_bio;
+
+ bio_copy_data(tbio, fbio);
d = r10_bio->devs[i].devnum;
atomic_inc(&conf->mirrors[d].rdev->nr_pending);
@@ -2124,17 +2117,14 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
* that are active
*/
for (i = 0; i < conf->copies; i++) {
- int j, d;
+ int d;
tbio = r10_bio->devs[i].repl_bio;
if (!tbio || !tbio->bi_end_io)
continue;
if (r10_bio->devs[i].bio->bi_end_io != end_sync_write
&& r10_bio->devs[i].bio != fbio)
- for (j = 0; j < vcnt; j++)
- memcpy(page_address(tbio->bi_io_vec[j].bv_page),
- page_address(fbio->bi_io_vec[j].bv_page),
- PAGE_SIZE);
+ bio_copy_data(tbio, fbio);
d = r10_bio->devs[i].devnum;
atomic_inc(&r10_bio->remaining);
md_sync_acct(conf->mirrors[d].replacement->bdev,
--
1.9.1
^ permalink raw reply related
* Re: PROBLEM: write to jbod with 3TB and 160GB drives hits BUG/oops
From: NeilBrown @ 2015-04-24 1:55 UTC (permalink / raw)
To: Charles Bertsch; +Cc: linux-raid, BertschC@acm.org
In-Reply-To: <55398462.1000202@cox.net>
[-- Attachment #1: Type: text/plain, Size: 971 bytes --]
On Thu, 23 Apr 2015 16:46:42 -0700 Charles Bertsch <cbertsch@cox.net> wrote:
> 1. PROBLEM: write to jbod with 3TB and 160GB drives hits BUG/oops
>
> 2. Description. Using linux-3.18.5, linux-3.18.7, and linux-4.0 a
> two-drive jbod created using one 3TB drive and one 160GB drive will
> SOMETIMES hit a BUG call at linux-4.0/fs/buffer.c:3220
By "jbod" I assume you mean "linear array".
You say this happens without any filesystem on the array, yet the stack
traces clearly show ext2 in use.
Maybe some weird interaction is happening between the the filesystem and the
linear array.
But please confirm that the stack trace happened when there was no filesystem
on the array you were testing, and report what filesystems you do have which
use ext2.
Is there any chance you could use "git bisect" to find out exactly which
commit introduced the problem? That is the mostly likely path to a solution.
Thanks for the report,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox