* Re: Reshape of RAID5 array from 3 to 4 disks frozen
From: Vilhelm von Ehrenheim @ 2015-06-09 7:07 UTC (permalink / raw)
To: David Wahler; +Cc: linux-raid
In-Reply-To: <CAGivzjHHm=faBv_+12ZMKE3=Se=tbja8uLJ1P6xvcMO2kgPYbQ@mail.gmail.com>
Hi again!
Thank you so much for this! Really cool. I patched the kernel changing
the line as you said and now the reshape is now continuing. Holding my
breath that it will finish but all looks ok so far.
I needed to change the patch a little as it was for an older kernel so
the lines changed a bit. This is what i used for linux 4.0.5:
--- a/drivers/md/raid5.c 2015-06-08 23:05:02.808214213 +0200
+++ b/drivers/md/raid5.c 2015-06-08 23:05:47.601355604 +0200
@@ -3855,7 +3855,7 @@
*/
if (s.failed > conf->max_degraded) {
sh->check_state = 0;
- sh->reconstruct_state = 0;
+ //sh->reconstruct_state = 0;
if (s.to_read+s.to_write+s.written)
handle_failed_stripe(conf, sh, &s, disks, &s.return_bi);
if (s.syncing + s.replacing)
Thank you again. :)
/ Vilhelm
On Mon, Jun 8, 2015 at 9:31 AM, David Wahler <dwahler@gmail.com> wrote:
> On Mon, Jun 8, 2015 at 1:19 AM, Vilhelm von Ehrenheim
> <vonehrenheim@gmail.com> wrote:
>> One thing that is strange and that seem to be connected to the reshape
>> is this error, present in dmesg:
>>
>> [ 360.625322] INFO: task md0_reshape:126 blocked for more than 120 seconds.
>> [ 360.625351] Not tainted 4.0.4-2-ARCH #1
>> [ 360.625367] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 360.625394] md0_reshape D ffff88040af57a58 0 126
>> 2 0x00000000
>> [ 360.625397] ffff88040af57a58 ffff88040cf58000 ffff8800da535b20
>> 00000001642a9888
>> [ 360.625399] ffff88040af57fd8 ffff8800da429000 ffff8800da429008
>> ffff8800da429208
>> [ 360.625401] 0000000096400e00 ffff88040af57a78 ffffffff81576707
>> ffff8800da429000
>> [ 360.625403] Call Trace:
>> [ 360.625410] [<ffffffff81576707>] schedule+0x37/0x90
>> [ 360.625428] [<ffffffffa0120de9>] get_active_stripe+0x5c9/0x760 [raid456]
>> [ 360.625432] [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
>> [ 360.625436] [<ffffffffa01246e0>] reshape_request+0x5b0/0x980 [raid456]
>> [ 360.625439] [<ffffffff81579053>] ? schedule_timeout+0x123/0x250
>> [ 360.625443] [<ffffffffa011743f>] sync_request+0x28f/0x400 [raid456]
>> [ 360.625449] [<ffffffffa00da486>] ? is_mddev_idle+0x136/0x170 [md_mod]
>> [ 360.625454] [<ffffffffa00de4ba>] md_do_sync+0x8ba/0xe70 [md_mod]
>> [ 360.625457] [<ffffffff81576002>] ? __schedule+0x362/0xa30
>> [ 360.625462] [<ffffffffa00d9e54>] md_thread+0x144/0x150 [md_mod]
>> [ 360.625464] [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
>> [ 360.625468] [<ffffffffa00d9d10>] ? md_start_sync+0xf0/0xf0 [md_mod]
>> [ 360.625471] [<ffffffff81093418>] kthread+0xd8/0xf0
>> [ 360.625473] [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
>> [ 360.625476] [<ffffffff8157a398>] ret_from_fork+0x58/0x90
>> [ 360.625478] [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
>>
>>
>> Also, looking at CPU usage md0_raid5 seems to be having problems as it
>> is stuck on 100% CPU on one core:
>>
>> PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
>> 125 root 20 0 0.0m 0.0m 100.0 0.0 35:57.44 R `- md0_raid5
>> 126 root 20 0 0.0m 0.0m 0.0 0.0 0:00.06 D `- md0_reshape
>>
>> Could this be why the reshape has stopped?
>>
>> Can I do something to get it going again or Is it possible to revert
>> to using 3 drives again without losing data? The data is not super
>> important, hence no backup solution, but it would mean a lot of lost
>> work.
>>
>> I'm thankful for any help I can get. Not sure what to do now.
>
> Hi Vilhelm,
>
> I ran into this exact situation several weeks ago. Fortunately Neil
> Brown was able to track it down; it turns out that the reshape
> operation can get stuck if it encounters bad blocks. See
> http://article.gmane.org/gmane.linux.raid/48673
>
> You can try applying the kernel patch from that message as a temporary
> hack to allow the reshape to complete. It worked fine for me, aside
> from a small amount of filesystem corruption that was fixable with
> fsck.
>
> -- David
^ permalink raw reply
* Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?
From: Can Jeuleers @ 2015-06-09 5:23 UTC (permalink / raw)
To: Pierre Wieser, linux-raid
In-Reply-To: <167089395.613.1433791723592.JavaMail.zimbra@wieser.fr>
On 08/06/15 21:28, Pierre Wieser wrote:
> Hi all,
>
> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> I wonder if it would be possible to migrate it to a bigger RAID 5
> with 3 x 6TB new disks.
I'd recommend against it:
https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild
Jan
^ permalink raw reply
* Re: [PATCH v2 0/8] Introduce strreplace
From: Theodore Ts'o @ 2015-06-09 0:55 UTC (permalink / raw)
To: Rasmus Villemoes
Cc: Andrew Morton, Greg Kroah-Hartman, Neil Brown, Andreas Dilger,
Steven Rostedt, Ingo Molnar, Joe Perches, Al Viro, linux-kernel,
linux-raid, linux-ext4
In-Reply-To: <1433806017-10823-1-git-send-email-linux@rasmusvillemoes.dk>
On Tue, Jun 09, 2015 at 01:26:48AM +0200, Rasmus Villemoes wrote:
> Doing single-character substitution on an entire string is open-coded
> in a few places, sometimes in a rather suboptimal way. This introduces
> a trivial helper, strreplace, for this task along with a few example
> conversions.
>
> Andrew, can I get you to take 1/8 through the mm tree? I'm not sure
> what the easiest path is for the remaining patches.
This is not super urgent, right? So we could let 1/8 go into
mainline, and then the rest of the patches could go in the next
release. That would be the simplest, although it would drag out how
long it would take for strreplace to be used everywhere.
- Ted
^ permalink raw reply
* [PATCH v2 6/8] drivers/md/md.c: Use strreplace
From: Rasmus Villemoes @ 2015-06-08 23:26 UTC (permalink / raw)
To: Andrew Morton, Neil Brown
Cc: Joe Perches, Al Viro, Rasmus Villemoes, linux-raid, linux-kernel
In-Reply-To: <1433806017-10823-1-git-send-email-linux@rasmusvillemoes.dk>
There's no point in starting over when we meet a '/'. This also
eliminates a stack variable and a little .text.
Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
v2: no changes.
drivers/md/md.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 27506302eb7a..2ea2f28551c5 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2024,7 +2024,6 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
{
char b[BDEVNAME_SIZE];
struct kobject *ko;
- char *s;
int err;
/* prevent duplicates */
@@ -2070,8 +2069,7 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
return -EBUSY;
}
bdevname(rdev->bdev,b);
- while ( (s=strchr(b, '/')) != NULL)
- *s = '!';
+ strreplace(b, '/', '!');
rdev->mddev = mddev;
printk(KERN_INFO "md: bind<%s>\n", b);
--
2.1.3
^ permalink raw reply related
* [PATCH v2 0/8] Introduce strreplace
From: Rasmus Villemoes @ 2015-06-08 23:26 UTC (permalink / raw)
To: Andrew Morton, Greg Kroah-Hartman, Neil Brown, Theodore Ts'o,
Andreas Dilger, Steven Rostedt, Ingo Molnar
Cc: Joe Perches, Al Viro, Rasmus Villemoes, linux-kernel, linux-raid,
linux-ext4
Doing single-character substitution on an entire string is open-coded
in a few places, sometimes in a rather suboptimal way. This introduces
a trivial helper, strreplace, for this task along with a few example
conversions.
Andrew, can I get you to take 1/8 through the mm tree? I'm not sure
what the easiest path is for the remaining patches.
Rasmus Villemoes (8):
lib: string: Introduce strreplace
kernel/trace/trace_events_filter.c: Use strreplace
blktrace: use strreplace in do_blk_trace_setup
lib/kobject.c: Use strreplace
drivers/base/core.c: Use strreplace
drivers/md/md.c: Use strreplace
fs/jbd2/journal.c: Use strreplace
fs/ext4/super.c: Use strreplace in ext4_fill_super
drivers/base/core.c | 9 ++++-----
drivers/md/md.c | 4 +---
fs/ext4/super.c | 4 +---
fs/jbd2/journal.c | 10 ++--------
include/linux/string.h | 1 +
kernel/trace/blktrace.c | 6 ++----
kernel/trace/trace_events_filter.c | 5 ++---
lib/kobject.c | 13 +++++--------
lib/string.c | 17 +++++++++++++++++
9 files changed, 35 insertions(+), 34 deletions(-)
--
2.1.3
^ permalink raw reply
* Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?
From: Wols Lists @ 2015-06-08 20:10 UTC (permalink / raw)
To: Pierre Wieser, linux-raid
In-Reply-To: <167089395.613.1433791723592.JavaMail.zimbra@wieser.fr>
On 08/06/15 20:28, Pierre Wieser wrote:
> Hi all,
>
> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> I wonder if it would be possible to migrate it to a bigger RAID 5
> with 3 x 6TB new disks.
>
Do you have a spare (I presume SATA) disk port?
> I've imagined something like that :
> - successively fail, remove a 2TB disk, add a 4TB disk, wait for end of recovery on three 2TB disks
If you've got a spare port, the newer mdadm's have, I believe, a "clone
and replace" option. Much better than failing then rebuilding.
If not, is it worth getting a SATA expansion board? If you've not got a
specialist mobo, surely a board is only going to cost a tenner or so,
and quality isn't *that* important seeing as it's only a temporary measure.
> - at the end of this first phase, I have the same ~6TB RAID 5 clean group with 3 x 4TB + 1 x 2TB disks
> - declare the last 2 TB disk faulty and remove it
> - the RAID 5 group state goes to clean, degraded
> - grow the RAID 5 group with --size=max option
> - grow the RAID 5 group with --array-size=~12TB option
> - last, grow the RAID 5 group with --raid-devices=3 and --backup-file=... options.
>
> And I have tested it on a small test RAID 5 group.
> As expected, this last command makes the RAID 5 group begins a reshaping operation.
> But this one keeps stucked at zero.
>
> So I have several questions :
>
Don't think I've answered any of them, but I might have raised new ones.
I just hope the tip saves you a bit of time.
Cheers,
Wol
^ permalink raw reply
* Migrating a RAID 5 from 4x2TB to 3x6TB ?
From: Pierre Wieser @ 2015-06-08 19:28 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1419435054.589.1433790714774.JavaMail.zimbra@wieser.fr>
Hi all,
I currently have an almost full RAID 5 built with 4 x 2 TB disks.
I wonder if it would be possible to migrate it to a bigger RAID 5
with 3 x 6TB new disks.
I've imagined something like that :
- successively fail, remove a 2TB disk, add a 4TB disk, wait for end of recovery on three 2TB disks
- at the end of this first phase, I have the same ~6TB RAID 5 clean group with 3 x 4TB + 1 x 2TB disks
- declare the last 2 TB disk faulty and remove it
- the RAID 5 group state goes to clean, degraded
- grow the RAID 5 group with --size=max option
- grow the RAID 5 group with --array-size=~12TB option
- last, grow the RAID 5 group with --raid-devices=3 and --backup-file=... options.
And I have tested it on a small test RAID 5 group.
As expected, this last command makes the RAID 5 group begins a reshaping operation.
But this one keeps stucked at zero.
So I have several questions :
- is it even theorically possible to grow a RAID 5 while decreasing the number of disks ?
- do you think the sequence i've imagined is correct ?
- why the reshaping operation does it stuck at zero ?
Any help or hint would be greatly appreciated :)
Thanks
Regards
Pierre
^ permalink raw reply
* Re: Reshape of RAID5 array from 3 to 4 disks frozen
From: David Wahler @ 2015-06-08 7:31 UTC (permalink / raw)
To: Vilhelm von Ehrenheim; +Cc: linux-raid
In-Reply-To: <CAJ1MkgJWoeKr+uT39AVwHOax_Z12ZF1G6v9nUz1OLvSkewQQqg@mail.gmail.com>
On Mon, Jun 8, 2015 at 1:19 AM, Vilhelm von Ehrenheim
<vonehrenheim@gmail.com> wrote:
> One thing that is strange and that seem to be connected to the reshape
> is this error, present in dmesg:
>
> [ 360.625322] INFO: task md0_reshape:126 blocked for more than 120 seconds.
> [ 360.625351] Not tainted 4.0.4-2-ARCH #1
> [ 360.625367] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 360.625394] md0_reshape D ffff88040af57a58 0 126
> 2 0x00000000
> [ 360.625397] ffff88040af57a58 ffff88040cf58000 ffff8800da535b20
> 00000001642a9888
> [ 360.625399] ffff88040af57fd8 ffff8800da429000 ffff8800da429008
> ffff8800da429208
> [ 360.625401] 0000000096400e00 ffff88040af57a78 ffffffff81576707
> ffff8800da429000
> [ 360.625403] Call Trace:
> [ 360.625410] [<ffffffff81576707>] schedule+0x37/0x90
> [ 360.625428] [<ffffffffa0120de9>] get_active_stripe+0x5c9/0x760 [raid456]
> [ 360.625432] [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
> [ 360.625436] [<ffffffffa01246e0>] reshape_request+0x5b0/0x980 [raid456]
> [ 360.625439] [<ffffffff81579053>] ? schedule_timeout+0x123/0x250
> [ 360.625443] [<ffffffffa011743f>] sync_request+0x28f/0x400 [raid456]
> [ 360.625449] [<ffffffffa00da486>] ? is_mddev_idle+0x136/0x170 [md_mod]
> [ 360.625454] [<ffffffffa00de4ba>] md_do_sync+0x8ba/0xe70 [md_mod]
> [ 360.625457] [<ffffffff81576002>] ? __schedule+0x362/0xa30
> [ 360.625462] [<ffffffffa00d9e54>] md_thread+0x144/0x150 [md_mod]
> [ 360.625464] [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
> [ 360.625468] [<ffffffffa00d9d10>] ? md_start_sync+0xf0/0xf0 [md_mod]
> [ 360.625471] [<ffffffff81093418>] kthread+0xd8/0xf0
> [ 360.625473] [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
> [ 360.625476] [<ffffffff8157a398>] ret_from_fork+0x58/0x90
> [ 360.625478] [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
>
>
> Also, looking at CPU usage md0_raid5 seems to be having problems as it
> is stuck on 100% CPU on one core:
>
> PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
> 125 root 20 0 0.0m 0.0m 100.0 0.0 35:57.44 R `- md0_raid5
> 126 root 20 0 0.0m 0.0m 0.0 0.0 0:00.06 D `- md0_reshape
>
> Could this be why the reshape has stopped?
>
> Can I do something to get it going again or Is it possible to revert
> to using 3 drives again without losing data? The data is not super
> important, hence no backup solution, but it would mean a lot of lost
> work.
>
> I'm thankful for any help I can get. Not sure what to do now.
Hi Vilhelm,
I ran into this exact situation several weeks ago. Fortunately Neil
Brown was able to track it down; it turns out that the reshape
operation can get stuck if it encounters bad blocks. See
http://article.gmane.org/gmane.linux.raid/48673
You can try applying the kernel patch from that message as a temporary
hack to allow the reshape to complete. It worked fine for me, aside
from a small amount of filesystem corruption that was fixable with
fsck.
-- David
^ permalink raw reply
* Reshape of RAID5 array from 3 to 4 disks frozen
From: Vilhelm von Ehrenheim @ 2015-06-08 6:19 UTC (permalink / raw)
To: linux-raid
Hi!
I recently added a new disk to my RAID5 array and started growing it.
I started the grow process with the following command (as i understand
it i should have had a backup file):
$ mdadm --grow --raid-devices=4 /dev/md0
The reshape process has frozen at `28%`. I can no longer mount the
array, stop it or anything it just seem to have frozen up.
Trying to mount the array just hangs
# mount /dev/md0 /mnt/storage/
And the same if I try to stop the array
# mdadm -S /dev/md0
I have also tried growing it down to 3 devices again but it is busy
with the last reshape:
# mdadm --grow /dev/md0 --raid-devices=3
mdadm: /dev/md0 is performing resync/recovery and cannot be reshaped
I tried to mark the new drive as faulty to see if the reshape would
stop but to no avail. It works to mark it as failed but nothing
happens.
After this I tried to reboot (a bit risky, i know) but the reshape
starts again from the same position still frozen at 28%.
I also tried to run a check instead of a reshape (as I read somewhere
this fixed a similar problem) but the device is busy
# echo check>/sys/block/md0/md/sync_action
-bash: echo: write error: Device or resource busy
Here is some info on the array:
# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Mar 28 17:31:15 2015
Raid Level : raid5
Array Size : 5860063744 (5588.59 GiB 6000.71 GB)
Used Dev Size : 2930031872 (2794.30 GiB 3000.35 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Jun 7 11:04:28 2015
State : clean, reshaping
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Reshape Status : 28% complete
Delta Devices : 1, (3->4)
Name : ocular:0 (local to host ocular)
UUID : e1f7a83b:2e43c552:84d09d04:b1416cb2
Events : 344582
Number Major Minor RaidDevice State
4 8 17 0 active sync /dev/sdb1
1 8 49 1 active sync /dev/sdd1
3 8 65 2 active sync /dev/sde1
5 8 33 3 active sync /dev/sdc1
and
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[4] sdc1[5] sde1[3] sdd1[1]
5860063744 blocks super 1.2 level 5, 256k chunk, algorithm 2
[4/4] [UUUU]
[=====>...............] reshape = 28.6%
(840259584/2930031872) finish=finish=33438525.6min speed=1K/sec
bitmap: 3/22 pages [12KB], 65536KB chunk
unused devices: <none>
I have also run extended SMART tests on all four disks and they all
passed without error.
One thing that is strange and that seem to be connected to the reshape
is this error, present in dmesg:
[ 360.625322] INFO: task md0_reshape:126 blocked for more than 120 seconds.
[ 360.625351] Not tainted 4.0.4-2-ARCH #1
[ 360.625367] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 360.625394] md0_reshape D ffff88040af57a58 0 126
2 0x00000000
[ 360.625397] ffff88040af57a58 ffff88040cf58000 ffff8800da535b20
00000001642a9888
[ 360.625399] ffff88040af57fd8 ffff8800da429000 ffff8800da429008
ffff8800da429208
[ 360.625401] 0000000096400e00 ffff88040af57a78 ffffffff81576707
ffff8800da429000
[ 360.625403] Call Trace:
[ 360.625410] [<ffffffff81576707>] schedule+0x37/0x90
[ 360.625428] [<ffffffffa0120de9>] get_active_stripe+0x5c9/0x760 [raid456]
[ 360.625432] [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
[ 360.625436] [<ffffffffa01246e0>] reshape_request+0x5b0/0x980 [raid456]
[ 360.625439] [<ffffffff81579053>] ? schedule_timeout+0x123/0x250
[ 360.625443] [<ffffffffa011743f>] sync_request+0x28f/0x400 [raid456]
[ 360.625449] [<ffffffffa00da486>] ? is_mddev_idle+0x136/0x170 [md_mod]
[ 360.625454] [<ffffffffa00de4ba>] md_do_sync+0x8ba/0xe70 [md_mod]
[ 360.625457] [<ffffffff81576002>] ? __schedule+0x362/0xa30
[ 360.625462] [<ffffffffa00d9e54>] md_thread+0x144/0x150 [md_mod]
[ 360.625464] [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
[ 360.625468] [<ffffffffa00d9d10>] ? md_start_sync+0xf0/0xf0 [md_mod]
[ 360.625471] [<ffffffff81093418>] kthread+0xd8/0xf0
[ 360.625473] [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
[ 360.625476] [<ffffffff8157a398>] ret_from_fork+0x58/0x90
[ 360.625478] [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
Also, looking at CPU usage md0_raid5 seems to be having problems as it
is stuck on 100% CPU on one core:
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
125 root 20 0 0.0m 0.0m 100.0 0.0 35:57.44 R `- md0_raid5
126 root 20 0 0.0m 0.0m 0.0 0.0 0:00.06 D `- md0_reshape
Could this be why the reshape has stopped?
Can I do something to get it going again or Is it possible to revert
to using 3 drives again without losing data? The data is not super
important, hence no backup solution, but it would mean a lot of lost
work.
I'm thankful for any help I can get. Not sure what to do now.
Br,
Vilhelm von Ehrenheim
^ permalink raw reply
* Fwd: [PATCH v3 0/8] MD: a caching layer for raid5/6
From: Davor Vusir @ 2015-06-04 19:29 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <cover.1433356864.git.shli@fb.com>
Sorry for intruding on the patch reviewing process and for the novice
question.
Is this caching layer also suitable for a smaller system with only one
raid5 where /boot and / resides? Or is it meant for dedicated data
storage(s) only?
Regards
Davor Vusir
-------- Vidarebefordrat meddelande --------
Ämne: [PATCH v3 0/8] MD: a caching layer for raid5/6
Datum: Wed, 3 Jun 2015 15:48:35 -0700
Från: Shaohua Li <shli@fb.com>
Till: linux-raid@vger.kernel.org
Kopia: Kernel-team@fb.com, songliubraving@fb.com, hch@infradead.org,
dan.j.williams@intel.com, neilb@suse.de
Hi,
This is the third version of the raid5/6 caching layer patches. The
patches add
a caching layer for raid5/6. The caching layer uses a SSD as a cache for
a raid
5/6. It works like the similar way of a hardware raid controller. The
purpose
is to improve raid performance (reduce read-modify-write) and fix write hole
issue. The main patch is patch 3 and the description has all details
about the
implementation. Please review!
Thanks,
Shaohua
V3:
-make reclaim multi-thread
-add statistics in sysfs
-bug fixes
V2:
-metadata write doesn't use FUA
-discard request is only issued when necessary
-bug fixes and cleanup
Shaohua Li (7):
raid5: directly use mddev->queue
raid5: A caching layer for RAID5/6
raid5: add some sysfs entries
md: don't allow resize/reshape with cache support
raid5: skip resync if caching is enabled
raid5: guarantee cache release stripes in correct way
raid5: multi-thread support for raid5 caching reclaim
Song Liu (1):
MD: add a new disk role to present cache device
drivers/md/Makefile | 2 +-
drivers/md/md.c | 14 +-
drivers/md/md.h | 4 +
drivers/md/raid5-cache.c | 3775
++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 177 +-
drivers/md/raid5.h | 25 +-
include/uapi/linux/raid/md_p.h | 73 +
7 files changed, 4022 insertions(+), 48 deletions(-)
create mode 100644 drivers/md/raid5-cache.c
--
1.8.1
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: block: add a bi_error field to struct bio
From: Mike Snitzer @ 2015-06-04 15:31 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: Jens Axboe, linux-raid, dm-devel, linux-btrfs
In-Reply-To: <1433338959-24808-2-git-send-email-hch@lst.de>
On Wed, Jun 03 2015 at 9:42P -0400,
Christoph Hellwig <hch@lst.de> wrote:
> Currently we have two different ways to signal an I/O error on a BIO:
>
> (1) by clearing the BIO_UPTODATE flag
> (2) by returning a Linux errno value to the bi_end_io callback
>
> The first one has the drawback of only communicating a single possible
> error (-EIO), and the second one has the drawback of not beeing persistent
> when bios are queued up, and are not passed along from child to parent
> bio in the ever more popular chaining scenario. Having both mechanisms
> available has the additional drawback of utterly confusing driver authors
> and introducing bugs where various I/O submitters only deal with one of
> them, and the others have to add boilerplate code to deal with both kinds
> of error returns.
>
> So add a new bi_error field to store an errno value directly in struct
> bio and remove the existing mechanisms to clean all this up.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
This patch _really_ concerns me because just in DM alone I found you
took liberties that you shouldn't have and created a regression. First
issue is a real bug (your proposed dm-io.c:dmio_complete change missed
that dm-io uses error_bits and not traditional error code like expected)
the other issue being you added extra branching that isn't needed and
made review more tedious (dm.c:clone_endio).
I'll defer to others to check their respective areas of expertise but my
experience with this review should really underscore the need for more
eyes on this patch. That said, I do appreciate your effort on cleaning
this up!
For DM, please add Signed-off-by: Mike Snitzer <snitzer@redhat.com> once
you've folded in this patch, thanks!
diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
index 86dbbc7..d2fc49f 100644
--- a/drivers/md/dm-bufio.c
+++ b/drivers/md/dm-bufio.c
@@ -545,7 +545,8 @@ static void dmio_complete(unsigned long error, void *context)
{
struct dm_buffer *b = context;
- b->bio.bi_end_io(&b->bio, error ? -EIO : 0);
+ b->bio.bi_error = error ? -EIO : 0;
+ b->bio.bi_end_io(&b->bio);
}
static void use_dmio(struct dm_buffer *b, int rw, sector_t block,
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 767bce9..85a7c3d 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -954,25 +954,22 @@ static void disable_write_same(struct mapped_device *md)
limits->max_write_same_sectors = 0;
}
-static void clone_endio(struct bio *bio, int error)
+static void clone_endio(struct bio *bio)
{
- int r = error;
+ int r = bio->bi_error;
struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone);
struct dm_io *io = tio->io;
struct mapped_device *md = tio->io->md;
dm_endio_fn endio = tio->ti->type->end_io;
- if (!bio_flagged(bio, BIO_UPTODATE) && !error)
- error = -EIO;
-
if (endio) {
- r = endio(tio->ti, bio, error);
+ r = endio(tio->ti, bio, bio->bi_error);
if (r < 0 || r == DM_ENDIO_REQUEUE)
/*
* error and requeue request are handled
* in dec_pending().
*/
- error = r;
+ ;
else if (r == DM_ENDIO_INCOMPLETE)
/* The target will handle the io */
return;
@@ -987,7 +984,7 @@ static void clone_endio(struct bio *bio, int error)
disable_write_same(md);
free_tio(md, tio);
- dec_pending(io, error);
+ dec_pending(io, r);
}
static struct dm_rq_target_io *tio_from_request(struct request *rq)
^ permalink raw reply related
* Re: [dm-devel] [PATCH] block: add a bi_error field to struct bio
From: Martin K. Petersen @ 2015-06-04 9:53 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Jens Axboe, device-mapper development, linux-raid, linux-btrfs
In-Reply-To: <1433338959-24808-2-git-send-email-hch@lst.de>
>>>>> "Christoph" == Christoph Hellwig <hch@lst.de> writes:
Christoph> The first one has the drawback of only communicating a single
Christoph> possible error (-EIO), and the second one has the drawback of
Christoph> not beeing persistent when bios are queued up, and are not
Christoph> passed along from child to parent bio in the ever more
Christoph> popular chaining scenario.
Christoph> So add a new bi_error field to store an errno value directly
Christoph> in struct bio and remove the existing mechanisms to clean all
Christoph> this up.
Having the error status separate from the bio has been a major headache.
I am entirely in favor of this patch.
It was a big chunk of changes to read through but I did not spot any
obvious problems or polarity reversals. It would be nice to get the
respective fs/md/target driver folks to check their portions, though.
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* [RFC/PATCH 6/8] drivers/md/md.c: Use strreplace
From: Rasmus Villemoes @ 2015-06-04 9:37 UTC (permalink / raw)
To: Andrew Morton, Neil Brown; +Cc: Rasmus Villemoes, linux-raid, linux-kernel
In-Reply-To: <1433410636-26812-1-git-send-email-linux@rasmusvillemoes.dk>
There's no point in starting over when we meet a '/'. This also
eliminates a stack variable and a little .text.
Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
drivers/md/md.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 27506302eb7a..2ea2f28551c5 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2024,7 +2024,6 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
{
char b[BDEVNAME_SIZE];
struct kobject *ko;
- char *s;
int err;
/* prevent duplicates */
@@ -2070,8 +2069,7 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
return -EBUSY;
}
bdevname(rdev->bdev,b);
- while ( (s=strchr(b, '/')) != NULL)
- *s = '!';
+ strreplace(b, '/', '!');
rdev->mddev = mddev;
printk(KERN_INFO "md: bind<%s>\n", b);
--
2.1.3
^ permalink raw reply related
* [RFC/PATCH 0/8] Introduce strreplace
From: Rasmus Villemoes @ 2015-06-04 9:37 UTC (permalink / raw)
To: Andrew Morton, Greg Kroah-Hartman, Neil Brown, Theodore Ts'o,
Andreas Dilger, Steven Rostedt, Ingo Molnar
Cc: Rasmus Villemoes, linux-kernel, linux-raid, linux-ext4
Doing single-character substitution on an entire string is open-coded
in a few places, sometimes in a rather suboptimal way. This introduces
a trivial helper, strreplace, for this task along with a few example
conversions.
Rasmus Villemoes (8):
lib: string: Introduce strreplace
kernel/trace/trace_events_filter.c: Use strreplace
blktrace: use strreplace in do_blk_trace_setup
lib/kobject.c: Use strreplace
drivers/base/core.c: Use strreplace
drivers/md/md.c: Use strreplace
fs/jbd2/journal.c: Use strreplace
fs/ext4/super.c: Use strreplace in ext4_fill_super
drivers/base/core.c | 5 +----
drivers/md/md.c | 4 +---
fs/ext4/super.c | 4 +---
fs/jbd2/journal.c | 10 ++--------
include/linux/string.h | 1 +
kernel/trace/blktrace.c | 6 ++----
kernel/trace/trace_events_filter.c | 5 ++---
lib/kobject.c | 4 +---
lib/string.c | 17 +++++++++++++++++
9 files changed, 28 insertions(+), 28 deletions(-)
--
2.1.3
^ permalink raw reply
* Re: 4.1-rc6 radi5 OOPS
From: Jes Sorensen @ 2015-06-04 1:44 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Xiao Ni
In-Reply-To: <20150604081557.55435f13@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Wed, 03 Jun 2015 17:57:43 -0400 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> NeilBrown <neilb@suse.de> writes:
>> > On Wed, 03 Jun 2015 16:20:21 -0400 Jes Sorensen <Jes.Sorensen@redhat.com>
>> > wrote:
>> >
>> >> Neil,
>> >>
>> >> I was running testing on the current 4.1-rc6 tree (Linus' top of trunk
>> >> 8cd9234c64c584432f6992fe944ca9e46ca8ea76) and I am seeing the following
>> >> OOPS which is reproducible.
>> >>
>> >> It shows up when running the mdadm test suite, 07changelevelintr to be
>> >> specific.
>> >>
>> >> Is this something you have seen?
>> >>
>> >> Cheers,
>> >> Jes
>> >>
>> >> ------------[ cut here ]------------
>> >> kernel BUG at drivers/md/raid5.c:5391!
>> >
>> > No, I haven't seen that. And I've been running the test suite quite a bit
>> > lately.
>> >
>> > Can you get it to print out the relevant numbers? Include
>> > readpos/writepos/safepos too.
>>
>> This enough? Let me know if you need more.
>>
>> I suspect this started happening with the changes that went in between
>> 4.1-rc5 and 4.1-rc6. I will try to bisect it tomorrow.
>>
>> Cheers,
>> Jes
>>
>> mddev->dev_sectors: 0x9800, reshape_sectors: 0x0200 stripe_addr:
>> fffffffffffffdff, sector_nr 0, readpos 511, writepos -513, safepos
>> 512
>
> Those negative numbers look VERY suspicious.
> I'm actually on leave this week so I won't be looking at it any more, but
> I'll see what I can find on Monday.
> Thanks,
> NeilBrown
Thanks - I'll try and dig more into this in the mean time.
Cheers,
Jes
^ permalink raw reply
* [PATCH v3 8/8] raid5: multi-thread support for raid5 caching reclaim
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1433356864.git.shli@fb.com>
There are several stalled stages in raid5 caching reclaim, which can
significantly harm reclaim performance. To mitigate the performance
issue, we introduce multi-thread support for reclaim. Since each thread
records reclaimed stripes in flush start/end block, it's safe for
multi-thread.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 343 +++++++++++++++++++++++++++++++++++++++--------
1 file changed, 288 insertions(+), 55 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 86e7b94..329aa38 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -224,6 +224,8 @@ enum {
};
#define STRIPE_LOCK_BITS 8
+
+#define MAX_RECLAIM_WORKERS 16
struct r5c_cache {
struct mddev *mddev;
struct md_rdev *rdev;
@@ -258,9 +260,11 @@ struct r5c_cache {
unsigned long reclaim_reason;
wait_queue_head_t reclaim_wait;
struct md_thread *reclaim_thread;
- __le64 *stripe_flush_data;
int quiesce_state;
+ struct md_thread *reclaim_workers[MAX_RECLAIM_WORKERS];
+ int reclaim_worker_cnt;
+
int in_recovery;
struct work_struct pending_io_work;
@@ -294,9 +298,8 @@ enum {
RECLAIM_DISK_BACKGROUND = 9, /* try to reclaim disk */
RECLAIM_FLUSH_ALL = 16, /* flush all data to raid */
- QUIESCE_NONE = 0,
+ QUIESCE_END = 0,
QUIESCE_START = 1,
- QUIESCE_END = 2,
ERROR_NOERROR = 0,
ERROR_PREPARE = 1, /* Had an error, flushing cache to raid */
@@ -1958,7 +1961,9 @@ static void r5c_select_stripes(struct r5c_cache *cache, struct list_head *list)
{
int stripes;
bool blocking;
+ static DEFINE_MUTEX(lock);
+ mutex_lock(&lock);
/*
* generally select full stripe, if no disk space, select first stripe
*/
@@ -1991,6 +1996,7 @@ static void r5c_select_stripes(struct r5c_cache *cache, struct list_head *list)
}
spin_unlock_irq(&cache->tree_lock);
+ mutex_unlock(&lock);
}
static void r5c_disks_flush_end(struct bio *bio, int err)
@@ -2028,11 +2034,44 @@ static int r5c_stripe_list_cmp(void *priv, struct list_head *a,
return !(stripe_a->raid_index < stripe_b->raid_index);
}
-static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
- struct list_head *stripe_list)
+struct r5c_reclaim_context {
+ struct list_head stripe_list;
+ __le64 *stripe_flush_data;
+ u64 seq;
+ sector_t meta;
+ struct completion comp;
+};
+
+static struct r5c_reclaim_context *
+r5c_alloc_reclaim_context(struct r5c_cache *cache)
+{
+ struct r5c_reclaim_context *context;
+
+ context = kzalloc(sizeof(*context), GFP_KERNEL);
+ if (!context)
+ return NULL;
+ INIT_LIST_HEAD(&context->stripe_list);
+ context->stripe_flush_data = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!context->stripe_flush_data) {
+ kfree(context);
+ return NULL;
+ }
+ init_completion(&context->comp);
+ return context;
+}
+
+static void r5c_free_reclaim_context(struct r5c_reclaim_context *context)
+{
+ kfree(context->stripe_flush_data);
+ kfree(context);
+}
+
+static void r5c_do_reclaim(struct r5c_cache *cache,
+ struct r5c_reclaim_context *context)
{
+ struct list_head *stripe_list = &context->stripe_list;
+ __le64 *stripe_flush_data = context->stripe_flush_data;
struct r5c_stripe *stripe;
- struct r5c_io_range *range;
u64 seq;
sector_t meta;
size_t size = 0;
@@ -2045,7 +2084,7 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
list_sort(NULL, stripe_list, r5c_stripe_list_cmp);
list_for_each_entry(stripe, stripe_list, lru) {
- cache->stripe_flush_data[size] =
+ stripe_flush_data[size] =
cpu_to_le64(stripe->raid_index);
size++;
}
@@ -2063,7 +2102,7 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
/* step 3: make sure data and parity settle down */
r5l_flush_block(&cache->log, R5LOG_TYPE_FLUSH_START,
- cache->stripe_flush_data, size, &seq, &meta);
+ stripe_flush_data, size, &seq, &meta);
/* step 4: continue write to raid */
list_for_each_entry(stripe, stripe_list, lru) {
@@ -2087,7 +2126,7 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
/* step 7: mark data is flushed to raid */
r5l_flush_block(&cache->log, R5LOG_TYPE_FLUSH_END,
- cache->stripe_flush_data, size, &seq, &meta);
+ stripe_flush_data, size, &seq, &meta);
/* step 8: mark stripe as dead */
while (!list_empty(stripe_list)) {
@@ -2100,6 +2139,23 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
r5c_put_stripe(stripe);
}
+ context->seq = seq;
+ context->meta = meta;
+}
+
+static void r5c_do_reclaim_and_write_super(struct r5c_cache *cache,
+ struct r5c_reclaim_context *context)
+{
+ struct r5c_io_range *range;
+ u64 seq;
+ sector_t meta;
+
+ if (list_empty(&context->stripe_list))
+ return;
+
+ r5c_do_reclaim(cache, context);
+ seq = context->seq;
+ meta = context->meta;
/* step 9: advance superblock checkpoint */
spin_lock_irq(&cache->tree_lock);
@@ -2118,14 +2174,67 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
r5l_write_super(&cache->log, seq, meta);
}
+static void r5c_reclaim_work(struct r5c_cache *cache,
+ struct r5c_reclaim_context *context)
+{
+ /*
+ * select stripe will freeze stripe, which will guarantee no
+ * new task pending in error mode
+ * */
+ r5c_select_stripes(cache, &context->stripe_list);
+
+ r5c_do_reclaim(cache, context);
+}
+
+static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
+ struct list_head *stripe_list)
+{
+ int size;
+ struct r5c_reclaim_context *context;
+
+ context = r5c_alloc_reclaim_context(cache);
+
+ while (!list_empty(stripe_list)) {
+ size = 0;
+ while (!list_empty(stripe_list) && size < RECLAIM_BATCH) {
+ list_move_tail(stripe_list->next,
+ &context->stripe_list);
+ size++;
+ }
+ if (!list_empty(stripe_list))
+ r5c_do_reclaim(cache, context);
+ else
+ r5c_do_reclaim_and_write_super(cache, context);
+ }
+
+ r5c_free_reclaim_context(context);
+}
+
+static void r5c_reclaim_thread_check_quiesce(struct r5c_cache *cache,
+ struct r5c_reclaim_context *context)
+{
+ if (cache->quiesce_state == QUIESCE_START) {
+ complete(&context->comp);
+ wait_event(cache->reclaim_wait, cache->quiesce_state ==
+ QUIESCE_END);
+ reinit_completion(&context->comp);
+ }
+}
+
static void r5c_reclaim_thread(struct md_thread *thread)
{
struct mddev *mddev = thread->mddev;
struct r5conf *conf = mddev->private;
struct r5c_cache *cache = conf->cache;
- LIST_HEAD(stripe_list);
+ struct r5c_reclaim_context *context = thread->private;
bool retry;
+ r5c_reclaim_thread_check_quiesce(cache, context);
+
+ if (thread != cache->reclaim_thread) {
+ r5c_reclaim_work(cache, context);
+ return;
+ }
do_retry:
retry = false;
@@ -2134,9 +2243,9 @@ static void r5c_reclaim_thread(struct md_thread *thread)
* select stripe will freeze stripe, which will guarantee no
* new task pending in error mode
* */
- r5c_select_stripes(cache, &stripe_list);
+ r5c_select_stripes(cache, &context->stripe_list);
- if (list_empty(&stripe_list)) {
+ if (list_empty(&context->stripe_list)) {
clear_bit(RECLAIM_MEM_FULL, &cache->reclaim_reason);
clear_bit(RECLAIM_MEM_BACKGROUND,
&cache->reclaim_reason);
@@ -2144,7 +2253,7 @@ static void r5c_reclaim_thread(struct md_thread *thread)
&cache->reclaim_reason);
clear_bit(RECLAIM_FLUSH_ALL, &cache->reclaim_reason);
} else
- r5c_reclaim_stripe_list(cache, &stripe_list);
+ r5c_do_reclaim_and_write_super(cache, context);
wake_up(&cache->reclaim_wait);
}
@@ -2179,20 +2288,80 @@ static void r5c_reclaim_thread(struct md_thread *thread)
cache->error_state = 0;
mddev_resume(cache->mddev);
}
- if (cache->quiesce_state == QUIESCE_START) {
- /* user IO already finished, we just stop reclaim */
- cache->reclaim_reason = 0;
- cache->quiesce_state = QUIESCE_END;
- wake_up(&cache->reclaim_wait);
- wait_event(cache->reclaim_wait, cache->quiesce_state ==
- QUIESCE_NONE);
+}
+
+static struct md_thread *r5c_init_reclaim_thread(struct r5c_cache *cache)
+{
+ struct r5c_reclaim_context *context;
+ struct md_thread *thread;
+
+ context = r5c_alloc_reclaim_context(cache);
+ if (!context)
+ return NULL;
+
+ thread = md_register_thread(r5c_reclaim_thread,
+ cache->mddev, "reclaim");
+ if (!thread) {
+ r5c_free_reclaim_context(context);
+ return NULL;
+ }
+ thread->private = context;
+
+ return thread;
+}
+
+static void r5c_exit_reclaim_thread(struct r5c_cache *cache,
+ struct md_thread **thread)
+{
+ struct r5c_reclaim_context *context;
+
+ context = (*thread)->private;
+ r5c_free_reclaim_context(context);
+
+ md_unregister_thread(thread);
+}
+
+static int r5c_init_reclaimers(struct r5c_cache *cache)
+{
+ struct md_thread *thread;
+
+ thread = r5c_init_reclaim_thread(cache);
+ if (!thread)
+ return -ENOMEM;
+ cache->reclaim_thread = thread;
+ cache->reclaim_thread->timeout = CHECKPOINT_TIMEOUT;
+ return 0;
+}
+
+static void r5c_exit_reclaimers(struct r5c_cache *cache)
+{
+ int i = cache->reclaim_worker_cnt;
+
+ while (i > 0) {
+ r5c_exit_reclaim_thread(cache, &cache->reclaim_workers[i - 1]);
+ i--;
}
+
+ r5c_exit_reclaim_thread(cache, &cache->reclaim_thread);
+}
+
+static void r5c_wakeup_reclaimer_threads(struct r5c_cache *cache)
+{
+ int i;
+
+ md_wakeup_thread(cache->reclaim_thread);
+
+ preempt_disable();
+ for (i = 0; i < cache->reclaim_worker_cnt; i++)
+ if (cache->reclaim_workers[i])
+ md_wakeup_thread(cache->reclaim_workers[i]);
+ preempt_enable();
}
static void r5c_wake_reclaimer(struct r5c_cache *cache, int reason)
{
set_bit(reason, &cache->reclaim_reason);
- md_wakeup_thread(cache->reclaim_thread);
+ r5c_wakeup_reclaimer_threads(cache);
}
static void r5c_wake_wait_reclaimer(struct r5c_cache *cache, int reason)
@@ -2205,17 +2374,25 @@ static void r5c_wake_wait_reclaimer(struct r5c_cache *cache, int reason)
void r5c_quiesce(struct r5conf *conf, int state)
{
struct r5c_cache *cache = conf->cache;
+ struct r5c_reclaim_context *context;
+ int i;
if (!cache || cache->error_state)
return;
if (state == 1) {
r5c_wake_wait_reclaimer(cache, RECLAIM_FLUSH_ALL);
+
cache->quiesce_state = QUIESCE_START;
- md_wakeup_thread(cache->reclaim_thread);
- wait_event(cache->reclaim_wait, cache->quiesce_state ==
- QUIESCE_END);
+ r5c_wakeup_reclaimer_threads(cache);
+
+ for (i = 0; i < cache->reclaim_worker_cnt; i++) {
+ context = cache->reclaim_workers[i]->private;
+ wait_for_completion(&context->comp);
+ }
+ context = cache->reclaim_thread->private;
+ wait_for_completion(&context->comp);
} else if (state == 0) {
- cache->quiesce_state = QUIESCE_NONE;
+ cache->quiesce_state = QUIESCE_END;
wake_up(&cache->reclaim_wait);
}
}
@@ -2751,7 +2928,6 @@ static int r5c_recover_stripes(struct r5c_load_ctx *ctx)
{
struct r5c_cache *cache = ctx->cache;
LIST_HEAD(list);
- int i;
r5l_check_stripes_checksum(ctx);
@@ -2762,18 +2938,7 @@ static int r5c_recover_stripes(struct r5c_load_ctx *ctx)
cache->in_recovery = 1;
- while (!list_empty(&ctx->stripes_with_parity)) {
- i = 0;
- /* Can't handle large stripe list */
- while (i < RECLAIM_BATCH &&
- !list_empty(&ctx->stripes_with_parity)) {
- list_move_tail(ctx->stripes_with_parity.next,
- &list);
- i++;
- }
- r5c_reclaim_stripe_list(cache, &list);
- BUG_ON(!list_empty(&list));
- }
+ r5c_reclaim_stripe_list(cache, &ctx->stripes_with_parity);
cache->in_recovery = 0;
return 0;
@@ -3181,16 +3346,20 @@ int r5c_min_stripe_cache_size(struct r5c_cache *cache)
{
struct r5conf *conf = cache->mddev->private;
return (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) *
- cache->reclaim_batch;
+ cache->reclaim_batch * (1 + cache->reclaim_worker_cnt);
}
-static void r5c_set_reclaim_batch(struct r5c_cache *cache, int batch)
+static void r5c_set_reclaim_batch(struct r5c_cache *cache, int batch,
+ int threads)
{
struct mddev *mddev = cache->mddev;
struct r5conf *conf = mddev->private;
int size;
- size = (cache->stripe_parity_pages << PAGE_SECTOR_SHIFT) * batch;
+ threads++;
+
+ size = (cache->stripe_parity_pages << PAGE_SECTOR_SHIFT) * batch *
+ threads;
if (size > cache->reserved_space) {
cache->reserved_space = size;
mutex_lock(&cache->log.io_mutex);
@@ -3207,7 +3376,7 @@ static void r5c_set_reclaim_batch(struct r5c_cache *cache, int batch)
cache->reserved_space = size;
}
- size = (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) * batch;
+ size = (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) * batch * threads;
mddev_lock(mddev);
if (size > conf->max_nr_stripes)
@@ -3241,7 +3410,7 @@ static ssize_t r5c_store_cache_reclaim_batch(struct mddev *mddev,
new = r5l_max_flush_stripes(&cache->log);
if (new != cache->reclaim_batch)
- r5c_set_reclaim_batch(cache, new);
+ r5c_set_reclaim_batch(cache, new, cache->reclaim_worker_cnt);
return len;
}
@@ -3337,6 +3506,68 @@ static struct md_sysfs_entry r5c_cache_memory_watermark =
__ATTR(cache_memory_watermark, S_IRUGO | S_IWUSR,
r5c_show_cache_memory_watermark, r5c_store_cache_memory_watermark);
+static ssize_t r5c_show_reclaim_threads(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%d\n", cache->reclaim_worker_cnt);
+}
+
+static void r5c_set_reclaim_thread_count(struct r5c_cache *cache, int cnt)
+{
+ struct md_thread *thread;
+ int old_cnt;
+
+ if (cache->reclaim_worker_cnt == cnt)
+ return;
+ if (cnt > MAX_RECLAIM_WORKERS)
+ cnt = MAX_RECLAIM_WORKERS;
+
+ old_cnt = cache->reclaim_worker_cnt;
+ if (old_cnt > cnt) {
+ cache->reclaim_worker_cnt = cnt;
+ /* make sure r5c_wake_reclaimer() isn't using thread */
+ synchronize_sched();
+ }
+
+ while (old_cnt > cnt) {
+ r5c_exit_reclaim_thread(cache,
+ &cache->reclaim_workers[old_cnt - 1]);
+ old_cnt--;
+ }
+ while (old_cnt < cnt) {
+ thread = r5c_init_reclaim_thread(cache);
+ if (!thread)
+ break;
+ cache->reclaim_workers[old_cnt++] = thread;
+ }
+
+ r5c_set_reclaim_batch(cache, cache->reclaim_batch, old_cnt);
+
+ cache->reclaim_worker_cnt = old_cnt;
+}
+
+static ssize_t r5c_store_reclaim_threads(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned int new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtouint(page, 0, &new))
+ return -EINVAL;
+
+ r5c_set_reclaim_thread_count(cache, new);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_reclaim_threads =
+ __ATTR(cache_reclaim_threads, S_IRUGO | S_IWUSR,
+ r5c_show_reclaim_threads, r5c_store_reclaim_threads);
+
static int r5c_init_sysfs(struct r5c_cache *cache)
{
struct mddev *mddev = cache->mddev;
@@ -3363,7 +3594,16 @@ static int r5c_init_sysfs(struct r5c_cache *cache)
&r5c_cache_memory_watermark.attr, NULL);
if (ret)
goto memory_watermark;
+
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_reclaim_threads.attr, NULL);
+ if (ret)
+ goto reclaim_threads;
+
return 0;
+reclaim_threads:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory_watermark.attr, NULL);
memory_watermark:
sysfs_remove_file_from_group(&mddev->kobj,
&r5c_cache_stat.attr, NULL);
@@ -3392,6 +3632,8 @@ static void r5c_exit_sysfs(struct r5c_cache *cache)
&r5c_cache_stat.attr, NULL);
sysfs_remove_file_from_group(&mddev->kobj,
&r5c_cache_memory_watermark.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_reclaim_threads.attr, NULL);
}
static void r5c_free_cache_data(struct r5c_cache *cache)
@@ -3446,10 +3688,6 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
cache->stripe_parity_pages = (cache->stripe_size -
cache->stripe_data_size) >> PAGE_SECTOR_SHIFT;
- cache->stripe_flush_data = kmalloc(PAGE_SIZE, GFP_KERNEL);
- if (!cache->stripe_flush_data)
- goto io_range_kc;
-
cache->io_range_kc = KMEM_CACHE(r5c_io_range, 0);
if (!cache->io_range_kc)
goto io_range_kc;
@@ -3497,18 +3735,15 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
r5c_calculate_watermark(cache);
- cache->reclaim_thread = md_register_thread(r5c_reclaim_thread,
- mddev, "reclaim");
- if (!cache->reclaim_thread)
+ if (r5c_init_reclaimers(cache))
goto err_page;
- cache->reclaim_thread->timeout = CHECKPOINT_TIMEOUT;
r5c_shrink_cache_memory(cache, cache->max_pages);
if (r5c_init_sysfs(cache))
goto err_sysfs;
return cache;
err_sysfs:
- md_unregister_thread(&cache->reclaim_thread);
+ r5c_exit_reclaimers(cache);
err_page:
r5c_free_cache_data(cache);
@@ -3520,7 +3755,6 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
stripe_kc:
kmem_cache_destroy(cache->io_range_kc);
io_range_kc:
- kfree(cache->stripe_flush_data);
kfree(cache);
return NULL;
}
@@ -3528,7 +3762,7 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
void r5c_exit_cache(struct r5c_cache *cache)
{
r5c_exit_sysfs(cache);
- md_unregister_thread(&cache->reclaim_thread);
+ r5c_exit_reclaimers(cache);
r5l_exit_log(&cache->log);
r5c_free_cache_data(cache);
@@ -3537,6 +3771,5 @@ void r5c_exit_cache(struct r5c_cache *cache)
kmem_cache_destroy(cache->stripe_kc);
kmem_cache_destroy(cache->io_range_kc);
- kfree(cache->stripe_flush_data);
kfree(cache);
}
--
1.8.1
^ permalink raw reply related
* [PATCH v3 7/8] raid5: guarantee cache release stripes in correct way
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1433356864.git.shli@fb.com>
When cache trys to flush data to raid, we must be very careful about
stripe release. The issue is if we release a stripe and the stripe is
handling by raid5d and we add a new bio into the stripe later, the
stripe will enter the raid5 state machine several times, which
r5cache_write_parity can't handle well. This could happen if
get_active_stripe() sleeps in adding bio to several stripes. To solve
this issue, we guarantee stripe release after all bio are added to
corresponding stripes. This is a performance win too if the bug happens.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 28 +++++++++-------
drivers/md/raid5.c | 84 +++++++++++++++++++++++++++++++-----------------
drivers/md/raid5.h | 11 ++++++-
3 files changed, 80 insertions(+), 43 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 04b1684..86e7b94 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -1670,7 +1670,7 @@ static void r5c_write_bio(struct r5c_cache *cache, struct bio *bio)
r5c_enter_error_mode(cache, -ENOMEM);
r5c_check_wait_error_mode(cache);
error_mode:
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, NULL);
}
static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
@@ -1686,7 +1686,7 @@ static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
stripe = r5c_get_stripe(cache, stripe_index);
if (!stripe) {
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, NULL);
return;
}
@@ -1725,7 +1725,7 @@ static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
} else
split = bio;
- raid5_make_request(cache->mddev, split);
+ raid5_make_request(cache->mddev, split, NULL);
start = tmp;
}
@@ -1823,7 +1823,7 @@ void r5c_write_end(struct mddev *mddev, struct bio *bi)
}
static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
- int start, int pages)
+ int start, int pages, struct raid5_plug_context *pc)
{
sector_t base;
struct bio *bio;
@@ -1858,7 +1858,7 @@ static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
atomic64_inc(&cache->out_cache_rq);
atomic64_add(bio_sectors(bio), &cache->out_cache_sectors);
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, pc);
}
}
@@ -1871,7 +1871,8 @@ static void r5c_put_stripe_dirty(struct r5c_cache *cache,
}
}
-static void r5c_flush_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe)
+static void r5c_flush_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ struct raid5_plug_context *pc)
{
unsigned long *stripe_bits;
int chunk_stripes;
@@ -1900,7 +1901,7 @@ static void r5c_flush_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe)
while (end < cache->stripe_data_pages &&
stripe->data_pages[end])
end++;
- r5c_flush_one(cache, stripe, start, end - start);
+ r5c_flush_one(cache, stripe, start, end - start, pc);
}
r5c_put_stripe_dirty(cache, stripe);
}
@@ -2034,11 +2035,14 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
struct r5c_io_range *range;
u64 seq;
sector_t meta;
- struct blk_plug plug;
size_t size = 0;
+ struct raid5_plug_context pc;
if (list_empty(stripe_list))
return;
+
+ raid5_context_init(&pc);
+
list_sort(NULL, stripe_list, r5c_stripe_list_cmp);
list_for_each_entry(stripe, stripe_list, lru) {
cache->stripe_flush_data[size] =
@@ -2047,11 +2051,11 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
}
size *= sizeof(__le64);
- blk_start_plug(&plug);
/* step 1: start write to raid */
list_for_each_entry(stripe, stripe_list, lru)
- r5c_flush_stripe(cache, stripe);
- blk_finish_plug(&plug);
+ r5c_flush_stripe(cache, stripe, &pc);
+
+ raid5_context_unplug(&pc, cache->mddev, false);
/* step 2: wait parity write to cache */
list_for_each_entry_reverse(stripe, stripe_list, lru)
@@ -2158,7 +2162,7 @@ static void r5c_reclaim_thread(struct md_thread *thread)
wake_up(&cache->error_wait);
while ((bio = bio_list_pop(&cache->retry_bio_list)) != NULL)
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, NULL);
if (++cache->retry_cnt < MAX_RETRY) {
cache->next_retry_time = jiffies +
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index b1c942c..6cb10af 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -4973,26 +4973,26 @@ static struct stripe_head *__get_priority_stripe(struct r5conf *conf, int group)
return sh;
}
-struct raid5_plug_cb {
- struct blk_plug_cb cb;
- struct list_head list;
- struct list_head temp_inactive_list[NR_STRIPE_HASH_LOCKS];
-};
+void raid5_context_init(struct raid5_plug_context *context)
+{
+ int i;
+ INIT_LIST_HEAD(&context->list);
+ for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++)
+ INIT_LIST_HEAD(context->temp_inactive_list + i);
+}
-static void raid5_unplug(struct blk_plug_cb *blk_cb, bool from_schedule)
+void raid5_context_unplug(struct raid5_plug_context *context,
+ struct mddev *mddev, bool from_schedule)
{
- struct raid5_plug_cb *cb = container_of(
- blk_cb, struct raid5_plug_cb, cb);
struct stripe_head *sh;
- struct mddev *mddev = cb->cb.data;
struct r5conf *conf = mddev->private;
int cnt = 0;
int hash;
- if (cb->list.next && !list_empty(&cb->list)) {
+ if (context->list.next && !list_empty(&context->list)) {
spin_lock_irq(&conf->device_lock);
- while (!list_empty(&cb->list)) {
- sh = list_first_entry(&cb->list, struct stripe_head, lru);
+ while (!list_empty(&context->list)) {
+ sh = list_first_entry(&context->list, struct stripe_head, lru);
list_del_init(&sh->lru);
/*
* avoid race release_stripe_plug() sees
@@ -5006,15 +5006,38 @@ static void raid5_unplug(struct blk_plug_cb *blk_cb, bool from_schedule)
* case, the count is always > 1 here
*/
hash = sh->hash_lock_index;
- __release_stripe(conf, sh, &cb->temp_inactive_list[hash]);
+ __release_stripe(conf, sh, &context->temp_inactive_list[hash]);
cnt++;
}
spin_unlock_irq(&conf->device_lock);
}
- release_inactive_stripe_list(conf, cb->temp_inactive_list,
+ release_inactive_stripe_list(conf, context->temp_inactive_list,
NR_STRIPE_HASH_LOCKS);
if (mddev->queue)
trace_block_unplug(mddev->queue, cnt, !from_schedule);
+}
+
+static void raid5_context_plug(struct raid5_plug_context *context,
+ struct stripe_head *sh)
+{
+ if (!test_and_set_bit(STRIPE_ON_UNPLUG_LIST, &sh->state))
+ list_add_tail(&sh->lru, &context->list);
+ else
+ release_stripe(sh);
+}
+
+struct raid5_plug_cb {
+ struct blk_plug_cb cb;
+ struct raid5_plug_context context;
+};
+
+static void raid5_unplug(struct blk_plug_cb *blk_cb, bool from_schedule)
+{
+ struct raid5_plug_cb *cb = container_of(
+ blk_cb, struct raid5_plug_cb, cb);
+ struct mddev *mddev = cb->cb.data;
+
+ raid5_context_unplug(&cb->context, mddev, from_schedule);
kfree(cb);
}
@@ -5033,20 +5056,14 @@ static void release_stripe_plug(struct mddev *mddev,
cb = container_of(blk_cb, struct raid5_plug_cb, cb);
- if (cb->list.next == NULL) {
- int i;
- INIT_LIST_HEAD(&cb->list);
- for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++)
- INIT_LIST_HEAD(cb->temp_inactive_list + i);
- }
+ if (cb->context.list.next == NULL)
+ raid5_context_init(&cb->context);
- if (!test_and_set_bit(STRIPE_ON_UNPLUG_LIST, &sh->state))
- list_add_tail(&sh->lru, &cb->list);
- else
- release_stripe(sh);
+ raid5_context_plug(&cb->context, sh);
}
-static void make_discard_request(struct mddev *mddev, struct bio *bi)
+static void make_discard_request(struct mddev *mddev, struct bio *bi,
+ struct raid5_plug_context *context)
{
struct r5conf *conf = mddev->private;
sector_t logical_sector, last_sector;
@@ -5128,7 +5145,10 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
clear_bit(STRIPE_DELAYED, &sh->state);
if (!test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
atomic_inc(&conf->preread_active_stripes);
- release_stripe_plug(mddev, sh);
+ if (context)
+ raid5_context_plug(context, sh);
+ else
+ release_stripe_plug(mddev, sh);
}
remaining = raid5_dec_bi_active_stripes(bi);
@@ -5138,7 +5158,8 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
}
}
-void raid5_make_request(struct mddev *mddev, struct bio * bi)
+void raid5_make_request(struct mddev *mddev, struct bio * bi,
+ struct raid5_plug_context *context)
{
struct r5conf *conf = mddev->private;
int dd_idx;
@@ -5168,7 +5189,7 @@ void raid5_make_request(struct mddev *mddev, struct bio * bi)
return;
if (unlikely(bi->bi_rw & REQ_DISCARD)) {
- make_discard_request(mddev, bi);
+ make_discard_request(mddev, bi, context);
return;
}
@@ -5296,7 +5317,10 @@ void raid5_make_request(struct mddev *mddev, struct bio * bi)
((bi->bi_rw & REQ_SYNC) || conf->cache) &&
!test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
atomic_inc(&conf->preread_active_stripes);
- release_stripe_plug(mddev, sh);
+ if (context)
+ raid5_context_plug(context, sh);
+ else
+ release_stripe_plug(mddev, sh);
} else {
/* cannot get stripe for read-ahead, just give-up */
clear_bit(BIO_UPTODATE, &bi->bi_flags);
@@ -5323,7 +5347,7 @@ static void make_request(struct mddev *mddev, struct bio *bi)
if (conf->cache)
r5c_handle_bio(conf->cache, bi);
else
- raid5_make_request(mddev, bi);
+ raid5_make_request(mddev, bi, NULL);
}
static sector_t raid5_size(struct mddev *mddev, sector_t sectors, int raid_disks);
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index f117b68..1adb2b8 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -607,8 +607,17 @@ static inline int algorithm_is_DDF(int layout)
extern void md_raid5_kick_device(struct r5conf *conf);
extern int raid5_set_cache_size(struct mddev *mddev, int size);
+struct raid5_plug_context {
+ struct list_head list;
+ struct list_head temp_inactive_list[NR_STRIPE_HASH_LOCKS];
+};
+void raid5_context_init(struct raid5_plug_context *context);
+void raid5_context_unplug(struct raid5_plug_context *context,
+ struct mddev *mddev, bool from_schedule);
+
void release_stripe(struct stripe_head *sh);
-void raid5_make_request(struct mddev *mddev, struct bio *bi);
+void raid5_make_request(struct mddev *mddev, struct bio *bi,
+ struct raid5_plug_context *context);
void r5c_handle_bio(struct r5c_cache *cache, struct bio *bi);
int r5c_write_parity(struct r5c_cache *cache, struct stripe_head *sh);
void r5c_flush_pending_parity(struct r5c_cache *cache);
--
1.8.1
^ permalink raw reply related
* [PATCH v3 6/8] raid5: skip resync if caching is enabled
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1433356864.git.shli@fb.com>
If caching is enabled, the caching layer will guarantee data
consistency, so skip resync for unclean shutdown
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 29f49c7..b1c942c 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -6975,6 +6975,13 @@ static int run(struct mddev *mddev)
if (mddev->queue)
blk_queue_logical_block_size(mddev->queue, STRIPE_SIZE);
conf->skip_copy = 1;
+
+ if (mddev->recovery_cp == 0) {
+ printk(KERN_NOTICE
+ "md/raid:%s: skip resync with caching enabled\n",
+ mdname(mddev));
+ mddev->recovery_cp = MaxSector;
+ }
}
return 0;
--
1.8.1
^ permalink raw reply related
* [PATCH v3 5/8] md: don't allow resize/reshape with cache support
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1433356864.git.shli@fb.com>
If cache support is enabled, don't allow resize/reshape in current
stage. In the future, we can flush all data from cache to raid before
resize/reshape and then allow resize/reshape.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 26561d8..29f49c7 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7207,6 +7207,10 @@ static int raid5_resize(struct mddev *mddev, sector_t sectors)
* worth it.
*/
sector_t newsize;
+ struct r5conf *conf = mddev->private;
+
+ if (conf->cache)
+ return -EINVAL;
sectors &= ~((sector_t)mddev->chunk_sectors - 1);
newsize = raid5_size(mddev, sectors, mddev->raid_disks);
if (mddev->external_size &&
@@ -7258,6 +7262,8 @@ static int check_reshape(struct mddev *mddev)
{
struct r5conf *conf = mddev->private;
+ if (conf->cache)
+ return -EINVAL;
if (mddev->delta_disks == 0 &&
mddev->new_layout == mddev->layout &&
mddev->new_chunk_sectors == mddev->chunk_sectors)
--
1.8.1
^ permalink raw reply related
* [PATCH v3 4/8] raid5: add some sysfs entries
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1433356864.git.shli@fb.com>
Add some sysfs entries.
-cache_memory. Control the cache memory size.
-cache_reclaim_batch. Control how many stripes reclaim should run in one
time.
-cache_memory_watermark. The background reclaim runs if cache memory
hits the watermark and stops after hit 1.5x of the watermark.
-cache_disk_watermark. The background reclaim runs if cache disk space
hits the watermark and stops after hit 1.5x of the watermark.
-cache_stat. statistics about cache.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 294 ++++++++++++++++++++++++++++++++++++++++++++++-
drivers/md/raid5.c | 3 +
drivers/md/raid5.h | 1 +
3 files changed, 297 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index c21d2f2..04b1684 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -278,6 +278,12 @@ struct r5c_cache {
struct kmem_cache *io_range_kc;
struct kmem_cache *stripe_kc;
struct bio_set *bio_set;
+
+ atomic64_t in_cache_rq;
+ atomic64_t out_cache_rq;
+ atomic64_t in_cache_sectors;
+ atomic64_t out_cache_sectors;
+ atomic64_t read_cache_sectors;
};
enum {
@@ -314,6 +320,12 @@ static inline int r5l_page_blocks(struct r5l_log *log, int pages)
return pages << log->page_block_shift;
}
+static inline int r5l_max_flush_stripes(struct r5l_log *log)
+{
+ return (log->block_size - sizeof(struct r5l_flush_block)) /
+ sizeof(__le64);
+}
+
static u32 r5l_calculate_checksum(struct r5l_log *log, u32 crc,
void *buf, size_t size, bool data)
{
@@ -1644,6 +1656,9 @@ static void r5c_write_bio(struct r5c_cache *cache, struct bio *bio)
stripe->existing_pages += new_pages;
r5c_unlock_stripe(cache, stripe, &flags);
+ atomic64_inc(&cache->in_cache_rq);
+ atomic64_add(bio_sectors(bio), &cache->in_cache_sectors);
+
if (r5l_queue_bio(&cache->log, bio, r5c_bio_task_end, io_range,
reserved_blocks))
goto put_error;
@@ -1692,6 +1707,8 @@ static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
split = bio;
r5c_copy_bio(split, &stripe->data_pages[start], true);
+ atomic64_add(bio_sectors(split),
+ &cache->read_cache_sectors);
bio_endio(split, 0);
@@ -1837,6 +1854,10 @@ static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
bio->bi_end_io = r5c_flush_endio;
bio->bi_rw = WRITE;
atomic_inc(&stripe->pending_bios);
+
+ atomic64_inc(&cache->out_cache_rq);
+ atomic64_add(bio_sectors(bio), &cache->out_cache_sectors);
+
raid5_make_request(cache->mddev, bio);
}
}
@@ -3102,6 +3123,273 @@ static int r5c_shrink_cache_memory(struct r5c_cache *cache, unsigned long size)
return 0;
}
+static ssize_t r5c_show_cache_memory(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->max_pages << PAGE_SHIFT);
+}
+
+static ssize_t r5c_store_cache_memory(struct mddev *mddev, const char *page,
+ size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+ LIST_HEAD(page_list);
+ u64 i;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new >>= PAGE_SHIFT;
+
+ if (new > cache->max_pages) {
+ i = cache->max_pages;
+ while (i < new) {
+ struct page *page = alloc_page(GFP_KERNEL);
+
+ if (!page)
+ break;
+ list_add(&page->lru, &page_list);
+ i++;
+ }
+
+ spin_lock_irq(&cache->pool_lock);
+ list_splice(&page_list, &cache->page_pool);
+ cache->free_pages += i - cache->max_pages;
+ cache->max_pages = i;
+ cache->total_pages = i;
+ r5c_calculate_watermark(cache);
+ spin_unlock_irq(&cache->pool_lock);
+ return len;
+ }
+ r5c_shrink_cache_memory(cache, new);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_memory = __ATTR(cache_memory,
+ S_IRUGO | S_IWUSR, r5c_show_cache_memory, r5c_store_cache_memory);
+
+int r5c_min_stripe_cache_size(struct r5c_cache *cache)
+{
+ struct r5conf *conf = cache->mddev->private;
+ return (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) *
+ cache->reclaim_batch;
+}
+
+static void r5c_set_reclaim_batch(struct r5c_cache *cache, int batch)
+{
+ struct mddev *mddev = cache->mddev;
+ struct r5conf *conf = mddev->private;
+ int size;
+
+ size = (cache->stripe_parity_pages << PAGE_SECTOR_SHIFT) * batch;
+ if (size > cache->reserved_space) {
+ cache->reserved_space = size;
+ mutex_lock(&cache->log.io_mutex);
+ cache->log.reserved_blocks = r5l_sector_to_block(&cache->log,
+ cache->reserved_space) + 1;
+ mutex_unlock(&cache->log.io_mutex);
+ r5c_wake_wait_reclaimer(cache,
+ RECLAIM_DISK_BACKGROUND);
+ } else {
+ mutex_lock(&cache->log.io_mutex);
+ cache->log.reserved_blocks -= r5l_sector_to_block(&cache->log,
+ cache->reserved_space - size);
+ mutex_unlock(&cache->log.io_mutex);
+ cache->reserved_space = size;
+ }
+
+ size = (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) * batch;
+
+ mddev_lock(mddev);
+ if (size > conf->max_nr_stripes)
+ raid5_set_cache_size(mddev, size);
+ mddev_unlock(mddev);
+
+ cache->reclaim_batch = batch;
+}
+
+static ssize_t r5c_show_cache_reclaim_batch(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%d\n", cache->reclaim_batch);
+}
+
+static ssize_t r5c_store_cache_reclaim_batch(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+
+ if (new > r5l_max_flush_stripes(&cache->log))
+ new = r5l_max_flush_stripes(&cache->log);
+
+ if (new != cache->reclaim_batch)
+ r5c_set_reclaim_batch(cache, new);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_reclaim_batch =
+ __ATTR(cache_reclaim_batch, S_IRUGO | S_IWUSR,
+ r5c_show_cache_reclaim_batch, r5c_store_cache_reclaim_batch);
+
+static ssize_t r5c_show_cache_stat(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld %lld %lld %lld %lld\n",
+ (u64)atomic64_read(&cache->in_cache_rq),
+ (u64)atomic64_read(&cache->in_cache_sectors),
+ (u64)atomic64_read(&cache->out_cache_rq),
+ (u64)atomic64_read(&cache->out_cache_sectors),
+ (u64)atomic64_read(&cache->read_cache_sectors));
+}
+
+static struct md_sysfs_entry r5c_cache_stat =
+ __ATTR(cache_stat, S_IRUGO, r5c_show_cache_stat, NULL);
+
+static ssize_t r5c_show_cache_disk_watermark(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->log.low_watermark *
+ cache->log.block_size);
+}
+
+static ssize_t r5c_store_cache_disk_watermark(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ struct r5l_log *log = &cache->log;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new /= log->block_size;
+
+ if (new * 3 / 2 >= log->total_blocks)
+ return -EINVAL;
+
+ mutex_lock(&log->io_mutex);
+ log->low_watermark = new;
+ log->high_watermark = new * 3 / 2;
+ mutex_unlock(&log->io_mutex);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_disk_watermark =
+ __ATTR(cache_disk_watermark, S_IRUGO | S_IWUSR,
+ r5c_show_cache_disk_watermark, r5c_store_cache_disk_watermark);
+
+static ssize_t r5c_show_cache_memory_watermark(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->low_watermark << PAGE_SHIFT);
+}
+
+static ssize_t r5c_store_cache_memory_watermark(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new >>= PAGE_SHIFT;
+
+ if (new * 2 >= cache->max_pages)
+ return -EINVAL;
+
+ spin_lock_irq(&cache->pool_lock);
+ cache->low_watermark = new;
+ cache->high_watermark = new << 1;
+ spin_unlock_irq(&cache->pool_lock);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_memory_watermark =
+ __ATTR(cache_memory_watermark, S_IRUGO | S_IWUSR,
+ r5c_show_cache_memory_watermark, r5c_store_cache_memory_watermark);
+
+static int r5c_init_sysfs(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ int ret;
+
+ ret = sysfs_add_file_to_group(&mddev->kobj, &r5c_cache_memory.attr,
+ NULL);
+ if (ret)
+ return ret;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+ if (ret)
+ goto err_reclaim;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+ if (ret)
+ goto disk_watermark;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_stat.attr, NULL);
+ if (ret)
+ goto stat;
+
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_memory_watermark.attr, NULL);
+ if (ret)
+ goto memory_watermark;
+ return 0;
+memory_watermark:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_stat.attr, NULL);
+stat:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+disk_watermark:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+err_reclaim:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory.attr, NULL);
+ return ret;
+}
+
+static void r5c_exit_sysfs(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_stat.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory_watermark.attr, NULL);
+}
+
static void r5c_free_cache_data(struct r5c_cache *cache)
{
struct r5c_stripe *stripe;
@@ -3212,8 +3500,11 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
cache->reclaim_thread->timeout = CHECKPOINT_TIMEOUT;
r5c_shrink_cache_memory(cache, cache->max_pages);
-
+ if (r5c_init_sysfs(cache))
+ goto err_sysfs;
return cache;
+err_sysfs:
+ md_unregister_thread(&cache->reclaim_thread);
err_page:
r5c_free_cache_data(cache);
@@ -3232,6 +3523,7 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
void r5c_exit_cache(struct r5c_cache *cache)
{
+ r5c_exit_sysfs(cache);
md_unregister_thread(&cache->reclaim_thread);
r5l_exit_log(&cache->log);
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 6cfba8f..26561d8 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5907,6 +5907,9 @@ raid5_set_cache_size(struct mddev *mddev, int size)
if (size <= 16 || size > 32768)
return -EINVAL;
+ if (conf->cache && size < r5c_min_stripe_cache_size(conf->cache))
+ size = r5c_min_stripe_cache_size(conf->cache);
+
conf->min_nr_stripes = size;
while (size < conf->max_nr_stripes &&
drop_one_stripe(conf))
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 40307ca..f117b68 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -617,4 +617,5 @@ void r5c_exit_cache(struct r5c_cache *cache);
void r5c_write_start(struct mddev *mddev, struct bio *bi);
void r5c_write_end(struct mddev *mddev, struct bio *bi);
void r5c_quiesce(struct r5conf *conf, int state);
+int r5c_min_stripe_cache_size(struct r5c_cache *cache);
#endif
--
1.8.1
^ permalink raw reply related
* [PATCH v3 2/8] raid5: directly use mddev->queue
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1433356864.git.shli@fb.com>
When the cache layer flushes data from cache disk to raid disks, it will
dipsatch IO to raid disks. At that time, we don't have a block device
attached to the bio, so directly use mddev->queue. That should not
impact IO dispatched to rdev, which has rdev block device attached.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 12 +++++-------
1 file changed, 5 insertions(+), 7 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 553d54b..4c122ad 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -223,7 +223,7 @@ static int raid6_idx_to_slot(int idx, struct stripe_head *sh,
return slot;
}
-static void return_io(struct bio *return_bi)
+static void return_io(struct r5conf *conf, struct bio *return_bi)
{
struct bio *bi = return_bi;
while (bi) {
@@ -231,8 +231,7 @@ static void return_io(struct bio *return_bi)
return_bi = bi->bi_next;
bi->bi_next = NULL;
bi->bi_iter.bi_size = 0;
- trace_block_bio_complete(bdev_get_queue(bi->bi_bdev),
- bi, 0);
+ trace_block_bio_complete(conf->mddev->queue, bi, 0);
bio_endio(bi, 0);
bi = return_bi;
}
@@ -1200,7 +1199,7 @@ static void ops_complete_biofill(void *stripe_head_ref)
}
clear_bit(STRIPE_BIOFILL_RUN, &sh->state);
- return_io(return_bi);
+ return_io(sh->raid_conf, return_bi);
set_bit(STRIPE_HANDLE, &sh->state);
release_stripe(sh);
@@ -4594,7 +4593,7 @@ static void handle_stripe(struct stripe_head *sh)
md_wakeup_thread(conf->mddev->thread);
}
- return_io(s.return_bi);
+ return_io(conf, s.return_bi);
clear_bit_unlock(STRIPE_ACTIVE, &sh->state);
}
@@ -5298,8 +5297,7 @@ static void make_request(struct mddev *mddev, struct bio * bi)
if ( rw == WRITE )
md_write_end(mddev);
- trace_block_bio_complete(bdev_get_queue(bi->bi_bdev),
- bi, 0);
+ trace_block_bio_complete(mddev->queue, bi, 0);
bio_endio(bi, 0);
}
}
--
1.8.1
^ permalink raw reply related
* [PATCH v3 1/8] MD: add a new disk role to present cache device
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1433356864.git.shli@fb.com>
From: Song Liu <songliubraving@fb.com>
Next patches will use a disk as raid5/6 caching. We need a new disk role
to present the cache device
Not sure if we should bump up the MD superblock version for the disk
role.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.c | 14 +++++++++++++-
drivers/md/md.h | 4 ++++
include/uapi/linux/raid/md_p.h | 1 +
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 2750630..6297087 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -1656,6 +1656,9 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *rdev)
case 0xfffe: /* faulty */
set_bit(Faulty, &rdev->flags);
break;
+ case 0xfffd: /* cache device */
+ set_bit(WriteCache, &rdev->flags);
+ break;
default:
rdev->saved_raid_disk = role;
if ((le32_to_cpu(sb->feature_map) &
@@ -1811,6 +1814,8 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
sb->dev_roles[i] = cpu_to_le16(0xfffe);
else if (test_bit(In_sync, &rdev2->flags))
sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
+ else if (test_bit(WriteCache, &rdev2->flags))
+ sb->dev_roles[i] = cpu_to_le16(0xfffd);
else if (rdev2->raid_disk >= 0)
sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
else
@@ -5780,7 +5785,8 @@ static int get_disk_info(struct mddev *mddev, void __user * arg)
else if (test_bit(In_sync, &rdev->flags)) {
info.state |= (1<<MD_DISK_ACTIVE);
info.state |= (1<<MD_DISK_SYNC);
- }
+ } else if (test_bit(WriteCache, &rdev->flags))
+ info.state |= (1<<MD_DISK_WRITECACHE);
if (test_bit(WriteMostly, &rdev->flags))
info.state |= (1<<MD_DISK_WRITEMOSTLY);
} else {
@@ -5895,6 +5901,8 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
else
clear_bit(WriteMostly, &rdev->flags);
+ if (info->state & (1<<MD_DISK_WRITECACHE))
+ set_bit(WriteCache, &rdev->flags);
/*
* check whether the device shows up in other nodes
*/
@@ -7263,6 +7271,10 @@ static int md_seq_show(struct seq_file *seq, void *v)
seq_printf(seq, "(F)");
continue;
}
+ if (test_bit(WriteCache, &rdev->flags)) {
+ seq_printf(seq, "(C)");
+ continue;
+ }
if (rdev->raid_disk < 0)
seq_printf(seq, "(S)"); /* spare */
if (test_bit(Replacement, &rdev->flags))
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 4046a6c..6857592 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -175,6 +175,10 @@ enum flag_bits {
* This device is seen locally but not
* by the whole cluster
*/
+ WriteCache, /* This device is used as write cache.
+ * Usually, this device should be faster
+ * than other devices in the array
+ */
};
#define BB_LEN_MASK (0x00000000000001FFULL)
diff --git a/include/uapi/linux/raid/md_p.h b/include/uapi/linux/raid/md_p.h
index 2ae6131..9d36b91 100644
--- a/include/uapi/linux/raid/md_p.h
+++ b/include/uapi/linux/raid/md_p.h
@@ -89,6 +89,7 @@
* read requests will only be sent here in
* dire need
*/
+#define MD_DISK_WRITECACHE 18 /* disk is used as the write cache in RAID-5/6 */
typedef struct mdp_device_descriptor_s {
__u32 number; /* 0 Device number in the entire set */
--
1.8.1
^ permalink raw reply related
* [PATCH v3 0/8] MD: a caching layer for raid5/6
From: Shaohua Li @ 2015-06-03 22:48 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
Hi,
This is the third version of the raid5/6 caching layer patches. The patches add
a caching layer for raid5/6. The caching layer uses a SSD as a cache for a raid
5/6. It works like the similar way of a hardware raid controller. The purpose
is to improve raid performance (reduce read-modify-write) and fix write hole
issue. The main patch is patch 3 and the description has all details about the
implementation. Please review!
Thanks,
Shaohua
V3:
-make reclaim multi-thread
-add statistics in sysfs
-bug fixes
V2:
-metadata write doesn't use FUA
-discard request is only issued when necessary
-bug fixes and cleanup
Shaohua Li (7):
raid5: directly use mddev->queue
raid5: A caching layer for RAID5/6
raid5: add some sysfs entries
md: don't allow resize/reshape with cache support
raid5: skip resync if caching is enabled
raid5: guarantee cache release stripes in correct way
raid5: multi-thread support for raid5 caching reclaim
Song Liu (1):
MD: add a new disk role to present cache device
drivers/md/Makefile | 2 +-
drivers/md/md.c | 14 +-
drivers/md/md.h | 4 +
drivers/md/raid5-cache.c | 3775 ++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 177 +-
drivers/md/raid5.h | 25 +-
include/uapi/linux/raid/md_p.h | 73 +
7 files changed, 4022 insertions(+), 48 deletions(-)
create mode 100644 drivers/md/raid5-cache.c
--
1.8.1
^ permalink raw reply
* Re: 4.1-rc6 radi5 OOPS
From: NeilBrown @ 2015-06-03 22:15 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid, Xiao Ni
In-Reply-To: <wrfjzj4glajs.fsf@jes.lga.redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]
On Wed, 03 Jun 2015 17:57:43 -0400 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> NeilBrown <neilb@suse.de> writes:
> > On Wed, 03 Jun 2015 16:20:21 -0400 Jes Sorensen <Jes.Sorensen@redhat.com>
> > wrote:
> >
> >> Neil,
> >>
> >> I was running testing on the current 4.1-rc6 tree (Linus' top of trunk
> >> 8cd9234c64c584432f6992fe944ca9e46ca8ea76) and I am seeing the following
> >> OOPS which is reproducible.
> >>
> >> It shows up when running the mdadm test suite, 07changelevelintr to be
> >> specific.
> >>
> >> Is this something you have seen?
> >>
> >> Cheers,
> >> Jes
> >>
> >> ------------[ cut here ]------------
> >> kernel BUG at drivers/md/raid5.c:5391!
> >
> > No, I haven't seen that. And I've been running the test suite quite a bit
> > lately.
> >
> > Can you get it to print out the relevant numbers? Include
> > readpos/writepos/safepos too.
>
> This enough? Let me know if you need more.
>
> I suspect this started happening with the changes that went in between
> 4.1-rc5 and 4.1-rc6. I will try to bisect it tomorrow.
>
> Cheers,
> Jes
>
> mddev->dev_sectors: 0x9800, reshape_sectors: 0x0200 stripe_addr: fffffffffffffdff, sector_nr 0, readpos 511, writepos -513, safepos 512
Those negative numbers look VERY suspicious.
I'm actually on leave this week so I won't be looking at it any more, but
I'll see what I can find on Monday.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: 4.1-rc6 radi5 OOPS
From: Jes Sorensen @ 2015-06-03 21:57 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Xiao Ni
In-Reply-To: <20150604064048.0cb2d7c9@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Wed, 03 Jun 2015 16:20:21 -0400 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> Neil,
>>
>> I was running testing on the current 4.1-rc6 tree (Linus' top of trunk
>> 8cd9234c64c584432f6992fe944ca9e46ca8ea76) and I am seeing the following
>> OOPS which is reproducible.
>>
>> It shows up when running the mdadm test suite, 07changelevelintr to be
>> specific.
>>
>> Is this something you have seen?
>>
>> Cheers,
>> Jes
>>
>> ------------[ cut here ]------------
>> kernel BUG at drivers/md/raid5.c:5391!
>
> No, I haven't seen that. And I've been running the test suite quite a bit
> lately.
>
> Can you get it to print out the relevant numbers? Include
> readpos/writepos/safepos too.
This enough? Let me know if you need more.
I suspect this started happening with the changes that went in between
4.1-rc5 and 4.1-rc6. I will try to bisect it tomorrow.
Cheers,
Jes
mddev->dev_sectors: 0x9800, reshape_sectors: 0x0200 stripe_addr: fffffffffffffdff, sector_nr 0, readpos 511, writepos -513, safepos 512
------------[ cut here ]------------
kernel BUG at drivers/md/raid5.c:5394!
invalid opcode: 0000 [#1] SMP
Modules linked in: raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter ip_tables tun bridge stp llc x86_pkg_temp_thermal coretemp kvm_intel kvm xfs crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw libcrc32c nfsd gf128mul ablk_helper auth_rpcgss oid_registry exportfs nfs_acl lockd cryptd iTCO_wdt grace sunrpc microcode raid0 ppdev iTCO_vendor_support shpchp i2c_i801 i2c_core pcspkr lpc_ich mfd_core parport_pc acpi_cpufreq parport video uinput autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 libahci mii dm_mirror dm_region_hash dm_log dm_mod ipv6
CPU: 1 PID: 14648 Comm: md0_resync Not tainted 4.1.0-rc2+ #5
Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012
task: ffff880234b0f080 ti: ffff88022aadc000 task.ti: ffff88022aadc000
RIP: 0010:[<ffffffffa05db722>] [<ffffffffa05db722>] reshape_request+0x8d2/0x8e0 [raid456]
RSP: 0018:ffff88022aadfb68 EFLAGS: 00010296
RAX: 0000000000000087 RBX: ffff88022a8d2400 RCX: 0000000000000000
RDX: 0000000000000001 RSI: ffff88023ee2d368 RDI: ffff88023ee2d368
RBP: ffff88022aadfc28 R08: 0000000000000400 R09: ffffffff81d6c864
R10: 000000000000058f R11: 000000000000058e R12: ffff88022aadfd0c
R13: 0000000000000000 R14: ffff88022aadfd0c R15: ffff8800a778c800
FS: 0000000000000000(0000) GS:ffff88023ee20000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc8ddece000 CR3: 0000000001a0b000 CR4: 00000000001406e0
Stack:
fffffffffffffdff 0000000000000200 0000000000000000 0000000000000200
0000000000000001 0000000000000000 0000000000000001 fffffffffffffdff
0000020000000001 ffffffff815b2d23 ffff880234b0f080 ffff880233748f00
Call Trace:
[<ffffffff815b2d23>] ? __schedule+0x383/0x8e0
[<ffffffffa05dba5e>] sync_request+0x32e/0x3a0 [raid456]
[<ffffffff81092708>] ? __wake_up+0x48/0x60
[<ffffffff8148b564>] md_do_sync+0x8f4/0xe90
[<ffffffff81487888>] md_thread+0x128/0x140
[<ffffffff81487760>] ? find_pers+0x80/0x80
[<ffffffff81487760>] ? find_pers+0x80/0x80
[<ffffffff81071b49>] kthread+0xc9/0xe0
[<ffffffff810edd76>] ? __audit_syscall_exit+0x1e6/0x280
[<ffffffff81071a80>] ? kthread_create_on_node+0x170/0x170
[<ffffffff815b6b92>] ret_from_fork+0x42/0x70
[<ffffffff81071a80>] ? kthread_create_on_node+0x170/0x170
Code: 48 89 44 24 08 48 8b 85 78 ff ff ff 48 c7 c7 e0 3d 5e a0 4c 8b 85 68 ff ff ff 8b 55 84 48 89 04 24 48 89 c1 31 c0 e8 fe 10 fd e0 <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41
RIP [<ffffffffa05db722>] reshape_request+0x8d2/0x8e0 [raid456]
RSP <ffff88022aadfb68>
---[ end trace 7eefd62cedfb2b13 ]---
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff81073c2f>] exit_creds+0x1f/0x70
PGD 2365b1067 PUD 235280067 PMD 0
Oops: 0000 [#2] SMP
Modules linked in: raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter ip_tables tun bridge stp llc x86_pkg_temp_thermal coretemp kvm_intel kvm xfs crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw libcrc32c nfsd gf128mul ablk_helper auth_rpcgss oid_registry exportfs nfs_acl lockd cryptd iTCO_wdt grace sunrpc microcode raid0 ppdev iTCO_vendor_support shpchp i2c_i801 i2c_core pcspkr lpc_ich mfd_core parport_pc acpi_cpufreq parport video uinput autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 libahci mii dm_mirror dm_region_hash dm_log dm_mod ipv6
CPU: 1 PID: 14535 Comm: mdadm Tainted: G D 4.1.0-rc2+ #5
Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012
task: ffff8800a6b67100 ti: ffff880232ae4000 task.ti: ffff880232ae4000
RIP: 0010:[<ffffffff81073c2f>] [<ffffffff81073c2f>] exit_creds+0x1f/0x70
RSP: 0018:ffff880232ae7ca8 EFLAGS: 00010292
RAX: 0000000000000000 RBX: ffff880234b0f080 RCX: ffff880232ae7d00
RDX: 0000000000005a31 RSI: 0000000000000296 RDI: 0000000000000000
RBP: ffff880232ae7cb8 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000000b R11: 0000000000000246 R12: ffff880234b0f080
R13: 000000000000000b R14: ffff8802338f0000 R15: 0000000000000004
FS: 00007f70bfc97740(0000) GS:ffff88023ee20000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000234966000 CR4: 00000000001406e0
Stack:
ffff880232ae7cb8 ffff880234b0f080 ffff880232ae7cd8 ffffffff8105265a
0000000000000000 ffff880234b0f080 ffff880232ae7d08 ffffffff81072058
ffff880200000004 ffff8802361046c0 ffff8800a778c950 0000000000000004
Call Trace:
[<ffffffff8105265a>] __put_task_struct+0x4a/0x130
[<ffffffff81072058>] kthread_stop+0x88/0x100
[<ffffffff814878e5>] md_unregister_thread+0x45/0x80
[<ffffffff8148e32d>] md_reap_sync_thread+0x1d/0x1a0
[<ffffffff8148e650>] action_store+0x1a0/0x290
[<ffffffff8105cd5d>] ? ns_capable+0x2d/0x60
[<ffffffff8148bb7b>] md_attr_store+0x7b/0xd0
[<ffffffff8120f45d>] sysfs_kf_write+0x3d/0x50
[<ffffffff8120ebba>] kernfs_fop_write+0x12a/0x180
[<ffffffff81199878>] __vfs_write+0x28/0xf0
[<ffffffff8119c479>] ? __sb_start_write+0x49/0xf0
[<ffffffff81230873>] ? security_file_permission+0x23/0xa0
[<ffffffff81199f69>] vfs_write+0xa9/0x1b0
[<ffffffff8119ad36>] SyS_write+0x46/0xb0
[<ffffffff810edb34>] ? __audit_syscall_entry+0xb4/0x110
[<ffffffff815b67d7>] system_call_fastpath+0x12/0x6a
Code: 0f 84 37 fe ff ff e9 10 fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 87 c8 09 00 00 48 8b bf c0 09 00 00 <8b> 00 48 c7 83 c0 09 00 00 00 00 00 00 f0 ff 0f 74 1f 48 8b bb
RIP [<ffffffff81073c2f>] exit_creds+0x1f/0x70
RSP <ffff880232ae7ca8>
CR2: 0000000000000000
---[ end trace 7eefd62cedfb2b14 ]---
>
> Thanks,
> NeilBrown
>
>
>> invalid opcode: 0000 [#1] SMP
>> Modules linked in: raid456 async_raid6_recov async_memcpy async_pq
>> async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4
>> xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter
>> ip_tables tun bridge stp llc xfs x86_pkg_temp_thermal coretemp
>> kvm_intel kvm crc32c_intel ghash_clmulni_intel libcrc32c aesni_intel
>> aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ppdev raid0
>> microcode pcspkr iTCO_wdt iTCO_vendor_support parport_pc i2c_i801
>> i2c_core parport shpchp lpc_ich mfd_core video acpi_cpufreq nfsd
>> auth_rpcgss oid_registry exportfs nfs_acl lockd grace sunrpc uinput
>> autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 mii
>> libahci dm_mirror dm_region_hash dm_log dm_mod ipv6
>> CPU: 0 PID: 15142 Comm: md0_resync Tainted: G W 4.1.0-rc6+ #2
>> Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS
>> S1200BT.86B.02.00.0035.030220120927 03/02/2012
>> task: ffff88022b2642d0 ti: ffff8802327e0000 task.ti: ffff8802327e0000
>> RIP: 0010:[<ffffffffa046d6f7>] [<ffffffffa046d6f7>]
>> reshape_request+0x8a7/0x8b0 [raid456]
>> RSP: 0018:ffff8802327e3b78 EFLAGS: 00010297
>> RAX: 0000000000000200 RBX: ffff88022b130400 RCX: 0000000000000200
>> RDX: 0000000000009801 RSI: 0000000000000200 RDI: 0000000000000001
>> RBP: ffff8802327e3c28 R08: 00000000000001ff R09: 0000000000000000
>> R10: fffffffffffffdff R11: 0000000000000001 R12: ffff8802327e3d0c
>> R13: 0000000000000000 R14: ffff8802327e3d0c R15: ffff880234577000
>> FS: 0000000000000000(0000) GS:ffff88023ee00000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00007f985b677140 CR3: 0000000001a0b000 CR4: 00000000001406f0
>> Stack:
>> 0000000000000000 0000000000000200 0000000000000001 0000000000000000
>> 0000000000000000 fffffffffffffdff 0000020000000001 ffffffff815b34d3
>> ffff88022b2642d0 ffff880235f2f080 ffff8802327e3c08 ffff8802327e4000
>> Call Trace:
>> [<ffffffff815b34d3>] ? __schedule+0x383/0x8e0
>> [<ffffffffa046da2e>] sync_request+0x32e/0x3a0 [raid456]
>> [<ffffffff81092738>] ? __wake_up+0x48/0x60
>> [<ffffffff8148b974>] md_do_sync+0x8f4/0xe90
>> [<ffffffff810779bc>] ? update_rq_clock.part.89+0x1c/0x40
>> [<ffffffff81487c98>] md_thread+0x128/0x140
>> [<ffffffff81487b70>] ? find_pers+0x80/0x80
>> [<ffffffff81487b70>] ? find_pers+0x80/0x80
>> [<ffffffff81071b39>] kthread+0xc9/0xe0
>> [<ffffffff810eddd6>] ? __audit_syscall_exit+0x1e6/0x280
>> [<ffffffff81071a70>] ? kthread_create_on_node+0x170/0x170
>> [<ffffffff815b7352>] ret_from_fork+0x42/0x70
>> [<ffffffff81071a70>] ? kthread_create_on_node+0x170/0x170
>> Code: 00 00 be 03 00 00 00 e8 18 50 c2 e0 49 8d 7f 50 48 c7 c2 39 55
>> 47 a0 31 f6 e8 56 21 da e0 48 8b 45 88 e9 ae fb ff ff 0f 0b 0f 0b
>> <0f> 0b 0f 1f 80 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54
>> RIP [<ffffffffa046d6f7>] reshape_request+0x8a7/0x8b0 [raid456]
>> RSP <ffff8802327e3b78>
>> ---[ end trace 1d72a07e246912ea ]---
>> BUG: unable to handle kernel NULL pointer dereference at (null)
>> IP: [<ffffffff81073c1f>] exit_creds+0x1f/0x70
>> PGD bdc0c067 PUD a6697067 PMD 0
>> Oops: 0000 [#2] SMP
>> Modules linked in: raid456 async_raid6_recov async_memcpy async_pq
>> async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4
>> xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter
>> ip_tables tun bridge stp llc xfs x86_pkg_temp_thermal coretemp
>> kvm_intel kvm crc32c_intel ghash_clmulni_intel libcrc32c aesni_intel
>> aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ppdev raid0
>> microcode pcspkr iTCO_wdt iTCO_vendor_support parport_pc i2c_i801
>> i2c_core parport shpchp lpc_ich mfd_core video acpi_cpufreq nfsd
>> auth_rpcgss oid_registry exportfs nfs_acl lockd grace sunrpc uinput
>> autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 mii
>> libahci dm_mirror dm_region_hash dm_log dm_mod ipv6
>> CPU: 1 PID: 15030 Comm: mdadm Tainted: G D W 4.1.0-rc6+ #2
>> Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS
>> S1200BT.86B.02.00.0035.030220120927 03/02/2012
>> task: ffff880233550110 ti: ffff8800a6cbc000 task.ti: ffff8800a6cbc000
>> RIP: 0010:[<ffffffff81073c1f>] [<ffffffff81073c1f>] exit_creds+0x1f/0x70
>> RSP: 0018:ffff8800a6cbfca8 EFLAGS: 00010292
>> RAX: 0000000000000000 RBX: ffff88022b2642d0 RCX: ffff8800a6cbfd00
>> RDX: 0000000000005110 RSI: 0000000000000296 RDI: 0000000000000000
>> RBP: ffff8800a6cbfcb8 R08: 0000000000000000 R09: 0000000000000000
>> R10: 000000000000000b R11: 0000000000000246 R12: ffff88022b2642d0
>> R13: 000000000000ff00 R14: 0000000000000004 R15: 0000000000000004
>> FS: 00007f70ff26f740(0000) GS:ffff88023ee20000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000000000000000 CR3: 00000000a66cf000 CR4: 00000000001406e0
>> Stack:
>> ffff8800a6cbfcb8 ffff88022b2642d0 ffff8800a6cbfcd8 ffffffff8105264a
>> 0000000000000000 ffff88022b2642d0 ffff8800a6cbfd08 ffffffff81072048
>> ffff880000000001 ffff880233cb9140 ffff880234577150 ffffffffa0477320
>> Call Trace:
>> [<ffffffff8105264a>] __put_task_struct+0x4a/0x130
>> [<ffffffff81072048>] kthread_stop+0x88/0x100
>> [<ffffffff81487cf5>] md_unregister_thread+0x45/0x80
>> [<ffffffff8148e73d>] md_reap_sync_thread+0x1d/0x1a0
>> [<ffffffff8148ea68>] action_store+0x1a8/0x2b0
>> [<ffffffff8105cd4d>] ? ns_capable+0x2d/0x60
>> [<ffffffff8148bf8b>] md_attr_store+0x7b/0xd0
>> [<ffffffff8120f77d>] sysfs_kf_write+0x3d/0x50
>> [<ffffffff8120eeda>] kernfs_fop_write+0x12a/0x180
>> [<ffffffff81199ba8>] __vfs_write+0x28/0xf0
>> [<ffffffff8119c7a9>] ? __sb_start_write+0x49/0xf0
>> [<ffffffff81230b93>] ? security_file_permission+0x23/0xa0
>> [<ffffffff8119a299>] vfs_write+0xa9/0x1b0
>> [<ffffffff8119b066>] SyS_write+0x46/0xb0
>> [<ffffffff810edb94>] ? __audit_syscall_entry+0xb4/0x110
>> [<ffffffff815b6f97>] system_call_fastpath+0x12/0x6a
>> Code: 0f 84 37 fe ff ff e9 10 fe ff ff 90 0f 1f 44 00 00 55 48 89 e5
>> 53 48 89 fb 48 83 ec 08 48 8b 87 c8 09 00 00 48 8b bf c0 09 00 00
>> <8b> 00 48 c7 83 c0 09 00 00 00 00 00 00 f0 ff 0f 74 1f 48 8b bb
>> RIP [<ffffffff81073c1f>] exit_creds+0x1f/0x70
>> RSP <ffff8800a6cbfca8>
>> CR2: 0000000000000000
>> ---[ end trace 1d72a07e246912eb ]---
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox