* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-10 20:09 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux-RAID
In-Reply-To: <5669DB3B.30101@turmel.org>
On Thu, Dec 10, 2015 at 2:06 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/09/2015 08:34 PM, Dallas Clement wrote:
>> Hi all. I'm trying to determine best and worst case expected
>> sequential write speeds for Linux software RAID with spinning disks.
>>
>> I have been assuming on the following:
>>
>> Best case RAID 6 sequential write speed is (N-2) * X, where is is
>> number of drives and X is write speed of a single drive.
>>
>> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
>>
>> Best case RAID 5 sequential write speed is (N-1) * X.
>>
>> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
>>
>> Could someone please confirm whether these formulas are accurate or not?
>
> Confirm these? No. In fact, I see no theoretical basis for stating a
> worst case speed as half the best case speed. Or any other fraction.
> It's dependent on numerous variables -- block size, processor load, I/O
> bandwidth at various choke points (Northbridge, southbridge, PCI/PCIe,
> SATA/SAS channels, port mux...), I/O latency vs. queue depth vs. drive
> buffers, sector positioning at block boundaries, drive firmware
> housekeeping, etc.
>
> Where'd you get the worst case formulas?
>
> Where'd you get the worst case formulas?
Google search I'm afraid. I think the assumption for RAID 5,6 worst
case is having to read and write the parity + data every cycle.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Phil Turmel @ 2015-12-10 20:06 UTC (permalink / raw)
To: Dallas Clement, linux-raid
In-Reply-To: <CAE9DZUR=uSzYfdqFkVFdyXx+iKb1SeXxo5eX7M_nTw-fnWBwNA@mail.gmail.com>
On 12/09/2015 08:34 PM, Dallas Clement wrote:
> Hi all. I'm trying to determine best and worst case expected
> sequential write speeds for Linux software RAID with spinning disks.
>
> I have been assuming on the following:
>
> Best case RAID 6 sequential write speed is (N-2) * X, where is is
> number of drives and X is write speed of a single drive.
>
> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
>
> Best case RAID 5 sequential write speed is (N-1) * X.
>
> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
>
> Could someone please confirm whether these formulas are accurate or not?
Confirm these? No. In fact, I see no theoretical basis for stating a
worst case speed as half the best case speed. Or any other fraction.
It's dependent on numerous variables -- block size, processor load, I/O
bandwidth at various choke points (Northbridge, southbridge, PCI/PCIe,
SATA/SAS channels, port mux...), I/O latency vs. queue depth vs. drive
buffers, sector positioning at block boundaries, drive firmware
housekeeping, etc.
Where'd you get the worst case formulas?
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: John Stoffel @ 2015-12-10 19:33 UTC (permalink / raw)
To: Dallas Clement; +Cc: Mark Knecht, John Stoffel, Linux-RAID
In-Reply-To: <CAE9DZUQsNmsuk=7PjFwvADNj6KdifWt_5qArbhme3f5EmViqyA@mail.gmail.com>
>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
Dallas> I tried a more extreme case.
Dallas> Device Start End Sectors Size Type
Dallas> /dev/sda1 2048 6999998464 6999996417 3.3T Linux filesystem
Dallas> /dev/sda2 7000000512 7814037134 814036623 388.2G Linux filesystem
Dallas> Now I'm seeing quite a bit more difference between inner and outer.
Dallas> [root@localhost ~]# dd if=/dev/zero of=/dev/sda1 bs=2048k count=1000
Dallas> 1000+0 records in
Dallas> 1000+0 records out
Dallas> 2097152000 bytes (2.1 GB) copied, 13.4422 s, 156 MB/s
Dallas> [root@localhost ~]# dd if=/dev/zero of=/dev/sda2 bs=2048k count=1000
Dallas> 1000+0 records in
Dallas> 1000+0 records out
Dallas> 2097152000 bytes (2.1 GB) copied, 21.9703 s, 95.5 MB/s
This is actually one of the tricks people used to do before there was
ready availability of SSDs. They would buy a bunch of disks and then
only use the outer tracks, while striping data across a whole bunch of
disks to get up the IOPs when they were IOP but not space limited.
Think databases with lots and lots of transactions.
Now it's simpler to just A) buy lots and lots of memory, B) bunches of
SSDs, C) both, D) beat the developers until they learn to write better
SQL.
Sorry, D) never happens. :-)
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: John Stoffel @ 2015-12-10 19:28 UTC (permalink / raw)
To: Dallas Clement; +Cc: John Stoffel, Linux-RAID
In-Reply-To: <CAE9DZUTO-Muvgp+ofAkoKD1KcunnZ821UeSJch4216iWYL61cQ@mail.gmail.com>
>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
Dallas> On Thu, Dec 10, 2015 at 9:14 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>>
Dallas> Hi all. I'm trying to determine best and worst case expected
Dallas> sequential write speeds for Linux software RAID with spinning disks.
>>
Dallas> I have been assuming on the following:
>>
Dallas> Best case RAID 6 sequential write speed is (N-2) * X, where is is
Dallas> number of drives and X is write speed of a single drive.
>>
Dallas> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
>>
Dallas> Best case RAID 5 sequential write speed is (N-1) * X.
>>
Dallas> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
>>
Dallas> Could someone please confirm whether these formulas are accurate or not?
>>
>>
Dallas> I am not even getting worst case write performance with an
Dallas> array of 12 spinning 7200 RPM SATA disks. Thus I suspect
Dallas> either the formulas I am using are wrong or I have alignment
Dallas> issues or something. My chunk size is 128 KB at the moment.
>>
>> I think you're over-estimating the speed of your disks. Remember that
>> disk speeds are faster on the outer tracks of the drive, and slower on
>> the inner tracks.
>>
>> I'd setup two partitions, one at the start and one at the outside and
>> do some simple:
>>
>> dd if=/dev/zero of=/dev/inner,outer bs=8192 count=100000 oflag=direct
>>
>> and look at those numbers. Then build up a table where you vary the
>> bs= from 512 to N, which could be whatever you want.
>>
>> That will give you a better estimate of individual drive performance.
>>
>> Then when you do your fio tests, vary the queue depth, block size,
>> inner/outer partition, etc, but all on a single disk at first to
>> compare with the first set of results and to see how they correlate.
>>
>> THEN you can start looking at the RAID performance numbers.
>>
>> And of course, the controller you use matters, how it's configured,
>> how it's setup for caching, etc. Lots and lots and lots of details to
>> be tracked.
>>
>> Change one thing at a time, then re-run your tests. Automating them
>> is key here.
>>
>>
Dallas> Hi John. Thanks for the help. I did what you recommended and created
Dallas> two equal size partitions on my Hitachi 4TB 7200RPM SATA disks.
Dallas> Device Start End Sectors Size Type
Dallas> /dev/sda1 2048 3907014656 3907012609 1.8T Linux filesystem
Dallas> /dev/sda2 3907016704 7814037134 3907020431 1.8T Linux filesystem
I would do it even differently, put a 10g partition at each end and
run your tests.
Dallas> I ran the dd test with varying block size. I started to see a
Dallas> difference in write speed with larger block size.
You will... that's the streaming write speed. But in real life,
unless you're streaming video or other large large files, you're never
doing to see that.
Dallas> [root@localhost ~]# dd if=/dev/zero of=/dev/sda1 bs=2048k count=1000
Dallas> oflag=direct
Dallas> 1000+0 records in
Dallas> 1000+0 records out
Dallas> 2097152000 bytes (2.1 GB) copied, 11.5475 s, 182 MB/s
Dallas> [root@localhost ~]# dd if=/dev/zero of=/dev/sda2 bs=2048k count=1000
Dallas> oflag=direct
Dallas> 1000+0 records in
Dallas> 1000+0 records out
Dallas> 2097152000 bytes (2.1 GB) copied, 13.6355 s, 154 MB/s
It will be an even higher difference if you move the partitions to the
ends even more.
Dallas> The difference is not as great as I suspected it might be. If
Dallas> I plug in this lower write speed of 154 MB/s in the RAID 6
Dallas> worst case write speed calculation mentioned earlier, I should
Dallas> be getting at least (12 - 2) * 154 MB/s / 2 = 770 MB/s. For
Dallas> this same bs=2048k and queue_depth=256 I am getting 678 MB/s
Dallas> which is almost 100 MB/s less than worst case.
At this point, you need to now look at your controllers and
motherboard and how they're configured. If all those drives are on
one controller, and if that controller is on a single lane of PCIe,
then you will see controller bandwidth issues as well.
So now you need to step back and look at the entire system. How is
the drive cabled? What is the system powered with?
Also, Linux RAID only recently, in recent linux kernels, got away from
single threaded RAID56 compute threads, so that could be an impact
too.
The best way would be to have your disks spread out across multiple
controllers, on multiple busses, all talking in parallel.
If you're looking for a more linear speedup test, build a small 10g
partition on each disk, then build a RAID0 linear stripped array, but
with the small stride number. Then you do your sequential write and
you should see a pretty linear increase in speed, up until you hit
controller, memory, cpu, SATA limits.
Another option, if you're looking for good performance, might be to
look at lvcache, which is what I've just done at home. I have a pair
of mirrored 4Tb disks, and a pair of mirrored 500Gb SSDs which I used
for boot, /, /var and cache. So far I'm quite happy with the
performance speedup. But I also haven't done *any* rigorous testing,
since I'm more concerned about durability first, then speed.
John
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-10 19:26 UTC (permalink / raw)
To: Mark Knecht; +Cc: John Stoffel, Linux-RAID
In-Reply-To: <CAK2H+ed+fe5Wr0B=h5AzK5_=ougQtW_6cJcUG_S_cg+WfzDb=Q@mail.gmail.com>
On Thu, Dec 10, 2015 at 12:54 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Thu, Dec 10, 2015 at 10:40 AM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
>>
>> <SNIP>
>>
>> Hi John. Thanks for the help. I did what you recommended and created
>> two equal size partitions on my Hitachi 4TB 7200RPM SATA disks.
>>
>> Device Start End Sectors Size Type
>> /dev/sda1 2048 3907014656 3907012609 1.8T Linux filesystem
>> /dev/sda2 3907016704 7814037134 3907020431 1.8T Linux filesystem
>>
>
> If you're goal was to test the speed at the end of the drive it seems to me
> that sda2
> should have started near the end of the drive and not presumably in the
> middle?
>
> Probably has nothing to do with the missing 100MB/s though.
>
> - Mark
I tried a more extreme case.
Device Start End Sectors Size Type
/dev/sda1 2048 6999998464 6999996417 3.3T Linux filesystem
/dev/sda2 7000000512 7814037134 814036623 388.2G Linux filesystem
Now I'm seeing quite a bit more difference between inner and outer.
[root@localhost ~]# dd if=/dev/zero of=/dev/sda1 bs=2048k count=1000
1000+0 records in
1000+0 records out
2097152000 bytes (2.1 GB) copied, 13.4422 s, 156 MB/s
[root@localhost ~]# dd if=/dev/zero of=/dev/sda2 bs=2048k count=1000
1000+0 records in
1000+0 records out
2097152000 bytes (2.1 GB) copied, 21.9703 s, 95.5 MB/s
Using this this worst case inner speed my expected worst case write
speed is (12 - 2) * 95.5 MB/s / 2 = 477.5 MB/s. So I guess if this
really is the worst case, then the 678 MB/s sequential write speed I
am seeing with RAID 6 on a large partition is not so bad.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-10 18:40 UTC (permalink / raw)
To: John Stoffel; +Cc: Linux-RAID
In-Reply-To: <22121.38606.101474.41382@quad.stoffel.home>
On Thu, Dec 10, 2015 at 9:14 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>
> Dallas> Hi all. I'm trying to determine best and worst case expected
> Dallas> sequential write speeds for Linux software RAID with spinning disks.
>
> Dallas> I have been assuming on the following:
>
> Dallas> Best case RAID 6 sequential write speed is (N-2) * X, where is is
> Dallas> number of drives and X is write speed of a single drive.
>
> Dallas> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
>
> Dallas> Best case RAID 5 sequential write speed is (N-1) * X.
>
> Dallas> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
>
> Dallas> Could someone please confirm whether these formulas are accurate or not?
>
>
> Dallas> I am not even getting worst case write performance with an
> Dallas> array of 12 spinning 7200 RPM SATA disks. Thus I suspect
> Dallas> either the formulas I am using are wrong or I have alignment
> Dallas> issues or something. My chunk size is 128 KB at the moment.
>
> I think you're over-estimating the speed of your disks. Remember that
> disk speeds are faster on the outer tracks of the drive, and slower on
> the inner tracks.
>
> I'd setup two partitions, one at the start and one at the outside and
> do some simple:
>
> dd if=/dev/zero of=/dev/inner,outer bs=8192 count=100000 oflag=direct
>
> and look at those numbers. Then build up a table where you vary the
> bs= from 512 to N, which could be whatever you want.
>
> That will give you a better estimate of individual drive performance.
>
> Then when you do your fio tests, vary the queue depth, block size,
> inner/outer partition, etc, but all on a single disk at first to
> compare with the first set of results and to see how they correlate.
>
> THEN you can start looking at the RAID performance numbers.
>
> And of course, the controller you use matters, how it's configured,
> how it's setup for caching, etc. Lots and lots and lots of details to
> be tracked.
>
> Change one thing at a time, then re-run your tests. Automating them
> is key here.
>
>
Hi John. Thanks for the help. I did what you recommended and created
two equal size partitions on my Hitachi 4TB 7200RPM SATA disks.
Device Start End Sectors Size Type
/dev/sda1 2048 3907014656 3907012609 1.8T Linux filesystem
/dev/sda2 3907016704 7814037134 3907020431 1.8T Linux filesystem
I ran the dd test with varying block size. I started to see a
difference in write speed with larger block size.
[root@localhost ~]# dd if=/dev/zero of=/dev/sda1 bs=2048k count=1000
oflag=direct
1000+0 records in
1000+0 records out
2097152000 bytes (2.1 GB) copied, 11.5475 s, 182 MB/s
[root@localhost ~]# dd if=/dev/zero of=/dev/sda2 bs=2048k count=1000
oflag=direct
1000+0 records in
1000+0 records out
2097152000 bytes (2.1 GB) copied, 13.6355 s, 154 MB/s
The difference is not as great as I suspected it might be. If I plug
in this lower write speed of 154 MB/s in the RAID 6 worst case write
speed calculation mentioned earlier, I should be getting at least (12
- 2) * 154 MB/s / 2 = 770 MB/s. For this same bs=2048k and
queue_depth=256 I am getting 678 MB/s which is almost 100 MB/s less
than worst case.
^ permalink raw reply
* Re: Reshape stalls and fails when SELinux is enabled
From: Wols Lists @ 2015-12-10 18:24 UTC (permalink / raw)
To: Edward Kuns, Enrico Tagliavini; +Cc: Linux-RAID
In-Reply-To: <CACsGCyQX4DKfT-6CYd7pa8P-Rq12HhP2tA79F27rw1PbNHe_QQ@mail.gmail.com>
On 10/12/15 17:06, Edward Kuns wrote:
> I ran into this as well. I unwisely worked around it by placing the
> backup file on /tmp. I don't recommend that choice!
Most people don't realise there are various tmp directories, with
different behaviours. /tmp is defined by LSB or somesuch to be real
temporary storage - no guarantees - stuff may disappear at any time.
Which is why my /tmp is a tmpfs - stored in ram. So yes, a backup file
on /tmp is not a good idea ...
/var/tmp is defined as holding data that must survive a reboot - where
emacs and vi and that lot are recommended to store their crash dumps,
recovery files, etc for example.
I didn't see
> your AVC on your EMail, but I recommend you reach out to the SELinux
> folks for labeling help. I've reach out to them in the past and have
> found them very helpful (as are the folks here). Getting the labeling
> corrected will not only help you but everyone else in the long term.
>
> I agree that it would be good for mdadm to handle this kind of failure
> in a more productive fashion as well. But I'm only a lurker here. I
> don't have enough knowledge to suggest what would be better. One
> thing that really helped me debug this was commenting out the lines
>
> #StandardOutput=null
> #StandardError=null
>
> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
> -- note you have to run systemctl daemon-reload after changing the
> file for the change to be noticed.
>
> Once I did that, the system logs had much more useful information.
> I'm not sure where that file is sourced, whether it's from the
> linux-raid people or from the distributions or somewhere in between.
> Does anyone here know if there's a reason why standard out and
> standard error are sent to /dev/null for this service?
>
I wouldn't know for certain, but given the amount of junk so many
programs spew to stdout or stderr (which nobody ever sees because
they're running in a gui), I expect it's because that's what happens by
default anyway :-) They don't want to clutter the logs with all that
junk and chatter.
I often somehow get stdout and stderr for various programs redirected to
konsole, because I'll use dbus-launch to start a program. And then when
the program has exited I'll get loads of rubbish from other programs ...
the chatter is horrendous.
Cheers,
Wol
^ permalink raw reply
* Re: Reshape stalls and fails when SELinux is enabled
From: Edward Kuns @ 2015-12-10 17:06 UTC (permalink / raw)
To: Enrico Tagliavini; +Cc: Linux-RAID
In-Reply-To: <CAMQd7mYscUDroop=4eLz4UwTEiibbF0OqDN1b28yiqJyfW7gGw@mail.gmail.com>
I ran into this as well. I unwisely worked around it by placing the
backup file on /tmp. I don't recommend that choice! I didn't see
your AVC on your EMail, but I recommend you reach out to the SELinux
folks for labeling help. I've reach out to them in the past and have
found them very helpful (as are the folks here). Getting the labeling
corrected will not only help you but everyone else in the long term.
I agree that it would be good for mdadm to handle this kind of failure
in a more productive fashion as well. But I'm only a lurker here. I
don't have enough knowledge to suggest what would be better. One
thing that really helped me debug this was commenting out the lines
#StandardOutput=null
#StandardError=null
in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
-- note you have to run systemctl daemon-reload after changing the
file for the change to be noticed.
Once I did that, the system logs had much more useful information.
I'm not sure where that file is sourced, whether it's from the
linux-raid people or from the distributions or somewhere in between.
Does anyone here know if there's a reason why standard out and
standard error are sent to /dev/null for this service?
Eddie
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: Phil Turmel @ 2015-12-10 16:55 UTC (permalink / raw)
To: P. Remek, linux-raid
In-Reply-To: <CABdHLQ6Lo8WvgSf=MAuYdVoJJMHmLuV5gkE+N4MDMfxvAbG5OA@mail.gmail.com>
On 12/10/2015 10:29 AM, P. Remek wrote:
> Hello,
>
> I am trying to create mdadm raid on top of dm-multipath devices.
> Everything works but after reboot the mdadm array is assembled from
> the original (not multipath devices) devices and the multipath devices
> are not created at all.
>
> Aparently it is because the mdadm software raid is started before the
> dm-multipath creates the multipathed devices. Is there a way how to
> make the dm-multipath start before the mdadm is started?
Order of device discovery is not guaranteed, but base devices will
almost always show up before multipath devices. You have to filter out
the base devices from mdadm consideration:
Add a DEVICES statement to your mdadm.conf that only matches your
multipath device names, not the base names. Then update your initramfs
so it applies to early boot as well.
Phil
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: John Stoffel @ 2015-12-10 16:28 UTC (permalink / raw)
To: P. Remek; +Cc: linux-raid
In-Reply-To: <CABdHLQ6Lo8WvgSf=MAuYdVoJJMHmLuV5gkE+N4MDMfxvAbG5OA@mail.gmail.com>
P> I am trying to create mdadm raid on top of dm-multipath devices.
P> Everything works but after reboot the mdadm array is assembled from
P> the original (not multipath devices) devices and the multipath
P> devices are not created at all.
Can you give more details about the OS, kernel version and
configuration of the setup?
It sounds like you have some local disks and some remote disks that
you want to use in the RAID. But it's not clear how you're setting
things up.
P> Aparently it is because the mdadm software raid is started before
P> the dm-multipath creates the multipathed devices. Is there a way
P> how to make the dm-multipath start before the mdadm is started?
Hmm... I think devicemapper should be starting pretty early. Do you
have your initramfs setup properly? You might need to setup things so
they are working properly, the do:
update-grub2
to make sure the initramfs gets properly updated. You might also want
to make sure that LVM is compiled into your kernel, but MDADM is a
module, which should force them to be started in the proper order.
But really, you need to provide details. Can you setup a small test
MD RAID and show all the details?
John
^ permalink raw reply
* start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-10 15:29 UTC (permalink / raw)
To: linux-raid
Hello,
I am trying to create mdadm raid on top of dm-multipath devices.
Everything works but after reboot the mdadm array is assembled from
the original (not multipath devices) devices and the multipath devices
are not created at all.
Aparently it is because the mdadm software raid is started before the
dm-multipath creates the multipathed devices. Is there a way how to
make the dm-multipath start before the mdadm is started?
Regards,
Remek
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: John Stoffel @ 2015-12-10 15:14 UTC (permalink / raw)
To: Dallas Clement; +Cc: linux-raid
In-Reply-To: <CAE9DZUR=uSzYfdqFkVFdyXx+iKb1SeXxo5eX7M_nTw-fnWBwNA@mail.gmail.com>
>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
Dallas> Hi all. I'm trying to determine best and worst case expected
Dallas> sequential write speeds for Linux software RAID with spinning disks.
Dallas> I have been assuming on the following:
Dallas> Best case RAID 6 sequential write speed is (N-2) * X, where is is
Dallas> number of drives and X is write speed of a single drive.
Dallas> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
Dallas> Best case RAID 5 sequential write speed is (N-1) * X.
Dallas> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
Dallas> Could someone please confirm whether these formulas are accurate or not?
Dallas> I am not even getting worst case write performance with an
Dallas> array of 12 spinning 7200 RPM SATA disks. Thus I suspect
Dallas> either the formulas I am using are wrong or I have alignment
Dallas> issues or something. My chunk size is 128 KB at the moment.
I think you're over-estimating the speed of your disks. Remember that
disk speeds are faster on the outer tracks of the drive, and slower on
the inner tracks.
I'd setup two partitions, one at the start and one at the outside and
do some simple:
dd if=/dev/zero of=/dev/inner,outer bs=8192 count=100000 oflag=direct
and look at those numbers. Then build up a table where you vary the
bs= from 512 to N, which could be whatever you want.
That will give you a better estimate of individual drive performance.
Then when you do your fio tests, vary the queue depth, block size,
inner/outer partition, etc, but all on a single disk at first to
compare with the first set of results and to see how they correlate.
THEN you can start looking at the RAID performance numbers.
And of course, the controller you use matters, how it's configured,
how it's setup for caching, etc. Lots and lots and lots of details to
be tracked.
Change one thing at a time, then re-run your tests. Automating them
is key here.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-10 14:38 UTC (permalink / raw)
To: Alexander Afonyashin; +Cc: Linux-RAID
In-Reply-To: <CAETWcfs_UmRKi+x3Wf+z2aaG6+x-qyCgBHk+OUdjE7r0XpCYEw@mail.gmail.com>
On Thu, Dec 10, 2015 at 12:36 AM, Alexander Afonyashin
<a.afonyashin@madnet-team.ru> wrote:
> Hi,
>
> Did you set stride/stripe configuration to filesystem that is aligned
> with raid configuration?
>
> Regards,
> Alexander
>
> On Thu, Dec 10, 2015 at 4:34 AM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
>> Hi all. I'm trying to determine best and worst case expected
>> sequential write speeds for Linux software RAID with spinning disks.
>>
>> I have been assuming on the following:
>>
>> Best case RAID 6 sequential write speed is (N-2) * X, where is is
>> number of drives and X is write speed of a single drive.
>>
>> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
>>
>> Best case RAID 5 sequential write speed is (N-1) * X.
>>
>> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
>>
>> Could someone please confirm whether these formulas are accurate or not?
>>
>>
>> I am not even getting worst case write performance with an array of 12
>> spinning 7200 RPM SATA disks. Thus I suspect either the formulas I
>> am using are wrong or I have alignment issues or something. My chunk
>> size is 128 KB at the moment.
>>
>> Thanks,
>>
>> Dallas
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Did you set stride/stripe configuration to filesystem that is aligned
with raid configuration?
Hi Alexander. I've just been reading / writing the raw raid device
with no filesystem at the moment. Using fio, with direct=1. I seem
to get the best results with bs=512k and queue_depth=256. Even with
this, I am still short of the worst case write performance.
^ permalink raw reply
* Re: Reshape stalls and fails when SELinux is enabled
From: Enrico Tagliavini @ 2015-12-10 9:00 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAMQd7mbdqsw5BiRSKbgBqwNFrz4V8T-A3hOftWReRk1XgXc=+Q@mail.gmail.com>
Looks like I'm not the only one and it happens in real world cases,
not just a test on a VM:
https://marc.info/?l=linux-raid&m=144973573922461&w=2
Is there a place where I can open a bug to keep track of this issue?
On the wiki it says that "Linux RAID issues are discussed in the
linux-raid mailing list to be found at
http://vger.kernel.org/vger-lists.html#linux-raid", but it's easy to
loose track eventually. Can't find a component "mdadm" at
https://bugzilla.kernel.org/ maybe one should be created under the
"Tools" product?
Thank you.
Best regards.
Enrico Tagliavini
On 3 December 2015 at 12:04, Enrico Tagliavini
<enrico.tagliavini@gmail.com> wrote:
> Hi everybody,
>
> yesterday I tested a conversion of a 2 disk raid1 to a 4 disk raid5.
> The reshape never completed, I rebooted the VM and the raid device was
> unrecoverable. Backup-file was not useful unfortunately. Take a
> comfortable seat, this is a bit long story, sorry :).
>
> Environment:
> - VM running on my laptop (Fedora 22), created with virt-manager,
> qemu based. All disk images are qcow2 virtio devices.
> - VM OS: centos 7 64 bit, fully updated as of today.
> - mdadm version: 3.3.2-2.el7_1.1
> - kernel version: 3.10.0-229.20.1.el7
> - selinux policy targeted: 3.13.1-23.el7_1.21
>
> How to reproduce:
>
> * Attach four 1 GB disks to the VM, in the following example they are
> vd[b,c,d,e].
> * create a raid1 device with two disks: mdadm --create /dev/md0 -e
> 1.2 -n 2 -l raid1 -N raidtest /dev/vdb /dev/vdc
> * [optional?]: I created a FS on it: mkfs.xfs -L test /dev/md0
> * [optional?]: I created three files on it and noted the checksum, to
> check for eventual corruptions
> - mount /dev/md0 /mnt/raidtest/ && cd /mnt/raidtest/
> - echo uno > first.txt ; echo due > second.txt ; touch third.txt
> ; shred -s 512M -f third.txt
> - for file in *; do sha1sum $file >> sha1sum.txt; done
> * check /proc/mdstat to make sure the raid1 sync operation is
> complete (should really be fast but better safe than sorry)
> * unmount /mnt/raidtest/
> * add the two additional disks as spares:
> - mdadm --manage /dev/md0 --add-spare /dev/vdd
> - mdadm --manage /dev/md0 --add-spare /dev/vde
> * grow and reshape: mdadm --grow /dev/md0 -n 4 --level=5
> --backup-file /root/backup-md0 # (I also tried with
> /var/local/backup-md0)
>
> At this point an AVC will happen. Even if I'm in an interactive
> session and, as such SELinux should not limit it normally, mdadm
> process switches to mdadm_t type (maybe a forked process with its own
> session group?) and is not allowed to write a file in the /root/ or
> /var/ folders. This is ok, however mdadm keeps going instead of
> aborting the reshape. It's running without a backup file, that's not
> what the admin asked for since the --backup-file option is specified.
> But even worst than this is that my reshape got stuck and never
> completed. I waited a couple of hours but it remained at 0%. Something
> was actually written to the backup file (which is weird given the AVC,
> but it can be the original mdadm process not running under mdadm_t).
>
> At this point I was kind of curious to test what would happen if a
> distracted admin like me wont notice the problem and, days later,
> would reboot the server due to security updates or anything else. The
> result is an unrecoverable md array. I tried to assemble it back with
> the backup file
>
> mdadm --assemble /dev/md0 -u 6f53ec3e:d9868fef:12d3e243:8489561b
> --backup-file /root/backup-md
>
> But no way
>
> mdadm: [sorry I copied wrong and the device name was lost] has an
> active reshape - checking if critical section needs to be restored
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
>
> I retried the entire procedure from scratch, but this time with before
> mdadm --grow I set SELinux in permissive mode with setenforce 0.
> Everything was butter smooth this time. Reshape was almost instant for
> such a small array, data was checksumming correctly and my array was
> level 5.
>
> Now there might be a problem with the SELinux policy here, but
> honestly I think mdadm should just abort, whatever the reason of the
> problem was. There might be other scenarios not involving SELinux
> causing the same problem.
>
> It would also be nice to suggest the user, if SELinux is active, to
> change the context of the backup file to something SELinux will permit
> (mdadm_map_t, mdadm_var_run_t?).
>
> Attached you can also find the AVC denial for my entire day of testing.
>
> Thank you for your help.
> Best regards.
>
> Enrico Tagliavini
^ permalink raw reply
* Re: RAID 6 "Failed to restore critical section for reshape, sorry." - recovery advice?
From: Mikael Abrahamsson @ 2015-12-10 8:22 UTC (permalink / raw)
To: George Rapp; +Cc: linux-raid
In-Reply-To: <CAF-Kpgb57-LoNy=-jy76ZcE=YPABmARzfsHNitJNZyw3v2AShw@mail.gmail.com>
On Wed, 9 Dec 2015, George Rapp wrote:
> # mdadm --assemble --verbose --invalid-backup --force
> --update=revert-reshape /dev/md4 $OVERLAYS
> mdadm: looking for devices for /dev/md4
> mdadm: /dev/mapper/sdg4 is identified as a member of /dev/md4, slot 2.
> mdadm: /dev/mapper/sdi1 is identified as a member of /dev/md4, slot 5.
> mdadm: /dev/mapper/sdc4 is identified as a member of /dev/md4, slot 1.
> mdadm: /dev/mapper/sdh1 is identified as a member of /dev/md4, slot 3.
> mdadm: /dev/mapper/sdj1 is identified as a member of /dev/md4, slot 4.
> mdadm: /dev/mapper/sdd4 is identified as a member of /dev/md4, slot 0.
> mdadm: /dev/md4 has an active reshape - checking if critical section
> needs to be restored
> mdadm: Failed to find backup of critical section
> mdadm: continuing without restoring backup
> mdadm: added /dev/mapper/sdc4 to /dev/md4 as 1
> mdadm: added /dev/mapper/sdg4 to /dev/md4 as 2
> mdadm: added /dev/mapper/sdh1 to /dev/md4 as 3
> mdadm: added /dev/mapper/sdj1 to /dev/md4 as 4
> mdadm: added /dev/mapper/sdi1 to /dev/md4 as 5
> mdadm: added /dev/mapper/sdd4 to /dev/md4 as 0
> mdadm: /dev/md4: Need a backup file to complete reshape of this array.
> mdadm: Please provided one with "--backup-file=..."
> mdadm: (Don't specify --update=revert-reshape again, that part succeeded.)
>
> How can the array have an active reshape if the reshape pos'n is 0 on
> all devices? Doesn't that mean that the reshape never actually
> started? If so, can I just revert -- somehow -- to a 5-device RAID 6
> array to recover my data?
Just a shot in the dark, what happens if you add a backup file to the
above command, but without revert-reshape? Ie state both --invalid-backup
but also supply a backup-file. The text above could indicate that this
might help.
If you get the array up and running again but it's still reshaping but at
position 0, issue a --continue to it (this has worked for others). Also, I
would get the latest git version of mdadm and try with that one if you're
still using v3.2.2 as per the link to your august original email.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Alexander Afonyashin @ 2015-12-10 6:36 UTC (permalink / raw)
To: Dallas Clement; +Cc: Linux-RAID
In-Reply-To: <CAE9DZUR=uSzYfdqFkVFdyXx+iKb1SeXxo5eX7M_nTw-fnWBwNA@mail.gmail.com>
Hi,
Did you set stride/stripe configuration to filesystem that is aligned
with raid configuration?
Regards,
Alexander
On Thu, Dec 10, 2015 at 4:34 AM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> Hi all. I'm trying to determine best and worst case expected
> sequential write speeds for Linux software RAID with spinning disks.
>
> I have been assuming on the following:
>
> Best case RAID 6 sequential write speed is (N-2) * X, where is is
> number of drives and X is write speed of a single drive.
>
> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
>
> Best case RAID 5 sequential write speed is (N-1) * X.
>
> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
>
> Could someone please confirm whether these formulas are accurate or not?
>
>
> I am not even getting worst case write performance with an array of 12
> spinning 7200 RPM SATA disks. Thus I suspect either the formulas I
> am using are wrong or I have alignment issues or something. My chunk
> size is 128 KB at the moment.
>
> Thanks,
>
> Dallas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-10 1:34 UTC (permalink / raw)
To: linux-raid
Hi all. I'm trying to determine best and worst case expected
sequential write speeds for Linux software RAID with spinning disks.
I have been assuming on the following:
Best case RAID 6 sequential write speed is (N-2) * X, where is is
number of drives and X is write speed of a single drive.
Worst case RAID 6 sequential write speed is (N-2) * X / 2.
Best case RAID 5 sequential write speed is (N-1) * X.
Worst case RAID 5 sequential write speed is (N-1) * X / 2.
Could someone please confirm whether these formulas are accurate or not?
I am not even getting worst case write performance with an array of 12
spinning 7200 RPM SATA disks. Thus I suspect either the formulas I
am using are wrong or I have alignment issues or something. My chunk
size is 128 KB at the moment.
Thanks,
Dallas
^ permalink raw reply
* RAID 6 "Failed to restore critical section for reshape, sorry." - recovery advice?
From: George Rapp @ 2015-12-09 23:12 UTC (permalink / raw)
To: linux-raid
Hello RAID folks -
Reference my old thread at
http://marc.info/?l=linux-raid&m=143880359028232&w=2 which I'm just
now getting back around to working on, the delay having been caused by
my need to clone a failing disk.
Recall that I was attempting to grow a 5-disk RAID 6 array to 6 disks,
but writing to the backup-file was inhibited by SELinux.
# mdadm --add /dev/md4 /dev/sdi1
# mdadm --grow --raid-devices=6
--backup-file=/home/gwr/2015/2015-08/grow_md4.bak /dev/md4
The second command threw a bunch of SELinux errors about access to
/home/gwr/c/grow_md4.bak. The reshape operation sat for many minutes
at 0% progress, according to /proc/mdstat. However, the file
/home/gwr/c/grow_md4.bak *was* created with a size of 6295552 bytes/
I appeared to experience a segmentation fault or other runtime error
when attempting to stop the RAID 6 array with "#mdadm --stop
/dev/md4". System log from that situation has been uploaded to
https://app.box.com/s/3pksam3c7n79anpnzvsrwekzqwtsvlf6 Notably, the
backup file was created, but contains all zero/null characters.
My most recent raid.status file (generated using command "# mdadm
--examine /dev/sd[cdg]4 /dev/sd[hij]1" - and, yes, I know my partition
layout is a mess) has been uploaded to
https://app.box.com/s/pbienbpdanr0rq224b9ag2qu36vk76iv What I find
interesting about this status file is that the reshape of the array
appears to have made no progress (note the presence of " Reshape pos'n
: 0 " on all six devices).
I have been using the recovery advice found at
https://raid.wiki.kernel.org/index.php/RAID_Recovery and
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID,
especially the part about creating overlay files so as not to damage
my actual disks. The overlay devices are in the variable $OVERLAYS.
When I attempt to assemble the RAID 6 array using the backup file, I get this:
# mdadm --assemble --verbose --force
--backup-file=/home/gwr/2015/2015-08/grow_md6.bak /dev/md4 $OVERLAYS
mdadm: looking for devices for /dev/md4
mdadm: /dev/mapper/sdg4 is identified as a member of /dev/md4, slot 2.
mdadm: /dev/mapper/sdi1 is identified as a member of /dev/md4, slot 5.
mdadm: /dev/mapper/sdc4 is identified as a member of /dev/md4, slot 1.
mdadm: /dev/mapper/sdh1 is identified as a member of /dev/md4, slot 3.
mdadm: /dev/mapper/sdj1 is identified as a member of /dev/md4, slot 4.
mdadm: /dev/mapper/sdd4 is identified as a member of /dev/md4, slot 0.
mdadm: /dev/md4 has an active reshape - checking if critical section
needs to be restored
mdadm: No backup metadata on /home/gwr/2015/2015-08/grow_md6.bak
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
and when I omit the backup file, I get this:
# mdadm --assemble --verbose --force --invalid-backup /dev/md4 $OVERLAYS
mdadm: looking for devices for /dev/md4
mdadm: /dev/mapper/sdg4 is identified as a member of /dev/md4, slot 2.
mdadm: /dev/mapper/sdi1 is identified as a member of /dev/md4, slot 5.
mdadm: /dev/mapper/sdc4 is identified as a member of /dev/md4, slot 1.
mdadm: /dev/mapper/sdh1 is identified as a member of /dev/md4, slot 3.
mdadm: /dev/mapper/sdj1 is identified as a member of /dev/md4, slot 4.
mdadm: /dev/mapper/sdd4 is identified as a member of /dev/md4, slot 0.
mdadm: /dev/md4 has an active reshape - checking if critical section
needs to be restored
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/mapper/sdc4 to /dev/md4 as 1
mdadm: added /dev/mapper/sdg4 to /dev/md4 as 2
mdadm: added /dev/mapper/sdh1 to /dev/md4 as 3
mdadm: added /dev/mapper/sdj1 to /dev/md4 as 4
mdadm: added /dev/mapper/sdi1 to /dev/md4 as 5
mdadm: added /dev/mapper/sdd4 to /dev/md4 as 0
mdadm: /dev/md4: Need a backup file to complete reshape of this array.
mdadm: Please provided one with "--backup-file=..."
I even tried --update=revert-reshape; no luck:
# mdadm --assemble --verbose --invalid-backup --force
--update=revert-reshape /dev/md4 $OVERLAYS
mdadm: looking for devices for /dev/md4
mdadm: /dev/mapper/sdg4 is identified as a member of /dev/md4, slot 2.
mdadm: /dev/mapper/sdi1 is identified as a member of /dev/md4, slot 5.
mdadm: /dev/mapper/sdc4 is identified as a member of /dev/md4, slot 1.
mdadm: /dev/mapper/sdh1 is identified as a member of /dev/md4, slot 3.
mdadm: /dev/mapper/sdj1 is identified as a member of /dev/md4, slot 4.
mdadm: /dev/mapper/sdd4 is identified as a member of /dev/md4, slot 0.
mdadm: /dev/md4 has an active reshape - checking if critical section
needs to be restored
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/mapper/sdc4 to /dev/md4 as 1
mdadm: added /dev/mapper/sdg4 to /dev/md4 as 2
mdadm: added /dev/mapper/sdh1 to /dev/md4 as 3
mdadm: added /dev/mapper/sdj1 to /dev/md4 as 4
mdadm: added /dev/mapper/sdi1 to /dev/md4 as 5
mdadm: added /dev/mapper/sdd4 to /dev/md4 as 0
mdadm: /dev/md4: Need a backup file to complete reshape of this array.
mdadm: Please provided one with "--backup-file=..."
mdadm: (Don't specify --update=revert-reshape again, that part succeeded.)
How can the array have an active reshape if the reshape pos'n is 0 on
all devices? Doesn't that mean that the reshape never actually
started? If so, can I just revert -- somehow -- to a 5-device RAID 6
array to recover my data?
Thanks.
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: Wols Lists @ 2015-12-09 13:51 UTC (permalink / raw)
To: Shaohua Li, NeilBrown; +Cc: Christoph Hellwig, linux-raid
In-Reply-To: <20151209012812.GA2403138@devbig084.prn1.facebook.com>
On 09/12/15 01:28, Shaohua Li wrote:
> It does remove the scary __GFP_NOFAIL, but the approach is essentially
> idential to a 'retry after allocation failure'. Why not just let the mm
> (with __GFP_NOFAIL) to do the retry then?
Because that's almost as dangerous?
Forgive me if I'm talking out of my hat, but I remember an article on
lwn reasonably recently about kernel memory allocation, and it was
something to do with small allocations never failing and the actual
behaviour being COMPLETELY different to what most users THINK is happening.
(iirc, the actual behaviour was NOFAIL, whether it was requested or not,
so if you think that adding NOFAIL to your code is going to help, you
may be in for a nasty shock.)
Whatever, if the disk code is prone to deadlocks, and it's calling
memory code which is prone to deadlocks, then better safe than sorry...
Cheers,
Wol
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: NeilBrown @ 2015-12-09 6:34 UTC (permalink / raw)
To: Shaohua Li; +Cc: Christoph Hellwig, linux-raid
In-Reply-To: <20151209012812.GA2403138@devbig084.prn1.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 2428 bytes --]
On Wed, Dec 09 2015, Shaohua Li wrote:
> On Wed, Dec 09, 2015 at 11:36:30AM +1100, NeilBrown wrote:
>> On Thu, Dec 03 2015, Christoph Hellwig wrote:
>>
>> > Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
>> >
>> > I've looked into replacing these with mempools and biosets, and for the
>> > bio and the meta_page that's pretty trivial as they have short life times
>> > and do make guaranteed progress. I'm massively struggling with the iounit
>> > allocation, though. These can live on for a long time over log I/O, cache
>> > flushing and last but not least RAID I/O, and every attempt at something
>> > mempool-like results in reproducible deadlocks. I wonder if we need to
>> > figure out some more efficient data structure to communicate the completion
>> > status that doesn't rely on these fairly long living allocations from
>> > the I/O path.
>>
>> Presumably the root cause of these deadlocks is that the raid5d thread
>> has called
>> handle_stripe -> ops_run_io ->r5l_write_stripe -> r5l_log_stripe
>> -> r5l_get_meta -> r5l_new_meta
>>
>> and r5l_new_meta is blocked on memory allocation, which won't complete
>> until some raid5 stripes get written out, which requires raid5d to do
>> something more useful than sitting and waiting.
>>
>> I suspect a good direction towards a solution would be to allow the
>> memory allocation to fail, to cleanly propagate that failure indication
>> up through r5l_log_stripe to r5l_write_stripe which falls back to adding
>> the stripe_head to ->no_space_stripes.
>>
>> Then we only release stripes from no_space_stripes when a memory
>> allocation might succeed.
>>
>> There are lots of missing details, and possibly we would need a separate
>> list rather than re-using no_space_stripes.
>> But the key idea is that raid5d should never block (except beneath
>> submit_bio on some other device) and when it cannot make progress
>> without blocking, it should queue the stripe_head for later handling.
>>
>> Does that make sense?
>
> It does remove the scary __GFP_NOFAIL, but the approach is essentially
> idential to a 'retry after allocation failure'. Why not just let the mm
> (with __GFP_NOFAIL) to do the retry then?
>
Because deadlocks.
If raid5d is waiting for the mm to allocated memory, then it cannot
retire write requests which could free up memory.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Unable to (un)-grow raid6
From: Phil Turmel @ 2015-12-09 3:12 UTC (permalink / raw)
To: David Waite; +Cc: Linux-RAID
In-Reply-To: <10980501-F73B-4A63-AF7D-D7100C9C6B72@alkaline-solutions.com>
On 12/08/2015 06:42 PM, David Waite wrote:
> Thanks for your help - it is now reshaping the array with sdia3 and
> sdib3 as hot spares.
>
> It appears the version of mdadm on the NAS (3.1.4) is pretty old, and
> does not have a replace command - just marking disks as faulty. I
> will build and load in a newer version of mdadm. I assume there is no
> issue in doing this while a reshaping is going on?
No issue. You don't even have to install it. However, like most
features, there's a kernel part and a userspace part. If the new mdadm
that supports --replace can't talk to the matching kernel part, it won't
work.
Either way, you can't actually do the --replace until the reshape finishes.
{Convention on kernel.org lists is to reply-to-all, trim replies, and
bottom post. Please do.}
Phil
^ permalink raw reply
* Update
From: robert @ 2015-12-09 2:43 UTC (permalink / raw)
To: Recipients
Good day, hoping you read this email and respond to me in good time.I do not intend to solicit for funds but your time and energy in using my own resources to assist the less privileged becauseI am medically ill and confined at the moment hence I request your indulgence.I will give you a comprehensive brief once I hear from you. Thanks and reply. Robert Grondahl
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: Shaohua Li @ 2015-12-09 1:28 UTC (permalink / raw)
To: NeilBrown; +Cc: Christoph Hellwig, linux-raid
In-Reply-To: <876108qwz5.fsf@notabene.neil.brown.name>
On Wed, Dec 09, 2015 at 11:36:30AM +1100, NeilBrown wrote:
> On Thu, Dec 03 2015, Christoph Hellwig wrote:
>
> > Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
> >
> > I've looked into replacing these with mempools and biosets, and for the
> > bio and the meta_page that's pretty trivial as they have short life times
> > and do make guaranteed progress. I'm massively struggling with the iounit
> > allocation, though. These can live on for a long time over log I/O, cache
> > flushing and last but not least RAID I/O, and every attempt at something
> > mempool-like results in reproducible deadlocks. I wonder if we need to
> > figure out some more efficient data structure to communicate the completion
> > status that doesn't rely on these fairly long living allocations from
> > the I/O path.
>
> Presumably the root cause of these deadlocks is that the raid5d thread
> has called
> handle_stripe -> ops_run_io ->r5l_write_stripe -> r5l_log_stripe
> -> r5l_get_meta -> r5l_new_meta
>
> and r5l_new_meta is blocked on memory allocation, which won't complete
> until some raid5 stripes get written out, which requires raid5d to do
> something more useful than sitting and waiting.
>
> I suspect a good direction towards a solution would be to allow the
> memory allocation to fail, to cleanly propagate that failure indication
> up through r5l_log_stripe to r5l_write_stripe which falls back to adding
> the stripe_head to ->no_space_stripes.
>
> Then we only release stripes from no_space_stripes when a memory
> allocation might succeed.
>
> There are lots of missing details, and possibly we would need a separate
> list rather than re-using no_space_stripes.
> But the key idea is that raid5d should never block (except beneath
> submit_bio on some other device) and when it cannot make progress
> without blocking, it should queue the stripe_head for later handling.
>
> Does that make sense?
It does remove the scary __GFP_NOFAIL, but the approach is essentially
idential to a 'retry after allocation failure'. Why not just let the mm
(with __GFP_NOFAIL) to do the retry then?
Thanks,
Shaohua
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: NeilBrown @ 2015-12-09 0:36 UTC (permalink / raw)
To: Christoph Hellwig, shli; +Cc: linux-raid
In-Reply-To: <1449072638-15409-1-git-send-email-hch@lst.de>
[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]
On Thu, Dec 03 2015, Christoph Hellwig wrote:
> Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
>
> I've looked into replacing these with mempools and biosets, and for the
> bio and the meta_page that's pretty trivial as they have short life times
> and do make guaranteed progress. I'm massively struggling with the iounit
> allocation, though. These can live on for a long time over log I/O, cache
> flushing and last but not least RAID I/O, and every attempt at something
> mempool-like results in reproducible deadlocks. I wonder if we need to
> figure out some more efficient data structure to communicate the completion
> status that doesn't rely on these fairly long living allocations from
> the I/O path.
Presumably the root cause of these deadlocks is that the raid5d thread
has called
handle_stripe -> ops_run_io ->r5l_write_stripe -> r5l_log_stripe
-> r5l_get_meta -> r5l_new_meta
and r5l_new_meta is blocked on memory allocation, which won't complete
until some raid5 stripes get written out, which requires raid5d to do
something more useful than sitting and waiting.
I suspect a good direction towards a solution would be to allow the
memory allocation to fail, to cleanly propagate that failure indication
up through r5l_log_stripe to r5l_write_stripe which falls back to adding
the stripe_head to ->no_space_stripes.
Then we only release stripes from no_space_stripes when a memory
allocation might succeed.
There are lots of missing details, and possibly we would need a separate
list rather than re-using no_space_stripes.
But the key idea is that raid5d should never block (except beneath
submit_bio on some other device) and when it cannot make progress
without blocking, it should queue the stripe_head for later handling.
Does that make sense?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH v2 1/3] badblocks: Add core badblock management code
From: Verma, Vishal L @ 2015-12-08 23:47 UTC (permalink / raw)
To: Williams, Dan J
Cc: neilb@suse.com, linux-block@vger.kernel.org, jmoyer@redhat.com,
linux-raid@vger.kernel.org, linux-scsi@vger.kernel.org,
axboe@fb.com, linux-nvdimm@lists.01.org,
James.Bottomley@hansenpartnership.com
In-Reply-To: <CAPcyv4gJrdP23sh2cQU94fpq6+Z9GBjxWf0RpBp_4+kydpQAsA@mail.gmail.com>
On Tue, 2015-12-08 at 13:18 -0800, Dan Williams wrote:
> On Tue, Dec 8, 2015 at 1:08 PM, Verma, Vishal L
> <vishal.l.verma@intel.com> wrote:
> > On Wed, 2015-12-09 at 08:03 +1100, NeilBrown wrote:
> > > On Sat, Dec 05 2015, Verma, Vishal L wrote:
> > > > >
> > > > > > +int badblocks_clear(struct badblocks *bb, sector_t s, int
> > > > > > sectors)
> > > > > > +{
> > > > > [...]
> > > > > > +#define DO_DEBUG 1
> > > > >
> > > > > Why have this at all if it's unconditionally defined and
> > > > > always
> > > > > set.
> > > >
> > > > Neil - any reason or anything you had in mind for this? Or is it
> > > > just an
> > > > artifact and can be removed.
> > >
> > > Like the comment says:
> > >
> > > /* Allow clearing via sysfs *only* for testing/debugging.
> > > * Normally only a successful write may clear a badblock
> > > */
> > >
> > > The DO_DEBUG define and ifdefs are documentation identifying bits
> > > of
> > > code that should be removed when it all seems to be working.
> > > Maybe now is a good time to remove that code.
> > >
> > Hm, I think it would be nice to continue to have the ability to
> > clear
> > badblocks using sysfs at least for a while more, as we test the
> > various
> > error handling paths for NVDIMMS (Dan, thoughts?).
> >
> > We could either remove it later or (I'm leaning towards) make it a
> > config option similar to FAIL_MAKE_REQUEST and friends..
>
> "later" as in before v4.5-rc1? We can always carry this debug feature
> locally for testing. We don't want userspace growing ABI attachments
> to this capability now that it's more than just md tooling that will
> see this.
Agreed. The following incremental patch removes sysfs support.
All the latest badblocks patches can also be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/vishal/nvdimm.git gendisk-badblocks
8<-----
From 5f0e7ac31d27a132f314106f1db33af22fde03ed Mon Sep 17 00:00:00 2001
From: Vishal Verma <vishal.l.verma@intel.com>
Date: Tue, 8 Dec 2015 16:28:31 -0700
Subject: [PATCH v4 4/3] badblocks: remove support for clearing via sysfs
sysfs support for clearing badblocks was originally meant for testing
only. With the move to generalize the interface, remove this support so
that userspace doesn't start treating this as an ABI.
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
---
block/badblocks.c | 15 ---------------
1 file changed, 15 deletions(-)
diff --git a/block/badblocks.c b/block/badblocks.c
index f0ac279..e5d2a91 100644
--- a/block/badblocks.c
+++ b/block/badblocks.c
@@ -503,16 +503,6 @@ ssize_t badblocks_store(struct badblocks *bb, const
char *page, size_t len,
int length;
char newline;
- /* Allow clearing via sysfs *only* for testing/debugging.
- * Normally only a successful write may clear a badblock
- */
- int clear = 0;
-
- if (page[0] == '-') {
- clear = 1;
- page++;
- }
-
switch (sscanf(page, "%llu %d%c", §or, &length, &newline))
{
case 3:
if (newline != '\n')
@@ -525,11 +515,6 @@ ssize_t badblocks_store(struct badblocks *bb, const
char *page, size_t len,
return -EINVAL;
}
- if (clear) {
- badblocks_clear(bb, sector, length);
- return len;
- }
-
if (badblocks_set(bb, sector, length, !unack))
return -ENOSPC;
else
--
2.5.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox