question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

[Jason Keltz]

Hi.

I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system. I've 
experimented with setting "speed_limit_min" and "speed_limit_max" kernel 
variables so that I get the best balance of performance during a RAID 
rebuild of one of the RAID1 pairs. If, for example, I set 
speed_limit_min AND speed_limit_max to 80000 then fail a disk when there 
is no other disk activity, then I do get a rebuild rate of around 80 
MB/s. However, if I then start up a write intensive operation on the MD 
array (eg. a dd, or a mkfs on an LVM logical volume that is created on 
that MD), then, my write operation seems to get "full power", and my 
rebuild drops to around 25 MB/s. This means that the rebuild of my 
RAID10 disk is going to take a huge amount of time (>12 hours!!!). When 
I set speed_limit_min and speed_limit_max to the same value, am I not 
guaranteeing the rebuild speed? Is this a bug that I should be reporting 
to Red Hat, or a "feature"?

Thanks in advance for any help that you can provide...

Jason.

Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

[Jason Keltz]

On 10/20/2014 12:19 PM, Jason Keltz wrote:
> Hi.
>
> I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system. 
> I've experimented with setting "speed_limit_min" and "speed_limit_max" 
> kernel variables so that I get the best balance of performance during 
> a RAID rebuild of one of the RAID1 pairs. If, for example, I set 
> speed_limit_min AND speed_limit_max to 80000 then fail a disk when 
> there is no other disk activity, then I do get a rebuild rate of 
> around 80 MB/s. However, if I then start up a write intensive 
> operation on the MD array (eg. a dd, or a mkfs on an LVM logical 
> volume that is created on that MD), then, my write operation seems to 
> get "full power", and my rebuild drops to around 25 MB/s. This means 
> that the rebuild of my RAID10 disk is going to take a huge amount of 
> time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to 
> the same value, am I not guaranteeing the rebuild speed? Is this a bug 
> that I should be reporting to Red Hat, or a "feature"?
>
> Thanks in advance for any help that you can provide...
>
> Jason.

I would like to add that I downloaded the latest version of Ubuntu, and 
am running it on the same server with the same MD.
When I set speed_limit_min and speed_limit_max to 80000, I was able to 
start two large dds on the md array, and the rebuild stuck at around 71 
MB/s, which is close enough.  This leads me to believe that the problem 
above is probably a RHEL6 issue.  However, after I stopped the two dd 
operations,  and raised both speed_limit_min and speed_limit_max to 
120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes 
.. now it seems to be at 100 MB/s... but doesn't seem to get any higher 
(even though I had 120 MB/s and above on the RHEL system without any 
load)... Hmm.

Jason.

Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2825 bytes --]

On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz <jas@cse.yorku.ca> wrote:

> On 10/20/2014 12:19 PM, Jason Keltz wrote:
> > Hi.
> >
> > I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system. 
> > I've experimented with setting "speed_limit_min" and "speed_limit_max" 
> > kernel variables so that I get the best balance of performance during 
> > a RAID rebuild of one of the RAID1 pairs. If, for example, I set 
> > speed_limit_min AND speed_limit_max to 80000 then fail a disk when 
> > there is no other disk activity, then I do get a rebuild rate of 
> > around 80 MB/s. However, if I then start up a write intensive 
> > operation on the MD array (eg. a dd, or a mkfs on an LVM logical 
> > volume that is created on that MD), then, my write operation seems to 
> > get "full power", and my rebuild drops to around 25 MB/s. This means 
> > that the rebuild of my RAID10 disk is going to take a huge amount of 
> > time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to 
> > the same value, am I not guaranteeing the rebuild speed? Is this a bug 
> > that I should be reporting to Red Hat, or a "feature"?
> >
> > Thanks in advance for any help that you can provide...
> >
> > Jason.
> 
> I would like to add that I downloaded the latest version of Ubuntu, and 
> am running it on the same server with the same MD.
> When I set speed_limit_min and speed_limit_max to 80000, I was able to 
> start two large dds on the md array, and the rebuild stuck at around 71 
> MB/s, which is close enough.  This leads me to believe that the problem 
> above is probably a RHEL6 issue.  However, after I stopped the two dd 
> operations,  and raised both speed_limit_min and speed_limit_max to 
> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes 
> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher 
> (even though I had 120 MB/s and above on the RHEL system without any 
> load)... Hmm.
>

md certainly cannot "guarantee" any speed - it can only deliver what the
underlying devices deliver.
I know the kernels logs say something about a "guarantee".  That was added
before my time and I haven't had occasion to remove it.

md will normally just try to recover as fast as it can unless that exceeds
one of the limits - then it will back-off.
What speed it actually achieved depends on other load and the behaviour of
the IO scheduler.

"RHEL6" and "Ubuntu" don't mean a lot to me.  Specific kernel version might,
though in the case of Redhat I know that backport lots of stuff so even the
kernel version isn't very helpful.  I'm must prefer having report against
mainline kernels.

Rotating drives do get lower transfer speeds at higher addresses.  That might
explain the 120 / 100 difference.

NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

[Jason Keltz]

On 28/10/2014 6:38 PM, NeilBrown wrote:
> On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz<jas@cse.yorku.ca>  wrote:
>
>> On 10/20/2014 12:19 PM, Jason Keltz wrote:
>>> Hi.
>>>
>>> I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system.
>>> I've experimented with setting "speed_limit_min" and "speed_limit_max"
>>> kernel variables so that I get the best balance of performance during
>>> a RAID rebuild of one of the RAID1 pairs. If, for example, I set
>>> speed_limit_min AND speed_limit_max to 80000 then fail a disk when
>>> there is no other disk activity, then I do get a rebuild rate of
>>> around 80 MB/s. However, if I then start up a write intensive
>>> operation on the MD array (eg. a dd, or a mkfs on an LVM logical
>>> volume that is created on that MD), then, my write operation seems to
>>> get "full power", and my rebuild drops to around 25 MB/s. This means
>>> that the rebuild of my RAID10 disk is going to take a huge amount of
>>> time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to
>>> the same value, am I not guaranteeing the rebuild speed? Is this a bug
>>> that I should be reporting to Red Hat, or a "feature"?
>>>
>>> Thanks in advance for any help that you can provide...
>>>
>>> Jason.
>> I would like to add that I downloaded the latest version of Ubuntu, and
>> am running it on the same server with the same MD.
>> When I set speed_limit_min and speed_limit_max to 80000, I was able to
>> start two large dds on the md array, and the rebuild stuck at around 71
>> MB/s, which is close enough.  This leads me to believe that the problem
>> above is probably a RHEL6 issue.  However, after I stopped the two dd
>> operations,  and raised both speed_limit_min and speed_limit_max to
>> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes
>> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher
>> (even though I had 120 MB/s and above on the RHEL system without any
>> load)... Hmm.
>>
> md certainly cannot "guarantee" any speed - it can only deliver what the
> underlying devices deliver.
> I know the kernels logs say something about a "guarantee".  That was added
> before my time and I haven't had occasion to remove it.
>
> md will normally just try to recover as fast as it can unless that exceeds
> one of the limits - then it will back-off.
> What speed it actually achieved depends on other load and the behaviour of
> the IO scheduler.
>
> "RHEL6" and "Ubuntu" don't mean a lot to me.  Specific kernel version might,
> though in the case of Redhat I know that backport lots of stuff so even the
> kernel version isn't very helpful.  I'm must prefer having report against
> mainline kernels.
>
> Rotating drives do get lower transfer speeds at higher addresses.  That might
> explain the 120 / 100 difference.
Hi Neil,
Thanks very much for your response.
I must say that I'm a little puzzled though. I'm coming from using a 
3Ware hardware RAID controller where I could configure how much of the 
disk bandwidth is to be used for a rebuild versus I/O.   From what I 
understand, you're saying that MD can only use the disk bandwidth 
available to it.  It seems that it doesn't take any priority in the I/O 
chain.  It will only attempt to use no less than min bandwidth, and no 
more than max bandwidth for the rebuild, but if you're on a busy system, 
and other system I/O needs that disk bandwidth, then there's nothing it 
can do about it.  I guess I just don't understand why.  Why can't md be 
given a priority in the kernel to allow the admin to decide how much 
bandwidth goes to system I/O versus rebuild I/O.  Even in a busy system, 
I still want to allocate at least some minimum bandwidth to MD.  In 
fact, in the event of a disk failure, I want to have a whole lot of the 
disk bandwidth dedicated to MD.  It's something about short term pain 
for long term gain? I'd rather not have the users suffer at all, but if 
they do have to suffer, I'd rather them suffer for a few hours, knowing 
that after that, the RAID system is in a perfectly good state with no 
bad disks as opposed to letting a bad disk resync take days because the 
system is really busy... days during which another failure might occur!

Jason.

Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 5940 bytes --]

On Tue, 28 Oct 2014 22:34:07 -0400 Jason Keltz <jas@cse.yorku.ca> wrote:

> On 28/10/2014 6:38 PM, NeilBrown wrote:
> > On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz<jas@cse.yorku.ca>  wrote:
> >
> >> On 10/20/2014 12:19 PM, Jason Keltz wrote:
> >>> Hi.
> >>>
> >>> I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system.
> >>> I've experimented with setting "speed_limit_min" and "speed_limit_max"
> >>> kernel variables so that I get the best balance of performance during
> >>> a RAID rebuild of one of the RAID1 pairs. If, for example, I set
> >>> speed_limit_min AND speed_limit_max to 80000 then fail a disk when
> >>> there is no other disk activity, then I do get a rebuild rate of
> >>> around 80 MB/s. However, if I then start up a write intensive
> >>> operation on the MD array (eg. a dd, or a mkfs on an LVM logical
> >>> volume that is created on that MD), then, my write operation seems to
> >>> get "full power", and my rebuild drops to around 25 MB/s. This means
> >>> that the rebuild of my RAID10 disk is going to take a huge amount of
> >>> time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to
> >>> the same value, am I not guaranteeing the rebuild speed? Is this a bug
> >>> that I should be reporting to Red Hat, or a "feature"?
> >>>
> >>> Thanks in advance for any help that you can provide...
> >>>
> >>> Jason.
> >> I would like to add that I downloaded the latest version of Ubuntu, and
> >> am running it on the same server with the same MD.
> >> When I set speed_limit_min and speed_limit_max to 80000, I was able to
> >> start two large dds on the md array, and the rebuild stuck at around 71
> >> MB/s, which is close enough.  This leads me to believe that the problem
> >> above is probably a RHEL6 issue.  However, after I stopped the two dd
> >> operations,  and raised both speed_limit_min and speed_limit_max to
> >> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes
> >> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher
> >> (even though I had 120 MB/s and above on the RHEL system without any
> >> load)... Hmm.
> >>
> > md certainly cannot "guarantee" any speed - it can only deliver what the
> > underlying devices deliver.
> > I know the kernels logs say something about a "guarantee".  That was added
> > before my time and I haven't had occasion to remove it.
> >
> > md will normally just try to recover as fast as it can unless that exceeds
> > one of the limits - then it will back-off.
> > What speed it actually achieved depends on other load and the behaviour of
> > the IO scheduler.
> >
> > "RHEL6" and "Ubuntu" don't mean a lot to me.  Specific kernel version might,
> > though in the case of Redhat I know that backport lots of stuff so even the
> > kernel version isn't very helpful.  I'm must prefer having report against
> > mainline kernels.
> >
> > Rotating drives do get lower transfer speeds at higher addresses.  That might
> > explain the 120 / 100 difference.
> Hi Neil,
> Thanks very much for your response.
> I must say that I'm a little puzzled though. I'm coming from using a 
> 3Ware hardware RAID controller where I could configure how much of the 
> disk bandwidth is to be used for a rebuild versus I/O.   From what I 
> understand, you're saying that MD can only use the disk bandwidth 
> available to it.  It seems that it doesn't take any priority in the I/O 
> chain.  It will only attempt to use no less than min bandwidth, and no 
> more than max bandwidth for the rebuild, but if you're on a busy system, 
> and other system I/O needs that disk bandwidth, then there's nothing it 
> can do about it.  I guess I just don't understand why.  Why can't md be 
> given a priority in the kernel to allow the admin to decide how much 
> bandwidth goes to system I/O versus rebuild I/O.  Even in a busy system, 
> I still want to allocate at least some minimum bandwidth to MD.  In 
> fact, in the event of a disk failure, I want to have a whole lot of the 
> disk bandwidth dedicated to MD.  It's something about short term pain 
> for long term gain? I'd rather not have the users suffer at all, but if 
> they do have to suffer, I'd rather them suffer for a few hours, knowing 
> that after that, the RAID system is in a perfectly good state with no 
> bad disks as opposed to letting a bad disk resync take days because the 
> system is really busy... days during which another failure might occur!
> 
> Jason.

It isn't so much "that MD can only use..." but rather "that MD does only
use ...".

This is how the code has "always" worked and no-one has ever bothered to
change it, or to ask for it to be changed (that I recall).

There are difficulties in guaranteeing a minimum when the array uses
partitions from devices on which other partitions are used for other things.
In that case I don't think it is practical to make guarantees, but that
needn't stop us making guarantees when we can I guess.

If the configured bandwidth exceeded the physically available bandwidth I
don't think we would want to exclude non-resync IO completely, so the
guaranty would have to be:
   N MB/sec or M% of available, whichever is less

We could even implement the different approach in a back-compatible way.
Introduce a new setting "max_sync_percent".  By default that is unset and the
current algorithm applies.
If it is set to something below 100, non-resync IO is throttled to
an appropriate fraction of the actual resync throughput whenever that is
below sync_speed_min.

Or something like that.

Some care would be needed in comparing throughput and sync throughput is
measured per-device, while non-resync throughput might be measured per-array.
Maybe the throttling would happen per-device??

All we need now is for someone to firm up the design and then write the code.

NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

[Jason Keltz]

On 10/28/2014 10:57 PM, NeilBrown wrote:
> On Tue, 28 Oct 2014 22:34:07 -0400 Jason Keltz <jas@cse.yorku.ca> wrote:
>
>> On 28/10/2014 6:38 PM, NeilBrown wrote:
>>> On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz<jas@cse.yorku.ca>  wrote:
>>>
>>>> On 10/20/2014 12:19 PM, Jason Keltz wrote:
>>>>> Hi.
>>>>>
>>>>> I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system.
>>>>> I've experimented with setting "speed_limit_min" and "speed_limit_max"
>>>>> kernel variables so that I get the best balance of performance during
>>>>> a RAID rebuild of one of the RAID1 pairs. If, for example, I set
>>>>> speed_limit_min AND speed_limit_max to 80000 then fail a disk when
>>>>> there is no other disk activity, then I do get a rebuild rate of
>>>>> around 80 MB/s. However, if I then start up a write intensive
>>>>> operation on the MD array (eg. a dd, or a mkfs on an LVM logical
>>>>> volume that is created on that MD), then, my write operation seems to
>>>>> get "full power", and my rebuild drops to around 25 MB/s. This means
>>>>> that the rebuild of my RAID10 disk is going to take a huge amount of
>>>>> time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to
>>>>> the same value, am I not guaranteeing the rebuild speed? Is this a bug
>>>>> that I should be reporting to Red Hat, or a "feature"?
>>>>>
>>>>> Thanks in advance for any help that you can provide...
>>>>>
>>>>> Jason.
>>>> I would like to add that I downloaded the latest version of Ubuntu, and
>>>> am running it on the same server with the same MD.
>>>> When I set speed_limit_min and speed_limit_max to 80000, I was able to
>>>> start two large dds on the md array, and the rebuild stuck at around 71
>>>> MB/s, which is close enough.  This leads me to believe that the problem
>>>> above is probably a RHEL6 issue.  However, after I stopped the two dd
>>>> operations,  and raised both speed_limit_min and speed_limit_max to
>>>> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes
>>>> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher
>>>> (even though I had 120 MB/s and above on the RHEL system without any
>>>> load)... Hmm.
>>>>
>>> md certainly cannot "guarantee" any speed - it can only deliver what the
>>> underlying devices deliver.
>>> I know the kernels logs say something about a "guarantee".  That was added
>>> before my time and I haven't had occasion to remove it.
>>>
>>> md will normally just try to recover as fast as it can unless that exceeds
>>> one of the limits - then it will back-off.
>>> What speed it actually achieved depends on other load and the behaviour of
>>> the IO scheduler.
>>>
>>> "RHEL6" and "Ubuntu" don't mean a lot to me.  Specific kernel version might,
>>> though in the case of Redhat I know that backport lots of stuff so even the
>>> kernel version isn't very helpful.  I'm must prefer having report against
>>> mainline kernels.
>>>
>>> Rotating drives do get lower transfer speeds at higher addresses.  That might
>>> explain the 120 / 100 difference.
>> Hi Neil,
>> Thanks very much for your response.
>> I must say that I'm a little puzzled though. I'm coming from using a
>> 3Ware hardware RAID controller where I could configure how much of the
>> disk bandwidth is to be used for a rebuild versus I/O.   From what I
>> understand, you're saying that MD can only use the disk bandwidth
>> available to it.  It seems that it doesn't take any priority in the I/O
>> chain.  It will only attempt to use no less than min bandwidth, and no
>> more than max bandwidth for the rebuild, but if you're on a busy system,
>> and other system I/O needs that disk bandwidth, then there's nothing it
>> can do about it.  I guess I just don't understand why.  Why can't md be
>> given a priority in the kernel to allow the admin to decide how much
>> bandwidth goes to system I/O versus rebuild I/O.  Even in a busy system,
>> I still want to allocate at least some minimum bandwidth to MD.  In
>> fact, in the event of a disk failure, I want to have a whole lot of the
>> disk bandwidth dedicated to MD.  It's something about short term pain
>> for long term gain? I'd rather not have the users suffer at all, but if
>> they do have to suffer, I'd rather them suffer for a few hours, knowing
>> that after that, the RAID system is in a perfectly good state with no
>> bad disks as opposed to letting a bad disk resync take days because the
>> system is really busy... days during which another failure might occur!
>>
>> Jason.
> It isn't so much "that MD can only use..." but rather "that MD does only
> use ...".
Got it..

> This is how the code has "always" worked and no-one has ever bothered to
> change it, or to ask for it to be changed (that I recall).
I'm actually not surprised to hear that since I spent a considerable  
time trying to find articles talking about this topic, and couldn't find 
a single thing! :)  I, on the other hand, am replacing a hardware RAID 
system with MD, so I've been "spoiled" already! (not with the 
performance of hardware RAID, but the functionality) :)

> There are difficulties in guaranteeing a minimum when the array uses
> partitions from devices on which other partitions are used for other things.
> In that case I don't think it is practical to make guarantees, but that
> needn't stop us making guarantees when we can I guess.
>
> If the configured bandwidth exceeded the physically available bandwidth I
> don't think we would want to exclude non-resync IO completely, so the
> guaranty would have to be:
>     N MB/sec or M% of available, whichever is less
>
> We could even implement the different approach in a back-compatible way.
> Introduce a new setting "max_sync_percent".  By default that is unset and the
> current algorithm applies.
> If it is set to something below 100, non-resync IO is throttled to
> an appropriate fraction of the actual resync throughput whenever that is
> below sync_speed_min.
>
> Or something like that.
That actually sounds great!  I can certainly understand and appreciate 
how it would be difficult to handle arrays using partitions from 
multiple devices. Maybe the functionality only works if you're using 
full disk devices. :)  (Okay, on second thought, that's about 99% of the 
people using MD -- apparently, word has it that only you and I are using 
MD full disk devices).

> Some care would be needed in comparing throughput and sync throughput is
> measured per-device, while non-resync throughput might be measured per-array.
> Maybe the throttling would happen per-device??
>
> All we need now is for someone to firm up the design and then write the code.
Maybe someone on the list will step forward :D

The truth is, as people start to combine larger and larger disks, and 
rebuild times go up and up and up, this type of request will become more 
common....

Jas.

Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.

[Peter Grandi]

>>>>>> If, for example, I set speed_limit_min AND speed_limit_max to
>>>>>> 80000 then fail a disk when there is no other disk activity, then
>>>>>> I do get a rebuild rate of around 80 MB/s. However, if I then
>>>>>> start up a write intensive operation on the MD array (eg. a dd,
>>>>>> or a mkfs on an LVM logical volume that is created on that MD),
>>>>>> then, my write operation seems to get "full power", and my
>>>>>> rebuild drops to around 25 MB/s.

Linux MD RAID is fundamentally an IO address remapper, and actual IO is
scheduled and executed by the Linux block (page) IO subsystem. This
separation is beneficial in many ways.

Also the bandwidth delivered by a storage device is not a single
number. Disk drives transfer rate depend a lot on degree of randomness
of access, and outer vs. inner regions. Common disks can therefore
deliver bandwidth between 150MB/s and 0.5MB/s depending on the overall
traffic investing them.

Therefore in order to deliver a consistent transfer rate to the MD
resync kernel process the Linux block (page) IO subsystem would have to
be quite clever in controlling the rates of usage of all the processes
using a disk.

>>> I'm coming from using a 3Ware hardware RAID cotntroller where I
>>> could configure how much of the disk bandwidth is to be used for a
>>> rebuild versus I/O.

That (usually) works because the disk is completely dedicated to the
RAID card and the RAID card can schedule all IO to it.

>>> From what I understand, you're saying that MD [ ... ] and other
>>> system I/O needs that disk bandwidth, then there's nothing it can do
>>> about it. I guess I just don't understand why. Why can't md be given
>>> a priority in the kernel to allow the admin to decide how much
>>> bandwidth goes to system I/O versus rebuild I/O.

There is something you can do about it: rewrite the block IO subsystem
in the Linux kernel so that it can be configured to allocate IOPS and/or
bandwidth quotas to different processes, among them the MD resync kernel
process (extra awesome if that is also isochronous).

Because as well summarized below that process is just one of many
possible users of a given disk in a Linux based system:

>> There are difficulties in guaranteeing a minimum when the array uses
>> partitions from devices on which other partitions are used for other
>> things.

Put another way, designing MD RAID as fundamentally an IO address
remapper, and letting the MD resync kernel process run as "just another
process", has some big advantages and gives a lot of flexibility, but
means relying on the kernel block subsystem to do actual IO, and
accepting its current limitations. That is a free choice.

> The truth is, as people start to combine larger and larger
> disks, and rebuild times go up and up and up, this type of
> request will become more common....

Request of the type "I decided to use a physical storage design that
behaves in a way that I don't like so MD should do magic and work around
my decision" are already common. :-).

Using «larger and larger disks» is a *choice people make* and if they
don't like the obvious consequences they should then not make that
choice; they can instead choose to use smaller disks, or just the outer
part of larger disks (which can` be cheaper).

A critical metric for physical storage is IOPS/GB ratios (and their
variability dependent on workload) and ooking at those ratios I
personally think that common disks larger than 1TB are not suitable for
many cases of "typical" live data usage, and in the day job we sometimes
build MD RAID sets made of 146GB 15k disks, because fortunately my
colleagues understand the relevant tradeoffs too.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html