RAID1 and load-balancing during read

RAID1 and load-balancing during read

[Dimitrios Apostolou]

Hello list, 

I just created a RAID1 array consisting of two disks. After experiments with 
processes *reading* from the device (badblocks, dd) and the iostat program, I 
can see that only one disk is being utilised for reading. To be exact, every 
time I execute the command one of the two disks is being randomly used, but 
the other one has absolutely no activity. 

My question is: why isn't load balancing happening? Is there an option I'm 
missing? Until now I though it was the default for all RAID1 implementations. 
Even md man page mentions in the RAID1 section: 

  The driver attempts to distribute read requests across all devices to 
maximise performance.


Thanks in advance, 
Dimitris

Re: RAID1 and load-balancing during read

[Iustin Pop]

On Mon, Sep 10, 2007 at 10:29:30PM +0300, Dimitrios Apostolou wrote:
> Hello list, 
> 
> I just created a RAID1 array consisting of two disks. After experiments with 
> processes *reading* from the device (badblocks, dd) and the iostat program, I 
> can see that only one disk is being utilised for reading. To be exact, every 
> time I execute the command one of the two disks is being randomly used, but 
> the other one has absolutely no activity. 
> 
> My question is: why isn't load balancing happening? Is there an option I'm 
> missing? Until now I though it was the default for all RAID1 implementations. 

Did you read the archives of this list? This question has been answered,
like, 4 times already in the last months.

And yes, the driver does do load balancing. Just not as RAID0 does,
since it's not RAID0.

regards,
iustin

Re: RAID1 and load-balancing during read

[Dimitrios Apostolou]

On Monday 10 September 2007 22:35:30 Iustin Pop wrote:
> On Mon, Sep 10, 2007 at 10:29:30PM +0300, Dimitrios Apostolou wrote:
> > Hello list,
> >
> > I just created a RAID1 array consisting of two disks. After experiments
> > with processes *reading* from the device (badblocks, dd) and the iostat
> > program, I can see that only one disk is being utilised for reading. To
> > be exact, every time I execute the command one of the two disks is being
> > randomly used, but the other one has absolutely no activity.
> >
> > My question is: why isn't load balancing happening? Is there an option
> > I'm missing? Until now I though it was the default for all RAID1
> > implementations.
>
> Did you read the archives of this list? This question has been answered,
> like, 4 times already in the last months.
>
> And yes, the driver does do load balancing. Just not as RAID0 does,
> since it's not RAID0.

Of course I did a quick search in the archives but couldn't find anything. 
I'll search better, thanks anyway. Moreover, I think I found the answer in 
the code after posting. There is a comment somewhere in read_balance() 
saying "Don't change to another disk for sequential reads". I have to study 
it a bit to figure out *why* you chose that way. 


Thanks, 
Dimitris

Re: RAID1 and load-balancing during read

[Iustin Pop]

On Mon, Sep 10, 2007 at 10:51:37PM +0300, Dimitrios Apostolou wrote:
> On Monday 10 September 2007 22:35:30 Iustin Pop wrote:
> > On Mon, Sep 10, 2007 at 10:29:30PM +0300, Dimitrios Apostolou wrote:
> > > Hello list,
> > >
> > > I just created a RAID1 array consisting of two disks. After experiments
> > > with processes *reading* from the device (badblocks, dd) and the iostat
> > > program, I can see that only one disk is being utilised for reading. To
> > > be exact, every time I execute the command one of the two disks is being
> > > randomly used, but the other one has absolutely no activity.
> > >
> > > My question is: why isn't load balancing happening? Is there an option
> > > I'm missing? Until now I though it was the default for all RAID1
> > > implementations.
> >
> > Did you read the archives of this list? This question has been answered,
> > like, 4 times already in the last months.
> >
> > And yes, the driver does do load balancing. Just not as RAID0 does,
> > since it's not RAID0.
> 
> Of course I did a quick search in the archives but couldn't find anything. 
Hmm, it's true that searching does not point out an easy to find
response.

> I'll search better, thanks anyway. Moreover, I think I found the answer in 
> the code after posting. There is a comment somewhere in read_balance() 
> saying "Don't change to another disk for sequential reads". I have to study 
> it a bit to figure out *why* you chose that way. 
Well, from what I understand, you cannot make a mirror behave like a
stripe, plain and simple. There is no simple algorithm that makes
sequential raid behave better.

OTOH, random I/O or multiple threads are being sped up by raid1. And
people have said on the list that using the raid10 module with only two
disks and (IIRC) in offset or far mode will give better read
performance, albeit it reduces write performance.

Hmmm, I think a patch is needed to md.4 in order to explain this right
at the source of the confusion.

thanks,
iustin

Re: RAID1 and load-balancing during read

[Goswin von Brederlow]

Iustin Pop <iusty@k1024.org> writes:

> On Mon, Sep 10, 2007 at 10:51:37PM +0300, Dimitrios Apostolou wrote:
>> On Monday 10 September 2007 22:35:30 Iustin Pop wrote:
>> > On Mon, Sep 10, 2007 at 10:29:30PM +0300, Dimitrios Apostolou wrote:
>> > > Hello list,
>> > >
>> > > I just created a RAID1 array consisting of two disks. After experiments
>> > > with processes *reading* from the device (badblocks, dd) and the iostat
>> > > program, I can see that only one disk is being utilised for reading. To
>> > > be exact, every time I execute the command one of the two disks is being
>> > > randomly used, but the other one has absolutely no activity.
>> > >
>> > > My question is: why isn't load balancing happening? Is there an option
>> > > I'm missing? Until now I though it was the default for all RAID1
>> > > implementations.
>> >
>> > Did you read the archives of this list? This question has been answered,
>> > like, 4 times already in the last months.
>> >
>> > And yes, the driver does do load balancing. Just not as RAID0 does,
>> > since it's not RAID0.
>> 
>> Of course I did a quick search in the archives but couldn't find anything. 
> Hmm, it's true that searching does not point out an easy to find
> response.
>
>> I'll search better, thanks anyway. Moreover, I think I found the answer in 
>> the code after posting. There is a comment somewhere in read_balance() 
>> saying "Don't change to another disk for sequential reads". I have to study 
>> it a bit to figure out *why* you chose that way. 
> Well, from what I understand, you cannot make a mirror behave like a
> stripe, plain and simple. There is no simple algorithm that makes
> sequential raid behave better.

As I understand it the problem is the hardware. Reading a chunk of
data from a disk means that the head has to seek to the right track
and the disk has to spin to the right position. After that you can
read a full revolution of the disk worth of data sequentially.

Now consider what happens if you read 4K per disk in stripes. The disk
seeks to the right track, spins to the right position and reads
4k. Then it waits for 4k to rotate below the head, read 4k, waits 4k,
read 4k, waits 4k, .... That way both disks are busy without any gain.

What you would need to do is read one track from one disk, the next
track from the other and so on. But how should the kernel know where
tracks start and end. That is highly device dependent and differs
between the outside and inside of the platter. The geometry values
reported by the disk is purely fictional so the CHS values are no
help.

> OTOH, random I/O or multiple threads are being sped up by raid1. And
> people have said on the list that using the raid10 module with only two
> disks and (IIRC) in offset or far mode will give better read
> performance, albeit it reduces write performance.

I found that near copies behave like raid1, offset copies are slower
in both reading and writing (beats me why) and far copies are slightly
faster than near copies in write and twice as fast in read. All for
sequential read/write. For random writes far copies should be slower
to write.

> Hmmm, I think a patch is needed to md.4 in order to explain this right
> at the source of the confusion.
>
> thanks,
> iustin

MfG
        Goswin

Re: RAID1 and load-balancing during read

[Neil Brown]

On Tuesday September 11, brederlo@informatik.uni-tuebingen.de wrote:
> 
> I found that near copies behave like raid1, offset copies are slower
> in both reading and writing (beats me why) and far copies are slightly
> faster than near copies in write and twice as fast in read. All for
> sequential read/write. For random writes far copies should be slower
> to write.

I would expect read performance on an 'offset' layout to be
significantly affected by the chunk size.

You probably want the chunk size to be about 4 times the cylinder
size.
The ideal would be exactly the cylinder size, and exactly aligned to
cylinders.  But as cylinder sizes change across the device, this is
not possible.

Unfortunately I don't know how big cylinders are.  I suspect if you
hunt through the docs for the device you might find out.

NeilBrown

Re: RAID1 and load-balancing during read

[Dimitrios Apostolou]

Goswin von Brederlow wrote:
> As I understand it the problem is the hardware. Reading a chunk of
> data from a disk means that the head has to seek to the right track
> and the disk has to spin to the right position. After that you can
> read a full revolution of the disk worth of data sequentially.
> 
> Now consider what happens if you read 4K per disk in stripes. The disk
> seeks to the right track, spins to the right position and reads
> 4k. Then it waits for 4k to rotate below the head, read 4k, waits 4k,
> read 4k, waits 4k, .... That way both disks are busy without any gain.

I'm not sure about that. All disks have some sort of read-ahead option. 
Usually this read-ahead option is configurable too. What if the 
read-ahead was set to 2*chunk size (for RAID1 with 2 disks). The 
algorithm for reading in parallel from both disks would then be:

1st disk:
current_pos1= read(chunk_size)
seek(current_pos1 + chunk_size)

2nd disk:
seek(current_pos2+chunk_size)
current_pos2= read(chunk_size)


Of course if drive read-ahead works properly (which should be for small 
values) the seek() functions would not really cause the drive to seek.

As for the example you mentioned I'm quite sure that reading 4K from a 
drive and then seeking 4K forward and reading again would not result in 
any slowdown, since most of today's IDE drives have a default read-ahead 
of 128KB.

> 
> What you would need to do is read one track from one disk, the next
> track from the other and so on. But how should the kernel know where
> tracks start and end. That is highly device dependent and differs
> between the outside and inside of the platter. The geometry values
> reported by the disk is purely fictional so the CHS values are no
> help.
> 
>> OTOH, random I/O or multiple threads are being sped up by raid1. And
>> people have said on the list that using the raid10 module with only two
>> disks and (IIRC) in offset or far mode will give better read
>> performance, albeit it reduces write performance.
> 
> I found that near copies behave like raid1, offset copies are slower
> in both reading and writing (beats me why) and far copies are slightly
> faster than near copies in write and twice as fast in read. All for
> sequential read/write. For random writes far copies should be slower
> to write.
> 
>> Hmmm, I think a patch is needed to md.4 in order to explain this right
>> at the source of the confusion.
>>
>> thanks,
>> iustin
> 
> MfG
>         Goswin

Thanks,
Dimitris

Re: RAID1 and load-balancing during read

[Dimitrios Apostolou]

Iustin Pop wrote:
> On Mon, Sep 10, 2007 at 10:51:37PM +0300, Dimitrios Apostolou wrote:
>> On Monday 10 September 2007 22:35:30 Iustin Pop wrote:
>>> On Mon, Sep 10, 2007 at 10:29:30PM +0300, Dimitrios Apostolou wrote:
>>>> Hello list,
>>>>
>>>> I just created a RAID1 array consisting of two disks. After experiments
>>>> with processes *reading* from the device (badblocks, dd) and the iostat
>>>> program, I can see that only one disk is being utilised for reading. To
>>>> be exact, every time I execute the command one of the two disks is being
>>>> randomly used, but the other one has absolutely no activity.
>>>>
>>>> My question is: why isn't load balancing happening? Is there an option
>>>> I'm missing? Until now I though it was the default for all RAID1
>>>> implementations.
>>> Did you read the archives of this list? This question has been answered,
>>> like, 4 times already in the last months.
>>>
>>> And yes, the driver does do load balancing. Just not as RAID0 does,
>>> since it's not RAID0.
>> Of course I did a quick search in the archives but couldn't find anything. 
> Hmm, it's true that searching does not point out an easy to find
> response.

I now found archived replies to the subject, thanks for pointing that 
out. The outcome is that because of the way RAID1 works, doubling 
performance for sequential access is not feasible, or it is very hard.

I would be curious however, is anyone aware of hardware RAID 
implementations that actually improve sequential read performance?

> 
>> I'll search better, thanks anyway. Moreover, I think I found the answer in 
>> the code after posting. There is a comment somewhere in read_balance() 
>> saying "Don't change to another disk for sequential reads". I have to study 
>> it a bit to figure out *why* you chose that way. 
> Well, from what I understand, you cannot make a mirror behave like a
> stripe, plain and simple. There is no simple algorithm that makes
> sequential raid behave better.
> 
> OTOH, random I/O or multiple threads are being sped up by raid1. And

Indeed, and it is notable that on a benchmark I performed yesterday, 
RAID1 performed better (about 25%) than RAID0 on random seeks! (Of 
course all other numbers were worse).

> people have said on the list that using the raid10 module with only two
> disks and (IIRC) in offset or far mode will give better read
> performance, albeit it reduces write performance.
> 
> Hmmm, I think a patch is needed to md.4 in order to explain this right
> at the source of the confusion.

I will agree that this needs some clarification in the docs. Unless 
another patch is implemented that actually improves the performance. :-p

> 
> thanks,
> iustin

Thanks,
Dimitris

Re: RAID1 and load-balancing during read

[Dimitrios Apostolou]

Dimitrios Apostolou wrote:
> Iustin Pop wrote:
>> Hmmm, I think a patch is needed to md.4 in order to explain this right
>> at the source of the confusion.
> 
> I will agree that this needs some clarification in the docs. Unless 
> another patch is implemented that actually improves the performance. :-p

Just saw the list archives... Thanks for the patch!


Dimitris

Re: RAID1 and load-balancing during read

[Dimitrios Apostolou]

Dimitrios Apostolou wrote:
> Goswin von Brederlow wrote:
>> As I understand it the problem is the hardware. Reading a chunk of
>> data from a disk means that the head has to seek to the right track
>> and the disk has to spin to the right position. After that you can
>> read a full revolution of the disk worth of data sequentially.
>>
>> Now consider what happens if you read 4K per disk in stripes. The disk
>> seeks to the right track, spins to the right position and reads
>> 4k. Then it waits for 4k to rotate below the head, read 4k, waits 4k,
>> read 4k, waits 4k, .... That way both disks are busy without any gain.
> 
> I'm not sure about that. All disks have some sort of read-ahead option. 
> Usually this read-ahead option is configurable too. What if the 
> read-ahead was set to 2*chunk size (for RAID1 with 2 disks). The 
> algorithm for reading in parallel from both disks would then be:
> 
> 1st disk:
> current_pos1= read(chunk_size)
> seek(current_pos1 + chunk_size)
> 
> 2nd disk:
> seek(current_pos2+chunk_size)
> current_pos2= read(chunk_size)
> 
> 
> Of course if drive read-ahead works properly (which should be for small 
> values) the seek() functions would not really cause the drive to seek.
> 
> As for the example you mentioned I'm quite sure that reading 4K from a 
> drive and then seeking 4K forward and reading again would not result in 
> any slowdown, since most of today's IDE drives have a default read-ahead 
> of 128KB.
> 
On a second thought I realise that this doesn't work as expected... Even 
though this will not cause slowdown, it will not cause speedup either. 
It is obvious that if a drive reads sequentially with 40MB/s, it won't 
surpass that speed no matter what seeks you do. Readahead might help to 
avoid disk head seeks, but read speed just can't increase. I don't know 
what I was thinking when writing this... :-s


Thanks for your replies,
Dimitris