RAID1: Disks alternating on reads

RAID1: Disks alternating on reads

[Andrew Herdman]

Pardon me if my assumption was incorrect, but I was under the belief that
when using software RAID1, that when reads occurred on the RAID device that
it would read from both drives in a striped fashion similar to how RAID0
works to improve the speed of the md devices.   I am actually seeing this,
but it appears that the reads on each drive are continuing for 10 to 20
seconds before moving onto the next drive and then another 10-20 seconds and
back again.  This is not allowing for any performance increase, it just lets
the drives rest alternately.

This link http://www.whine.com/~andrew/alternating_disk.png is a gkrellm
snapshot of the activity, hda and hde are the two disks in the RAID array.
Each disk is the master on it's own channel.

I also included the mdadm --query --detail info below for each RAID
partition

how:~# mdadm --query --detail /dev/md/1
/dev/md/1:
        Version : 00.90.00
  Creation Time : Sat Jul 19 08:30:42 2003
     Raid Level : raid1
     Array Size : 62400 (60.94 MiB 63.90 MB)
    Device Size : 62400 (60.94 MiB 63.90 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Mon Sep 15 08:41:38 2003
          State : dirty, no-errors
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0


    Number   Major   Minor   RaidDevice State
       0       3        1        0      active sync
/dev/ide/host0/bus0/target0/lun0/part1
       1      33        1        1      active sync
/dev/ide/host2/bus0/target0/lun0/part1
           UUID : faa4b7b2:4832fae8:ac2fa294:2fce1d85
         Events : 0.38

how:~# mdadm --query --detail /dev/md/2
/dev/md/2:
        Version : 00.90.00
  Creation Time : Sat Jul 19 08:32:18 2003
     Raid Level : raid1
     Array Size : 119140544 (113.62 GiB 121.100 GB)
    Device Size : 119140544 (113.62 GiB 121.100 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Mon Sep 15 08:41:38 2003
          State : dirty, no-errors
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0


    Number   Major   Minor   RaidDevice State
       0       3        2        0      active sync
/dev/ide/host0/bus0/target0/lun0/part2
       1      33        2        1      active sync
/dev/ide/host2/bus0/target0/lun0/part2
           UUID : 0ad35f02:56a32094:c8301e3e:45895c16
         Events : 0.38

how:~# mdadm --query --detail /dev/md/3
/dev/md/3:
        Version : 00.90.00
  Creation Time : Sat Jul 19 08:31:37 2003
     Raid Level : raid1
     Array Size : 499904 (488.19 MiB 511.90 MB)
    Device Size : 499904 (488.19 MiB 511.90 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 3
    Persistence : Superblock is persistent

    Update Time : Mon Sep 15 08:41:38 2003
          State : dirty, no-errors
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0


    Number   Major   Minor   RaidDevice State
       0       3        3        0      active sync
/dev/ide/host0/bus0/target0/lun0/part3
       1      33        3        1      active sync
/dev/ide/host2/bus0/target0/lun0/part3
           UUID : f8b2d95b:93055c43:d78427ac:197a610f
         Events : 0.38

how:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md/2             112G   94G   16G  86% /
/dev/md/1              60M   16M   42M  28% /boot

Have I configured something incorrectly?  Or is this really the way RAID1
works?

Thanks
 Andrew
--
Andrew Herdman
andrew@whine.com - Primary E-Mail

Re: RAID1: Disks alternating on reads

[Catalin BOIE]

On Sun, 21 Sep 2003, Andrew Herdman wrote:

I put the same question. I get no response.

> Pardon me if my assumption was incorrect, but I was under the belief that
> when using software RAID1, that when reads occurred on the RAID device that
> it would read from both drives in a striped fashion similar to how RAID0
> works to improve the speed of the md devices.   I am actually seeing this,
> but it appears that the reads on each drive are continuing for 10 to 20
> seconds before moving onto the next drive and then another 10-20 seconds and
> back again.  This is not allowing for any performance increase, it just lets
> the drives rest alternately.
>
> This link http://www.whine.com/~andrew/alternating_disk.png is a gkrellm
> snapshot of the activity, hda and hde are the two disks in the RAID array.
> Each disk is the master on it's own channel.
>
> I also included the mdadm --query --detail info below for each RAID
> partition
>
> how:~# mdadm --query --detail /dev/md/1
> /dev/md/1:
>         Version : 00.90.00
>   Creation Time : Sat Jul 19 08:30:42 2003
>      Raid Level : raid1
>      Array Size : 62400 (60.94 MiB 63.90 MB)
>     Device Size : 62400 (60.94 MiB 63.90 MB)
>    Raid Devices : 2
>   Total Devices : 2
> Preferred Minor : 1
>     Persistence : Superblock is persistent
>
>     Update Time : Mon Sep 15 08:41:38 2003
>           State : dirty, no-errors
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
>
>
>     Number   Major   Minor   RaidDevice State
>        0       3        1        0      active sync
> /dev/ide/host0/bus0/target0/lun0/part1
>        1      33        1        1      active sync
> /dev/ide/host2/bus0/target0/lun0/part1
>            UUID : faa4b7b2:4832fae8:ac2fa294:2fce1d85
>          Events : 0.38
>
> how:~# mdadm --query --detail /dev/md/2
> /dev/md/2:
>         Version : 00.90.00
>   Creation Time : Sat Jul 19 08:32:18 2003
>      Raid Level : raid1
>      Array Size : 119140544 (113.62 GiB 121.100 GB)
>     Device Size : 119140544 (113.62 GiB 121.100 GB)
>    Raid Devices : 2
>   Total Devices : 2
> Preferred Minor : 2
>     Persistence : Superblock is persistent
>
>     Update Time : Mon Sep 15 08:41:38 2003
>           State : dirty, no-errors
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
>
>
>     Number   Major   Minor   RaidDevice State
>        0       3        2        0      active sync
> /dev/ide/host0/bus0/target0/lun0/part2
>        1      33        2        1      active sync
> /dev/ide/host2/bus0/target0/lun0/part2
>            UUID : 0ad35f02:56a32094:c8301e3e:45895c16
>          Events : 0.38
>
> how:~# mdadm --query --detail /dev/md/3
> /dev/md/3:
>         Version : 00.90.00
>   Creation Time : Sat Jul 19 08:31:37 2003
>      Raid Level : raid1
>      Array Size : 499904 (488.19 MiB 511.90 MB)
>     Device Size : 499904 (488.19 MiB 511.90 MB)
>    Raid Devices : 2
>   Total Devices : 2
> Preferred Minor : 3
>     Persistence : Superblock is persistent
>
>     Update Time : Mon Sep 15 08:41:38 2003
>           State : dirty, no-errors
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
>
>
>     Number   Major   Minor   RaidDevice State
>        0       3        3        0      active sync
> /dev/ide/host0/bus0/target0/lun0/part3
>        1      33        3        1      active sync
> /dev/ide/host2/bus0/target0/lun0/part3
>            UUID : f8b2d95b:93055c43:d78427ac:197a610f
>          Events : 0.38
>
> how:~# df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md/2             112G   94G   16G  86% /
> /dev/md/1              60M   16M   42M  28% /boot
>
> Have I configured something incorrectly?  Or is this really the way RAID1
> works?
>
> Thanks
>  Andrew
> --
> Andrew Herdman
> andrew@whine.com - Primary E-Mail
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

---
Catalin(ux) BOIE
catab@deuroconsult.ro

Re: RAID1: Disks alternating on reads

[Juri Haberland]

"Andrew Herdman" <andrew@whine.com> wrote:
> Pardon me if my assumption was incorrect, but I was under the belief that
> when using software RAID1, that when reads occurred on the RAID device that
> it would read from both drives in a striped fashion similar to how RAID0
> works to improve the speed of the md devices.   I am actually seeing this,
> but it appears that the reads on each drive are continuing for 10 to 20
> seconds before moving onto the next drive and then another 10-20 seconds and
> back again.  This is not allowing for any performance increase, it just lets
> the drives rest alternately.
> 
> This link http://www.whine.com/~andrew/alternating_disk.png is a gkrellm
> snapshot of the activity, hda and hde are the two disks in the RAID array.
> Each disk is the master on it's own channel.

Was that a single large file that you read or multiple files?

As far as I could observe, the RAID1 code reads one file from one disk
and never changes disks during this read - so you get the performance
of one disk.
*But* the code distributes different writes to different disks.

Try the following:
dd if=/dev/mdX of=/dev/null&

Now watch gkrellm and see that it reads only from one disk.
Then start another instance of dd. You will see that this instance reads
from the other RAID disk.

This way one disk can seek for a file while the other one is transfering
one. A single read won't be faster but concurrent access to the same
filesystem will be much faster than on a non-RAID filesystem.

Cheers,
Juri

-- 
Juri Haberland  <juri@koschikode.com> 

Re: RAID1: Disks alternating on reads

[Andrew Herdman]

Juri;

Thanks for your reply, some comments inline below.

----- Original Message ----- 
From: "Juri Haberland" <list-linux.raid@spoiled.org>
To: <linux-raid@vger.kernel.org>
Sent: Monday, September 22, 2003 5:39 AM
Subject: Re: RAID1: Disks alternating on reads


> "Andrew Herdman" <andrew@whine.com> wrote:
> > Pardon me if my assumption was incorrect, but I was under the belief
that
> > when using software RAID1, that when reads occurred on the RAID device
that
> > it would read from both drives in a striped fashion similar to how RAID0
> > works to improve the speed of the md devices.   I am actually seeing
this,
> > but it appears that the reads on each drive are continuing for 10 to 20
> > seconds before moving onto the next drive and then another 10-20 seconds
and
> > back again.  This is not allowing for any performance increase, it just
lets
> > the drives rest alternately.
> >
> > This link http://www.whine.com/~andrew/alternating_disk.png is a gkrellm
> > snapshot of the activity, hda and hde are the two disks in the RAID
array.
> > Each disk is the master on it's own channel.
>
> Was that a single large file that you read or multiple files?

Multiple files 1-2MB in size

> As far as I could observe, the RAID1 code reads one file from one disk
> and never changes disks during this read - so you get the performance
> of one disk.
> *But* the code distributes different writes to different disks.

Well this is what I am seeing, so it's being consistent then.

> Try the following:
> dd if=/dev/mdX of=/dev/null&

Looks just like the graph in gkrellm I showed earlier.

> Now watch gkrellm and see that it reads only from one disk.
> Then start another instance of dd. You will see that this instance reads
> from the other RAID disk.

Well look at that, it is now starting to consistently use both disks.  I did
notice that after a length of time of running both instances that it stopped
using hda for an extended time.  Starting a 3rd instance again invoked the
use of hda.  The implementation doesn't seem perfect, but appears to be more
than adequate.

> This way one disk can seek for a file while the other one is transfering
> one. A single read won't be faster but concurrent access to the same
> filesystem will be much faster than on a non-RAID filesystem.

Does this mean that the md driver is filesystem aware?  Or is it just
because i'm using EXT3?  How does md know how many file accesses i'm doing?

Thanks for your explaination, it certainly clears things up.

Andrew

Re: RAID1: Disks alternating on reads

[Lars Marowsky-Bree]

On 2003-09-22T09:39:05,
   Juri Haberland <list-linux.raid@spoiled.org> said:

> As far as I could observe, the RAID1 code reads one file from one disk
> and never changes disks during this read - so you get the performance
> of one disk.
> *But* the code distributes different writes to different disks.

Reads of course, not writes ;-)


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering		ever tried. ever failed. no matter.
SuSE Labs				try again. fail again. fail better.
Research & Development, SUSE LINUX AG		-- Samuel Beckett

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RAID1: Disks alternating on reads

[Bailey, Scott]

> Pardon me if my assumption was incorrect, but I was under the belief that
> when using software RAID1, that when reads occurred on the RAID device
that
> it would read from both drives in a striped fashion similar to how RAID0
> works to improve the speed of the md devices.   I am actually seeing this,
> but it appears that the reads on each drive are continuing for 10 to 20
> seconds before moving onto the next drive and then another 10-20 seconds
and
> back again.  This is not allowing for any performance increase, it just
lets
> the drives rest alternately.

My impression from crawling through the code a few months back was that this
behavior was a design feature. On a read request, the request is sent to the
device with heads "closest" to the target sector, EXCEPT that for sequential
reads, the device gets reused on the assumption that the disk can be kept
streaming, EXCEPT that after some maximum number of reads, another drive
gets chosen to give the previous device a rest.

I speculate your files are contiguous (or nearly so) with the result that
the initially closest drive gets hammered until its 'work quota' gets
exceeded, and then the next drive will get pounded. And so on. To test this,
I think you could reduce MAX_WORK_PER_DISK in raid1.c from the default 128
(in 2.4.21) to something smaller (perhaps 8 sectors for a filesystem with 4K
block size?) and see if the load evens out.

Good luck,

	Scott Bailey
	scott.bailey at eds.com

Re: RAID1: Disks alternating on reads

[Juri Haberland]

Lars Marowsky-Bree <lmb@suse.de> wrote:
> On 2003-09-22T09:39:05,
>    Juri Haberland <list-linux.raid@spoiled.org> said:
> 
>> As far as I could observe, the RAID1 code reads one file from one disk
>> and never changes disks during this read - so you get the performance
>> of one disk.
>> *But* the code distributes different writes to different disks.
> 
> Reads of course, not writes ;-)

Yes, of course :)

Thanks,
Juri

-- 
Juri Haberland  <juri@koschikode.com> 

Re: RAID1: Disks alternating on reads

[Juri Haberland]

"Andrew Herdman" <andrew@whine.com> wrote:
> From: "Juri Haberland" <list-linux.raid@spoiled.org>

>> This way one disk can seek for a file while the other one is transfering
>> one. A single read won't be faster but concurrent access to the same
>> filesystem will be much faster than on a non-RAID filesystem.
> 
> Does this mean that the md driver is filesystem aware?  Or is it just
> because i'm using EXT3?  How does md know how many file accesses i'm doing?

No, AFAIK it's not filesystem aware and it shouldn't be. For a better
explanation please look at the mail from Scott Bailey. He obviously
looked at the code which I didn't...

Cheers,
Juri

-- 
Juri Haberland  <juri@koschikode.com>