Software RAID5 write issues

Software RAID5 write issues

[Steven Haigh]

Hi all,

After a week and a bit of googling, experimenting and frustration I'm  
posting here and hoping I can get some clues on what could be wrong  
with my 5 disk RAID5 SATA array.

The array in question is:
md1 : active raid5 sdg1[0] sdf1[1] sde1[3] sdd1[2] sdc1[4]
       1172131840 blocks level 5, 1024k chunk, algorithm 2 [5/5] [UUUUU]

All 5 drives are connection to a sil_sata controller (a 3112 & a 3114)  
set up as a simple SATA controller (ie no RAID here).

Once the system buffer is full, write speeds to the array are usually  
under 20MB/sec.

I am currently running CentOS 5.3 (kernel  
2.6.18-128.1.10.el5.centos.plus).

I have lodged a bug report against RHEL 5.3, as I believe something is  
not quite right here, but haven't been able to narrow down the exact  
issue.
	https://bugzilla.redhat.com/show_bug.cgi?id=502499

Using bonnie++ to benchmark the array, it shows sequential block reads  
at 90MB/sec but writes at 11MB/sec across the RAID5 array - a  
difference I really didn't expect.

Any pointers on how to try to tackle this one and figure out the root  
cause of the problem would be VERY helpful!

--
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

Re: Software RAID5 write issues

[John Robinson]

On 10/06/2009 18:17, Steven Haigh wrote:
> After a week and a bit of googling, experimenting and frustration I'm 
> posting here and hoping I can get some clues on what could be wrong with 
> my 5 disk RAID5 SATA array.
> 
> The array in question is:
> md1 : active raid5 sdg1[0] sdf1[1] sde1[3] sdd1[2] sdc1[4]
>       1172131840 blocks level 5, 1024k chunk, algorithm 2 [5/5] [UUUUU]
> 
> All 5 drives are connection to a sil_sata controller (a 3112 & a 3114) 
> set up as a simple SATA controller (ie no RAID here).
> 
> Once the system buffer is full, write speeds to the array are usually 
> under 20MB/sec.
> 
> I am currently running CentOS 5.3 (kernel 2.6.18-128.1.10.el5.centos.plus).
> 
> I have lodged a bug report against RHEL 5.3, as I believe something is 
> not quite right here, but haven't been able to narrow down the exact issue.
>     https://bugzilla.redhat.com/show_bug.cgi?id=502499
> 
> Using bonnie++ to benchmark the array, it shows sequential block reads 
> at 90MB/sec but writes at 11MB/sec across the RAID5 array - a difference 
> I really didn't expect.
> 
> Any pointers on how to try to tackle this one and figure out the root 
> cause of the problem would be VERY helpful!

Do your drives rattle furiously while writing? Do you have a write 
intent bitmap? I found my write performance quite sucky until changing 
the bitmap settings, see 
http://marc.info/?l=linux-raid&m=124027469610660&w=2 for the thread. In 
short, changing the bitmap from its default 2M chunk size to 32M got me 
a substantial speed-up; here's my bonnie++ output on a 3-drive RAID-5:

# bonnie++ -d /mnt/store/bonnie/ -u john -n 256 -m beast
Using uid:500, gid:500.
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
beast            4G 72011  79 59334  10 35775   4 66455  64 174049   6 
215.8   0
                     ------Sequential Create------ --------Random 
Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 256 61524  81 377753  96 22143  28 53153  73 425291  99 
  8327  11
beast,4G,72011,79,59334,10,35775,4,66455,64,174049,6,215.8,0,256,61524,81,377753,96,22143,28,53153,73,425291,99,8327,11


Cheers,

John.

Re: Software RAID5 write issues

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 5549 bytes --]

On Thu, 2009-06-11 at 03:17 +1000, Steven Haigh wrote:
> Hi all,
> 
> After a week and a bit of googling, experimenting and frustration I'm  
> posting here and hoping I can get some clues on what could be wrong  
> with my 5 disk RAID5 SATA array.
> 
> The array in question is:
> md1 : active raid5 sdg1[0] sdf1[1] sde1[3] sdd1[2] sdc1[4]
>        1172131840 blocks level 5, 1024k chunk, algorithm 2 [5/5] [UUUUU]
> 
> All 5 drives are connection to a sil_sata controller (a 3112 & a 3114)  
> set up as a simple SATA controller (ie no RAID here).
> 
> Once the system buffer is full, write speeds to the array are usually  
> under 20MB/sec.
> 
> I am currently running CentOS 5.3 (kernel  
> 2.6.18-128.1.10.el5.centos.plus).
> 
> I have lodged a bug report against RHEL 5.3, as I believe something is  
> not quite right here, but haven't been able to narrow down the exact  
> issue.
> 	https://bugzilla.redhat.com/show_bug.cgi?id=502499
> 
> Using bonnie++ to benchmark the array, it shows sequential block reads  
> at 90MB/sec but writes at 11MB/sec across the RAID5 array - a  
> difference I really didn't expect.
> 
> Any pointers on how to try to tackle this one and figure out the root  
> cause of the problem would be VERY helpful!

OK, so I read the bug report.  There are two distinctly different
problems you are experiencing.  One, is a slow down specific to our
recent kernels.  The slow down in your case takes your normally abysmal
raid and makes it even worse.  The original bug report was mainly about
the slowdown, so I'll address that in the bug report.  However, in
regards to your raid setup, I'll try to address why your array performs
so poorly regardless of kernel version and maybe that will help you
build up a better raid setup.

You have 4 motherboard SATA ports, and 4 SATA ports on a PCI card.
Right now you have your two OS drives on motherboard SATA ports, two of
the five raid5 drives on motherboard SATA ports, and the three remaining
raid5 drives on the PCI card SATA ports.  You need to get as many of the
raid5 SATA disks on motherboard ports as possible.  I would decide if
you are more concerned about the raid5 array performing well (common, as
it's usually the data you access most often) or the base OS array
performing well (not so common, as it gets loaded largely into cache and
doesn't get hit nearly so often as the data drive).  If you can deal
with slowing down the OS drives, then I would move one of the OS drives
to the PCI card and move one of the raid5 drives to the motherboard SATA
port (and whichever drive you just moved over to the PCI card, I would
mark it's raid1 arrays as write-mostly so that you don't read from it
normally).  If your BIOS will allow you to select drives on the PCI card
as boot drives, and you can tolerate the slow down, then I would move
both of the OS drives to the PCI card (and don't worry about using
write-mostly on the raid1 arrays any more) and get 4 of the 5 raid5
drives onto motherboard SATA ports.

Your big problem is that with 3 out of 5 raid5 drives on that PCI card,
and sharing bandwidth, your total theoretical raid speed is abysmal.
When the three drives are sharing bandwidth on the card, they tend to
split it up fairly evenly.  That means each drive gets roughly 1/3 of
the PCI card's total available bandwidth over the PCI bus, which is
generally poor in the first place.  Understand that a slow drive drags
down *all* the drives in a raid5 array.  The faster drives just end up
idling while waiting on the slower drive to finish its work (the faster
drives will run ahead up to a point, then they eventually just get so
far ahead that there isn't anything else for them to do until the
slowest drive finishes up its stuff so old block requests can be
completed, etc).  On the other hand, if you get 4 of the 5 drives on the
motherboard ports, then that 5th drive on the PCI card won't be
splitting bandwidth up and the overall array performance will shoot up
(assuming the OS drives aren't also heavily loaded).

If you move one OS drive to the PCI card, then that leaves two raid5
drives on the card.  In that case, I would seriously consider dropping
back to a 4 drive array if you can handle the space reduction.  I would
also seriously consider using raid4 instead of raid5 depending on your
normal usage pattern.  If the data on the raid5 array is written once
and then read over and over again, a raid4 can be beneficial in that you
can stick the parity drive off on the PCI card and it won't be read from
unless there is a drive failure or one the rare occasions when you write
new data.  If, on the other hand, you write lots of new data, then
either don't use raid4, or put the parity drive on a motherboard port
where it won't hog so much bandwidth on the PCI card.  Ideally, I would
say get both OS drives on the PCI card, and if you need all 5 drives for
the data raid, then use raid4 with the parity on the PCI card if the
array is mostly static, use raid5 otherwise.  If you only move one OS
drive to the PCI card and still have two raid5 drives on the PCI card,
then again think about whether your data is static or not and possibly
use raid4 in an attempt to reduce the traffic on the PCI card.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Software RAID5 write issues

[Steven Haigh]

On 11/06/2009, at 6:57 AM, Doug Ledford wrote:

> On Thu, 2009-06-11 at 03:17 +1000, Steven Haigh wrote:
>> Hi all,
>>
>> After a week and a bit of googling, experimenting and frustration I'm
>> posting here and hoping I can get some clues on what could be wrong
>> with my 5 disk RAID5 SATA array.
>>
>> The array in question is:
>> md1 : active raid5 sdg1[0] sdf1[1] sde1[3] sdd1[2] sdc1[4]
>>       1172131840 blocks level 5, 1024k chunk, algorithm 2 [5/5]  
>> [UUUUU]
>>
>> All 5 drives are connection to a sil_sata controller (a 3112 & a  
>> 3114)
>> set up as a simple SATA controller (ie no RAID here).
>>
>> Once the system buffer is full, write speeds to the array are usually
>> under 20MB/sec.
>>
>> I am currently running CentOS 5.3 (kernel
>> 2.6.18-128.1.10.el5.centos.plus).
>>
>> I have lodged a bug report against RHEL 5.3, as I believe something  
>> is
>> not quite right here, but haven't been able to narrow down the exact
>> issue.
>> 	https://bugzilla.redhat.com/show_bug.cgi?id=502499
>>
>> Using bonnie++ to benchmark the array, it shows sequential block  
>> reads
>> at 90MB/sec but writes at 11MB/sec across the RAID5 array - a
>> difference I really didn't expect.
>>
>> Any pointers on how to try to tackle this one and figure out the root
>> cause of the problem would be VERY helpful!
>
> OK, so I read the bug report.  There are two distinctly different
> problems you are experiencing.  One, is a slow down specific to our
> recent kernels.  The slow down in your case takes your normally  
> abysmal
> raid and makes it even worse.  The original bug report was mainly  
> about
> the slowdown, so I'll address that in the bug report.  However, in
> regards to your raid setup, I'll try to address why your array  
> performs
> so poorly regardless of kernel version and maybe that will help you
> build up a better raid setup.
>
> You have 4 motherboard SATA ports, and 4 SATA ports on a PCI card.
> Right now you have your two OS drives on motherboard SATA ports, two  
> of
> the five raid5 drives on motherboard SATA ports, and the three  
> remaining
> raid5 drives on the PCI card SATA ports.  You need to get as many of  
> the
> raid5 SATA disks on motherboard ports as possible.  I would decide if
> you are more concerned about the raid5 array performing well  
> (common, as
> it's usually the data you access most often) or the base OS array
> performing well (not so common, as it gets loaded largely into cache  
> and
> doesn't get hit nearly so often as the data drive).  If you can deal
> with slowing down the OS drives, then I would move one of the OS  
> drives
> to the PCI card and move one of the raid5 drives to the motherboard  
> SATA
> port (and whichever drive you just moved over to the PCI card, I would
> mark it's raid1 arrays as write-mostly so that you don't read from it
> normally).  If your BIOS will allow you to select drives on the PCI  
> card
> as boot drives, and you can tolerate the slow down, then I would move
> both of the OS drives to the PCI card (and don't worry about using
> write-mostly on the raid1 arrays any more) and get 4 of the 5 raid5
> drives onto motherboard SATA ports.
>

This would also be an interesting test - as from memory, now that I  
have updated the firmware on the 3114 card, the BIOS will see those  
drives and allow me to boot from them (hopefully). I will experiment  
here and post the results.

> Your big problem is that with 3 out of 5 raid5 drives on that PCI  
> card,
> and sharing bandwidth, your total theoretical raid speed is abysmal.
> When the three drives are sharing bandwidth on the card, they tend to
> split it up fairly evenly.  That means each drive gets roughly 1/3 of
> the PCI card's total available bandwidth over the PCI bus, which is
> generally poor in the first place.  Understand that a slow drive drags
> down *all* the drives in a raid5 array.  The faster drives just end up
> idling while waiting on the slower drive to finish its work (the  
> faster
> drives will run ahead up to a point, then they eventually just get so
> far ahead that there isn't anything else for them to do until the
> slowest drive finishes up its stuff so old block requests can be
> completed, etc).  On the other hand, if you get 4 of the 5 drives on  
> the
> motherboard ports, then that 5th drive on the PCI card won't be
> splitting bandwidth up and the overall array performance will shoot up
> (assuming the OS drives aren't also heavily loaded).

Isn't the PCI Bus limited to around 133MB/sec? If so, even with 3  
drives on the same controller, you would expect divided equally that  
each drive would get ~44MB/sec before overheads - not around 7MB/sec  
per drive. I know I'm not going to get phenomenal performance with my  
setup, but as most the data is archiving (and then copied to tape), I  
would like to get things at least up to a reasonable level instead of  
having a write speed of ~12% of the read speed.

> If you move one OS drive to the PCI card, then that leaves two raid5
> drives on the card.  In that case, I would seriously consider dropping
> back to a 4 drive array if you can handle the space reduction.  I  
> would
> also seriously consider using raid4 instead of raid5 depending on your
> normal usage pattern.  If the data on the raid5 array is written once
> and then read over and over again, a raid4 can be beneficial in that  
> you
> can stick the parity drive off on the PCI card and it won't be read  
> from
> unless there is a drive failure or one the rare occasions when you  
> write
> new data.  If, on the other hand, you write lots of new data, then
> either don't use raid4, or put the parity drive on a motherboard port
> where it won't hog so much bandwidth on the PCI card.  Ideally, I  
> would
> say get both OS drives on the PCI card, and if you need all 5 drives  
> for
> the data raid, then use raid4 with the parity on the PCI card if the
> array is mostly static, use raid5 otherwise.  If you only move one OS
> drive to the PCI card and still have two raid5 drives on the PCI card,
> then again think about whether your data is static or not and possibly
> use raid4 in an attempt to reduce the traffic on the PCI card.


Hmmm - a very interesting read - but I am a little confused when it  
comes to PCI bandwidth. I would assume (maybe wrongly) that if I can  
READ from the array at 95MB/sec (as measured by bonnie++), then I  
should be able to write to the same array at a little faster than 11MB/ 
sec - as a read would usually read from 4 of 5 drives, however a write  
would go to all drives. This being said, I wouldn't expect one extra  
write to equal 12% of a read speed!

The other thing I wonder is if it has something to do with the  
sil_sata driver - as ALL the drives in the RAID5 are handled by that  
kernel module. The boot RAID1 is on the ICH5 SATA controller - and  
suffers no performance issues at all. It shows a good 40MB/sec+ read  
AND write speeds per drive.

--
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

Re: Software RAID5 write issues

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 4876 bytes --]

On Thu, 2009-06-11 at 10:05 +1000, Steven Haigh wrote:
> Isn't the PCI Bus limited to around 133MB/sec? If so, even with 3  
> drives on the same controller, you would expect divided equally that  
> each drive would get ~44MB/sec before overheads - not around 7MB/sec  
> per drive. I know I'm not going to get phenomenal performance with my  
> setup, but as most the data is archiving (and then copied to tape), I  
> would like to get things at least up to a reasonable level instead of  
> having a write speed of ~12% of the read speed.

It is, but it's shared amongst all the cards on that particular bus, and
in particular older motherboards would daisy chain busses such that a
later bus only gets part of that bandwidth because earlier busses are
using it too.  Plus the PCI bus limitation is 133MB/s theoretical
maximum, in practice you get less due to things like arbitration and
such.

> Hmmm - a very interesting read - but I am a little confused when it  
> comes to PCI bandwidth. I would assume (maybe wrongly) that if I can  
> READ from the array at 95MB/sec (as measured by bonnie++), then I  
> should be able to write to the same array at a little faster than 11MB/ 
> sec - as a read would usually read from 4 of 5 drives, however a write  
> would go to all drives. This being said, I wouldn't expect one extra  
> write to equal 12% of a read speed!

There are two factors involved in this (I'm speculating of course, but
here goes).

One, a read doesn't involve every drive in the array.  For any given
stripe, you will actually only read from 4 of the 5 drives.  Since 3 of
the drives are on the card, that means that 3 out of 5 stripes, one of
those drives will be the parity drive and therefore not used in the read
process.  So, for 3 out of 5 stripes, you actually read from two of the
drives behind the card and two on the motherboard.  The other two you
read from three of the drives behind the card and one on the
motherboard.  That accounts for a reasonable amount of difference all by
itself.  As an example, I have an external SATA drive case here that
holds 4 drives on a repeater and uses a single eSATA cable to run the 4
drives.  When accessing a single drive, I get 132MB/s throughput.  When
I access two drives, it drops to 60MB/s throughput per drive.  When
accessing three drives, it drops to 39MB/s throughput per drive.  So,
you can see where, on read, the lack of need to access all three drives
can really help on specific stripes.  In other words, reading from only
4 drives at a time *helps* your performance because whatever two drives
are in use behind the PCI card run faster and can keep up better with
the two drives on the motherboard.  Since writes always go to all 5
drives, you always get the slower speed (and you are writing 25% more
data to disk relative to the amount of actual data transfered than when
you are reading to boot).

Two, you use a 1MB chunk size.  Given a 5 drive raid5, that gives a 4MB
stripe width.  My guess is that your stripe size is large enough,
relative to your average write size, that your array is more often than
not performing a read/modify/write cycle for writing instead of a full
stripe write.  In a full stripe write, the md stack will write out all
four data chunks and a calculated parity chunk without regard to what
parity was before and what data was there before.  If, on the other
hand, it doesn't have enough data to write an entire stripe by the time
it is flushing things out, then it has to do a read/modify/write cycle.
The particulars of what's most efficient in this case depend on how many
chunks are being overwritten in the stripe, but regardless it means
reading in parts of the stripe and parity first, then doing xor
operations, then writing new data and new parity back out.  This will
mean that at least some of the 5 drives are doing both reads and writes
in a single stripe operation.  So, I think the combination of
read/modify/write cycles combined with the poor performance of drives
behind the PCI card actually can result in the drastic difference you
are seeing between read and write speeds.

> The other thing I wonder is if it has something to do with the  
> sil_sata driver - as ALL the drives in the RAID5 are handled by that  
> kernel module. The boot RAID1 is on the ICH5 SATA controller - and  
> suffers no performance issues at all. It shows a good 40MB/sec+ read  
> AND write speeds per drive.

It's entirely possible that the driver plays a role in this, yes.  I
don't have any hardware that uses that driver so I couldn't say.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

RE: Software RAID5 write issues

[Leslie Rhorer]

> You have 4 motherboard SATA ports, and 4 SATA ports on a PCI card.
> Right now you have your two OS drives on motherboard SATA ports, two of
> the five raid5 drives on motherboard SATA ports, and the three remaining
> raid5 drives on the PCI card SATA ports.  You need to get as many of the
> raid5 SATA disks on motherboard ports as possible.

'Or at least off the PCI card.  Depending on the motherboard, the
performance of the embedded ports may not be all that terrific, either.  I
have one Asus motherboard which is otherwise great, but whose on-board SATA
performance is anything but stellar.  If he can, I might suggest a different
SATA controller, either PCI Express or PCI-X.  There are a number of fairly
inexpensive PCI Express and PCI-X SATA controllers that provide very decent
performance - certainly better than he is seeing.  I have a couple of SiI
3124 based SATA controllers coupled with port multipliers and a couple of
Highpoint multilane SAS controllers that can all deliver in excess of
100MBps reads and in excess of 65 MBps writes on RAID5 and RAID6 arrays
across a 1G network.

> I would decide if
> you are more concerned about the raid5 array performing well (common, as
> it's usually the data you access most often) or the base OS array
> performing well (not so common, as it gets loaded largely into cache and
> doesn't get hit nearly so often as the data drive).  If you can deal
> with slowing down the OS drives, then I would move one of the OS drives
> to the PCI card and move one of the raid5 drives to the motherboard SATA
> port (and whichever drive you just moved over to the PCI card, I would
> mark it's raid1 arrays as write-mostly so that you don't read from it
> normally).

I think I would recommend replacing the controller and maybe going to a port
multiplier solution before I would shuffling the drives around using the
same old PCI card.  The PM is likely not going to be as fast or efficient as
a multilane solution, but a PCI Express or PCI-X card combined with a PM is
still probably going to be much faster than a PCI card.  It also allows for
more economic future expansion.

> Your big problem is that with 3 out of 5 raid5 drives on that PCI card,
> and sharing bandwidth, your total theoretical raid speed is abysmal.

I agree, or at least it's probably a big part of the problem.  He could of
course always have other problems, as well.

> When the three drives are sharing bandwidth on the card, they tend to
> split it up fairly evenly.  That means each drive gets roughly 1/3 of
> the PCI card's total available bandwidth over the PCI bus, which is
> generally poor in the first place.  Understand that a slow drive drags
> down *all* the drives in a raid5 array.  The faster drives just end up
> idling while waiting on the slower drive to finish its work (the faster
> drives will run ahead up to a point, then they eventually just get so
> far ahead that there isn't anything else for them to do until the
> slowest drive finishes up its stuff so old block requests can be
> completed, etc).  On the other hand, if you get 4 of the 5 drives on the
> motherboard ports, then that 5th drive on the PCI card won't be
> splitting bandwidth up and the overall array performance will shoot up
> (assuming the OS drives aren't also heavily loaded).

Yeah. I'm not even using SATA drives for my OS partitions.  I've got a bunch
of PATA drives lying around, good for little else, so rather than spend
money on new SATA drives or fiddle with booting from the arrays, I just put
a PATA drive on each system and use it for booting and OS.  While PATA ports
are getting somewhat rarer, most of even the most modern motherboards have
at least one IDE chain.  An ordinary old IDE drive with a capacity in the 80
- 160G range makes a perfectly good boot drive.  If he has a couple of old
unused IDE drives laying around, he might consider using  them.

> If you move one OS drive to the PCI card, then that leaves two raid5
> drives on the card.  In that case, I would seriously consider dropping
> back to a 4 drive array if you can handle the space reduction.  I would
> also seriously consider using raid4 instead of raid5 depending on your
> normal usage pattern.  If the data on the raid5 array is written once
> and then read over and over again, a raid4 can be beneficial in that you
> can stick the parity drive off on the PCI card and it won't be read from
> unless there is a drive failure or one the rare occasions when you write
> new data.

He's complaining more about write performance than read performance, so I
expect he would not be fond of this solution.