2x6 or 3x4 raid10 arrays ?

2x6 or 3x4 raid10 arrays ?

[Franck Routier]

Hi,

I am installing a database (postgresql) server.
I am considering two options:
- either setup two 6 disks raid10 arrays
- or setup three 4 disks raid10 arrays

You guessed I have 12 disks :)

Raw performance is better on 6 disks arrays, but having 3 arrays allows
me to setup 3 tablespaces and maybe to achieve better parallelism.

I am under the impression that I will get better results with requests
spread over 3 less effective arrays rather than two slightly more effive
one.

Does it make any sense, or am I totally missing the point ?

Thanks,
Franck

Re: 2x6 or 3x4 raid10 arrays ?

[Tim Southerwood]

Franck Routier wrote:
> Hi,
> 
> I am installing a database (postgresql) server.
> I am considering two options:
> - either setup two 6 disks raid10 arrays
> - or setup three 4 disks raid10 arrays
> 
> You guessed I have 12 disks :)
> 
> Raw performance is better on 6 disks arrays, but having 3 arrays allows
> me to setup 3 tablespaces and maybe to achieve better parallelism.
> 
> I am under the impression that I will get better results with requests
> spread over 3 less effective arrays rather than two slightly more effive
> one.
> 
> Does it make any sense, or am I totally missing the point ?
> 
> Thanks,
> Franck
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi

No, it makes total sense.

Having just done a lot of work on optimising Postgresql:

How many distinct table spaces depends on your expected usage pattern - 
ie if you have one table that is hammered for updates all the time you 
might benefit from having the table storage on a physically separate 
tablespace to any indexes for that table.

If you are hammering some tables for updates whilst querying other 
tables at a high rate, placing the updating tables on a different 
tablespace to the ones you are querying may benefit.

The nice thing about Postgresql 8.1 upwards (at least I haven't tried 
this under 8.0) is that you can ALTER TABLE|INDEX to use a different 
tablespace at run time on a live database, so experimentation is easy.

However, the single biggest improvement I found was to ensure that the 
WAL is redirected to a otherwise quiescent disk.

In my case, I arranged two physically separate storage volumes thus:

VOL1: OS (/) + WAL
VOL2: DB storage (minus WAL) + /var/log

Taking /var/log off VOL1 rendered it fairly quiet after all applications 
had started and having the WAL on a quiet volume gave me a tenfold 
improvement in INSERT rate, so not insignificant.

If you are expecting heavy insert/update accesses, I would suggest you 
take two disks off as a RAID1 mirror and devote them entirely to the WAL.

Cheers

Tim

Re: 2x6 or 3x4 raid10 arrays ?

[Brendan Conoboy]

Franck Routier wrote:
> I am under the impression that I will get better results with requests
> spread over 3 less effective arrays rather than two slightly more effive
> one.

Depends on your workload.  If you use a larger chunk size, regardless of 
raid layout, you can try to get each db transaction to only hit one 
spindle.  Seek time being a limiting factor, this may scale better when 
you have dozens of simultaneous requests.

-- 
Brendan Conoboy / Red Hat, Inc. / blc@redhat.com

Re: 2x6 or 3x4 raid10 arrays ?

[Janek Kozicki]

Franck Routier said:     (by the date of Thu, 28 Feb 2008 11:45:54 +0100)

> Hi,
> 
> I am installing a database (postgresql) server.
> I am considering two options:
> - either setup two 6 disks raid10 arrays
> - or setup three 4 disks raid10 arrays
> 
> You guessed I have 12 disks :)

wouldn't be a single raid10 far=2, the fastest solution? But you get
only half of the capacity.

-- 
Janek Kozicki                                                         |

Re: 2x6 or 3x4 raid10 arrays ?

[Janek Kozicki]

Janek Kozicki said:     (by the date of Thu, 28 Feb 2008 19:25:00 +0100)

sorry about replying to myself.

* two 6 disks raid10 arrays : theoretical max speed 6 times single disc
* three 4 disks raid10 arrays : theoretical max speed 4 times single disc
* single raid10 far=2  : theoretical max speed 12 times single disc (!)

isn't that true?

-- 
Janek Kozicki                                                         |

Re: 2x6 or 3x4 raid10 arrays ?

[Brendan Conoboy]

Janek Kozicki wrote:
> * two 6 disks raid10 arrays : theoretical max speed 6 times single disc
> * three 4 disks raid10 arrays : theoretical max speed 4 times single disc
> * single raid10 far=2  : theoretical max speed 12 times single disc (!)
> 
> isn't that true?

If speed is raw throughput then that's approximately right.  Unless 
you're streaming huge piles of data, that sort of speed isn't what you 
need to tune for, though.  Look at it this way:

two 6 disk raid 10 arrays : theoretical max seeks per operation: 6 times 
the seeks of a single disk

three 4 disk raid 10 arrays: theoretical max seeks per operation: 4 
times the seeks of a single disk

one 12 disk raid 10 arrays: theoretical max seeks per operation: 12 
times the seeks of a single disk!

Seeks are bad.  You have to tune for your workload.

-- 
Brendan Conoboy / Red Hat, Inc. / blc@redhat.com

Re: 2x6 or 3x4 raid10 arrays ?

[Bill Davidsen]

Janek Kozicki wrote:
> Janek Kozicki said:     (by the date of Thu, 28 Feb 2008 19:25:00 +0100)
>
> sorry about replying to myself.
>
> * two 6 disks raid10 arrays : theoretical max speed 6 times single disc
> * three 4 disks raid10 arrays : theoretical max speed 4 times single disc
> * single raid10 far=2  : theoretical max speed 12 times single disc (!)
>
> isn't that true?
>
>   
True for throughput, not for seek. Also, what I have seen and read 
indicates that smaller chunks help a lot for database with lots of seeks.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: 2x6 or 3x4 raid10 arrays ?

[Keld Jørn Simonsen]

On Sat, Mar 01, 2008 at 03:07:59PM -0500, Bill Davidsen wrote:
> Janek Kozicki wrote:
> >Janek Kozicki said:     (by the date of Thu, 28 Feb 2008 19:25:00 +0100)
> >
> >sorry about replying to myself.
> >
> >* two 6 disks raid10 arrays : theoretical max speed 6 times single disc
> >* three 4 disks raid10 arrays : theoretical max speed 4 times single disc
> >* single raid10 far=2  : theoretical max speed 12 times single disc (!)
> >
> >isn't that true?
> >
> >  
> True for throughput, not for seek. Also, what I have seen and read 
> indicates that smaller chunks help a lot for database with lots of seeks.

Why not for seek? I would have thought that seeks are almost the same,
and actually smaller for raid10,n2 with more disks, as raid10,f2 limits
read searches to the first part of the disks, and head movement thus are
much smaller, say 1/12 of the whole disk in stead of 1/4 for each of the
disks in a 4 disk array.

Best regards
keld

Re: 2x6 or 3x4 raid10 arrays ?

[Nat Makarevitch]

Franck Routier <franck.routier <at> axege.com> writes:

> database (postgresql) server.

AFAIK if the average size of an I/O operation as long as the corresponding
variance are low... go for a single RAID10,f2 with a stripe size slightly
superior to this average. This way you will have most requests mobilizing only a
single spindle and all your spindles acting in parallel. If this average size
varies upon tables one may create a RAID (with the adequate stripe size) per
database partition.

-- 
http://www.makarevitch.org/rant/raid/

Re: 2x6 or 3x4 raid10 arrays ?

[Keld Jørn Simonsen]

On Thu, Feb 28, 2008 at 10:36:22PM +0000, Nat Makarevitch wrote:
> Franck Routier <franck.routier <at> axege.com> writes:
> 
> > database (postgresql) server.
> 
> AFAIK if the average size of an I/O operation as long as the corresponding
> variance are low... go for a single RAID10,f2 with a stripe size slightly
> superior to this average. This way you will have most requests mobilizing only a
> single spindle and all your spindles acting in parallel. If this average size
> varies upon tables one may create a RAID (with the adequate stripe size) per
> database partition.

I believe that a full chunk is read for each read access.
Or, at least, if one operation can be done within one chunk,
not more than that chunk is operated upon.

And chunks are recommended to be between 256 kiB and 1 MiB.
Most random database reads are much smaller than 256 kiB.
So the probability that one random read can be done with just one 
seek + read operation is very high, as far as I understand it.

This would lead to that it is not important whether to use 
two arrays of 6 disks each, or 3 arrays of 4 disks each. 
Or for that sake 1 array of 12 disks.

Some other factors may be more important: such as the ability to survive
disk crashes. raid10,f2 is good for surviving 1 disk crash. If you have
3 raids of 4 disks, it can survive a disk crash in each of these raids.
Furthermore some combinations of crashes of 2 disks within a raid can
also be survived. There are 16 combinations of failing disksi, with 0 to
4 disks failing:

   0000 Y  0001 Y 0010 Y  0011 N 0100 Y 0101 N 0110 Y 0111 N
   1000 Y  1001 Y 1010 N  1011 N 1100 N 1101 N 1110 N 1111 N

So if 2 disks fails, then in 2 out of 6 cases this would be ok,
given a succes rates of 1.33 for surviving one or two disk crashes.
For 3 raid sets of 4 drives each, this should be able to survive 4
(3 * 1.33) disks malfunctioning on average, while it is only guaranteed
to survive 1 bad disk.

With more disks in the array, more combinations would be possible to
survive. I do not have a formula for it, but surely it should exist.
Maybe arrays with an odd number of drives would have better chances of
surviving, given that in the 4-drive example a number of combinations
of failing drives were fatal because all even chunks were distributed on
disks 1 and 3, while odd chunks were on disks 2 and 4). 
I would like to know if somebody could come up with a
formula, and what results one would get for  a 2 x 6 disk array, and 1 x
12 disks array. 

Another more important item could be speed: especially for average seek
times. More disks in an array would for raid10,f2 reduce the max and
average seek times as searching on each disks would be reduced in time,
for n disks in a raid only 1/n of each disk will be used for reading. 
   

Best regards
keld

Re: 2x6 or 3x4 raid10 arrays ?

[Keld Jørn Simonsen]

On Sat, Mar 01, 2008 at 09:40:20PM +0100, Keld Jørn Simonsen wrote:
> On Thu, Feb 28, 2008 at 10:36:22PM +0000, Nat Makarevitch wrote:
> > Franck Routier <franck.routier <at> axege.com> writes:
> > 
> > > database (postgresql) server.
> > 
> > AFAIK if the average size of an I/O operation as long as the corresponding
> > variance are low... go for a single RAID10,f2 with a stripe size slightly
> > superior to this average. This way you will have most requests mobilizing only a
> > single spindle and all your spindles acting in parallel. If this average size
> > varies upon tables one may create a RAID (with the adequate stripe size) per
> > database partition.
> 
> I believe that a full chunk is read for each read access.
> Or, at least, if one operation can be done within one chunk,
> not more than that chunk is operated upon.
> 
> And chunks are recommended to be between 256 kiB and 1 MiB.
> Most random database reads are much smaller than 256 kiB.
> So the probability that one random read can be done with just one 
> seek + read operation is very high, as far as I understand it.
> 
> This would lead to that it is not important whether to use 
> two arrays of 6 disks each, or 3 arrays of 4 disks each. 
> Or for that sake 1 array of 12 disks.
> 
> Some other factors may be more important: such as the ability to survive
> disk crashes. raid10,f2 is good for surviving 1 disk crash. If you have
> 3 raids of 4 disks, it can survive a disk crash in each of these raids.
> Furthermore some combinations of crashes of 2 disks within a raid can
> also be survived. There are 16 combinations of failing disksi, with 0 to
> 4 disks failing:
> 
>    0000 Y  0001 Y 0010 Y  0011 N 0100 Y 0101 N 0110 Y 0111 N
>    1000 Y  1001 Y 1010 N  1011 N 1100 N 1101 N 1110 N 1111 N
> 
> So if 2 disks fails, then in 2 out of 6 cases this would be ok,
> given a succes rates of 1.33 for surviving one or two disk crashes.
> For 3 raid sets of 4 drives each, this should be able to survive 4
> (3 * 1.33) disks malfunctioning on average, while it is only guaranteed
> to survive 1 bad disk.
> 
> With more disks in the array, more combinations would be possible to
> survive. I do not have a formula for it, but surely it should exist.
> Maybe arrays with an odd number of drives would have better chances of
> surviving, given that in the 4-drive example a number of combinations
> of failing drives were fatal because all even chunks were distributed on
> disks 1 and 3, while odd chunks were on disks 2 and 4). 
> I would like to know if somebody could come up with a
> formula, and what results one would get for  a 2 x 6 disk array, and 1 x
> 12 disks array. 

I would actually think that for a 12 disk raid, it would be hard to
survive more than 2 disk in average. For 2 times 6 disks the probability
of surviving about 3 bad disks is possible. But I would like a formula.

So smaller raid sizes are good here. For example with 6 sets of
raid10,f2 raids, you can loose 1 disk in each array, thus surviving
about 6 bad disks on average. Again, if just 2 disks fail, in the same 
raid, you are lost. So no guarantees on surviving 6 bad disks.

> Another more important item could be speed: especially for average seek
> times. More disks in an array would for raid10,f2 reduce the max and
> average seek times as searching on each disks would be reduced in time,
> for n disks in a raid only 1/n of each disk will be used for reading. 

Speed would also improve with bigger arrays, as IO bandwidth per
operation would become bigger. Eg. if you traverse all entries in a
table, you can have IO to all 12 disks at the same time. Having it all
in one big array make the system automatically distribute the database
accesses to all 12 disks, while you need to do the distribution by hand
between the arrays if you make 3 sets of 4 disks.

Best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2x6 or 3x4 raid10 arrays ?

[Nat Makarevitch]

Keld Jørn Simonsen <keld <at> dkuug.dk> writes:

> I believe that a full chunk is read for each read access.

I've read various definitions for "chunk". Is a "stripe" a 'cluster' (in the
"group of disk sectors" meaning) on a single physical drive (device, let's say
"spindle"), and a 'chunk' a set of stripes made with a single stripe of each
spindle? For what I understand this is the definition used in the 'md' world
(see 'man mdadm'), therefore I will use it thereafter.

Yes, AFAIK a full chunk is concerned by each access.

> Most random database reads are much smaller than 256 kiB.
> So the probability that one random read can be done with just one 
> seek + read operation is very high, as far as I understand it.

Indeed. In fact I proposed to define the chunk size with respect to the (known)
average size of read/written data blocks. Most database servers are able to show
this (you let your application run normally for hours/day, then obtain the
information), one can also use some instrumentation (blktrace...)

> This would lead to that it is not important whether to use 
> two arrays of 6 disks each, or 3 arrays of 4 disks each. 
> Or for that sake 1 array of 12 disks.

I beg to disagree. Creating more than one array may be OK when you very
precisely know your load profile per table, but in most cases this is not true,
or this profile will vary, therefore your best bet is "to maintain, for each
request, as much disk heads available as possible", carpet-bomb the array with
all requests and let the elevator(s) optimize. Another way to see it, in some
reciprocal way, is to say that you don't want to have any head sleeping when
there is a request to serve. 

> Some other factors may be more important: such as the ability to survive
> disk crashes

That's very true, however one may not neglect logistics. If I'm pretty sure that
I can change a spindle in less than 2 hours after a failure I will prefer using
all disks less one on a single array and letting the last one as a connected
(but powered off) spare. The alarm trips, some automatic or manual procedure
powers the spare and mounts it in the array, while the procedure aiming at
physically extracting the failed device and replacing it (it will become the new
spare) rolls. With more latency-prone logistics one may reserve more disks as
spares.


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2x6 or 3x4 raid10 arrays ?

[Keld Jørn Simonsen]

On Sat, Mar 01, 2008 at 10:05:06PM +0000, Nat Makarevitch wrote:
> Keld Jørn Simonsen <keld <at> dkuug.dk> writes:
> 
> > I believe that a full chunk is read for each read access.
> 
> I've read various definitions for "chunk". Is a "stripe" a 'cluster' (in the
> "group of disk sectors" meaning) on a single physical drive (device, let's say
> "spindle"), and a 'chunk' a set of stripes made with a single stripe of each
> spindle? For what I understand this is the definition used in the 'md' world
> (see 'man mdadm'), therefore I will use it thereafter.

I have understood chunk to be a set of sectors on a single device.
If it was to be understood as a set of space on multiple devices, then
the chunk size would normaly be relative to the number of devices
involved. And that is not what we talk about in eg mdadm chunk=256

> Yes, AFAIK a full chunk is concerned by each access.

Given your definition of chunk, I do not beleve so. A read of a chunk on
eg a 12 device raid, would not involve 12 reads, one on each device.

> > This would lead to that it is not important whether to use 
> > two arrays of 6 disks each, or 3 arrays of 4 disks each. 
> > Or for that sake 1 array of 12 disks.
> 
> I beg to disagree. Creating more than one array may be OK when you very
> precisely know your load profile per table, but in most cases this is not true,
> or this profile will vary, therefore your best bet is "to maintain, for each
> request, as much disk heads available as possible", carpet-bomb the array with
> all requests and let the elevator(s) optimize. Another way to see it, in some
> reciprocal way, is to say that you don't want to have any head sleeping when
> there is a request to serve. 

This is for bigger operations. I believe that for smaller operations,
such as a random read in a database, you would only like to have one IO
operation on one device. Seek times are important here. You can in one
average seek time of say 10 milliseconds on SATA-II transfer 800 kB
data, which is much more than normal size of a record in a table.
Having 12 operations for a record of say 1 kB would be very expensive.
The elevator is better of serving a number of concurrent requests.

> > Some other factors may be more important: such as the ability to survive
> > disk crashes
> 
> That's very true, however one may not neglect logistics. If I'm pretty sure that
> I can change a spindle in less than 2 hours after a failure I will prefer using
> all disks less one on a single array and letting the last one as a connected
> (but powered off) spare. The alarm trips, some automatic or manual procedure
> powers the spare and mounts it in the array, while the procedure aiming at
> physically extracting the failed device and replacing it (it will become the new
> spare) rolls. With more latency-prone logistics one may reserve more disks as
> spares.

Yes, rebuild time would also be a factor. Smaller raids are quicker to
rebuild, I think. Or maybe this is irrelevant, at least for raid10,f2 -
as rebuilding is done concurrently from all devices involved, and
probebly limited by the write speed of the replacing device.

Best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2x6 or 3x4 raid10 arrays ?

[Nat Makarevitch]

Keld Jørn Simonsen <keld <at> dkuug.dk> writes:

> I have understood chunk to be a set of sectors on a single device.

You are right, I completely botched my reply

>> Creating more than one array may be OK when you very
>> precisely know your load profile per table

>> your best bet is "to maintain, for each
>> request, as much disk heads available as possible", carpet-bomb the array
>> with all requests and let the elevator(s) optimize

> This is for bigger operations. I believe that for smaller operations,
> such as a random read in a database, you would only like to have one IO
> operation on one device.

We agree on this. My reply was tailored upon the prerequisite of a chunk size
defined to be slightly superior to the size of the data needed for most requests.

Using multiple arrays may be useful if all tables are accessed at similar rates
while some have much bigger average amount of bytes implied by any request:
their array need a bigger chunk size.

> > > Some other factors may be more important: such as the ability to survive
> > > disk crashes
> > 
> > That's very true, however one may not neglect logistics.

> Yes, rebuild time would also be a factor.

I think so

> Smaller raids are quicker to rebuild

It depends on device performance (putting online a recent small-capacity device
will lead to quick rebuild), workload during rebuild...

> for raid10,f2 -
> probebly limited by the write speed of the replacing device.

I think so.


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2x6 or 3x4 raid10 arrays ?

[Bill Davidsen]

Franck Routier wrote:
> Hi,
>
> I am installing a database (postgresql) server.
> I am considering two options:
> - either setup two 6 disks raid10 arrays
> - or setup three 4 disks raid10 arrays
>   

One more thought on this, if you use is such that you have one table 
which is really getting heavy read use, have more than two copies. Bad 
for disk utilization, good for having the data on an idle spindle.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark