Disk Order

Disk Order

[Jonathan Tripathy]

Hi Everyone,

Our server currently has 6 SATA ports on the motherboard filled with 6 
drives in RAID10. I have a need to install an additional PCIe SATA 
controller in a server in order to obtain 2 more SATA ports to install 
an additional 2 new drives (As I want a new RAID1 array). However I 
would like to move one of the current drives onto the new controller, 
and one of the new drives onto the motherboard. This should give me a 
little redundancy in case the new controller card were to fail (I don't 
want the new controller to bring down 2 drives).

When setting up MD RAID in this configuration, how would I go about tell 
MD RAID which disk is which in the array (referring to the near/far 
configurations)? I'm assuming by default that MD RAID just uses 
alphabetical order in order to determine the drive order?

Let's say the RAID10 drives will be sda sdb sdc sdd sde sdg*, where sdg 
is the second of the RAID0 strip of sde.
And the two new RAID1 drives are sdf sdh*.
The drives with the asterisks beside them will be on the new PCIe 
controller.

Do I have to do anything special for this configuration? Will there be a 
performance hit since one drive in the RAID10 array will be on a 
different controller?

Thanks

Re: Disk Order

[Peter Grandi]

> Our server currently has 6 SATA ports on the motherboard
> filled with 6 drives in RAID10. [ ... ] additional PCIe SATA
> controller in a server [ ... ] give me a little redundancy in
> case the new controller card were to fail (I don't want the
> new controller to bring down 2 drives).

It would be nice to split each pair of your new RAID1 and the
existing RAID10 over the two host adapters (the controllers are
the PCBs on the bottom of the drive...) you will have.

As in one member of each pair on the motherboard and the other
on the PCIe card.

Ideally therefore get a 4-drive PCIe card, so you can put on it
3 members of your existing RAID10 set, plus one member of your
new RAID1 set. SPAN.com tend to have reasonable non-RAID SATA
cards.

> When setting up MD RAID in this configuration, how would I go
> about tell MD RAID which disk is which in the array (referring
> to the near/far configurations)?

If you are using a recent MD setup each member block device (not
necessarily disk) has an MD descriptor block with an UUID (or a
name) for the array and one for the member, and ideally you
would use that.

You can use 'mdadm -E /dev/sd?' to get the relevant information,
and them put the relevant UUID (or name) in '/etc/mdadm.conf'

> I'm assuming by default that MD RAID just uses alphabetical
> order in order to determine the drive order? Let's say the
> RAID10 drives will be sda sdb sdc sdd sde sdg* [ ... ]

Drive names can change on any reboot. You cannot rely on them.

> Will there be a performance hit since one drive in the RAID10
> array will be on a different controller?

PCIe host adapters tend to have lower host bus bandwith than
motherboard ones, but this should not matter in your case (if
your host adapter has a 4-lane PCIe interface).

Re: Disk Order

[Rudy Zijlstra]

Jonathan,

On 12-08-12 23:10, Jonathan Tripathy wrote:
> Hi Everyone,
>
> Our server currently has 6 SATA ports on the motherboard filled with 6 
> drives in RAID10. I have a need to install an additional PCIe SATA 
> controller in a server in order to obtain 2 more SATA ports to install 
> an additional 2 new drives (As I want a new RAID1 array). However I 
> would like to move one of the current drives onto the new controller, 
> and one of the new drives onto the motherboard. This should give me a 
> little redundancy in case the new controller card were to fail (I 
> don't want the new controller to bring down 2 drives).
>
> When setting up MD RAID in this configuration, how would I go about 
> tell MD RAID which disk is which in the array (referring to the 
> near/far configurations)? I'm assuming by default that MD RAID just 
> uses alphabetical order in order to determine the drive order?
mdadm does not care about the disk ordering once the raid has been 
defined. It uses the raid metadata on the disks to figure out which disk 
is doing what. This includes which disk belongs to which raid.

There are 2 options:
- use metadata .90 (no longer suggested) and have the kernel autodetect 
during boot
- use a later metadata (1.2 is default now), and have  mdadm in the 
initrd. This will use a config file that has mdadm search on UID. Again, 
independent of which sdx the disks are.

You could test by swapping disk order in your current system. Although i 
would suggest to do testing with a test machine ;)

Cheers,


Rudy

Re: Disk Order

[Jonathan Tripathy]

On 13/08/2012 00:16, Rudy Zijlstra wrote:
> Jonathan,
>
> On 12-08-12 23:10, Jonathan Tripathy wrote:
>> Hi Everyone,
>>
>> Our server currently has 6 SATA ports on the motherboard filled with 
>> 6 drives in RAID10. I have a need to install an additional PCIe SATA 
>> controller in a server in order to obtain 2 more SATA ports to 
>> install an additional 2 new drives (As I want a new RAID1 array). 
>> However I would like to move one of the current drives onto the new 
>> controller, and one of the new drives onto the motherboard. This 
>> should give me a little redundancy in case the new controller card 
>> were to fail (I don't want the new controller to bring down 2 drives).
>>
>> When setting up MD RAID in this configuration, how would I go about 
>> tell MD RAID which disk is which in the array (referring to the 
>> near/far configurations)? I'm assuming by default that MD RAID just 
>> uses alphabetical order in order to determine the drive order?
> mdadm does not care about the disk ordering once the raid has been 
> defined. It uses the raid metadata on the disks to figure out which 
> disk is doing what. This includes which disk belongs to which raid.
>
> There are 2 options:
> - use metadata .90 (no longer suggested) and have the kernel 
> autodetect during boot
> - use a later metadata (1.2 is default now), and have  mdadm in the 
> initrd. This will use a config file that has mdadm search on UID. 
> Again, independent of which sdx the disks are.
>
> You could test by swapping disk order in your current system. Although 
> i would suggest to do testing with a test machine ;)
>
Thanks for your reply.

Currently, we use CentOS and use the installer to set up the MD RAID. It 
is using metadata version 0.90. The only thing in mdadm.conf is the UUID 
of the array and the number of disks per array. It's good to know that 
mdadm doesn't look as the sdx value once the RAID is setup. What I would 
like to know is how to configure the disk order in the array. Or maybe 
it's not very important to do this? For example, going by these 
diagrams: 
http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10

how do I know/configure which disk is A1 and which is A2? I just want to 
know the disk order after i split my array across different adapters.

Thanks

Re: Disk Order

[Phil Turmel]

Hi Jonathan,

On 08/12/2012 05:10 PM, Jonathan Tripathy wrote:
> Hi Everyone,
> 
> Our server currently has 6 SATA ports on the motherboard filled with 6
> drives in RAID10. I have a need to install an additional PCIe SATA
> controller in a server in order to obtain 2 more SATA ports to install
> an additional 2 new drives (As I want a new RAID1 array). However I
> would like to move one of the current drives onto the new controller,
> and one of the new drives onto the motherboard. This should give me a
> little redundancy in case the new controller card were to fail (I don't
> want the new controller to bring down 2 drives).

Reasonable plan.

> When setting up MD RAID in this configuration, how would I go about tell
> MD RAID which disk is which in the array (referring to the near/far
> configurations)? I'm assuming by default that MD RAID just uses
> alphabetical order in order to determine the drive order?

No, MD uses the order they were specified in the "--create" operation,
and records that in metadata on each device.  Re-assembly can specify
them in any order--MD will put them into the correct role.

> Let's say the RAID10 drives will be sda sdb sdc sdd sde sdg*, where sdg
> is the second of the RAID0 strip of sde.
> And the two new RAID1 drives are sdf sdh*.
> The drives with the asterisks beside them will be on the new PCIe
> controller.

Probe order during boot is not guaranteed--you might discover that the
new controller gets /dev/sda and /dev/sdb.  Or with parallel probing you
might get /dev/sda[acefgh] on the mobo and /dev/sd[bd] on the new
controller.  (Not terribly likely, in my limited experience.)

You should consider recording what drive serial number is hooked to what
port, and compare that to your device names.  You might find my "lsdrv"
script useful for this, as that was its original purpose.[1]

Or just print a copy of "ls -l /dev/disk/by-id/".

> Do I have to do anything special for this configuration? Will there be a
> performance hit since one drive in the RAID10 array will be on a
> different controller?

Might be faster, if you have really fast drives.  Unlikely to be slower.

HTH,

Phil

[1] http://github.com/pturmel/lsdrv

Re: Disk Order

[Phil Turmel]

Hi Jonathan,

On 08/12/2012 07:47 PM, Jonathan Tripathy wrote:

[trim /]

> how do I know/configure which disk is A1 and which is A2? I just want to
> know the disk order after i split my array across different adapters.

The output of "mdadm -D /dev/md0" will show you the order in the
"RaidDevice" column.

HTH,

Phil

Re: Disk Order

[Jonathan Tripathy]

> No, MD uses the order they were specified in the "--create" operation,
> and records that in metadata on each device.  Re-assembly can specify
> them in any order--MD will put them into the correct role.
>
>
Thanks for this. So basically, the naming of the device is only 
important at the time of creating the array (as that defines the 
layout). As soon as I create the array, mdadm puts it's own metadata on 
the drive so it can identify them afterwards? That sounds very good indeed.


Now, onto the topic of PCIe adapter. I was looking at this: 
http://www.cclonline.com/product/42541/PEXSAT32/IDE-SATA-SCSI-Cards/StarTech-2-Port-SATA-6-Gbps-PCI-Express-SATA-Controller-Card/CNT1075/

Is that a bad model? Seems a little on the cheap side? Can anyone 
suggest a good model?

Many Thanks for all the help

Re: Disk Order

[Phil Turmel]

On 08/12/2012 08:04 PM, Jonathan Tripathy wrote:
> 
> Now, onto the topic of PCIe adapter. I was looking at this:
> http://www.cclonline.com/product/42541/PEXSAT32/IDE-SATA-SCSI-Cards/StarTech-2-Port-SATA-6-Gbps-PCI-Express-SATA-Controller-Card/CNT1075/
> 
> 
> Is that a bad model? Seems a little on the cheap side? Can anyone
> suggest a good model?

Amazon users aren't impressed.  But the limitation is really the PCIe x1
v2 bus interface: 500MB/sec.  Two SSDs would certainly overwhelm it, and
any other similar card.

I have a similar card, the Highpoint 620, in an older system at home
(PCIe v1): it maxes out at 200MB/sec.

So, it depends on what you want it to do.

{I'll let others make card recommendations--I prefer to *listen* to those.}

Phil

Re: Disk Order

[Rudy Zijlstra]

On 13-08-12 02:37, Phil Turmel wrote:
> On 08/12/2012 08:04 PM, Jonathan Tripathy wrote:
>> Now, onto the topic of PCIe adapter. I was looking at this:
>> http://www.cclonline.com/product/42541/PEXSAT32/IDE-SATA-SCSI-Cards/StarTech-2-Port-SATA-6-Gbps-PCI-Express-SATA-Controller-Card/CNT1075/
>>
>>
>> Is that a bad model? Seems a little on the cheap side? Can anyone
>> suggest a good model?
> Amazon users aren't impressed.  But the limitation is really the PCIe x1
> v2 bus interface: 500MB/sec.  Two SSDs would certainly overwhelm it, and
> any other similar card.
>
> I have a similar card, the Highpoint 620, in an older system at home
> (PCIe v1): it maxes out at 200MB/sec.
>
> So, it depends on what you want it to do.
>
> {I'll let others make card recommendations--I prefer to *listen* to those.}
>
http://azerty.nl/0-1036-67180/promise-sata300-tx4-controll.html

That card usually gives good results. If you can find the following, 
those are very good:

http://azerty.nl/0-1640-381070/lsi-megaraid-sas-9211-8i-sto.html

There are other, perhaps even more simple cards from LSI with very good 
properties

Cheers,


Rudy -- who uses LSI cards, and still has some promise cards in use.