why partition arrays?

why partition arrays?

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

As the Debian mdadm maintainer, I am often subjected to questions
about partitionable arrays; people seem to want to use them in
favour of normal arrays. I don't understand why.

There's possibly an argument to be made about flexibility when it
comes to resizing partitions within the array, but even most MD
array types can be resized now.

There's possibly an argument about saving space because of fewer
sectors used/wasted with superblock information, but I am not going
to buy that.

Why would anyone want to create a partitionable array and put
partitions in it, rather than creating separate arrays for each
filesystem? Intuitively, this makes way more sense as then the
partitions are independent of each other; one array can fail and the
rest still works -- part of the reason why you partition in the
first place.

Would anyone help me answer this FAQ?

(btw: [0] and [1] are obviously for public consumption; they are
available under the terms of the artistic licence 2.0)

0. http://svn.debian.org/wsvn/pkg-mdadm/mdadm/trunk/debian/FAQ?op=file&rev=0&sc=0
1. http://svn.debian.org/wsvn/pkg-mdadm/mdadm/trunk/debian/README.recipes?op=file&rev=0&sc=0

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: madduck.bogus@madduck.net
 
"the liar at any rate recognises that recreation, not instruction, is
 the aim of conversation, and is a far more civilised being than the
 blockhead who loudly expresses his disbelief in a story which is told
 simply for the amusement of the company."
                                                        -- oscar wilde

[-- Attachment #2: Digital signature (GPG/PGP) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: why partition arrays?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 3426 bytes --]

On Wed, 2006-10-18 at 14:42 +0200, martin f krafft wrote:
> Why would anyone want to create a partitionable array and put
> partitions in it, rather than creating separate arrays for each
> filesystem? Intuitively, this makes way more sense as then the
> partitions are independent of each other; one array can fail and the
> rest still works -- part of the reason why you partition in the
> first place.
> 
> Would anyone help me answer this FAQ?

There are a couple reasons I can think.

First, not all md types make sense to be split up, aka multipath.  For
those types, when a disk fails, the *entire* disk is considered to be
failed, but with different arrays you won't fail over to the next path
until each md array has attempted to access the bad path.  This can have
obvious bad consequences for certain array types that do automatic
failover from one port to another (you can end up getting the array in a
loop of switching ports repeatedly to satisfy the fact that one array
failed over during a path down, then the path came back up, and another
array stayed on the old path because it didn't send any commands during
the path down time period).

Second, convenience.  Assume you have a 6 disk raid5 array.  If a disk
fails and you are using a partitioned md array, then all the partitions
on the disk will already be handled without using that disk.  No need to
manually fail any still active array members from other arrays.

Third, safety.  Again with the raid5 array.  If you use multiple arrays
on a single disk, and that disk fails, but it only failed on one array,
then you now need to manually fail that disk from the other arrays
before shutting down or hot swapping the disk.  Generally speaking,
that's not a big deal, but people do occasionally have fat finger
syndrome and this is a good opportunity for someone to accidentally fail
the wrong disk, and when you then go to remove the disk you create a two
disk failure instead of one and now you are in real trouble.

Forth, to respond to what you wrote about independent of each other --
part of the reason why you partition.  I would argue that's not true.
If your goal is to salvage as much use from a failing disk as possible,
then OK.  But, generally speaking, people that have something of value
on their disks don't want to salvage any part of a failing disk, they
want that disk gone and replaced immediately.  There simply is little to
no value in an already malfunctioning disk.  They're too cheap and the
data stored on them too valuable to risk loosing something in an effort
to further utilize broken hardware.  This of course is written with the
understanding that the latest md raid code will do read error rewrites
to compensate for minor disk issues, so anything that will throw a disk
out of an array is more than just a minor sector glitch.

> (btw: [0] and [1] are obviously for public consumption; they are
> available under the terms of the artistic licence 2.0)
> 
> 0. http://svn.debian.org/wsvn/pkg-mdadm/mdadm/trunk/debian/FAQ?op=file&rev=0&sc=0
> 1. http://svn.debian.org/wsvn/pkg-mdadm/mdadm/trunk/debian/README.recipes?op=file&rev=0&sc=0
> 
-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: why partition arrays?

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

also sprach Doug Ledford <dledford@redhat.com> [2006.10.18.1526 +0200]:
> There are a couple reasons I can think.

Thanks for your elaborate response. If you don't mind, I shall link
to it from the FAQ.

I have one other question: do partitionable and traditional arrays
actually differ in format? Put differently: can I assemble
a traditional array as a partitionable one simply by specifying:

  mdadm --create ... /dev/md0 ...
  mdadm --stop /dev/md0
  mdadm --assemble --auto=part ... /dev/md0 ...

? Or do the superblocks actually differ?

Thanks,

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: madduck.bogus@madduck.net
 
the images rushed around his mind and tried
to find somewhere to settle down and make sense.
            -- douglas adams, "the hitchhiker's guide to the galaxy"

[-- Attachment #2: Digital signature (GPG/PGP) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: why partition arrays?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

On Wed, 2006-10-18 at 15:43 +0200, martin f krafft wrote:
> also sprach Doug Ledford <dledford@redhat.com> [2006.10.18.1526 +0200]:
> > There are a couple reasons I can think.
> 
> Thanks for your elaborate response. If you don't mind, I shall link
> to it from the FAQ.

Sure.

> I have one other question: do partitionable and traditional arrays
> actually differ in format? Put differently: can I assemble
> a traditional array as a partitionable one simply by specifying:
> 
>   mdadm --create ... /dev/md0 ...
>   mdadm --stop /dev/md0
>   mdadm --assemble --auto=part ... /dev/md0 ...
> 
> ? Or do the superblocks actually differ?

Neil would be more authoritative about what would differ in the
superblocks, but yes, it is possible to do as you listed above.  In
fact, if you create a partitioned array, and your mkinitrd doesn't
restart it as a partitioned array, you'll wonder how to mount your
filesystems since the system will happily start that originally
partitioned array as non partitioned.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: why partition arrays?

[Ken Walker]

So is LVM better for partitions on a large raid5, or any raid, than separate
partitions on that array.

I'm still in my learning curve :)

for example, if one has Linux running on a two disk mirror array, raid1, and
the first disk is partitioned, say 5 partitions, with those partitions
mirrored on the second disk, and each identical partition is then run as a
mirror raid1.

What your saying is that, if a single partition fails, to remove the drive
you have to fail all the array partitions on the drive your taking out, then
rebuild the partitions and then add to the dirty raid the new partitions one
at a time.


Will LVM remove all this, so if you have a mirror as a single raid
partition, and use LVM to create the partitions on that mirror, if a disk
goes down, can it be removed, replaced, and then just added to the single
raid, with LVM having had no idea what was going on in the background and
just plod along merrily.

Is LVM stable, or can it cause more problems than separate raids on a array.

Ken





Second, convenience.  Assume you have a 6 disk raid5 array.  If a disk
fails and you are using a partitioned md array, then all the partitions
on the disk will already be handled without using that disk.  No need to
manually fail any still active array members from other arrays.

Third, safety.  Again with the raid5 array.  If you use multiple arrays
on a single disk, and that disk fails, but it only failed on one array,
then you now need to manually fail that disk from the other arrays
before shutting down or hot swapping the disk.  Generally speaking,
that's not a big deal, but people do occasionally have fat finger
syndrome and this is a good opportunity for someone to accidentally fail
the wrong disk, and when you then go to remove the disk you create a two
disk failure instead of one and now you are in real trouble.

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org]On Behalf Of martin f krafft
Sent: 18 October 2006 2:43pm
To: linux-raid mailing list
Subject: Re: why partition arrays?


also sprach Doug Ledford <dledford@redhat.com> [2006.10.18.1526 +0200]:
> There are a couple reasons I can think.

Thanks for your elaborate response. If you don't mind, I shall link
to it from the FAQ.

I have one other question: do partitionable and traditional arrays
actually differ in format? Put differently: can I assemble
a traditional array as a partitionable one simply by specifying:

  mdadm --create ... /dev/md0 ...
  mdadm --stop /dev/md0
  mdadm --assemble --auto=part ... /dev/md0 ...

? Or do the superblocks actually differ?

Thanks,

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: madduck.bogus@madduck.net
 
the images rushed around his mind and tried
to find somewhere to settle down and make sense.
            -- douglas adams, "the hitchhiker's guide to the galaxy"

RE: why partition arrays?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2683 bytes --]

On Thu, 2006-10-19 at 12:25 +0100, Ken Walker wrote:
> So is LVM better for partitions on a large raid5, or any raid, than separate
> partitions on that array.

In some ways yes, although it introduces a certain amount of uncertainty
in tuning of block devices.

> I'm still in my learning curve :)
> 
> for example, if one has Linux running on a two disk mirror array, raid1, and
> the first disk is partitioned, say 5 partitions, with those partitions
> mirrored on the second disk, and each identical partition is then run as a
> mirror raid1.
> 
> What your saying is that, if a single partition fails, to remove the drive
> you have to fail all the array partitions on the drive your taking out, then
> rebuild the partitions and then add to the dirty raid the new partitions one
> at a time.

Yep.

> Will LVM remove all this, so if you have a mirror as a single raid
> partition, and use LVM to create the partitions on that mirror, if a disk
> goes down, can it be removed, replaced, and then just added to the single
> raid, with LVM having had no idea what was going on in the background and
> just plod along merrily.

Yep.  In addition, with LVM, if you added two new disks, also in a raid1
array, then you could add that to your current volume group as another
physical volume, and the LVM code would happily extend your volume to
span both RAID1 arrays and increase the size.  Since the md code can now
grow things, this isn't as impressive as it used to be, but it's
probably a little easier to handle the lvm stuff than the md growth
stuff if for no other reason than they have graphical LVM tools that you
can do this with.

> Is LVM stable, or can it cause more problems than separate raids on a array.

Current incarnations are very stable.  I mentioned earlier that it can
introduce some tuning issues.  If you are dealing with a raid device
directly, then it's relatively straight forward to set the stripe size,
chunk size, etc. according to the number of raid disks and then set the
elevator and possibly things like read ahead values to optimize the raid
array's performance for different needs.  When you introduce LVM on top
of raid, there is the possibility that there will be interactions
between the two that have a detrimental impact on performance (this may
not always be the case, and it may not be unfixable, I'm just saying
it's an additional layer you have to deal with).

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: why partition arrays?

[Bodo Thiesen]

Ken Walker <ken.walker@manchester.ac.uk> wrote:

> Is LVM stable, or can it cause more problems than separate raids on a array.

I don't know what you consider "stable". But let me explain what we have 
running here for now, maybe you will then have a better feeling on how 
stable LVM is:

After two hard disks crashed, and we loosed more ore less semi-important 
data, we considered to set up all disk space using RAID6.

The initial situation was: We had 7 hard discs with more or less different 
sizes. So just using the block devices for the raids was not optimal. We 
didn't want to waste TOO much space (we were going to "waste" 2/5 through 
RAID6 already, that is 40%, but another 50% by orienting on the smallest 
disc was out of the questions). So we decided to create four RAID6 arrays 
each using five block devices. There were the sizes ~100GB, ~75GB and 
2*~55GB. So we could fully utilize our new discs of ~300GB each, and still 
meaningful use the other disks - all smaller in size. But with this strange 
concept we immediately decided, that just using IBM disk labels (aka 
partitions) were not really meaningful, especially as after swapping discs 
(consider changing the chassis, just disconnecting all discs ... hmm, what 
was hdb and what was hdd? And hde? Hmmm ...), so we decided the following 
structure:

hda -> vg called raida -> creating LVs called raida1..raida4
hdb -> vg called raidb -> creating LVs called raidb1..raidb4

and so on

On some discs, not all of raidX1..raidX4 were actually created, e.g. on 
our 200GB disc, only raidX2..raidX4 were creates thus missing the 100GB 
component.

Now, raid*1 would make up md1, raid*2 would build md2 and so on.

Each of md1..md4 where then created another VG on, all put in one volume 
group (called 'space' ;), and from that VG we created our actual logical 
volumes like homes etc.

BTW: I left off the dm-crypt step, additionally the raid LVs are all 
encrypted.

That system is now stable at least since 14. Feb 2006.

So, now decide for your own, if you consider LVM stable - I would ;)

Regards, Bodo

Re: why partition arrays?

[Henrik Holst]

Bodo Thiesen wrote:
> Ken Walker <ken.walker@manchester.ac.uk> wrote:
> 
>> Is LVM stable, or can it cause more problems than separate raids on a array.

[description of street smart raid setup]

(The same function could probably be achieved with logical partitions
and ordinary software raid levels.)

> So, now decide for your own, if you consider LVM stable - I would ;)
> 
> Regards, Bodo

Have you lost any disc (i.e. "physical volumes") since February? Or lost
the meta-data?

I would not recommend anyone to use LVM if they are less than experts on
Linux systems. Setting up a LVM system is easy: administrating and
salvaging the same, was much more work. (I used it ~3 years ago)

/Henrik Holst

Re: why partition arrays?

[Bodo Thiesen]

Henrik Holst <henrik.holst@idgmail.se> wrote:

> Have you lost any disc (i.e. "physical volumes") since February?

In fact, we have. One disc failed, we removed it brought a new disc, set 
it up like the other ones and finally added it to the degraded RAID 
array. Now everything is fine again ;)

> Or lost the meta-data?

LVM meta-data or RAID meta-data?

LVM: In this case we may really get a problem. But on the other hand: How 
would you try to loose those meta-data?

RAID6: There we would be quite save, as we have all LVs of raid1 called 
raid?1 and so on. So if it wouldn't assemble anymore we could still fall 
back to just recreate it.

> I would not recommend anyone to use LVM if they are less than experts on
> Linux systems.

LVM is such easy to use, I'd recommend anyone to use LVM in favor of 
creating partitions etc. because I think, repartitioning a disc is MUCH 
MORE error prone, than just creating a new LV. And you can do stuff like 
enlarging an LV what you can't with partitions.

> Setting up a LVM system is easy: administrating and
> salvaging the same, was much more work. (I used it ~3 years ago)

Was that LVM on RAID? I'm talking about LVM on RAID.

Regards, Bodo

Re: why partition arrays?

[Nix]

On 21 Oct 2006, Bodo Thiesen yowled:
> was hdb and what was hdd? And hde? Hmmm ...), so we decided the following 
> structure:
> 
> hda -> vg called raida -> creating LVs called raida1..raida4
> hdb -> vg called raidb -> creating LVs called raidb1..raidb4

I'm interested: why two VGs? Why not have one VG covering all RAID arrays,
and then another one for any unRAIDed space (if any)?

-- 
`When we are born we have plenty of Hydrogen but as we age our
 Hydrogen pool becomes depleted.'

Re: why partition arrays?

[Mario 'BitKoenig' Holbe]

martin f krafft <madduck@madduck.net> wrote:
> Why would anyone want to create a partitionable array and put
> partitions in it, rather than creating separate arrays for each
> filesystem? Intuitively, this makes way more sense as then the
> partitions are independent of each other; one array can fail and the
> rest still works -- part of the reason why you partition in the

Intuitively, especially the independence is a really mixed blessing.
First: If a disk fails somehow, md usually makes sure there is no
further access to it which could probably worsen the situation, i.e.
freeze busses, controllers, etc. Yes, this is a bit softened with the
new read-error-correction code, but IMHO still valid - and gets IMHO
even more and more valid with cheaper and cheaper controllers.
With multiple raids over partitions the disk is still a candidate to be
accessed subsequently.
Second: Bigger independence does also mean bigger concurrency. RAID1 for
example tries to equalize reads and directs reads to the mirror with
it's heads "closest" to the read-position (how good the "close"
estimation will ever be). Since partitions are somehow connected
together (especially by the disk's heads), concurrent access to multiple
RAIDs could torpedize this optimization. Perhaps this got a bit better
in 2.6, I don't know, in 2.4 you can watch this very well.


regards
   Mario
-- 
But after a while I learned the trick of speaking fast. You don't have
to think any faster; just use twice as many words to say everything.
                                -- Paul Graham

Re: why partition arrays?

[Bill Davidsen]

Henrik Holst wrote:

>Bodo Thiesen wrote:
>  
>
>>Ken Walker <ken.walker@manchester.ac.uk> wrote:
>>
>>    
>>
>>>Is LVM stable, or can it cause more problems than separate raids on a array.
>>>      
>>>
>
>[description of street smart raid setup]
>
>(The same function could probably be achieved with logical partitions
>and ordinary software raid levels.)
>
>  
>
>>So, now decide for your own, if you consider LVM stable - I would ;)
>>
>>Regards, Bodo
>>    
>>
>
>Have you lost any disc (i.e. "physical volumes") since February? Or lost
>the meta-data?
>
>I would not recommend anyone to use LVM if they are less than experts on
>Linux systems. Setting up a LVM system is easy: administrating and
>salvaging the same, was much more work. (I used it ~3 years ago)
>
My read on LVM is that (a) it's one more thing for the admin to learn, 
(b) because it's seldom used the admin will be working from 
documentation if it has a problem, and (c) there is no bug-free 
software, therefore the use of LVM on top of RAID will be less reliable 
than a RAID-only solution. I can't quantify that, the net effect may be 
too small to measure. However, the cost and chance of a finger check 
from (a) and (b) are significant.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: why partition arrays?

[dean gaudet]

On Tue, 24 Oct 2006, Bill Davidsen wrote:

> My read on LVM is that (a) it's one more thing for the admin to learn, (b)
> because it's seldom used the admin will be working from documentation if it
> has a problem, and (c) there is no bug-free software, therefore the use of LVM
> on top of RAID will be less reliable than a RAID-only solution. I can't
> quantify that, the net effect may be too small to measure. However, the cost
> and chance of a finger check from (a) and (b) are significant.

this is essentially why i gave up on LVM as well.

add in the following tidbits:

- snapshots stopped working in 2.6.  may be fixed by now, but i gave up 
hope and this was the biggest feature i desired from LVM.

- it's way better for performance to have only one active filesystem on a 
group of spindles

- you can emulate pvmove with md superblockless raid1 sufficiently well 
for most purposes (although as we've discussed here it would be nice if md 
directly supported "proactive replacement")

and more i'm forgetting.

-dean