[linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[js]

Hi list,

I'm looking into making a home brewed nas setup. This is not a 
production, highly critical environment.

The result needs to be 1 continuous physical storage space.
Expandable without reformating, removing data, etc,..
Acceptable redundancy.
I don't mind 50% loss of data capacity because of raid 1.  Storage is 
(relatively) cheap.

I don't have this material to test.  So it's more like a theoretical setup.

I was thinking of creating sets of mdadm RAID 1 setups and unify these 
raid 1 devices with lvm2 into 1 big logical volume which is resizeable.
For example:
I have 4 identical 250 GB sata2 disks.

I use mdadm to create:

/dev/md0 raid1 consisting out of /dev/sda1 and /dev/sdb1.
/dev/md1 raid1 consisting out of /dev/sdc1 and /dev/sdd1.

Then I would like to:

create LVM physical volumes out of /dev/md0 and /dev/md1.
create a LVM volume group named storage consisting out of LVM physical 
device /dev/md0 and /dev/md1
create a LVM logical volume that covers the maximum size of the LVM 
volume group which should be around 500GB.
create a filesystem on this LVM logical volume such as ext3.

Now when I need more storage:

I add 2 250GB sata2 disks.
I create a new raid1 /dev/md2 consisting out of /dev/sde1 and /dev/sdf1
Create a LVM physical volume out of /dev/md2.
Add the /dev/md2 LVM physical volume to the LVM volume group.
expand the LVM logical volume so it becomes around 750GB in size.
resize the EXT3 filesystem.


Now my question to you is.  Is this completely bunkers?  Or is it 
reasonable?
Any tips, tricks, advice, comments welcome.

Thanks,

js

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Stuart D. Gathman]

On Sun, 22 Jul 2007, js wrote:

> I'm looking into making a home brewed nas setup. This is not a 
> production, highly critical environment.
> 
> The result needs to be 1 continuous physical storage space.
> Expandable without reformating, removing data, etc,..
> Acceptable redundancy.
> I don't mind 50% loss of data capacity because of raid 1.  Storage is 
> (relatively) cheap.
> 
> I don't have this material to test.  So it's more like a theoretical setup.
...
> Now my question to you is.  Is this completely bunkers?  Or is it 
> reasonable?

This is how all our systems are setup.  It is the only reasonable way
to do raid1.  Hardware RAID is nasty when you need to add/replace disks
and the new models are bigger.

> Any tips, tricks, advice, comments welcome.

The main drawback to md raid1 is that it always resynchronizes
the *entire* partition when a disk goes offline temporarily.  I also
like to split a drive into several pieces, say split a pair of 240G into two
120G md partitions.  This gives me some flexibility to migrate partitions
between physical drives (for performance tuning, or swapping physical
drives around).  To migrate, set the mirror "faulty" with mdadm,
"hot remove" it, then "hot add" the new partition.  If you don't like
leaving the md drive unmirrored while resyncing to its new location,
reserve 3 physical drives in each md device, but use only 2.  Then
you can a 3rd mirror to migrate, then set faulty and hot remove the
old after the new is finished synchronizing.

Of course, I *hate* having to manually manage the md partitions in Linux.  I
cut my teeth on AIX LVM - which has robust useful raid1 built it.
(The latest experimental LVM mirroring in Linux still doesn't cut it.)
AIX is probably too expensive for home brew, but I understand Open Solaris
has similar features (but haven't had time to try it).

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Luca Berra]

On Sun, Jul 22, 2007 at 02:57:33PM -0400, Stuart D. Gathman wrote:
>The main drawback to md raid1 is that it always resynchronizes
>the *entire* partition when a disk goes offline temporarily.  I also
False. md uses bitmaps to avoid this.

L.
-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Stuart D. Gathman]

On Mon, 23 Jul 2007, Luca Berra wrote:

> On Sun, Jul 22, 2007 at 02:57:33PM -0400, Stuart D. Gathman wrote:
> >The main drawback to md raid1 is that it always resynchronizes
> >the *entire* partition when a disk goes offline temporarily.  I also
> False. md uses bitmaps to avoid this.

I am using md on dozens of systems up through Centos-5.  If there are any
bitmaps, they don't work.  Or maybe they are a new feature not in kernel
2.6.18.  The only change I've noticed since RH7.3 is that sync speed is no
longer limited to 10K.  Or maybe I'm not looking at the right thing.

When replacing a disk, a full resync is required anyway - so no problem.
Where it is a problem is when there is an abnormal system shutdown,
so that md devices are not closed properly.  The system boots immediately,
but performance is slow until the mirror have been resynced.  This can
take hours.  And the system is vulnerable to single disk failure in the source
drive for those hours.

How would I see these bitmaps in action?

BTW, to upgrade a 7.2 system to Centos5, I plan to install on a disk
with matching partition sizes and replace boot drive.  Is there any
chance that md raid1 will sync from the old system overtop the new?
(Because the new wouild have lower sequence numbers?)  Or is there some
kind of UUID in the raid superblock to prevent this?  (Of course I plan
to change partition types away from RAID Auto just to be safe...)

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Luca Berra]

On Mon, Jul 23, 2007 at 12:43:17PM -0400, Stuart D. Gathman wrote:
>On Mon, 23 Jul 2007, Luca Berra wrote:
>
>> On Sun, Jul 22, 2007 at 02:57:33PM -0400, Stuart D. Gathman wrote:
>> >The main drawback to md raid1 is that it always resynchronizes
>> >the *entire* partition when a disk goes offline temporarily.  I also
>> False. md uses bitmaps to avoid this.
>
>I am using md on dozens of systems up through Centos-5.  If there are any
>bitmaps, they don't work.  Or maybe they are a new feature not in kernel
>2.6.18.  The only change I've noticed since RH7.3 is that sync speed is no
>longer limited to 10K.  Or maybe I'm not looking at the right thing.
man mdadm
Bitmaps are a feature of md since mdadm-2.0 (kernel 2.6.13 or 2.6.14)

...

>How would I see these bitmaps in action?

man mdadm
but basically
mdadm -G /dev/md?? -b internal

>BTW, to upgrade a 7.2 system to Centos5, I plan to install on a disk
>with matching partition sizes and replace boot drive.  Is there any
>chance that md raid1 will sync from the old system overtop the new?
>(Because the new wouild have lower sequence numbers?)  Or is there some
>kind of UUID in the raid superblock to prevent this?  (Of course I plan
>to change partition types away from RAID Auto just to be safe...)
unfortunately redhat insisted for a long time on using the in-kernel md
auto-detect, which iirc just ignores md uuid. i think this is still true
in redhat5.

changing partition type will prevent this
creating the new raid with a different minor will prevent this
or change the minor of the old raid.
... but we're getting offtopic.
if you have further questions post them on linux-raid.

L.


-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Stuart D. Gathman]

On Mon, 23 Jul 2007, Luca Berra wrote:

> On Sun, Jul 22, 2007 at 02:57:33PM -0400, Stuart D. Gathman wrote:
> >The main drawback to md raid1 is that it always resynchronizes
> >the *entire* partition when a disk goes offline temporarily.  I also
> False. md uses bitmaps to avoid this.

The LVM mirroring uses bitmaps.  Are you sure md raid does?  Since what
version?

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Mon, 23 Jul 2007, Luca Berra wrote:
> 
>> On Sun, Jul 22, 2007 at 02:57:33PM -0400, Stuart D. Gathman wrote:
>>> The main drawback to md raid1 is that it always resynchronizes
>>> the *entire* partition when a disk goes offline temporarily.  I also
>> False. md uses bitmaps to avoid this.
> 
> The LVM mirroring uses bitmaps.  Are you sure md raid does?  Since what
> version?
> 

drivers/md/md.c:

   - persistent bitmap code
     Copyright (C) 2003-2004, Paul Clements, SteelEye Technology, Inc.

Cheers,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFGpcKf6YSQoMYUY94RAq4JAKDJYwRqXm3ZsAqLgrhbup4/MEvNNACfauDJ
GNLb7n3pHU55pe+aKXenLAw=
=oj8i
-----END PGP SIGNATURE-----

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Stuart D. Gathman]

On Mon, 23 Jul 2007, Luca Berra wrote:

> On Sun, Jul 22, 2007 at 02:57:33PM -0400, Stuart D. Gathman wrote:
> >The main drawback to md raid1 is that it always resynchronizes
> >the *entire* partition when a disk goes offline temporarily.  I also

> False. md uses bitmaps to avoid this.

Ok, I just tested this with Centos-5.0, kernel-2.6.18-8.1.8.el5.
I shutdown with md1 50% resynced, and when it boots, it starts over
at the beginning.  Maybe we are talking about different things, but
it is a major annoyance that it restarts the sync from the beginning
if it misses just one update (e.g. unexpected power failure or reboot).

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Luca Berra]

On Mon, Aug 20, 2007 at 08:33:26PM -0400, Stuart D. Gathman wrote:
>On Mon, 23 Jul 2007, Luca Berra wrote:
>
>> On Sun, Jul 22, 2007 at 02:57:33PM -0400, Stuart D. Gathman wrote:
>> >The main drawback to md raid1 is that it always resynchronizes
>> >the *entire* partition when a disk goes offline temporarily.  I also
>
>> False. md uses bitmaps to avoid this.
>
>Ok, I just tested this with Centos-5.0, kernel-2.6.18-8.1.8.el5.
>I shutdown with md1 50% resynced, and when it boots, it starts over
>at the beginning.  Maybe we are talking about different things, but
>it is a major annoyance that it restarts the sync from the beginning
>if it misses just one update (e.g. unexpected power failure or reboot).
>
did you add the bitmap to your md array?

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: [linux-lvm] Flexible storga: LVM setup on top of mdadm sets. Good idea or not?

[Stuart D. Gathman]

On Tue, 21 Aug 2007, Luca Berra wrote:

> >> False. md uses bitmaps to avoid this.
> >
> >Ok, I just tested this with Centos-5.0, kernel-2.6.18-8.1.8.el5.
> >I shutdown with md1 50% resynced, and when it boots, it starts over
> >at the beginning.  Maybe we are talking about different things, but
> >it is a major annoyance that it restarts the sync from the beginning
> >if it misses just one update (e.g. unexpected power failure or reboot).
> >
> did you add the bitmap to your md array?

I installed Centos5 from the CDs on raid1 devices.  I do see a bitmap
option in the map page for mdadm.  How do I check whether Centos5 install
neglected to activate it?  Is there any way to add it after the install?
(LVM is on md1 and root,swap are on LVM.)  If not, what would be the
install procedure to actually use a bitmap?  Note: special restrictions
on the boot partition are acceptable.  Special restrictions on the root
partition/LV are not.

# mdadm --examine /dev/hde3
/dev/hde3:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 14c44605:1e3381da:9833dc4f:b4bdc3c0
  Creation Time : Mon Aug 20 15:13:47 2007
     Raid Level : raid1
    Device Size : 78059712 (74.44 GiB 79.93 GB)
     Array Size : 78059712 (74.44 GiB 79.93 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1

    Update Time : Tue Aug 21 10:06:36 2007
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : bb50a9f9 - correct
         Events : 0.8


      Number   Major   Minor   RaidDevice State
this     0      33        3        0      active sync   /dev/hde3

   0     0      33        3        0      active sync   /dev/hde3
   1     1      34        3        1      active sync   /dev/hdg3

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.