New features?

New features?

[John Rowe]

All this discussion has led me to wonder if we users of linux RAID have
a clear consensus of what our priorities are, ie what are the things we
really want to see soon as opposed to the many things that would be nice
but not worth delaying the important things for. FWIW, here are mine, in
order although the first two are roughly equal priority.

1 "Warm swap" - replacing drives without taking down the array but maybe
having to type in a few commands. Presumably a sata or sata/raid
interface issue. (True hot swap is nice but not worth delaying warm-
swap.)

2 Adding new disks to arrays. Allows incremental upgrades and to take
advantage of the hard disk equivalent of Moore's law.

3. RAID level conversion (1 to 5, 5 to 6, with single-disk to RAID 1 a
lower priority).

4. Uneven disk sizes, eg adding a 400GB disk to a 2x200GB mirror to
create a 400GB mirror. Together with 2 and 3, allows me to continuously
expand a disk array.

(Not knowing the code, I wonder if 2, 3 and 4 could be accomplished by
allowing an "external" RAID device to have several internal devices and
with changes accomplished the old one to shrink and the new one to grow
until the old one no longer exists.)

Thanks for listening.

John

Re: New features?

[Neil Brown]

On Tuesday October 31, rowe@excc.ex.ac.uk wrote:
> All this discussion has led me to wonder if we users of linux RAID have
> a clear consensus of what our priorities are, ie what are the things we
> really want to see soon as opposed to the many things that would be nice
> but not worth delaying the important things for. FWIW, here are mine, in
> order although the first two are roughly equal priority.
> 
> 1 "Warm swap" - replacing drives without taking down the array but maybe
> having to type in a few commands. Presumably a sata or sata/raid
> interface issue. (True hot swap is nice but not worth delaying warm-
> swap.)

I believe that 2.6.18 has SATA hot-swap, so this should be available
know ... providing you can find out what commands to use.

> 
> 2 Adding new disks to arrays. Allows incremental upgrades and to take
> advantage of the hard disk equivalent of Moore's law.

Works for raid5 and linear.  Raid6 one day.

> 
> 3. RAID level conversion (1 to 5, 5 to 6, with single-disk to RAID 1 a
> lower priority).

A single disk is large than a RAID1 built from it, so this is
non-trivial.  What exactly do you want to do there.

> 
> 4. Uneven disk sizes, eg adding a 400GB disk to a 2x200GB mirror to
> create a 400GB mirror. Together with 2 and 3, allows me to continuously
> expand a disk array.

So you have a RAID1 (md) from sda and sdb, both 200GB, and you now have a
sdc which is 400GB.
So
   mdadm /dev/md0 -a /dev/sdc
   mdadm /dev/md0 -f /dev/sda
   mdadm /dev/md0 -r /dev/sda
   # wait for recovery
   mdadm /dev/md0 -f /dev/sdb
   mdadm /dev/md0 -r /dev/sdb
   mdadm -C /dev/md1 -l linear -n 2 /dev/sda /dev/sdb
   mdadm /dev/md0 -a /dev/md1
   # wait for recovery
   mdadm --grow /dev/md0 --size=max

You do run with a degraded array for a while, but you can do it
entirely online.
It might be possible to decrease the time when the array is degraded,
but it is too late at night to think about that.

NeilBrown

Re: New features?

[John Rowe]

Thanks for this Neil, good to know that most of what I would like is
already available. I think your reply highlights what I almost put in
there as my first priority: documentation, specifically a HOWTO.

> I believe that 2.6.18 has SATA hot-swap, so this should be available
> know ... providing you can find out what commands to use.

Exactly!

> > 2 Adding new disks to arrays. Allows incremental upgrades and to take
> > advantage of the hard disk equivalent of Moore's law.
> 
> Works for raid5 and linear.  Raid6 one day.

Am I misinterpreting the mdadm 2.5 man pages when it says:

        Grow (or shrink) an array, or otherwise reshape it in some way.
        Currently supported growth options including changing the active
        size of component devices in RAID level 1/4/5/6 and changing the
        number of active devices in RAID1.

> > 3. RAID level conversion (1 to 5, 5 to 6, with single-disk to RAID 1 a
> > lower priority).
> 
> A single disk is large than a RAID1 built from it, so this is
> non-trivial.  What exactly do you want to do there.

Single to disk is less important, but adding a third disk to a RAID1
pair to make a RAID5 would be nice as would be adding one or more disks
to a RAID5 to make a RAID6.

John

Re: New features?

[Mike Hardy]

Neil Brown wrote:
> On Tuesday October 31, rowe@excc.ex.ac.uk wrote:

>> 1 "Warm swap" - replacing drives without taking down the array but maybe
>> having to type in a few commands. Presumably a sata or sata/raid
>> interface issue. (True hot swap is nice but not worth delaying warm-
>> swap.)
> 
> I believe that 2.6.18 has SATA hot-swap, so this should be available
> know ... providing you can find out what commands to use.

I forgot 2.6.18 has SATA hot-swap, has anyone tested that?

FWIW, SCSI (or SAS now, using SCSI or SATA drives) has full hot-swap
with completely online drive exchanges. I have done this on recent
kernels in production and it works.

> 
>> 2 Adding new disks to arrays. Allows incremental upgrades and to take
>> advantage of the hard disk equivalent of Moore's law.
> 
> Works for raid5 and linear.  Raid6 one day.

Also works for raid1!


>> 4. Uneven disk sizes, eg adding a 400GB disk to a 2x200GB mirror to
>> create a 400GB mirror. Together with 2 and 3, allows me to continuously
>> expand a disk array.
> 
> So you have a RAID1 (md) from sda and sdb, both 200GB, and you now have a
> sdc which is 400GB.
> So
>    mdadm /dev/md0 -a /dev/sdc
>    mdadm /dev/md0 -f /dev/sda
>    mdadm /dev/md0 -r /dev/sda
>    # wait for recovery

Could be:

    mdadm /dev/md0 -a /dev/sdc
    mdadm --grow /dev/md0 --raid-devices=3 # 3-disk mirror
    # wait for recovery
    # don't forget grub-install /dev/sda (or similar)!
    mdadm /dev/md0 -f /dev/sda
    mdadm /dev/md0 -r /dev/sda
    mdadm --grow /dev/md0 --raid-devices=2 # 2-disk again

    # Run a 'smartctl -d ata -t long /dev/sdb' before next line...

>    mdadm /dev/md0 -f /dev/sdb
>    mdadm /dev/md0 -r /dev/sdb
>    mdadm -C /dev/md1 -l linear -n 2 /dev/sda /dev/sdb
>    mdadm /dev/md0 -a /dev/md1
>    # wait for recovery
>    mdadm --grow /dev/md0 --size=max
> 
> You do run with a degraded array for a while, but you can do it
> entirely online.
> It might be possible to decrease the time when the array is degraded,
> but it is too late at night to think about that.

All I did was decrease the degradation time, but hey it could help. And
don't forget the long SMART test before running degraded for real. Could
save you some pain.

-Mike

Re: New features?

[Frido Ferdinand]

Hi,

On Tue, Oct 31, 2006 at 08:50:19AM -0800, Mike Hardy wrote:
> >> 1 "Warm swap" - replacing drives without taking down the array but maybe
> >> having to type in a few commands. Presumably a sata or sata/raid
> >> interface issue. (True hot swap is nice but not worth delaying warm-
> >> swap.)
> > 
> > I believe that 2.6.18 has SATA hot-swap, so this should be available
> > know ... providing you can find out what commands to use.
> 
> I forgot 2.6.18 has SATA hot-swap, has anyone tested that?

Yeah, I've tracked the sata EH patches and now 2.6.18 for a while and
hotswap works. However if you pull the disk on a raidset the disk is set
as faulty and the device (/dev/sda for example) dissapears. If you
replug it, the device does not regain it's original devicename but but
will use the latest free 'slot' available (in a four disk layout that's
/dev/sde). Also trying to --remove the disk doesn't work since the
devicefile is gone. So be sure to --remove disks _before_ you pull it.

Anyone know if there's work being done to fix this issue, does 
this also happen on scsi ?

Met vriendelijke groet,

		-- Frido Ferdinand

mdadm --assemble weirdness?

[Patrik Jonsson]

Hi,

I just got a new SATA controller and moved some of my drives around, so
the device numbers of course got all messed up. I expected mdadm to be
able to handle that, which it did when I moved one device. However, when
I moved the next 3, things got weird. Here's the output from mdadm
--assemble --scan -v:

... <removed unmatching devices>
mdadm: /dev/sdj1 is identified as a member of /dev/md5, slot 3.
mdadm: /dev/sdi1 is identified as a member of /dev/md5, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md5, slot 1.
mdadm: /dev/sdg1 is identified as a member of /dev/md5, slot 0.
mdadm: /dev/sdf1 is identified as a member of /dev/md5, slot 6.
mdadm: /dev/sde1 is identified as a member of /dev/md5, slot 7.
mdadm: /dev/sdd1 is identified as a member of /dev/md5, slot 9.
mdadm: /dev/sdc1 is identified as a member of /dev/md5, slot 8.
mdadm: /dev/sdb1 is identified as a member of /dev/md5, slot 5.
mdadm: /dev/sda1 is identified as a member of /dev/md5, slot 4.
mdadm: added /dev/sdh1 to /dev/md5 as 1
mdadm: added /dev/sdi1 to /dev/md5 as 2
mdadm: added /dev/sdj1 to /dev/md5 as 3
mdadm: added /dev/sda1 to /dev/md5 as 4
mdadm: added /dev/sdb1 to /dev/md5 as 5
mdadm: added /dev/sdf1 to /dev/md5 as 6
mdadm: added /dev/sde1 to /dev/md5 as 7
mdadm: added /dev/sdc1 to /dev/md5 as 8
mdadm: added /dev/sdd1 to /dev/md5 as 9
mdadm: added /dev/sdg1 to /dev/md5 as 0
mdadm: /dev/md5 assembled from 7 drives - not enough to start the array.

The added drives are the correct ones, but how it adds 10 drives but
then thinks that it's only assembled 7 mystifies me. Is this a bug?

Regards,

/Patrik

Re: mdadm --assemble weirdness?

[Neil Brown]

On Tuesday November 28, patrik@ucolick.org wrote:
> Hi,
> 
> I just got a new SATA controller and moved some of my drives around, so
> the device numbers of course got all messed up. I expected mdadm to be
> able to handle that, which it did when I moved one device. However, when
> I moved the next 3, things got weird. Here's the output from mdadm
> --assemble --scan -v:
> 
> ... <removed unmatching devices>
--snip--
> mdadm: /dev/md5 assembled from 7 drives - not enough to start the array.
> 
> The added drives are the correct ones, but how it adds 10 drives but
> then thinks that it's only assembled 7 mystifies me. Is this a bug?

Weird...

What version of mdadm?

Can you show us "mdadm --examine" of a couple of devices?

Any kernel logs while this is happening?

NeilBrown

Re: mdadm --assemble weirdness?

[Patrik Jonsson]

Neil Brown wrote:
>
> Weird...
>
> What version of mdadm?
>
> Can you show us "mdadm --examine" of a couple of devices?
>
> Any kernel logs while this is happening?
>   
I just got the LATEST off of your website, whatever the number was. I
got the array running, i noticed that in the dmesg from booting it said
something about "kicking non-fresh drive sdxyz" so I did a --assemble
--scan --force and then re-added the final drive which got the array
back to syncing. I would have prefered not to resync but whatever. The
strange thing is how they came to be non-fresh, these were the drives
that had changed controller. They were always there and I never did
anything to the array apart from trying to assemble it.

If I try to assemble an array and it can only find part of the devices,
do those devices that are found have their superblocks updated before
the assemble is rejected? If that's the case and I try again, the
missing drives will be "non-fresh" even though nothing was ever done.
That's the only cause I can imagine...

cheers,

/Patrik

Re: mdadm --assemble weirdness?

[Patrik Jonsson]

[-- Attachment #1: Type: text/plain, Size: 3148 bytes --]

Patrik Jonsson wrote
> I just got the LATEST off of your website, whatever the number was. I
> got the array running, i noticed that in the dmesg from booting it said
> something about "kicking non-fresh drive sdxyz" so I did a --assemble
> --scan --force and then re-added the final drive which got the array
> back to syncing. I would have prefered not to resync but whatever. The
> strange thing is how they came to be non-fresh, these were the drives
> that had changed controller. They were always there and I never did
> anything to the array apart from trying to assemble it.

Ouch! -- resync failed because of a read error on another drive... If
I'm correct, the data should still be safe because the resyncing drive
should have been rewritten with identical data, right? So it should
still be possible to recreate the array with --assume-clean and not lose
data? Now I'm getting sufficiently close to the edge that I want to
reconfirm the steps, though.

To successfully do a recreate I need the correct device order, right?
Currently, mdadm says:

/dev/md5:
        Version : 00.90.03
  Creation Time : Thu Jun 16 18:44:56 2005
     Raid Level : raid5
     Array Size : 2193136704 (2091.54 GiB 2245.77 GB)
    Device Size : 243681856 (232.39 GiB 249.53 GB)
   Raid Devices : 10
  Total Devices : 10
Preferred Minor : 5
    Persistence : Superblock is persistent

    Update Time : Tue Nov 28 18:24:47 2006
          State : clean, degraded
 Active Devices : 8
Working Devices : 9
 Failed Devices : 1
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : b438b4a3:de389878:2cbbe06c:ebaab31f
         Events : 0.4455550

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       1       8      129        1      active sync   /dev/sdi1
       2       8      113        2      active sync   /dev/sdh1
       3       0        0        3      removed
       4       8        1        4      active sync   /dev/sda1
       5       0        0        5      removed
       6       8       81        6      active sync   /dev/sdf1
       7       8       65        7      active sync   /dev/sde1
       8       8       33        8      active sync   /dev/sdc1
       9       8       49        9      active sync   /dev/sdd1

      10       8      145        -      spare   /dev/sdj1
      11       8       17        -      faulty spare   /dev/sdb1

I'm pretty sure that sdj1 is supposed to go as raiddevice 3 and sdb1 as
5. The correct way to proceed would then be:

mdadm -S /dev/md5   to stop the array
mdadm --create /dev/md5 --assume-clean -l 5 -n 10 /dev/sdg1 /dev/sdi1
/dev/sdh1 /dev/sdj1 /dev/sda1 /dev/sdb1 /dev/sdf1 /dev/sde1 /dev/sdc1
/dev/sdd1  to recreate

mdadm -o /dev/md5   to set readonly
then mount read-only and see if it looks like a file system. If it does
the order was correct if not the order was incorrect.
If order is correct then proceed to do a raid5 "repair" and things
should be safe.

Does this seem reasonable?

Thanks a bunch,

/Patrik


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

--no-degraded in mdadm.conf?

[Patrik Jonsson]

[-- Attachment #1: Type: text/plain, Size: 126 bytes --]

Hi,
I'm wondering if there's a corresponding 'no-degraded' flag that can be
put in the mdadm.conf?

cheers,

/Patrik


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: New features?

[Bill Davidsen]

John Rowe wrote:

>All this discussion has led me to wonder if we users of linux RAID have
>a clear consensus of what our priorities are, ie what are the things we
>really want to see soon as opposed to the many things that would be nice
>but not worth delaying the important things for. FWIW, here are mine, in
>order although the first two are roughly equal priority.
>
>1 "Warm swap" - replacing drives without taking down the array but maybe
>having to type in a few commands. Presumably a sata or sata/raid
>interface issue. (True hot swap is nice but not worth delaying warm-
>swap.)
>  
>
That seems to work now. It does assume that you have hardware hot swap 
capability.

>2 Adding new disks to arrays. Allows incremental upgrades and to take
>advantage of the hard disk equivalent of Moore's law.
>  
>
Also seems to work.

>3. RAID level conversion (1 to 5, 5 to 6, with single-disk to RAID 1 a
>lower priority).
>  
>
Single to RAID-N is possible, but involves a good bit of magic with 
leaving room for superblocks, etc.

>4. Uneven disk sizes, eg adding a 400GB disk to a 2x200GB mirror to
>create a 400GB mirror. Together with 2 and 3, allows me to continuously
>expand a disk array.
>  
>
???

-- 

bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

New features?

[Frido Ferdinand]

Hi,

On Tue, Oct 31, 2006 at 08:50:19AM -0800, Mike Hardy wrote:
> >> 1 "Warm swap" - replacing drives without taking down the array but maybe
> >> having to type in a few commands. Presumably a sata or sata/raid
> >> interface issue. (True hot swap is nice but not worth delaying warm-
> >> swap.)
> > 
> > I believe that 2.6.18 has SATA hot-swap, so this should be available
> > know ... providing you can find out what commands to use.
> 
> I forgot 2.6.18 has SATA hot-swap, has anyone tested that?

Yeah, I've tracked the sata EH patches and now 2.6.18 for a while and
hotswap works. However if you pull the disk on a raidset the disk is set
as faulty and the device (/dev/sda for example) dissapears. If you
replug it, the device does not regain it's original devicename but but
will use the latest free 'slot' available (in a four disk layout that's
/dev/sde). Also trying to --remove the disk doesn't work since the
devicefile is gone. So be sure to --remove disks _before_ you pull it.

Anyone know if there's work being done to fix this issue, does 
this also happen on scsi ?

Regards,

        -- Frido

Re: New features?

[Neil Brown]

On Tuesday October 31, frido@zolder.org wrote:
> Hi,
> 
> On Tue, Oct 31, 2006 at 08:50:19AM -0800, Mike Hardy wrote:
> > >> 1 "Warm swap" - replacing drives without taking down the array but maybe
> > >> having to type in a few commands. Presumably a sata or sata/raid
> > >> interface issue. (True hot swap is nice but not worth delaying warm-
> > >> swap.)
> > > 
> > > I believe that 2.6.18 has SATA hot-swap, so this should be available
> > > know ... providing you can find out what commands to use.
> > 
> > I forgot 2.6.18 has SATA hot-swap, has anyone tested that?
> 
> Yeah, I've tracked the sata EH patches and now 2.6.18 for a while and
> hotswap works. However if you pull the disk on a raidset the disk is set
> as faulty and the device (/dev/sda for example) dissapears. If you
> replug it, the device does not regain it's original devicename but but
> will use the latest free 'slot' available (in a four disk layout that's
> /dev/sde). Also trying to --remove the disk doesn't work since the
> devicefile is gone. So be sure to --remove disks _before_ you pull it.
> 
> Anyone know if there's work being done to fix this issue, does 
> this also happen on scsi ?

The 'correct' way to fix "--remove doesn't work after" is to get udev
to do the "mdadm --remove" before deleting the entry from /dev.

mdadm could probably be changed to be able to remove the device
anyway.  The only difficulty is: how do you tell it which device to
remove", given that there is no name in /dev to use.
Suggestions?

Having the device come back with a different name shouldn't really be
a problem.  You can still find it an re-add it.

NeilBrown

Re: New features?

[Gabor Gombas]

On Fri, Nov 03, 2006 at 02:39:31PM +1100, Neil Brown wrote:

> mdadm could probably be changed to be able to remove the device
> anyway.  The only difficulty is: how do you tell it which device to
> remove", given that there is no name in /dev to use.
> Suggestions?

Major:minor? If /sys/block still holds an entry for the removed disk,
then the user can figure it out from the name. Or mdadm could just
accept a path under /sys/block instead of a device node.

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

Re: New features?

[Neil Brown]

On Friday November 3, gombasg@sztaki.hu wrote:
> On Fri, Nov 03, 2006 at 02:39:31PM +1100, Neil Brown wrote:
> 
> > mdadm could probably be changed to be able to remove the device
> > anyway.  The only difficulty is: how do you tell it which device to
> > remove", given that there is no name in /dev to use.
> > Suggestions?
> 
> Major:minor? If /sys/block still holds an entry for the removed disk,
> then the user can figure it out from the name. Or mdadm could just
> accept a path under /sys/block instead of a device node.

I like the /sys/block idea.  So if given a directory we look for a
'dev' file and read major:minor from that.
I guess I could also just allow
  mdadm /dev/mdX --remove failed

and all failed devices get removed.

Thanks for the suggestion.
NeilBrown