no good deed goes unpunished...

no good deed goes unpunished...

[Dave Stevens]

Hello list, Phil,

I read your advice in the text below and it seems very reasonable. I  
want to do a test to see if the data is readable and this is what I  
did with the live distro:

mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2

and then mdadm said:

mdadm: can not open device /dev/sdb2: device or resource busy
mdadm: /dev/sdb2 has no superblock - assembly aborted

I don't know if these errors are related - i.e., if there is no  
superblock is that the busy resource in the first message? And if the  
superblock is absent can I put it back or somehow recreate it?

Dave






From: Phil Turmel <philip@turmel.org>
To: Dave Stevens <geek@uniserve.com>, linux-raid@vger.kernel.org
Subject: Re: two raid issues
Date: Sun, 08 Mar 2015 09:52:04 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101  
Thunderbird/31.5.0

Good morning Dave,

On 03/07/2015 05:42 PM, Dave Stevens wrote:
> Hello the raid list,
>
> I have inherited a server set up by people who are no longer around. It
> worked fine until recently and then after a routine update refused to
> boot. I've got the machine in my office and have been examining the
> problem, or rather problems, I think there are two.

Three, at least.

> First, the bootable partition on /dev/sda1 won't successfully boot to a
> xen kernel, kernel version is 2.6.18.something-xen. The intent is to
> boot to a raid-10 array of four 750GB drives, each partitioned into a
> small and a large partition as detailed below.
>
> Boot proceeds normally according to on-screen messages until this:
>
> md: md0: raid array is not clean  -- starting background reconstruction
> raid10: not enough operational mirrors for md0
> md: pers -> () failed

Yup.  Degraded to the point of not running.

> Immediately after these messages is another stating that an attempt has
> been made to kill init, kernel panic and reboot.

No way to pivot to your root filesystem, so your initramfs gives up.
Depending on the distro, it may be possible to pass a kernel command
line option to drop into a repair shell at that point.

> I've tried to give these messages verbatim but have no way (I think) to
> reproduce them other than manually.

Repair shell, if available.

> So at first I looked around, read through the wiki and found advice to
> NOT write anything to the array, which seems reasonable. I looked at a
> live microknoppix distro called runtime live that gave me a command
> shell to run the examine command with the output below:

LiveCD boot is good, and the following report is very detailed, thanks:

> /dev/sda2:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>   Creation Time : Sun Nov 29 15:33:50 2009
>      Raid Level : raid10
>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>    Raid Devices : 4
>   Total Devices : 3
> Preferred Minor : 0
>
>     Update Time : Wed Feb 18 19:28:22 2015
>           State : active
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 2
>   Spare Devices : 0
>        Checksum : 7c76593b - correct
>          Events : 32945477

This is important:  ^^^^^^^^

>          Layout : near=2
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     0       8        2        0      active sync   /dev/sda2
>
>    0     0       8        2        0      active sync   /dev/sda2
>    1     1       0        0        1      active sync
>    2     2       8       34        2      active sync   /dev/sdc2
>    3     3       0        0        3      faulty removed
> /dev/sdb2:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>   Creation Time : Sun Nov 29 15:33:50 2009
>      Raid Level : raid10
>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>    Raid Devices : 4
>   Total Devices : 3
> Preferred Minor : 0
>
>     Update Time : Sat Nov 22 13:18:12 2014
>           State : clean
>  Active Devices : 3
> Working Devices : 3
>  Failed Devices : 1
>   Spare Devices : 0
>        Checksum : 7d850ca4 - correct
>          Events : 32945477

With this:          ^^^^^^^^

>          Layout : near=2
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     1       8       18        1      active sync   /dev/sdb2
>
>    0     0       8        2        0      active sync   /dev/sda2
>    1     1       8       18        1      active sync   /dev/sdb2
>    2     2       8       34        2      active sync   /dev/sdc2
>    3     3       0        0        3      faulty removed
> /dev/sdc2:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>   Creation Time : Sun Nov 29 15:33:50 2009
>      Raid Level : raid10
>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>    Raid Devices : 4
>   Total Devices : 3
> Preferred Minor : 0
>
>     Update Time : Wed Feb 18 19:30:13 2015
>           State : active
>  Active Devices : 1
> Working Devices : 1
>  Failed Devices : 2
>   Spare Devices : 0
>        Checksum : 7c7659de - correct
>          Events : 32945479

And this:           ^^^^^^^^

>          Layout : near=2
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     2       8       34        2      active sync   /dev/sdc2
>
>    0     0       0        0        0      removed
>    1     1       0        0        1      faulty removed
>    2     2       8       34        2      active sync   /dev/sdc2
>    3     3       0        0        3      faulty removed
> /dev/sdd2:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>   Creation Time : Sun Nov 29 15:33:50 2009
>      Raid Level : raid10
>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>    Raid Devices : 4
>   Total Devices : 5
> Preferred Minor : 0
>
>     Update Time : Wed Sep  4 07:51:50 2013
>           State : active
>  Active Devices : 4
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 1
>        Checksum : 77c1a3a7 - correct
>          Events : 53

Whoa!              ^^^^

>          Layout : near=2
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     3       8       50        3      active sync   /dev/sdd2
>
>    0     0       8        2        0      active sync   /dev/sda2
>    1     1       8       18        1      active sync   /dev/sdb2
>    2     2       8       34        2      active sync   /dev/sdc2
>    3     3       8       50        3      active sync   /dev/sdd2
>    4     4       8       66        4      spare   /dev/sde2
> /dev/sde2:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>   Creation Time : Sun Nov 29 15:33:50 2009
>      Raid Level : raid10
>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>    Raid Devices : 4
>   Total Devices : 5
> Preferred Minor : 0
>
>     Update Time : Sun Sep  8 13:25:42 2013
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>   Spare Devices : 0
>        Checksum : 77c775be - correct
>          Events : 7934

And Whoa again!    ^^^^^^

>          Layout : near=2
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     3       8       66        3      active sync   /dev/sde2
>
>    0     0       8        2        0      active sync   /dev/sda2
>    1     1       8       18        1      active sync   /dev/sdb2
>    2     2       8       34        2      active sync   /dev/sdc2
>    3     3       8       66        3      active sync   /dev/sde2
>
> This makes sense to me as far as it goes but I don't see what to do
> next. As I understand it the four partitions from sda2 to sdd2 would
> form the array with sde as hot spare. It has been my assumption that if
> a drive failed that sde would sync and take over. I don't know if this
> is in fact the case and don't see a path forward. Of course the backups
> are inadequate.

Based on the events and update timestamps, sdd died sometime around Wed
Sep 4 07:51:50 2013, at which point sde stepped in.  It too failed
shortly after ~ Sun Sep 8 13:25:42 2013.  You then ran degraded for over
a year until sdb also failed ~ Sat Nov 22 13:18:12 2014.  You were then
running doubly-degraded (luckily on non-adjacent members) until this Feb
14 when sda was booted out.  Leaving only one running drive.

{ I wouldn't keep such people around, either. }

Your best bet is to force assembly of the last two working drives to get
the system running, then take an immediate backup of all critical files.
  Do the forced assembly with the livecd, then do a clean shutdown.  You
should then be able to boot the original OS and take your backup.

Then you need to completely rebuild your system with proper log
monitoring, array monitoring, and verification of your drives.

Phil


-- 
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey

Re: no good deed goes unpunished...

[Roger Heflin]

you should probably do cat /proc/mdstat

It is likely that the livecd may have already tried to assemble it,
and that the busy is because it is already in use.

If it is already assembled and you want to redo it first you will need
to stop the assembled array to be able to redo it.

On Thu, Mar 26, 2015 at 6:52 PM, Dave Stevens <geek@uniserve.com> wrote:
> Hello list, Phil,
>
> I read your advice in the text below and it seems very reasonable. I want to
> do a test to see if the data is readable and this is what I did with the
> live distro:
>
> mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>
> and then mdadm said:
>
> mdadm: can not open device /dev/sdb2: device or resource busy
> mdadm: /dev/sdb2 has no superblock - assembly aborted
>
> I don't know if these errors are related - i.e., if there is no superblock
> is that the busy resource in the first message? And if the superblock is
> absent can I put it back or somehow recreate it?
>
> Dave
>
>
>
>
>
>
> From: Phil Turmel <philip@turmel.org>
> To: Dave Stevens <geek@uniserve.com>, linux-raid@vger.kernel.org
> Subject: Re: two raid issues
> Date: Sun, 08 Mar 2015 09:52:04 -0400
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101
> Thunderbird/31.5.0
>
> Good morning Dave,
>
> On 03/07/2015 05:42 PM, Dave Stevens wrote:
>>
>> Hello the raid list,
>>
>> I have inherited a server set up by people who are no longer around. It
>> worked fine until recently and then after a routine update refused to
>> boot. I've got the machine in my office and have been examining the
>> problem, or rather problems, I think there are two.
>
>
> Three, at least.
>
>> First, the bootable partition on /dev/sda1 won't successfully boot to a
>> xen kernel, kernel version is 2.6.18.something-xen. The intent is to
>> boot to a raid-10 array of four 750GB drives, each partitioned into a
>> small and a large partition as detailed below.
>>
>> Boot proceeds normally according to on-screen messages until this:
>>
>> md: md0: raid array is not clean  -- starting background reconstruction
>> raid10: not enough operational mirrors for md0
>> md: pers -> () failed
>
>
> Yup.  Degraded to the point of not running.
>
>> Immediately after these messages is another stating that an attempt has
>> been made to kill init, kernel panic and reboot.
>
>
> No way to pivot to your root filesystem, so your initramfs gives up.
> Depending on the distro, it may be possible to pass a kernel command
> line option to drop into a repair shell at that point.
>
>> I've tried to give these messages verbatim but have no way (I think) to
>> reproduce them other than manually.
>
>
> Repair shell, if available.
>
>> So at first I looked around, read through the wiki and found advice to
>> NOT write anything to the array, which seems reasonable. I looked at a
>> live microknoppix distro called runtime live that gave me a command
>> shell to run the examine command with the output below:
>
>
> LiveCD boot is good, and the following report is very detailed, thanks:
>
>> /dev/sda2:
>>           Magic : a92b4efc
>>         Version : 0.90.00
>>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>>   Creation Time : Sun Nov 29 15:33:50 2009
>>      Raid Level : raid10
>>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>>    Raid Devices : 4
>>   Total Devices : 3
>> Preferred Minor : 0
>>
>>     Update Time : Wed Feb 18 19:28:22 2015
>>           State : active
>>  Active Devices : 2
>> Working Devices : 2
>>  Failed Devices : 2
>>   Spare Devices : 0
>>        Checksum : 7c76593b - correct
>>          Events : 32945477
>
>
> This is important:  ^^^^^^^^
>
>>          Layout : near=2
>>      Chunk Size : 256K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     0       8        2        0      active sync   /dev/sda2
>>
>>    0     0       8        2        0      active sync   /dev/sda2
>>    1     1       0        0        1      active sync
>>    2     2       8       34        2      active sync   /dev/sdc2
>>    3     3       0        0        3      faulty removed
>> /dev/sdb2:
>>           Magic : a92b4efc
>>         Version : 0.90.00
>>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>>   Creation Time : Sun Nov 29 15:33:50 2009
>>      Raid Level : raid10
>>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>>    Raid Devices : 4
>>   Total Devices : 3
>> Preferred Minor : 0
>>
>>     Update Time : Sat Nov 22 13:18:12 2014
>>           State : clean
>>  Active Devices : 3
>> Working Devices : 3
>>  Failed Devices : 1
>>   Spare Devices : 0
>>        Checksum : 7d850ca4 - correct
>>          Events : 32945477
>
>
> With this:          ^^^^^^^^
>
>>          Layout : near=2
>>      Chunk Size : 256K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     1       8       18        1      active sync   /dev/sdb2
>>
>>    0     0       8        2        0      active sync   /dev/sda2
>>    1     1       8       18        1      active sync   /dev/sdb2
>>    2     2       8       34        2      active sync   /dev/sdc2
>>    3     3       0        0        3      faulty removed
>> /dev/sdc2:
>>           Magic : a92b4efc
>>         Version : 0.90.00
>>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>>   Creation Time : Sun Nov 29 15:33:50 2009
>>      Raid Level : raid10
>>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>>    Raid Devices : 4
>>   Total Devices : 3
>> Preferred Minor : 0
>>
>>     Update Time : Wed Feb 18 19:30:13 2015
>>           State : active
>>  Active Devices : 1
>> Working Devices : 1
>>  Failed Devices : 2
>>   Spare Devices : 0
>>        Checksum : 7c7659de - correct
>>          Events : 32945479
>
>
> And this:           ^^^^^^^^
>
>>          Layout : near=2
>>      Chunk Size : 256K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     2       8       34        2      active sync   /dev/sdc2
>>
>>    0     0       0        0        0      removed
>>    1     1       0        0        1      faulty removed
>>    2     2       8       34        2      active sync   /dev/sdc2
>>    3     3       0        0        3      faulty removed
>> /dev/sdd2:
>>           Magic : a92b4efc
>>         Version : 0.90.00
>>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>>   Creation Time : Sun Nov 29 15:33:50 2009
>>      Raid Level : raid10
>>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>>    Raid Devices : 4
>>   Total Devices : 5
>> Preferred Minor : 0
>>
>>     Update Time : Wed Sep  4 07:51:50 2013
>>           State : active
>>  Active Devices : 4
>> Working Devices : 5
>>  Failed Devices : 0
>>   Spare Devices : 1
>>        Checksum : 77c1a3a7 - correct
>>          Events : 53
>
>
> Whoa!              ^^^^
>
>>          Layout : near=2
>>      Chunk Size : 256K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     3       8       50        3      active sync   /dev/sdd2
>>
>>    0     0       8        2        0      active sync   /dev/sda2
>>    1     1       8       18        1      active sync   /dev/sdb2
>>    2     2       8       34        2      active sync   /dev/sdc2
>>    3     3       8       50        3      active sync   /dev/sdd2
>>    4     4       8       66        4      spare   /dev/sde2
>> /dev/sde2:
>>           Magic : a92b4efc
>>         Version : 0.90.00
>>            UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>>   Creation Time : Sun Nov 29 15:33:50 2009
>>      Raid Level : raid10
>>   Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>>      Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>>    Raid Devices : 4
>>   Total Devices : 5
>> Preferred Minor : 0
>>
>>     Update Time : Sun Sep  8 13:25:42 2013
>>           State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 0
>>   Spare Devices : 0
>>        Checksum : 77c775be - correct
>>          Events : 7934
>
>
> And Whoa again!    ^^^^^^
>
>>          Layout : near=2
>>      Chunk Size : 256K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     3       8       66        3      active sync   /dev/sde2
>>
>>    0     0       8        2        0      active sync   /dev/sda2
>>    1     1       8       18        1      active sync   /dev/sdb2
>>    2     2       8       34        2      active sync   /dev/sdc2
>>    3     3       8       66        3      active sync   /dev/sde2
>>
>> This makes sense to me as far as it goes but I don't see what to do
>> next. As I understand it the four partitions from sda2 to sdd2 would
>> form the array with sde as hot spare. It has been my assumption that if
>> a drive failed that sde would sync and take over. I don't know if this
>> is in fact the case and don't see a path forward. Of course the backups
>> are inadequate.
>
>
> Based on the events and update timestamps, sdd died sometime around Wed
> Sep 4 07:51:50 2013, at which point sde stepped in.  It too failed
> shortly after ~ Sun Sep 8 13:25:42 2013.  You then ran degraded for over
> a year until sdb also failed ~ Sat Nov 22 13:18:12 2014.  You were then
> running doubly-degraded (luckily on non-adjacent members) until this Feb
> 14 when sda was booted out.  Leaving only one running drive.
>
> { I wouldn't keep such people around, either. }
>
> Your best bet is to force assembly of the last two working drives to get
> the system running, then take an immediate backup of all critical files.
>  Do the forced assembly with the livecd, then do a clean shutdown.  You
> should then be able to boot the original OS and take your backup.
>
> Then you need to completely rebuild your system with proper log
> monitoring, array monitoring, and verification of your drives.
>
> Phil
>
>
> --
> "As long as politics is the shadow cast on society by big business,
> the attenuation of the shadow will not change the substance."
>
> -- John Dewey
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

re: no good deed goes unpunished...

[Dave Stevens]

From:	Roger Heflin <rogerheflin@gmail.com>
To:	Dave Stevens <geek@uniserve.com>
Cc:	Linux RAID <linux-raid@vger.kernel.org>
Subject: Re: no good deed goes unpunished...
Date:	Thu, 26 Mar 2015 19:04:45 -0500
Sender:	linux-raid-owner@vger.kernel.org

you should probably do cat /proc/mdstat

It is likely that the livecd may have already tried to assemble it,
and that the busy is because it is already in use.

If it is already assembled and you want to redo it first you will need
to stop the assembled array to be able to redo it.

so this is what I did while running the live distro:

# cat /proc/mdstat

Personalities :

md12 : inactive   sda2[0](S)
	732467520 blocks

md10 : inactive sdc2[2](S) sdb2[1](S)
	1464935040 blocks

unused devices: <none>


It seemed to me that matters were needlessly complicated by possible  
residual issues from my earlier unsuccessful attempts so I rebooted.


------------------------------ reboot ----------------------

# mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2

mdadm : /dev/md13 assembles from 1 drive - not enough to start the array

# cat /proc/mdstat

Personalities :

md13 : inactive sdc2[2](S) sdb2[1](S)

	1464935040 blocks

unused devices : <none>

# mkdir aa

# mount /dev/md13 aa

mount: /dev/md13: can't read superblock

So I still don't see what to do. Advice welcome.

Dave


-- 
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey

Re: no good deed goes unpunished...

[Roger Heflin]

I don't know what to make of it.

Everyone will ask you for this info:
mdadm --examine /dev/sdc2              and the same output against /dev/sdb2

On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
> From:   Roger Heflin <rogerheflin@gmail.com>
> To:     Dave Stevens <geek@uniserve.com>
> Cc:     Linux RAID <linux-raid@vger.kernel.org>
> Subject: Re: no good deed goes unpunished...
> Date:   Thu, 26 Mar 2015 19:04:45 -0500
> Sender: linux-raid-owner@vger.kernel.org
>
> you should probably do cat /proc/mdstat
>
> It is likely that the livecd may have already tried to assemble it,
> and that the busy is because it is already in use.
>
> If it is already assembled and you want to redo it first you will need
> to stop the assembled array to be able to redo it.
>
> so this is what I did while running the live distro:
>
> # cat /proc/mdstat
>
> Personalities :
>
> md12 : inactive   sda2[0](S)
>         732467520 blocks
>
> md10 : inactive sdc2[2](S) sdb2[1](S)
>         1464935040 blocks
>
> unused devices: <none>
>
>
> It seemed to me that matters were needlessly complicated by possible
> residual issues from my earlier unsuccessful attempts so I rebooted.
>
>
> ------------------------------ reboot ----------------------
>
> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>
> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>
> # cat /proc/mdstat
>
> Personalities :
>
> md13 : inactive sdc2[2](S) sdb2[1](S)
>
>         1464935040 blocks
>
> unused devices : <none>
>
> # mkdir aa
>
> # mount /dev/md13 aa
>
> mount: /dev/md13: can't read superblock
>
> So I still don't see what to do. Advice welcome.
>
> Dave
>
>
>
> --
> "As long as politics is the shadow cast on society by big business,
> the attenuation of the shadow will not change the substance."
>
> -- John Dewey
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no good deed goes unpunished...

[Dave Stevens]

Quoting Roger Heflin <rogerheflin@gmail.com>:

> I don't know what to make of it.
>
> Everyone will ask you for this info:
> mdadm --examine /dev/sdc2              and the same output against /dev/sdb2

ok, like this:

# cat /proc/mdstat

Personalities :

md12 : inactive   sda2[0](S)
	732467520 blocks

md10 : inactive sdc2[2](S) sdb2[1](S)
	1464935040 blocks

unused devices: <none>


------------------------------ reboot ----------------------

# mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2

mdadm : /dev/md13 assembles from 1 drive - not enough to start the array

# cat /proc/mdstat

Personalities :

md13 : inactive sdc2[2](S) sdb2[1](S)

	1464935040 blocks

unused devices : <none>

# mkdir aa

# mount /dev/md13 aa

mount: /dev/md13: can't read superblock


Dave




>
> On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
>> From:   Roger Heflin <rogerheflin@gmail.com>
>> To:     Dave Stevens <geek@uniserve.com>
>> Cc:     Linux RAID <linux-raid@vger.kernel.org>
>> Subject: Re: no good deed goes unpunished...
>> Date:   Thu, 26 Mar 2015 19:04:45 -0500
>> Sender: linux-raid-owner@vger.kernel.org
>>
>> you should probably do cat /proc/mdstat
>>
>> It is likely that the livecd may have already tried to assemble it,
>> and that the busy is because it is already in use.
>>
>> If it is already assembled and you want to redo it first you will need
>> to stop the assembled array to be able to redo it.
>>
>> so this is what I did while running the live distro:
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md12 : inactive   sda2[0](S)
>>         732467520 blocks
>>
>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>         1464935040 blocks
>>
>> unused devices: <none>
>>
>>
>> It seemed to me that matters were needlessly complicated by possible
>> residual issues from my earlier unsuccessful attempts so I rebooted.
>>
>>
>> ------------------------------ reboot ----------------------
>>
>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>
>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>
>>         1464935040 blocks
>>
>> unused devices : <none>
>>
>> # mkdir aa
>>
>> # mount /dev/md13 aa
>>
>> mount: /dev/md13: can't read superblock
>>
>> So I still don't see what to do. Advice welcome.
>>
>> Dave
>>
>>
>>
>> --
>> "As long as politics is the shadow cast on society by big business,
>> the attenuation of the shadow will not change the substance."
>>
>> -- John Dewey
>>
>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey

Re: no good deed goes unpunished...

[Brad Campbell]

md12 has claimed sda2 and md10 has sdb&c. Try this

mdadm --stop /dev/md12
mdadm --stop /dev/md10
mdadm --assemble --force /dev/md13 /dev/sd[abc]2



On 28/03/15 08:37, Dave Stevens wrote:
> Quoting Roger Heflin <rogerheflin@gmail.com>:
>
>> I don't know what to make of it.
>>
>> Everyone will ask you for this info:
>> mdadm --examine /dev/sdc2              and the same output against
>> /dev/sdb2
>
> ok, like this:
>
> # cat /proc/mdstat
>
> Personalities :
>
> md12 : inactive   sda2[0](S)
>      732467520 blocks
>
> md10 : inactive sdc2[2](S) sdb2[1](S)
>      1464935040 blocks
>
> unused devices: <none>
>
>
> ------------------------------ reboot ----------------------
>
> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>
> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>
> # cat /proc/mdstat
>
> Personalities :
>
> md13 : inactive sdc2[2](S) sdb2[1](S)
>
>      1464935040 blocks
>
> unused devices : <none>
>
> # mkdir aa
>
> # mount /dev/md13 aa
>
> mount: /dev/md13: can't read superblock
>
>
> Dave
>
>
>
>
>>
>> On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
>>> From:   Roger Heflin <rogerheflin@gmail.com>
>>> To:     Dave Stevens <geek@uniserve.com>
>>> Cc:     Linux RAID <linux-raid@vger.kernel.org>
>>> Subject: Re: no good deed goes unpunished...
>>> Date:   Thu, 26 Mar 2015 19:04:45 -0500
>>> Sender: linux-raid-owner@vger.kernel.org
>>>
>>> you should probably do cat /proc/mdstat
>>>
>>> It is likely that the livecd may have already tried to assemble it,
>>> and that the busy is because it is already in use.
>>>
>>> If it is already assembled and you want to redo it first you will need
>>> to stop the assembled array to be able to redo it.
>>>
>>> so this is what I did while running the live distro:
>>>
>>> # cat /proc/mdstat
>>>
>>> Personalities :
>>>
>>> md12 : inactive   sda2[0](S)
>>>         732467520 blocks
>>>
>>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>>         1464935040 blocks
>>>
>>> unused devices: <none>
>>>
>>>
>>> It seemed to me that matters were needlessly complicated by possible
>>> residual issues from my earlier unsuccessful attempts so I rebooted.
>>>
>>>
>>> ------------------------------ reboot ----------------------
>>>
>>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>>
>>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>>
>>> # cat /proc/mdstat
>>>
>>> Personalities :
>>>
>>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>>
>>>         1464935040 blocks
>>>
>>> unused devices : <none>
>>>
>>> # mkdir aa
>>>
>>> # mount /dev/md13 aa
>>>
>>> mount: /dev/md13: can't read superblock
>>>
>>> So I still don't see what to do. Advice welcome.
>>>
>>> Dave
>>>
>>>
>>>
>>> --
>>> "As long as politics is the shadow cast on society by big business,
>>> the attenuation of the shadow will not change the substance."
>>>
>>> -- John Dewey
>>>
>>>
>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
>


-- 
Dolphins are so intelligent that within a few weeks they can
train Americans to stand at the edge of the pool and throw them
fish.

Re: no good deed goes unpunished...

[Phil Turmel]

Hi Dave,

{Convention on kernel.org is reply-to-all, trim quotes, and avoid
top-posting.  Please trim.}

On 03/27/2015 08:37 PM, Dave Stevens wrote:
> # cat /proc/mdstat
> 
> Personalities :
> 
> md12 : inactive   sda2[0](S)
>     732467520 blocks
> 
> md10 : inactive sdc2[2](S) sdb2[1](S)
>     1464935040 blocks

If your livecd tried to be helpful by assembling arrays, but couldn't
complete, you end up with partially-assembled inactive arrays like shown
above.  The member devices shown are *busy*, under the expectation that
the remaining device(s) will show up soon. :-)

As Roger tried to suggest, you need to run:

mdadm --stop /dev/md10
mdadm --stop /dev/md12

That'll release the member devices you need.  However, the device
assembly pairs above look suspicious.  I suggest you show mdadm -E for
those devices again to make sure you work with the correct two disks.
(sda and sdc from the original report.)

Then you can do:

mdadm --assemble --force --verbose /dev/mdX /dev/sdY2 /dev/sdZ2

with the correct substitutions for X, Y, and Z.

If that doesn't work, show the output.

Phil

Re: no good deed goes unpunished...

[Dave Stevens]

Quoting Phil Turmel <philip@turmel.org>:

> Hi Dave,
>
> {Convention on kernel.org is reply-to-all, trim quotes, and avoid
> top-posting.  Please trim.}

ok, thanks

>
> On 03/27/2015 08:37 PM, Dave Stevens wrote:
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md12 : inactive   sda2[0](S)
>>     732467520 blocks
>>
>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>     1464935040 blocks
>
> If your livecd tried to be helpful by assembling arrays, but couldn't
> complete, you end up with partially-assembled inactive arrays like shown
> above.  The member devices shown are *busy*, under the expectation that
> the remaining device(s) will show up soon. :-)
>
> As Roger tried to suggest, you need to run:
>
> mdadm --stop /dev/md10
> mdadm --stop /dev/md12
>
> That'll release the member devices you need.  However, the device
> assembly pairs above look suspicious.  I suggest you show mdadm -E for
> those devices again to make sure you work with the correct two disks.
> (sda and sdc from the original report.)
>
> Then you can do:
>
> mdadm --assemble --force --verbose /dev/mdX /dev/sdY2 /dev/sdZ2
>
> with the correct substitutions for X, Y, and Z.
>
> If that doesn't work, show the output.
>
> Phil

will do, thanks

D

> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey

Re: no good deed goes unpunished...

[Dave Stevens]

Quoting Brad Campbell <brad@fnarfbargle.com>:

> md12 has claimed sda2 and md10 has sdb&c. Try this
>
> mdadm --stop /dev/md12
> mdadm --stop /dev/md10
> mdadm --assemble --force /dev/md13 /dev/sd[abc]2

I owe you a beer, Brad! Now I only have to mount the LVM partition,  
retrieve the Xen domains and rejig the management tools so I can do a  
meaningful backup and restore but this is the bit I couldn't get my  
head around.

Thanks to all who helped. Does anyone care to provide a reference to a  
good intro to software RAID? This experience was educational all  
right, but I would on the whole prefer to be more orderly and have a  
good reference.

Dave

>
>
>
> On 28/03/15 08:37, Dave Stevens wrote:
>> Quoting Roger Heflin <rogerheflin@gmail.com>:
>>
>>> I don't know what to make of it.
>>>
>>> Everyone will ask you for this info:
>>> mdadm --examine /dev/sdc2              and the same output against
>>> /dev/sdb2
>>
>> ok, like this:
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md12 : inactive   sda2[0](S)
>>     732467520 blocks
>>
>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>     1464935040 blocks
>>
>> unused devices: <none>
>>
>>
>> ------------------------------ reboot ----------------------
>>
>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>
>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>
>>     1464935040 blocks
>>
>> unused devices : <none>
>>
>> # mkdir aa
>>
>> # mount /dev/md13 aa
>>
>> mount: /dev/md13: can't read superblock
>>
>>
>> Dave
>>
>>
>>
>>
>>>
>>> On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
>>>> From:   Roger Heflin <rogerheflin@gmail.com>
>>>> To:     Dave Stevens <geek@uniserve.com>
>>>> Cc:     Linux RAID <linux-raid@vger.kernel.org>
>>>> Subject: Re: no good deed goes unpunished...
>>>> Date:   Thu, 26 Mar 2015 19:04:45 -0500
>>>> Sender: linux-raid-owner@vger.kernel.org
>>>>
>>>> you should probably do cat /proc/mdstat
>>>>
>>>> It is likely that the livecd may have already tried to assemble it,
>>>> and that the busy is because it is already in use.
>>>>
>>>> If it is already assembled and you want to redo it first you will need
>>>> to stop the assembled array to be able to redo it.
>>>>
>>>> so this is what I did while running the live distro:
>>>>
>>>> # cat /proc/mdstat
>>>>
>>>> Personalities :
>>>>
>>>> md12 : inactive   sda2[0](S)
>>>>        732467520 blocks
>>>>
>>>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>>>        1464935040 blocks
>>>>
>>>> unused devices: <none>
>>>>
>>>>
>>>> It seemed to me that matters were needlessly complicated by possible
>>>> residual issues from my earlier unsuccessful attempts so I rebooted.
>>>>
>>>>
>>>> ------------------------------ reboot ----------------------
>>>>
>>>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>>>
>>>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>>>
>>>> # cat /proc/mdstat
>>>>
>>>> Personalities :
>>>>
>>>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>>>
>>>>        1464935040 blocks
>>>>
>>>> unused devices : <none>
>>>>
>>>> # mkdir aa
>>>>
>>>> # mount /dev/md13 aa
>>>>
>>>> mount: /dev/md13: can't read superblock
>>>>
>>>> So I still don't see what to do. Advice welcome.
>>>>
>>>> Dave
>>>>
>>>>
>>>>
>>>> --
>>>> "As long as politics is the shadow cast on society by big business,
>>>> the attenuation of the shadow will not change the substance."
>>>>
>>>> -- John Dewey
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>>
>
>
> -- 
> Dolphins are so intelligent that within a few weeks they can
> train Americans to stand at the edge of the pool and throw them
> fish.
>



-- 
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey

Re: no good deed goes unpunished...

[Phil Turmel]

On 03/31/2015 05:41 PM, Dave Stevens wrote:
> Quoting Brad Campbell <brad@fnarfbargle.com>:
> 
>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>
>> mdadm --stop /dev/md12
>> mdadm --stop /dev/md10
>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
> 
> I owe you a beer, Brad! Now I only have to mount the LVM partition,
> retrieve the Xen domains and rejig the management tools so I can do a
> meaningful backup and restore but this is the bit I couldn't get my head
> around.

If you did this with those three disks, you are likely to have random
data corruption from the older mirror.  If so, stop and re-assemble with
just the last good disks.

Phil

Re: no good deed goes unpunished...

[Phil Turmel]

On 03/31/2015 05:52 PM, Phil Turmel wrote:
> On 03/31/2015 05:41 PM, Dave Stevens wrote:
>> Quoting Brad Campbell <brad@fnarfbargle.com>:
>>
>>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>>
>>> mdadm --stop /dev/md12
>>> mdadm --stop /dev/md10
>>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>>
>> I owe you a beer, Brad! Now I only have to mount the LVM partition,
>> retrieve the Xen domains and rejig the management tools so I can do a
>> meaningful backup and restore but this is the bit I couldn't get my head
>> around.
> 
> If you did this with those three disks, you are likely to have random
> data corruption from the older mirror.  If so, stop and re-assemble with
> just the last good disks.

Let me clarify: you only had *two* working disks for a month or so
before the array finally died.  Only assemble those two.

Phil

Re: no good deed goes unpunished...

[Dave Stevens]

Quoting Phil Turmel <philip@turmel.org>:

> On 03/31/2015 05:41 PM, Dave Stevens wrote:
>> Quoting Brad Campbell <brad@fnarfbargle.com>:
>>
>>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>>
>>> mdadm --stop /dev/md12
>>> mdadm --stop /dev/md10
>>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>>
>> I owe you a beer, Brad! Now I only have to mount the LVM partition,
>> retrieve the Xen domains and rejig the management tools so I can do a
>> meaningful backup and restore but this is the bit I couldn't get my head
>> around.
>
> If you did this with those three disks, you are likely to have random
> data corruption from the older mirror.  If so, stop and re-assemble with
> just the last good disks.
>
> Phil
>

OK, thanks!

Dave



-- 
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey

Re: no good deed goes unpunished...

[Dave Stevens]

Quoting Phil Turmel <philip@turmel.org>:

snip!
> On 03/31/2015 05:52 PM, Phil Turmel wrote:
> If you did this with those three disks, you are likely to have random
>> data corruption from the older mirror.  If so, stop and re-assemble with
>> just the last good disks.
>
> Let me clarify: you only had *two* working disks for a month or so
> before the array finally died.  Only assemble those two.

yes that's clear. I've done that, found the LVM volgroup and can now  
see the Xen domains, working on getting data off.

many thanks!

D

>
> Phil
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey