Desperate: mdadm fails to assemble raid saying devices are busy.

Desperate: mdadm fails to assemble raid saying devices are busy.

[Sachin Garg]

Hi,

I am on Arch Linux. I have a 4 device RAID 5 of which 1 device is no longer
working. I am unable to assemble the raid.

The active devices are:/dev/sda2, /dev/sdb1, /dev/sdc2

The errors I get are like:

# mdadm -Avf /dev/md127
mdadm: looking for devices for /dev/md127
mdadm: no RAID superblock on /dev/dm-6
mdadm: no RAID superblock on /dev/dm-5
mdadm: no RAID superblock on /dev/dm-4
mdadm: no RAID superblock on /dev/dm-3
mdadm: no RAID superblock on /dev/dm-2
mdadm: no RAID superblock on /dev/dm-1
mdadm: no RAID superblock on /dev/dm-0
mdadm: /dev/sdc2 is busy - skipping
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb1 is busy - skipping
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 is busy - skipping
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda

I would like to know what is making these devices busy. I do not see any
indication from lsof or from the device mapper that these devices are being
used. I have even changed the  initramfs and the fstab etc. to prevent this
array from being started at boot time - still unable to assemble it.

This array contains my /home and I do not know how to recover it.

Help please ...

Thanks,
Sachin

Re: Desperate: mdadm fails to assemble raid saying devices are busy.

[Roger Heflin]

cat /proc/mdstat

It is probably already partially assembled (and since no force not
active--note there is a kernel optionmd_mod.start_dirty_degraded=1
that will enable if the array can at least work ) so not operational,
typically you can stop it and it will no longer show up in mdstat then
you can do what you are trying to do.

lsof will not show things that are being held open/busy in the kernel,
lsof only shows userspace useage.

On Sat, Sep 21, 2013 at 9:32 AM, Sachin Garg
<sgarg.bugreporter@gmail.com> wrote:
> Hi,
>
> I am on Arch Linux. I have a 4 device RAID 5 of which 1 device is no longer
> working. I am unable to assemble the raid.
>
> The active devices are:/dev/sda2, /dev/sdb1, /dev/sdc2
>
> The errors I get are like:
>
> # mdadm -Avf /dev/md127
> mdadm: looking for devices for /dev/md127
> mdadm: no RAID superblock on /dev/dm-6
> mdadm: no RAID superblock on /dev/dm-5
> mdadm: no RAID superblock on /dev/dm-4
> mdadm: no RAID superblock on /dev/dm-3
> mdadm: no RAID superblock on /dev/dm-2
> mdadm: no RAID superblock on /dev/dm-1
> mdadm: no RAID superblock on /dev/dm-0
> mdadm: /dev/sdc2 is busy - skipping
> mdadm: no RAID superblock on /dev/sdc1
> mdadm: no RAID superblock on /dev/sdc
> mdadm: /dev/sdb1 is busy - skipping
> mdadm: no RAID superblock on /dev/sdb
> mdadm: no RAID superblock on /dev/sda3
> mdadm: /dev/sda2 is busy - skipping
> mdadm: no RAID superblock on /dev/sda1
> mdadm: no RAID superblock on /dev/sda
>
> I would like to know what is making these devices busy. I do not see any
> indication from lsof or from the device mapper that these devices are being
> used. I have even changed the  initramfs and the fstab etc. to prevent this
> array from being started at boot time - still unable to assemble it.
>
> This array contains my /home and I do not know how to recover it.
>
> Help please ...
>
> Thanks,
> Sachin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Desperate: mdadm fails to assemble raid saying devices are busy.

[Sachin Garg]

Hi Roger & Mikael,

Thanks a lot for your replies. I tried the following and still to no avail.

Here are the results of the various commands:

# cat /proc/mdstat
Personalities :
md0 : inactive sda2[0](S) sdb1[4](S) sdc2[1](S)
      480215144 blocks super 1.2

unused devices: <none>

# mdadm -V
mdadm - v3.2.6 - 25th October 2012

# uname -a
Linux triveni 3.10.6-2-ARCH #1 SMP PREEMPT Tue Aug 13 09:47:01 CEST 2013
x86_64 GNU/Linux

# mdadm -Esv
ARRAY /dev/md/0  level=raid5 metadata=1.2 num-devices=4
UUID=01b153ea:a93c7325:c13bf9cf:3195666f name=sysresccd:0
   devices=/dev/sdc2,/dev/sdb1,/dev/sda2

Based on Roger's mail, set the kernel option
"md_mod.start_dirty_degraded=1":

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=/dev/mapper/triveni--lvm--320gp3-root rw
md_mod.start_dirty_degraded=1

Still, I get:

# mdadm -Avfs
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/dm-6
mdadm: no RAID superblock on /dev/dm-5
mdadm: no RAID superblock on /dev/dm-4
mdadm: no RAID superblock on /dev/dm-3
mdadm: no RAID superblock on /dev/dm-2
mdadm: no RAID superblock on /dev/dm-1
mdadm: no RAID superblock on /dev/dm-0
mdadm: /dev/sdc2 is busy - skipping
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb1 is busy - skipping
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 is busy - skipping
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda



On 21-Sep-13 1:44 PM, Roger Heflin wrote:
> cat /proc/mdstat
> 
> It is probably already partially assembled (and since no force not
> active--note there is a kernel optionmd_mod.start_dirty_degraded=1
> that will enable if the array can at least work ) so not operational,
> typically you can stop it and it will no longer show up in mdstat then
> you can do what you are trying to do.
> 
> lsof will not show things that are being held open/busy in the kernel,
> lsof only shows userspace useage.
> 
> On Sat, Sep 21, 2013 at 9:32 AM, Sachin Garg
> <sgarg.bugreporter@gmail.com> wrote:
>> Hi,
>>
>> I am on Arch Linux. I have a 4 device RAID 5 of which 1 device is no longer
>> working. I am unable to assemble the raid.
>>
>> The active devices are:/dev/sda2, /dev/sdb1, /dev/sdc2
>>
>> The errors I get are like:
>>
>> # mdadm -Avf /dev/md127
>> mdadm: looking for devices for /dev/md127
>> mdadm: no RAID superblock on /dev/dm-6
>> mdadm: no RAID superblock on /dev/dm-5
>> mdadm: no RAID superblock on /dev/dm-4
>> mdadm: no RAID superblock on /dev/dm-3
>> mdadm: no RAID superblock on /dev/dm-2
>> mdadm: no RAID superblock on /dev/dm-1
>> mdadm: no RAID superblock on /dev/dm-0
>> mdadm: /dev/sdc2 is busy - skipping
>> mdadm: no RAID superblock on /dev/sdc1
>> mdadm: no RAID superblock on /dev/sdc
>> mdadm: /dev/sdb1 is busy - skipping
>> mdadm: no RAID superblock on /dev/sdb
>> mdadm: no RAID superblock on /dev/sda3
>> mdadm: /dev/sda2 is busy - skipping
>> mdadm: no RAID superblock on /dev/sda1
>> mdadm: no RAID superblock on /dev/sda
>>
>> I would like to know what is making these devices busy. I do not see any
>> indication from lsof or from the device mapper that these devices are being
>> used. I have even changed the  initramfs and the fstab etc. to prevent this
>> array from being started at boot time - still unable to assemble it.
>>
>> This array contains my /home and I do not know how to recover it.
>>
>> Help please ...
>>
>> Thanks,
>> Sachin
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Desperate: mdadm fails to assemble raid saying devices are busy.

[Mikael Abrahamsson]

On Mon, 23 Sep 2013, Sachin Garg wrote:

> Hi Roger & Mikael,
>
> Thanks a lot for your replies. I tried the following and still to no avail.
>
> Here are the results of the various commands:
>
> # cat /proc/mdstat
> Personalities :
> md0 : inactive sda2[0](S) sdb1[4](S) sdc2[1](S)
>      480215144 blocks super 1.2

Do "mdadm --stop /dev/md0" and then try

"mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb1 /dev/sdc2"

See if this removes the busy problem. Also post output from this 
operation as well as what comes up in dmesg when you do this.


-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: (Solved) Desperate: mdadm fails to assemble raid saying devices are busy.

[Sachin Garg]

On 23-Sep-13 11:43 AM, Mikael Abrahamsson wrote:
> Do "mdadm --stop /dev/md0" and then try
> 
> "mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb1 /dev/sdc2"
> 
> See if this removes the busy problem. Also post output from this
> operation as well as what comes up in dmesg when you do this.

Thanks, Mikael - that worked. The output of the various commands are
given below.

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0

[22874.978883] md: md0 stopped.
[22874.978900] md: unbind<sda2>
[22875.018193] md: export_rdev(sda2)
[22875.018284] md: unbind<sdb1>
[22875.059733] md: export_rdev(sdb1)
[22875.059821] md: unbind<sdc2>
[22875.079735] md: export_rdev(sdc2)
[22875.652012] md: md0 stopped.

# mdadm --assemble /dev/md0 /dev/sda2 /dev/sdb1 /dev/sdc2
mdadm: /dev/md0 assembled from 3 drives - not enough to start the array
while not clean - consider --force.

[22875.653625] md: bind<sdc2>
[22875.653943] md: bind<sdb1>
[22875.654302] md: bind<sda2>

# mdadm --assemble --force /dev/md0 /dev/sda2 /dev/sdb1 /dev/sdc2

This worked!!!

# mdadm -Dv /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Sat Mar  2 11:03:45 2013
     Raid Level : raid5
     Array Size : 468475392 (446.77 GiB 479.72 GB)
  Used Dev Size : 156158464 (148.92 GiB 159.91 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Mon Sep 23 15:41:08 2013
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : sysresccd:0
           UUID : 01b153ea:a93c7325:c13bf9cf:3195666f
         Events : 118445

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       34        1      active sync   /dev/sdc2
       2       0        0        2      removed
       4       8       17        3      active sync   /dev/sdb1

Now, I need to find a new hard drive and put it in Slot 2. What would be
the best way to do that, apart from RTFM ?

Thanks for all your help.

Best,
Sachin

Re: (Solved) Desperate: mdadm fails to assemble raid saying devices are busy.

[Mikael Abrahamsson]

On Mon, 23 Sep 2013, Sachin Garg wrote:

> Now, I need to find a new hard drive and put it in Slot 2. What would be 
> the best way to do that, apart from RTFM ?

I recommend you to do RTFM anyhow, it'll help you in the long run. "--add" 
is what you're looking for for above operation.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Desperate: mdadm fails to assemble raid saying devices are busy.

[Mikael Abrahamsson]

On Sat, 21 Sep 2013, Sachin Garg wrote:

> This array contains my /home and I do not know how to recover it.

what does "cat /proc/mdstat" say? Also post mdadm and kernel version.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se