* Re: no good deed goes unpunished...
From: Phil Turmel @ 2015-03-28 1:28 UTC (permalink / raw)
To: Dave Stevens, Roger Heflin; +Cc: linux-raid
In-Reply-To: <20150327173709.19022q59t5hxh4tx@webmail.uniserve.com>
Hi Dave,
{Convention on kernel.org is reply-to-all, trim quotes, and avoid
top-posting. Please trim.}
On 03/27/2015 08:37 PM, Dave Stevens wrote:
> # cat /proc/mdstat
>
> Personalities :
>
> md12 : inactive sda2[0](S)
> 732467520 blocks
>
> md10 : inactive sdc2[2](S) sdb2[1](S)
> 1464935040 blocks
If your livecd tried to be helpful by assembling arrays, but couldn't
complete, you end up with partially-assembled inactive arrays like shown
above. The member devices shown are *busy*, under the expectation that
the remaining device(s) will show up soon. :-)
As Roger tried to suggest, you need to run:
mdadm --stop /dev/md10
mdadm --stop /dev/md12
That'll release the member devices you need. However, the device
assembly pairs above look suspicious. I suggest you show mdadm -E for
those devices again to make sure you work with the correct two disks.
(sda and sdc from the original report.)
Then you can do:
mdadm --assemble --force --verbose /dev/mdX /dev/sdY2 /dev/sdZ2
with the correct substitutions for X, Y, and Z.
If that doesn't work, show the output.
Phil
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Brad Campbell @ 2015-03-28 0:57 UTC (permalink / raw)
To: Dave Stevens, Roger Heflin; +Cc: linux-raid
In-Reply-To: <20150327173709.19022q59t5hxh4tx@webmail.uniserve.com>
md12 has claimed sda2 and md10 has sdb&c. Try this
mdadm --stop /dev/md12
mdadm --stop /dev/md10
mdadm --assemble --force /dev/md13 /dev/sd[abc]2
On 28/03/15 08:37, Dave Stevens wrote:
> Quoting Roger Heflin <rogerheflin@gmail.com>:
>
>> I don't know what to make of it.
>>
>> Everyone will ask you for this info:
>> mdadm --examine /dev/sdc2 and the same output against
>> /dev/sdb2
>
> ok, like this:
>
> # cat /proc/mdstat
>
> Personalities :
>
> md12 : inactive sda2[0](S)
> 732467520 blocks
>
> md10 : inactive sdc2[2](S) sdb2[1](S)
> 1464935040 blocks
>
> unused devices: <none>
>
>
> ------------------------------ reboot ----------------------
>
> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>
> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>
> # cat /proc/mdstat
>
> Personalities :
>
> md13 : inactive sdc2[2](S) sdb2[1](S)
>
> 1464935040 blocks
>
> unused devices : <none>
>
> # mkdir aa
>
> # mount /dev/md13 aa
>
> mount: /dev/md13: can't read superblock
>
>
> Dave
>
>
>
>
>>
>> On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
>>> From: Roger Heflin <rogerheflin@gmail.com>
>>> To: Dave Stevens <geek@uniserve.com>
>>> Cc: Linux RAID <linux-raid@vger.kernel.org>
>>> Subject: Re: no good deed goes unpunished...
>>> Date: Thu, 26 Mar 2015 19:04:45 -0500
>>> Sender: linux-raid-owner@vger.kernel.org
>>>
>>> you should probably do cat /proc/mdstat
>>>
>>> It is likely that the livecd may have already tried to assemble it,
>>> and that the busy is because it is already in use.
>>>
>>> If it is already assembled and you want to redo it first you will need
>>> to stop the assembled array to be able to redo it.
>>>
>>> so this is what I did while running the live distro:
>>>
>>> # cat /proc/mdstat
>>>
>>> Personalities :
>>>
>>> md12 : inactive sda2[0](S)
>>> 732467520 blocks
>>>
>>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>> 1464935040 blocks
>>>
>>> unused devices: <none>
>>>
>>>
>>> It seemed to me that matters were needlessly complicated by possible
>>> residual issues from my earlier unsuccessful attempts so I rebooted.
>>>
>>>
>>> ------------------------------ reboot ----------------------
>>>
>>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>>
>>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>>
>>> # cat /proc/mdstat
>>>
>>> Personalities :
>>>
>>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>>
>>> 1464935040 blocks
>>>
>>> unused devices : <none>
>>>
>>> # mkdir aa
>>>
>>> # mount /dev/md13 aa
>>>
>>> mount: /dev/md13: can't read superblock
>>>
>>> So I still don't see what to do. Advice welcome.
>>>
>>> Dave
>>>
>>>
>>>
>>> --
>>> "As long as politics is the shadow cast on society by big business,
>>> the attenuation of the shadow will not change the substance."
>>>
>>> -- John Dewey
>>>
>>>
>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
>
--
Dolphins are so intelligent that within a few weeks they can
train Americans to stand at the edge of the pool and throw them
fish.
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-28 0:37 UTC (permalink / raw)
To: Roger Heflin; +Cc: linux-raid
In-Reply-To: <CAAMCDef5ejTu4aoYxMRxaRPvV9kKO4qwzbo9jAuPnFDYDoHo0A@mail.gmail.com>
Quoting Roger Heflin <rogerheflin@gmail.com>:
> I don't know what to make of it.
>
> Everyone will ask you for this info:
> mdadm --examine /dev/sdc2 and the same output against /dev/sdb2
ok, like this:
# cat /proc/mdstat
Personalities :
md12 : inactive sda2[0](S)
732467520 blocks
md10 : inactive sdc2[2](S) sdb2[1](S)
1464935040 blocks
unused devices: <none>
------------------------------ reboot ----------------------
# mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
# cat /proc/mdstat
Personalities :
md13 : inactive sdc2[2](S) sdb2[1](S)
1464935040 blocks
unused devices : <none>
# mkdir aa
# mount /dev/md13 aa
mount: /dev/md13: can't read superblock
Dave
>
> On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
>> From: Roger Heflin <rogerheflin@gmail.com>
>> To: Dave Stevens <geek@uniserve.com>
>> Cc: Linux RAID <linux-raid@vger.kernel.org>
>> Subject: Re: no good deed goes unpunished...
>> Date: Thu, 26 Mar 2015 19:04:45 -0500
>> Sender: linux-raid-owner@vger.kernel.org
>>
>> you should probably do cat /proc/mdstat
>>
>> It is likely that the livecd may have already tried to assemble it,
>> and that the busy is because it is already in use.
>>
>> If it is already assembled and you want to redo it first you will need
>> to stop the assembled array to be able to redo it.
>>
>> so this is what I did while running the live distro:
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md12 : inactive sda2[0](S)
>> 732467520 blocks
>>
>> md10 : inactive sdc2[2](S) sdb2[1](S)
>> 1464935040 blocks
>>
>> unused devices: <none>
>>
>>
>> It seemed to me that matters were needlessly complicated by possible
>> residual issues from my earlier unsuccessful attempts so I rebooted.
>>
>>
>> ------------------------------ reboot ----------------------
>>
>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>
>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>
>> 1464935040 blocks
>>
>> unused devices : <none>
>>
>> # mkdir aa
>>
>> # mount /dev/md13 aa
>>
>> mount: /dev/md13: can't read superblock
>>
>> So I still don't see what to do. Advice welcome.
>>
>> Dave
>>
>>
>>
>> --
>> "As long as politics is the shadow cast on society by big business,
>> the attenuation of the shadow will not change the substance."
>>
>> -- John Dewey
>>
>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: mdadm RAID6 "active" with spares and failed disks; need help
From: Matt Callaghan @ 2015-03-27 23:48 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54C0C73A.4080409@lentijn.sess.ink>
Back at it with fresh brain and fresh hardware. (several months ago I
got part-way through Valentine's ideas but not all the way -- decided to
get a clean setup before progressing further)
I have built a new (fresh/clean) server, and compiled+installed the
latest mdadm v3.3.2.
The 8x drives from this RAID6 array have also been moved to the new
temporary server.
Now of course, in the new server, the device labels are different.
I need to map the previous "known labels" in the old server (/dev/sdX)
to the "new labels" in order to get the drive ordering for re-assembly
right.
http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
e.g. before I had:
{{{
/dev/sd[nmlpiokj]1
}}}
, and now I have:
{{{
/dev/sd[abcdefghi]1
}}}
Unfortunately I don't have any smartctl output saved from the previous
server and I can't find a way to map device drive label to serial numbers.
Any thoughts how I could do this based on the data I have saved in that
forum post?
~Matt
-------- Original Message --------
From: Valentijn <v@lentijn.sess.ink>
Sent: 1/22/2015, 4:47:38 AM
To: Matt Callaghan <matt_callaghan@sympatico.ca>, Wols Lists
<antlists@youngman.org.uk>, linux-raid@vger.kernel.org
Cc:
Subject: Re: mdadm RAID6 "active" with spares and failed disks; need help
> Hi Matt,
As long as your data is still somewhere on these disks, all is not -
necessarily - lost. You could still try using dumpe2fs (and later
e2fsck) and/or dumpe2fs with different superblocks. And even if you
cannot find your file system by any means, you could try to use the
"foremost" utility to scrape off images, documents and the like from
these disks.
So I still don't think all is lost. However, I do think that will cost
more time. You may want to dedicate a spare machine to this task,
because of the resources.
I see that your mdadm says this, somewhere along your odyssee:
mdadm: /dev/sdk1 appears to contain an ext2fs file system
size=1695282944K mtime=Tue Apr 12 11:10:24 1977
... which could mean (I'm not sure, I'm just guessing) that due to the
internal bitmap, your fs has been overwritten.
Your new array in fact said:
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Jan 7 09:46:44 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c7603819 - correct
Events : 0
... as far as I understand, this means that 8 blocks from the
superblock, some - whatever size - sectors were occupied by the Internal
Bitmap, which, in turn, would mean your filesystem superblock has been
overwritten.
The good news is: there is more than one superblock.
BTW, didn't you have the right raid drive ordering from the original
disks? You did have output of "mdadm --examine" after the array broke
down, didn't you? So your "create" statement is, by definition, correct
if a new "--examine" output shows the same output - hence the filesystem
is correct if the latter is the case?
So please try if "dumpe2fs -h -o superblock=32768" does anything. Or
98304, 163840, 229376. Dumpe2fs just dumps the fs header, nothing more.
If dumpe2fs doesn't do anything (but complain that it "Couldn't find
valid filesystem superblock"), then you could still try if "foremost"
finds anything. It's not that hard to use, you simply dedicate some
storage to it and tell it to scrape your array. It *will* find things
and it's up to you to see if
1) documents, images and the like are all 64K or 128K or less - and/or
contain large blocks of rubbish. This probably means you have the wrong
array config, because foremost in this case only finds single "chunks"
with correct data - if a file is longer, it doesn't find it and/or spews
out random data from other images
2) documents, images etcetera are OK. This means your array is OK. You
then can use foremost to scrape off everything (it may take weeks but it
could work), or simply try to find where the filesystem superblock hangs
out (if the array is in good order, the fs superblock must be somewhere,
right?)
Please, please try to do as little as possible on the real disks. Use
dmsetup to create snapshots. Copy the disks. Use hardware that is in
good state - you don't want to loose your data that you just found back
because the memory is flakey, do you? ;-)
I hope it's going to work.
Best regards,
Valentijn
On 01/21/15 01:34, Matt Callaghan wrote:
I tried again with the --bitmap=none, clearly that was a miss on my part.
However, still even with that correction, and attempting across varying
combinations of "drive ordering", the filesystem appears corrupt.
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Roger Heflin @ 2015-03-27 18:42 UTC (permalink / raw)
To: Dave Stevens; +Cc: linux-raid
In-Reply-To: <20150327103301.16746hhb6t1sroel@webmail.uniserve.com>
I don't know what to make of it.
Everyone will ask you for this info:
mdadm --examine /dev/sdc2 and the same output against /dev/sdb2
On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
> From: Roger Heflin <rogerheflin@gmail.com>
> To: Dave Stevens <geek@uniserve.com>
> Cc: Linux RAID <linux-raid@vger.kernel.org>
> Subject: Re: no good deed goes unpunished...
> Date: Thu, 26 Mar 2015 19:04:45 -0500
> Sender: linux-raid-owner@vger.kernel.org
>
> you should probably do cat /proc/mdstat
>
> It is likely that the livecd may have already tried to assemble it,
> and that the busy is because it is already in use.
>
> If it is already assembled and you want to redo it first you will need
> to stop the assembled array to be able to redo it.
>
> so this is what I did while running the live distro:
>
> # cat /proc/mdstat
>
> Personalities :
>
> md12 : inactive sda2[0](S)
> 732467520 blocks
>
> md10 : inactive sdc2[2](S) sdb2[1](S)
> 1464935040 blocks
>
> unused devices: <none>
>
>
> It seemed to me that matters were needlessly complicated by possible
> residual issues from my earlier unsuccessful attempts so I rebooted.
>
>
> ------------------------------ reboot ----------------------
>
> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>
> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>
> # cat /proc/mdstat
>
> Personalities :
>
> md13 : inactive sdc2[2](S) sdb2[1](S)
>
> 1464935040 blocks
>
> unused devices : <none>
>
> # mkdir aa
>
> # mount /dev/md13 aa
>
> mount: /dev/md13: can't read superblock
>
> So I still don't see what to do. Advice welcome.
>
> Dave
>
>
>
> --
> "As long as politics is the shadow cast on society by big business,
> the attenuation of the shadow will not change the substance."
>
> -- John Dewey
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-27 17:33 UTC (permalink / raw)
To: linux-raid
From: Roger Heflin <rogerheflin@gmail.com>
To: Dave Stevens <geek@uniserve.com>
Cc: Linux RAID <linux-raid@vger.kernel.org>
Subject: Re: no good deed goes unpunished...
Date: Thu, 26 Mar 2015 19:04:45 -0500
Sender: linux-raid-owner@vger.kernel.org
you should probably do cat /proc/mdstat
It is likely that the livecd may have already tried to assemble it,
and that the busy is because it is already in use.
If it is already assembled and you want to redo it first you will need
to stop the assembled array to be able to redo it.
so this is what I did while running the live distro:
# cat /proc/mdstat
Personalities :
md12 : inactive sda2[0](S)
732467520 blocks
md10 : inactive sdc2[2](S) sdb2[1](S)
1464935040 blocks
unused devices: <none>
It seemed to me that matters were needlessly complicated by possible
residual issues from my earlier unsuccessful attempts so I rebooted.
------------------------------ reboot ----------------------
# mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
# cat /proc/mdstat
Personalities :
md13 : inactive sdc2[2](S) sdb2[1](S)
1464935040 blocks
unused devices : <none>
# mkdir aa
# mount /dev/md13 aa
mount: /dev/md13: can't read superblock
So I still don't see what to do. Advice welcome.
Dave
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: missing arrays after OS upgrade
From: Daniel Sanabria @ 2015-03-27 9:40 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <5512A6EB.10104@turmel.org>
Thanks Phil,
The array is now up and running following your advice.
Thanks again,
Daniel
^ permalink raw reply
* Re: /dev/md0 can't be created
From: Xiao Ni @ 2015-03-27 6:06 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <200110393.3671122.1427355395924.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 4527 bytes --]
I'm lucky, I reproduced it again and I ran the command udevadm monitor > monitor before test.
I'll put it in as an attachment.
I found there is a not whole message in the last line.
Xiao
----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: "NeilBrown" <neilb@suse.de>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, March 26, 2015 3:36:35 PM
> Subject: Re: /dev/md0 can't be created
>
> Hi Neil
>
> I can reproduce it now. Do you want me to run udevadm monitor before the
> test?
> And I checked the /var/log/messages, it show the information:
>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-2>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-3>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-4>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-5>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-6>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-7>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-9>
> Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-8>
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-7 operational as
> raid disk 5
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-6 operational as
> raid disk 4
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-5 operational as
> raid disk 3
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-4 operational as
> raid disk 2
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-3 operational as
> raid disk 1
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-2 operational as
> raid disk 0
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: allocated 0kB
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: raid level 5 active with 6
> out of 7 devices, algorithm 2
> Oct 20 19:59:59 ibm-z10-25 kernel: md/raid456: discard support disabled due
> to uncertainty.
> Oct 20 19:59:59 ibm-z10-25 kernel: Set
> raid456.devices_handle_discard_safely=Y to override.
> Oct 20 19:59:59 ibm-z10-25 kernel: md0: detected capacity change from 0 to
> 1881145344
> Oct 20 19:59:59 ibm-z10-25 kernel: md: recovery of RAID array md0
> Oct 20 19:59:59 ibm-z10-25 kernel: md: minimum _guaranteed_ speed: 1000
> KB/sec/disk.
> Oct 20 19:59:59 ibm-z10-25 kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for recovery.
> Oct 20 19:59:59 ibm-z10-25 kernel: md: using 128k window, over a total of
> 306176k.
> Oct 20 19:59:59 ibm-z10-25 systemd-udevd: inotify_add_watch(7, /dev/md0, 10)
> failed: No such file or directory
>
> Xiao
>
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Wednesday, March 25, 2015 2:35:29 PM
> > Subject: Re: /dev/md0 can't be created
> >
> > On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> >
> > > Hi all
> > >
> > > I have encountered so many times, the raid device is created
> > > successfully, but the directory
> > > /dev/md0 can't be created. It can't reproduce 100%.
> > >
> > > [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> > > Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > > md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3]
> > > loop2[2] loop1[1] loop0[0]
> > > 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]
> > > bitmap: 0/1 pages [0KB], 65536KB chunk
> > >
> > > unused devices: <none>
> > > [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> > > ls: cannot access /dev/md0: No such file or directory
> > >
> > > The underline devices are loop devices which are created with big
> > > file.
> > >
> > > The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm -
> > > v3.3.2
> > > - 21st August 2014)
> > > I'll try to reproduce this with upstream kernel and mdadm. But I think it
> > > shouldn't be the problem about kernel.
> > >
> > > What do you think I should check for this? And which tool is
> > > responsible for creating the directory? Maybe
> > > I can add some log to it to find the reason.
> > >
> >
> > /dev/md0 is created by udev.
> > Run
> > udevadm monitor
> >
> > to see the events that udev is processing. When and ADD event for "md0" is
> > processed, /dev/md0 should get created.
> >
> > NeilBrown
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
[-- Attachment #2: monitor --]
[-- Type: application/octet-stream, Size: 24576 bytes --]
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
KERNEL[6515.301274] change /devices/virtual/block/loop0 (block)
KERNEL[6515.303462] change /devices/virtual/block/loop0 (block)
UDEV [6515.319066] change /devices/virtual/block/loop0 (block)
UDEV [6515.331930] change /devices/virtual/block/loop0 (block)
KERNEL[6517.641752] change /devices/virtual/block/loop1 (block)
KERNEL[6517.642637] change /devices/virtual/block/loop1 (block)
UDEV [6517.660489] change /devices/virtual/block/loop1 (block)
UDEV [6517.677432] change /devices/virtual/block/loop1 (block)
KERNEL[6520.022457] change /devices/virtual/block/loop2 (block)
KERNEL[6520.023667] change /devices/virtual/block/loop2 (block)
UDEV [6520.039374] change /devices/virtual/block/loop2 (block)
UDEV [6520.058352] change /devices/virtual/block/loop2 (block)
KERNEL[6522.368026] change /devices/virtual/block/loop3 (block)
KERNEL[6522.368622] change /devices/virtual/block/loop3 (block)
UDEV [6522.383221] change /devices/virtual/block/loop3 (block)
UDEV [6522.402220] change /devices/virtual/block/loop3 (block)
KERNEL[6525.645736] change /devices/virtual/block/loop4 (block)
KERNEL[6525.647442] change /devices/virtual/block/loop4 (block)
UDEV [6525.663895] change /devices/virtual/block/loop4 (block)
UDEV [6525.679432] change /devices/virtual/block/loop4 (block)
KERNEL[6529.974901] change /devices/virtual/block/loop5 (block)
KERNEL[6529.976536] change /devices/virtual/block/loop5 (block)
UDEV [6529.993692] change /devices/virtual/block/loop5 (block)
UDEV [6530.009535] change /devices/virtual/block/loop5 (block)
KERNEL[6535.055372] change /devices/virtual/block/loop6 (block)
KERNEL[6535.057058] change /devices/virtual/block/loop6 (block)
UDEV [6535.072879] change /devices/virtual/block/loop6 (block)
UDEV [6535.087172] change /devices/virtual/block/loop6 (block)
KERNEL[6540.574575] change /devices/virtual/block/loop7 (block)
KERNEL[6540.576235] change /devices/virtual/block/loop7 (block)
UDEV [6540.593438] change /devices/virtual/block/loop7 (block)
UDEV [6540.607785] change /devices/virtual/block/loop7 (block)
KERNEL[6540.754883] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6540.755896] add /devices/virtual/block/md0 (block)
UDEV [6540.755917] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6540.757982] change /devices/virtual/block/loop0 (block)
UDEV [6540.769174] change /devices/virtual/block/loop0 (block)
UDEV [6540.770964] add /devices/virtual/block/md0 (block)
KERNEL[6541.267525] change /devices/virtual/block/loop0 (block)
KERNEL[6541.268027] change /devices/virtual/block/loop1 (block)
UDEV [6541.280223] change /devices/virtual/block/loop1 (block)
UDEV [6541.280839] change /devices/virtual/block/loop0 (block)
KERNEL[6541.750291] change /devices/virtual/block/loop1 (block)
KERNEL[6541.751248] change /devices/virtual/block/loop2 (block)
UDEV [6541.763540] change /devices/virtual/block/loop1 (block)
UDEV [6541.763573] change /devices/virtual/block/loop2 (block)
KERNEL[6543.691132] change /devices/virtual/block/loop2 (block)
KERNEL[6543.691992] change /devices/virtual/block/loop3 (block)
UDEV [6543.703832] change /devices/virtual/block/loop3 (block)
UDEV [6543.705042] change /devices/virtual/block/loop2 (block)
KERNEL[6547.431384] change /devices/virtual/block/loop3 (block)
KERNEL[6547.433030] change /devices/virtual/block/loop4 (block)
UDEV [6547.447432] change /devices/virtual/block/loop3 (block)
UDEV [6547.447464] change /devices/virtual/block/loop4 (block)
KERNEL[6552.923835] change /devices/virtual/block/loop4 (block)
KERNEL[6552.925517] change /devices/virtual/block/loop5 (block)
UDEV [6552.939977] change /devices/virtual/block/loop5 (block)
UDEV [6552.943144] change /devices/virtual/block/loop4 (block)
KERNEL[6557.602230] change /devices/virtual/block/loop5 (block)
KERNEL[6557.604138] change /devices/virtual/block/loop6 (block)
UDEV [6557.618449] change /devices/virtual/block/loop6 (block)
UDEV [6557.619513] change /devices/virtual/block/loop5 (block)
KERNEL[6561.675460] change /devices/virtual/block/loop6 (block)
UDEV [6561.689512] change /devices/virtual/block/loop6 (block)
KERNEL[6561.743692] change /devices/virtual/block/md0 (block)
UDEV [6561.784740] change /devices/virtual/block/md0 (block)
KERNEL[6605.925900] change /devices/virtual/block/md0 (block)
UDEV [6606.093197] change /devices/virtual/block/md0 (block)
KERNEL[6609.034238] change /devices/virtual/block/md0 (block)
UDEV [6609.683910] change /devices/virtual/block/md0 (block)
KERNEL[6609.806307] change /devices/virtual/block/md0 (block)
UDEV [6609.943069] change /devices/virtual/block/md0 (block)
KERNEL[6610.665984] change /devices/virtual/block/md0 (block)
UDEV [6611.279813] change /devices/virtual/block/md0 (block)
KERNEL[6611.567864] change /devices/virtual/block/md0 (block)
UDEV [6611.770054] change /devices/virtual/block/md0 (block)
KERNEL[6612.124655] change /devices/virtual/block/md0 (block)
UDEV [6612.182382] change /devices/virtual/block/md0 (block)
KERNEL[6612.532394] change /devices/virtual/block/md0 (block)
UDEV [6613.142899] change /devices/virtual/block/md0 (block)
KERNEL[6613.450778] change /devices/virtual/block/md0 (block)
UDEV [6613.941184] change /devices/virtual/block/md0 (block)
KERNEL[6613.989039] change /devices/virtual/block/md0 (block)
UDEV [6614.025015] change /devices/virtual/block/md0 (block)
KERNEL[6614.056150] change /devices/virtual/block/md0 (block)
UDEV [6614.099238] change /devices/virtual/block/md0 (block)
KERNEL[6614.776258] change /devices/virtual/block/md0 (block)
UDEV [6614.922649] change /devices/virtual/block/md0 (block)
KERNEL[6614.962054] change /devices/virtual/block/md0 (block)
UDEV [6615.000911] change /devices/virtual/block/md0 (block)
KERNEL[6615.169124] change /devices/virtual/block/md0 (block)
UDEV [6615.355895] change /devices/virtual/block/md0 (block)
KERNEL[6660.683679] change /devices/virtual/block/md0 (block)
UDEV [6660.757371] change /devices/virtual/block/md0 (block)
KERNEL[6660.771043] change /devices/virtual/block/md0 (block)
UDEV [6660.814874] change /devices/virtual/block/md0 (block)
KERNEL[6671.628859] change /devices/virtual/block/md0 (block)
UDEV [6671.830377] change /devices/virtual/block/md0 (block)
KERNEL[6687.755208] change /devices/virtual/block/md0 (block)
KERNEL[6687.885216] change /devices/virtual/block/md0 (block)
KERNEL[6687.885488] change /devices/virtual/block/md0 (block)
KERNEL[6687.886975] remove /devices/virtual/bdi/9:0 (bdi)
UDEV [6687.888247] change /devices/virtual/block/md0 (block)
KERNEL[6687.888276] remove /devices/virtual/block/md0 (block)
UDEV [6687.889881] remove /devices/virtual/bdi/9:0 (bdi)
UDEV [6687.903134] change /devices/virtual/block/md0 (block)
UDEV [6687.915560] change /devices/virtual/block/md0 (block)
KERNEL[6687.919211] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6687.919838] add /devices/virtual/block/md0 (block)
UDEV [6687.920574] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6687.926955] change /devices/virtual/block/md0 (block)
UDEV [6687.926978] remove /devices/virtual/block/md0 (block)
KERNEL[6687.927010] change /devices/virtual/block/loop0 (block)
KERNEL[6687.927025] change /devices/virtual/block/loop1 (block)
KERNEL[6687.928120] change /devices/virtual/block/loop2 (block)
KERNEL[6687.928795] change /devices/virtual/block/loop3 (block)
KERNEL[6687.936185] change /devices/virtual/block/loop4 (block)
KERNEL[6687.936216] change /devices/virtual/block/loop5 (block)
KERNEL[6687.936243] change /devices/virtual/block/loop6 (block)
KERNEL[6687.936271] change /devices/virtual/block/loop2 (block)
UDEV [6687.937469] change /devices/virtual/block/loop0 (block)
KERNEL[6687.941864] change /devices/virtual/block/loop4 (block)
KERNEL[6687.943995] change /devices/virtual/block/loop5 (block)
UDEV [6687.953942] change /devices/virtual/block/loop6 (block)
UDEV [6687.958603] change /devices/virtual/block/loop1 (block)
KERNEL[6687.963136] change /devices/virtual/block/loop0 (block)
UDEV [6687.963160] change /devices/virtual/block/loop4 (block)
UDEV [6687.969767] change /devices/virtual/block/loop5 (block)
UDEV [6687.973885] change /devices/virtual/block/loop0 (block)
UDEV [6687.976070] change /devices/virtual/block/loop2 (block)
UDEV [6687.981657] change /devices/virtual/block/loop3 (block)
UDEV [6687.982961] change /devices/virtual/block/loop4 (block)
UDEV [6687.994403] change /devices/virtual/block/loop5 (block)
UDEV [6687.994436] change /devices/virtual/block/loop2 (block)
KERNEL[6688.012417] change /devices/virtual/block/md0 (block)
UDEV [6688.039546] add /devices/virtual/block/md0 (block)
UDEV [6688.074203] change /devices/virtual/block/md0 (block)
UDEV [6688.102564] change /devices/virtual/block/md0 (block)
KERNEL[6734.583172] change /devices/virtual/block/md0 (block)
UDEV [6735.326056] change /devices/virtual/block/md0 (block)
KERNEL[6735.342604] change /devices/virtual/block/md0 (block)
UDEV [6735.441686] change /devices/virtual/block/md0 (block)
KERNEL[6737.082876] change /devices/virtual/block/md0 (block)
UDEV [6737.679813] change /devices/virtual/block/md0 (block)
KERNEL[6738.181769] change /devices/virtual/block/md0 (block)
UDEV [6738.533050] change /devices/virtual/block/md0 (block)
KERNEL[6738.991194] change /devices/virtual/block/md0 (block)
UDEV [6739.179237] change /devices/virtual/block/md0 (block)
KERNEL[6739.299921] change /devices/virtual/block/md0 (block)
UDEV [6739.464539] change /devices/virtual/block/md0 (block)
KERNEL[6739.512987] change /devices/virtual/block/md0 (block)
UDEV [6739.685571] change /devices/virtual/block/md0 (block)
KERNEL[6739.767087] change /devices/virtual/block/md0 (block)
UDEV [6740.199527] change /devices/virtual/block/md0 (block)
KERNEL[6740.223356] change /devices/virtual/block/md0 (block)
UDEV [6740.348953] change /devices/virtual/block/md0 (block)
KERNEL[6740.722135] change /devices/virtual/block/md0 (block)
UDEV [6740.917654] change /devices/virtual/block/md0 (block)
KERNEL[6740.995449] change /devices/virtual/block/md0 (block)
UDEV [6741.027057] change /devices/virtual/block/md0 (block)
KERNEL[6741.176307] change /devices/virtual/block/md0 (block)
UDEV [6741.287848] change /devices/virtual/block/md0 (block)
KERNEL[6788.019597] change /devices/virtual/block/md0 (block)
UDEV [6788.054290] change /devices/virtual/block/md0 (block)
KERNEL[6788.075712] change /devices/virtual/block/md0 (block)
UDEV [6788.105865] change /devices/virtual/block/md0 (block)
KERNEL[6788.114089] change /devices/virtual/block/md0 (block)
KERNEL[6788.237174] change /devices/virtual/block/md0 (block)
KERNEL[6788.237429] change /devices/virtual/block/md0 (block)
KERNEL[6788.238775] remove /devices/virtual/bdi/9:0 (bdi)
UDEV [6788.239246] change /devices/virtual/block/md0 (block)
KERNEL[6788.240270] remove /devices/virtual/block/md0 (block)
UDEV [6788.241486] remove /devices/virtual/bdi/9:0 (bdi)
UDEV [6788.253115] change /devices/virtual/block/md0 (block)
UDEV [6788.263858] change /devices/virtual/block/md0 (block)
KERNEL[6788.267274] change /devices/virtual/block/loop0 (block)
UDEV [6788.271644] remove /devices/virtual/block/md0 (block)
UDEV [6788.282811] change /devices/virtual/block/loop0 (block)
KERNEL[6788.292175] change /devices/virtual/block/loop1 (block)
UDEV [6788.307988] change /devices/virtual/block/loop1 (block)
KERNEL[6788.325582] change /devices/virtual/block/loop2 (block)
UDEV [6788.340242] change /devices/virtual/block/loop2 (block)
KERNEL[6788.350532] change /devices/virtual/block/loop3 (block)
UDEV [6788.365899] change /devices/virtual/block/loop3 (block)
KERNEL[6788.375606] change /devices/virtual/block/loop4 (block)
UDEV [6788.389328] change /devices/virtual/block/loop4 (block)
KERNEL[6788.400577] change /devices/virtual/block/loop5 (block)
UDEV [6788.414682] change /devices/virtual/block/loop5 (block)
KERNEL[6788.425803] change /devices/virtual/block/loop6 (block)
UDEV [6788.440313] change /devices/virtual/block/loop6 (block)
KERNEL[6788.510976] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6788.511568] add /devices/virtual/block/md0 (block)
UDEV [6788.512115] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6788.513826] change /devices/virtual/block/loop0 (block)
UDEV [6788.525079] add /devices/virtual/block/md0 (block)
UDEV [6788.525435] change /devices/virtual/block/loop0 (block)
KERNEL[6788.558815] change /devices/virtual/block/loop0 (block)
KERNEL[6788.559677] change /devices/virtual/block/loop1 (block)
UDEV [6788.569728] change /devices/virtual/block/loop0 (block)
UDEV [6788.570191] change /devices/virtual/block/loop1 (block)
KERNEL[6788.600595] change /devices/virtual/block/loop1 (block)
KERNEL[6788.601480] change /devices/virtual/block/loop2 (block)
UDEV [6788.612566] change /devices/virtual/block/loop1 (block)
UDEV [6788.612629] change /devices/virtual/block/loop2 (block)
KERNEL[6788.650466] change /devices/virtual/block/loop2 (block)
KERNEL[6788.651371] change /devices/virtual/block/loop3 (block)
UDEV [6788.661917] change /devices/virtual/block/loop2 (block)
UDEV [6788.662598] change /devices/virtual/block/loop3 (block)
KERNEL[6788.700620] change /devices/virtual/block/loop3 (block)
KERNEL[6788.701543] change /devices/virtual/block/loop4 (block)
UDEV [6788.712727] change /devices/virtual/block/loop3 (block)
UDEV [6788.713231] change /devices/virtual/block/loop4 (block)
KERNEL[6788.750621] change /devices/virtual/block/loop4 (block)
KERNEL[6788.751586] change /devices/virtual/block/loop5 (block)
UDEV [6788.763155] change /devices/virtual/block/loop5 (block)
UDEV [6788.763308] change /devices/virtual/block/loop4 (block)
KERNEL[6788.800558] change /devices/virtual/block/loop5 (block)
KERNEL[6788.801416] change /devices/virtual/block/loop6 (block)
UDEV [6788.812158] change /devices/virtual/block/loop5 (block)
UDEV [6788.812471] change /devices/virtual/block/loop6 (block)
KERNEL[6788.842301] change /devices/virtual/block/loop6 (block)
KERNEL[6788.843113] change /devices/virtual/block/loop7 (block)
UDEV [6788.853459] change /devices/virtual/block/loop6 (block)
UDEV [6788.854106] change /devices/virtual/block/loop7 (block)
KERNEL[6788.883836] change /devices/virtual/block/loop7 (block)
UDEV [6788.894871] change /devices/virtual/block/loop7 (block)
KERNEL[6789.011564] change /devices/virtual/block/md0 (block)
UDEV [6789.065268] change /devices/virtual/block/md0 (block)
KERNEL[6884.248066] change /devices/virtual/block/md0 (block)
UDEV [6885.360337] change /devices/virtual/block/md0 (block)
KERNEL[6885.695050] change /devices/virtual/block/md0 (block)
UDEV [6888.854765] change /devices/virtual/block/md0 (block)
KERNEL[6889.225064] change /devices/virtual/block/md0 (block)
UDEV [6889.929240] change /devices/virtual/block/md0 (block)
KERNEL[6890.840079] change /devices/virtual/block/md0 (block)
UDEV [6891.820037] change /devices/virtual/block/md0 (block)
KERNEL[6893.293889] change /devices/virtual/block/md0 (block)
KERNEL[6893.302149] change /devices/virtual/block/md0 (block)
KERNEL[6893.302171] change /devices/virtual/block/md0 (block)
UDEV [6894.249836] change /devices/virtual/block/md0 (block)
UDEV [6894.283789] change /devices/virtual/block/md0 (block)
UDEV [6894.317230] change /devices/virtual/block/md0 (block)
KERNEL[6894.368336] change /devices/virtual/block/md0 (block)
UDEV [6896.026075] change /devices/virtual/block/md0 (block)
KERNEL[6896.045684] change /devices/virtual/block/md0 (block)
UDEV [6896.684701] change /devices/virtual/block/md0 (block)
KERNEL[6896.745566] change /devices/virtual/block/md0 (block)
UDEV [6896.998419] change /devices/virtual/block/md0 (block)
KERNEL[6926.410642] change /devices/virtual/block/md0 (block)
UDEV [6926.449836] change /devices/virtual/block/md0 (block)
KERNEL[6926.538568] change /devices/virtual/block/md0 (block)
KERNEL[6926.539464] change /devices/virtual/block/md0 (block)
KERNEL[6926.539641] change /devices/virtual/block/md0 (block)
UDEV [6926.577725] change /devices/virtual/block/md0 (block)
UDEV [6931.596644] change /devices/virtual/block/md0 (block)
UDEV [6931.628094] change /devices/virtual/block/md0 (block)
KERNEL[6933.881018] change /devices/virtual/block/md0 (block)
UDEV [6933.922429] change /devices/virtual/block/md0 (block)
KERNEL[6962.269802] change /devices/virtual/block/md0 (block)
KERNEL[6977.715345] change /devices/virtual/block/md0 (block)
KERNEL[6977.716614] change /devices/virtual/block/md0 (block)
UDEV [6977.717886] change /devices/virtual/block/md0 (block)
KERNEL[6977.719960] remove /devices/virtual/bdi/9:0 (bdi)
KERNEL[6977.722557] remove /devices/virtual/block/md0 (block)
UDEV [6977.723251] remove /devices/virtual/bdi/9:0 (bdi)
UDEV [6977.727476] change /devices/virtual/block/md0 (block)
UDEV [6977.737814] change /devices/virtual/block/md0 (block)
UDEV [6977.744947] remove /devices/virtual/block/md0 (block)
KERNEL[6977.751407] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6977.752291] add /devices/virtual/block/md0 (block)
UDEV [6977.752687] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[6977.755950] change /devices/virtual/block/md0 (block)
KERNEL[6977.756472] change /devices/virtual/block/loop0 (block)
KERNEL[6977.763863] change /devices/virtual/block/loop1 (block)
KERNEL[6977.764093] change /devices/virtual/block/loop2 (block)
UDEV [6977.766600] add /devices/virtual/block/md0 (block)
KERNEL[6977.766981] change /devices/virtual/block/loop3 (block)
KERNEL[6977.775461] change /devices/virtual/block/loop4 (block)
KERNEL[6977.775484] change /devices/virtual/block/loop5 (block)
KERNEL[6977.775503] change /devices/virtual/block/loop6 (block)
KERNEL[6977.775522] change /devices/virtual/block/loop7 (block)
UDEV [6977.781891] change /devices/virtual/block/loop0 (block)
UDEV [6977.797031] change /devices/virtual/block/loop1 (block)
UDEV [6977.798102] change /devices/virtual/block/loop3 (block)
UDEV [6977.800268] change /devices/virtual/block/loop2 (block)
UDEV [6977.802618] change /devices/virtual/block/loop7 (block)
UDEV [6977.807059] change /devices/virtual/block/loop5 (block)
UDEV [6977.807082] change /devices/virtual/block/md0 (block)
KERNEL[6977.818065] change /devices/virtual/block/loop7 (block)
UDEV [6977.820035] change /devices/virtual/block/loop6 (block)
KERNEL[6977.822047] change /devices/virtual/block/loop0 (block)
UDEV [6977.822965] change /devices/virtual/block/loop4 (block)
UDEV [6977.834246] change /devices/virtual/block/loop7 (block)
UDEV [6977.835720] change /devices/virtual/block/loop0 (block)
KERNEL[6977.939436] change /devices/virtual/block/md0 (block)
UDEV [6977.981041] change /devices/virtual/block/md0 (block)
KERNEL[7026.071351] change /devices/virtual/block/md0 (block)
UDEV [7026.984276] change /devices/virtual/block/md0 (block)
KERNEL[7027.751297] change /devices/virtual/block/md0 (block)
UDEV [7029.623042] change /devices/virtual/block/md0 (block)
KERNEL[7030.138935] change /devices/virtual/block/md0 (block)
UDEV [7031.286231] change /devices/virtual/block/md0 (block)
KERNEL[7031.429465] change /devices/virtual/block/md0 (block)
UDEV [7031.619533] change /devices/virtual/block/md0 (block)
KERNEL[7033.103921] change /devices/virtual/block/md0 (block)
UDEV [7034.158973] change /devices/virtual/block/md0 (block)
KERNEL[7034.955973] change /devices/virtual/block/md0 (block)
UDEV [7035.549463] change /devices/virtual/block/md0 (block)
KERNEL[7035.745108] change /devices/virtual/block/md0 (block)
UDEV [7036.201920] change /devices/virtual/block/md0 (block)
KERNEL[7036.816814] change /devices/virtual/block/md0 (block)
UDEV [7038.079945] change /devices/virtual/block/md0 (block)
KERNEL[7038.837269] change /devices/virtual/block/md0 (block)
UDEV [7040.239793] change /devices/virtual/block/md0 (block)
KERNEL[7040.334491] change /devices/virtual/block/md0 (block)
UDEV [7040.365864] change /devices/virtual/block/md0 (block)
KERNEL[7040.367420] change /devices/virtual/block/md0 (block)
UDEV [7040.422668] change /devices/virtual/block/md0 (block)
KERNEL[7041.462403] change /devices/virtual/block/md0 (block)
UDEV [7042.279653] change /devices/virtual/block/md0 (block)
KERNEL[7042.901076] change /devices/virtual/block/md0 (block)
UDEV [7042.972826] change /devices/virtual/block/md0 (block)
KERNEL[7082.239840] change /devices/virtual/block/md0 (block)
UDEV [7082.610794] change /devices/virtual/block/md0 (block)
KERNEL[7082.995887] change /devices/virtual/block/md0 (block)
KERNEL[7083.471178] change /devices/virtual/block/md0 (block)
KERNEL[7083.471420] change /devices/virtual/block/md0 (block)
KERNEL[7083.472801] remove /devices/virtual/bdi/9:0 (bdi)
KERNEL[7083.473626] remove /devices/virtual/block/md0 (block)
UDEV [7083.475440] remove /devices/virtual/bdi/9:0 (bdi)
KERNEL[7083.484140] add /devices/virtual/bdi/9:0 (bdi)
KERNEL[7083.484782] add /devices/virtual/block/md0 (block)
UDEV [7083.485216] add /devices/virtual/bdi/9:0 (bdi)
UDEV [7083.488284] change /devices/virtual/block/md0 (block)
UDEV [7083.499367] change /devices/virtual/block/md0 (block)
KERNEL[7083.508459] change /devices/virtual/block/loop0 (block)
UDEV [7083.509181] change /devices/virtual/block/md0 (block)
UDEV [7083.515422] remove /devices/virtual/block/md0 (block)
UDEV [7083.528049] change /devices/virtual/block/loop0 (block)
UDEV [7083.531441] add /devices/virtual/block/md0 (block)
KERNEL[7083.551131] change /devices/virtual/block/loop1 (block)
UDEV [7083.567953] change /devices/virtual/block/loop1 (block)
KERNEL[7083.591765] change /devices/virtual/block/loop2 (block)
UDEV [7083.625259] change /devices/virtual/block/loop2 (block)
KERNEL[7083.633389] change /devices/virtual/block/loop3 (block)
UDEV [7083.652052] change /devices/virtual/block/loop3 (block)
KERNEL[7083.666807] change /devices/virtual/block/loop4 (block)
UDEV [7083.686840] change /devices/virtual/block/loop4 (block)
KERNEL[7083.700199] change /devices/virtual/block/loop5 (block)
UDEV [7083.728246] change /devices/virtual/block/loop5 (block)
KERNEL[7083.741776] change /devices/virtual/block/loop6 (block)
UDEV [7083.763122] change /devices/virtual/block/loop6 (block)
KERNEL[7083.775143] change /devices/virtual/block/loop7 (block)
UDEV [7083.789863] change /devices/virtual/block/loop7 (block)
KERNEL[7083.861960] change /devices/virtual/block/loop0 (block)
UDEV [7083.871309] change /devices/virtual/block/loop0 (block)
KERNEL[7083.900120] change /devices/virtual/block/loop0 (block)
KERNEL[7083.900996] change /devices/virtual/block/loop1 (block)
UDEV [7083.911511] change /devices/virtual/block/loop0 (block)
UDEV [7083.912274] change /devices/virtual/block/loop1 (block)
KERNEL[7083.966790] change /devices/virtual/block/loop1 (block)
KERNEL[7083.967754] change /devices/virtual/block/loop2 (block)
UDEV [7083.979239] change /devices/virtual/block/loop1 (block)
UDEV [7083.979539] change /devices/virtual/block/loop2 (block)
KERNEL[7084.008355] change /devices/virtual/block/loop2 (block)
KERNEL[7084.009243] change /devices/virtual/block/loop3 (block)
UDEV [7084.020182] change /devices/virtual/block/loop2 (block)
UDEV [7084.020669] change /devices/virtual/block/loop3 (block)
KERNEL[7084.058466] change /devices/virtual/block/loop3 (block)
KERNEL[7084.059510] change /devices/virtual/block/loop4 (block)
UDEV [7084.070194] change /devices/virtual/block/loop3 (block)
UDEV [7084.071165] change /devices/virtual/block/loop4 (block)
KERNEL[7084.108374] change /devices/virtual/block/loop4 (block)
KERNEL[7084.109179] change /devices/virtual/block/loop5 (block)
UDEV [7084.120583] change /devices/virtual/block/loop5 (block)
UDEV [7084.120631] change /devices/virtual/block/loop4 (block)
KERNEL[7084.158432] change /devices/virtual/block/loop5 (block)
KERNEL[7084.15
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Roger Heflin @ 2015-03-27 0:04 UTC (permalink / raw)
To: Dave Stevens; +Cc: Linux RAID
In-Reply-To: <20150326165210.160752yq9xdwbd7u@webmail.uniserve.com>
you should probably do cat /proc/mdstat
It is likely that the livecd may have already tried to assemble it,
and that the busy is because it is already in use.
If it is already assembled and you want to redo it first you will need
to stop the assembled array to be able to redo it.
On Thu, Mar 26, 2015 at 6:52 PM, Dave Stevens <geek@uniserve.com> wrote:
> Hello list, Phil,
>
> I read your advice in the text below and it seems very reasonable. I want to
> do a test to see if the data is readable and this is what I did with the
> live distro:
>
> mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>
> and then mdadm said:
>
> mdadm: can not open device /dev/sdb2: device or resource busy
> mdadm: /dev/sdb2 has no superblock - assembly aborted
>
> I don't know if these errors are related - i.e., if there is no superblock
> is that the busy resource in the first message? And if the superblock is
> absent can I put it back or somehow recreate it?
>
> Dave
>
>
>
>
>
>
> From: Phil Turmel <philip@turmel.org>
> To: Dave Stevens <geek@uniserve.com>, linux-raid@vger.kernel.org
> Subject: Re: two raid issues
> Date: Sun, 08 Mar 2015 09:52:04 -0400
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101
> Thunderbird/31.5.0
>
> Good morning Dave,
>
> On 03/07/2015 05:42 PM, Dave Stevens wrote:
>>
>> Hello the raid list,
>>
>> I have inherited a server set up by people who are no longer around. It
>> worked fine until recently and then after a routine update refused to
>> boot. I've got the machine in my office and have been examining the
>> problem, or rather problems, I think there are two.
>
>
> Three, at least.
>
>> First, the bootable partition on /dev/sda1 won't successfully boot to a
>> xen kernel, kernel version is 2.6.18.something-xen. The intent is to
>> boot to a raid-10 array of four 750GB drives, each partitioned into a
>> small and a large partition as detailed below.
>>
>> Boot proceeds normally according to on-screen messages until this:
>>
>> md: md0: raid array is not clean -- starting background reconstruction
>> raid10: not enough operational mirrors for md0
>> md: pers -> () failed
>
>
> Yup. Degraded to the point of not running.
>
>> Immediately after these messages is another stating that an attempt has
>> been made to kill init, kernel panic and reboot.
>
>
> No way to pivot to your root filesystem, so your initramfs gives up.
> Depending on the distro, it may be possible to pass a kernel command
> line option to drop into a repair shell at that point.
>
>> I've tried to give these messages verbatim but have no way (I think) to
>> reproduce them other than manually.
>
>
> Repair shell, if available.
>
>> So at first I looked around, read through the wiki and found advice to
>> NOT write anything to the array, which seems reasonable. I looked at a
>> live microknoppix distro called runtime live that gave me a command
>> shell to run the examine command with the output below:
>
>
> LiveCD boot is good, and the following report is very detailed, thanks:
>
>> /dev/sda2:
>> Magic : a92b4efc
>> Version : 0.90.00
>> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>> Creation Time : Sun Nov 29 15:33:50 2009
>> Raid Level : raid10
>> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>> Raid Devices : 4
>> Total Devices : 3
>> Preferred Minor : 0
>>
>> Update Time : Wed Feb 18 19:28:22 2015
>> State : active
>> Active Devices : 2
>> Working Devices : 2
>> Failed Devices : 2
>> Spare Devices : 0
>> Checksum : 7c76593b - correct
>> Events : 32945477
>
>
> This is important: ^^^^^^^^
>
>> Layout : near=2
>> Chunk Size : 256K
>>
>> Number Major Minor RaidDevice State
>> this 0 8 2 0 active sync /dev/sda2
>>
>> 0 0 8 2 0 active sync /dev/sda2
>> 1 1 0 0 1 active sync
>> 2 2 8 34 2 active sync /dev/sdc2
>> 3 3 0 0 3 faulty removed
>> /dev/sdb2:
>> Magic : a92b4efc
>> Version : 0.90.00
>> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>> Creation Time : Sun Nov 29 15:33:50 2009
>> Raid Level : raid10
>> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>> Raid Devices : 4
>> Total Devices : 3
>> Preferred Minor : 0
>>
>> Update Time : Sat Nov 22 13:18:12 2014
>> State : clean
>> Active Devices : 3
>> Working Devices : 3
>> Failed Devices : 1
>> Spare Devices : 0
>> Checksum : 7d850ca4 - correct
>> Events : 32945477
>
>
> With this: ^^^^^^^^
>
>> Layout : near=2
>> Chunk Size : 256K
>>
>> Number Major Minor RaidDevice State
>> this 1 8 18 1 active sync /dev/sdb2
>>
>> 0 0 8 2 0 active sync /dev/sda2
>> 1 1 8 18 1 active sync /dev/sdb2
>> 2 2 8 34 2 active sync /dev/sdc2
>> 3 3 0 0 3 faulty removed
>> /dev/sdc2:
>> Magic : a92b4efc
>> Version : 0.90.00
>> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>> Creation Time : Sun Nov 29 15:33:50 2009
>> Raid Level : raid10
>> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>> Raid Devices : 4
>> Total Devices : 3
>> Preferred Minor : 0
>>
>> Update Time : Wed Feb 18 19:30:13 2015
>> State : active
>> Active Devices : 1
>> Working Devices : 1
>> Failed Devices : 2
>> Spare Devices : 0
>> Checksum : 7c7659de - correct
>> Events : 32945479
>
>
> And this: ^^^^^^^^
>
>> Layout : near=2
>> Chunk Size : 256K
>>
>> Number Major Minor RaidDevice State
>> this 2 8 34 2 active sync /dev/sdc2
>>
>> 0 0 0 0 0 removed
>> 1 1 0 0 1 faulty removed
>> 2 2 8 34 2 active sync /dev/sdc2
>> 3 3 0 0 3 faulty removed
>> /dev/sdd2:
>> Magic : a92b4efc
>> Version : 0.90.00
>> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>> Creation Time : Sun Nov 29 15:33:50 2009
>> Raid Level : raid10
>> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>> Raid Devices : 4
>> Total Devices : 5
>> Preferred Minor : 0
>>
>> Update Time : Wed Sep 4 07:51:50 2013
>> State : active
>> Active Devices : 4
>> Working Devices : 5
>> Failed Devices : 0
>> Spare Devices : 1
>> Checksum : 77c1a3a7 - correct
>> Events : 53
>
>
> Whoa! ^^^^
>
>> Layout : near=2
>> Chunk Size : 256K
>>
>> Number Major Minor RaidDevice State
>> this 3 8 50 3 active sync /dev/sdd2
>>
>> 0 0 8 2 0 active sync /dev/sda2
>> 1 1 8 18 1 active sync /dev/sdb2
>> 2 2 8 34 2 active sync /dev/sdc2
>> 3 3 8 50 3 active sync /dev/sdd2
>> 4 4 8 66 4 spare /dev/sde2
>> /dev/sde2:
>> Magic : a92b4efc
>> Version : 0.90.00
>> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
>> Creation Time : Sun Nov 29 15:33:50 2009
>> Raid Level : raid10
>> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
>> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
>> Raid Devices : 4
>> Total Devices : 5
>> Preferred Minor : 0
>>
>> Update Time : Sun Sep 8 13:25:42 2013
>> State : clean
>> Active Devices : 4
>> Working Devices : 4
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : 77c775be - correct
>> Events : 7934
>
>
> And Whoa again! ^^^^^^
>
>> Layout : near=2
>> Chunk Size : 256K
>>
>> Number Major Minor RaidDevice State
>> this 3 8 66 3 active sync /dev/sde2
>>
>> 0 0 8 2 0 active sync /dev/sda2
>> 1 1 8 18 1 active sync /dev/sdb2
>> 2 2 8 34 2 active sync /dev/sdc2
>> 3 3 8 66 3 active sync /dev/sde2
>>
>> This makes sense to me as far as it goes but I don't see what to do
>> next. As I understand it the four partitions from sda2 to sdd2 would
>> form the array with sde as hot spare. It has been my assumption that if
>> a drive failed that sde would sync and take over. I don't know if this
>> is in fact the case and don't see a path forward. Of course the backups
>> are inadequate.
>
>
> Based on the events and update timestamps, sdd died sometime around Wed
> Sep 4 07:51:50 2013, at which point sde stepped in. It too failed
> shortly after ~ Sun Sep 8 13:25:42 2013. You then ran degraded for over
> a year until sdb also failed ~ Sat Nov 22 13:18:12 2014. You were then
> running doubly-degraded (luckily on non-adjacent members) until this Feb
> 14 when sda was booted out. Leaving only one running drive.
>
> { I wouldn't keep such people around, either. }
>
> Your best bet is to force assembly of the last two working drives to get
> the system running, then take an immediate backup of all critical files.
> Do the forced assembly with the livecd, then do a clean shutdown. You
> should then be able to boot the original OS and take your backup.
>
> Then you need to completely rebuild your system with proper log
> monitoring, array monitoring, and verification of your drives.
>
> Phil
>
>
> --
> "As long as politics is the shadow cast on society by big business,
> the attenuation of the shadow will not change the substance."
>
> -- John Dewey
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* no good deed goes unpunished...
From: Dave Stevens @ 2015-03-26 23:52 UTC (permalink / raw)
To: linux-raid
Hello list, Phil,
I read your advice in the text below and it seems very reasonable. I
want to do a test to see if the data is readable and this is what I
did with the live distro:
mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
and then mdadm said:
mdadm: can not open device /dev/sdb2: device or resource busy
mdadm: /dev/sdb2 has no superblock - assembly aborted
I don't know if these errors are related - i.e., if there is no
superblock is that the busy resource in the first message? And if the
superblock is absent can I put it back or somehow recreate it?
Dave
From: Phil Turmel <philip@turmel.org>
To: Dave Stevens <geek@uniserve.com>, linux-raid@vger.kernel.org
Subject: Re: two raid issues
Date: Sun, 08 Mar 2015 09:52:04 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101
Thunderbird/31.5.0
Good morning Dave,
On 03/07/2015 05:42 PM, Dave Stevens wrote:
> Hello the raid list,
>
> I have inherited a server set up by people who are no longer around. It
> worked fine until recently and then after a routine update refused to
> boot. I've got the machine in my office and have been examining the
> problem, or rather problems, I think there are two.
Three, at least.
> First, the bootable partition on /dev/sda1 won't successfully boot to a
> xen kernel, kernel version is 2.6.18.something-xen. The intent is to
> boot to a raid-10 array of four 750GB drives, each partitioned into a
> small and a large partition as detailed below.
>
> Boot proceeds normally according to on-screen messages until this:
>
> md: md0: raid array is not clean -- starting background reconstruction
> raid10: not enough operational mirrors for md0
> md: pers -> () failed
Yup. Degraded to the point of not running.
> Immediately after these messages is another stating that an attempt has
> been made to kill init, kernel panic and reboot.
No way to pivot to your root filesystem, so your initramfs gives up.
Depending on the distro, it may be possible to pass a kernel command
line option to drop into a repair shell at that point.
> I've tried to give these messages verbatim but have no way (I think) to
> reproduce them other than manually.
Repair shell, if available.
> So at first I looked around, read through the wiki and found advice to
> NOT write anything to the array, which seems reasonable. I looked at a
> live microknoppix distro called runtime live that gave me a command
> shell to run the examine command with the output below:
LiveCD boot is good, and the following report is very detailed, thanks:
> /dev/sda2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Wed Feb 18 19:28:22 2015
> State : active
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 2
> Spare Devices : 0
> Checksum : 7c76593b - correct
> Events : 32945477
This is important: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 0 8 2 0 active sync /dev/sda2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 0 0 1 active sync
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdb2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Sat Nov 22 13:18:12 2014
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 7d850ca4 - correct
> Events : 32945477
With this: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 1 8 18 1 active sync /dev/sdb2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdc2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Wed Feb 18 19:30:13 2015
> State : active
> Active Devices : 1
> Working Devices : 1
> Failed Devices : 2
> Spare Devices : 0
> Checksum : 7c7659de - correct
> Events : 32945479
And this: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 2 8 34 2 active sync /dev/sdc2
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdd2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 5
> Preferred Minor : 0
>
> Update Time : Wed Sep 4 07:51:50 2013
> State : active
> Active Devices : 4
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 77c1a3a7 - correct
> Events : 53
Whoa! ^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 50 3 active sync /dev/sdd2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 50 3 active sync /dev/sdd2
> 4 4 8 66 4 spare /dev/sde2
> /dev/sde2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 5
> Preferred Minor : 0
>
> Update Time : Sun Sep 8 13:25:42 2013
> State : clean
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
> Checksum : 77c775be - correct
> Events : 7934
And Whoa again! ^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 66 3 active sync /dev/sde2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 66 3 active sync /dev/sde2
>
> This makes sense to me as far as it goes but I don't see what to do
> next. As I understand it the four partitions from sda2 to sdd2 would
> form the array with sde as hot spare. It has been my assumption that if
> a drive failed that sde would sync and take over. I don't know if this
> is in fact the case and don't see a path forward. Of course the backups
> are inadequate.
Based on the events and update timestamps, sdd died sometime around Wed
Sep 4 07:51:50 2013, at which point sde stepped in. It too failed
shortly after ~ Sun Sep 8 13:25:42 2013. You then ran degraded for over
a year until sdb also failed ~ Sat Nov 22 13:18:12 2014. You were then
running doubly-degraded (luckily on non-adjacent members) until this Feb
14 when sda was booted out. Leaving only one running drive.
{ I wouldn't keep such people around, either. }
Your best bet is to force assembly of the last two working drives to get
the system running, then take an immediate backup of all critical files.
Do the forced assembly with the livecd, then do a clean shutdown. You
should then be able to boot the original OS and take your backup.
Then you need to completely rebuild your system with proper log
monitoring, array monitoring, and verification of your drives.
Phil
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: /dev/md0 can't be created
From: Xiao Ni @ 2015-03-26 7:36 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150325173529.13a6efd7@notabene.brown>
Hi Neil
I can reproduce it now. Do you want me to run udevadm monitor before the test?
And I checked the /var/log/messages, it show the information:
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-2>
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-3>
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-4>
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-5>
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-6>
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-7>
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-9>
Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-8>
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-7 operational as raid disk 5
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-6 operational as raid disk 4
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-5 operational as raid disk 3
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-4 operational as raid disk 2
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-3 operational as raid disk 1
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-2 operational as raid disk 0
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: allocated 0kB
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: raid level 5 active with 6 out of 7 devices, algorithm 2
Oct 20 19:59:59 ibm-z10-25 kernel: md/raid456: discard support disabled due to uncertainty.
Oct 20 19:59:59 ibm-z10-25 kernel: Set raid456.devices_handle_discard_safely=Y to override.
Oct 20 19:59:59 ibm-z10-25 kernel: md0: detected capacity change from 0 to 1881145344
Oct 20 19:59:59 ibm-z10-25 kernel: md: recovery of RAID array md0
Oct 20 19:59:59 ibm-z10-25 kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Oct 20 19:59:59 ibm-z10-25 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Oct 20 19:59:59 ibm-z10-25 kernel: md: using 128k window, over a total of 306176k.
Oct 20 19:59:59 ibm-z10-25 systemd-udevd: inotify_add_watch(7, /dev/md0, 10) failed: No such file or directory
Xiao
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, March 25, 2015 2:35:29 PM
> Subject: Re: /dev/md0 can't be created
>
> On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> > Hi all
> >
> > I have encountered so many times, the raid device is created
> > successfully, but the directory
> > /dev/md0 can't be created. It can't reproduce 100%.
> >
> > [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> > Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3]
> > loop2[2] loop1[1] loop0[0]
> > 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]
> > bitmap: 0/1 pages [0KB], 65536KB chunk
> >
> > unused devices: <none>
> > [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> > ls: cannot access /dev/md0: No such file or directory
> >
> > The underline devices are loop devices which are created with big file.
> >
> > The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm - v3.3.2
> > - 21st August 2014)
> > I'll try to reproduce this with upstream kernel and mdadm. But I think it
> > shouldn't be the problem about kernel.
> >
> > What do you think I should check for this? And which tool is
> > responsible for creating the directory? Maybe
> > I can add some log to it to find the reason.
> >
>
> /dev/md0 is created by udev.
> Run
> udevadm monitor
>
> to see the events that udev is processing. When and ADD event for "md0" is
> processed, /dev/md0 should get created.
>
> NeilBrown
>
^ permalink raw reply
* Re: Badblocks and degraded array.
From: NeilBrown @ 2015-03-25 23:50 UTC (permalink / raw)
To: Wakko Warner; +Cc: linux-raid
In-Reply-To: <20150325231400.GA9242@animx.eu.org>
[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]
On Wed, 25 Mar 2015 19:14:00 -0400 Wakko Warner <wakko@animx.eu.org> wrote:
> Firstly, I'm not in need of assistance, just looking for information.
>
> I had a system with 4x 500gb disks in raid 5. One drive (slot 2) was
> kicked. I removed and reseated the drive (which is OK). During rebuild, it
> hit a bad block on another drive (slot 1) which it kicked. Is it possible
> that if there's no redundancy to not kick a drive if it has a bad block?
Only if you have bad-block-logs enabled. This is a relatively new feature.
>
> In the end, my solution was to create a dm target using linear and zero as
> needed (zero where the bad block was) then a snapshot target ontop of that
> since there was no possibility to write to that section that was a zero
> target. I had LVM on top of the raid and my /usr was the one in the bad
> block. Fortunately, no files were in that bad block. I dumped the usr
> volume elsewhere, removed all the mappings (md, dm, and lvm), assembled the
> array again and dumped the volume back which corrected the bad sector. All
> this was done using another installation.
>
> This system will be retired anyway so the data isn't really useful. But
> having the experience is.
>
> On a side note, it seems that everytime I encounter a bad sector on a drive,
> it's always 8 sectors. Does anyone know if hard drives have been 4k
> sectored longer than AF drives? This disk is 512 physical according to
> fdisk. I've even noticed this on IDE drives.
>
Linux tends to do IO in multiples of 4k so it is unlikely to report a smaller
block. That may or may not be relevant for your particular experiences.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Badblocks and degraded array.
From: Wakko Warner @ 2015-03-25 23:14 UTC (permalink / raw)
To: linux-raid
Firstly, I'm not in need of assistance, just looking for information.
I had a system with 4x 500gb disks in raid 5. One drive (slot 2) was
kicked. I removed and reseated the drive (which is OK). During rebuild, it
hit a bad block on another drive (slot 1) which it kicked. Is it possible
that if there's no redundancy to not kick a drive if it has a bad block?
In the end, my solution was to create a dm target using linear and zero as
needed (zero where the bad block was) then a snapshot target ontop of that
since there was no possibility to write to that section that was a zero
target. I had LVM on top of the raid and my /usr was the one in the bad
block. Fortunately, no files were in that bad block. I dumped the usr
volume elsewhere, removed all the mappings (md, dm, and lvm), assembled the
array again and dumped the volume back which corrected the bad sector. All
this was done using another installation.
This system will be retired anyway so the data isn't really useful. But
having the experience is.
On a side note, it seems that everytime I encounter a bad sector on a drive,
it's always 8 sectors. Does anyone know if hard drives have been 4k
sectored longer than AF drives? This disk is 512 physical according to
fdisk. I've even noticed this on IDE drives.
--
Microsoft has beaten Volkswagen's world record. Volkswagen only created 22
million bugs.
^ permalink raw reply
* [PATCH] Fix minor typo in mdadm manpage.
From: Andrew Burgess @ 2015-03-25 17:17 UTC (permalink / raw)
To: linux-raid; +Cc: Andrew Burgess
Appologies if this is the wrong mailing list for this patch.
This is a very small patch for the manual page for the mdadm utility.
Thanks,
Andrew
---
Change 'is and of' to 'is one of' in mdadm manual page.
Signed-off-by: Andrew Burgess <andrew.burgess@embecosm.com>
---
mdadm.8.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mdadm.8.in b/mdadm.8.in
index a630310..b4a21d9 100644
--- a/mdadm.8.in
+++ b/mdadm.8.in
@@ -214,7 +214,7 @@ to detect and assemble arrays \(em possibly in an
.P
If a device is given before any options, or if the first option is
-and of
+one of
.BR \-\-add ,
.BR \-\-re\-add ,
.BR \-\-add\-spare ,
--
2.2.2
^ permalink raw reply related
* Re: RAID6 won't reassemble after disk controller failure
From: Scott Sinno @ 2015-03-25 16:16 UTC (permalink / raw)
To: Mikael Abrahamsson
Cc: linux-raid,
Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
In-Reply-To: <alpine.DEB.2.02.1503251707440.20507@uplift.swm.pp.se>
Mikael Abrahamsson wrote:
> I will not make any more recommendations on how to proceed with this, it
> sounds like failing sdbt houldn't be needed and if I were you, I would
> assemble the array, keep it running until someone else who knows more
> can potentially help you with what might be wrong with your older
> version of mdadm (or the newer one might not do everything it should, or
> it's something your kernel is doing wrong).
>
Understood. Many thanks again for the assist!!! I'm certainly intent
on rolling out a newer version of mdadm now..will proceed on that
endeavor with prudent caution.
^ permalink raw reply
* Re: RAID6 won't reassemble after disk controller failure
From: Mikael Abrahamsson @ 2015-03-25 16:09 UTC (permalink / raw)
To: Scott Sinno
Cc: linux-raid,
Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
In-Reply-To: <5512D79B.704@sinno.net>
On Wed, 25 Mar 2015, Scott Sinno wrote:
> Mikael Abrahamsson wrote:
>> Ok, next thing to recommend would be to get the latest mdadm from Niels
>> git repo and try with that one. I know some who have fixed similar
>> problems to yours by doing this.
>>
>> "git clone git://neil.brown.name/mdadm mdadm" and compile that and see
>> if --assemble --force works.
>>
>
> That constitutes a huge step forward. Was able to assemble the array
> with all 12 drives using this binary! Thanks so much!!!
>
> However, after stopping and re-assembling the array, using the 'stock'
> CentOS version, it yields this complaint unless '--force' is also used.
> I'm inclined to believe I should simply fail /dev/sdbt and move on, yes?
I will not make any more recommendations on how to proceed with this, it
sounds like failing sdbt houldn't be needed and if I were you, I would
assemble the array, keep it running until someone else who knows more can
potentially help you with what might be wrong with your older version of
mdadm (or the newer one might not do everything it should, or it's
something your kernel is doing wrong).
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: RAID6 won't reassemble after disk controller failure
From: Scott Sinno @ 2015-03-25 15:43 UTC (permalink / raw)
To: Mikael Abrahamsson
Cc: linux-raid,
Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
In-Reply-To: <alpine.DEB.2.02.1503251628120.20507@uplift.swm.pp.se>
Mikael Abrahamsson wrote:
> Ok, next thing to recommend would be to get the latest mdadm from Niels
> git repo and try with that one. I know some who have fixed similar
> problems to yours by doing this.
>
> "git clone git://neil.brown.name/mdadm mdadm" and compile that and see
> if --assemble --force works.
>
That constitutes a huge step forward. Was able to assemble the array
with all 12 drives using this binary! Thanks so much!!!
However, after stopping and re-assembling the array, using the 'stock'
CentOS version, it yields this complaint unless '--force' is also used.
I'm inclined to believe I should simply fail /dev/sdbt and move on, yes?
[root@dsfs01 ~/mdadm]./mdadm -A /dev/md10
mdadm: ignoring /dev/sdbk as it reports /dev/sdbt as failed
mdadm: ignoring /dev/sdbl as it reports /dev/sdbt as failed
mdadm: ignoring /dev/sdbm as it reports /dev/sdbt as failed
mdadm: /dev/md10 assembled from 9 drives - not enough to start the array.
[root@dsfs01 ~/mdadm]mdadm
[root@dsfs01 ~/mdadm]./mdadm -A --force /dev/md10
mdadm: /dev/md10 has been started with 12 drives.
[root@dsfs01 ~/mdadm]mdadm -S /dev/md10
mdadm: stopped /dev/md10
[root@dsfs01 ~/mdadm]mdadm -A /dev/md10
mdadm: ignoring /dev/sdbk as it reports /dev/sdbt as failed
mdadm: ignoring /dev/sdbl as it reports /dev/sdbt as failed
mdadm: ignoring /dev/sdbm as it reports /dev/sdbt as failed
mdadm: /dev/md10 assembled from 9 drives - not enough to start the array.
[root@dsfs01 ~/mdadm]mdadm -A --force /dev/md10
mdadm: /dev/md10 has been started with 12 drives.
^ permalink raw reply
* Re: RAID6 won't reassemble after disk controller failure
From: Mikael Abrahamsson @ 2015-03-25 15:30 UTC (permalink / raw)
To: Scott Sinno
Cc: linux-raid,
Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
In-Reply-To: <5512D3AA.5000003@sinno.net>
On Wed, 25 Mar 2015, Scott Sinno wrote:
> It doesn't appear to be very telling. Note that per the provided
> '-E'(examine) output, event count is off a bit on some of the drives,
> and several report 'missing'
Ok, next thing to recommend would be to get the latest mdadm from Niels
git repo and try with that one. I know some who have fixed similar
problems to yours by doing this.
"git clone git://neil.brown.name/mdadm mdadm" and compile that and see if
--assemble --force works.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: RAID6 won't reassemble after disk controller failure
From: Scott Sinno @ 2015-03-25 15:29 UTC (permalink / raw)
To: Mikael Abrahamsson
Cc: linux-raid,
Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
In-Reply-To: <5512D3AA.5000003@sinno.net>
Scott Sinno wrote:
> It doesn't appear to be very telling. Note that per the provided
> '-E'(examine) output, event count is off a bit on some of the drives,
> and several report 'missing'
Clarification - the drives themselves are not 'missing', they're healthy
and present, 'mdadm -E' reports some components missing in the 'Array
State' field.
^ permalink raw reply
* Re: RAID6 won't reassemble after disk controller failure
From: Scott Sinno @ 2015-03-25 15:26 UTC (permalink / raw)
To: Mikael Abrahamsson
Cc: linux-raid,
Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
In-Reply-To: <alpine.DEB.2.02.1503251621390.20507@uplift.swm.pp.se>
Mikael Abrahamsson wrote:
> I would recommend you to not do that unless there is no other way out.
Wholeheartedly agreed, thus looking to this list for guidance first :-)
>
> Please post dmesg output from the --assemble --force attempt, it might
> contain reason why the assembly was unsuccessful.
>
It doesn't appear to be very telling. Note that per the provided
'-E'(examine) output, event count is off a bit on some of the drives,
and several report 'missing'
#dmesg output
md: bind<sdbk>
md: bind<sdbl>
md: bind<sdbm>
md: bind<sdbn>
md: bind<sdbo>
md: bind<sdbp>
md: bind<sdbq>
md: bind<sdbi>
md: bind<sdbr>
md: bind<sdbs>
md: bind<sdbt>
md: bind<sdbj>
md: md10 stopped.
md: unbind<sdbj>
md: export_rdev(sdbj)
md: unbind<sdbt>
md: export_rdev(sdbt)
md: unbind<sdbs>
md: export_rdev(sdbs)
md: unbind<sdbr>
md: export_rdev(sdbr)
md: unbind<sdbi>
md: export_rdev(sdbi)
md: unbind<sdbq>
md: export_rdev(sdbq)
md: unbind<sdbp>
md: export_rdev(sdbp)
md: unbind<sdbo>
md: export_rdev(sdbo)
md: unbind<sdbn>
md: export_rdev(sdbn)
md: unbind<sdbm>
md: export_rdev(sdbm)
md: unbind<sdbl>
md: export_rdev(sdbl)
md: unbind<sdbk>
md: export_rdev(sdbk)
md: md10 stopped.
md: bind<sdbk>
md: bind<sdbl>
md: bind<sdbm>
md: bind<sdbn>
md: bind<sdbo>
md: bind<sdbp>
md: bind<sdbq>
md: bind<sdbi>
md: bind<sdbr>
md: bind<sdbs>
md: bind<sdbt>
md: bind<sdbj>
md: md10 stopped.
md: unbind<sdbj>
md: export_rdev(sdbj)
md: unbind<sdbt>
md: export_rdev(sdbt)
md: unbind<sdbs>
md: export_rdev(sdbs)
md: unbind<sdbr>
md: export_rdev(sdbr)
md: unbind<sdbi>
md: export_rdev(sdbi)
md: unbind<sdbq>
md: export_rdev(sdbq)
md: unbind<sdbp>
md: export_rdev(sdbp)
md: unbind<sdbo>
md: export_rdev(sdbo)
md: unbind<sdbn>
md: export_rdev(sdbn)
md: unbind<sdbm>
md: export_rdev(sdbm)
md: unbind<sdbl>
md: export_rdev(sdbl)
md: unbind<sdbk>
md: export_rdev(sdbk)
^ permalink raw reply
* Re: RAID6 won't reassemble after disk controller failure
From: Mikael Abrahamsson @ 2015-03-25 15:22 UTC (permalink / raw)
To: Scott Sinno
Cc: linux-raid,
Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
In-Reply-To: <5512D21E.1010506@sinno.net>
On Wed, 25 Mar 2015, Scott Sinno wrote:
> JBOD controller failed this morning - we've replaced it, and all
> drives are being presented to the system again, however 'mdadm' refuses
> to assemble the ARRAY. I've tried assembling with '--force' flags but
> it still won't...any help most appreciated. I'm guessing I'll need to
> recreate with '--assemble-clean'
I would recommend you to not do that unless there is no other way out.
Please post dmesg output from the --assemble --force attempt, it might
contain reason why the assembly was unsuccessful.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* RAID6 won't reassemble after disk controller failure
From: Scott Sinno @ 2015-03-25 15:19 UTC (permalink / raw)
To: linux-raid
Cc: Vaughan, Garrison (GSFC-606.2)[INTERNATIONAL TECHNOLOGY COALITION, INC]
JBOD controller failed this morning - we've replaced it, and all
drives are being presented to the system again, however 'mdadm' refuses
to assemble the ARRAY. I've tried assembling with '--force' flags but
it still won't...any help most appreciated. I'm guessing I'll need to
recreate with '--assemble-clean'
[root@dsfs01 ~]cat /etc/redhat-release
CentOS release 6.6 (Final)
[root@dsfs01 ~]mdadm --version
mdadm - v3.3 - 3rd September 2013
[root@dsfs01 ~]uname -r
3.14.33
[root@dsfs01 ~]mdadm -A --force --verbose /dev/md10 /dev/sdbj /dev/sdbk
/dev/sdbl /dev/sdbm /dev/sdbn /dev/sdbo /dev/sdbp /dev/sdbq /dev/sdbi
/dev/sdbr /dev/sdbs /dev/sdbt
mdadm: looking for devices for /dev/md10
mdadm: /dev/sdbj is identified as a member of /dev/md10, slot 0.
mdadm: /dev/sdbk is identified as a member of /dev/md10, slot 1.
mdadm: /dev/sdbl is identified as a member of /dev/md10, slot 2.
mdadm: /dev/sdbm is identified as a member of /dev/md10, slot 3.
mdadm: /dev/sdbn is identified as a member of /dev/md10, slot 4.
mdadm: /dev/sdbo is identified as a member of /dev/md10, slot 5.
mdadm: /dev/sdbp is identified as a member of /dev/md10, slot 6.
mdadm: /dev/sdbq is identified as a member of /dev/md10, slot 7.
mdadm: /dev/sdbi is identified as a member of /dev/md10, slot 8.
mdadm: /dev/sdbr is identified as a member of /dev/md10, slot 9.
mdadm: /dev/sdbs is identified as a member of /dev/md10, slot 10.
mdadm: /dev/sdbt is identified as a member of /dev/md10, slot 11.
mdadm: added /dev/sdbk to /dev/md10 as 1
mdadm: added /dev/sdbl to /dev/md10 as 2
mdadm: added /dev/sdbm to /dev/md10 as 3
mdadm: added /dev/sdbn to /dev/md10 as 4
mdadm: added /dev/sdbo to /dev/md10 as 5
mdadm: added /dev/sdbp to /dev/md10 as 6 (possibly out of date)
mdadm: added /dev/sdbq to /dev/md10 as 7 (possibly out of date)
mdadm: added /dev/sdbi to /dev/md10 as 8 (possibly out of date)
mdadm: added /dev/sdbr to /dev/md10 as 9 (possibly out of date)
mdadm: added /dev/sdbs to /dev/md10 as 10 (possibly out of date)
mdadm: added /dev/sdbt to /dev/md10 as 11 (possibly out of date)
mdadm: added /dev/sdbj to /dev/md10 as 0
mdadm -E /dev/sdbj
/dev/sdbj:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : clean
Device UUID : 3bd0e7f2:cfc24add:122a7354:049efe95
Update Time : Wed Mar 25 08:21:34 2015
Checksum : 4a952f6a - correct
Events : 7641
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 0
Array State : AAAAAA...... ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbk
/dev/sdbk:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : clean
Device UUID : 4ebd5ec0:24367c71:bd5d3b52:3910ebcb
Update Time : Wed Mar 25 08:21:34 2015
Checksum : dfec35ad - correct
Events : 7641
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 1
Array State : AAAA........ ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbl
/dev/sdbl:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : clean
Device UUID : 8cf70b24:9cd55339:08f408d7:7719469b
Update Time : Wed Mar 25 08:21:34 2015
Checksum : 5f99aeed - correct
Events : 7641
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 2
Array State : AAAA........ ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbm
/dev/sdbm:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : clean
Device UUID : 59816854:0a6eb50b:b8cb756e:497c2d77
Update Time : Wed Mar 25 08:21:34 2015
Checksum : d5ac0baa - correct
Events : 7641
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 3
Array State : AAAA........ ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbn
/dev/sdbn:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : 52c08937:e782b635:05d7f31c:7e7fa2ae
Update Time : Wed Mar 25 08:21:30 2015
Checksum : c8e56e1f - correct
Events : 7641
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 4
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbo
/dev/sdbo:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : 2902c7b0:e0e0b31e:70e94404:1d8f10c4
Update Time : Wed Mar 25 08:21:30 2015
Checksum : 27df2ffb - correct
Events : 7641
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 5
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbp
/dev/sdbp:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : d83bf256:a2f59330:e3209784:ee201d00
Update Time : Wed Mar 25 08:21:30 2015
Checksum : 9c4947ad - correct
Events : 7638
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 6
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbq
/dev/sdbq:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : d2556b61:a6fa2608:7ad69e55:ff09a3f7
Update Time : Wed Mar 25 08:21:30 2015
Checksum : 46e30555 - correct
Events : 7638
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 7
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbi
/dev/sdbi:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : 4a19a417:b2c5909e:496d5266:23228c19
Update Time : Wed Mar 25 08:21:30 2015
Checksum : c62242cc - correct
Events : 7638
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 8
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbr
/dev/sdbr:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : 1074ca9b:d9db5522:17055a3c:b6e69869
Update Time : Wed Mar 25 08:21:30 2015
Checksum : f422101b - correct
Events : 7638
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 9
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbs
/dev/sdbs:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : 13afb79f:7d24a231:b5e165ab:6faff1c7
Update Time : Wed Mar 25 08:21:30 2015
Checksum : d4c0391b - correct
Events : 7638
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 10
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
mdadm -E /dev/sdbt
/dev/sdbt:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 903cfc6d:921b0a84:c0b4bc7e:27e28c86
Name : dsfs01:10 (local to host dsfs01)
Creation Time : Tue Nov 5 03:37:14 2013
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 39068871680 (37258.98 GiB 40006.52 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=688 sectors
State : active
Device UUID : 3cc36ae5:6965ba18:d5ed98bb:ec757657
Update Time : Wed Mar 25 08:21:30 2015
Checksum : a14360ce - correct
Events : 7638
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 11
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
^ permalink raw reply
* Re: missing arrays after OS upgrade
From: Phil Turmel @ 2015-03-25 12:15 UTC (permalink / raw)
To: Daniel Sanabria; +Cc: linux-raid
In-Reply-To: <CAHscji1spYx8vvUQ2=PsU4VRCb4+kOqAstJqyK0SbYJ6=OSYiQ@mail.gmail.com>
Hi Daniel,
{Convention on kernel.org is to trim quotes, then interleave replies or
bottom-post.}
On 03/25/2015 07:49 AM, Daniel Sanabria wrote:
> Thank you very much Phil,
>
> Below the output of lsdrv followed by /proc/mdstat and mdadm.conf
> (will clean up dmesg a bit before attaching it here):
When you do, turn off line wrap. I forgot to suggest that.
> PCI [ahci] 07:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02)
> ├scsi 0:x:x:x [Empty]
> └scsi 1:x:x:x [Empty]
> PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
> ├scsi 2:0:0:0 ATA WDC WD5000AAKS-0 {WD-WMAWF0085724}
> │└sda 465.76g [8:0] Partitioned (dos)
> │ ├sda1 199.00m [8:1] ext4 {4e51f903-37ca-4479-9197-fac7b2280557}
> │ │└Mounted as /dev/sda1 @ /boot
> │ ├sda2 29.30g [8:2] MD raid10,near2 (0/2) (w/ sdc1) in_sync {9af006ca-8845-bbd3-bfe7-8010bc810f04}
> │ │└md126 29.30g [9:126] MD v0.90 raid10,near2 (2) clean, 64k Chunk {9af006ca:8845bbd3:bfe78010:bc810f04}
> │ │ │ PV LVM2_member 28.03g used, 1.26g free {cE4ePh-RWO8-Wgdy-YPOY-ehyC-KI6u-io1cyH}
> │ │ └VG vg_bigblackbox 29.29g 1.26g free {VWfuwI-5v2q-w8qf-FEbc-BdGW-3mKX-pZd7hR}
> │ │ ├dm-6 7.81g [253:6] LV LogVol_opt ext4 {b08d7f5e-f15f-4241-804e-edccecab6003}
> │ │ │└Mounted as /dev/mapper/vg_bigblackbox-LogVol_opt @ /opt
> │ │ ├dm-0 9.77g [253:0] LV LogVol_root ext4 {4dabd6b0-b1a3-464d-8ed7-0aab93fab6c3}
> │ │ │└Mounted as /dev/mapper/vg_bigblackbox-LogVol_root @ /
> │ │ ├dm-8 1.95g [253:8] LV LogVol_tmp ext4 {f6b46363-170b-4038-83bd-2c5f9f6a1973}
> │ │ │└Mounted as /dev/mapper/vg_bigblackbox-LogVol_tmp @ /tmp
> │ │ └dm-4 8.50g [253:4] LV LogVol_var ext4 {ab165c61-3d62-4c55-8639-6c2c2bf4b021}
> │ │ └Mounted as /dev/mapper/vg_bigblackbox-LogVol_var @ /var
> │ ├sda3 244.14g [8:3] MD raid5 (3) inactive {2cff15d1-e411-447b-fd5d-472103e44022}
> │ ├sda4 1.00k [8:4] Partitioned (dos)
> │ ├sda5 30.00g [8:5] MD raid0 (0/3) (w/ sdb5,sdc5) in_sync 'reading.homeunix.com:3' {acd5374f-7262-8c93-6a90-6c4b5f675ce5}
> │ │└md124 90.00g [9:124] MD v1.2 raid0 (3) clean, 512k Chunk, None (None) None {acd5374f:72628c93:6a906c4b:5f675ce5}
> │ │ │ PV LVM2_member 86.00g used, 3.99g free {VmsWRd-8qHt-bauf-lvAn-FC97-KyH5-gk89ox}
> │ │ └VG libvirt_lvm 89.99g 3.99g free {t8GQck-f2Eu-iD2V-fnJQ-kBm6-QyKw-dR31PB}
> │ │ ├dm-9 8.00g [253:9] LV builder Partitioned (dos)
> │ │ ├dm-10 8.00g [253:10] LV builder2 Partitioned (dos)
> │ │ ├dm-11 8.00g [253:11] LV builder3 Partitioned (dos)
> │ │ ├dm-13 8.00g [253:13] LV builder5.3 Partitioned (dos)
> │ │ ├dm-12 8.00g [253:12] LV builder5.6 Partitioned (dos)
> │ │ ├dm-5 8.00g [253:5] LV centos_updt Partitioned (dos)
> │ │ ├dm-7 8.00g [253:7] LV cms Partitioned (dos)
> │ │ └dm-2 30.00g [253:2] LV win7 Partitioned (dos)
> │ └sda6 3.39g [8:6] Empty/Unknown
> ├scsi 2:0:1:0 ATA WDC WD5000AAKS-0 {WD-WCASY7694185}
> │└sdb 465.76g [8:16] Partitioned (dos)
> │ ├sdb2 244.14g [8:18] MD raid5 (3) inactive {2cff15d1-e411-447b-fd5d-472103e44022}
> │ ├sdb3 7.81g [8:19] swap {9194f492-881a-4fc3-ac09-ca4e1cc2985a}
> │ ├sdb4 1.00k [8:20] Partitioned (dos)
> │ ├sdb5 30.00g [8:21] MD raid0 (1/3) (w/ sda5,sdc5) in_sync 'reading.homeunix.com:3' {acd5374f-7262-8c93-6a90-6c4b5f675ce5}
> │ │└md124 90.00g [9:124] MD v1.2 raid0 (3) clean, 512k Chunk, None (None) None {acd5374f:72628c93:6a906c4b:5f675ce5}
> │ │ PV LVM2_member 86.00g used, 3.99g free {VmsWRd-8qHt-bauf-lvAn-FC97-KyH5-gk89ox}
> │ └sdb6 3.39g [8:22] Empty/Unknown
> ├scsi 3:0:0:0 ATA WDC WD5000AAKS-0 {WD-WCASZ0505379}
> │└sdc 465.76g [8:32] Partitioned (dos)
> │ ├sdc1 29.30g [8:33] MD raid10,near2 (1/2) (w/ sda2) in_sync {9af006ca-8845-bbd3-bfe7-8010bc810f04}
> │ │└md126 29.30g [9:126] MD v0.90 raid10,near2 (2) clean, 64k Chunk {9af006ca:8845bbd3:bfe78010:bc810f04}
> │ │ PV LVM2_member 28.03g used, 1.26g free {cE4ePh-RWO8-Wgdy-YPOY-ehyC-KI6u-io1cyH}
> │ ├sdc2 244.14g [8:34] MD raid5 (3) inactive {2cff15d1-e411-447b-fd5d-472103e44022}
> │ ├sdc3 1.00k [8:35] Partitioned (dos)
> │ ├sdc5 30.00g [8:37] MD raid0 (2/3) (w/ sda5,sdb5) in_sync 'reading.homeunix.com:3' {acd5374f-7262-8c93-6a90-6c4b5f675ce5}
> │ │└md124 90.00g [9:124] MD v1.2 raid0 (3) clean, 512k Chunk, None (None) None {acd5374f:72628c93:6a906c4b:5f675ce5}
> │ │ PV LVM2_member 86.00g used, 3.99g free {VmsWRd-8qHt-bauf-lvAn-FC97-KyH5-gk89ox}
> │ └sdc6 3.39g [8:38] Empty/Unknown
> └scsi 3:0:1:0 ATA WDC WD30EZRX-00D {WD-WCC4N1294906}
> └sdd 2.73t [8:48] Partitioned (gpt)
> ├sdd1 2.00t [8:49] MD raid5 (0/3) (w/ sde1,sdf1) in_sync 'lamachine:128' {f2372cb9-d381-6fd6-ce86-d826882ec82e}
> │└md127 4.00t [9:127] MD v1.2 raid5 (3) read-auto, 512k Chunk {f2372cb9:d3816fd6:ce86d826:882ec82e}
> │ │ PV LVM2_member 3.50t used, 511.75g free {NrF3vV-nS1f-wtjY-SfZk-xwUE-YbSB-TWWdLu}
> │ └VG vg_media 4.00t 511.75g free {cPGpZK-OdZ6-sW71-8Rtf-x23f-BgEU-WGEmid}
> │ └dm-1 3.50t [253:1] LV lv_media ext4 {ef90c386-76e6-4843-a427-88412948d49f}
> └sdd2 500.00g [8:50] MD raid0 (0/3) (w/ sde2,sdf2) in_sync 'lamachine:129' {895dae98-d1a4-96de-4f59-0b8bcb8ac12a}
> └md125 1.46t [9:125] MD v1.2 raid0 (3) clean, 512k Chunk, None (None) None {895dae98:d1a496de:4f590b8b:cb8ac12a}
> │ PV LVM2_member 512.00g used, 987.62g free {70aHF7-aTfj-X2y6-qYXF-szQu-QBFW-q4UlfG}
> └VG vg_virt_dir 1.46t 987.62g free {SQXqhD-j5dR-Xsdh-NBoR-YMqk-YwfF-hyf9ur}
> └dm-3 512.00g [253:3] LV lv_virt_dir1 ext4 {1929e452-ae83-4766-bf0e-8575a7d1a3c8}
> PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
> ├scsi 4:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4NCWT13RF}
> │└sde 2.73t [8:64] Partitioned (gpt)
> │ ├sde1 2.00t [8:65] MD raid5 (1/3) (w/ sdd1,sdf1) in_sync 'lamachine:128' {f2372cb9-d381-6fd6-ce86-d826882ec82e}
> │ │└md127 4.00t [9:127] MD v1.2 raid5 (3) read-auto, 512k Chunk {f2372cb9:d3816fd6:ce86d826:882ec82e}
> │ │ PV LVM2_member 3.50t used, 511.75g free {NrF3vV-nS1f-wtjY-SfZk-xwUE-YbSB-TWWdLu}
> │ └sde2 500.00g [8:66] MD raid0 (1/3) (w/ sdd2,sdf2) in_sync 'lamachine:129' {895dae98-d1a4-96de-4f59-0b8bcb8ac12a}
> │ └md125 1.46t [9:125] MD v1.2 raid0 (3) clean, 512k Chunk, None (None) None {895dae98:d1a496de:4f590b8b:cb8ac12a}
> │ PV LVM2_member 512.00g used, 987.62g free {70aHF7-aTfj-X2y6-qYXF-szQu-QBFW-q4UlfG}
> └scsi 5:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4NPRDD6D7}
> └sdf 2.73t [8:80] Partitioned (gpt)
> ├sdf1 2.00t [8:81] MD raid5 (2/3) (w/ sdd1,sde1) in_sync 'lamachine:128' {f2372cb9-d381-6fd6-ce86-d826882ec82e}
> │└md127 4.00t [9:127] MD v1.2 raid5 (3) read-auto, 512k Chunk {f2372cb9:d3816fd6:ce86d826:882ec82e}
> │ PV LVM2_member 3.50t used, 511.75g free {NrF3vV-nS1f-wtjY-SfZk-xwUE-YbSB-TWWdLu}
> └sdf2 500.00g [8:82] MD raid0 (2/3) (w/ sdd2,sde2) in_sync 'lamachine:129' {895dae98-d1a4-96de-4f59-0b8bcb8ac12a}
> └md125 1.46t [9:125] MD v1.2 raid0 (3) clean, 512k Chunk, None (None) None {895dae98:d1a496de:4f590b8b:cb8ac12a}
> PV LVM2_member 512.00g used, 987.62g free {70aHF7-aTfj-X2y6-qYXF-szQu-QBFW-q4UlfG}
> PCI [pata_jmicron] 07:00.1 IDE interface: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02)
> ├scsi 6:x:x:x [Empty]
> └scsi 7:x:x:x [Empty]
>
> [root@localhost ~]# cat /proc/mdstat
> Personalities : [raid10] [raid0] [raid6] [raid5] [raid4]
> md124 : active raid0 sda5[0] sdb5[1] sdc5[2]
> 94367232 blocks super 1.2 512k chunks
>
> md125 : active raid0 sdf2[2] sdd2[0] sde2[1]
> 1572470784 blocks super 1.2 512k chunks
>
> md127 : active (auto-read-only) raid5 sdf1[3] sde1[1] sdd1[0]
> 4294705152 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
> bitmap: 0/16 pages [0KB], 65536KB chunk
>
> md126 : active raid10 sda2[0] sdc1[1]
> 30719936 blocks 2 near-copies [2/2] [UU]
>
> unused devices: <none>
> [root@localhost ~]# cat /etc/mdadm.conf
> # mdadm.conf written out by anaconda
> MAILADDR root
> AUTO +imsm +1.x -all
> ARRAY /dev/md/126 level=raid10 num-devices=2 UUID=9af006ca:8845bbd3:bfe78010:bc810f04
Ok. Your array didn't assemble because mdadm.conf said not to. The
AUTO clause is excluding unidentified v0.90 metadata. The UUID
....:bc810f04 was called out explicitly, so it assembled.
You should be safe to just assemble the array normally:
mdadm -A /dev/md2 /dev/sd{a3,b2,c2}
Then you need to create a new mdadm.conf. I suggest you use "AUTO -all"
and explicitly list all of your arrays. You only need the device name
and the UUID on each ARRAY line. You can run "mdadm -Es
>>..../mdadm.conf" to add live arrays to the file, then trim out the
unnecessary details.
When mdadm.conf has your arrays, with the names you want them to keep,
update your initramfs. (Command varies by distro.)
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: missing arrays after OS upgrade
From: Daniel Sanabria @ 2015-03-25 11:49 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <5511FB3B.8030607@turmel.org>
Thank you very much Phil,
Below the output of lsdrv followed by /proc/mdstat and mdadm.conf
(will clean up dmesg a bit before attaching it here):
Again, thanks for your input on this,
Daniel
PCI [ahci] 07:00.0 SATA controller: JMicron Technology Corp. JMB363
SATA/IDE Controller (rev 02)
├scsi 0:x:x:x [Empty]
└scsi 1:x:x:x [Empty]
PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10
Family) 4 port SATA IDE Controller #1
├scsi 2:0:0:0 ATA WDC WD5000AAKS-0 {WD-WMAWF0085724}
│└sda 465.76g [8:0] Partitioned (dos)
│ ├sda1 199.00m [8:1] ext4 {4e51f903-37ca-4479-9197-fac7b2280557}
│ │└Mounted as /dev/sda1 @ /boot
│ ├sda2 29.30g [8:2] MD raid10,near2 (0/2) (w/ sdc1) in_sync
{9af006ca-8845-bbd3-bfe7-8010bc810f04}
│ │└md126 29.30g [9:126] MD v0.90 raid10,near2 (2) clean, 64k Chunk
{9af006ca:8845bbd3:bfe78010:bc810f04}
│ │ │ PV LVM2_member 28.03g used, 1.26g free
{cE4ePh-RWO8-Wgdy-YPOY-ehyC-KI6u-io1cyH}
│ │ └VG vg_bigblackbox 29.29g 1.26g free
{VWfuwI-5v2q-w8qf-FEbc-BdGW-3mKX-pZd7hR}
│ │ ├dm-6 7.81g [253:6] LV LogVol_opt ext4
{b08d7f5e-f15f-4241-804e-edccecab6003}
│ │ │└Mounted as /dev/mapper/vg_bigblackbox-LogVol_opt @ /opt
│ │ ├dm-0 9.77g [253:0] LV LogVol_root ext4
{4dabd6b0-b1a3-464d-8ed7-0aab93fab6c3}
│ │ │└Mounted as /dev/mapper/vg_bigblackbox-LogVol_root @ /
│ │ ├dm-8 1.95g [253:8] LV LogVol_tmp ext4
{f6b46363-170b-4038-83bd-2c5f9f6a1973}
│ │ │└Mounted as /dev/mapper/vg_bigblackbox-LogVol_tmp @ /tmp
│ │ └dm-4 8.50g [253:4] LV LogVol_var ext4
{ab165c61-3d62-4c55-8639-6c2c2bf4b021}
│ │ └Mounted as /dev/mapper/vg_bigblackbox-LogVol_var @ /var
│ ├sda3 244.14g [8:3] MD raid5 (3) inactive
{2cff15d1-e411-447b-fd5d-472103e44022}
│ ├sda4 1.00k [8:4] Partitioned (dos)
│ ├sda5 30.00g [8:5] MD raid0 (0/3) (w/ sdb5,sdc5) in_sync
'reading.homeunix.com:3' {acd5374f-7262-8c93-6a90-6c4b5f675ce5}
│ │└md124 90.00g [9:124] MD v1.2 raid0 (3) clean, 512k Chunk, None
(None) None {acd5374f:72628c93:6a906c4b:5f675ce5}
│ │ │ PV LVM2_member 86.00g used, 3.99g free
{VmsWRd-8qHt-bauf-lvAn-FC97-KyH5-gk89ox}
│ │ └VG libvirt_lvm 89.99g 3.99g free {t8GQck-f2Eu-iD2V-fnJQ-kBm6-QyKw-dR31PB}
│ │ ├dm-9 8.00g [253:9] LV builder Partitioned (dos)
│ │ ├dm-10 8.00g [253:10] LV builder2 Partitioned (dos)
│ │ ├dm-11 8.00g [253:11] LV builder3 Partitioned (dos)
│ │ ├dm-13 8.00g [253:13] LV builder5.3 Partitioned (dos)
│ │ ├dm-12 8.00g [253:12] LV builder5.6 Partitioned (dos)
│ │ ├dm-5 8.00g [253:5] LV centos_updt Partitioned (dos)
│ │ ├dm-7 8.00g [253:7] LV cms Partitioned (dos)
│ │ └dm-2 30.00g [253:2] LV win7 Partitioned (dos)
│ └sda6 3.39g [8:6] Empty/Unknown
├scsi 2:0:1:0 ATA WDC WD5000AAKS-0 {WD-WCASY7694185}
│└sdb 465.76g [8:16] Partitioned (dos)
│ ├sdb2 244.14g [8:18] MD raid5 (3) inactive
{2cff15d1-e411-447b-fd5d-472103e44022}
│ ├sdb3 7.81g [8:19] swap {9194f492-881a-4fc3-ac09-ca4e1cc2985a}
│ ├sdb4 1.00k [8:20] Partitioned (dos)
│ ├sdb5 30.00g [8:21] MD raid0 (1/3) (w/ sda5,sdc5) in_sync
'reading.homeunix.com:3' {acd5374f-7262-8c93-6a90-6c4b5f675ce5}
│ │└md124 90.00g [9:124] MD v1.2 raid0 (3) clean, 512k Chunk, None
(None) None {acd5374f:72628c93:6a906c4b:5f675ce5}
│ │ PV LVM2_member 86.00g used, 3.99g free
{VmsWRd-8qHt-bauf-lvAn-FC97-KyH5-gk89ox}
│ └sdb6 3.39g [8:22] Empty/Unknown
├scsi 3:0:0:0 ATA WDC WD5000AAKS-0 {WD-WCASZ0505379}
│└sdc 465.76g [8:32] Partitioned (dos)
│ ├sdc1 29.30g [8:33] MD raid10,near2 (1/2) (w/ sda2) in_sync
{9af006ca-8845-bbd3-bfe7-8010bc810f04}
│ │└md126 29.30g [9:126] MD v0.90 raid10,near2 (2) clean, 64k Chunk
{9af006ca:8845bbd3:bfe78010:bc810f04}
│ │ PV LVM2_member 28.03g used, 1.26g free
{cE4ePh-RWO8-Wgdy-YPOY-ehyC-KI6u-io1cyH}
│ ├sdc2 244.14g [8:34] MD raid5 (3) inactive
{2cff15d1-e411-447b-fd5d-472103e44022}
│ ├sdc3 1.00k [8:35] Partitioned (dos)
│ ├sdc5 30.00g [8:37] MD raid0 (2/3) (w/ sda5,sdb5) in_sync
'reading.homeunix.com:3' {acd5374f-7262-8c93-6a90-6c4b5f675ce5}
│ │└md124 90.00g [9:124] MD v1.2 raid0 (3) clean, 512k Chunk, None
(None) None {acd5374f:72628c93:6a906c4b:5f675ce5}
│ │ PV LVM2_member 86.00g used, 3.99g free
{VmsWRd-8qHt-bauf-lvAn-FC97-KyH5-gk89ox}
│ └sdc6 3.39g [8:38] Empty/Unknown
└scsi 3:0:1:0 ATA WDC WD30EZRX-00D {WD-WCC4N1294906}
└sdd 2.73t [8:48] Partitioned (gpt)
├sdd1 2.00t [8:49] MD raid5 (0/3) (w/ sde1,sdf1) in_sync
'lamachine:128' {f2372cb9-d381-6fd6-ce86-d826882ec82e}
│└md127 4.00t [9:127] MD v1.2 raid5 (3) read-auto, 512k Chunk
{f2372cb9:d3816fd6:ce86d826:882ec82e}
│ │ PV LVM2_member 3.50t used, 511.75g free
{NrF3vV-nS1f-wtjY-SfZk-xwUE-YbSB-TWWdLu}
│ └VG vg_media 4.00t 511.75g free {cPGpZK-OdZ6-sW71-8Rtf-x23f-BgEU-WGEmid}
│ └dm-1 3.50t [253:1] LV lv_media ext4 {ef90c386-76e6-4843-a427-88412948d49f}
└sdd2 500.00g [8:50] MD raid0 (0/3) (w/ sde2,sdf2) in_sync
'lamachine:129' {895dae98-d1a4-96de-4f59-0b8bcb8ac12a}
└md125 1.46t [9:125] MD v1.2 raid0 (3) clean, 512k Chunk, None
(None) None {895dae98:d1a496de:4f590b8b:cb8ac12a}
│ PV LVM2_member 512.00g used, 987.62g free
{70aHF7-aTfj-X2y6-qYXF-szQu-QBFW-q4UlfG}
└VG vg_virt_dir 1.46t 987.62g free {SQXqhD-j5dR-Xsdh-NBoR-YMqk-YwfF-hyf9ur}
└dm-3 512.00g [253:3] LV lv_virt_dir1 ext4
{1929e452-ae83-4766-bf0e-8575a7d1a3c8}
PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10
Family) 2 port SATA IDE Controller #2
├scsi 4:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4NCWT13RF}
│└sde 2.73t [8:64] Partitioned (gpt)
│ ├sde1 2.00t [8:65] MD raid5 (1/3) (w/ sdd1,sdf1) in_sync
'lamachine:128' {f2372cb9-d381-6fd6-ce86-d826882ec82e}
│ │└md127 4.00t [9:127] MD v1.2 raid5 (3) read-auto, 512k Chunk
{f2372cb9:d3816fd6:ce86d826:882ec82e}
│ │ PV LVM2_member 3.50t used, 511.75g free
{NrF3vV-nS1f-wtjY-SfZk-xwUE-YbSB-TWWdLu}
│ └sde2 500.00g [8:66] MD raid0 (1/3) (w/ sdd2,sdf2) in_sync
'lamachine:129' {895dae98-d1a4-96de-4f59-0b8bcb8ac12a}
│ └md125 1.46t [9:125] MD v1.2 raid0 (3) clean, 512k Chunk, None
(None) None {895dae98:d1a496de:4f590b8b:cb8ac12a}
│ PV LVM2_member 512.00g used, 987.62g free
{70aHF7-aTfj-X2y6-qYXF-szQu-QBFW-q4UlfG}
└scsi 5:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4NPRDD6D7}
└sdf 2.73t [8:80] Partitioned (gpt)
├sdf1 2.00t [8:81] MD raid5 (2/3) (w/ sdd1,sde1) in_sync
'lamachine:128' {f2372cb9-d381-6fd6-ce86-d826882ec82e}
│└md127 4.00t [9:127] MD v1.2 raid5 (3) read-auto, 512k Chunk
{f2372cb9:d3816fd6:ce86d826:882ec82e}
│ PV LVM2_member 3.50t used, 511.75g free
{NrF3vV-nS1f-wtjY-SfZk-xwUE-YbSB-TWWdLu}
└sdf2 500.00g [8:82] MD raid0 (2/3) (w/ sdd2,sde2) in_sync
'lamachine:129' {895dae98-d1a4-96de-4f59-0b8bcb8ac12a}
└md125 1.46t [9:125] MD v1.2 raid0 (3) clean, 512k Chunk, None
(None) None {895dae98:d1a496de:4f590b8b:cb8ac12a}
PV LVM2_member 512.00g used, 987.62g free
{70aHF7-aTfj-X2y6-qYXF-szQu-QBFW-q4UlfG}
PCI [pata_jmicron] 07:00.1 IDE interface: JMicron Technology Corp.
JMB363 SATA/IDE Controller (rev 02)
├scsi 6:x:x:x [Empty]
└scsi 7:x:x:x [Empty]
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4]
md124 : active raid0 sda5[0] sdb5[1] sdc5[2]
94367232 blocks super 1.2 512k chunks
md125 : active raid0 sdf2[2] sdd2[0] sde2[1]
1572470784 blocks super 1.2 512k chunks
md127 : active (auto-read-only) raid5 sdf1[3] sde1[1] sdd1[0]
4294705152 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
bitmap: 0/16 pages [0KB], 65536KB chunk
md126 : active raid10 sda2[0] sdc1[1]
30719936 blocks 2 near-copies [2/2] [UU]
unused devices: <none>
[root@localhost ~]# cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md/126 level=raid10 num-devices=2
UUID=9af006ca:8845bbd3:bfe78010:bc810f04
On 25 March 2015 at 00:03, Phil Turmel <philip@turmel.org> wrote:
> On 03/24/2015 06:48 PM, Daniel Sanabria wrote:
>> Hi,
>>
>> One of our servers went trough an OS upgrade (from fedora20 to
>> fedora21) and althou some of the arrays seem to be auto-detected one
>> of them seem to be missing.
>
> You may have had some device names reordered. Building a config that
> relies on the names is common but unwise.
>
> With such a thin report, there's not much to go on. Please run lsdrv
> [1] and paste its output in a reply.
>
> Then add the contents of /proc/mdstat, the relevant-looking parts of
> dmesg, and your mdadm.conf file.
>
> We may need more later, but this will permit intelligent follow-up
> questions.
>
> Phil
>
> [1] https://github.com/pturmel/lsdrv
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: /dev/md0 can't be created
From: NeilBrown @ 2015-03-25 6:35 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <12135556.2792023.1427264134774.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1415 bytes --]
On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> Hi all
>
> I have encountered so many times, the raid device is created successfully, but the directory
> /dev/md0 can't be created. It can't reproduce 100%.
>
> [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> unused devices: <none>
> [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> ls: cannot access /dev/md0: No such file or directory
>
> The underline devices are loop devices which are created with big file.
>
> The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm - v3.3.2 - 21st August 2014)
> I'll try to reproduce this with upstream kernel and mdadm. But I think it shouldn't be the problem about kernel.
>
> What do you think I should check for this? And which tool is responsible for creating the directory? Maybe
> I can add some log to it to find the reason.
>
/dev/md0 is created by udev.
Run
udevadm monitor
to see the events that udev is processing. When and ADD event for "md0" is
processed, /dev/md0 should get created.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox