* Re: md resync ignoring unreadable sectors
From: Phil Turmel @ 2015-02-07 23:42 UTC (permalink / raw)
To: Roman Mamedov, Eyal Lebedinsky; +Cc: linux-raid
In-Reply-To: <20150208040438.156a8554@natsu>
Hi Roman,
On 02/07/2015 06:04 PM, Roman Mamedov wrote:
> On Sun, 08 Feb 2015 09:39:47 +1100
> Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
>> I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
>> around the bad area also showed it to be all zeroes.
>
> I wouldn't expect mdadm to have any headers or unused areas as far as 133 MB
> into a RAID member.
Look at mdadm -E for that drive and your partition start sector. I bet
Eyal is right. Latest mdadm gives me a 128MB data offset.
Phil
^ permalink raw reply
* Re: md resync ignoring unreadable sectors
From: Eyal Lebedinsky @ 2015-02-07 23:43 UTC (permalink / raw)
Cc: linux-raid
In-Reply-To: <20150208040438.156a8554@natsu>
On 08/02/15 10:04, Roman Mamedov wrote:
> On Sun, 08 Feb 2015 09:39:47 +1100
> Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
>
>> The error did not show up during normal operation (or during scrub), only during the smartctl long test.
>> What triggered the error for you?
>
> Just appeared during boot-up after a reboot (after 50 days uptime) which was
> performed for some hardware upgrades (RAM, SATA controller). The error doesn't
> go away after swapping the SATA controller for different one.
>
>> I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
>> around the bad area also showed it to be all zeroes.
>
> I wouldn't expect mdadm to have any headers or unused areas as far as 133 MB
> into a RAID member.
Roman,
You may want to read
https://raid.wiki.kernel.org/index.php/RAID_superblock_formats
For me:
# parted -l
...
Number Start End Size File system Name Flags
1 1049kB 4001GB 4001GB
...
# mdadm --examine /dev/sdc1
...
Data Offset : 262144 sectors
...
With required alignment etc. it may reach your 133MB.
cheers
--
Eyal Lebedinsky (eyal@eyal.emu.id.au)
^ permalink raw reply
* Re: md resync ignoring unreadable sectors
From: Roman Mamedov @ 2015-02-07 23:49 UTC (permalink / raw)
To: Phil Turmel; +Cc: Eyal Lebedinsky, linux-raid
In-Reply-To: <54D6A2CC.3050102@turmel.org>
On Sat, 07 Feb 2015 18:42:04 -0500
Phil Turmel <philip@turmel.org> wrote:
> Hi Roman,
>
> On 02/07/2015 06:04 PM, Roman Mamedov wrote:
> > On Sun, 08 Feb 2015 09:39:47 +1100
> > Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
>
> >> I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
> >> around the bad area also showed it to be all zeroes.
> >
> > I wouldn't expect mdadm to have any headers or unused areas as far as 133 MB
> > into a RAID member.
>
> Look at mdadm -E for that drive and your partition start sector. I bet
> Eyal is right. Latest mdadm gives me a 128MB data offset.
Oh indeed:
Data Offset : 262144 sectors
The unreadable area was at 260200 sectors.
Thanks
--
With respect,
Roman
^ permalink raw reply
* Re: md resync ignoring unreadable sectors
From: Eyal Lebedinsky @ 2015-02-07 23:49 UTC (permalink / raw)
Cc: linux-raid
In-Reply-To: <20150208040438.156a8554@natsu>
On 08/02/15 10:04, Roman Mamedov wrote:
> On Sun, 08 Feb 2015 09:39:47 +1100
> Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
>
>> The error did not show up during normal operation (or during scrub), only during the smartctl long test.
>> What triggered the error for you?
>
> Just appeared during boot-up after a reboot (after 50 days uptime) which was
> performed for some hardware upgrades (RAM, SATA controller). The error doesn't
> go away after swapping the SATA controller for different one.
>
>> I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
>> around the bad area also showed it to be all zeroes.
>
> I wouldn't expect mdadm to have any headers or unused areas as far as 133 MB
> into a RAID member.
>
You can also read the earlier discussion starting 26/Feb/14:
Subject: how to handle bad sectors in md control areas?
cheers
--
Eyal Lebedinsky (eyal@eyal.emu.id.au)
^ permalink raw reply
* Re: [PATCH 3/3] dm ioctl: Delete an unnecessary check before the function call "dm_table_destroy"
From: SF Markus Elfring @ 2015-02-08 9:55 UTC (permalink / raw)
To: Mike Snitzer
Cc: Alasdair Kergon, Neil Brown, dm-devel, linux-raid, Julia Lawall,
kernel-janitors, LKML
In-Reply-To: <20150206211234.GA14803@redhat.com>
> Your proposed patch (while technically correct) hurts code clarity.
How many source code readability and understanding challenges does each
additional condition check cause?
Can the affected place become also a bit more efficient?
Regards,
Markus
^ permalink raw reply
* Re: [PATCH 3/3] dm ioctl: Delete an unnecessary check before the function call "dm_table_destroy"
From: Mike Snitzer @ 2015-02-08 14:02 UTC (permalink / raw)
To: SF Markus Elfring
Cc: Alasdair Kergon, Neil Brown, device-mapper development,
linux-raid@vger.kernel.org, Julia Lawall, kernel-janitors, LKML
In-Reply-To: <54D7329C.206@users.sourceforge.net>
On Sun, Feb 8, 2015 at 4:55 AM, SF Markus Elfring
<elfring@users.sourceforge.net> wrote:
>> Your proposed patch (while technically correct) hurts code clarity.
>
> How many source code readability and understanding challenges does each
> additional condition check cause?
Please don't make a mountain out of a mole hill in an attempt to
defend your robotic patch (I'm quite tired of some of these static
analyzer patch submissions).
FYI, I did stage your other patch for 3.20, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20&id=d0ce7e911c97c7c6df1081dcedfefced82a0c6bf
> Can the affected place become also a bit more efficient?
Efficiency isn't a concern in this instance (it isn't a hot IO path).
And even if it were, a branch (with current code) is more efficient vs
a a jump + branch (your proposed patch) -- in the case that no active
table exists. Now if it likely that old_map does exist then yes your
patch is always a very slight win.
But given the duality of the calling function (deals with loading a
new map and destroying the old map if it exists) I prefer to keep the
code as is. Sorry.
^ permalink raw reply
* An old "write-mostly" read balance issue
From: Dark Penguin @ 2015-02-08 15:40 UTC (permalink / raw)
To: linux-raid
There is an old issue about RAID1 read-balancing when "write-mostly"
disks are present.
The problem is, according to the manual, "md driver will avoid reading
from these devices if at all possible".
One way to understand this statement is that these drives will never be
read from, except when the main drive can not be read from. There are A
LOT of situations when this is the expected and desired behaviour:
- People mirroring an SSD with an HDD and suffering a performance loss;
- People mirroring a fast HDD with a slow HDD for reliability, for
example, mirroring a 300Gb WD Raptor to a 300Gb partition on a 3Tb 5900
"green" drive for backup; since the larger drive may be used for
something other than this RAID, many would prefer it to be spared the
workload.
- In my case, I have a home RAID1 storage, which is idle 95% of the
time, and 95% of the remaining 5% I only read from it. So I want one of
the drives to spin down and never turn on, in order to avoid wearing
down the mechanics. They say, "The best way to keep a device from
breaking is to turn it off and not use it". :) But even if I simply
retrieve the contents of my volume, that request is apparently enough to
load the first drive to 100% for a split second, which causes the second
drive to spin up, which is extremely undesirable.
I've spent a lot of time looking for the answer "why does it spin up",
and "normal forum users" couldn not even help me, but then I found out
that there is another way to read that statement: apparently, there are
other people who would like to see whatever little benefit reading from
the second drive could give them. I can not say which side is a
majority, but I respect their wishes as well, and personally I'm fine
with any default behaviour as long as I have what I need.
I've found a patch for that:
http://marc.info/?l=linux-raid&m=135982797322422
Apparently, it can be used with any kernel, but I'm not good enough to
make sure nothing's broken everytime I upgrade the kernel, and frankly,
I think there are A LOT of people who wish to see the behaviour I would
expect. So my plea is for the developers to accept this patch and make
this behaviour optional, if not default. At least give us a compile
option to build the kernel this way! There are people out there who use
RAID1 at home and not in production, and therefore care less about
performance than home storage idling, and who understand the words "if
at all possible" in the more obvious way! I think that's the whole
reason why the "write-mostly" option is there in the first place, but if
there are people who don't agree with me - I'm not going to argue, they
can have it their way, just give us the option to do what we want, too!
--
darkpenguin
^ permalink raw reply
* Re: [PATCH 3/3] dm ioctl: Delete an unnecessary check before the function call "dm_table_destroy"
From: SF Markus Elfring @ 2015-02-08 15:52 UTC (permalink / raw)
To: Mike Snitzer
Cc: Alasdair Kergon, Neil Brown, device-mapper development,
linux-raid@vger.kernel.org, Julia Lawall, kernel-janitors, LKML
In-Reply-To: <CAMM=eLfpAiTeqvX17C3agsLXP03UD_Q7DnmhGnO5GjEC0pZa8Q@mail.gmail.com>
> FYI, I did stage your other patch for 3.20, see:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20&id=d0ce7e911c97c7c6df1081dcedfefced82a0c6bf
Thanks for your acceptance of the suggested clean-up around
vfree() function calls at least.
Additional source code places can also be reconsidered at other times,
can't they?
Regards,
Markus
^ permalink raw reply
* Re: md resync ignoring unreadable sectors
From: John Stoffel @ 2015-02-08 17:23 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Eyal Lebedinsky, linux-raid
In-Reply-To: <20150208040438.156a8554@natsu>
>>>>> "Roman" == Roman Mamedov <rm@romanrm.net> writes:
Roman> On Sun, 08 Feb 2015 09:39:47 +1100
Roman> Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
>> The error did not show up during normal operation (or during scrub), only during the smartctl long test.
>> What triggered the error for you?
Roman> Just appeared during boot-up after a reboot (after 50 days uptime) which was
Roman> performed for some hardware upgrades (RAM, SATA controller). The error doesn't
Roman> go away after swapping the SATA controller for different one.
>> I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
>> around the bad area also showed it to be all zeroes.
Roman> I wouldn't expect mdadm to have any headers or unused areas as
Roman> far as 133 MB into a RAID member.
Roman,
I would immediately add in a third RAID1 member, wait for it to
resync, then pull out the bad drive and write zeros to the entire
drive to force any and all bad sectors to get over-written and
hopefully reallocated from good sectors.
But I'd also treat the drive as suspect and replace it ASAP. Keet it
around as a scratch drive, or a temp space area you don't care about
if you like, but not of important data if at all possible.
John
^ permalink raw reply
* raid md126, md127 problem after reboot, howto fix?
From: Marc Widmer @ 2015-02-08 19:03 UTC (permalink / raw)
To: linux-raid
Hi List
I have no deep unterstand about raids, beside setting them up initially and
replacing disks if needed. So this error has never happened to me before:
After a reboot i have a really strange behaviour on my server. Disks are
not marked faulty, but raid is "fallend apart".
/proc/mdstat shows me:
md126 : active raid1 sda1[0]
10485696 blocks [2/1] [U_]
md127 : active raid1 sda2[0]
721558464 blocks [2/1] [U_]
md1 : active raid1 sdb1[1]
10485696 blocks [2/1] [_U]
md2 : active raid1 sdb2[1]
721558464 blocks [2/1] [_U]
wished would be something similar to:
md1 : active raid1 sdb1[1] sda1[0]
10238912 blocks [2/2] [UU]
md2 : active raid1 sdb2[1] sda2[0]
1942746048 blocks [2/2] [UU]
Currently only md1, md2 are running. nmon shows me, that only disks sdb is
active, sda is not doing anything.
I run debian squeeze.
I am a bit concerned what to do, because at the moment i run on one disk
only and if things go wrong i end up with a server not running (downtime)
and possible data loss (beside backups).
Any ideas what i should do? Howto put the raid back together, possibly in
live mode, without rebooting in rescue mode and risk long downtime?
Any help would be greatly appreciated as by now the only thing i had to do
was resyncing a disk after usual hd crash.
Best
marc
^ permalink raw reply
* Re: raid md126, md127 problem after reboot, howto fix?
From: Wols Lists @ 2015-02-08 21:29 UTC (permalink / raw)
To: Marc Widmer, linux-raid
In-Reply-To: <54D7B317.5010809@gmail.com>
On 08/02/15 19:03, Marc Widmer wrote:
> Hi List
>
> I have no deep unterstand about raids, beside setting them up initially and
> replacing disks if needed. So this error has never happened to me before:
>
> After a reboot i have a really strange behaviour on my server. Disks are
> not marked faulty, but raid is "fallend apart".
>
> /proc/mdstat shows me:
>
> md126 : active raid1 sda1[0]
> 10485696 blocks [2/1] [U_]
>
> md127 : active raid1 sda2[0]
> 721558464 blocks [2/1] [U_]
>
> md1 : active raid1 sdb1[1]
> 10485696 blocks [2/1] [_U]
>
> md2 : active raid1 sdb2[1]
> 721558464 blocks [2/1] [_U]
>
> wished would be something similar to:
> md1 : active raid1 sdb1[1] sda1[0]
> 10238912 blocks [2/2] [UU]
>
> md2 : active raid1 sdb2[1] sda2[0]
> 1942746048 blocks [2/2] [UU]
>
> Currently only md1, md2 are running. nmon shows me, that only disks sdb is
> active, sda is not doing anything.
>
> I run debian squeeze.
What version of mdadm are you running? 3.2.6 or thereabouts?
>
> I am a bit concerned what to do, because at the moment i run on one disk
> only and if things go wrong i end up with a server not running (downtime)
> and possible data loss (beside backups).
>
> Any ideas what i should do? Howto put the raid back together, possibly in
> live mode, without rebooting in rescue mode and risk long downtime?
>
> Any help would be greatly appreciated as by now the only thing i had to do
> was resyncing a disk after usual hd crash.
>
The reason I ask is this looks like a bug I had - if I'm right it's a
known problem and you need to upgrade mdadm.
Cheers,
Wol
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: NeilBrown @ 2015-02-09 1:13 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150206113930.GK23123@twins.programming.kicks-ass.net>
[-- Attachment #1: Type: text/plain, Size: 4744 bytes --]
On Fri, 6 Feb 2015 12:39:30 +0100 Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, Feb 06, 2015 at 08:51:33AM +1100, NeilBrown wrote:
> > That is exactly what is happening here. However I don't think that is an
> > "observed problem" but rather an "observed false-positive".
> >
> > If nothing inside the outer loop blocks, then in particular
> > generic_make_request will not be called, so nothing will be added to the
> > queue that blk_schedule_flush_plug flushes.
> > So the first time through the loop, a call the 'schedule()' may not actually
> > block, but every subsequent time it will.
> > So there is no actual problem here.
> >
> > So I'd be included to add sched_annotate_sleep() in blk_flush_plug_list().
> >
> > Peter: what do you think is the best way to silence this warning.
>
> > > Call Trace:
>
> > > [<ffffffff8027ee62>] __might_sleep+0x82/0x90
> > > [<ffffffff803bee06>] generic_make_request_checks+0x36/0x2d0
> > > [<ffffffff803bf0b3>] generic_make_request+0x13/0x100
> > > [<ffffffff8054983b>] raid1_unplug+0x12b/0x170
> > > [<ffffffff803c1302>] blk_flush_plug_list+0xa2/0x230
> > > [<ffffffff80646383>] io_schedule+0x43/0x80
> > > [<ffffffff80646787>] bit_wait_io+0x27/0x50
>
> Well, I don't know. I don't particularly like the whole blk_flush_plug()
> thing scheduling while on its way to schedule. If you ever end up
> calling io_schedule() from it there's 'fun'.
>
> Also, how likely is it to actually schedule when doing all that? This
> block layer stuff is somewhat impenetrable for me, too many callbacks.
>
> You have some words on how its unlikely, but I can't even find _where_
> it would schedule :/ All I see is a loop calling ->make_request_fn() and
> god only knows where that ends up.
>
> So there appear to be two blk_flush_plug() variants, one with an
> @from_schedule = true, which seems to really try not to schedule, which
> seems to suggest the 'false' one (the one above) is meant to schedule?
>
> If scheduling is the rule rather than the exception, the above is
> properly broken.
>
> But again, I don't know.
I had to re-read the code (And your analysis) a couple of times to be sure ...
As you say, when schedule() calls blk_schedule_flush_plug(), the
@from_schedule=true variant is used and the unplug code doesn't block.
So there is no problem there.
However, when io_schedule() explicitly calls blk_flush_plug(), then
@from_schedule=false variant is used, and the unplug functions are allowed to
allocate memory and block and maybe even call mempool_alloc() which might
call io_schedule().
This shouldn't be a problem as blk_flush_plug() spliced out the plug list, so
any recursive call will find an empty list and do nothing.
Worst case is that a wait_event loop that calls io_schedule() (i.e.
wait_on_bit_io()) might not block in the first call to io_schedule()
if the unplugging needed to wait. Every subsequent call will block as
required as there is nothing else to add requests to the plug queue.
So as long as wait_on_bio_io() can cope with a single false wakeup (which it
can), there is no problem here.
>
> If you're confident that scheduling is rare for _ALL_ (current and
> future) block device implementations, not just the raid one, then you
> can annotate blk_flush_plug_list() I suppose.
>
> Otherwise I would suggest adding them one at a time in whatever blk
> device thing likes to go schedule on us. Also, add a comment that
> explains why its rare for the future us who need to look at it again.
It isn't that scheduling is "rare" - it is that it can only occur once in a
loop which doesn't expect it.
So I propose the following, though I haven't tested it.
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e628cb11b560..b0f12ab3df23 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4374,6 +4374,11 @@ void __sched io_schedule(void)
delayacct_blkio_start();
atomic_inc(&rq->nr_iowait);
+ /* Any sleeping in blk_flush_plug() should not
+ * trigger the "do not call blocking ops" warning
+ * as it can only happen once in a wait_event loop.
+ */
+ sched_annotate_sleep();
blk_flush_plug(current);
current->in_iowait = 1;
schedule();
@@ -4390,6 +4395,11 @@ long __sched io_schedule_timeout(long timeout)
delayacct_blkio_start();
atomic_inc(&rq->nr_iowait);
+ /* Any sleeping in blk_flush_plug() should not
+ * trigger the "do not call blocking ops" warning
+ * as it can only happen once in a wait_event loop.
+ */
+ sched_annotate_sleep();
blk_flush_plug(current);
current->in_iowait = 1;
ret = schedule_timeout(timeout);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: Peter Zijlstra @ 2015-02-09 9:10 UTC (permalink / raw)
To: NeilBrown; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150209121357.29f19d36@notabene.brown>
On Mon, Feb 09, 2015 at 12:13:57PM +1100, NeilBrown wrote:
> I had to re-read the code (And your analysis) a couple of times to be sure ...
Sorry :-)
> However, when io_schedule() explicitly calls blk_flush_plug(), then
> @from_schedule=false variant is used, and the unplug functions are allowed to
> allocate memory and block and maybe even call mempool_alloc() which might
> call io_schedule().
>
> This shouldn't be a problem as blk_flush_plug() spliced out the plug list, so
> any recursive call will find an empty list and do nothing.
Unless, something along the way stuck something back on, right? So
should we stick an:
WARN_ON(current->in_iowait);
somewhere near where things are added to this plug list? (and move the
blk_flush_plug() call inside of where that's actually true of course).
> Worst case is that a wait_event loop that calls io_schedule() (i.e.
> wait_on_bit_io()) might not block in the first call to io_schedule()
> if the unplugging needed to wait. Every subsequent call will block as
> required as there is nothing else to add requests to the plug queue.
Again, assuming @cond will not actually stick something on this list.
Which if we add the above we'll get warned about.
> It isn't that scheduling is "rare" - it is that it can only occur once in a
> loop which doesn't expect it.
With the above WARN stuck in, agreed.
> So I propose the following, though I haven't tested it.
>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index e628cb11b560..b0f12ab3df23 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4374,6 +4374,11 @@ void __sched io_schedule(void)
>
> delayacct_blkio_start();
> atomic_inc(&rq->nr_iowait);
> + /* Any sleeping in blk_flush_plug() should not
> + * trigger the "do not call blocking ops" warning
> + * as it can only happen once in a wait_event loop.
> + */
Might I suggest the 'regular' multi-line comment style, and a reference
to the above WARN that makes everything actually work?
/*
* multi-line
* comments have an empty
* line at the start... As per CodingStyle ch. 8
*/
> + sched_annotate_sleep();
> blk_flush_plug(current);
Also, at this point, should we put it in blk_flush_plug()?
The only thing that really goes wrong then is if people 'forget' to put
a loop around io_schedule().
^ permalink raw reply
* Re: raid md126, md127 problem after reboot, howto fix?
From: Sebastian Parschauer @ 2015-02-09 9:41 UTC (permalink / raw)
To: Wols Lists, Marc Widmer, linux-raid
In-Reply-To: <54D7D551.2030800@youngman.org.uk>
On 08.02.2015 22:29, Wols Lists wrote:
> On 08/02/15 19:03, Marc Widmer wrote:
>> Hi List
>>
>> I have no deep unterstand about raids, beside setting them up initially and
>> replacing disks if needed. So this error has never happened to me before:
>>
>> After a reboot i have a really strange behaviour on my server. Disks are
>> not marked faulty, but raid is "fallend apart".
>>
>> /proc/mdstat shows me:
>>
>> md126 : active raid1 sda1[0]
>> 10485696 blocks [2/1] [U_]
>>
>> md127 : active raid1 sda2[0]
>> 721558464 blocks [2/1] [U_]
>>
>> md1 : active raid1 sdb1[1]
>> 10485696 blocks [2/1] [_U]
>>
>> md2 : active raid1 sdb2[1]
>> 721558464 blocks [2/1] [_U]
>>
>> wished would be something similar to:
>> md1 : active raid1 sdb1[1] sda1[0]
>> 10238912 blocks [2/2] [UU]
>>
>> md2 : active raid1 sdb2[1] sda2[0]
>> 1942746048 blocks [2/2] [UU]
>>
>> Currently only md1, md2 are running. nmon shows me, that only disks sdb is
>> active, sda is not doing anything.
>>
>> I run debian squeeze.
>
> What version of mdadm are you running? 3.2.6 or thereabouts?
>>
>> I am a bit concerned what to do, because at the moment i run on one disk
>> only and if things go wrong i end up with a server not running (downtime)
>> and possible data loss (beside backups).
>>
>> Any ideas what i should do? Howto put the raid back together, possibly in
>> live mode, without rebooting in rescue mode and risk long downtime?
>>
>> Any help would be greatly appreciated as by now the only thing i had to do
>> was resyncing a disk after usual hd crash.
>>
> The reason I ask is this looks like a bug I had - if I'm right it's a
> known problem and you need to upgrade mdadm.
Yeah, pretty much sounds like the bad udev rules with Squeeze and old
mdadm. E.g. deactivation of the MD udev rules and assembling via init
scripts is a way to workaround this.
I have some test VMs providing MD RAID-1 on iSCSI targets and have seen
the same issue when logging in to the targets. Deactivation of the udev
rules and manual assembly helped.
Cheers,
Sebastian
^ permalink raw reply
* Re: raid md126, md127 problem after reboot, howto fix?
From: Sebastian Parschauer @ 2015-02-09 9:55 UTC (permalink / raw)
To: Marc Widmer, linux-raid
In-Reply-To: <54D7B317.5010809@gmail.com>
On 08.02.2015 20:03, Marc Widmer wrote:
> Hi List
>
> I have no deep unterstand about raids, beside setting them up initially and
> replacing disks if needed. So this error has never happened to me before:
>
> After a reboot i have a really strange behaviour on my server. Disks are
> not marked faulty, but raid is "fallend apart".
>
> /proc/mdstat shows me:
>
> md126 : active raid1 sda1[0]
> 10485696 blocks [2/1] [U_]
>
> md127 : active raid1 sda2[0]
> 721558464 blocks [2/1] [U_]
>
> md1 : active raid1 sdb1[1]
> 10485696 blocks [2/1] [_U]
>
> md2 : active raid1 sdb2[1]
> 721558464 blocks [2/1] [_U]
>
> wished would be something similar to:
> md1 : active raid1 sdb1[1] sda1[0]
> 10238912 blocks [2/2] [UU]
>
> md2 : active raid1 sdb2[1] sda2[0]
> 1942746048 blocks [2/2] [UU]
>
> Currently only md1, md2 are running. nmon shows me, that only disks sdb is
> active, sda is not doing anything.
>
> I run debian squeeze.
>
> I am a bit concerned what to do, because at the moment i run on one disk
> only and if things go wrong i end up with a server not running (downtime)
> and possible data loss (beside backups).
>
> Any ideas what i should do? Howto put the raid back together, possibly in
> live mode, without rebooting in rescue mode and risk long downtime?
Just stop all arrays which aren't in use at the moment and assemble them
manually. If they are already running degraded, then add the now
unassociated disks to their respective running arrays.
$ mdadm --stop /dev/md126
$ mdadm --stop /dev/md127
$ mdadm /dev/md1 --add /dev/sda1
$ mdadm /dev/md2 --add /dev/sda2
This could require some syncing but then everything should be normal again.
Cheers,
Sebastian
^ permalink raw reply
* [PATCH RESEND] Monitor: fix for regression with container devices
From: Artur Paszkiewicz @ 2015-02-09 10:13 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, pawel.baldysiak, Artur Paszkiewicz
This patch fixes 2 problems introduced by commit 9a518d8: not closing a
file descriptor and ignoring container devices. Array state is always
"inactive" for containers, so we make sure that the device is not a
container by reading also the "level" sysfs entry.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-by: Pawel Baldysiak <pawel.baldysiak@intel.com>
---
Monitor.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Monitor.c b/Monitor.c
index 971d2ec..66d67ba 100644
--- a/Monitor.c
+++ b/Monitor.c
@@ -483,11 +483,17 @@ static int check_array(struct state *st, struct mdstat_ent *mdstat,
strncmp(buf,"inact",5) == 0) {
if (fd >= 0)
close(fd);
- if (!st->err)
- alert("DeviceDisappeared", dev, NULL, ainfo);
- st->err++;
- return 0;
+ fd = sysfs_open(st->devnm, NULL, "level");
+ if (fd < 0 || read(fd, buf, 10) != 0) {
+ if (fd >= 0)
+ close(fd);
+ if (!st->err)
+ alert("DeviceDisappeared", dev, NULL, ainfo);
+ st->err++;
+ return 0;
+ }
}
+ close(fd);
}
fd = open(dev, O_RDONLY);
if (fd < 0) {
--
2.1.4
^ permalink raw reply related
* Re: raid md126, md127 problem after reboot, howto fix?
From: Wols Lists @ 2015-02-09 10:22 UTC (permalink / raw)
To: Sebastian Parschauer, Marc Widmer, linux-raid
In-Reply-To: <54D883FA.8080401@profitbricks.com>
On 09/02/15 09:55, Sebastian Parschauer wrote:
>> > Any ideas what i should do? Howto put the raid back together, possibly in
>> > live mode, without rebooting in rescue mode and risk long downtime?
> Just stop all arrays which aren't in use at the moment and assemble them
> manually. If they are already running degraded, then add the now
> unassociated disks to their respective running arrays.
>
> $ mdadm --stop /dev/md126
> $ mdadm --stop /dev/md127
> $ mdadm /dev/md1 --add /dev/sda1
> $ mdadm /dev/md2 --add /dev/sda2
>
> This could require some syncing but then everything should be normal again.
This isn't just a Debian/Ubuntu problem - I run gentoo.
And isn't there a re-add option? That will hopefully just require a
recovery rather than a total resync.
Cheers,
Wol
^ permalink raw reply
* Re: raid md126, md127 problem after reboot, howto fix?
From: Sebastian Parschauer @ 2015-02-09 10:33 UTC (permalink / raw)
To: Wols Lists, Marc Widmer, linux-raid
In-Reply-To: <54D88A7A.4000207@youngman.org.uk>
On 09.02.2015 11:22, Wols Lists wrote:
> On 09/02/15 09:55, Sebastian Parschauer wrote:
>>>> Any ideas what i should do? Howto put the raid back together, possibly in
>>>> live mode, without rebooting in rescue mode and risk long downtime?
>> Just stop all arrays which aren't in use at the moment and assemble them
>> manually. If they are already running degraded, then add the now
>> unassociated disks to their respective running arrays.
>>
>> $ mdadm --stop /dev/md126
>> $ mdadm --stop /dev/md127
>> $ mdadm /dev/md1 --add /dev/sda1
>> $ mdadm /dev/md2 --add /dev/sda2
>>
>> This could require some syncing but then everything should be normal again.
>
> This isn't just a Debian/Ubuntu problem - I run gentoo.
Okay.
> And isn't there a re-add option? That will hopefully just require a
> recovery rather than a total resync.
Yes, sure, --re-add should be tried first. If it doesn't work, then the
--add option will definitely work.
Cheers,
Sebastian
^ permalink raw reply
* Upgrading storage server
From: Adam Goryachev @ 2015-02-09 12:35 UTC (permalink / raw)
To: linux-raid
Hi all,
After making a whole string of mistakes in building a iSCSI server about
2 years ago, I'm now looking to replace it without all the wrong
turns/mistakes. I was hoping you could all offer some advice on hardware
selection/choices.
The target usage as above is an iSCSI server as the backend to a bunch
of VM's. Currently I have two identical storage servers, using 7 x SSD
with Linux MD Raid, then using LVM to divide it up for each VM, and then
DRBD on top to sync the two servers together, on the top is ietd to
share the multiple DRBD devices out. The two servers have a single
10Gbps connection between them for DRBD to sync the data. They also have
a second 10Gbps ethernet for iscsi to use, with a pair of 1Gbps for
management (on board). I have 8 x PC's running Xen with 2 x 1Gbps
ethernet for iSCSI and one 1Gbps ethernet for the "user"/management LAN.
Current hardware of the storage servers are:
7 x Intel 480GB SSD Model SSDSC2CW480A3
1 x Intel 180GB SSD Model SSDSC2CT180A4 (for the OS)
1 x LSI Logic SAS2308 PCI-Express (8 x SATA connections)
1 x Intel Dual port 10Gbps 82599EB SFI/SFP+ Ethernet
1 x Intel Xeon CPU E3-1230 V2 @ 3.30GHz
Motherboard Intel S1200
http://ark.intel.com/products/67494/Intel-Server-Board-S1200BTLR
What I'm hoping to achieve is to purchase two new (identical) servers,
using current recommended (and well supported for the new few years)
parts, and then move the two existing servers to a remote site,
combining with DRBD proxy to give a full, "live" off-site backup
solution. (Note, by backup I mean Disaster Recovery, not backup).
I would also like to be able to grow the total size of the data further
if needed, currently I have 7 x 480G in RAID5, which is likely somewhat
sub-optimal. Options include moving to larger size SSD, or at perhaps
splitting into 2 x RAID5 arrays. The advantage of larger SSD's would be
a smaller "system", with lower complexity, while using more smaller
drives would provide (potentially) better performance, since each drive
(regardless of size) has the same overall performance (both throughput
and IOPS).
I would appreciate any advise or suggestions you can make to help me
avoid the many mistakes I made last time.
Regards,
Adam
--
Adam Goryachev
Website Managers
www.websitemanagers.com.au
^ permalink raw reply
* Re: Upgrading storage server
From: Joe Landman @ 2015-02-09 14:47 UTC (permalink / raw)
To: Adam Goryachev, linux-raid
In-Reply-To: <54D8A9AD.3060700@websitemanagers.com.au>
On 02/09/2015 07:35 AM, Adam Goryachev wrote:
> Hi all,
>
> After making a whole string of mistakes in building a iSCSI server
> about 2 years ago, I'm now looking to replace it without all the wrong
> turns/mistakes. I was hoping you could all offer some advice on
> hardware selection/choices.
>
> The target usage as above is an iSCSI server as the backend to a bunch
> of VM's. Currently I have two identical storage servers, using 7 x SSD
> with Linux MD Raid, then using LVM to divide it up for each VM, and
> then DRBD on top to sync the two servers together, on the top is ietd
> to share the multiple DRBD devices out. The two servers have a single
> 10Gbps connection between them for DRBD to sync the data. They also
> have a second 10Gbps ethernet for iscsi to use, with a pair of 1Gbps
> for management (on board). I have 8 x PC's running Xen with 2 x 1Gbps
> ethernet for iSCSI and one 1Gbps ethernet for the "user"/management LAN.
>
> Current hardware of the storage servers are:
> 7 x Intel 480GB SSD Model SSDSC2CW480A3
> 1 x Intel 180GB SSD Model SSDSC2CT180A4 (for the OS)
We always use 2 drives in an MD RAID1 for OS.
> 1 x LSI Logic SAS2308 PCI-Express (8 x SATA connections)
Ok. This is a lower end card on the performance side.
> 1 x Intel Dual port 10Gbps 82599EB SFI/SFP+ Ethernet
> 1 x Intel Xeon CPU E3-1230 V2 @ 3.30GHz
> Motherboard Intel S1200
> http://ark.intel.com/products/67494/Intel-Server-Board-S1200BTLR
>
> What I'm hoping to achieve is to purchase two new (identical) servers,
> using current recommended (and well supported for the new few years)
> parts, and then move the two existing servers to a remote site,
> combining with DRBD proxy to give a full, "live" off-site backup
> solution. (Note, by backup I mean Disaster Recovery, not backup).
>
> I would also like to be able to grow the total size of the data
> further if needed, currently I have 7 x 480G in RAID5, which is likely
> somewhat sub-optimal. Options include moving to larger size SSD, or at
> perhaps splitting into 2 x RAID5 arrays.
Yes, RAIDx for x=5,6 are generally suboptimal for SSDs due to write
amplification from the RMW cycle. RAID10's are generally much gentler
on SSDs from a longevity scenario.
> The advantage of larger SSD's would be a smaller "system", with lower
> complexity, while using more smaller drives would provide
> (potentially) better performance, since each drive (regardless of
> size) has the same overall performance (both throughput and IOPS).
Are you performance limited now, or will you be shortly? If so the
performance arguments make sense.
>
> I would appreciate any advise or suggestions you can make to help me
> avoid the many mistakes I made last time.
I'm biased given what we do. If you are going to build it yourself, I'd
recommend sticking to known working elements that aren't a pain to setup
and manage. Focus on RAID10 for the primary storage, move the OS to a
completely different controller. Build the OS drives as MD RAID1.
You might want to investigate dm multipath as well as DRBD/md, and Ceph
RBD. I'm a huge fan and user of MD RAID, but you are asking much higher
level architectural questions, and MD RAID would be one of several
technologies you would use for this.
>
> Regards,
> Adam
>
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
e: landman@scalableinformatics.com
w: http://scalableinformatics.com
t: @scalableinfo
p: +1 734 786 8423 x121
c: +1 734 612 4615
^ permalink raw reply
* mdsadm -A won't assemble my array
From: G. Michael Carter @ 2015-02-09 20:21 UTC (permalink / raw)
To: linux-raid
Some time last night my machine had a kernel panic. Two of the arrays
didn't start up.
One I managed to fix as a mdadm -E clued me in that three of the
drives were ok. So I just reassembled the three and added the fourth.
Then it just started no problem.
My big array however I'm not so lucky.
I've got a state of
Raid level: 5
/dev/sdb: AA.. (state: clean)
/dev/sdk: AAAA (state: active)
/dev/sdo: A.AA (state: clean)
/dev/sdp: A.AA (state: clean)
Thus can only get two drives to match in any config. How do I get out
of this mess?
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: G. Michael Carter @ 2015-02-09 23:05 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CANjusZesi-8zxpZ9rCOEToSXwFoiApeN-ivO51qY=80R6_xb_g@mail.gmail.com>
After doing a lot more reading... I think I'm getting down to running
something like this. assemble force isn't doing much.
mdadm --create --assume-clean --level=5 --verbose --chunk 512K
--raid-devices=4 /dev/md3 /dev/sdb /dev/sdk /dev/sdo /dev/sdp
But as per the big warning... says to check with you guys first, also
need help writing the command (since it seems to be a one shot type
thing)
Here's the key information I think I need from the examine:
Raid level: 5
Chunk Size: 512K
Used Dev Size: 7813774336
/dev/sdb: AA.. (state: clean - active device 0)
/dev/sdk: AAAA (state: active - active device 1)
/dev/sdo: A.AA (state: clean - active device 2)
/dev/sdp: A.AA (state: clean - active device 3)
thanks
On Mon, Feb 9, 2015 at 3:21 PM, G. Michael Carter <mikey@carterfamily.ca> wrote:
> Some time last night my machine had a kernel panic. Two of the arrays
> didn't start up.
>
> One I managed to fix as a mdadm -E clued me in that three of the
> drives were ok. So I just reassembled the three and added the fourth.
> Then it just started no problem.
>
> My big array however I'm not so lucky.
>
> I've got a state of
>
>
> Raid level: 5
> /dev/sdb: AA.. (state: clean)
> /dev/sdk: AAAA (state: active)
> /dev/sdo: A.AA (state: clean)
> /dev/sdp: A.AA (state: clean)
>
> Thus can only get two drives to match in any config. How do I get out
> of this mess?
--
G. Michael Carter
Contact: H: 1-519-940-8935 | W: 1-905-267-8494 | M: 1-519-215-1869 |
F: 1-519-941-0009
Google Talk: xmpp:mikeycarter1974@gmail.com
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: Phil Turmel @ 2015-02-10 0:13 UTC (permalink / raw)
To: G. Michael Carter, linux-raid
In-Reply-To: <CANjusZcnX=BArJJLMiBN4hfzRRLdUtD6mSq-mDZ6OhPB=KMX3w@mail.gmail.com>
Hi Michael,
[Convention on kernel.org is reply-to-all, to trim replies and to
bottom-post, or interleave your reply]
On 02/09/2015 06:05 PM, G. Michael Carter wrote:
> On Mon, Feb 9, 2015 at 3:21 PM, G. Michael Carter <mikey@carterfamily.ca> wrote:
>> Some time last night my machine had a kernel panic. Two of the arrays
>> didn't start up.
>>
>> One I managed to fix as a mdadm -E clued me in that three of the
>> drives were ok. So I just reassembled the three and added the fourth.
>> Then it just started no problem.
>>
>> My big array however I'm not so lucky.
>>
>> I've got a state of
>>
>>
>> Raid level: 5
>> /dev/sdb: AA.. (state: clean)
>> /dev/sdk: AAAA (state: active)
>> /dev/sdo: A.AA (state: clean)
>> /dev/sdp: A.AA (state: clean)
Please show us *all* of your mdadm -E output for this array. Pasted
inline is preferred. Also show a map of your device names versus drive
serial numbers. An excerpt from "ls -l /dev/disk/by-id/" will do. You
have many drives, and the kernel doesn't guarantee consistent naming.
>> Thus can only get two drives to match in any config. How do I get out
>> of this mess?
> After doing a lot more reading... I think I'm getting down to running
> something like this. assemble force isn't doing much.
This throwaway line is critical. --assemble --force is the right answer
to this situation, and if its not working, something else should be
investigated. Do *not* use --create.
Show your kernel and mdadm versions. Show the content of /proc/mdstat.
Show the output of:
mdadm --assemble --force --verbose /dev/mdX /dev/sd[bkop]
and the tail of "dmesg" that corresponds to the above.
> mdadm --create --assume-clean --level=5 --verbose --chunk 512K
> --raid-devices=4 /dev/md3 /dev/sdb /dev/sdk /dev/sdo /dev/sdp
>
> But as per the big warning... says to check with you guys first, also
> need help writing the command (since it seems to be a one shot type
> thing)
>
> Here's the key information I think I need from the examine:
>
> Raid level: 5
> Chunk Size: 512K
> Used Dev Size: 7813774336
> /dev/sdb: AA.. (state: clean - active device 0)
> /dev/sdk: AAAA (state: active - active device 1)
> /dev/sdo: A.AA (state: clean - active device 2)
> /dev/sdp: A.AA (state: clean - active device 3)
Oh, and this isn't nearly enough information to advise on --create, in
the remote chance it turns out to be the right answer.
Phil
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: G. Michael Carter @ 2015-02-10 1:36 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <54D94D3C.3060808@turmel.org>
Was actually dreading the complete info question, as the machine has
no CD/DVD, it is my netboot station and holds all my ISO mirrors. But
turns out starting it's network interface from the emergency shell was
easier than I thought.
---- uname
Linux andromeda 3.16.6-203.fc20.x86_64 #1 SMP Sat Oct 25 12:44:32 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
---- mdadm -V
mdadm - v3.3 - 3rd September 2013
---- disk-by-id
total 0
lrwxrwxrwx 1 root root 9 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part3 -> ../../sdc3
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part4 -> ../../sdc4
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST2000DL003-9VT166_5YD5QSG3
-> ../../sda
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST2000DL003-9VT166_5YD604E0
-> ../../sde
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-1CH166_Z1F2H9YC
-> ../../sdm
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-9YN166_S1F026CS
-> ../../sdf
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-9YN166_W1F0GD7Y
-> ../../sdn
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-9YN166_W1F0JSVP
-> ../../sdh
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_9XK0A9AD -> ../../sdd
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_9XK0AV1G -> ../../sdl
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_9XK0N7GY -> ../../sdg
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_Z291009B -> ../../sdi
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_Z2911DKS -> ../../sdj
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W3009GE3
-> ../../sdp
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W300E08A
-> ../../sdk
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300PYF2
-> ../../sdb
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300Q4YS
-> ../../sdo
lrwxrwxrwx 1 root root 9 Feb 9 20:16
lvm-pv-uuid-fIZX8P-yR3l-KR3t-b206-KiDM-PhpK-JwLevw -> ../../md0
lrwxrwxrwx 1 root root 9 Feb 9 20:16
lvm-pv-uuid-Uvrg7m-X8Hr-JJn9-CAlu-Lno6-K3rp-A2p6fj -> ../../md1
lrwxrwxrwx 1 root root 9 Feb 9 20:16
lvm-pv-uuid-XetEWl-bxcb-WOHF-jNIg-M10Q-xxz1-HDeCwH -> ../../md2
lrwxrwxrwx 1 root root 9 Feb 9 20:16 md-name-andromeda:0 -> ../../md0
lrwxrwxrwx 1 root root 9 Feb 9 20:16 md-name-andromeda:1 -> ../../md1
lrwxrwxrwx 1 root root 9 Feb 9 20:16 md-name-andromeda:2 -> ../../md2
lrwxrwxrwx 1 root root 9 Feb 9 20:16
md-uuid-569d52c7:91ba146a:2dc88abf:1dbd4f12 -> ../../md0
lrwxrwxrwx 1 root root 9 Feb 9 20:16
md-uuid-8b1dbda6:fc378fa5:774dcb4f:c273dca5 -> ../../md1
lrwxrwxrwx 1 root root 9 Feb 9 20:16
md-uuid-fb065d4d:c906243c:945b8291:73539d13 -> ../../md2
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5002da066d5 -> ../../sdd
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5002dad7e24 -> ../../sdl
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50036281493 -> ../../sdi
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50036406c91 -> ../../sdj
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50044338d40 -> ../../sde
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50045add261 -> ../../sda
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5004a1255d8 -> ../../sdf
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5004ffa2772 -> ../../sdm
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50050dd4721 -> ../../sdn
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c500510652c5 -> ../../sdh
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c500608925d1 -> ../../sdp
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5006434f070 -> ../../sdb
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50064360fff -> ../../sdo
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5e83a97f4233045f -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part1 ->
../../sdc1
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part2 ->
../../sdc2
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part3 ->
../../sdc3
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part4 ->
../../sdc4
---- mdadm -E
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : clean
Device UUID : e9e7af60:e1bc1c7f:107157b4:4099c48a
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:13:20 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 57eaf19 - correct
Events : 158964
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : active
Device UUID : ed09357e:36655c6d:b7430500:63d5e540
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:10:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 131587ad - correct
Events : 158964
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : clean
Device UUID : de849807:80d7f071:9909f3f2:78022d94
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:11:55 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e73f681f - correct
Events : 158962
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : clean
Device UUID : 67fcd9a9:432b0c8b:178cc556:67b003b3
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:11:55 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 254563d8 - correct
Events : 158962
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
---- blkid
/dev/sda: UUID="8b1dbda6-fc37-8fa5-774d-cb4fc273dca5"
UUID_SUB="82a7750b-74d9-c901-766e-7802b2737d3e" LABEL="andromeda:1"
TYPE="linux_raid_member"
/dev/sdb: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="e9e7af60-e1bc-1c7f-1071-57b44099c48a" LABEL="andromeda:3"
TYPE="linux_raid_member"
/dev/sdc: PTUUID="99e8af68-ba3a-4343-9751-3217c4e3d9a1" PTTYPE="gpt"
/dev/sdc1: PARTUUID="69de1013-cf8a-4131-bdab-03705decd42e"
/dev/sdc2: UUID="944c019a-4fb0-434d-b998-361d90352230" TYPE="ext4"
PARTLABEL="ext4" PARTUUID="cf7f5556-567e-450a-9712-e3d2bc735537"
/dev/sdc3: UUID="9dbf5b08-afdd-47c5-b92b-5350dc26524c" TYPE="ext4"
PARTUUID="46335bee-eea2-441a-8422-207b5c5fa44c"
/dev/sdc4: UUID="7407b6b8-517f-49ca-aac0-01120d98fcdc" TYPE="swap"
PARTUUID="f14bdff5-23f9-42f0-a6e3-b859076ca0bc"
/dev/sdd: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="9786712b-834a-47c1-4402-a4eba529e89e" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sde: UUID="8b1dbda6-fc37-8fa5-774d-cb4fc273dca5"
UUID_SUB="44280e97-717e-09d5-92f5-3af95a5c7364" LABEL="andromeda:1"
TYPE="linux_raid_member"
/dev/sdf: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="fdb635cc-f399-943e-bbb0-67809f0ac896" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdg: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="08104eec-720d-4cae-ed0e-3ef06d4938ff" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdh: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="49b3eafa-f4ab-2bb3-7faa-6996dfc4bb00" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdi: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="a70e7fa1-6909-b030-e899-0f85555f0094" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdj: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="c74f4f4f-0b4b-65d4-b8e9-90ce689678c8" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdk: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="ed09357e-3665-5c6d-b743-050063d5e540" LABEL="andromeda:3"
TYPE="linux_raid_member"
/dev/sdl: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="bd04495d-ce3b-09c7-0cac-37b98f74b3a7" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdm: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="5c6e4d63-3f4d-a05e-d64e-62b339f4f767" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdn: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="1b2773d1-391b-3c78-42e0-fd405416d9e7" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdo: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="de849807-80d7-f071-9909-f3f278022d94" LABEL="andromeda:3"
TYPE="linux_raid_member"
/dev/sdp: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="67fcd9a9-432b-0c8b-178c-c55667b003b3" LABEL="andromeda:3"
TYPE="linux_raid_member"
---- dmesg
[ 576.890380] md: md3 stopped.
[ 576.891629] md: unbind<sdb>
[ 576.900841] md: export_rdev(sdb)
[ 576.902147] md: unbind<sdo>
[ 576.906824] md: export_rdev(sdo)
[ 576.908029] md: unbind<sdk>
[ 576.911845] md: export_rdev(sdk)
[ 576.913030] md: unbind<sdp>
[ 576.916862] md: export_rdev(sdp)
[ 585.164936] md: md3 stopped.
[ 585.360571] md: bind<sdk>
[ 585.360833] md: bind<sdp>
[ 585.361050] md: bind<sdo>
[ 585.361261] md: bind<sdb>
[ 585.361294] md: md3 stopped.
[ 585.361298] md: unbind<sdb>
[ 585.386616] md: export_rdev(sdb)
[ 585.387767] md: unbind<sdo>
[ 585.398644] md: export_rdev(sdo)
[ 585.399633] md: unbind<sdp>
[ 585.404662] md: export_rdev(sdp)
[ 585.405681] md: unbind<sdk>
[ 585.410647] md: export_rdev(sdk)
---- mdadm -A output
mdadm: looking for devices for /dev/md3
mdadm: /dev/sdb is identified as a member of /dev/md3, slot 0.
mdadm: /dev/sdk is identified as a member of /dev/md3, slot 1.
mdadm: /dev/sdo is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdp is identified as a member of /dev/md3, slot 2.
mdadm: added /dev/sdk to /dev/md3 as 1
mdadm: added /dev/sdp to /dev/md3 as 2 (possibly out of date)
mdadm: added /dev/sdo to /dev/md3 as 3 (possibly out of date)
mdadm: added /dev/sdb to /dev/md3 as 0
mdadm: /dev/md3 assembled from 2 drives - not enough to start the array.
---- /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid1 sda[0] sde[1]
1953383488 blocks super 1.2 [2/2] [UU]
bitmap: 0/15 pages [0KB], 65536KB chunk
md0 : active raid5 sdl[0] sdn[3] sdd[5] sdm[1]
8790405120 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md2 : active raid6 sdj[0] sdi[1] sdg[3] sdh[2]
5860270080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
** md3 is shutdown which is why I'm assuming it's not there.
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: Phil Turmel @ 2015-02-10 2:04 UTC (permalink / raw)
To: G. Michael Carter; +Cc: linux-raid
In-Reply-To: <CANjusZdgzDxN4ZxsXFc20OU1Y9_LoScYQcZmffyRGHWYUviY0Q@mail.gmail.com>
Hi Michael,
On 02/09/2015 08:36 PM, G. Michael Carter wrote:
> Was actually dreading the complete info question, as the machine has
> no CD/DVD, it is my netboot station and holds all my ISO mirrors. But
> turns out starting it's network interface from the emergency shell was
> easier than I thought.
Almost all good livecds can be put on a thumb drive to boot from,
instead of using a real CD. I highly recommend sysrescuecd.org, FWIW.
> ---- uname
> Linux andromeda 3.16.6-203.fc20.x86_64 #1 SMP Sat Oct 25 12:44:32 UTC
> 2014 x86_64 x86_64 x86_64 GNU/Linux
Not too old, good. There been a steady stream of small bugfixes since
3.16. I'm not sure what fedora's been backporting.
> ---- mdadm -V
> mdadm - v3.3 - 3rd September 2013
Bugfixes to this, too.
> ---- disk-by-id
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W3009GE3
> -> ../../sdp
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W300E08A
> -> ../../sdk
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300PYF2
> -> ../../sdb
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300Q4YS
> -> ../../sdo
I think you missed the 'excerpt' part, but no harm done. Anyways, if I
recall Seagate model numbering (past misery), these are green drives.
Very bad for raid service. After we revive your array, you'll want to
do some reading on 'timeout mismatch'.[1] (You are dangerously close to
option "D" there, and option "C" is your only choice w/ green drives.)
> ---- mdadm -E
> /dev/sdb:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : clean
> Device UUID : e9e7af60:e1bc1c7f:107157b4:4099c48a
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:13:20 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 57eaf19 - correct
> Events : 158964
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdk:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : active
> Device UUID : ed09357e:36655c6d:b7430500:63d5e540
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:10:53 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 131587ad - correct
> Events : 158964
matching events.
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdo:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : clean
> Device UUID : de849807:80d7f071:9909f3f2:78022d94
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:11:55 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : e73f681f - correct
> Events : 158962
Events off by two. Pretty minor.
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdp:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : clean
> Device UUID : 67fcd9a9:432b0c8b:178cc556:67b003b3
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:11:55 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 254563d8 - correct
> Events : 158962
Also off by two. Again, minor.
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
> ---- dmesg
> [ 576.890380] md: md3 stopped.
> [ 576.891629] md: unbind<sdb>
> [ 576.900841] md: export_rdev(sdb)
> [ 576.902147] md: unbind<sdo>
> [ 576.906824] md: export_rdev(sdo)
> [ 576.908029] md: unbind<sdk>
> [ 576.911845] md: export_rdev(sdk)
> [ 576.913030] md: unbind<sdp>
> [ 576.916862] md: export_rdev(sdp)
> [ 585.164936] md: md3 stopped.
> [ 585.360571] md: bind<sdk>
> [ 585.360833] md: bind<sdp>
> [ 585.361050] md: bind<sdo>
> [ 585.361261] md: bind<sdb>
> [ 585.361294] md: md3 stopped.
> [ 585.361298] md: unbind<sdb>
> [ 585.386616] md: export_rdev(sdb)
> [ 585.387767] md: unbind<sdo>
> [ 585.398644] md: export_rdev(sdo)
> [ 585.399633] md: unbind<sdp>
> [ 585.404662] md: export_rdev(sdp)
> [ 585.405681] md: unbind<sdk>
> [ 585.410647] md: export_rdev(sdk)
> ---- mdadm -A output
> mdadm: looking for devices for /dev/md3
> mdadm: /dev/sdb is identified as a member of /dev/md3, slot 0.
> mdadm: /dev/sdk is identified as a member of /dev/md3, slot 1.
> mdadm: /dev/sdo is identified as a member of /dev/md3, slot 3.
> mdadm: /dev/sdp is identified as a member of /dev/md3, slot 2.
> mdadm: added /dev/sdk to /dev/md3 as 1
> mdadm: added /dev/sdp to /dev/md3 as 2 (possibly out of date)
> mdadm: added /dev/sdo to /dev/md3 as 3 (possibly out of date)
> mdadm: added /dev/sdb to /dev/md3 as 0
> mdadm: /dev/md3 assembled from 2 drives - not enough to start the array.
Please redo this with an explicit command line so we can see what's
going on:
mdadm --assemble --force --verbose /dev/md3 /dev/sd[bkop]
> ---- /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1]
> md1 : active raid1 sda[0] sde[1]
> 1953383488 blocks super 1.2 [2/2] [UU]
> bitmap: 0/15 pages [0KB], 65536KB chunk
>
> md0 : active raid5 sdl[0] sdn[3] sdd[5] sdm[1]
> 8790405120 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md2 : active raid6 sdj[0] sdi[1] sdg[3] sdh[2]
> 5860270080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> unused devices: <none>
>
> ** md3 is shutdown which is why I'm assuming it's not there.
Yup. If --assemble --force doesn't work with your installed OS,
temporarily boot from a recent system rescue cd and do the above over
again (especially the /dev/disk/by-id excerpt).
Phil
[1] http://marc.info/?l=linux-raid&m=135811522817345&w=1
More history:
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=139050322510249&w=2
You might want to read more from those threads than just the mails I've
pointed out...
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox