* Raid 5: all devices marked spare, cannot assemble
From: Paul Boven @ 2015-03-12 12:21 UTC (permalink / raw)
To: linux-raid
Hi folks,
I have a rather curious issue with one of our storage machines. The
machine has 36x 4TB disks (SuperMicro 847 chassis) which are divided
over 4 dual SAS-HBAs and the on-board SAS. These disks are in RAID5
configurations, 6 raids of 6 disks each. Recently the machine ran out of
memory (it has 32GB, and no swapspace as it boots from SATA-DOM) and the
last entries in the syslog are from the OOM-killer. The machine is
running Ubuntu 14.04.02 LTS, mdadm 3.2.5-5ubuntu4.1.
After doing a hard reset, the machine booted fine but one of the raids
needed to resync. Worse, another of the raid5s will not assemble at all.
All the drives are marked SPARE. Relevant output from /proc/mdstat (one
working and the broken array):
md14 : active raid5 sdc1[2] sdag1[6] sde1[4] sdi1[3] sdz1[0] sdu1[1]
19534425600 blocks super 1.2 level 5, 512k chunk, algorithm 2
[6/6] [UUUUUU]
md15 : inactive sdd1[6](S) sdad1[0](S) sdy1[3](S) sdv1[4](S) sdm1[2](S)
sdq1[1](S)
23441313792 blocks super 1.2
Using 'mdadm --examine' on each of the drives from the broken md15, I get:
sdd1: Spare, Events: 0
sdad1: Active device 0, Events 194
sdy1: Active device 3, Events 194
sdv1: Active device 4, Events 194
sdm1: Active device 2, Events 194
sdq1: Active device 1, Events 194
This numbering corresponds to how the raid5 was created when I installed
the machine:
mdadm --create /dev/md15 -l 5 -n 6 /dev/sdad1 /dev/sdq1 /dev/sdm1
/dev/sdy1 /dev/sdv1 /dev/sdd1
Possible clues from /var/log/syslog:
md/raid:md13: not clean -- starting background reconstruction
(at 14 seconds uptime).
md15 isn't even mentioned in the boot-time syslog, only once I manually
try to assemble it did I get these errors:
md: kicking non-fresh sdd1 from array!
md: unbind<sdd1>
md: export_rdev(sdd1)
md/raid:md15: not clean -- starting background reconstruction
md/raid:md15: device sdy1 operational as raid disk 3
md/raid:md15: device sdv1 operational as raid disk 4
md/raid:md15: device sdad1 operational as raid disk 0
md/raid:md15: device sdq1 operational as raid disk 1
md/raid:md15: device sdm1 operational as raid disk 2
md/raid:md15: allocated 0kB
md/raid:md15: cannot start dirty degraded array.
RAID conf printout:
--- level:5 rd:6 wd:5
disk 0, o:1, dev:sdad1
disk 1, o:1, dev:sdq1
disk 2, o:1, dev:sdm1
disk 3, o:1, dev:sdy1
disk 4, o:1, dev:sdv1
md/raid:md15: failed to run raid set.
md: pers->run() failed ...
So the questions I'd like to pose are:
* Why does this raid5 not assemble? Only one drive (sdd) seems to be
missing (marked spare), although I see no real issues with it and can
read from it fine. There should still be enough drives to start the array.
# mdadm --assemble /dev/md15 --run
Returns without any error message, but leaves /proc/mdstat unchanged.
* How can the data be recovered, and the machine brought into production
again
And of course
* What went wrong, and how can we guard against this?
Any insights and help are much appreciated.
Regards, Paul Boven.
--
Paul Boven <boven@jive.nl> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
^ permalink raw reply
* (unknown),
From: pepa6.es @ 2015-03-12 11:49 UTC (permalink / raw)
Proposal,
Respond to my personal email; mrs.zhangxiao1962@outlook.
com
Yours Sincerely.
Mrs. Zhang Xiao (Accounts book Keeper)
Angang
Steel Company Limited
396 Nan Zhong Hua Lu, Tie Dong District Anshan,
Liaoning 114021, China.
^ permalink raw reply
* Re: dm log userspace: use mempool_create_kmalloc_pool()
From: Mike Snitzer @ 2015-03-11 12:58 UTC (permalink / raw)
To: David Rientjes
Cc: Alasdair Kergon, Neil Brown, linux-kernel, linux-raid, dm-devel
In-Reply-To: <alpine.DEB.2.10.1503071642540.2772@chino.kir.corp.google.com>
On Sat, Mar 07 2015 at 7:43pm -0500,
David Rientjes <rientjes@google.com> wrote:
> Mempools created for kmalloc caches should use
> mempool_create_kmalloc_pool().
>
> Cc: Alasdair Kergon <agk@redhat.com>
> Cc: Mike Snitzer <snitzer@redhat.com>
> Cc: Neil Brown <neilb@suse.de>
> Signed-off-by: David Rientjes <rientjes@google.com>
Thanks David, I'll get this queued up.
^ permalink raw reply
* mdraid with device-mapper multipath devices
From: Sushma Gurram @ 2015-03-10 1:43 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi,
Let's say I have a system with two multipath SSDs (dm-0 and dm-1) and a raid0 is created using both.
When the underlying drive for dm-0 is accidentally removed, the multipathd cannot flush the multipath map (because it's being used by mdraid) and hence the dm-0 entry continues to exist and hence mdraid also has the dm-0 entry in "active sync" state. However, I/O to dm-0 would fail since there is no drive underneath, while I/O to dm-1 would succeed.
Now if the failed drive is reinserted, device mapper/multipath would update the dm-0 maps and now dm-0 handles a valid/existing drive. Since mdraid still thinks dm-0 is active sync, I/O continues as if raid0 array was clean.
I haven't tried with other raid levels, but I suspect a similar behavior.
Here are prints in kernel log on drive removal, with drive being part of raid config and without it being part of config
***Removal with dm-X configured as path of md****
Mar 9 17:57:07 ion-ws1lvp4w kernel: [ 2173.868995] mpt2sas0: removing handle(0x0012), sas_addr(0x50011731001395ea)
Mar 9 17:57:07 ion-ws1lvp4w multipathd: sdq: remove path (uevent)
Mar 9 17:57:08 ion-ws1lvp4w multipathd: mpathh: map in use
Mar 9 17:57:08 ion-ws1lvp4w multipathd: mpathh: can't flush
Mar 9 17:57:08 ion-ws1lvp4w multipathd: mpathh: load table [0 3750748848 multipath 0 0 0 0]
Mar 9 17:57:08 ion-ws1lvp4w multipathd: sdq: path removed from map mpathh
***Removal with no configuration***
Mar 9 18:01:32 ion-ws1lvp4w kernel: [ 2438.314508] mpt2sas0: removing handle(0x0012), sas_addr(0x50011731001395ea)
Mar 9 18:01:33 ion-ws1lvp4w multipathd: 65:0: mark as failed
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: remaining active paths: 0
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391320] device-mapper: multipath: Failing path 65:0.
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391336] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391341] quiet_error: 7 callbacks suppressed
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391344] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391385] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391391] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w multipathd: sdq: remove path (uevent)
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395616] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395619] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395638] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395640] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.396691] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.396697] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: map flushed
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: stop event checker thread (140284200109824)
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: removed map after removing all paths
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: adding map
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: devmap dm-7 added
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: adding map
Ideally the dm-X (multipath device) should be removed so that mdraid updates its constituent drive information. dm-X is not being removed because its in use by raid. Any insight into how we could fix this issue?
Thanks,
Sushma
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply
* Re: Raid 10 Issue - Swapping Data from Array to Array [SOLVED]
From: Stefan Lamby @ 2015-03-09 8:51 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <54F9A421.2020907@turmel.org>
Hi Phil.
I did what you suggested and ended up with a working raid 10 array.
Thank you so much for your support and time.
God blees you.
Stefan
> Phil Turmel <philip@turmel.org> hat am 6. März 2015 um 13:57 geschrieben:
>
>
> On 03/06/2015 05:09 AM, Stefan Lamby wrote:
> >
> >
> > This is what I got right now.
> > What do you recommend to do?
>
> 1) pvcreate on the new array
> 2) vgextend to add the new array to the volume group
> 3) pvmove to get the data into the new array
> 4) vgreduce to disconnect the old array from lvm
> 5) pvremove to wipe the old array's lvm meta
> 6) stop the old array
> 7) mdadm --zero-superblock to clear the old members
> 8) mdadm --add to put those members into the new array
>
> Regards,
>
> Phil Turmel
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid 10 Issue
From: Wilson, Jonathan @ 2015-03-08 14:59 UTC (permalink / raw)
To: Phil Turmel; +Cc: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <54F8B792.9060206@turmel.org>
On Thu, 2015-03-05 at 15:07 -0500, Phil Turmel wrote:
> On 03/05/2015 12:56 PM, Stefan Lamby wrote:
> > Hello List.
> >
> > I was setting up a new machine using ubuntu 14.04.02 lts using its installer,
> > configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks and
> > now I like to add 2 more disks to the array so i want to end up with 4 disks, no
> > spare.
> >
> > Searching the internet I found that I am not able to --grow the array with the
> > mdadm version this ubuntu is using (v3.2.5).
> > Is that right?
> >
> > So I decided to build a new array that way and try to move my data afterwards,
> > which failed:
> > (Is it OK to do it that way or do you recommend another?)
>
> No, you should be able to do this. Probably without any shutdown.
> Please show the full layout of your drives, partitions, and lvm.
>
> I suggest lsdrv[1] for working layouts. If your email is set to use
> utf8, just paste the result in a reply.
>
> Regards,
>
> Phil Turmel
>
> [1] https://github.com/pturmel/lsdrv
OT: What a fantastic little script, a couple of my biggest annoyances
with my set up was that it was a pain to track "os disk designation" to
"serial no." making identifying the physical device a chore.
Also one of my cheap "4 port sata" cards doesn't identify port number on
the device and while it was possible to work it out by tracing device
serial-sata cable-socket then mentally comparing sd*-device serial=port
it was something I was putting off as "a pain" but also being bugged by
my md member devices numbers being out of whack with my sd designations:
sda[4] sdb[2] sdc[1] sdd[3] and with 12 devices the last thing you want
when something is failing is to have to hunt around to work out which
device physically relates to which os identified device.
With this I can now easily mark my disks with a simple set of
identification, likewise the cable on both ends, and make a sketch of
ports and their sequences and which disk in which slot relates.
Obviously if I was starting my system from scratch I would have done all
this from the get go... but after 4 cases, various upgrades and
additions, differing numbers of MB sata ports, and various sata cards
and multiple variations of md's/partition layouts it all became a huge
muddle and one elegant script has made it simple to finally do what I
have put of for far to long.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: two raid issues
From: Phil Turmel @ 2015-03-08 13:52 UTC (permalink / raw)
To: Dave Stevens, linux-raid
In-Reply-To: <20150307144256.17321vlciup2v4kw@webmail.uniserve.com>
Good morning Dave,
On 03/07/2015 05:42 PM, Dave Stevens wrote:
> Hello the raid list,
>
> I have inherited a server set up by people who are no longer around. It
> worked fine until recently and then after a routine update refused to
> boot. I've got the machine in my office and have been examining the
> problem, or rather problems, I think there are two.
Three, at least.
> First, the bootable partition on /dev/sda1 won't successfully boot to a
> xen kernel, kernel version is 2.6.18.something-xen. The intent is to
> boot to a raid-10 array of four 750GB drives, each partitioned into a
> small and a large partition as detailed below.
>
> Boot proceeds normally according to on-screen messages until this:
>
> md: md0: raid array is not clean -- starting background reconstruction
> raid10: not enough operational mirrors for md0
> md: pers -> () failed
Yup. Degraded to the point of not running.
> Immediately after these messages is another stating that an attempt has
> been made to kill init, kernel panic and reboot.
No way to pivot to your root filesystem, so your initramfs gives up.
Depending on the distro, it may be possible to pass a kernel command
line option to drop into a repair shell at that point.
> I've tried to give these messages verbatim but have no way (I think) to
> reproduce them other than manually.
Repair shell, if available.
> So at first I looked around, read through the wiki and found advice to
> NOT write anything to the array, which seems reasonable. I looked at a
> live microknoppix distro called runtime live that gave me a command
> shell to run the examine command with the output below:
LiveCD boot is good, and the following report is very detailed, thanks:
> /dev/sda2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Wed Feb 18 19:28:22 2015
> State : active
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 2
> Spare Devices : 0
> Checksum : 7c76593b - correct
> Events : 32945477
This is important: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 0 8 2 0 active sync /dev/sda2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 0 0 1 active sync
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdb2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Sat Nov 22 13:18:12 2014
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 7d850ca4 - correct
> Events : 32945477
With this: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 1 8 18 1 active sync /dev/sdb2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdc2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Wed Feb 18 19:30:13 2015
> State : active
> Active Devices : 1
> Working Devices : 1
> Failed Devices : 2
> Spare Devices : 0
> Checksum : 7c7659de - correct
> Events : 32945479
And this: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 2 8 34 2 active sync /dev/sdc2
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdd2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 5
> Preferred Minor : 0
>
> Update Time : Wed Sep 4 07:51:50 2013
> State : active
> Active Devices : 4
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 77c1a3a7 - correct
> Events : 53
Whoa! ^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 50 3 active sync /dev/sdd2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 50 3 active sync /dev/sdd2
> 4 4 8 66 4 spare /dev/sde2
> /dev/sde2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 5
> Preferred Minor : 0
>
> Update Time : Sun Sep 8 13:25:42 2013
> State : clean
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
> Checksum : 77c775be - correct
> Events : 7934
And Whoa again! ^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 66 3 active sync /dev/sde2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 66 3 active sync /dev/sde2
>
> This makes sense to me as far as it goes but I don't see what to do
> next. As I understand it the four partitions from sda2 to sdd2 would
> form the array with sde as hot spare. It has been my assumption that if
> a drive failed that sde would sync and take over. I don't know if this
> is in fact the case and don't see a path forward. Of course the backups
> are inadequate.
Based on the events and update timestamps, sdd died sometime around Wed
Sep 4 07:51:50 2013, at which point sde stepped in. It too failed
shortly after ~ Sun Sep 8 13:25:42 2013. You then ran degraded for over
a year until sdb also failed ~ Sat Nov 22 13:18:12 2014. You were then
running doubly-degraded (luckily on non-adjacent members) until this Feb
14 when sda was booted out. Leaving only one running drive.
{ I wouldn't keep such people around, either. }
Your best bet is to force assembly of the last two working drives to get
the system running, then take an immediate backup of all critical files.
Do the forced assembly with the livecd, then do a clean shutdown. You
should then be able to boot the original OS and take your backup.
Then you need to completely rebuild your system with proper log
monitoring, array monitoring, and verification of your drives.
Phil
^ permalink raw reply
* [patch] dm log userspace: use mempool_create_kmalloc_pool()
From: David Rientjes @ 2015-03-08 0:43 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer
Cc: Neil Brown, linux-kernel, linux-raid, dm-devel
Mempools created for kmalloc caches should use
mempool_create_kmalloc_pool().
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Neil Brown <neilb@suse.de>
Signed-off-by: David Rientjes <rientjes@google.com>
---
drivers/md/dm-log-userspace-base.c | 19 ++++---------------
1 file changed, 4 insertions(+), 15 deletions(-)
diff --git a/drivers/md/dm-log-userspace-base.c b/drivers/md/dm-log-userspace-base.c
--- a/drivers/md/dm-log-userspace-base.c
+++ b/drivers/md/dm-log-userspace-base.c
@@ -74,18 +74,6 @@ struct log_c {
uint32_t integrated_flush;
};
-static mempool_t *flush_entry_pool;
-
-static void *flush_entry_alloc(gfp_t gfp_mask, void *pool_data)
-{
- return kmalloc(sizeof(struct flush_entry), gfp_mask);
-}
-
-static void flush_entry_free(void *element, void *pool_data)
-{
- kfree(element);
-}
-
static int userspace_do_request(struct log_c *lc, const char *uuid,
int request_type, char *data, size_t data_size,
char *rdata, size_t *rdata_size)
@@ -537,6 +525,8 @@ static int flush_by_group(struct log_c *lc, struct list_head *flush_list,
return r;
}
+static mempool_t *flush_entry_pool;
+
/*
* userspace_flush
*
@@ -886,9 +876,8 @@ static int __init userspace_dirty_log_init(void)
{
int r = 0;
- flush_entry_pool = mempool_create(100, flush_entry_alloc,
- flush_entry_free, NULL);
-
+ flush_entry_pool = mempool_create_kmalloc_pool(100,
+ sizeof(struct flush_entry));
if (!flush_entry_pool) {
DMWARN("Unable to create flush_entry_pool: No memory.");
return -ENOMEM;
^ permalink raw reply
* two raid issues
From: Dave Stevens @ 2015-03-07 22:42 UTC (permalink / raw)
To: linux-raid
Hello the raid list,
I have inherited a server set up by people who are no longer around.
It worked fine until recently and then after a routine update refused
to boot. I've got the machine in my office and have been examining the
problem, or rather problems, I think there are two.
First, the bootable partition on /dev/sda1 won't successfully boot to
a xen kernel, kernel version is 2.6.18.something-xen. The intent is to
boot to a raid-10 array of four 750GB drives, each partitioned into a
small and a large partition as detailed below.
Boot proceeds normally according to on-screen messages until this:
md: md0: raid array is not clean -- starting background reconstruction
raid10: not enough operational mirrors for md0
md: pers -> () failed
Immediately after these messages is another stating that an attempt
has been made to kill init, kernel panic and reboot.
I've tried to give these messages verbatim but have no way (I think)
to reproduce them other than manually.
So at first I looked around, read through the wiki and found advice to
NOT write anything to the array, which seems reasonable. I looked at a
live microknoppix distro called runtime live that gave me a command
shell to run the examine command with the output below:
/dev/sda2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Wed Feb 18 19:28:22 2015
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Checksum : 7c76593b - correct
Events : 32945477
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 0 8 2 0 active sync /dev/sda2
0 0 8 2 0 active sync /dev/sda2
1 1 0 0 1 active sync
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
/dev/sdb2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sat Nov 22 13:18:12 2014
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 7d850ca4 - correct
Events : 32945477
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 1 8 18 1 active sync /dev/sdb2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
/dev/sdc2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Wed Feb 18 19:30:13 2015
State : active
Active Devices : 1
Working Devices : 1
Failed Devices : 2
Spare Devices : 0
Checksum : 7c7659de - correct
Events : 32945479
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 2 8 34 2 active sync /dev/sdc2
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
/dev/sdd2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Wed Sep 4 07:51:50 2013
State : active
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Checksum : 77c1a3a7 - correct
Events : 53
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 3 8 50 3 active sync /dev/sdd2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 8 50 3 active sync /dev/sdd2
4 4 8 66 4 spare /dev/sde2
/dev/sde2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Sun Sep 8 13:25:42 2013
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 77c775be - correct
Events : 7934
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 3 8 66 3 active sync /dev/sde2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 8 66 3 active sync /dev/sde2
This makes sense to me as far as it goes but I don't see what to do
next. As I understand it the four partitions from sda2 to sdd2 would
form the array with sde as hot spare. It has been my assumption that
if a drive failed that sde would sync and take over. I don't know if
this is in fact the case and don't see a path forward. Of course the
backups are inadequate.
Any and all ideas welcome.
Dave
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: Raid 10 Issue - Booting in case raid failed
From: Phil Turmel @ 2015-03-06 20:12 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <322760937.22899.1425668800663.JavaMail.open-xchange@app04.ox.hosteurope.de>
On 03/06/2015 02:06 PM, Stefan Lamby wrote:
> Hi list.
>
> If everything will work out OK, I will end up with an raid 10 array with 4
> devices.
>
> My partition design and layout structure will be found at the end, if needed.
>
> There are a few questions left for me in case I have to boot with a failed disk:
>
> 1) As you might have seen from the partition design, only partition sda1 has the
> boot flag set. As far as I guess, the ubuntu installer was using grub-install
> only for sda. I am kind of afraid what will happen, in case sda will fail in the
> future. Will it be a good idea to grub-install to all the other devices also?
Yes, sort of.
> 2) What about the boot flag, if I need to grub-install the other devices also?
> Should it be O or 1? Do I have to leave it set to false and in case things go
> wrong boot from a live cd and set it to on to boot from another device?
Set to true.
But the "sort of" comes from your reliance on grub support for MD raid,
and having the appropriate mirrors containing the boot folder. You've
left enough space before your first partition (48g) to easily hold a
plain raid1 x4 boot partition and a raid6 root partition (use a small
chunk size for that). Then your system could boot with any two drives
missing, and let you know what's possible with the large raid10.
Also note that this kind of boot redundancy only helps if the bad drive
is entirely missing at boot time. If you really need boot redundancy,
you have fewer choices: BIOS fakeraid, or hardware raid with a BIOS
extension, or EFI boot with a monolithic kernel/initramfs on each device.
HTH,
Phil
^ permalink raw reply
* Re: raid 10 - Recovering raid - what stats are most helpfull?
From: Phil Turmel @ 2015-03-06 19:50 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <270478433.23047.1425669790387.JavaMail.open-xchange@app04.ox.hosteurope.de>
On 03/06/2015 02:23 PM, Stefan Lamby wrote:
> What information about the raid would you suggest to collect and to save
> somewhere to have it handy in case it is needed?
Most important is the complete "mdadm -E" output for every member
device. With that you need a list that correlates the member device
names to the underlying drive serial numbers. An *excerpt* from "ls -l
/dev/disk/by-id/" would suffice. Or lsdrv output. I wrote lsdrv to
generate a functional summary that includes all of the useful serial
numbers.
Regards,
Phil Turmel
--
200 Creekwood Trail
Fayetteville, GA 30214
Mobile: (404) 713-7284
Home: (770) 719-0684
^ permalink raw reply
* Re: raid 10 - Recovering raid - what stats are most helpfull?
From: Stefan Lamby @ 2015-03-06 19:23 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <54F85645.9000603@turmel.org>
> Phil Turmel <philip@turmel.org> hat am 5. März 2015 um 14:12 geschrieben:
>
>
> On 03/05/2015 03:18 AM, Stefan Lamby wrote:
> >
> > Hi.
> > Is it OK to ask for support here to this list or is it the wrong place?
> > If so, could you please give a hint where to go?
>
> This is the right place. Post your array details (mdadm -D), member
> details (mdadm -E) and a summary of what happened. Include excerpts
> from your dmesg and/or syslogs that look like they might be relevant.
>
> These should all be text files, so pasting them directly into your email
> would be preferred.
>
> Regards,
>
> Phil Turmel
>
>
Hi Phil,
hello List.
This would be also very interesting - just in case:
Hopefully it will never happen, but if the raid 10 array crashes, what
information will be the most usefull for later recovery? Is this the output of
cat /proc/mdstat or as mentioned by Phil the output of mdadm --detail or what
ever?
I would like to be prepared and write a small shell script which will write this
to a file to be included in the backups.
What information about the raid would you suggest to collect and to save
somewhere to have it handy in case it is needed?
Thank you very much for your help.
Stefan
P.S.: Phil, thank you so much for the hints regarding moving the data. It looks
very promising. I'll know more in detail tomorrow. pvmove is taking all night, I
guess.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid 10 Issue - Booting in case raid failed
From: Stefan Lamby @ 2015-03-06 19:06 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <1315687872.8638.1425636591443.JavaMail.open-xchange@app04.ox.hosteurope.de>
Hi list.
If everything will work out OK, I will end up with an raid 10 array with 4
devices.
My partition design and layout structure will be found at the end, if needed.
There are a few questions left for me in case I have to boot with a failed disk:
1) As you might have seen from the partition design, only partition sda1 has the
boot flag set. As far as I guess, the ubuntu installer was using grub-install
only for sda. I am kind of afraid what will happen, in case sda will fail in the
future. Will it be a good idea to grub-install to all the other devices also?
2) What about the boot flag, if I need to grub-install the other devices also?
Should it be O or 1? Do I have to leave it set to false and in case things go
wrong boot from a live cd and set it to on to boot from another device?
What do you recommend?
Thanks
Stefan
Here is my layout - please do not care
root@kvm15:~# fdisk -l /dev/sda
Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 Köpfe, 63 Sektoren/Spur, 243201 Zylinder, zusammen 3907029168 Sektoren
Einheiten = Sektoren von 1 × 512 = 512 Bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Festplattenidentifikation: 0x00071c2b
Gerät boot. Anfang Ende Blöcke Id System
/dev/sda1 * 98435072 3907028991 1904296960 fd Linux raid autodetect
Here as an example for all other disks
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 Köpfe, 63 Sektoren/Spur, 243201 Zylinder, zusammen 3907029168 Sektoren
Einheiten = Sektoren von 1 × 512 = 512 Bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Festplattenidentifikation: 0x0008624b
Gerät boot. Anfang Ende Blöcke Id System
/dev/sdb1 98435072 3907028991 1904296960 fd Linux raid autodetect
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family)
SATA AHCI Controller
├scsi 2:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M1LPT1AE}
│└sda 1.82t [8:0] Partitioned (dos)
│ └sda1 1.77t [8:1] MD raid10,near2 (0/2) (w/ sdb1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ │ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
│ └VG vg_raid10 5.32t (w/ md10) 3.30t free
{HbjouC-RgUe-YYNB-z2ns-4kzK-RwJH-RHWSWq}
│ ├dm-0 479.39g [252:0] LV home ext4 {2d67d9cc-0378-4669-9d72-7b7c7071dea8}
│ │└Mounted as /dev/mapper/vg_raid10-home @ /home
│ ├dm-1 93.13g [252:1] LV root ext4 {c14e4524-e95c-45c2-bfa0-75d529ed48fe}
│ │└Mounted as /dev/mapper/vg_raid10-root @ /
│ ├dm-4 23.28g [252:4] LV swap swap {9e1a582f-1c88-44a2-be90-aafcb96805c7}
│ ├dm-3 46.56g [252:3] LV tmp ext4 {ac67d0d9-049c-4cf2-9a0e-591cdb6a3559}
│ │└Mounted as /dev/mapper/vg_raid10-tmp @ /tmp
│ └dm-2 393.13g [252:2] LV var ext4 {ff71c558-c1f8-4410-8e2a-dc9c77c27a03}
│ └Mounted as /dev/mapper/vg_raid10-var @ /var
├scsi 3:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5LAR62D}
│└sdb 1.82t [8:16] Partitioned (dos)
│ └sdb1 1.77t [8:17] MD raid10,near2 (1/2) (w/ sda1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
├scsi 4:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M7YA1ANR}
│└sdc 1.82t [8:32] Partitioned (dos)
│ └sdc1 1.77t [8:33] MD raid10,near2 (1/4) (w/ sdd1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
│ └md10 3.55t [9:10] MD v1.2 raid10,near2 (4) active DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
│ │ PV LVM2_member 1.01t used, 2.54t free
{wYV8fH-uOp2-E88P-EIp5-6U33-chYg-S94vvy}
│ └VG vg_raid10 5.32t (w/ md0) 3.30t free
{HbjouC-RgUe-YYNB-z2ns-4kzK-RwJH-RHWSWq}
└scsi 5:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5AFRYVP}
└sdd 1.82t [8:48] Partitioned (dos)
└sdd1 1.77t [8:49] MD raid10,near2 (3/4) (w/ sdc1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
└md10 3.55t [9:10] MD v1.2 raid10,near2 (4) active DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
PV LVM2_member 1.01t used, 2.54t free
{wYV8fH-uOp2-E88P-EIp5-6U33-chYg-S94vvy}
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid 10 Issue - Swapping Data from Array to Array
From: Phil Turmel @ 2015-03-06 12:57 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <1315687872.8638.1425636591443.JavaMail.open-xchange@app04.ox.hosteurope.de>
On 03/06/2015 05:09 AM, Stefan Lamby wrote:
>
>
> This is what I got right now.
> What do you recommend to do?
1) pvcreate on the new array
2) vgextend to add the new array to the volume group
3) pvmove to get the data into the new array
4) vgreduce to disconnect the old array from lvm
5) pvremove to wipe the old array's lvm meta
6) stop the old array
7) mdadm --zero-superblock to clear the old members
8) mdadm --add to put those members into the new array
Regards,
Phil Turmel
^ permalink raw reply
* Re: Raid 10 Issue - Swapping Data from Array to Array
From: Stefan Lamby @ 2015-03-06 10:09 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <54F8B792.9060206@turmel.org>
> Phil Turmel <philip@turmel.org> hat am 5. März 2015 um 21:07 geschrieben:
>
>
> On 03/05/2015 12:56 PM, Stefan Lamby wrote:
> > Hello List.
> >
> > I was setting up a new machine using ubuntu 14.04.02 lts using its
> > installer,
> > configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks
> > and
> > now I like to add 2 more disks to the array so i want to end up with 4
> > disks, no
> > spare.
> >
> > Searching the internet I found that I am not able to --grow the array with
> > the
> > mdadm version this ubuntu is using (v3.2.5).
> > Is that right?
> >
> > So I decided to build a new array that way and try to move my data
> > afterwards,
> > which failed:
> > (Is it OK to do it that way or do you recommend another?)
>
> No, you should be able to do this. Probably without any shutdown.
> Please show the full layout of your drives, partitions, and lvm.
>
> I suggest lsdrv[1] for working layouts. If your email is set to use
> utf8, just paste the result in a reply.
>
> Regards,
>
> Phil Turmel
>
> [1] https://github.com/pturmel/lsdrv
>
Hi Phil.
I like your suggestion using lsdrv. Pretty nice. Here is the output (including
the newly created array):
root@kvm15:~/lsdrv/lsdrv# ./lsdrv
PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2
port SATA IDE Controller #2
├scsi 0:0:0:0 HL-DT-ST DVD-RAM GH60L {K1XA5SF1137}
│└sr0 3.68g [11:0] udf 'UDF_Volume'
└scsi 1:x:x:x [Empty]
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family)
SATA AHCI Controller
├scsi 2:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M1LPT1AE}
│└sda 1.82t [8:0] Partitioned (dos)
│ └sda1 1.77t [8:1] MD raid10,near2 (0/2) (w/ sdb1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ │ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
│ └VG vg_raid10 1.77t 780.45g free {HbjouC-RgUe-YYNB-z2ns-4kzK-RwJH-RHWSWq}
│ ├dm-0 479.39g [252:0] LV home ext4 {2d67d9cc-0378-4669-9d72-7b7c7071dea8}
│ │└Mounted as /dev/mapper/vg_raid10-home @ /home
│ ├dm-1 93.13g [252:1] LV root ext4 {c14e4524-e95c-45c2-bfa0-75d529ed48fe}
│ │└Mounted as /dev/mapper/vg_raid10-root @ /
│ ├dm-4 23.28g [252:4] LV swap swap {9e1a582f-1c88-44a2-be90-aafcb96805c7}
│ ├dm-3 46.56g [252:3] LV tmp ext4 {ac67d0d9-049c-4cf2-9a0e-591cdb6a3559}
│ │└Mounted as /dev/mapper/vg_raid10-tmp @ /tmp
│ └dm-2 393.13g [252:2] LV var ext4 {ff71c558-c1f8-4410-8e2a-dc9c77c27a03}
│ └Mounted as /dev/mapper/vg_raid10-var @ /var
├scsi 3:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5LAR62D}
│└sdb 1.82t [8:16] Partitioned (dos)
│ └sdb1 1.77t [8:17] MD raid10,near2 (1/2) (w/ sda1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
├scsi 4:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M7YA1ANR}
│└sdc 1.82t [8:32] Partitioned (dos)
│ └sdc1 1.77t [8:33] MD raid10,near2 (1/4) (w/ sdd1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
│ └md10 3.55t [9:10] MD v1.2 raid10,near2 (4) clean DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
│ Empty/Unknown
└scsi 5:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5AFRYVP}
└sdd 1.82t [8:48] Partitioned (dos)
└sdd1 1.77t [8:49] MD raid10,near2 (3/4) (w/ sdc1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
└md10 3.55t [9:10] MD v1.2 raid10,near2 (4) clean DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
Empty/Unknown
This is what I got right now.
What do you recommend to do?
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid 10 Issue [SOLVED]
From: Stefan Lamby @ 2015-03-06 9:32 UTC (permalink / raw)
To: Robin Hill; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <20150306085407.GA32079@cthulhu.home.robinhill.me.uk>
> Robin Hill <robin@robinhill.me.uk> hat am 6. März 2015 um 09:54 geschrieben:
>
>
> On Thu Mar 05, 2015 at 06:56:00PM +0100, Stefan Lamby wrote:
>
> > Hello List.
> >
> > I was setting up a new machine using ubuntu 14.04.02 lts using its
> > installer,
> > configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks
> > and
> > now I like to add 2 more disks to the array so i want to end up with 4
> > disks, no
> > spare.
> >
> > Searching the internet I found that I am not able to --grow the array with
> > the
> > mdadm version this ubuntu is using (v3.2.5).
> > Is that right?
> >
> > So I decided to build a new array that way and try to move my data
> > afterwards,
> > which failed:
> > (Is it OK to do it that way or do you recommend another?)
> >
> > root@kvm15:~# mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10
> > missing missing /dev/sdc1 /dev/sdd1
> > mdadm: layout defaults to n2
> > mdadm: layout defaults to n2
> > mdadm: chunk size defaults to 512K
> > mdadm: /dev/sdc1 appears to be part of a raid array:
> > level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> > mdadm: /dev/sdd1 appears to be part of a raid array:
> > level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> > mdadm: size set to 1904165376K
> > Continue creating array? y
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: RUN_ARRAY failed: Input/output error
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<
> > root@kvm15:~#
> >
> IIRC, in a RAID10 setup, the redundant pair is held on adjacent drives.
> You've specified two adjacent drives as missing, so the array cannot be
> run. Try doing:
> mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10 \
> missing /dev/sdc1 missing /dev/sdd1
>
Dang!
One shot one catch, thank you so much!
Dear Maintainer,
please take a one-to-one copy of Robins explaination and print it if another
dumb user like me tries to do the same and is caught also. This could help avoid
a lot of frustration. There are posts in the internet which claim the order of
the given devices is critical.
Thank you all for taking the time to help.
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Last working drive in RAID1
From: Chris @ 2015-03-06 9:21 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150306085222.19dd30ce@notabene.brown>
Am Fri, 6 Mar 2015 08:52:22 +1100
schrieb NeilBrown <neilb@suse.de>:
> > Or, quite possibly, the device is plugged back in, at which point
> > the device name is there for it (as long as you use the same port,
> > of course). In which case the filesystem may very well resume
> > successfully.
>
> I was with you right up to this last point.
> When a device is unplugged and then plugged back in, it will always
> get a new name.
Right, that is what I see when "rotating" disks or connecting to a
docking station. And I did also see the last disk of an external
storage raid not going away (fail) when unplugged.
The times I had devices really break though, I don't think they
triggered an unplug event. They still seemed fully connected but had
motor-start, bus errors or something. In some occasions the failure was
only intermittent, and the device node continued to work again (within
the controller timeout or without a permanent error remaining after a
reset).
So if the last working drive could be marked as failed, when it
actually failed, that would also provide the proper information for
system failover on replicated hosts, in the many cases when a
controller/bus/drive fails without an unplug event. Cases that the udev
rule only idea does not seem to cover.
Regards,
Chris
^ permalink raw reply
* Re: Raid 10 Issue
From: Robin Hill @ 2015-03-06 8:54 UTC (permalink / raw)
To: Stefan Lamby; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <1888475554.226351.1425578160090.JavaMail.open-xchange@app09.ox.hosteurope.de>
[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]
On Thu Mar 05, 2015 at 06:56:00PM +0100, Stefan Lamby wrote:
> Hello List.
>
> I was setting up a new machine using ubuntu 14.04.02 lts using its installer,
> configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks and
> now I like to add 2 more disks to the array so i want to end up with 4 disks, no
> spare.
>
> Searching the internet I found that I am not able to --grow the array with the
> mdadm version this ubuntu is using (v3.2.5).
> Is that right?
>
> So I decided to build a new array that way and try to move my data afterwards,
> which failed:
> (Is it OK to do it that way or do you recommend another?)
>
> root@kvm15:~# mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10
> missing missing /dev/sdc1 /dev/sdd1
> mdadm: layout defaults to n2
> mdadm: layout defaults to n2
> mdadm: chunk size defaults to 512K
> mdadm: /dev/sdc1 appears to be part of a raid array:
> level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> mdadm: /dev/sdd1 appears to be part of a raid array:
> level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> mdadm: size set to 1904165376K
> Continue creating array? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: RUN_ARRAY failed: Input/output error
> <<<<<<<<<<<<<<<<<<<<<<<<<<<
> root@kvm15:~#
>
IIRC, in a RAID10 setup, the redundant pair is held on adjacent drives.
You've specified two adjacent drives as missing, so the array cannot be
run. Try doing:
mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10 \
missing /dev/sdc1 missing /dev/sdd1
HTH,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: Last working drive in RAID1
From: Chris @ 2015-03-05 21:54 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54F8B5D2.8070304@turmel.org>
Am Thu, 05 Mar 2015 15:00:18 -0500
schrieb Phil Turmel <philip@turmel.org>:
> It has to stay there to give errors to the upper layers that are still
> hooked to it. When they are administratively "unhooked", aka
> unmounted or disassociated with mdadm --remove.
>
> Or, quite possibly, the device is plugged back in, at which point the
> device name is there for it (as long as you use the same port, of
> course). In which case the filesystem may very well resume
> successfully.
From reading this it makes sense that the md device stays there, just
as the the physical device nodes. (to give errors, and to recover)
However, as I understood this thread, md does not seem to inform upper
layers or the user (even not through its own --monitor?) properly.
To me, marking the last disk within an array as failed (*within* the
array) just seems to make more sense, so /proc/mdstat actually
iforms about the md error state (and the md device returning errors on
access).
Regards,
Chris
^ permalink raw reply
* Re: Last working drive in RAID1
From: NeilBrown @ 2015-03-05 21:52 UTC (permalink / raw)
To: Phil Turmel; +Cc: Wols Lists, Eric Mei, linux-raid
In-Reply-To: <54F8B5D2.8070304@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 2692 bytes --]
On Thu, 05 Mar 2015 15:00:18 -0500 Phil Turmel <philip@turmel.org> wrote:
> On 03/05/2015 10:55 AM, Wols Lists wrote:
> > Sorry to butt in, but I'm finding this conversation a bit surreal ...
> > take everything I say with a pinch of salt. But the really weird bit
> > was "what does linux do if /dev/sda disappears?"
> >
> > In the old days, with /dev/hd*, the * had a hard mapping to the
> > hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc
> > etc. I think I ran several systems with just hdb and hdd. Not a good
> > idea, but.
> >
> > Nowadays, with sd*, the letter is assigned in order of finding the
> > drive. So if sda is removed, linux moves all the other drives and what
> > was sdb becomes sda.
>
> On reboot, yes. Not live. If /dev/sda has anything using it when
> unplugged, it stays there and gives errors. Existing devices retain
> their names, and new devices are added to the end. Only if all users
> are completely unhooked will a kernel name get re-used live.
>
> > Which is why you're advised now to always refer
> > to drives by their BLKDEV or whatever, as linux provides no guarantees
> > whatsoever about sd*. The blockdev may only be a symlink to whatever
> > the sd*n code of the disk is, but it makes sure you get the disk you
> > want when the sd*n changes under you.
>
> Correct, but once assembled or mounted, the kernel names are locked.
>
> > Equally surreal is the comment about "what does raid1 do with no
> > working devices?". Surely it will do nothing, if there's no spinning
> > rust or whatever underneath it? You can't corrupt it if there's
> > nothing there to corrupt?
>
> It has to stay there to give errors to the upper layers that are still
> hooked to it. When they are administratively "unhooked", aka unmounted
> or disassociated with mdadm --remove.
>
> Or, quite possibly, the device is plugged back in, at which point the
> device name is there for it (as long as you use the same port, of
> course). In which case the filesystem may very well resume successfully.
I was with you right up to this last point.
When a device is unplugged and then plugged back in, it will always get a new
name. Detecting "is the same device" is far from fool-proof, particularly as
the device could have been plugged into some other machine and has 'fsck' etc
run.
Once a mounted device is unplugged, that mount is permanently unusable.
NeilBrown
>
> > Sorry again if this is inappropriate, but you're coming over as so
> > buried in the trees that you can't see the wood.
>
> Sometimes the expanded view is appropriate :-)
>
> Regards,
>
> Phil Turmel
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Last working drive in RAID1
From: Eric Mei @ 2015-03-05 20:23 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150305102622.016ec792@notabene.brown>
On 2015-03-04 4:26 PM, NeilBrown wrote:
> On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <meijia@gmail.com> wrote:
>
>> Hi Neil,
>>
>> I see, that does make sense. Thank you.
>>
>> But it impose a problem for HA. We have 2 nodes as active-standby pair,
>> if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access
>> to physical drives are gone), we hope the array failover to node 2. But
>> with lingering drive reference, mdadm will report array is still alive
>> thus failover won't happen.
>>
>> I guess it depends on what kind of error on the drive. If it's just a
>> media error we should keep it online as much as possible. But if the
>> drive is really bad or physically gone, keeping the stale reference
>> won't help anything. Back to your comparison with single drive /dev/sda,
>> I think MD as an array should do the same as /dev/sda, not the
>> individual drive inside MD, for them we should just let it go. How do
>> you think?
> If there were some what that md could be told that the device really was gone
> and just just returning errors, then I would be OK with it being marked as
> faulty and being removed from the array.
>
> I don't think there is any mechanism in the kernel to allow that. It would
> be easiest to capture a "REMOVE" event via udev, and have udev run "mdadm" to
> tell the md array that the device was gone.
>
> Currently there is no way to do that ... I guess we could change raid1 so
> that a 'fail' event that came from user-space would always cause the device
> to be marked failed, even when an IO error would not...
> To preserve current behaviour, it should require something like "faulty-force"
> to be written to the "state" file. We would need to check that raid1 copes
> with having zero working drives - currently it might always assume there is
> at least one device.
I guess we don't need to know exactly what happened physically, it
should be good enough to know "drive stopped working". If a drive
stopped working, keeping it doesn't add much value anyway. And I think
serious error detected in MD (e.g. superblock write error, bad block
table write error) might be a good criteria to make that judgement.
But as you said current code may assume at least one drive present, need
a more careful review.
Eric
^ permalink raw reply
* Re: Raid 10 Issue
From: Phil Turmel @ 2015-03-05 20:07 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <1888475554.226351.1425578160090.JavaMail.open-xchange@app09.ox.hosteurope.de>
On 03/05/2015 12:56 PM, Stefan Lamby wrote:
> Hello List.
>
> I was setting up a new machine using ubuntu 14.04.02 lts using its installer,
> configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks and
> now I like to add 2 more disks to the array so i want to end up with 4 disks, no
> spare.
>
> Searching the internet I found that I am not able to --grow the array with the
> mdadm version this ubuntu is using (v3.2.5).
> Is that right?
>
> So I decided to build a new array that way and try to move my data afterwards,
> which failed:
> (Is it OK to do it that way or do you recommend another?)
No, you should be able to do this. Probably without any shutdown.
Please show the full layout of your drives, partitions, and lvm.
I suggest lsdrv[1] for working layouts. If your email is set to use
utf8, just paste the result in a reply.
Regards,
Phil Turmel
[1] https://github.com/pturmel/lsdrv
^ permalink raw reply
* Re: Last working drive in RAID1
From: Phil Turmel @ 2015-03-05 20:00 UTC (permalink / raw)
To: Wols Lists, NeilBrown, Eric Mei; +Cc: linux-raid
In-Reply-To: <54F87C7C.8020501@youngman.org.uk>
On 03/05/2015 10:55 AM, Wols Lists wrote:
> Sorry to butt in, but I'm finding this conversation a bit surreal ...
> take everything I say with a pinch of salt. But the really weird bit
> was "what does linux do if /dev/sda disappears?"
>
> In the old days, with /dev/hd*, the * had a hard mapping to the
> hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc
> etc. I think I ran several systems with just hdb and hdd. Not a good
> idea, but.
>
> Nowadays, with sd*, the letter is assigned in order of finding the
> drive. So if sda is removed, linux moves all the other drives and what
> was sdb becomes sda.
On reboot, yes. Not live. If /dev/sda has anything using it when
unplugged, it stays there and gives errors. Existing devices retain
their names, and new devices are added to the end. Only if all users
are completely unhooked will a kernel name get re-used live.
> Which is why you're advised now to always refer
> to drives by their BLKDEV or whatever, as linux provides no guarantees
> whatsoever about sd*. The blockdev may only be a symlink to whatever
> the sd*n code of the disk is, but it makes sure you get the disk you
> want when the sd*n changes under you.
Correct, but once assembled or mounted, the kernel names are locked.
> Equally surreal is the comment about "what does raid1 do with no
> working devices?". Surely it will do nothing, if there's no spinning
> rust or whatever underneath it? You can't corrupt it if there's
> nothing there to corrupt?
It has to stay there to give errors to the upper layers that are still
hooked to it. When they are administratively "unhooked", aka unmounted
or disassociated with mdadm --remove.
Or, quite possibly, the device is plugged back in, at which point the
device name is there for it (as long as you use the same port, of
course). In which case the filesystem may very well resume successfully.
> Sorry again if this is inappropriate, but you're coming over as so
> buried in the trees that you can't see the wood.
Sometimes the expanded view is appropriate :-)
Regards,
Phil Turmel
^ permalink raw reply
* Re: Last working drive in RAID1
From: Eric Mei @ 2015-03-05 19:54 UTC (permalink / raw)
To: Wols Lists, NeilBrown; +Cc: linux-raid
In-Reply-To: <54F87C7C.8020501@youngman.org.uk>
On 2015-03-05 8:55 AM, Wols Lists wrote:
> On 04/03/15 23:26, NeilBrown wrote:
>> On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <meijia@gmail.com>
>> wrote:
>>
>>> Hi Neil,
>>>
>>> I see, that does make sense. Thank you.
>>>
>>> But it impose a problem for HA. We have 2 nodes as active-standby
>>> pair, if HW on node 1 have problem (e.g. SAS cable get pulled,
>>> thus all access to physical drives are gone), we hope the array
>>> failover to node 2. But with lingering drive reference, mdadm
>>> will report array is still alive thus failover won't happen.
>>>
>>> I guess it depends on what kind of error on the drive. If it's
>>> just a media error we should keep it online as much as possible.
>>> But if the drive is really bad or physically gone, keeping the
>>> stale reference won't help anything. Back to your comparison with
>>> single drive /dev/sda, I think MD as an array should do the same
>>> as /dev/sda, not the individual drive inside MD, for them we
>>> should just let it go. How do you think?
>> If there were some what that md could be told that the device
>> really was gone and just just returning errors, then I would be OK
>> with it being marked as faulty and being removed from the array.
>>
>> I don't think there is any mechanism in the kernel to allow that.
>> It would be easiest to capture a "REMOVE" event via udev, and have
>> udev run "mdadm" to tell the md array that the device was gone.
>>
>> Currently there is no way to do that ... I guess we could change
>> raid1 so that a 'fail' event that came from user-space would
>> always cause the device to be marked failed, even when an IO error
>> would not... To preserve current behaviour, it should require
>> something like "faulty-force" to be written to the "state" file.
>> We would need to check that raid1 copes with having zero working
>> drives - currently it might always assume there is at least one
>> device.
>>
> Sorry to butt in, but I'm finding this conversation a bit surreal ...
> take everything I say with a pinch of salt. But the really weird bit
> was "what does linux do if /dev/sda disappears?"
>
> In the old days, with /dev/hd*, the * had a hard mapping to the
> hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc
> etc. I think I ran several systems with just hdb and hdd. Not a good
> idea, but.
>
> Nowadays, with sd*, the letter is assigned in order of finding the
> drive. So if sda is removed, linux moves all the other drives and what
> was sdb becomes sda. Which is why you're advised now to always refer
> to drives by their BLKDEV or whatever, as linux provides no guarantees
> whatsoever about sd*. The blockdev may only be a symlink to whatever
> the sd*n code of the disk is, but it makes sure you get the disk you
> want when the sd*n changes under you.
>
> Equally surreal is the comment about "what does raid1 do with no
> working devices?". Surely it will do nothing, if there's no spinning
> rust or whatever underneath it? You can't corrupt it if there's
> nothing there to corrupt?
>
> Sorry again if this is inappropriate, but you're coming over as so
> buried in the trees that you can't see the wood.
>
> Cheers,
> Wol
Hi Wol,
I think Neil's intention regarding "/dev/sda" waslike this: For a single
drive if it's physically gone, its user will still keep the reference to
it; and RAID1 with single drive should behave the same, i.e. without
knowledge of what exactly happened on this drive, MD is not comfortable
to make decision for application about the status of the last drive,
thus refuse to mark it as failed. The whole thing has not much to do
with FS arrangement under /dev, which is usually managed by udev.
Eric
^ permalink raw reply
* LAON AANBIEDING OP 1,5% rente
From: DIAMOND SWISS LOAN COMPANY @ 2015-03-05 19:38 UTC (permalink / raw)
groeten
We zijn Diamond Zwitserse lening bedrijf het geven van leningen per post advertentie, Ter ondersteuning van banken zeer ernstig en georganiseerd. Als je benteen industriële, aannemer, met pensioen persoon of alle anderen willen krediet aan zijn of haar bedrijf te herstarten of het opzetten van een bedrijf
etc.
we geven leningen. Op 1,5% rente?
Neem vandaag nog contact met de volgende informatie hieronder.
naam:
adres:
leeftijd:
Geslacht:
Contact-Phone:
beroep:
Maandelijks inkomen:
Het benodigde bedrag van de lening:
Duur van de lening:
Het doel van de lening:
land:
Postcode:
Alle informatie moet worden gericht aan ons op: diamondswissloanscompanys@gmail.com
"Wij tonen u een betere manier om uw financiële vrijheid"
Met vriendelijke groet,
De heer Diamond Peters (Managing Director).
De heer Bill Anthony (Marketing Manager).
=============================================================================================================================
Greetings
We are DIAMOND SWISS LOAN COMPANY giving out loans by mail advert, In support of banks very serious and organized. If you are
an industrial, contractor, retire person or all others wanting credit to restart his or her business or to set up a company
etc.
we give out loans. At 1.5% interest rate?
Contact us today with the following information below.
Name:
Address:
Age:
Gender:
Contact-Phone:
Occupation:
Monthly income:
The required amount of loan:
Duration of loan:
The purpose of the loan:
Country:
ZIP code:
All information should be sent to us at: diamondswissloanscompanys@gmail.com
"We show you a better way to your financial freedom"
With kind regards,
Mr. Diamond Peters (Managing Director).
Mr. Bill Anthony (Marketing Manager).
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox