* Re: raid5-cache I/O path improvements V2
From: Christoph Hellwig @ 2015-09-17 1:53 UTC (permalink / raw)
To: Shaohua Li
Cc: Neil Brown, Christoph Hellwig, linux-raid@vger.kernel.org,
Kernel Team, dan.j.williams@intel.com
In-Reply-To: <20150915215458.GA1943628@devbig084.prn1.facebook.com>
On Tue, Sep 15, 2015 at 02:54:59PM -0700, Shaohua Li wrote:
> > I don't want a sysfs config if we can possibly avoid it.
> >
> > Christoph's code sets FUA on every block written to the log, both data
> > and metadata. Is that really what we want?
> >
> > I don't know much of the hardware details, but wouldn't setting FUA and
> > FLUSH on the last block written be just as effective and possibly faster
> > (by giving more flexibility to lower layers)??
>
> How is it different against without FUA, eg, doing a flush after several bios?
It's just a more complicated version of doing the same..
^ permalink raw reply
* Re: Fwd: mdadm: not rebuilding IMSM array without OROM
From: Artur Paszkiewicz @ 2015-09-17 8:43 UTC (permalink / raw)
To: Justin Capella, linux-raid
In-Reply-To: <CAMrEMU_du7HKa5x9dNpiBkp103-9NtF0fXD+Bv-Y6o41tEM85A@mail.gmail.com>
On 09/16/2015 07:54 PM, Justin Capella wrote:
> In reference to the following commit:
> http://git.neil.brown.name/?p=mdadm.git;a=commit;h=7eee461e91df702d8f588955b03e5bbb07e1f88d
>
> I am unclear how to proceed given my unfortunate environment, where I
> have replaced the motherboard. I am fearful of reconfiguring the raid
> in OROM in case it causes data loss. Is anyone aware of a tool to
> rebuild/configure the OROM settings from the autodetected ones?
>
> Should I just modify my startup scripts in my initrd by hand to include --force?
You can use IMSM_NO_PLATFORM environment variable (see man mdadm), but if
your new motherboard also supports IMSM, then you can just enable it.
The raid will be detected normally and this should not cause data loss.
Artur
^ permalink raw reply
* Re: Recommendations for RAID setup needed
From: Michael Tokarev @ 2015-09-18 11:25 UTC (permalink / raw)
To: Alex, Adam Goryachev; +Cc: Wols Lists, Linux RAID
In-Reply-To: <CAB1R3siXjNdihbJhzSWL7bbLGuj0SD7a9iA2wKs-C2dKAO6u0Q@mail.gmail.com>
16.09.2015 18:53, Alex wrote:
[]
>>> All four are WD30EFRX-68EUZN0. They're not the cheapest WD disks, but
>>> they're also not the ones with the 5yr warranty. The last array I
>>> built using disks with 5yr warranty exceeded their capacity before the
>>> warranty expired.
>>
>> Umm, 1st google result showed this:
>> http://community.wd.com/t5/Desktop-Mobile-Drives/New-WD30EFRX-Red-Drive-Idle3-Timer-Set-to-8-Seconds-High-LCC-in/td-p/648821/page/5
>
> I was really just looking for general input on RAID5 vs RAID6, but
> that is good information. I knew the drives weren't basic desktop
> drives and would be generally suitable for building a software RAID
> array.
>
> Are you familiar with the idle3 time? It appears the idle3-tools can
> be used to disable the idle3 timer entirely, which would disable
> parking the head at all, correct?
An addition. From hdparm(8) manpage:
-J Get/set the Western Digital (WD) Green Drive's "idle3" timeout
value. This timeout controls how often the drive parks its
heads and enters a low power consumption state. The factory
default is eight (8) seconds, which is a very poor choice for
use with Linux. Leaving it at the default will result in hun‐
dreds of thousands of head load/unload cycles in a very short
period of time. The drive mechanism is only rated for 300,000
to 1,000,000 cycles, so leaving it at the default could result
in premature failure, not to mention the performance impact of
the drive often having to wake-up before doing routine I/O.
WD supply a WDIDLE3.EXE DOS utility for tweaking this setting,
and you should use that program instead of hdparm if at all pos‐
sible. The reverse-engineered implementation in hdparm is not
as complete as the original official program, even though it
does seem to work on at a least a few drives. A full power
cycle is required for any change in setting to take effect,
regardless of which program is used to tweak things.
A setting of 30 seconds is recommended for Linux use. Permitted
values are from 8 to 12 seconds, and from 30 to 300 seconds in
30-second increments. Specify a value of zero (0) to disable
the WD idle3 timer completely (NOT RECOMMENDED!).
Thanks,
/mjt
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Recommendations for RAID setup needed
From: Alex @ 2015-09-18 12:55 UTC (permalink / raw)
To: Michael Tokarev; +Cc: Adam Goryachev, Wols Lists, Linux RAID
In-Reply-To: <55FBF4B1.6060501@msgid.tls.msk.ru>
Hi,
> An addition. From hdparm(8) manpage:
>
> -J Get/set the Western Digital (WD) Green Drive's "idle3" timeout
> value. This timeout controls how often the drive parks its
> heads and enters a low power consumption state. The factory
> default is eight (8) seconds, which is a very poor choice for
> use with Linux. Leaving it at the default will result in hun‐
> dreds of thousands of head load/unload cycles in a very short
> period of time. The drive mechanism is only rated for 300,000
> to 1,000,000 cycles, so leaving it at the default could result
> in premature failure, not to mention the performance impact of
> the drive often having to wake-up before doing routine I/O.
>
> WD supply a WDIDLE3.EXE DOS utility for tweaking this setting,
> and you should use that program instead of hdparm if at all pos‐
> sible. The reverse-engineered implementation in hdparm is not
> as complete as the original official program, even though it
> does seem to work on at a least a few drives. A full power
> cycle is required for any change in setting to take effect,
> regardless of which program is used to tweak things.
>
> A setting of 30 seconds is recommended for Linux use. Permitted
> values are from 8 to 12 seconds, and from 30 to 300 seconds in
> 30-second increments. Specify a value of zero (0) to disable
> the WD idle3 timer completely (NOT RECOMMENDED!).
Thanks very much for the info. I posted to the WD community forum and
didn't receive any responses, so this is much appreciated. I'd like to
understand why disabling it isn't recommended (that's what I was going
to do), but I'll have to just accept this for now and move on.
In addition to hdparm, there's also the idle3-tools package and the
idle3ctl binary.
Thanks,
Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* strange syslog message on daily mdadm cron job
From: Jens-U. Mozdzen @ 2015-09-18 13:57 UTC (permalink / raw)
To: Linux RAID
Hi *,
checking syslog I noticed the following messages, created on openSUSE
13.1 during each daily "mdadm" cron job:
--- cut here ---
mdadm[15265]: DeviceDisappeared event detected on md device
/dev/md/host01-data
mdadm[15265]: DeviceDisappeared event detected on md device
/dev/md/host01-cache
mdadm[15265]: NewArray event detected on md device /dev/md127
mdadm[15265]: NewArray event detected on md device /dev/md126
--- cut here ---
The command issued by the cron job is
eval /sbin/mdadm -F --oneshot --syslog -m "user@company.com" -s -c
"/etc/mdadm.conf"
The contents of /etc/mdadm.conf are
--- cut here ---
DEVICE containers partitions
ARRAY /dev/md/host01-cache UUID=3884ea75:28c82f09:05d0026d:1c95b268
ARRAY /dev/md/host01-data UUID=92a1b47d:6c889735:6d60a23f:f8d8067c
HOMEHOST host01
MAILADDR user@company.com
MAILFROM host01@company.com
--- cut here ---
Searching the net didn't pop up any obvious hints, but shows several
responses to problem reports that advice to leave mdadm.conf empty.
We do not experience any noticeable problems, nor does mdadm send any
error report emails. The latter is tested, we had reports of faulty
disks some time ago (related to controller problems, not actual disk
errors). "mdadm --detail" reports look perfectly fine for both arrays.
Are the syslog messages the result of a local configuration error, or
may there be some other cause?
Regards,
Jens
^ permalink raw reply
* [PATCH 1/2] md: clear CHANGE_PENDING in readonly array
From: Shaohua Li @ 2015-09-18 17:20 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
If faulty disks of an array are more than allowed degraded number, the
array enters error handling. It will be marked as read-only with
MD_CHANGE_PENDING/RECOVERY_NEEDED set. But currently recovery doesn't
clear CHANGE_PENDING bit for read-only array. If MD_CHANGE_PENDING is
set for a raid5 array, all returned IO will be hold on a list till the
bit is clear. But recovery nevery clears this bit, the IO is always in
pending state and nevery finish. This has bad effects like upper layer
can't get an IO error and the array can't be stopped.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 95824fb..c596b73 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -8209,6 +8209,7 @@ void md_check_recovery(struct mddev *mddev)
md_reap_sync_thread(mddev);
clear_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+ clear_bit(MD_CHANGE_PENDING, &mddev->flags);
goto unlock;
}
--
1.8.1
^ permalink raw reply related
* [PATCH 2/2] raid5: update analysis state for failed stripe
From: Shaohua Li @ 2015-09-18 17:20 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <bda150068cfdb69bf8b632c32dc2a7db17c919f4.1442596586.git.shli@fb.com>
handle_failed_stripe() makes the stripe fail, eg, all IO will return
with a failure, but it doesn't update stripe_head_state. Later
handle_stripe() has special handling for raid6 for handle_stripe_fill().
That check before handle_stripe_fill() doesn't skip the failed stripe
and we get a kernel crash in need_this_block. This patch clear the
analysis state to make sure no functions wrongly called after
handle_failed_stripe()
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 394cdf8..8e4fb89a 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3155,6 +3155,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
spin_unlock_irq(&sh->stripe_lock);
if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
wake_up(&conf->wait_for_overlap);
+ if (bi)
+ s->to_read--;
while (bi && bi->bi_iter.bi_sector <
sh->dev[i].sector + STRIPE_SECTORS) {
struct bio *nextbi =
@@ -3173,6 +3175,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
*/
clear_bit(R5_LOCKED, &sh->dev[i].flags);
}
+ s->to_write = 0;
+ s->written = 0;
if (test_and_clear_bit(STRIPE_FULL_WRITE, &sh->state))
if (atomic_dec_and_test(&conf->pending_full_writes))
--
1.8.1
^ permalink raw reply related
* Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-19 18:33 UTC (permalink / raw)
To: linux-raid
I came back earlier today to find some problems on my server, having
looked at the logs it looked like there was a power spike or something
that caused the server to cough and the array to kick. I had to
reboot the server to get any info at all as it had locked.
As the array has gone offline I thought rather then me just messing
around it would be wiser to take advice before causing any further
potential damage.
Here is the output of
mdadm -E on the active drives in the array.
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : arandomserver.arandomlan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : active
Device UUID : 968a5c1d:ab019235:fb578b6e:2f5ee382
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Sep 19 14:28:32 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9e8960f5 - correct
Events : 285433
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : arandomserver.arandomlan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 140e09af:56e14b4e:5035d724:c2005f0b
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Sep 19 18:23:15 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 87b27b15 - correct
Events : 285455
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : .AAA..A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : arandomserver.arandomlan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : cda4f5e5:a489dbb9:5c1ab6a0:b257c984
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Sep 19 18:23:15 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1f77f61b - correct
Events : 285455
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : .AAA..A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : arandomserver.arandomlan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 7d0b65b3:d2ba2023:4625c287:1db2de9b
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Sep 19 18:23:15 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 544df34e - correct
Events : 285455
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : .AAA..A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : arandomserver.arandomlan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : active
Device UUID : df5af6ce:9017c863:697da267:046c9709
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Sep 19 14:28:32 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : fe247a62 - correct
Events : 285433
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : arandomserver.arandomlan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : active
Device UUID : 10563c18:d7c7a547:ec8ee742:4d6cc1a3
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Sep 19 14:28:32 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a0b737ab - correct
Events : 285433
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : arandomserver.arandomlan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : a50dd0a1:eeb0b3df:76200476:818e004d
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Sep 19 18:23:15 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9eafc329 - correct
Events : 285455
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : .AAA..A ('A' == active, '.' == missing, 'R' == replacing)
This gives me a reasonable level of confidence as on first glance it
looks to me like when the server coughed three of the drives in the
array went offline thus the array immediately failed. One thing I
have noticed is the Device UUID value has changed form what the arrays
value was in fstab
cat /proc/mdstat gives me this....
Personalities :
md127 : inactive sdh1[5](S) sdf1[3](S) sdc1[4](S) sdd1[1](S)
sdi1[6](S) sdg1[0](S) sde1[2](S)
41022733300 blocks super 1.2
unused devices: <none>
I don't just want to try a forced assemble until someone casts their
eye over this (and anything else wanted) and lets me know the best way
to proceed.....
Thanks in advance.
Tony
p.s. Before anyone says you do of course have a backup....thats what
the array was doing when it failed as the old backup had to be moved
yesterday.
^ permalink raw reply
* Re: Failed Raid6 Array.....want some guidance before attempting restart
From: Mikael Abrahamsson @ 2015-09-20 5:34 UTC (permalink / raw)
To: Another Sillyname; +Cc: linux-raid
In-Reply-To: <CAOS+5GGkLcN5RM7yq=D5NQ-DOBRf_+EwVsucoptfaiJPdU_nDQ@mail.gmail.com>
On Sat, 19 Sep 2015, Another Sillyname wrote:
> I don't just want to try a forced assemble until someone casts their eye
> over this (and anything else wanted) and lets me know the best way to
> proceed.....
The event counts are off by just a little bit, so if I were in your
position I would stop the array and then --assemble --force with all the
devices. If this doesn't work (because some of them are marked as failed),
I would compile latest mdadm version from the git repository, and then try
--assemble --force with that mdadm version instead.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-20 9:19 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAOS+5GEcpOV6dkkGYHjkA3ah88MXqyJ0AFS7T--qaCc-Z1-=uw@mail.gmail.com>
Thanks
Would you.....
mdadm --assemble --force --scan
or
mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
or
another variant of the mdadm command?
On 20 September 2015 at 06:34, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sat, 19 Sep 2015, Another Sillyname wrote:
>
>> I don't just want to try a forced assemble until someone casts their eye
>> over this (and anything else wanted) and lets me know the best way to
>> proceed.....
>
>
> The event counts are off by just a little bit, so if I were in your position
> I would stop the array and then --assemble --force with all the devices. If
> this doesn't work (because some of them are marked as failed), I would
> compile latest mdadm version from the git repository, and then try
> --assemble --force with that mdadm version instead.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Mikael Abrahamsson @ 2015-09-20 9:54 UTC (permalink / raw)
To: Another Sillyname; +Cc: linux-raid
In-Reply-To: <CAOS+5GHLFcbN=ROQ_YBLFYzw6iu+TZ4RcZvHdDbE+V3pqS_rCw@mail.gmail.com>
On Sun, 20 Sep 2015, Another Sillyname wrote:
> Thanks
>
> Would you.....
>
> mdadm --assemble --force --scan
>
> or
>
> mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
This last one is what I use myself.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-21 1:32 UTC (permalink / raw)
To: Mikael Abrahamsson, linux-raid
In-Reply-To: <alpine.DEB.2.02.1509201153490.8750@uplift.swm.pp.se>
OK
The array has come back up...but showing two drives as missing.
mdadm --query --detail /dev/md127/dev/md127:
Version : 1.2
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 5860390400 (5588.90 GiB 6001.04 GB)
Raid Devices : 7
Total Devices : 5
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Sep 21 02:21:48 2015
State : active, degraded
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : arandomserver.arandomlan.com:1
UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Events : 285469
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 49 1 active sync /dev/sdd1
2 8 65 2 active sync /dev/sde1
3 8 81 3 active sync /dev/sdf1
8 0 0 8 removed
10 0 0 10 removed
6 8 129 6 active sync /dev/sdi1
Data appears to be intact (haven't done a full analysis yet).
Does this mean I should add the 'missing' drives back into the array
(one at a time obviously)!.
Also doesn't this mean I'm horribly exposed to any writes now as this
would move the current 5+2 further out of 'sync' with each other thus
meaning any further short term fail could smash the data set totally.
I'm minded to stop any writes to the array in the short term and
continue just doing the backup (this in itself will take about 30+
hours).
Ideas and observations?
On 20 September 2015 at 10:54, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sun, 20 Sep 2015, Another Sillyname wrote:
>
>> Thanks
>>
>> Would you.....
>>
>> mdadm --assemble --force --scan
>>
>> or
>>
>> mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
>
>
> This last one is what I use myself.
>
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-21 1:59 UTC (permalink / raw)
To: Mikael Abrahamsson, linux-raid
In-Reply-To: <CAOS+5GHf1N6s9bsrveesuhm6cnxRVnb6XSjXoHoLVU7Np-i-Bw@mail.gmail.com>
Ignore last...having thought about it for 10 minutes the obvious thing
to do is to add the drives back and allow the array to rebuild
offline......
For the following reasons....
1. e2fsck -f -n /dev/mdxx reports all the data appears intact and
that was what I believed anyway based on the information available to
me.
2. To finish the backup will take 30+ hours, that's 30+ hours of risk
time where a single drive failure will compromise the data set.
3. To 'add' the missing drives back into the array and allow the
rebuild will take about 10 hours (based on my previous experience
building this array), therefore the lower 'risk' course of action is
to rebuild the array, then and only then, to restart the backup.
There's over 20 hours less risk doing it this way.
I realise I could do the two concurrently but I'd rather keep the
array 'destressed' as much as possible until I've got at least one
level of resilience restored.
Having now added the drives back in as 'spares' mdstat is telling me a
little over 12 hours to do the rebuild so it's now finger crossing
time time then.
Thanks for the help and advice....and most of all the confirmation my
approach was the correct one.
On 21 September 2015 at 02:32, Another Sillyname
<anothersname@googlemail.com> wrote:
> OK
>
> The array has come back up...but showing two drives as missing.
>
> mdadm --query --detail /dev/md127/dev/md127:
> Version : 1.2
> Creation Time : Sun May 10 14:47:51 2015
> Raid Level : raid6
> Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
> Used Dev Size : 5860390400 (5588.90 GiB 6001.04 GB)
> Raid Devices : 7
> Total Devices : 5
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon Sep 21 02:21:48 2015
> State : active, degraded
> Active Devices : 5
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Name : arandomserver.arandomlan.com:1
> UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
> Events : 285469
>
> Number Major Minor RaidDevice State
> 0 8 97 0 active sync /dev/sdg1
> 1 8 49 1 active sync /dev/sdd1
> 2 8 65 2 active sync /dev/sde1
> 3 8 81 3 active sync /dev/sdf1
> 8 0 0 8 removed
> 10 0 0 10 removed
> 6 8 129 6 active sync /dev/sdi1
>
> Data appears to be intact (haven't done a full analysis yet).
>
> Does this mean I should add the 'missing' drives back into the array
> (one at a time obviously)!.
>
> Also doesn't this mean I'm horribly exposed to any writes now as this
> would move the current 5+2 further out of 'sync' with each other thus
> meaning any further short term fail could smash the data set totally.
>
> I'm minded to stop any writes to the array in the short term and
> continue just doing the backup (this in itself will take about 30+
> hours).
>
> Ideas and observations?
>
>
>
> On 20 September 2015 at 10:54, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> On Sun, 20 Sep 2015, Another Sillyname wrote:
>>
>>> Thanks
>>>
>>> Would you.....
>>>
>>> mdadm --assemble --force --scan
>>>
>>> or
>>>
>>> mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
>>
>>
>> This last one is what I use myself.
>>
>>
>> --
>> Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Mikael Abrahamsson @ 2015-09-21 7:12 UTC (permalink / raw)
To: Another Sillyname; +Cc: linux-raid
In-Reply-To: <CAOS+5GHtJAYdHEHnvb+Akfc+jkh5es8cb8z50YQwB23rnk+_Sw@mail.gmail.com>
On Mon, 21 Sep 2015, Another Sillyname wrote:
> Thanks for the help and advice....and most of all the confirmation my
> approach was the correct one.
It would be helpful if you posted the dmesg output from what you did. So
while this approach is ok, so you stated, you're vulnerable to read errors
on the remaining drives.
It's possible a newer mdadm would have --assemble --force in the two last
drives as well. You have an internal bitmap, which should have meant that
you shouldn't have had to do a full resync.
So what happened to you isn't optimal and it would be helpful to
understand what kernel version, mdadm version you used, and what the
output from your steps were.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-21 7:57 UTC (permalink / raw)
To: Mikael Abrahamsson, linux-raid
In-Reply-To: <alpine.DEB.2.02.1509210909380.8750@uplift.swm.pp.se>
Kernel....
Linux arandomserver1 3.12.10-300.fc20.x86_64 #1 SMP Thu Feb 6 22:11:48
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
mdadm
mdadm-3.3-4.fc20.x86_64
I'll post the dmesg once the rebuild has finished.
This particular server is due to be upgraded/rebuilt next month (which
is why the backups are currently happening), generally I rebuild the
server on about a 2 year cycle for upgrades and minimise incremental
upgrades, mainly because experience has taught me that an up, stable,
un-interfered with server suffers less failures (hardware and
software) then one being rebooted every few months.
In 15+ years this is honestly the first time I can remember a server I
owned having a failure that happened during normal run...rather then
during an upgrade or reboot.
Once again thanks for the help.
On 21 September 2015 at 08:12, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Mon, 21 Sep 2015, Another Sillyname wrote:
>
>> Thanks for the help and advice....and most of all the confirmation my
>> approach was the correct one.
>
>
> It would be helpful if you posted the dmesg output from what you did. So
> while this approach is ok, so you stated, you're vulnerable to read errors
> on the remaining drives.
>
> It's possible a newer mdadm would have --assemble --force in the two last
> drives as well. You have an internal bitmap, which should have meant that
> you shouldn't have had to do a full resync.
>
> So what happened to you isn't optimal and it would be helpful to understand
> what kernel version, mdadm version you used, and what the output from your
> steps were.
>
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Alexander Afonyashin @ 2015-09-21 7:57 UTC (permalink / raw)
To: Another Sillyname; +Cc: Mikael Abrahamsson, linux-raid
In-Reply-To: <CAOS+5GHtJAYdHEHnvb+Akfc+jkh5es8cb8z50YQwB23rnk+_Sw@mail.gmail.com>
Hi,
You may also try to increase rebuild rate by echo-ing min speed value:
echo 100000 > /sys/block/mdX/md/sync_speed_min
or via sysctl:
sysctl -w dev.raid.speed_limit_min=100000
Regards,
Alexander
On Mon, Sep 21, 2015 at 4:59 AM, Another Sillyname
<anothersname@googlemail.com> wrote:
> Ignore last...having thought about it for 10 minutes the obvious thing
> to do is to add the drives back and allow the array to rebuild
> offline......
>
> For the following reasons....
>
> 1. e2fsck -f -n /dev/mdxx reports all the data appears intact and
> that was what I believed anyway based on the information available to
> me.
>
> 2. To finish the backup will take 30+ hours, that's 30+ hours of risk
> time where a single drive failure will compromise the data set.
>
> 3. To 'add' the missing drives back into the array and allow the
> rebuild will take about 10 hours (based on my previous experience
> building this array), therefore the lower 'risk' course of action is
> to rebuild the array, then and only then, to restart the backup.
> There's over 20 hours less risk doing it this way.
>
> I realise I could do the two concurrently but I'd rather keep the
> array 'destressed' as much as possible until I've got at least one
> level of resilience restored.
>
> Having now added the drives back in as 'spares' mdstat is telling me a
> little over 12 hours to do the rebuild so it's now finger crossing
> time time then.
>
> Thanks for the help and advice....and most of all the confirmation my
> approach was the correct one.
>
>
>
> On 21 September 2015 at 02:32, Another Sillyname
> <anothersname@googlemail.com> wrote:
>> OK
>>
>> The array has come back up...but showing two drives as missing.
>>
>> mdadm --query --detail /dev/md127/dev/md127:
>> Version : 1.2
>> Creation Time : Sun May 10 14:47:51 2015
>> Raid Level : raid6
>> Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
>> Used Dev Size : 5860390400 (5588.90 GiB 6001.04 GB)
>> Raid Devices : 7
>> Total Devices : 5
>> Persistence : Superblock is persistent
>>
>> Intent Bitmap : Internal
>>
>> Update Time : Mon Sep 21 02:21:48 2015
>> State : active, degraded
>> Active Devices : 5
>> Working Devices : 5
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Name : arandomserver.arandomlan.com:1
>> UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
>> Events : 285469
>>
>> Number Major Minor RaidDevice State
>> 0 8 97 0 active sync /dev/sdg1
>> 1 8 49 1 active sync /dev/sdd1
>> 2 8 65 2 active sync /dev/sde1
>> 3 8 81 3 active sync /dev/sdf1
>> 8 0 0 8 removed
>> 10 0 0 10 removed
>> 6 8 129 6 active sync /dev/sdi1
>>
>> Data appears to be intact (haven't done a full analysis yet).
>>
>> Does this mean I should add the 'missing' drives back into the array
>> (one at a time obviously)!.
>>
>> Also doesn't this mean I'm horribly exposed to any writes now as this
>> would move the current 5+2 further out of 'sync' with each other thus
>> meaning any further short term fail could smash the data set totally.
>>
>> I'm minded to stop any writes to the array in the short term and
>> continue just doing the backup (this in itself will take about 30+
>> hours).
>>
>> Ideas and observations?
>>
>>
>>
>> On 20 September 2015 at 10:54, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>> On Sun, 20 Sep 2015, Another Sillyname wrote:
>>>
>>>> Thanks
>>>>
>>>> Would you.....
>>>>
>>>> mdadm --assemble --force --scan
>>>>
>>>> or
>>>>
>>>> mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
>>>
>>>
>>> This last one is what I use myself.
>>>
>>>
>>> --
>>> Mikael Abrahamsson email: swmike@swm.pp.se
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-21 8:05 UTC (permalink / raw)
To: Alexander Afonyashin, Mikael Abrahamsson, linux-raid
In-Reply-To: <CAETWcfuFzUO0aCX-YX_JYKH+iUDOAS6LJUrxu_TYT0g9x2Jqmg@mail.gmail.com>
I think at the moment I'm in leave it alone and let it run
mode.....it'll be done in about 6 hours anyway and I'm adverse to
'tampering' with anything while I'm this exposed without any
resilience.
I meant to state in the earlier message when the rebuild happens next
month I'll be installing Fedora 22 (and all the latest updates). This
is a high demand server (not high load but requiring high
availability) so once rebuilt and running stable for a month it'll get
'locked down' without any changes for another couple of years.
On 21 September 2015 at 08:57, Alexander Afonyashin
<a.afonyashin@madnet-team.ru> wrote:
> Hi,
>
> You may also try to increase rebuild rate by echo-ing min speed value:
>
> echo 100000 > /sys/block/mdX/md/sync_speed_min
>
> or via sysctl:
>
> sysctl -w dev.raid.speed_limit_min=100000
>
> Regards,
> Alexander
>
> On Mon, Sep 21, 2015 at 4:59 AM, Another Sillyname
> <anothersname@googlemail.com> wrote:
>> Ignore last...having thought about it for 10 minutes the obvious thing
>> to do is to add the drives back and allow the array to rebuild
>> offline......
>>
>> For the following reasons....
>>
>> 1. e2fsck -f -n /dev/mdxx reports all the data appears intact and
>> that was what I believed anyway based on the information available to
>> me.
>>
>> 2. To finish the backup will take 30+ hours, that's 30+ hours of risk
>> time where a single drive failure will compromise the data set.
>>
>> 3. To 'add' the missing drives back into the array and allow the
>> rebuild will take about 10 hours (based on my previous experience
>> building this array), therefore the lower 'risk' course of action is
>> to rebuild the array, then and only then, to restart the backup.
>> There's over 20 hours less risk doing it this way.
>>
>> I realise I could do the two concurrently but I'd rather keep the
>> array 'destressed' as much as possible until I've got at least one
>> level of resilience restored.
>>
>> Having now added the drives back in as 'spares' mdstat is telling me a
>> little over 12 hours to do the rebuild so it's now finger crossing
>> time time then.
>>
>> Thanks for the help and advice....and most of all the confirmation my
>> approach was the correct one.
>>
>>
>>
>> On 21 September 2015 at 02:32, Another Sillyname
>> <anothersname@googlemail.com> wrote:
>>> OK
>>>
>>> The array has come back up...but showing two drives as missing.
>>>
>>> mdadm --query --detail /dev/md127/dev/md127:
>>> Version : 1.2
>>> Creation Time : Sun May 10 14:47:51 2015
>>> Raid Level : raid6
>>> Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
>>> Used Dev Size : 5860390400 (5588.90 GiB 6001.04 GB)
>>> Raid Devices : 7
>>> Total Devices : 5
>>> Persistence : Superblock is persistent
>>>
>>> Intent Bitmap : Internal
>>>
>>> Update Time : Mon Sep 21 02:21:48 2015
>>> State : active, degraded
>>> Active Devices : 5
>>> Working Devices : 5
>>> Failed Devices : 0
>>> Spare Devices : 0
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Name : arandomserver.arandomlan.com:1
>>> UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
>>> Events : 285469
>>>
>>> Number Major Minor RaidDevice State
>>> 0 8 97 0 active sync /dev/sdg1
>>> 1 8 49 1 active sync /dev/sdd1
>>> 2 8 65 2 active sync /dev/sde1
>>> 3 8 81 3 active sync /dev/sdf1
>>> 8 0 0 8 removed
>>> 10 0 0 10 removed
>>> 6 8 129 6 active sync /dev/sdi1
>>>
>>> Data appears to be intact (haven't done a full analysis yet).
>>>
>>> Does this mean I should add the 'missing' drives back into the array
>>> (one at a time obviously)!.
>>>
>>> Also doesn't this mean I'm horribly exposed to any writes now as this
>>> would move the current 5+2 further out of 'sync' with each other thus
>>> meaning any further short term fail could smash the data set totally.
>>>
>>> I'm minded to stop any writes to the array in the short term and
>>> continue just doing the backup (this in itself will take about 30+
>>> hours).
>>>
>>> Ideas and observations?
>>>
>>>
>>>
>>> On 20 September 2015 at 10:54, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>>> On Sun, 20 Sep 2015, Another Sillyname wrote:
>>>>
>>>>> Thanks
>>>>>
>>>>> Would you.....
>>>>>
>>>>> mdadm --assemble --force --scan
>>>>>
>>>>> or
>>>>>
>>>>> mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
>>>>
>>>>
>>>> This last one is what I use myself.
>>>>
>>>>
>>>> --
>>>> Mikael Abrahamsson email: swmike@swm.pp.se
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-21 8:09 UTC (permalink / raw)
To: Alexander Afonyashin, Mikael Abrahamsson, linux-raid
In-Reply-To: <CAOS+5GHK7_26qXWv-Xo_bnZ5yCtDTKsUn67Nb8K8sw2rBOJ0AQ@mail.gmail.com>
Current watch status
Every 30.0s: cat /proc/mdstat
Mon Sep 21 09:07:03 2015
Personalities : [raid6] [raid5] [raid4]
mdxxx : active raid6 sdh1[8] sdc1[7] sdg1[0] sdi1[6] sdf1[3] sde1[2] sdd1[1]
29301952000 blocks super 1.2 level 6, 512k chunk, algorithm 2
[7/5] [UUUU__U]
[==========>..........] recovery = 53.4%
(3133878924/5860390400) finish=358.2min speed=126856K/sec
bitmap: 6/44 pages [24KB], 65536KB chunk
unused devices: <none>
On 21 September 2015 at 09:05, Another Sillyname
<anothersname@googlemail.com> wrote:
> I think at the moment I'm in leave it alone and let it run
> mode.....it'll be done in about 6 hours anyway and I'm adverse to
> 'tampering' with anything while I'm this exposed without any
> resilience.
>
> I meant to state in the earlier message when the rebuild happens next
> month I'll be installing Fedora 22 (and all the latest updates). This
> is a high demand server (not high load but requiring high
> availability) so once rebuilt and running stable for a month it'll get
> 'locked down' without any changes for another couple of years.
>
>
>
>
>
> On 21 September 2015 at 08:57, Alexander Afonyashin
> <a.afonyashin@madnet-team.ru> wrote:
>> Hi,
>>
>> You may also try to increase rebuild rate by echo-ing min speed value:
>>
>> echo 100000 > /sys/block/mdX/md/sync_speed_min
>>
>> or via sysctl:
>>
>> sysctl -w dev.raid.speed_limit_min=100000
>>
>> Regards,
>> Alexander
>>
>> On Mon, Sep 21, 2015 at 4:59 AM, Another Sillyname
>> <anothersname@googlemail.com> wrote:
>>> Ignore last...having thought about it for 10 minutes the obvious thing
>>> to do is to add the drives back and allow the array to rebuild
>>> offline......
>>>
>>> For the following reasons....
>>>
>>> 1. e2fsck -f -n /dev/mdxx reports all the data appears intact and
>>> that was what I believed anyway based on the information available to
>>> me.
>>>
>>> 2. To finish the backup will take 30+ hours, that's 30+ hours of risk
>>> time where a single drive failure will compromise the data set.
>>>
>>> 3. To 'add' the missing drives back into the array and allow the
>>> rebuild will take about 10 hours (based on my previous experience
>>> building this array), therefore the lower 'risk' course of action is
>>> to rebuild the array, then and only then, to restart the backup.
>>> There's over 20 hours less risk doing it this way.
>>>
>>> I realise I could do the two concurrently but I'd rather keep the
>>> array 'destressed' as much as possible until I've got at least one
>>> level of resilience restored.
>>>
>>> Having now added the drives back in as 'spares' mdstat is telling me a
>>> little over 12 hours to do the rebuild so it's now finger crossing
>>> time time then.
>>>
>>> Thanks for the help and advice....and most of all the confirmation my
>>> approach was the correct one.
>>>
>>>
>>>
>>> On 21 September 2015 at 02:32, Another Sillyname
>>> <anothersname@googlemail.com> wrote:
>>>> OK
>>>>
>>>> The array has come back up...but showing two drives as missing.
>>>>
>>>> mdadm --query --detail /dev/md127/dev/md127:
>>>> Version : 1.2
>>>> Creation Time : Sun May 10 14:47:51 2015
>>>> Raid Level : raid6
>>>> Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
>>>> Used Dev Size : 5860390400 (5588.90 GiB 6001.04 GB)
>>>> Raid Devices : 7
>>>> Total Devices : 5
>>>> Persistence : Superblock is persistent
>>>>
>>>> Intent Bitmap : Internal
>>>>
>>>> Update Time : Mon Sep 21 02:21:48 2015
>>>> State : active, degraded
>>>> Active Devices : 5
>>>> Working Devices : 5
>>>> Failed Devices : 0
>>>> Spare Devices : 0
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 512K
>>>>
>>>> Name : arandomserver.arandomlan.com:1
>>>> UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
>>>> Events : 285469
>>>>
>>>> Number Major Minor RaidDevice State
>>>> 0 8 97 0 active sync /dev/sdg1
>>>> 1 8 49 1 active sync /dev/sdd1
>>>> 2 8 65 2 active sync /dev/sde1
>>>> 3 8 81 3 active sync /dev/sdf1
>>>> 8 0 0 8 removed
>>>> 10 0 0 10 removed
>>>> 6 8 129 6 active sync /dev/sdi1
>>>>
>>>> Data appears to be intact (haven't done a full analysis yet).
>>>>
>>>> Does this mean I should add the 'missing' drives back into the array
>>>> (one at a time obviously)!.
>>>>
>>>> Also doesn't this mean I'm horribly exposed to any writes now as this
>>>> would move the current 5+2 further out of 'sync' with each other thus
>>>> meaning any further short term fail could smash the data set totally.
>>>>
>>>> I'm minded to stop any writes to the array in the short term and
>>>> continue just doing the backup (this in itself will take about 30+
>>>> hours).
>>>>
>>>> Ideas and observations?
>>>>
>>>>
>>>>
>>>> On 20 September 2015 at 10:54, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>>>> On Sun, 20 Sep 2015, Another Sillyname wrote:
>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Would you.....
>>>>>>
>>>>>> mdadm --assemble --force --scan
>>>>>>
>>>>>> or
>>>>>>
>>>>>> mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
>>>>>
>>>>>
>>>>> This last one is what I use myself.
>>>>>
>>>>>
>>>>> --
>>>>> Mikael Abrahamsson email: swmike@swm.pp.se
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Guidance on reshape stuck at 0% after --grow
From: Guillaume Paumier @ 2015-09-22 2:15 UTC (permalink / raw)
To: linux-raid
Hello,
First of all, my apologies if this is explained somewhere; I haven't
been able to find existing documentation about this exact situation.
Yesterday, I added a new (identical) disk to an existing, clean,
8-disk RAID6 array (with XFS), and attempted to grow the array:
# mdadm --add /dev/md0 /dev/sdb1
# mdadm --grow --raid-devices=9 /dev/md0
Over 24 hours later, the reshape is still in the same state as when it
started, i.e. at 0%. Looking at cat /proc/mdstat , no data seems to
have been moved, and the speed is shown as 0K/sec. The "finish"
estimate just keeps increasing. I don't see any process related to
mdadm in top. Unfortunately, I omitted the backup file in the --grow
command.
I'm looking for guidance regarding what to do next. I'm not sure
whether I should wait longer (how long?), or try to re-assemble the
seemingly-intact original array (how do I stop/restart the reshape
cleanly?), or something else. Any advice would be greatly appreciated.
Here's some additional information about the array after the --grow command:
______________________________________________
# mdadm -D /dev/md0
______________________________________________
/dev/md0:
Version : 1.0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 3907016448 (3726.02 GiB 4000.78 GB)
Raid Devices : 9
Total Devices : 9
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Sep 20 16:48:15 2015
State : clean, reshaping
Active Devices : 9
Working Devices : 9
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 128K
Reshape Status : 0% complete
Delta Devices : 1, (8->9)
Name : (redacted):0
UUID : eea59047:120a0365:353da182:6787e030
Events : 35499
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 97 2 active sync /dev/sdg1
3 8 113 3 active sync /dev/sdh1
4 8 129 4 active sync /dev/sdi1
5 8 145 5 active sync /dev/sdj1
7 8 81 6 active sync /dev/sdf1
6 8 65 7 active sync /dev/sde1
8 8 17 8 active sync /dev/sdb1
______________________________________________
cat /proc/mdstat
______________________________________________
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdb1[8] sdg1[2] sdj1[5] sdc1[0] sdf1[7] sde1[6]
sdh1[3] sdi1[4] sdd1[1]
23442098688 blocks super 1.0 level 6, 128k chunk, algorithm 2
[9/9] [UUUUUUUUU]
[>....................] reshape = 0.0% (0/3907016448)
finish=377198019201.6min speed=0K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
unused devices: <none>
______________________________________________
# dmesg | grep md
______________________________________________
<pre>
[ 4.468239] md: bind<sdd1>
[ 4.477539] md: bind<sdi1>
[ 4.489898] md: bind<sdh1>
[ 4.499959] md: bind<sde1>
[ 4.507908] md: bind<sdf1>
[ 4.534961] md: bind<sdc1>
[ 4.544582] md: bind<sdj1>
[ 4.550060] md: bind<sdg1>
[ 4.642375] md: raid6 personality registered for level 6
[ 4.642378] md: raid5 personality registered for level 5
[ 4.642407] md: raid4 personality registered for level 4
[ 4.644839] md/raid:md0: device sdg1 operational as raid disk 2
[ 4.644843] md/raid:md0: device sdj1 operational as raid disk 5
[ 4.644844] md/raid:md0: device sdc1 operational as raid disk 0
[ 4.644845] md/raid:md0: device sdf1 operational as raid disk 6
[ 4.644846] md/raid:md0: device sde1 operational as raid disk 7
[ 4.644847] md/raid:md0: device sdh1 operational as raid disk 3
[ 4.644848] md/raid:md0: device sdi1 operational as raid disk 4
[ 4.644850] md/raid:md0: device sdd1 operational as raid disk 1
[ 4.645566] md/raid:md0: allocated 0kB
[ 4.646409] md/raid:md0: raid level 6 active with 8 out of 8
devices, algorithm 2
[ 4.646575] created bitmap (30 pages) for device md0
[ 4.647292] md0: bitmap initialized from disk: read 2 pages, set 0
of 59617 bits
[ 4.699645] md0: detected capacity change from 0 to 24004709056512
[ 4.700805] md0: unknown partition table
[ 4.909274] XFS (md0): Mounting V4 Filesystem
[ 5.335425] XFS (md0): Ending clean mount
[ 1642.872936] md0: unknown partition table
[ 1665.183589] md: bind<sdb1>
[ 1665.217944] md0: unknown partition table
[ 1868.222033] md: reshape of RAID array md0
[ 1868.222037] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 1868.222038] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for reshape.
[ 1868.222047] md: using 128k window, over a total of 3907016448k.
[ 1868.276409] md0: unknown partition table
[ 2001.573583] md0: unknown partition table
______________________________________________
# mdadm --version
______________________________________________
mdadm - v3.3.1 - 5th June 2014
--
Guillaume Paumier
^ permalink raw reply
* Re: Recommendations for RAID setup needed
From: David C. Rankin @ 2015-09-22 4:16 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <55F85E9F.1070100@youngman.org.uk>
On 9/15/2015 1:08 PM, Wols Lists wrote:
> Throwing something completely different into the mix, how about
> considering btrfs? It's not 100% solid yet, so you need to be careful
> with it, but if you back up with rsync and the "in place" option, it'll
> give you full backups for the cost of incremental.
>
> What you MUST do is KEEP AN EYE ON DISK SPACE! The main failure mode for
> btrfs I'm aware of, is that a disk full can cause a fatal error. As in
> "I've just trashed the disk - it's 'format c:' time". So if you hit 80%
> or so, alarm bells should be ringing. Very loud.
This is the exact reason NOT to use btrfs at the moment. The issue is
with the 'snap shotting' feature which can quietly fill your disk with
snap shots to the point of space exhaustion that has resulted in the
complete loss of information in a number of instances.
Why would you recommend a filesystem when "It's not 100% solid yet" to
someone looking for enhanced data integrity?
Your warning is well taken, and while btrfs is getting better all the
time, for a raid install where the object is to eliminate (or greatly
lessen) the chance of data loss, stick with a tried and true filesystem
unless you want to become an unwitting beta-tester with your data on the
line...
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: Recommendations for RAID setup needed
From: David C. Rankin @ 2015-09-22 4:18 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150915235731.5fcc5f79@natsu>
On 9/15/2015 1:57 PM, Roman Mamedov wrote:
> And this is just an absolute deranged baseless FUD today, or maybe something
> that might have been true about 5 years ago -- eons in Btrfs development.
Not sure that is 100% true. We still see data losses reported to the
various distro mailing lists. openSUSE most recently.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: Recommendations for RAID setup needed
From: Roman Mamedov @ 2015-09-22 5:03 UTC (permalink / raw)
To: David C. Rankin; +Cc: linux-raid
In-Reply-To: <5600D61A.6000908@suddenlinkmail.com>
[-- Attachment #1: Type: text/plain, Size: 861 bytes --]
On Mon, 21 Sep 2015 23:16:26 -0500
"David C. Rankin" <drankinatty@suddenlinkmail.com> wrote:
> This is the exact reason NOT to use btrfs at the moment. The issue is
> with the 'snap shotting' feature which can quietly fill your disk with
> snap shots to the point of space exhaustion that has resulted in the
> complete loss of information in a number of instances.
Snapshots is not something Btrfs "quietly" makes on its own, snapshots are
something *you* create. Snapshots of static and unmodified data are essentially
free in terms of disk space (due to CoW only the changes from the point
snapshot was taken and further on, will require additional space); and if you
heavily modify your files and want to also keep a trail of snapshots, it's of
course up to you to ensure you have enough disk space for that.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* md: How to change homehost without re-assemble md
From: Apple Yin @ 2015-09-22 8:25 UTC (permalink / raw)
To: linux-raid
Hi Neil
As I know, "--update homehost --homehost xxx" is only for mdadm
assemble/create mode. Is there a way to change homehost without stop
md first ? I use the "homehost" to store my private property and want
to change it when md is run. Thanks!
BRs,
AppleYin
^ permalink raw reply
* Re: Fwd: Failed Raid6 Array.....want some guidance before attempting restart
From: Another Sillyname @ 2015-09-22 9:33 UTC (permalink / raw)
To: Alexander Afonyashin, Mikael Abrahamsson, linux-raid
In-Reply-To: <CAOS+5GHYvupo4OC3o0f0R-Nrc49=2EUJ4X7AioaMr3bf=zwUww@mail.gmail.com>
Just to let you know the array seems to have rebuilt fully.
I'm just repopulating the live SQL databases and doing the backup,
then will start the full investigation into what happened and the
restore logs.
Thanks for all the help guys.
On 21 September 2015 at 09:09, Another Sillyname
<anothersname@googlemail.com> wrote:
> Current watch status
>
> Every 30.0s: cat /proc/mdstat
> Mon Sep 21 09:07:03 2015
>
> Personalities : [raid6] [raid5] [raid4]
> mdxxx : active raid6 sdh1[8] sdc1[7] sdg1[0] sdi1[6] sdf1[3] sde1[2] sdd1[1]
> 29301952000 blocks super 1.2 level 6, 512k chunk, algorithm 2
> [7/5] [UUUU__U]
> [==========>..........] recovery = 53.4%
> (3133878924/5860390400) finish=358.2min speed=126856K/sec
> bitmap: 6/44 pages [24KB], 65536KB chunk
>
> unused devices: <none>
>
> On 21 September 2015 at 09:05, Another Sillyname
> <anothersname@googlemail.com> wrote:
>> I think at the moment I'm in leave it alone and let it run
>> mode.....it'll be done in about 6 hours anyway and I'm adverse to
>> 'tampering' with anything while I'm this exposed without any
>> resilience.
>>
>> I meant to state in the earlier message when the rebuild happens next
>> month I'll be installing Fedora 22 (and all the latest updates). This
>> is a high demand server (not high load but requiring high
>> availability) so once rebuilt and running stable for a month it'll get
>> 'locked down' without any changes for another couple of years.
>>
>>
>>
>>
>>
>> On 21 September 2015 at 08:57, Alexander Afonyashin
>> <a.afonyashin@madnet-team.ru> wrote:
>>> Hi,
>>>
>>> You may also try to increase rebuild rate by echo-ing min speed value:
>>>
>>> echo 100000 > /sys/block/mdX/md/sync_speed_min
>>>
>>> or via sysctl:
>>>
>>> sysctl -w dev.raid.speed_limit_min=100000
>>>
>>> Regards,
>>> Alexander
>>>
>>> On Mon, Sep 21, 2015 at 4:59 AM, Another Sillyname
>>> <anothersname@googlemail.com> wrote:
>>>> Ignore last...having thought about it for 10 minutes the obvious thing
>>>> to do is to add the drives back and allow the array to rebuild
>>>> offline......
>>>>
>>>> For the following reasons....
>>>>
>>>> 1. e2fsck -f -n /dev/mdxx reports all the data appears intact and
>>>> that was what I believed anyway based on the information available to
>>>> me.
>>>>
>>>> 2. To finish the backup will take 30+ hours, that's 30+ hours of risk
>>>> time where a single drive failure will compromise the data set.
>>>>
>>>> 3. To 'add' the missing drives back into the array and allow the
>>>> rebuild will take about 10 hours (based on my previous experience
>>>> building this array), therefore the lower 'risk' course of action is
>>>> to rebuild the array, then and only then, to restart the backup.
>>>> There's over 20 hours less risk doing it this way.
>>>>
>>>> I realise I could do the two concurrently but I'd rather keep the
>>>> array 'destressed' as much as possible until I've got at least one
>>>> level of resilience restored.
>>>>
>>>> Having now added the drives back in as 'spares' mdstat is telling me a
>>>> little over 12 hours to do the rebuild so it's now finger crossing
>>>> time time then.
>>>>
>>>> Thanks for the help and advice....and most of all the confirmation my
>>>> approach was the correct one.
>>>>
>>>>
>>>>
>>>> On 21 September 2015 at 02:32, Another Sillyname
>>>> <anothersname@googlemail.com> wrote:
>>>>> OK
>>>>>
>>>>> The array has come back up...but showing two drives as missing.
>>>>>
>>>>> mdadm --query --detail /dev/md127/dev/md127:
>>>>> Version : 1.2
>>>>> Creation Time : Sun May 10 14:47:51 2015
>>>>> Raid Level : raid6
>>>>> Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
>>>>> Used Dev Size : 5860390400 (5588.90 GiB 6001.04 GB)
>>>>> Raid Devices : 7
>>>>> Total Devices : 5
>>>>> Persistence : Superblock is persistent
>>>>>
>>>>> Intent Bitmap : Internal
>>>>>
>>>>> Update Time : Mon Sep 21 02:21:48 2015
>>>>> State : active, degraded
>>>>> Active Devices : 5
>>>>> Working Devices : 5
>>>>> Failed Devices : 0
>>>>> Spare Devices : 0
>>>>>
>>>>> Layout : left-symmetric
>>>>> Chunk Size : 512K
>>>>>
>>>>> Name : arandomserver.arandomlan.com:1
>>>>> UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
>>>>> Events : 285469
>>>>>
>>>>> Number Major Minor RaidDevice State
>>>>> 0 8 97 0 active sync /dev/sdg1
>>>>> 1 8 49 1 active sync /dev/sdd1
>>>>> 2 8 65 2 active sync /dev/sde1
>>>>> 3 8 81 3 active sync /dev/sdf1
>>>>> 8 0 0 8 removed
>>>>> 10 0 0 10 removed
>>>>> 6 8 129 6 active sync /dev/sdi1
>>>>>
>>>>> Data appears to be intact (haven't done a full analysis yet).
>>>>>
>>>>> Does this mean I should add the 'missing' drives back into the array
>>>>> (one at a time obviously)!.
>>>>>
>>>>> Also doesn't this mean I'm horribly exposed to any writes now as this
>>>>> would move the current 5+2 further out of 'sync' with each other thus
>>>>> meaning any further short term fail could smash the data set totally.
>>>>>
>>>>> I'm minded to stop any writes to the array in the short term and
>>>>> continue just doing the backup (this in itself will take about 30+
>>>>> hours).
>>>>>
>>>>> Ideas and observations?
>>>>>
>>>>>
>>>>>
>>>>> On 20 September 2015 at 10:54, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>>>>> On Sun, 20 Sep 2015, Another Sillyname wrote:
>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Would you.....
>>>>>>>
>>>>>>> mdadm --assemble --force --scan
>>>>>>>
>>>>>>> or
>>>>>>>
>>>>>>> mdadm --assemble --force /dev/mdxx /dev/sd[c-i]1
>>>>>>
>>>>>>
>>>>>> This last one is what I use myself.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mikael Abrahamsson email: swmike@swm.pp.se
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Guidance on reshape stuck at 0% after --grow
From: Mikael Abrahamsson @ 2015-09-22 10:29 UTC (permalink / raw)
To: Guillaume Paumier; +Cc: linux-raid
In-Reply-To: <CAKWBTuFk4Tz20B4aZHgPF3FoEYXDqwQeKeAwN3Nt-cSAopUQ0Q@mail.gmail.com>
On Mon, 21 Sep 2015, Guillaume Paumier wrote:
> I'm looking for guidance regarding what to do next. I'm not sure
> whether I should wait longer (how long?), or try to re-assemble the
> seemingly-intact original array (how do I stop/restart the reshape
> cleanly?), or something else. Any advice would be greatly appreciated.
You're not the first one here with this problem.
I can't find it right now, but I believe someone issued --grow --continue
on the array to get it out of the state that you have right now.
I don't know if it's this:
http://www.spinics.net/lists/raid/msg49077.html
I do not recommend a reboot or anything else, others have had problems
with this. Also make sure you collect as much information as possible,
mdadm --examine, copy the superblocks in binary form using dd to somewhere
etc, so you can get back to the state where you are now.
Look through the mailing list archives from the past 6 months, you're not
alone in having this problem.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox