* storage with "very high Average Read/Write Request Time"
From: Peter Grandi @ 2014-10-25 14:47 UTC (permalink / raw)
To: Linux RAID; +Cc: quanjun hu, Stan Hoeppner
In-Reply-To: <54442A48.6050503@hardwarefreak.com>
I am copying here (without endorsement or comment) some extracts by
different people from a thread in the XFS mailing list that seems mostly
about RAID tuning:
http://article.gmane.org/gmane.comp.file-systems.xfs.general/65029
>>> Hi, I am using xfs on a raid 5 (~100TB) and put log on external ssd
>>> device, the mount information is: /dev/sdc on
>>> /data/fhgfs/fhgfs_storage type xfs
>>> (rw,relatime,attr2,delaylog,logdev=/dev/sdb1,sunit=512,swidth=15872,noquota).
>>> when doing only reading / only writing , the speed is very
>>> fast(~1.5G), but when do both the speed is very slow (100M), and
>>> high r_await(160) and w_await(200000).
>>> 1. how can I reduce average request time?
>>> 2. can I use ssd as write/read cache for xfs?
http://article.gmane.org/gmane.comp.file-systems.xfs.general/65034
>> There is a ratio of 31 (thirty one) between 'swidth' and 'sunit' and
>> assuming that this reflects the geometry of the RAID5 set and given
>> commonly available disk sizes it can be guessed that with amazing
>> "bravery" someone has configured a RAID5 out of 32 (thirty two) high
>> capacity/low IOPS 3TB drives, or something similar. It is even
>> "braver" than that: if the device name "/data/fhgfs/fhgfs_storage" is
>> dedscriptive, this "brave" RAID5 set is supposed to hold the object
>> storage layer of a BeeFS highly parallel filesystem, and therefore
>> will likely have mostly-random accesses. This issue should be moved
>> to the 'linux-raid' mailing list as from the reported information it
>> has nothing to do with XFS.
http://article.gmane.org/gmane.comp.file-systems.xfs.general/65036
> You apparently have 31 effective SATA 7.2k RPM spindles with 256 KiB
> chunk, 7.75 MiB stripe width, in RAID5. That should yield 3-4.6 GiB/s
> of streaming throughput assuming no cable, expander, nor HBA
> limitations. You're achieving only 1/3rd to 1/2 of this. Which
> hardware RAID controller is this? What are the specs? Cache RAM,
> host and back end cable count and type?
> When you say read or write is fast individually, but read+write is
> slow, what types of files are you reading and writing, and how many in
> parallel? This combined pattern is likely the cause of the slowdown
> due to excessive seeking in the drives. As others mentioned this isn't
> an XFS problem.
> The problem is that your RAID geometry doesn't match your workload.
> Your very wide parity stripe is apparently causing excessive seeking
> with your read+write workload due to read-modify-write operations. To
> mitigate this, and to increase resiliency, you should switch to RAID6
> with a smaller chunk. If you need maximum capacity make a single
> RAID6 array with 16 KiB chunk size. This will yield a 496 KiB stripe
> width, increasing the odds that all writes are a full stripe, and
> hopefully eliminating much of the RMW problem.
> A better option might be making three 10 drive RAID6 arrays (two
> spares) with 32 KiB chunk, 256 KiB stripe width, and concatenating
> the 3 arrays with mdadm --linear. You'd have 24 spindles of capacity
> and throughput instead of 31, but no more RMW operations, or at least
> very few. You'd format the linear md device with
> mkfs.xfs -d su=32k,sw=8 /dev/mdX
> As long as your file accesses are spread fairly evenly across at
> least 3 directories you should achieve excellent parallel throughput,
> though single file streaming throughput will peak at 800-1200 MiB/s,
> that of 8 drives. With a little understanding of how this setup
> works, you can write two streaming files and read a third without any
> of the 3 competing with one another for disk seeks/bandwidth--which
> is your current problem. Or you could do one read and one write to
> each of 3 directories, and no pair of two would interfere with the
> other pairs. Scale up from here.
> Basically what we're doing is isolating each RAID LUN into a set of
> directories. When you write to one of those directories the file
> goes into only one of the 3 RAID arrays. Doing this isolates RMWs
> for a given write to only a subset of your disks, and minimizes the
> amount of seeks generated by parallel accesses.
^ permalink raw reply
* Re: storage with "very high Average Read/Write Request Time"
From: Peter Grandi @ 2014-10-25 22:07 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <21579.47082.770855.521224@tree.ty.sabi.co.uk>
>>> Hi, I am using xfs on a raid 5 (~100TB) and put log on external ssd
>>> device, the mount information is: /dev/sdc [ ... ] when doing only
>>> reading / only writing , the speed is very fast(~1.5G), but when do
>>> both the speed is very slow (100M), and high r_await(160) and
>>> w_await(200000).
>> There is a ratio of 31 (thirty one) between 'swidth' and 'sunit' and
>> assuming that this reflects the geometry of the RAID5 set and given
>> commonly available disk sizes it can be guessed that with amazing
>> "bravery" someone has configured a RAID5 out of 32 (thirty two) high
>> capacity/low IOPS 3TB drives, or something similar. [ ... ]
>> You apparently have 31 effective SATA 7.2k RPM spindles with 256 KiB
>> chunk, 7.75 MiB stripe width, in RAID5.
That's highly likely, could be nearline SAS drives, but not a big
difference. But given the context SATA is likely as it is cheaper.
Conceivably the storage could also be a chunk of a RAID SAN over a
single FC 16Gb/s link or FCoE over dual bonded roundrobin 10Gb/s links,
but this seems to me to fit less well with the other scant clues
available (e.g. "on a raid 5 (~100TB)" and "log on external ssd
device").
>> That should yield 3-4.6 GiB/s of streaming throughput assuming no
>> cable, expander, nor HBA limitations. You're achieving only 1/3rd to
>> 1/2 of this.
That 1/3 to 1/2 may be not too bad, also considering that the RAID set
is so wide and there will be quite a bit of variance of rotational
positions across it, and that perhaps the hw has channels with a 2GB/s
max chokepoint.
Anyhow these are are some mostly reasonable questions on the details:
>> Which hardware RAID controller is this? What are the specs? Cache
>> RAM, host and back end cable count and type?
I'll do some plausible (hopefully) additional speculation...
The mention of '/dev/sdc' suggests that this is a hw RAID HA (aka HBA),
and the context gives me vibes that:
* The 3TB disks are likely 3.5in, so given typical enclosure geometries
the likely count of 32 suggests that the disks are in two classic
16-slot enclosures.
* That the RAID set seems to have 32 drives and there is the ceiling of
1.5GB/s suggests that there is a single not very recent hw RAID HA and
the two enclosures are daisy-chained.
* Because oldish RAID HAs typically max at 2GiB/s overall as they have
8x lane PCIe 1.x host bus connectors; or they are recent PCIe 2.0 ones
but plugged into oldish PCIe 1.x host bus slots (or perhaps even
plugged into slots with less than 8x lanes).
There is an enormous difference in read vs. write in «r_await(160) and
w_await(200000)» and the latter is apocalyptic at 200 seconds ('man
iostat' confirms that 'r_await' and 'w_await' are in ms). I have seen
that kind of horror before and it suggests that this is one of several
common types of hw RAID HA which a massively misdesigned buggy IO
scheduler and cache manager in the firmware.
>>> 1. how can I reduce average request time?
Some better alternative geometries have been already suggested, some of
which I don't like...
But overall if the speculation above applies, the current setup seems to
be "audaciously" aimed at the lowest possible upfront price, "targeting"
a workload made almost entirely of reads of data archived elsewhere, or
single-stream writes or reads, and that needs changing.
To change that I think that the two main suggestions are:
1. Stop using the hw RAID mode and use instead Linux MD raid.
3. Significantly boost IOPS when doing concurrent read-writes as per
typical HPC distributed filesystem object stores. If the aim was
indeed "lowest possible upfront price" that's not to easy.
As to the latter point it depends a bit whether the object store is
meant to hold transient or permanent data, and the nature of cluster
jobs expected, but here are some general thoughts:
* Ideally reach the same capacity with 1TB drives, as they have a much
better IOPS-per-GB ratio than 3TB drives. I consider for various
reasons current 2TB drives as already having a too low IOPS-per-GB
ratio for most cases of "live" data, for example.
* Change the geometry of the RAID sets, for example multiple sets with
each set being:
- RAID10 with 16 members, or RAID10 with 8 members.
- RAID5 with 3 members, with hot spares.
- RAID5 with 5 members, with hot spares.
- With some reluctance, RAID6 with 6-8-10 members.
* Some apposite tuning of the MD RAID parameters depending on the RAID
set geometry and expected workload, the usual on stripe cache sizes,
elevators, read-ahead, dirty page lifetime and ERC/TLER timeouts.
The reason behind the multiple sets is to reduce correlation, by
reducing RMW, and for example also permitting parallel 'fsck' and
probably also better backups impact.
Having a single very large storage pool per server is particularly
"insipid" if this is meant to be an object store for HPC parallel
filesystems like BeeGFS, because usually they are configured to slice
each files into 1MB segments and distributed these segments across many
available object stores.
>>> 2. can I use ssd as write/read cache for xfs?
I think that if the workload is that of a typical HPC distributed
filesystem object store that's not going to help much as object stores
cannot cache that much because of overall randomish access patterns.
Caches don't "always" increase IOPS across the board. :-)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Raid1 element stuck in (S) state
From: micah anderson @ 2014-10-27 14:18 UTC (permalink / raw)
To: linux-raid
Hi,
i've got a raid1 setup, where one drive died, it was replaced with a new
one, but its stuck in a (S) state and I can't seem to get it added into
the array, /proc/mdstat looks like this:
md3 : active raid1 sdc1[2](S) sdd1[1]
976759672 blocks super 1.2 [2/1] [_U]
where sdc1 is the replaced drive.
What is the right way to get this added back?
thanks!
micah
^ permalink raw reply
* RAID1 removing failed disk returns EBUSY
From: Joe Lawrence @ 2014-10-27 20:27 UTC (permalink / raw)
To: linux-raid; +Cc: Bill Kuzeja
Hi Neil,
We've encountered changes in MD and mdadm that have broken our automated
disk removal script. In the past, we've been able to run the following
after a RAID1 disk component removal:
% echo fail > /sys/block/md3/md/dev-sdr5/state
% echo remove > /sys/block/md3/md/dev-sdr5/state
However, the latest RHEL6.6 code drop has rebased to sufficiently recent
MD kernel and mdadm changes, in which the previous commands occasionally
fail like so:
* MD array is usually resyncing or checking
* Component disk /dev/sdr removed via HBA sysfs PCI removal
* Following UDEV rule fires:
SUBSYSTEM=="block", ACTION=="remove", ENV{ID_PATH}=="?*", \
RUN+="/sbin/mdadm -If $name --path $env{ID_PATH}"
% mdadm --detail /dev/md3
/dev/md3:
Version : 1.1
Creation Time : Tue Oct 14 17:31:59 2014
Raid Level : raid1
Array Size : 25149440 (23.98 GiB 25.75 GB)
Used Dev Size : 25149440 (23.98 GiB 25.75 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Oct 15 14:22:34 2014
State : active, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
Name : localhost.localdomain:3
UUID : 40ed68ee:ba41d4cd:28c361ed:be7470b8
Events : 142
Number Major Minor RaidDevice State
0 65 21 0 faulty
1 65 5 1 active sync /dev/sdj5
All attempts to remove this device fail:
% echo remove > /sys/block/md3/md/dev-sdr5/state
-bash: echo: write error: Device or resource busy
This can be traced to state_store():
} else if (cmd_match(buf, "remove")) {
if (rdev->raid_disk >= 0)
err = -EBUSY;
After much debugging and systemtapping, I think I've figured out that the
sysfs scripting may fail after the following combination of changes:
mdadm 8af530b07fce "Enhance incremental removal."
kernel 30b8feb730f9 "md/raid5: avoid deadlock when raid5 array has unack
badblocks during md_stop_writes"
With these two changes:
1 - On the user side, mdadm is trying to set the array_state to read-auto
on incremental removal (as invoked by UDEV rule).
2 - Kernel side, md_set_readonly() will set the MD_RECOVERY_FROZEN flag,
wake up the mddev->thread and if there is a sync_thread, it will set
MD_RECOVERY_INTR and then wait until the sync_thread is set to NULL.
When md_check_recovery() gets a chance to run as part of the
raid1d() mddev->thread, it may or may not ever get to
an invocation of remove_and_add_spares(), for there are but *many*
conditional early exits along the way -- for example, if
MD_RECOVERY_FROZEN is set, the following condition will bounce out of
the routine:
if (!test_and_clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
test_bit(MD_RECOVERY_FROZEN, &mddev->recovery))
goto unlock;
the next time around, MD_RECOVERY_NEEDED will have been cleared, so
all future tests will return 0 and the negation will always take the
early exit path.
Back in md_set_readonly(), it may notice that the MD is still in use,
so it clears the MD_RECOVERY_FROZEN and then returns -EBUSY, without
setting mddev->ro. But the damage has been done as conditions have
been set such that md_check_recovery() will never call
remove_and_add_spares().
This would also explain why an "idle" sync_action clears the wedge: it
sets MD_RECOVERY_NEEDED allowing md_check_recovery() to continue executing
to remove_and_add_spares().
As far as I can tell, this is what is happening to prevent the "remove"
write to /sys/block/md3/md/dev-sdr5/state from succeeding. There are
certainly a lot of little bit-states between disk removal, UDEV mdadm, and
various MD kernel threads, so apologies if I missed an important
transition.
Would you consider writing "idle" to the MD array sync_action file as a
safe and reasonable intermediate workaround step for our script?
And of course, any suggestions to whether this is intended behavior (ie,
the removed component disk is failed, but stuck in the array)?
This is fairly easy for us to reproduce with multiple MD arrays per disk
(one per partition) and interrupting a raid check on all of them
(especially when they are delayed waiting for the first to finish) by
removing the component disk via sysfs PCI removal. We can provide
additional debug or testing if required.
Regards,
-- Joe
^ permalink raw reply
* Re: Raid1 element stuck in (S) state
From: Joe Lawrence @ 2014-10-27 20:57 UTC (permalink / raw)
To: micah anderson; +Cc: linux-raid
In-Reply-To: <87k33lwq7s.fsf@muck.riseup.net>
On Mon, 27 Oct 2014 10:18:47 -0400
micah anderson <micah@debian.org> wrote:
>
> Hi,
>
> i've got a raid1 setup, where one drive died, it was replaced with a new
> one, but its stuck in a (S) state and I can't seem to get it added into
> the array, /proc/mdstat looks like this:
>
> md3 : active raid1 sdc1[2](S) sdd1[1]
> 976759672 blocks super 1.2 [2/1] [_U]
>
> where sdc1 is the replaced drive.
Hi Micah,
What does the output from mdadm --detail /dev/md3 look like?
-- Joe
^ permalink raw reply
* Re: Raid1 element stuck in (S) state
From: micah @ 2014-10-28 4:45 UTC (permalink / raw)
To: Joe Lawrence, micah anderson; +Cc: linux-raid
In-Reply-To: <20141027165716.44eaab04@jlaw-desktop.mno.stratus.com>
Hi Joe,
Joe Lawrence <joe.lawrence@stratus.com> writes:
> On Mon, 27 Oct 2014 10:18:47 -0400
> micah anderson <micah@debian.org> wrote:
>
>>
>> Hi,
>>
>> i've got a raid1 setup, where one drive died, it was replaced with a new
>> one, but its stuck in a (S) state and I can't seem to get it added into
>> the array, /proc/mdstat looks like this:
>>
>> md3 : active raid1 sdc1[2](S) sdd1[1]
>> 976759672 blocks super 1.2 [2/1] [_U]
>>
>> where sdc1 is the replaced drive.
>
> Hi Micah,
>
> What does the output from mdadm --detail /dev/md3 look like?
# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Fri Oct 21 12:22:03 2011
Raid Level : raid1
Array Size : 976759672 (931.51 GiB 1000.20 GB)
Used Dev Size : 976759672 (931.51 GiB 1000.20 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon Oct 27 21:45:01 2014
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Name : unassigned-hostname:3
UUID : 736c4da2:1e53a976:6b0ff39a:b0ca93c2
Events : 2459508
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 49 1 active sync /dev/sdd1
2 8 33 - spare /dev/sdc1
#
^ permalink raw reply
* Very small bug on assemble --force
From: Ethan Wilson @ 2014-10-28 14:27 UTC (permalink / raw)
To: linux-raid
Very small bug report:
mdadm assemble --force fails to assemble at the very last step after
having adjusted the event count:
root@storage1:/root# mdadm --assemble /dev/md106 --config
/etc/mdadm/mdadm.secondary.conf --force
mdadm: forcing event count in
/dev/mapper/mpathid-350014ee0ac5a112e-part1(3) from 1058 upto 1061
mdadm: forcing event count in
/dev/mapper/mpathid-350014ee056feaaf4-part1(4) from 1058 upto 1061
mdadm: forcing event count in
/dev/mapper/mpathid-350014ee001a93ede-part1(6) from 1058 upto 1061
mdadm: clearing FAULTY flag for device 1 in /dev/md106 for
/dev/mapper/mpathid-350014ee0ac5a112e-part1
mdadm: clearing FAULTY flag for device 5 in /dev/md106 for
/dev/mapper/mpathid-350014ee056feaaf4-part1
mdadm: clearing FAULTY flag for device 8 in /dev/md106 for
/dev/mapper/mpathid-350014ee001a93ede-part1
mdadm: Marking array /dev/md106 as 'clean'
mdadm: /dev/md106 assembled from 8 drives - not enough to start the array.
mdadm is wrong here: 8 devices is indeed the total number of devices for
this array.
the last command started it inactive like this:
md106 : inactive dm-101[0](S) dm-24[9](S) dm-107[6](S) dm-103[5](S)
dm-123[4](S) dm-95[3](S) dm-105[2](S) dm-106[1](S)
4193255424 blocks super 1.2
(maybe the blocks count is also wrong? Not adjusted for parity maybe?
4193255424 / 8.0 * 6.0 == 3144941568.0 almost correct)
it can be fixed simply by doing:
root@storage1:/root# mdadm --stop /dev/md106
mdadm: stopped /dev/md106
root@storage1:/root# mdadm --assemble /dev/md106 --config
/etc/mdadm/secondary.conf --no-degraded
mdadm: /dev/md106 has been started with 8 drives.
now it works:
md106 : active raid6 dm-101[0] dm-24[9] dm-107[6] dm-103[5] dm-123[4]
dm-95[3] dm-105[2] dm-106[1]
3144938496 blocks super 1.2 level 6, 512k chunk, algorithm 2
[8/8] [UUUUUUUU]
bitmap: 0/1 pages [0KB], 1048576KB chunk
Regards
EW
^ permalink raw reply
* can i recover an all spare raid10 array ?
From: Roland RoLaNd @ 2014-10-28 16:22 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
I have two raid arrays on my system:
raid1: /dev/sdd1 /dev/sdh1
raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
two disks had bad sectors: sdd and sdf <<-- they both got hot swapped. i added sdf back to raid10 and recovery took place but adding sdd1 to raid1 proved to be troublesome
as i didn't have anything important on '/' i formatted and installed ubuntu 14 on raid1
now system is up on raid 1, but raid10 (md127) is inactive
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
17580804096 blocks super 1.2
md2 : active raid1 sdh4[0] sdd4[1]
2921839424 blocks super 1.2 [2/2] [UU]
[==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
md1 : active raid1 sdh3[0] sdd3[1]
7996352 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdh2[0] sdd2[1]
292544 blocks super 1.2 [2/2] [UU]
unused devices: <none>
if i try to assemble md127
mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
mdadm: /dev/sde1 is busy - skipping
mdadm: /dev/sda1 is busy - skipping
mdadm: /dev/sdf1 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdg1 is busy - skipping
if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
mdadm: cannot get array info for /dev/md127
if i try:
mdadm --stop /dev/md127
mdadm: stopped /dev/md127
then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
returns:
assembled from 5 drives and 1 rebuilding - not enough to start the array
what does it mean ? is my data lost ?
if i examine one of the md127 raid 10 array disks it shows this:
mdadm --examine /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
Name : ubuntu:data (local to host ubuntu)
Creation Time : Sat May 10 21:54:56 2014
Raid Level : raid10
Raid Devices : 8
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
Update Time : Tue Oct 28 10:07:18 2014
Checksum : 409deeb4 - correct
Events : 8655
Layout : near=2
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA ('A' == active, '.' == missing)
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
the remaining two disks:
mdadm --examine /dev/sdj1
mdadm: No md superblock detected on /dev/sdj1.
mdadm --examine /dev/sdi1
mdadm: No md superblock detected on /dev/sdi1. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: can i recover an all spare raid10 array ?
From: Robin Hill @ 2014-10-28 17:01 UTC (permalink / raw)
To: Roland RoLaNd; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <DUB121-W342C9D8FABB014826217FA8A9F0@phx.gbl>
[-- Attachment #1: Type: text/plain, Size: 4168 bytes --]
On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
> I have two raid arrays on my system:
> raid1: /dev/sdd1 /dev/sdh1
> raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
>
>
> two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
> i added sdf back to raid10 and recovery took place but adding sdd1 to
> raid1 proved to be troublesome
> as i didn't have anything important on '/' i formatted and installed
> ubuntu 14 on raid1
>
> now system is up on raid 1, but raid10 (md127) is inactive
>
> cat /proc/mdstat
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
> 17580804096 blocks super 1.2
>
> md2 : active raid1 sdh4[0] sdd4[1]
> 2921839424 blocks super 1.2 [2/2] [UU]
> [==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
>
> md1 : active raid1 sdh3[0] sdd3[1]
> 7996352 blocks super 1.2 [2/2] [UU]
>
> md0 : active raid1 sdh2[0] sdd2[1]
> 292544 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
> if i try to assemble md127
>
>
> mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
> mdadm: /dev/sde1 is busy - skipping
> mdadm: /dev/sda1 is busy - skipping
> mdadm: /dev/sdf1 is busy - skipping
> mdadm: /dev/sdb1 is busy - skipping
> mdadm: /dev/sdc1 is busy - skipping
> mdadm: /dev/sdg1 is busy - skipping
>
>
> if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
> mdadm: cannot get array info for /dev/md127
>
> if i try:
>
> mdadm --stop /dev/md127
> mdadm: stopped /dev/md127
>
> then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>
> returns:
>
> assembled from 5 drives and 1 rebuilding - not enough to start the array
>
> what does it mean ? is my data lost ?
>
> if i examine one of the md127 raid 10 array disks it shows this:
>
> mdadm --examine /dev/sde1
> /dev/sde1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
> Name : ubuntu:data (local to host ubuntu)
> Creation Time : Sat May 10 21:54:56 2014
> Raid Level : raid10
> Raid Devices : 8
>
> Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
> Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
>
> Update Time : Tue Oct 28 10:07:18 2014
> Checksum : 409deeb4 - correct
> Events : 8655
>
> Layout : near=2
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAAAAAAA ('A' == active, '.' == missing)
>
> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
>
>
> the remaining two disks:
>
> mdadm --examine /dev/sdj1
> mdadm: No md superblock detected on /dev/sdj1.
> mdadm --examine /dev/sdi1
> mdadm: No md superblock detected on /dev/sdi1.
The --examine output indicates the RAID10 array was 8 members, not 6.
As it stands, you are missing two array members (presumably a mirrored
pair as mdadm won't start the array). Without these you're missing 512K
of every 2M in the array, so your data is toast (well, with a lot of
effort you may recover some files under 1.5M in size).
Were you expecting sdi1 and sdj1 to have been part of the original
RAID10 array? Have you removed the superblocks from them at any point?
For completeness, what mdadm and kernel versions are you running?
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* mdadm creates corrupt superblock
From: Hans Kraus @ 2014-10-28 18:06 UTC (permalink / raw)
To: Linux-RAID
Hi,
I debugged my problem a bit more: it seems that mdadm creates a defect
superblock. That's repeatedly happening, but only for one drive. I
copied zeros to that drive (via ddrescue /dev/zero ...) and the drive
looks OK. The info I got:
=================================================================================================================================================================================================
root@nashorn:/home/kraush/work/smartctrl# mdadm --zero-superblock /dev/sde1
mdadm: Unrecognised md component device - /dev/sde1
root@nashorn:/home/kraush/work/smartctrl# mdadm --add /dev/md126 /dev/sde1
mdadm: add new device failed for /dev/sde1 as 4: Invalid argument
root@nashorn:/home/kraush/work/smartctrl# mdadm -E /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
Name : nashorn:126 (local to host nashorn)
Creation Time : Mon Oct 27 15:58:38 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
Array Size : 0
Used Dev Size : 0
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=1953260976 sectors
State : clean
Device UUID : 4d1420c6:79978477:e86412b1:24beebfc
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Oct 28 11:53:49 2014
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6f02d040 - expected 891eedff
Events : 0
Device Role : spare
Array State : RR ('A' == active, '.' == missing, 'R' == replacing)
root@nashorn:/home/kraush/work/smartctrl#
-----------------------------------------------------------------------------------------------------------
root@nashorn:/home/kraush/work/smartctrl# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md127 : active (auto-read-only) raid6 sda1[0] sdk1[6](S) sdl2[7](S)
sdh1[5] sdg1[4] sdf1[3] sdc1[2] sdb1[1]
1953017856 blocks super 1.2 level 6, 512k chunk, algorithm 2
[6/6] [UUUUUU]
bitmap: 0/4 pages [0KB], 65536KB chunk
md10 : active (auto-read-only) raid1 sdi2[0] sdl3[1]
87833408 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md126 : active raid1 sdd1[3] sdj1[2]
488254464 blocks super 1.2 [2/2] [UU]
bitmap: 0/4 pages [0KB], 65536KB chunk
md0 : active raid1 sdl1[4] sdd2[3] sdi1[5]
156157824 blocks super 1.2 [3/1] [U__]
[===================>.] recovery = 95.6% (149290048/156157824)
finish=8.4min speed=13496K/sec
bitmap: 1/2 pages [4KB], 65536KB chunk
unused devices: <none>
root@nashorn:/home/kraush/work/smartctrl#
-----------------------------------------------------------------------------------------------------------
dmesg:
[33299.387382] md: invalid superblock checksum on sde1
[33299.387385] md: sde1 does not have a valid v1.2 superblock, not
importing!
[33299.387408] md: md_import_device returned -22
root@nashorn:/home/kraush/work/smartctrl#
-----------------------------------------------------------------------------------------------------------
root@nashorn:/home/kraush/work/mdadm# mdadm --version
mdadm - v3.3-161-gfed12d4 - 21st August 2014
=================================================================================================================================================================================================
Kind regards, Hans
^ permalink raw reply
* FW: can i recover an all spare raid10 array ?
From: Roland RoLaNd @ 2014-10-28 18:10 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <DUB121-W268A97DFB2DCC142617CED8A9F0@phx.gbl>
sending again as it failed to deliver the first time for an unknown reason
From: r_o_l_a_n_d@hotmail.com
To: robin@robinhill.me.uk
CC: linux-raid@vger.kernel.org
Subject: RE: can i recover an all spare raid10 array ?
Date: Tue, 28 Oct 2014 19:29:25 +0200
Thanks for pitching in.
here are the responses to you questions:
- yes i expected both of them to be part of the array though one of them was just added to the array and didnt finish recovering when raid1 "/" crashed
- i have not removed their superblocks or at least not in a way that i amaware of
- mdadm: 3.2.5-5ubuntu4.1
- uname -a: 3.13.0-24-generic
PS:
I just followed this recovery page:
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
I managed to reach the last step, whenever i tried to mount it kept asking me for the right file system
> Date: Tue, 28 Oct 2014 17:01:11 +0000
> From: robin@robinhill.me.uk
> To: r_o_l_a_n_d@hotmail.com
> CC: linux-raid@vger.kernel.org
> Subject: Re: can i recover an all spare raid10 array ?
>
> On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
>
>> I have two raid arrays on my system:
>> raid1: /dev/sdd1 /dev/sdh1
>> raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
>>
>>
>> two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
>> i added sdf back to raid10 and recovery took place but adding sdd1 to
>> raid1 proved to be troublesome
>> as i didn't have anything important on '/' i formatted and installed
>> ubuntu 14 on raid1
>>
>> now system is up on raid 1, but raid10 (md127) is inactive
>>
>> cat /proc/mdstat
>>
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
>> md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
>> 17580804096 blocks super 1.2
>>
>> md2 : active raid1 sdh4[0] sdd4[1]
>> 2921839424 blocks super 1.2 [2/2] [UU]
>> [==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
>>
>> md1 : active raid1 sdh3[0] sdd3[1]
>> 7996352 blocks super 1.2 [2/2] [UU]
>>
>> md0 : active raid1 sdh2[0] sdd2[1]
>> 292544 blocks super 1.2 [2/2] [UU]
>>
>> unused devices: <none>
>> if i try to assemble md127
>>
>>
>> mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>> mdadm: /dev/sde1 is busy - skipping
>> mdadm: /dev/sda1 is busy - skipping
>> mdadm: /dev/sdf1 is busy - skipping
>> mdadm: /dev/sdb1 is busy - skipping
>> mdadm: /dev/sdc1 is busy - skipping
>> mdadm: /dev/sdg1 is busy - skipping
>>
>>
>> if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
>> mdadm: cannot get array info for /dev/md127
>>
>> if i try:
>>
>> mdadm --stop /dev/md127
>> mdadm: stopped /dev/md127
>>
>> then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>>
>> returns:
>>
>> assembled from 5 drives and 1 rebuilding - not enough to start the array
>>
>> what does it mean ? is my data lost ?
>>
>> if i examine one of the md127 raid 10 array disks it shows this:
>>
>> mdadm --examine /dev/sde1
>> /dev/sde1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
>> Name : ubuntu:data (local to host ubuntu)
>> Creation Time : Sat May 10 21:54:56 2014
>> Raid Level : raid10
>> Raid Devices : 8
>>
>> Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>> Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
>>
>> Update Time : Tue Oct 28 10:07:18 2014
>> Checksum : 409deeb4 - correct
>> Events : 8655
>>
>> Layout : near=2
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAAAAAAA ('A' == active, '.' == missing)
>>
>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
>>
>>
>> the remaining two disks:
>>
>> mdadm --examine /dev/sdj1
>> mdadm: No md superblock detected on /dev/sdj1.
>> mdadm --examine /dev/sdi1
>> mdadm: No md superblock detected on /dev/sdi1.
>
> The --examine output indicates the RAID10 array was 8 members, not 6.
> As it stands, you are missing two array members (presumably a mirrored
> pair as mdadm won't start the array). Without these you're missing 512K
> of every 2M in the array, so your data is toast (well, with a lot of
> effort you may recover some files under 1.5M in size).
>
> Were you expecting sdi1 and sdj1 to have been part of the original
> RAID10 array? Have you removed the superblocks from them at any point?
> For completeness, what mdadm and kernel versions are you running?
>
> Cheers,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* FW: can i recover an all spare raid10 array ?
From: Roland RoLaNd @ 2014-10-28 18:11 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <DUB121-W26D4B9D346A4A0A0407CD38A9F0@phx.gbl>
sending again as it failed the first time for some unknown reason
________________________________
> From: r_o_l_a_n_d@hotmail.com
> To: robin@robinhill.me.uk
> CC: linux-raid@vger.kernel.org
> Subject: RE: can i recover an all spare raid10 array ?
> Date: Tue, 28 Oct 2014 19:30:50 +0200
>
>
> Correction:
> i couldn't force assemble the read devices so i issued instead:
>
> mdadm --create /dev/md089 --assume-clean --level=10 --verbose
> --raid-devices=8 missing /dev/dm-1 /dev/dm-0 /dev/dm-5 /dev/dm-3
> /dev/dm-2 missing /dev/dm-4
>
> which got it into degraded state
>
>
> ________________________________
> From: r_o_l_a_n_d@hotmail.com
> To: robin@robinhill.me.uk
> CC: linux-raid@vger.kernel.org
> Subject: RE: can i recover an all spare raid10 array ?
> Date: Tue, 28 Oct 2014 19:29:25 +0200
>
> Thanks for pitching in.
> here are the responses to you questions:
>
>
> - yes i expected both of them to be part of the array though one of
> them was just added to the array and didnt finish recovering when raid1
> "/" crashed
>
> - i have not removed their superblocks or at least not in a way that i
> amaware of
>
> - mdadm: 3.2.5-5ubuntu4.1
> - uname -a: 3.13.0-24-generic
>
>
> PS:
> I just followed this recovery page:
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
>
> I managed to reach the last step, whenever i tried to mount it kept
> asking me for the right file system
>
>
>> Date: Tue, 28 Oct 2014 17:01:11 +0000
>> From: robin@robinhill.me.uk
>> To: r_o_l_a_n_d@hotmail.com
>> CC: linux-raid@vger.kernel.org
>> Subject: Re: can i recover an all spare raid10 array ?
>>
>> On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
>>
>>> I have two raid arrays on my system:
>>> raid1: /dev/sdd1 /dev/sdh1
>>> raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
>>>
>>>
>>> two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
>>> i added sdf back to raid10 and recovery took place but adding sdd1 to
>>> raid1 proved to be troublesome
>>> as i didn't have anything important on '/' i formatted and installed
>>> ubuntu 14 on raid1
>>>
>>> now system is up on raid 1, but raid10 (md127) is inactive
>>>
>>> cat /proc/mdstat
>>>
>>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6]
> [raid5] [raid4] [raid10]
>>> md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S)
> sdf1[4](S) sda1[3](S)
>>> 17580804096 blocks super 1.2
>>>
>>> md2 : active raid1 sdh4[0] sdd4[1]
>>> 2921839424 blocks super 1.2 [2/2] [UU]
>>> [==>..................] resync = 10.4%
> (304322368/2921839424) finish=672.5min speed=64861K/sec
>>>
>>> md1 : active raid1 sdh3[0] sdd3[1]
>>> 7996352 blocks super 1.2 [2/2] [UU]
>>>
>>> md0 : active raid1 sdh2[0] sdd2[1]
>>> 292544 blocks super 1.2 [2/2] [UU]
>>>
>>> unused devices: <none>
>>> if i try to assemble md127
>>>
>>>
>>> mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1
> /dev/sdb1 /dev/sdc1 /dev/sdg1
>>> mdadm: /dev/sde1 is busy - skipping
>>> mdadm: /dev/sda1 is busy - skipping
>>> mdadm: /dev/sdf1 is busy - skipping
>>> mdadm: /dev/sdb1 is busy - skipping
>>> mdadm: /dev/sdc1 is busy - skipping
>>> mdadm: /dev/sdg1 is busy - skipping
>>>
>>>
>>> if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
>>> mdadm: cannot get array info for /dev/md127
>>>
>>> if i try:
>>>
>>> mdadm --stop /dev/md127
>>> mdadm: stopped /dev/md127
>>>
>>> then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1
> /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>>>
>>> returns:
>>>
>>> assembled from 5 drives and 1 rebuilding - not enough to start the array
>>>
>>> what does it mean ? is my data lost ?
>>>
>>> if i examine one of the md127 raid 10 array disks it shows this:
>>>
>>> mdadm --examine /dev/sde1
>>> /dev/sde1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
>>> Name : ubuntu:data (local to host ubuntu)
>>> Creation Time : Sat May 10 21:54:56 2014
>>> Raid Level : raid10
>>> Raid Devices : 8
>>>
>>> Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>>> Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
>>> Data Offset : 262144 sectors
>>> Super Offset : 8 sectors
>>> State : clean
>>> Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
>>>
>>> Update Time : Tue Oct 28 10:07:18 2014
>>> Checksum : 409deeb4 - correct
>>> Events : 8655
>>>
>>> Layout : near=2
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 2
>>> Array State : AAAAAAAA ('A' == active, '.' == missing)
>>>
>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this
> mean i still have my data ?
>>>
>>>
>>> the remaining two disks:
>>>
>>> mdadm --examine /dev/sdj1
>>> mdadm: No md superblock detected on /dev/sdj1.
>>> mdadm --examine /dev/sdi1
>>> mdadm: No md superblock detected on /dev/sdi1.
>>
>> The --examine output indicates the RAID10 array was 8 members, not 6.
>> As it stands, you are missing two array members (presumably a mirrored
>> pair as mdadm won't start the array). Without these you're missing 512K
>> of every 2M in the array, so your data is toast (well, with a lot of
>> effort you may recover some files under 1.5M in size).
>>
>> Were you expecting sdi1 and sdj1 to have been part of the original
>> RAID10 array? Have you removed the superblocks from them at any point?
>> For completeness, what mdadm and kernel versions are you running?
>>
>> Cheers,
>> Robin
>> --
>> ___
>> ( ' } | Robin Hill <robin@robinhill.me.uk> |
>> / / ) | Little Jim says .... |
>> // !! | "He fallen in de water !!" |
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: can i recover an all spare raid10 array ?
From: Robin Hill @ 2014-10-28 18:34 UTC (permalink / raw)
To: Roland RoLaNd; +Cc: Robin Hill, linux-raid@vger.kernel.org
In-Reply-To: <DUB121-W26D4B9D346A4A0A0407CD38A9F0@phx.gbl>
[-- Attachment #1: Type: text/plain, Size: 7591 bytes --]
Please don't top post, it makes conversations very difficult to follow.
Responses should go at the bottom, or interleaved with the previous post
if responding to particular points. I've moved your previous responses
to keep the conversation flow straight.
On Tue Oct 28, 2014 at 07:30:50PM +0200, Roland RoLaNd wrote:
>
> > From: r_o_l_a_n_d@hotmail.com
> > To: robin@robinhill.me.uk
> > CC: linux-raid@vger.kernel.org
> > Subject: Re: can i recover an all spare raid10 array ?
> > Date: Tue, 28 Oct 2014 19:29:25 +0200
> >
> > > Date: Tue, 28 Oct 2014 17:01:11 +0000
> > > From: robin@robinhill.me.uk
> > > To: r_o_l_a_n_d@hotmail.com
> > > CC: linux-raid@vger.kernel.org
> > > Subject: Re: can i recover an all spare raid10 array ?
> > >
> > > On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
> > >
> > > > I have two raid arrays on my system:
> > > > raid1: /dev/sdd1 /dev/sdh1
> > > > raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
> > > >
> > > >
> > > > two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
> > > > i added sdf back to raid10 and recovery took place but adding sdd1 to
> > > > raid1 proved to be troublesome
> > > > as i didn't have anything important on '/' i formatted and installed
> > > > ubuntu 14 on raid1
> > > >
> > > > now system is up on raid 1, but raid10 (md127) is inactive
> > > >
> > > > cat /proc/mdstat
> > > >
> > > > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > > > md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
> > > > 17580804096 blocks super 1.2
> > > >
> > > > md2 : active raid1 sdh4[0] sdd4[1]
> > > > 2921839424 blocks super 1.2 [2/2] [UU]
> > > > [==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
> > > >
> > > > md1 : active raid1 sdh3[0] sdd3[1]
> > > > 7996352 blocks super 1.2 [2/2] [UU]
> > > >
> > > > md0 : active raid1 sdh2[0] sdd2[1]
> > > > 292544 blocks super 1.2 [2/2] [UU]
> > > >
> > > > unused devices: <none>
> > > > if i try to assemble md127
> > > >
> > > >
> > > > mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
> > > > mdadm: /dev/sde1 is busy - skipping
> > > > mdadm: /dev/sda1 is busy - skipping
> > > > mdadm: /dev/sdf1 is busy - skipping
> > > > mdadm: /dev/sdb1 is busy - skipping
> > > > mdadm: /dev/sdc1 is busy - skipping
> > > > mdadm: /dev/sdg1 is busy - skipping
> > > >
> > > >
> > > > if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
> > > > mdadm: cannot get array info for /dev/md127
> > > >
> > > > if i try:
> > > >
> > > > mdadm --stop /dev/md127
> > > > mdadm: stopped /dev/md127
> > > >
> > > > then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
> > > >
> > > > returns:
> > > >
> > > > assembled from 5 drives and 1 rebuilding - not enough to start the array
> > > >
> > > > what does it mean ? is my data lost ?
> > > >
> > > > if i examine one of the md127 raid 10 array disks it shows this:
> > > >
> > > > mdadm --examine /dev/sde1
> > > > /dev/sde1:
> > > > Magic : a92b4efc
> > > > Version : 1.2
> > > > Feature Map : 0x0
> > > > Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
> > > > Name : ubuntu:data (local to host ubuntu)
> > > > Creation Time : Sat May 10 21:54:56 2014
> > > > Raid Level : raid10
> > > > Raid Devices : 8
> > > >
> > > > Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
> > > > Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
> > > > Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
> > > > Data Offset : 262144 sectors
> > > > Super Offset : 8 sectors
> > > > State : clean
> > > > Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
> > > >
> > > > Update Time : Tue Oct 28 10:07:18 2014
> > > > Checksum : 409deeb4 - correct
> > > > Events : 8655
> > > >
> > > > Layout : near=2
> > > > Chunk Size : 512K
> > > >
> > > > Device Role : Active device 2
> > > > Array State : AAAAAAAA ('A' == active, '.' == missing)
> > > >
> > > > Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
> > > >
> > > >
> > > > the remaining two disks:
> > > >
> > > > mdadm --examine /dev/sdj1
> > > > mdadm: No md superblock detected on /dev/sdj1.
> > > > mdadm --examine /dev/sdi1
> > > > mdadm: No md superblock detected on /dev/sdi1.
> > >
> > > The --examine output indicates the RAID10 array was 8 members, not 6.
> > > As it stands, you are missing two array members (presumably a mirrored
> > > pair as mdadm won't start the array). Without these you're missing 512K
> > > of every 2M in the array, so your data is toast (well, with a lot of
> > > effort you may recover some files under 1.5M in size).
> > >
> > > Were you expecting sdi1 and sdj1 to have been part of the original
> > > RAID10 array? Have you removed the superblocks from them at any point?
> > > For completeness, what mdadm and kernel versions are you running?
> > >
> > > Cheers,
> > > Robin
> >
> > Thanks for pitching in.here are the responses to you questions:
> >
> > - yes i expected both of them to be part of the array though one of
> > them was just added to the array and didnt finish recovering when
> > raid1 "/" crashed
> >
According to your --examine earlier, the RAID10 rebuild had completed
(it shows the array clean and having all disks active). Are you certain
that the new RAID1 array isn't using disks that used to be part of the
RAID10 array? Regardless, I'd expect the disks to have a superblock if
they were part of either array (unless they've been repartitioned?).
> > - i have not removed their superblocks or at least not in a way that i
> > amaware of
> >
> > - mdadm: 3.2.5-5ubuntu4.1
> > - uname -a: 3.13.0-24-generic
> >
That's a pretty old mdadm version, but I don't see anything in the
change logs that looks relevant. Others may be more familiar with issues
though.
> >
> > PS:
> > I just followed this recovery page:
> > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> > I managed to reach the last step, whenever i tried to mount it kept
> > asking me for the right file system
> >
That's good documentation anyway. As long as you stick to the overlay
devices your original data is untouched. It's amazing how many people
run --create on their original disks and lose any chance of getting the
data back.
> Correction:i couldn't force assemble the read devices so i issued instead:
> mdadm --create /dev/md089 --assume-clean --level=10 --verbose --raid-devices=8 missing /dev/dm-1 /dev/dm-0 /dev/dm-5 /dev/dm-3 /dev/dm-2 missing /dev/dm-4
> which got it into degraded state
>
What error did you get when you tried to force assemble (both from mdadm
and anything reported via dmesg)? The device order you're using would
suggest that the missing disks wouldn't be mirrors of each other, so the
data should be okay.
Can you post the --examine results for all the RAID members? Both for
the original partitions and for the overlay devices after you recreated
the array. There may be differences in data offset, etc. which will
break the filesystem.
Cheers,
Robin
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* RE: can i recover an all spare raid10 array ?
From: Roland RoLaNd @ 2014-10-28 19:11 UTC (permalink / raw)
To: Robin Hill; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <20141028183422.GA30582@cthulhu.home.robinhill.me.uk>
----------------------------------------
> Date: Tue, 28 Oct 2014 18:34:22 +0000
> From: robin@robinhill.me.uk
> To: r_o_l_a_n_d@hotmail.com
> CC: robin@robinhill.me.uk; linux-raid@vger.kernel.org
> Subject: Re: can i recover an all spare raid10 array ?
>
> Please don't top post, it makes conversations very difficult to follow.
> Responses should go at the bottom, or interleaved with the previous post
> if responding to particular points. I've moved your previous responses
> to keep the conversation flow straight.
>
> On Tue Oct 28, 2014 at 07:30:50PM +0200, Roland RoLaNd wrote:
>>
>>> From: r_o_l_a_n_d@hotmail.com
>>> To: robin@robinhill.me.uk
>>> CC: linux-raid@vger.kernel.org
>>> Subject: Re: can i recover an all spare raid10 array ?
>>> Date: Tue, 28 Oct 2014 19:29:25 +0200
>>>
>>>> Date: Tue, 28 Oct 2014 17:01:11 +0000
>>>> From: robin@robinhill.me.uk
>>>> To: r_o_l_a_n_d@hotmail.com
>>>> CC: linux-raid@vger.kernel.org
>>>> Subject: Re: can i recover an all spare raid10 array ?
>>>>
>>>> On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
>>>>
>>>>> I have two raid arrays on my system:
>>>>> raid1: /dev/sdd1 /dev/sdh1
>>>>> raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
>>>>>
>>>>>
>>>>> two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
>>>>> i added sdf back to raid10 and recovery took place but adding sdd1 to
>>>>> raid1 proved to be troublesome
>>>>> as i didn't have anything important on '/' i formatted and installed
>>>>> ubuntu 14 on raid1
>>>>>
>>>>> now system is up on raid 1, but raid10 (md127) is inactive
>>>>>
>>>>> cat /proc/mdstat
>>>>>
>>>>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
>>>>> md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
>>>>> 17580804096 blocks super 1.2
>>>>>
>>>>> md2 : active raid1 sdh4[0] sdd4[1]
>>>>> 2921839424 blocks super 1.2 [2/2] [UU]
>>>>> [==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
>>>>>
>>>>> md1 : active raid1 sdh3[0] sdd3[1]
>>>>> 7996352 blocks super 1.2 [2/2] [UU]
>>>>>
>>>>> md0 : active raid1 sdh2[0] sdd2[1]
>>>>> 292544 blocks super 1.2 [2/2] [UU]
>>>>>
>>>>> unused devices: <none>
>>>>> if i try to assemble md127
>>>>>
>>>>>
>>>>> mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>>>>> mdadm: /dev/sde1 is busy - skipping
>>>>> mdadm: /dev/sda1 is busy - skipping
>>>>> mdadm: /dev/sdf1 is busy - skipping
>>>>> mdadm: /dev/sdb1 is busy - skipping
>>>>> mdadm: /dev/sdc1 is busy - skipping
>>>>> mdadm: /dev/sdg1 is busy - skipping
>>>>>
>>>>>
>>>>> if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
>>>>> mdadm: cannot get array info for /dev/md127
>>>>>
>>>>> if i try:
>>>>>
>>>>> mdadm --stop /dev/md127
>>>>> mdadm: stopped /dev/md127
>>>>>
>>>>> then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>>>>>
>>>>> returns:
>>>>>
>>>>> assembled from 5 drives and 1 rebuilding - not enough to start the array
>>>>>
>>>>> what does it mean ? is my data lost ?
>>>>>
>>>>> if i examine one of the md127 raid 10 array disks it shows this:
>>>>>
>>>>> mdadm --examine /dev/sde1
>>>>> /dev/sde1:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x0
>>>>> Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
>>>>> Name : ubuntu:data (local to host ubuntu)
>>>>> Creation Time : Sat May 10 21:54:56 2014
>>>>> Raid Level : raid10
>>>>> Raid Devices : 8
>>>>>
>>>>> Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>>>>> Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
>>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> State : clean
>>>>> Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
>>>>>
>>>>> Update Time : Tue Oct 28 10:07:18 2014
>>>>> Checksum : 409deeb4 - correct
>>>>> Events : 8655
>>>>>
>>>>> Layout : near=2
>>>>> Chunk Size : 512K
>>>>>
>>>>> Device Role : Active device 2
>>>>> Array State : AAAAAAAA ('A' == active, '.' == missing)
>>>>>
>>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
>>>>>
>>>>>
>>>>> the remaining two disks:
>>>>>
>>>>> mdadm --examine /dev/sdj1
>>>>> mdadm: No md superblock detected on /dev/sdj1.
>>>>> mdadm --examine /dev/sdi1
>>>>> mdadm: No md superblock detected on /dev/sdi1.
>>>>
>>>> The --examine output indicates the RAID10 array was 8 members, not 6.
>>>> As it stands, you are missing two array members (presumably a mirrored
>>>> pair as mdadm won't start the array). Without these you're missing 512K
>>>> of every 2M in the array, so your data is toast (well, with a lot of
>>>> effort you may recover some files under 1.5M in size).
>>>>
>>>> Were you expecting sdi1 and sdj1 to have been part of the original
>>>> RAID10 array? Have you removed the superblocks from them at any point?
>>>> For completeness, what mdadm and kernel versions are you running?
>>>>
>>>> Cheers,
>>>> Robin
>>>
>>> Thanks for pitching in.here are the responses to you questions:
>>>
>>> - yes i expected both of them to be part of the array though one of
>>> them was just added to the array and didnt finish recovering when
>>> raid1 "/" crashed
>>>
> According to your --examine earlier, the RAID10 rebuild had completed
> (it shows the array clean and having all disks active). Are you certain
> that the new RAID1 array isn't using disks that used to be part of the
> RAID10 array? Regardless, I'd expect the disks to have a superblock if
> they were part of either array (unless they've been repartitioned?).
>
the examine earlier was to one of the 6 disks that belong to the current inactive array.. they're all clean
as for raid1/10 arrays, that's what i thought as it happened with me before, but lsblk shows the following:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 2.7T 0 disk
└─sda1 8:1 0 2.7T 0 part
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 2.7T 0 part
sdc 8:32 0 2.7T 0 disk
└─sdc1 8:33 0 2.7T 0 part
sdd 8:48 0 2.7T 0 disk
├─sdd1 8:49 0 1M 0 part
├─sdd2 8:50 0 286M 0 part
│ └─md0 9:0 0 285.7M 0 raid1 /boot
├─sdd3 8:51 0 7.6G 0 part
│ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
└─sdd4 8:52 0 2.7T 0 part
└─md2 9:2 0 2.7T 0 raid1 /
sde 8:64 0 2.7T 0 disk
└─sde1 8:65 0 2.7T 0 part
sdf 8:80 0 2.7T 0 disk
└─sdf1 8:81 0 2.7T 0 part
sdg 8:96 0 2.7T 0 disk
└─sdg1 8:97 0 2.7T 0 part
sdh 8:112 0 2.7T 0 disk
├─sdh1 8:113 0 1M 0 part
├─sdh2 8:114 0 286M 0 part
│ └─md0 9:0 0 285.7M 0 raid1 /boot
├─sdh3 8:115 0 7.6G 0 part
│ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
└─sdh4 8:116 0 2.7T 0 part
└─md2 9:2 0 2.7T 0 raid1 /
sdi 8:128 0 2.7T 0 disk
└─sdi1 8:129 0 2.7T 0 part
sdj 8:144 0 2.7T 0 disk
└─sdj1 8:145 0 2.7T 0 part
>>> - i have not removed their superblocks or at least not in a way that i
>>> amaware of
>>>
>>> - mdadm: 3.2.5-5ubuntu4.1
>>> - uname -a: 3.13.0-24-generic
>>>
> That's a pretty old mdadm version, but I don't see anything in the
> change logs that looks relevant. Others may be more familiar with issues
> though.
that's the latest in my current ubuntu repository
>
>>>
>>> PS:
>>> I just followed this recovery page:
>>> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
>>> I managed to reach the last step, whenever i tried to mount it kept
>>> asking me for the right file system
>>>
> That's good documentation anyway. As long as you stick to the overlay
> devices your original data is untouched. It's amazing how many people
> run --create on their original disks and lose any chance of getting the
> data back.
unfortunately i used to be/am one of those people.
had bad experiences with this before, so i took it slow and went with the overlay documentation.
all ebooks i could found about raid speak about different between multiple raid levels but none are thorough when it comes to setting up/troubleshooting raid.
and once i do fix my issue, i move on to the next firefighting situation so i lose interest due to lack of time.
>
>> Correction:i couldn't force assemble the read devices so i issued instead:
>> mdadm --create /dev/md089 --assume-clean --level=10 --verbose --raid-devices=8 missing /dev/dm-1 /dev/dm-0 /dev/dm-5 /dev/dm-3 /dev/dm-2 missing /dev/dm-4
>> which got it into degraded state
>>
>
> What error did you get when you tried to force assemble (both from mdadm
> and anything reported via dmesg)? The device order you're using would
> suggest that the missing disks wouldn't be mirrors of each other, so the
> data should be okay.
mdadm --assemble --force /dev/md100 $OVERLAYS
mdadm: /dev/md100 assembled from 5 drives and 1 rebuilding - not enough to start the array.
dmesg:
[ 6025.573964] md: md100 stopped.
[ 6025.595810] md: bind<dm-0>
[ 6025.596086] md: bind<dm-5>
[ 6025.596364] md: bind<dm-2>
[ 6025.596612] md: bind<dm-1>
[ 6025.596840] md: bind<dm-4>
[ 6025.597026] md: bind<dm-3>
>
> Can you post the --examine results for all the RAID members? Both for
> the original partitions and for the overlay devices after you recreated
> the array. There may be differences in data offset, etc. which will
> break the filesystem.
Original partitions:
http://pastebin.com/nHCxidvE
overlay:
http://pastebin.com/eva4cnu6
>
> Cheers,
> Robin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: can i recover an all spare raid10 array ?
From: Robin Hill @ 2014-10-28 20:02 UTC (permalink / raw)
To: Roland RoLaNd; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <DUB121-W224F0805F3F731496386978A9F0@phx.gbl>
[-- Attachment #1: Type: text/plain, Size: 12811 bytes --]
On Tue Oct 28, 2014 at 09:11:21PM +0200, Roland RoLaNd wrote:
>
>
> ----------------------------------------
> > Date: Tue, 28 Oct 2014 18:34:22 +0000
> > From: robin@robinhill.me.uk
> > To: r_o_l_a_n_d@hotmail.com
> > CC: robin@robinhill.me.uk; linux-raid@vger.kernel.org
> > Subject: Re: can i recover an all spare raid10 array ?
> >
> > Please don't top post, it makes conversations very difficult to follow.
> > Responses should go at the bottom, or interleaved with the previous post
> > if responding to particular points. I've moved your previous responses
> > to keep the conversation flow straight.
> >
> > On Tue Oct 28, 2014 at 07:30:50PM +0200, Roland RoLaNd wrote:
> >>
> >>> From: r_o_l_a_n_d@hotmail.com
> >>> To: robin@robinhill.me.uk
> >>> CC: linux-raid@vger.kernel.org
> >>> Subject: Re: can i recover an all spare raid10 array ?
> >>> Date: Tue, 28 Oct 2014 19:29:25 +0200
> >>>
> >>>> Date: Tue, 28 Oct 2014 17:01:11 +0000
> >>>> From: robin@robinhill.me.uk
> >>>> To: r_o_l_a_n_d@hotmail.com
> >>>> CC: linux-raid@vger.kernel.org
> >>>> Subject: Re: can i recover an all spare raid10 array ?
> >>>>
> >>>> On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
> >>>>
> >>>>> I have two raid arrays on my system:
> >>>>> raid1: /dev/sdd1 /dev/sdh1
> >>>>> raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
> >>>>>
> >>>>>
> >>>>> two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
> >>>>> i added sdf back to raid10 and recovery took place but adding sdd1 to
> >>>>> raid1 proved to be troublesome
> >>>>> as i didn't have anything important on '/' i formatted and installed
> >>>>> ubuntu 14 on raid1
> >>>>>
> >>>>> now system is up on raid 1, but raid10 (md127) is inactive
> >>>>>
> >>>>> cat /proc/mdstat
> >>>>>
> >>>>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> >>>>> md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
> >>>>> 17580804096 blocks super 1.2
> >>>>>
> >>>>> md2 : active raid1 sdh4[0] sdd4[1]
> >>>>> 2921839424 blocks super 1.2 [2/2] [UU]
> >>>>> [==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
> >>>>>
> >>>>> md1 : active raid1 sdh3[0] sdd3[1]
> >>>>> 7996352 blocks super 1.2 [2/2] [UU]
> >>>>>
> >>>>> md0 : active raid1 sdh2[0] sdd2[1]
> >>>>> 292544 blocks super 1.2 [2/2] [UU]
> >>>>>
> >>>>> unused devices: <none>
> >>>>> if i try to assemble md127
> >>>>>
> >>>>>
> >>>>> mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
> >>>>> mdadm: /dev/sde1 is busy - skipping
> >>>>> mdadm: /dev/sda1 is busy - skipping
> >>>>> mdadm: /dev/sdf1 is busy - skipping
> >>>>> mdadm: /dev/sdb1 is busy - skipping
> >>>>> mdadm: /dev/sdc1 is busy - skipping
> >>>>> mdadm: /dev/sdg1 is busy - skipping
> >>>>>
> >>>>>
> >>>>> if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
> >>>>> mdadm: cannot get array info for /dev/md127
> >>>>>
> >>>>> if i try:
> >>>>>
> >>>>> mdadm --stop /dev/md127
> >>>>> mdadm: stopped /dev/md127
> >>>>>
> >>>>> then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
> >>>>>
> >>>>> returns:
> >>>>>
> >>>>> assembled from 5 drives and 1 rebuilding - not enough to start the array
> >>>>>
> >>>>> what does it mean ? is my data lost ?
> >>>>>
> >>>>> if i examine one of the md127 raid 10 array disks it shows this:
> >>>>>
> >>>>> mdadm --examine /dev/sde1
> >>>>> /dev/sde1:
> >>>>> Magic : a92b4efc
> >>>>> Version : 1.2
> >>>>> Feature Map : 0x0
> >>>>> Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
> >>>>> Name : ubuntu:data (local to host ubuntu)
> >>>>> Creation Time : Sat May 10 21:54:56 2014
> >>>>> Raid Level : raid10
> >>>>> Raid Devices : 8
> >>>>>
> >>>>> Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
> >>>>> Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
> >>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
> >>>>> Data Offset : 262144 sectors
> >>>>> Super Offset : 8 sectors
> >>>>> State : clean
> >>>>> Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
> >>>>>
> >>>>> Update Time : Tue Oct 28 10:07:18 2014
> >>>>> Checksum : 409deeb4 - correct
> >>>>> Events : 8655
> >>>>>
> >>>>> Layout : near=2
> >>>>> Chunk Size : 512K
> >>>>>
> >>>>> Device Role : Active device 2
> >>>>> Array State : AAAAAAAA ('A' == active, '.' == missing)
> >>>>>
> >>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
> >>>>>
> >>>>>
> >>>>> the remaining two disks:
> >>>>>
> >>>>> mdadm --examine /dev/sdj1
> >>>>> mdadm: No md superblock detected on /dev/sdj1.
> >>>>> mdadm --examine /dev/sdi1
> >>>>> mdadm: No md superblock detected on /dev/sdi1.
> >>>>
> >>>> The --examine output indicates the RAID10 array was 8 members, not 6.
> >>>> As it stands, you are missing two array members (presumably a mirrored
> >>>> pair as mdadm won't start the array). Without these you're missing 512K
> >>>> of every 2M in the array, so your data is toast (well, with a lot of
> >>>> effort you may recover some files under 1.5M in size).
> >>>>
> >>>> Were you expecting sdi1 and sdj1 to have been part of the original
> >>>> RAID10 array? Have you removed the superblocks from them at any point?
> >>>> For completeness, what mdadm and kernel versions are you running?
> >>>>
> >>>> Cheers,
> >>>> Robin
> >>>
> >>> Thanks for pitching in.here are the responses to you questions:
> >>>
> >>> - yes i expected both of them to be part of the array though one of
> >>> them was just added to the array and didnt finish recovering when
> >>> raid1 "/" crashed
> >>>
> > According to your --examine earlier, the RAID10 rebuild had completed
> > (it shows the array clean and having all disks active). Are you certain
> > that the new RAID1 array isn't using disks that used to be part of the
> > RAID10 array? Regardless, I'd expect the disks to have a superblock if
> > they were part of either array (unless they've been repartitioned?).
> >
>
> the examine earlier was to one of the 6 disks that belong to the current inactive array.. they're all clean
> as for raid1/10 arrays, that's what i thought as it happened with me before, but lsblk shows the following:
>
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 2.7T 0 disk
> └─sda1 8:1 0 2.7T 0 part
> sdb 8:16 0 2.7T 0 disk
> └─sdb1 8:17 0 2.7T 0 part
> sdc 8:32 0 2.7T 0 disk
> └─sdc1 8:33 0 2.7T 0 part
> sdd 8:48 0 2.7T 0 disk
> ├─sdd1 8:49 0 1M 0 part
> ├─sdd2 8:50 0 286M 0 part
> │ └─md0 9:0 0 285.7M 0 raid1 /boot
> ├─sdd3 8:51 0 7.6G 0 part
> │ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
> └─sdd4 8:52 0 2.7T 0 part
> └─md2 9:2 0 2.7T 0 raid1 /
> sde 8:64 0 2.7T 0 disk
> └─sde1 8:65 0 2.7T 0 part
> sdf 8:80 0 2.7T 0 disk
> └─sdf1 8:81 0 2.7T 0 part
> sdg 8:96 0 2.7T 0 disk
> └─sdg1 8:97 0 2.7T 0 part
> sdh 8:112 0 2.7T 0 disk
> ├─sdh1 8:113 0 1M 0 part
> ├─sdh2 8:114 0 286M 0 part
> │ └─md0 9:0 0 285.7M 0 raid1 /boot
> ├─sdh3 8:115 0 7.6G 0 part
> │ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
> └─sdh4 8:116 0 2.7T 0 part
> └─md2 9:2 0 2.7T 0 raid1 /
> sdi 8:128 0 2.7T 0 disk
> └─sdi1 8:129 0 2.7T 0 part
> sdj 8:144 0 2.7T 0 disk
> └─sdj1 8:145 0 2.7T 0 part
>
>
> >>> - i have not removed their superblocks or at least not in a way that i
> >>> amaware of
> >>>
> >>> - mdadm: 3.2.5-5ubuntu4.1
> >>> - uname -a: 3.13.0-24-generic
> >>>
> > That's a pretty old mdadm version, but I don't see anything in the
> > change logs that looks relevant. Others may be more familiar with issues
> > though.
>
> that's the latest in my current ubuntu repository
>
> >
> >>>
> >>> PS:
> >>> I just followed this recovery page:
> >>> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> >>> I managed to reach the last step, whenever i tried to mount it kept
> >>> asking me for the right file system
> >>>
> > That's good documentation anyway. As long as you stick to the overlay
> > devices your original data is untouched. It's amazing how many people
> > run --create on their original disks and lose any chance of getting the
> > data back.
>
> unfortunately i used to be/am one of those people.
> had bad experiences with this before, so i took it slow and went with
> the overlay documentation.
> all ebooks i could found about raid speak about different between
> multiple raid levels but none are thorough when it comes to setting
> up/troubleshooting raid.
> and once i do fix my issue, i move on to the next firefighting
> situation so i lose interest due to lack of time.
>
> >
> >> Correction:i couldn't force assemble the read devices so i issued instead:
> >> mdadm --create /dev/md089 --assume-clean --level=10 --verbose --raid-devices=8 missing /dev/dm-1 /dev/dm-0 /dev/dm-5 /dev/dm-3 /dev/dm-2 missing /dev/dm-4
> >> which got it into degraded state
> >>
> >
> > What error did you get when you tried to force assemble (both from mdadm
> > and anything reported via dmesg)? The device order you're using would
> > suggest that the missing disks wouldn't be mirrors of each other, so the
> > data should be okay.
>
> mdadm --assemble --force /dev/md100 $OVERLAYS
> mdadm: /dev/md100 assembled from 5 drives and 1 rebuilding - not enough to start the array.
>
That's very odd - all the --examine results for the original disks show
the array as clean. That would suggest an issue with the installed
version of mdadm but it doesn't really matter in this case - see below.
> dmesg:
> [ 6025.573964] md: md100 stopped.
> [ 6025.595810] md: bind<dm-0>
> [ 6025.596086] md: bind<dm-5>
> [ 6025.596364] md: bind<dm-2>
> [ 6025.596612] md: bind<dm-1>
> [ 6025.596840] md: bind<dm-4>
> [ 6025.597026] md: bind<dm-3>
>
> >
> > Can you post the --examine results for all the RAID members? Both for
> > the original partitions and for the overlay devices after you recreated
> > the array. There may be differences in data offset, etc. which will
> > break the filesystem.
>
> Original partitions:
> http://pastebin.com/nHCxidvE
>
> overlay:
> http://pastebin.com/eva4cnu6
>
Right - these show you have the wrong order. The original partition
array device roles are:
sdc1: 2
sda1: 3
sdf1: 4
sdb1: 5
sdc1: 6
sdg1: 7
and your overlays are:
dm-1: 1
dm-0: 2
dm-5: 3
dm-3: 4
dm-2: 5
dm-4: 7
So the bad news is that you're missing roles 0 & 1, which will be
mirrors. That means your array is broken unless any other member disks
can be found.
If you're certain that sdi1 and sdj1 should be in the array then you can
try recreating the array (in the correct order) and using sdi1/sdj1 in
the missing slots and see if one option works. I'll assume the overlay
mapping is as follows (if not, remap as required):
sda1 -> dm-0
sdb1 -> dm-1
sdc1 -> dm-2
sde1 -> dm-3
sdf1 -> dm-4
sdg1 -> dm-5
sdi1 -> dm-6
sdj1 -> dm-7
For each of the following orders, you're going to need to:
- stop the existing array (mdadm -S /dev/md089)
- create a new array using --assume-clean
- check for an valid filesystem (fsck -n /dev/md089)
If the fsck returns without errors then try mounting the filesystem and
see if all looks okay, otherwise move on to the next order.
The orders to try are:
- /dev/dm-6 missing /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
- missing /dev/dm-6 /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
- /dev/dm-7 missing /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
- missing /dev/dm-7 /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
Good luck,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* RE: can i recover an all spare raid10 array ?
From: Roland RoLaNd @ 2014-10-28 20:17 UTC (permalink / raw)
To: Robin Hill; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <20141028200239.GA31861@cthulhu.home.robinhill.me.uk>
----------------------------------------
> Date: Tue, 28 Oct 2014 20:02:39 +0000
> From: robin@robinhill.me.uk
> To: r_o_l_a_n_d@hotmail.com
> CC: linux-raid@vger.kernel.org
> Subject: Re: can i recover an all spare raid10 array ?
>
> On Tue Oct 28, 2014 at 09:11:21PM +0200, Roland RoLaNd wrote:
>
>>
>>
>> ----------------------------------------
>>> Date: Tue, 28 Oct 2014 18:34:22 +0000
>>> From: robin@robinhill.me.uk
>>> To: r_o_l_a_n_d@hotmail.com
>>> CC: robin@robinhill.me.uk; linux-raid@vger.kernel.org
>>> Subject: Re: can i recover an all spare raid10 array ?
>>>
>>> Please don't top post, it makes conversations very difficult to follow.
>>> Responses should go at the bottom, or interleaved with the previous post
>>> if responding to particular points. I've moved your previous responses
>>> to keep the conversation flow straight.
>>>
>>> On Tue Oct 28, 2014 at 07:30:50PM +0200, Roland RoLaNd wrote:
>>>>
>>>>> From: r_o_l_a_n_d@hotmail.com
>>>>> To: robin@robinhill.me.uk
>>>>> CC: linux-raid@vger.kernel.org
>>>>> Subject: Re: can i recover an all spare raid10 array ?
>>>>> Date: Tue, 28 Oct 2014 19:29:25 +0200
>>>>>
>>>>>> Date: Tue, 28 Oct 2014 17:01:11 +0000
>>>>>> From: robin@robinhill.me.uk
>>>>>> To: r_o_l_a_n_d@hotmail.com
>>>>>> CC: linux-raid@vger.kernel.org
>>>>>> Subject: Re: can i recover an all spare raid10 array ?
>>>>>>
>>>>>> On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
>>>>>>
>>>>>>> I have two raid arrays on my system:
>>>>>>> raid1: /dev/sdd1 /dev/sdh1
>>>>>>> raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
>>>>>>>
>>>>>>>
>>>>>>> two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
>>>>>>> i added sdf back to raid10 and recovery took place but adding sdd1 to
>>>>>>> raid1 proved to be troublesome
>>>>>>> as i didn't have anything important on '/' i formatted and installed
>>>>>>> ubuntu 14 on raid1
>>>>>>>
>>>>>>> now system is up on raid 1, but raid10 (md127) is inactive
>>>>>>>
>>>>>>> cat /proc/mdstat
>>>>>>>
>>>>>>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
>>>>>>> md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
>>>>>>> 17580804096 blocks super 1.2
>>>>>>>
>>>>>>> md2 : active raid1 sdh4[0] sdd4[1]
>>>>>>> 2921839424 blocks super 1.2 [2/2] [UU]
>>>>>>> [==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
>>>>>>>
>>>>>>> md1 : active raid1 sdh3[0] sdd3[1]
>>>>>>> 7996352 blocks super 1.2 [2/2] [UU]
>>>>>>>
>>>>>>> md0 : active raid1 sdh2[0] sdd2[1]
>>>>>>> 292544 blocks super 1.2 [2/2] [UU]
>>>>>>>
>>>>>>> unused devices: <none>
>>>>>>> if i try to assemble md127
>>>>>>>
>>>>>>>
>>>>>>> mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>>>>>>> mdadm: /dev/sde1 is busy - skipping
>>>>>>> mdadm: /dev/sda1 is busy - skipping
>>>>>>> mdadm: /dev/sdf1 is busy - skipping
>>>>>>> mdadm: /dev/sdb1 is busy - skipping
>>>>>>> mdadm: /dev/sdc1 is busy - skipping
>>>>>>> mdadm: /dev/sdg1 is busy - skipping
>>>>>>>
>>>>>>>
>>>>>>> if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
>>>>>>> mdadm: cannot get array info for /dev/md127
>>>>>>>
>>>>>>> if i try:
>>>>>>>
>>>>>>> mdadm --stop /dev/md127
>>>>>>> mdadm: stopped /dev/md127
>>>>>>>
>>>>>>> then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
>>>>>>>
>>>>>>> returns:
>>>>>>>
>>>>>>> assembled from 5 drives and 1 rebuilding - not enough to start the array
>>>>>>>
>>>>>>> what does it mean ? is my data lost ?
>>>>>>>
>>>>>>> if i examine one of the md127 raid 10 array disks it shows this:
>>>>>>>
>>>>>>> mdadm --examine /dev/sde1
>>>>>>> /dev/sde1:
>>>>>>> Magic : a92b4efc
>>>>>>> Version : 1.2
>>>>>>> Feature Map : 0x0
>>>>>>> Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
>>>>>>> Name : ubuntu:data (local to host ubuntu)
>>>>>>> Creation Time : Sat May 10 21:54:56 2014
>>>>>>> Raid Level : raid10
>>>>>>> Raid Devices : 8
>>>>>>>
>>>>>>> Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>>>>>>> Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
>>>>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
>>>>>>> Data Offset : 262144 sectors
>>>>>>> Super Offset : 8 sectors
>>>>>>> State : clean
>>>>>>> Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
>>>>>>>
>>>>>>> Update Time : Tue Oct 28 10:07:18 2014
>>>>>>> Checksum : 409deeb4 - correct
>>>>>>> Events : 8655
>>>>>>>
>>>>>>> Layout : near=2
>>>>>>> Chunk Size : 512K
>>>>>>>
>>>>>>> Device Role : Active device 2
>>>>>>> Array State : AAAAAAAA ('A' == active, '.' == missing)
>>>>>>>
>>>>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
>>>>>>>
>>>>>>>
>>>>>>> the remaining two disks:
>>>>>>>
>>>>>>> mdadm --examine /dev/sdj1
>>>>>>> mdadm: No md superblock detected on /dev/sdj1.
>>>>>>> mdadm --examine /dev/sdi1
>>>>>>> mdadm: No md superblock detected on /dev/sdi1.
>>>>>>
>>>>>> The --examine output indicates the RAID10 array was 8 members, not 6.
>>>>>> As it stands, you are missing two array members (presumably a mirrored
>>>>>> pair as mdadm won't start the array). Without these you're missing 512K
>>>>>> of every 2M in the array, so your data is toast (well, with a lot of
>>>>>> effort you may recover some files under 1.5M in size).
>>>>>>
>>>>>> Were you expecting sdi1 and sdj1 to have been part of the original
>>>>>> RAID10 array? Have you removed the superblocks from them at any point?
>>>>>> For completeness, what mdadm and kernel versions are you running?
>>>>>>
>>>>>> Cheers,
>>>>>> Robin
>>>>>
>>>>> Thanks for pitching in.here are the responses to you questions:
>>>>>
>>>>> - yes i expected both of them to be part of the array though one of
>>>>> them was just added to the array and didnt finish recovering when
>>>>> raid1 "/" crashed
>>>>>
>>> According to your --examine earlier, the RAID10 rebuild had completed
>>> (it shows the array clean and having all disks active). Are you certain
>>> that the new RAID1 array isn't using disks that used to be part of the
>>> RAID10 array? Regardless, I'd expect the disks to have a superblock if
>>> they were part of either array (unless they've been repartitioned?).
>>>
>>
>> the examine earlier was to one of the 6 disks that belong to the current inactive array.. they're all clean
>> as for raid1/10 arrays, that's what i thought as it happened with me before, but lsblk shows the following:
>>
>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>> sda 8:0 0 2.7T 0 disk
>> └─sda1 8:1 0 2.7T 0 part
>> sdb 8:16 0 2.7T 0 disk
>> └─sdb1 8:17 0 2.7T 0 part
>> sdc 8:32 0 2.7T 0 disk
>> └─sdc1 8:33 0 2.7T 0 part
>> sdd 8:48 0 2.7T 0 disk
>> ├─sdd1 8:49 0 1M 0 part
>> ├─sdd2 8:50 0 286M 0 part
>> │ └─md0 9:0 0 285.7M 0 raid1 /boot
>> ├─sdd3 8:51 0 7.6G 0 part
>> │ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
>> └─sdd4 8:52 0 2.7T 0 part
>> └─md2 9:2 0 2.7T 0 raid1 /
>> sde 8:64 0 2.7T 0 disk
>> └─sde1 8:65 0 2.7T 0 part
>> sdf 8:80 0 2.7T 0 disk
>> └─sdf1 8:81 0 2.7T 0 part
>> sdg 8:96 0 2.7T 0 disk
>> └─sdg1 8:97 0 2.7T 0 part
>> sdh 8:112 0 2.7T 0 disk
>> ├─sdh1 8:113 0 1M 0 part
>> ├─sdh2 8:114 0 286M 0 part
>> │ └─md0 9:0 0 285.7M 0 raid1 /boot
>> ├─sdh3 8:115 0 7.6G 0 part
>> │ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
>> └─sdh4 8:116 0 2.7T 0 part
>> └─md2 9:2 0 2.7T 0 raid1 /
>> sdi 8:128 0 2.7T 0 disk
>> └─sdi1 8:129 0 2.7T 0 part
>> sdj 8:144 0 2.7T 0 disk
>> └─sdj1 8:145 0 2.7T 0 part
>>
>>
>>>>> - i have not removed their superblocks or at least not in a way that i
>>>>> amaware of
>>>>>
>>>>> - mdadm: 3.2.5-5ubuntu4.1
>>>>> - uname -a: 3.13.0-24-generic
>>>>>
>>> That's a pretty old mdadm version, but I don't see anything in the
>>> change logs that looks relevant. Others may be more familiar with issues
>>> though.
>>
>> that's the latest in my current ubuntu repository
>>
>>>
>>>>>
>>>>> PS:
>>>>> I just followed this recovery page:
>>>>> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
>>>>> I managed to reach the last step, whenever i tried to mount it kept
>>>>> asking me for the right file system
>>>>>
>>> That's good documentation anyway. As long as you stick to the overlay
>>> devices your original data is untouched. It's amazing how many people
>>> run --create on their original disks and lose any chance of getting the
>>> data back.
>>
>> unfortunately i used to be/am one of those people.
>> had bad experiences with this before, so i took it slow and went with
>> the overlay documentation.
>> all ebooks i could found about raid speak about different between
>> multiple raid levels but none are thorough when it comes to setting
>> up/troubleshooting raid.
>> and once i do fix my issue, i move on to the next firefighting
>> situation so i lose interest due to lack of time.
>>
>>>
>>>> Correction:i couldn't force assemble the read devices so i issued instead:
>>>> mdadm --create /dev/md089 --assume-clean --level=10 --verbose --raid-devices=8 missing /dev/dm-1 /dev/dm-0 /dev/dm-5 /dev/dm-3 /dev/dm-2 missing /dev/dm-4
>>>> which got it into degraded state
>>>>
>>>
>>> What error did you get when you tried to force assemble (both from mdadm
>>> and anything reported via dmesg)? The device order you're using would
>>> suggest that the missing disks wouldn't be mirrors of each other, so the
>>> data should be okay.
>>
>> mdadm --assemble --force /dev/md100 $OVERLAYS
>> mdadm: /dev/md100 assembled from 5 drives and 1 rebuilding - not enough to start the array.
>>
> That's very odd - all the --examine results for the original disks show
> the array as clean. That would suggest an issue with the installed
> version of mdadm but it doesn't really matter in this case - see below.
when i got ubuntu 14 installed, i issued apt-get update && apt-get upgrade -y
would that have affected anything ?
>
>> dmesg:
>> [ 6025.573964] md: md100 stopped.
>> [ 6025.595810] md: bind<dm-0>
>> [ 6025.596086] md: bind<dm-5>
>> [ 6025.596364] md: bind<dm-2>
>> [ 6025.596612] md: bind<dm-1>
>> [ 6025.596840] md: bind<dm-4>
>> [ 6025.597026] md: bind<dm-3>
>>
>>>
>>> Can you post the --examine results for all the RAID members? Both for
>>> the original partitions and for the overlay devices after you recreated
>>> the array. There may be differences in data offset, etc. which will
>>> break the filesystem.
>>
>> Original partitions:
>> http://pastebin.com/nHCxidvE
>>
>> overlay:
>> http://pastebin.com/eva4cnu6
>>
>
> Right - these show you have the wrong order. The original partition
> array device roles are:
> sdc1: 2
> sda1: 3
> sdf1: 4
> sdb1: 5
> sdc1: 6
> sdg1: 7
>
> and your overlays are:
> dm-1: 1
> dm-0: 2
> dm-5: 3
> dm-3: 4
> dm-2: 5
> dm-4: 7
>
> So the bad news is that you're missing roles 0 & 1, which will be
> mirrors. That means your array is broken unless any other member disks
> can be found
am i mistaken to think that the order of disks in an array can be known from the " Device Role : Active device Z " in mdadm --examine /dev/sdXN ?
>
> If you're certain that sdi1 and sdj1 should be in the array then you can
> try recreating the array (in the correct order) and using sdi1/sdj1 in
> the missing slots and see if one option works. I'll assume the overlay
> mapping is as follows (if not, remap as required):
> sda1 -> dm-0
> sdb1 -> dm-1
> sdc1 -> dm-2
> sde1 -> dm-3
> sdf1 -> dm-4
> sdg1 -> dm-5
> sdi1 -> dm-6
> sdj1 -> dm-7
>
> For each of the following orders, you're going to need to:
> - stop the existing array (mdadm -S /dev/md089)
> - create a new array using --assume-clean
> - check for an valid filesystem (fsck -n /dev/md089)
>
> If the fsck returns without errors then try mounting the filesystem and
> see if all looks okay, otherwise move on to the next order.
>
> The orders to try are:
> - /dev/dm-6 missing /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
> - missing /dev/dm-6 /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
> - /dev/dm-7 missing /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
> - missing /dev/dm-7 /dev/dm-3 /dev/dm-0 /dev/dm-4 /dev/dm-1 /dev/dm-2 /dev/dm-5
Thank you for all the help. appreciate it
>
> Good luck,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: can i recover an all spare raid10 array ?
From: Robin Hill @ 2014-10-28 20:46 UTC (permalink / raw)
To: Roland RoLaNd; +Cc: Robin Hill, linux-raid@vger.kernel.org
In-Reply-To: <DUB121-W16FDFC3155E2E6C24C71D88A9F0@phx.gbl>
[-- Attachment #1: Type: text/plain, Size: 12709 bytes --]
On Tue Oct 28, 2014 at 10:17:44PM +0200, Roland RoLaNd wrote:
>
>
> ----------------------------------------
> > Date: Tue, 28 Oct 2014 20:02:39 +0000
> > From: robin@robinhill.me.uk
> > To: r_o_l_a_n_d@hotmail.com
> > CC: linux-raid@vger.kernel.org
> > Subject: Re: can i recover an all spare raid10 array ?
> >
> > On Tue Oct 28, 2014 at 09:11:21PM +0200, Roland RoLaNd wrote:
> >
> >>
> >>
> >> ----------------------------------------
> >>> Date: Tue, 28 Oct 2014 18:34:22 +0000
> >>> From: robin@robinhill.me.uk
> >>> To: r_o_l_a_n_d@hotmail.com
> >>> CC: robin@robinhill.me.uk; linux-raid@vger.kernel.org
> >>> Subject: Re: can i recover an all spare raid10 array ?
> >>>
> >>> Please don't top post, it makes conversations very difficult to follow.
> >>> Responses should go at the bottom, or interleaved with the previous post
> >>> if responding to particular points. I've moved your previous responses
> >>> to keep the conversation flow straight.
> >>>
> >>> On Tue Oct 28, 2014 at 07:30:50PM +0200, Roland RoLaNd wrote:
> >>>>
> >>>>> From: r_o_l_a_n_d@hotmail.com
> >>>>> To: robin@robinhill.me.uk
> >>>>> CC: linux-raid@vger.kernel.org
> >>>>> Subject: Re: can i recover an all spare raid10 array ?
> >>>>> Date: Tue, 28 Oct 2014 19:29:25 +0200
> >>>>>
> >>>>>> Date: Tue, 28 Oct 2014 17:01:11 +0000
> >>>>>> From: robin@robinhill.me.uk
> >>>>>> To: r_o_l_a_n_d@hotmail.com
> >>>>>> CC: linux-raid@vger.kernel.org
> >>>>>> Subject: Re: can i recover an all spare raid10 array ?
> >>>>>>
> >>>>>> On Tue Oct 28, 2014 at 06:22:11PM +0200, Roland RoLaNd wrote:
> >>>>>>
> >>>>>>> I have two raid arrays on my system:
> >>>>>>> raid1: /dev/sdd1 /dev/sdh1
> >>>>>>> raid10: /dev/sde1 /dev/sda1 /dev/sdf1 /dec/sdb1 /dev/sdc1 /dev/sdg1
> >>>>>>>
> >>>>>>>
> >>>>>>> two disks had bad sectors: sdd and sdf <<-- they both got hot swapped.
> >>>>>>> i added sdf back to raid10 and recovery took place but adding sdd1 to
> >>>>>>> raid1 proved to be troublesome
> >>>>>>> as i didn't have anything important on '/' i formatted and installed
> >>>>>>> ubuntu 14 on raid1
> >>>>>>>
> >>>>>>> now system is up on raid 1, but raid10 (md127) is inactive
> >>>>>>>
> >>>>>>> cat /proc/mdstat
> >>>>>>>
> >>>>>>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> >>>>>>> md127 : inactive sde1[2](S) sdg1[8](S) sdc1[6](S) sdb1[5](S) sdf1[4](S) sda1[3](S)
> >>>>>>> 17580804096 blocks super 1.2
> >>>>>>>
> >>>>>>> md2 : active raid1 sdh4[0] sdd4[1]
> >>>>>>> 2921839424 blocks super 1.2 [2/2] [UU]
> >>>>>>> [==>..................] resync = 10.4% (304322368/2921839424) finish=672.5min speed=64861K/sec
> >>>>>>>
> >>>>>>> md1 : active raid1 sdh3[0] sdd3[1]
> >>>>>>> 7996352 blocks super 1.2 [2/2] [UU]
> >>>>>>>
> >>>>>>> md0 : active raid1 sdh2[0] sdd2[1]
> >>>>>>> 292544 blocks super 1.2 [2/2] [UU]
> >>>>>>>
> >>>>>>> unused devices: <none>
> >>>>>>> if i try to assemble md127
> >>>>>>>
> >>>>>>>
> >>>>>>> mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
> >>>>>>> mdadm: /dev/sde1 is busy - skipping
> >>>>>>> mdadm: /dev/sda1 is busy - skipping
> >>>>>>> mdadm: /dev/sdf1 is busy - skipping
> >>>>>>> mdadm: /dev/sdb1 is busy - skipping
> >>>>>>> mdadm: /dev/sdc1 is busy - skipping
> >>>>>>> mdadm: /dev/sdg1 is busy - skipping
> >>>>>>>
> >>>>>>>
> >>>>>>> if i try to add one of the disks: mdadm --add /dev/md127 /dev/sdj1
> >>>>>>> mdadm: cannot get array info for /dev/md127
> >>>>>>>
> >>>>>>> if i try:
> >>>>>>>
> >>>>>>> mdadm --stop /dev/md127
> >>>>>>> mdadm: stopped /dev/md127
> >>>>>>>
> >>>>>>> then running: mdadm --assemble /dev/md127 /dev/sde1 /dev/sda1 /dev/sdf1 /dev/sdb1 /dev/sdc1 /dev/sdg1
> >>>>>>>
> >>>>>>> returns:
> >>>>>>>
> >>>>>>> assembled from 5 drives and 1 rebuilding - not enough to start the array
> >>>>>>>
> >>>>>>> what does it mean ? is my data lost ?
> >>>>>>>
> >>>>>>> if i examine one of the md127 raid 10 array disks it shows this:
> >>>>>>>
> >>>>>>> mdadm --examine /dev/sde1
> >>>>>>> /dev/sde1:
> >>>>>>> Magic : a92b4efc
> >>>>>>> Version : 1.2
> >>>>>>> Feature Map : 0x0
> >>>>>>> Array UUID : ab90d4c8:41a55e14:635025cc:28f0ee76
> >>>>>>> Name : ubuntu:data (local to host ubuntu)
> >>>>>>> Creation Time : Sat May 10 21:54:56 2014
> >>>>>>> Raid Level : raid10
> >>>>>>> Raid Devices : 8
> >>>>>>>
> >>>>>>> Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
> >>>>>>> Array Size : 11720534016 (11177.57 GiB 12001.83 GB)
> >>>>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
> >>>>>>> Data Offset : 262144 sectors
> >>>>>>> Super Offset : 8 sectors
> >>>>>>> State : clean
> >>>>>>> Device UUID : a2a5db61:bd79f0ae:99d97f17:21c4a619
> >>>>>>>
> >>>>>>> Update Time : Tue Oct 28 10:07:18 2014
> >>>>>>> Checksum : 409deeb4 - correct
> >>>>>>> Events : 8655
> >>>>>>>
> >>>>>>> Layout : near=2
> >>>>>>> Chunk Size : 512K
> >>>>>>>
> >>>>>>> Device Role : Active device 2
> >>>>>>> Array State : AAAAAAAA ('A' == active, '.' == missing)
> >>>>>>>
> >>>>>>> Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB) <<--- does this mean i still have my data ?
> >>>>>>>
> >>>>>>>
> >>>>>>> the remaining two disks:
> >>>>>>>
> >>>>>>> mdadm --examine /dev/sdj1
> >>>>>>> mdadm: No md superblock detected on /dev/sdj1.
> >>>>>>> mdadm --examine /dev/sdi1
> >>>>>>> mdadm: No md superblock detected on /dev/sdi1.
> >>>>>>
> >>>>>> The --examine output indicates the RAID10 array was 8 members, not 6.
> >>>>>> As it stands, you are missing two array members (presumably a mirrored
> >>>>>> pair as mdadm won't start the array). Without these you're missing 512K
> >>>>>> of every 2M in the array, so your data is toast (well, with a lot of
> >>>>>> effort you may recover some files under 1.5M in size).
> >>>>>>
> >>>>>> Were you expecting sdi1 and sdj1 to have been part of the original
> >>>>>> RAID10 array? Have you removed the superblocks from them at any point?
> >>>>>> For completeness, what mdadm and kernel versions are you running?
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Robin
> >>>>>
> >>>>> Thanks for pitching in.here are the responses to you questions:
> >>>>>
> >>>>> - yes i expected both of them to be part of the array though one of
> >>>>> them was just added to the array and didnt finish recovering when
> >>>>> raid1 "/" crashed
> >>>>>
> >>> According to your --examine earlier, the RAID10 rebuild had completed
> >>> (it shows the array clean and having all disks active). Are you certain
> >>> that the new RAID1 array isn't using disks that used to be part of the
> >>> RAID10 array? Regardless, I'd expect the disks to have a superblock if
> >>> they were part of either array (unless they've been repartitioned?).
> >>>
> >>
> >> the examine earlier was to one of the 6 disks that belong to the current inactive array.. they're all clean
> >> as for raid1/10 arrays, that's what i thought as it happened with me before, but lsblk shows the following:
> >>
> >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> >> sda 8:0 0 2.7T 0 disk
> >> └─sda1 8:1 0 2.7T 0 part
> >> sdb 8:16 0 2.7T 0 disk
> >> └─sdb1 8:17 0 2.7T 0 part
> >> sdc 8:32 0 2.7T 0 disk
> >> └─sdc1 8:33 0 2.7T 0 part
> >> sdd 8:48 0 2.7T 0 disk
> >> ├─sdd1 8:49 0 1M 0 part
> >> ├─sdd2 8:50 0 286M 0 part
> >> │ └─md0 9:0 0 285.7M 0 raid1 /boot
> >> ├─sdd3 8:51 0 7.6G 0 part
> >> │ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
> >> └─sdd4 8:52 0 2.7T 0 part
> >> └─md2 9:2 0 2.7T 0 raid1 /
> >> sde 8:64 0 2.7T 0 disk
> >> └─sde1 8:65 0 2.7T 0 part
> >> sdf 8:80 0 2.7T 0 disk
> >> └─sdf1 8:81 0 2.7T 0 part
> >> sdg 8:96 0 2.7T 0 disk
> >> └─sdg1 8:97 0 2.7T 0 part
> >> sdh 8:112 0 2.7T 0 disk
> >> ├─sdh1 8:113 0 1M 0 part
> >> ├─sdh2 8:114 0 286M 0 part
> >> │ └─md0 9:0 0 285.7M 0 raid1 /boot
> >> ├─sdh3 8:115 0 7.6G 0 part
> >> │ └─md1 9:1 0 7.6G 0 raid1 [SWAP]
> >> └─sdh4 8:116 0 2.7T 0 part
> >> └─md2 9:2 0 2.7T 0 raid1 /
> >> sdi 8:128 0 2.7T 0 disk
> >> └─sdi1 8:129 0 2.7T 0 part
> >> sdj 8:144 0 2.7T 0 disk
> >> └─sdj1 8:145 0 2.7T 0 part
> >>
> >>
> >>>>> - i have not removed their superblocks or at least not in a way that i
> >>>>> amaware of
> >>>>>
> >>>>> - mdadm: 3.2.5-5ubuntu4.1
> >>>>> - uname -a: 3.13.0-24-generic
> >>>>>
> >>> That's a pretty old mdadm version, but I don't see anything in the
> >>> change logs that looks relevant. Others may be more familiar with issues
> >>> though.
> >>
> >> that's the latest in my current ubuntu repository
> >>
> >>>
> >>>>>
> >>>>> PS:
> >>>>> I just followed this recovery page:
> >>>>> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> >>>>> I managed to reach the last step, whenever i tried to mount it kept
> >>>>> asking me for the right file system
> >>>>>
> >>> That's good documentation anyway. As long as you stick to the overlay
> >>> devices your original data is untouched. It's amazing how many people
> >>> run --create on their original disks and lose any chance of getting the
> >>> data back.
> >>
> >> unfortunately i used to be/am one of those people.
> >> had bad experiences with this before, so i took it slow and went with
> >> the overlay documentation.
> >> all ebooks i could found about raid speak about different between
> >> multiple raid levels but none are thorough when it comes to setting
> >> up/troubleshooting raid.
> >> and once i do fix my issue, i move on to the next firefighting
> >> situation so i lose interest due to lack of time.
> >>
> >>>
> >>>> Correction:i couldn't force assemble the read devices so i issued instead:
> >>>> mdadm --create /dev/md089 --assume-clean --level=10 --verbose --raid-devices=8 missing /dev/dm-1 /dev/dm-0 /dev/dm-5 /dev/dm-3 /dev/dm-2 missing /dev/dm-4
> >>>> which got it into degraded state
> >>>>
> >>>
> >>> What error did you get when you tried to force assemble (both from mdadm
> >>> and anything reported via dmesg)? The device order you're using would
> >>> suggest that the missing disks wouldn't be mirrors of each other, so the
> >>> data should be okay.
> >>
> >> mdadm --assemble --force /dev/md100 $OVERLAYS
> >> mdadm: /dev/md100 assembled from 5 drives and 1 rebuilding - not enough to start the array.
> >>
> > That's very odd - all the --examine results for the original disks show
> > the array as clean. That would suggest an issue with the installed
> > version of mdadm but it doesn't really matter in this case - see below.
>
> when i got ubuntu 14 installed, i issued apt-get update && apt-get upgrade -y
> would that have affected anything ?
>
No - it's just that there must be some bug in that version of mdadm.
> >
> >> dmesg:
> >> [ 6025.573964] md: md100 stopped.
> >> [ 6025.595810] md: bind<dm-0>
> >> [ 6025.596086] md: bind<dm-5>
> >> [ 6025.596364] md: bind<dm-2>
> >> [ 6025.596612] md: bind<dm-1>
> >> [ 6025.596840] md: bind<dm-4>
> >> [ 6025.597026] md: bind<dm-3>
> >>
> >>>
> >>> Can you post the --examine results for all the RAID members? Both for
> >>> the original partitions and for the overlay devices after you recreated
> >>> the array. There may be differences in data offset, etc. which will
> >>> break the filesystem.
> >>
> >> Original partitions:
> >> http://pastebin.com/nHCxidvE
> >>
> >> overlay:
> >> http://pastebin.com/eva4cnu6
> >>
> >
> > Right - these show you have the wrong order. The original partition
> > array device roles are:
> > sdc1: 2
> > sda1: 3
> > sdf1: 4
> > sdb1: 5
> > sdc1: 6
> > sdg1: 7
> >
> > and your overlays are:
> > dm-1: 1
> > dm-0: 2
> > dm-5: 3
> > dm-3: 4
> > dm-2: 5
> > dm-4: 7
> >
> > So the bad news is that you're missing roles 0 & 1, which will be
> > mirrors. That means your array is broken unless any other member disks
> > can be found
>
> am i mistaken to think that the order of disks in an array can be known from the " Device Role : Active device Z " in mdadm --examine /dev/sdXN ?
>
That's correct, for version 1.x metadata anyway (I think 0.9 reports
things differently). This may be different from the number after the
device in /proc/mdstat though (which indicates the order the device was
added, so keeps increasing as the disks are replaced).
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: Very small bug on assemble --force
From: NeilBrown @ 2014-10-28 20:50 UTC (permalink / raw)
To: Ethan Wilson; +Cc: linux-raid
In-Reply-To: <544FA7B6.7030100@shiftmail.org>
[-- Attachment #1: Type: text/plain, Size: 2515 bytes --]
On Tue, 28 Oct 2014 15:27:02 +0100 Ethan Wilson <ethan.wilson@shiftmail.org>
wrote:
> Very small bug report:
Thanks for the bug report.
However, please always quote version of mdadm and version of Linux.
I think this might have been fixed already, in mdadm 3.3.1.
NeilBrown
> mdadm assemble --force fails to assemble at the very last step after
> having adjusted the event count:
>
>
> root@storage1:/root# mdadm --assemble /dev/md106 --config
> /etc/mdadm/mdadm.secondary.conf --force
> mdadm: forcing event count in
> /dev/mapper/mpathid-350014ee0ac5a112e-part1(3) from 1058 upto 1061
> mdadm: forcing event count in
> /dev/mapper/mpathid-350014ee056feaaf4-part1(4) from 1058 upto 1061
> mdadm: forcing event count in
> /dev/mapper/mpathid-350014ee001a93ede-part1(6) from 1058 upto 1061
> mdadm: clearing FAULTY flag for device 1 in /dev/md106 for
> /dev/mapper/mpathid-350014ee0ac5a112e-part1
> mdadm: clearing FAULTY flag for device 5 in /dev/md106 for
> /dev/mapper/mpathid-350014ee056feaaf4-part1
> mdadm: clearing FAULTY flag for device 8 in /dev/md106 for
> /dev/mapper/mpathid-350014ee001a93ede-part1
> mdadm: Marking array /dev/md106 as 'clean'
> mdadm: /dev/md106 assembled from 8 drives - not enough to start the array.
>
> mdadm is wrong here: 8 devices is indeed the total number of devices for
> this array.
> the last command started it inactive like this:
>
> md106 : inactive dm-101[0](S) dm-24[9](S) dm-107[6](S) dm-103[5](S)
> dm-123[4](S) dm-95[3](S) dm-105[2](S) dm-106[1](S)
> 4193255424 blocks super 1.2
> (maybe the blocks count is also wrong? Not adjusted for parity maybe?
> 4193255424 / 8.0 * 6.0 == 3144941568.0 almost correct)
>
> it can be fixed simply by doing:
>
> root@storage1:/root# mdadm --stop /dev/md106
> mdadm: stopped /dev/md106
>
> root@storage1:/root# mdadm --assemble /dev/md106 --config
> /etc/mdadm/secondary.conf --no-degraded
> mdadm: /dev/md106 has been started with 8 drives.
>
> now it works:
>
> md106 : active raid6 dm-101[0] dm-24[9] dm-107[6] dm-103[5] dm-123[4]
> dm-95[3] dm-105[2] dm-106[1]
> 3144938496 blocks super 1.2 level 6, 512k chunk, algorithm 2
> [8/8] [UUUUUUUU]
> bitmap: 0/1 pages [0KB], 1048576KB chunk
>
> Regards
> EW
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: mdadm creates corrupt superblock
From: NeilBrown @ 2014-10-28 21:00 UTC (permalink / raw)
To: Hans Kraus; +Cc: Linux-RAID
In-Reply-To: <544FDB1C.5030306@hanswkraus.com>
[-- Attachment #1: Type: text/plain, Size: 4245 bytes --]
On Tue, 28 Oct 2014 19:06:20 +0100 Hans Kraus <hans@hanswkraus.com> wrote:
> Hi,
>
> I debugged my problem a bit more: it seems that mdadm creates a defect
> superblock. That's repeatedly happening, but only for one drive. I
> copied zeros to that drive (via ddrescue /dev/zero ...) and the drive
> looks OK. The info I got:
>
and you send me the metadata of all your devices please?
mkdir /tmp/dump
mdadm --dump /tmp/dump /dev/sde1
tar czvf /tmp/dump.tgz /tmp/dump
and then send /tmp/dump.tgz
Thanks.
NeilBrown
=================================================================================================================================================================================================
> root@nashorn:/home/kraush/work/smartctrl# mdadm --zero-superblock /dev/sde1
> mdadm: Unrecognised md component device - /dev/sde1
> root@nashorn:/home/kraush/work/smartctrl# mdadm --add /dev/md126 /dev/sde1
> mdadm: add new device failed for /dev/sde1 as 4: Invalid argument
> root@nashorn:/home/kraush/work/smartctrl# mdadm -E /dev/sde1
> /dev/sde1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
> Name : nashorn:126 (local to host nashorn)
> Creation Time : Mon Oct 27 15:58:38 2014
> Raid Level : raid1
> Raid Devices : 2
>
> Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
> Array Size : 0
> Used Dev Size : 0
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=1953260976 sectors
> State : clean
> Device UUID : 4d1420c6:79978477:e86412b1:24beebfc
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Oct 28 11:53:49 2014
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 6f02d040 - expected 891eedff
> Events : 0
>
>
> Device Role : spare
> Array State : RR ('A' == active, '.' == missing, 'R' == replacing)
> root@nashorn:/home/kraush/work/smartctrl#
> -----------------------------------------------------------------------------------------------------------
> root@nashorn:/home/kraush/work/smartctrl# cat /proc/mdstat
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md127 : active (auto-read-only) raid6 sda1[0] sdk1[6](S) sdl2[7](S)
> sdh1[5] sdg1[4] sdf1[3] sdc1[2] sdb1[1]
> 1953017856 blocks super 1.2 level 6, 512k chunk, algorithm 2
> [6/6] [UUUUUU]
> bitmap: 0/4 pages [0KB], 65536KB chunk
>
> md10 : active (auto-read-only) raid1 sdi2[0] sdl3[1]
> 87833408 blocks super 1.2 [2/2] [UU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdd1[3] sdj1[2]
> 488254464 blocks super 1.2 [2/2] [UU]
> bitmap: 0/4 pages [0KB], 65536KB chunk
>
> md0 : active raid1 sdl1[4] sdd2[3] sdi1[5]
> 156157824 blocks super 1.2 [3/1] [U__]
> [===================>.] recovery = 95.6% (149290048/156157824)
> finish=8.4min speed=13496K/sec
> bitmap: 1/2 pages [4KB], 65536KB chunk
>
> unused devices: <none>
> root@nashorn:/home/kraush/work/smartctrl#
> -----------------------------------------------------------------------------------------------------------
> dmesg:
> [33299.387382] md: invalid superblock checksum on sde1
> [33299.387385] md: sde1 does not have a valid v1.2 superblock, not
> importing!
> [33299.387408] md: md_import_device returned -22
> root@nashorn:/home/kraush/work/smartctrl#
> -----------------------------------------------------------------------------------------------------------
> root@nashorn:/home/kraush/work/mdadm# mdadm --version
> mdadm - v3.3-161-gfed12d4 - 21st August 2014
> =================================================================================================================================================================================================
>
> Kind regards, Hans
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: RAID1 removing failed disk returns EBUSY
From: NeilBrown @ 2014-10-28 21:41 UTC (permalink / raw)
To: Joe Lawrence; +Cc: linux-raid, Bill Kuzeja
In-Reply-To: <20141027162748.593451be@jlaw-desktop.mno.stratus.com>
[-- Attachment #1: Type: text/plain, Size: 6472 bytes --]
On Mon, 27 Oct 2014 16:27:48 -0400 Joe Lawrence <joe.lawrence@stratus.com>
wrote:
> Hi Neil,
>
> We've encountered changes in MD and mdadm that have broken our automated
> disk removal script. In the past, we've been able to run the following
> after a RAID1 disk component removal:
>
> % echo fail > /sys/block/md3/md/dev-sdr5/state
> % echo remove > /sys/block/md3/md/dev-sdr5/state
>
> However, the latest RHEL6.6 code drop has rebased to sufficiently recent
> MD kernel and mdadm changes, in which the previous commands occasionally
> fail like so:
>
> * MD array is usually resyncing or checking
> * Component disk /dev/sdr removed via HBA sysfs PCI removal
> * Following UDEV rule fires:
>
> SUBSYSTEM=="block", ACTION=="remove", ENV{ID_PATH}=="?*", \
> RUN+="/sbin/mdadm -If $name --path $env{ID_PATH}"
>
> % mdadm --detail /dev/md3
> /dev/md3:
> Version : 1.1
> Creation Time : Tue Oct 14 17:31:59 2014
> Raid Level : raid1
> Array Size : 25149440 (23.98 GiB 25.75 GB)
> Used Dev Size : 25149440 (23.98 GiB 25.75 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Wed Oct 15 14:22:34 2014
> State : active, degraded
> Active Devices : 1
> Working Devices : 1
> Failed Devices : 1
> Spare Devices : 0
>
> Name : localhost.localdomain:3
> UUID : 40ed68ee:ba41d4cd:28c361ed:be7470b8
> Events : 142
>
> Number Major Minor RaidDevice State
> 0 65 21 0 faulty
> 1 65 5 1 active sync /dev/sdj5
>
> All attempts to remove this device fail:
>
> % echo remove > /sys/block/md3/md/dev-sdr5/state
> -bash: echo: write error: Device or resource busy
>
> This can be traced to state_store():
>
> } else if (cmd_match(buf, "remove")) {
> if (rdev->raid_disk >= 0)
> err = -EBUSY;
>
> After much debugging and systemtapping, I think I've figured out that the
> sysfs scripting may fail after the following combination of changes:
>
> mdadm 8af530b07fce "Enhance incremental removal."
> kernel 30b8feb730f9 "md/raid5: avoid deadlock when raid5 array has unack
> badblocks during md_stop_writes"
>
> With these two changes:
>
> 1 - On the user side, mdadm is trying to set the array_state to read-auto
> on incremental removal (as invoked by UDEV rule).
>
> 2 - Kernel side, md_set_readonly() will set the MD_RECOVERY_FROZEN flag,
> wake up the mddev->thread and if there is a sync_thread, it will set
> MD_RECOVERY_INTR and then wait until the sync_thread is set to NULL.
>
> When md_check_recovery() gets a chance to run as part of the
> raid1d() mddev->thread, it may or may not ever get to
> an invocation of remove_and_add_spares(), for there are but *many*
> conditional early exits along the way -- for example, if
> MD_RECOVERY_FROZEN is set, the following condition will bounce out of
> the routine:
>
> if (!test_and_clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
> test_bit(MD_RECOVERY_FROZEN, &mddev->recovery))
> goto unlock;
>
> the next time around, MD_RECOVERY_NEEDED will have been cleared, so
> all future tests will return 0 and the negation will always take the
> early exit path.
>
> Back in md_set_readonly(), it may notice that the MD is still in use,
> so it clears the MD_RECOVERY_FROZEN and then returns -EBUSY, without
> setting mddev->ro. But the damage has been done as conditions have
> been set such that md_check_recovery() will never call
> remove_and_add_spares().
>
> This would also explain why an "idle" sync_action clears the wedge: it
> sets MD_RECOVERY_NEEDED allowing md_check_recovery() to continue executing
> to remove_and_add_spares().
>
> As far as I can tell, this is what is happening to prevent the "remove"
> write to /sys/block/md3/md/dev-sdr5/state from succeeding. There are
> certainly a lot of little bit-states between disk removal, UDEV mdadm, and
> various MD kernel threads, so apologies if I missed an important
> transition.
>
> Would you consider writing "idle" to the MD array sync_action file as a
> safe and reasonable intermediate workaround step for our script?
>
> And of course, any suggestions to whether this is intended behavior (ie,
> the removed component disk is failed, but stuck in the array)?
>
> This is fairly easy for us to reproduce with multiple MD arrays per disk
> (one per partition) and interrupting a raid check on all of them
> (especially when they are delayed waiting for the first to finish) by
> removing the component disk via sysfs PCI removal. We can provide
> additional debug or testing if required.
>
Hi Joe,
thanks for the details analysis!!
I think the correct fix would be that MD_RECOVERY_NEEDED should be set after
clearing MD_RECOVERY_FROZEN, like the patch below.
Can you confirm that it works for you?
Writing 'idle' should in general be safe, so that could be used as an interim.
Thanks,
NeilBrown
diff --git a/drivers/md/md.c b/drivers/md/md.c
index c03d87b6890a..2c73fcb82593 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5261,6 +5261,7 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
printk("md: %s still in use.\n",mdname(mddev));
if (did_freeze) {
clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
md_wakeup_thread(mddev->thread);
}
err = -EBUSY;
@@ -5275,6 +5276,8 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
mddev->ro = 1;
set_disk_ro(mddev->gendisk, 1);
clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+ md_wakeup_thread(mddev->thread);
sysfs_notify_dirent_safe(mddev->sysfs_state);
err = 0;
}
@@ -5318,6 +5321,7 @@ static int do_md_stop(struct mddev *mddev, int mode,
mutex_unlock(&mddev->open_mutex);
if (did_freeze) {
clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
md_wakeup_thread(mddev->thread);
}
return -EBUSY;
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: Raid1 element stuck in (S) state
From: NeilBrown @ 2014-10-28 21:42 UTC (permalink / raw)
To: micah anderson; +Cc: linux-raid
In-Reply-To: <87k33lwq7s.fsf@muck.riseup.net>
[-- Attachment #1: Type: text/plain, Size: 660 bytes --]
On Mon, 27 Oct 2014 10:18:47 -0400 micah anderson <micah@debian.org> wrote:
>
> Hi,
>
> i've got a raid1 setup, where one drive died, it was replaced with a new
> one, but its stuck in a (S) state and I can't seem to get it added into
> the array, /proc/mdstat looks like this:
>
> md3 : active raid1 sdc1[2](S) sdd1[1]
> 976759672 blocks super 1.2 [2/1] [_U]
>
> where sdc1 is the replaced drive.
>
> What is the right way to get this added back?
>
I've a feeling this bug might have been fixed.
What versions of mdadm and Linux are you using?
Are there any errors in the kernel logs when you --add the device?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: [PATCH] Grow: fix resize of array component size to > 32bits
From: NeilBrown @ 2014-10-28 22:19 UTC (permalink / raw)
To: Justin Maggard; +Cc: linux-raid
In-Reply-To: <1414198502-10167-1-git-send-email-jmaggard10@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]
On Fri, 24 Oct 2014 17:55:02 -0700 Justin Maggard <jmaggard10@gmail.com>
wrote:
> If the request --size to --grow an array to is larger
> than 32bits, then mdadm may make the wrong choice and
> use ioctl instead of setting component_size via sysfs
> and the change is ignored.
Can you explain exactly why the current code is not sufficient? When does
it fail?
If you include the explanation in a re-submission of the patch, and I am
convinced, then I will gladly apply your patch.
Thanks,
NeilBrown
>
> Instead of using casts to check for a 32-bit overflow,
> just check for set bits outside of INT32_MAX.
> ---
> Grow.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Grow.c b/Grow.c
> index a9c8589..a614102 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1818,7 +1818,7 @@ int Grow_reshape(char *devname, int fd,
> if (s->size == MAX_SIZE)
> s->size = 0;
> array.size = s->size;
> - if (array.size != (signed)s->size) {
> + if (s->size & ~INT32_MAX) {
> /* got truncated to 32bit, write to
> * component_size instead
> */
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Help with raid1 to raid5 reshape after crash
From: NeilBrown @ 2014-10-28 22:24 UTC (permalink / raw)
To: shane-kernel; +Cc: linux-raid
In-Reply-To: <001001cfefe3$a82538a0$f86fa9e0$@csy.ca>
[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]
On Fri, 24 Oct 2014 16:38:40 -0700 <shane-kernel@csy.ca> wrote:
> Hello all,
>
> I was converting from a 2 disk raid1 array to a 4 disk raid5 (3+1s) in preparation to convert to a four disk raid6. I added the two additional component devices and did a:
> Mdadm --grow /dev/md1 --level 5 --raid-devices 3
I guess you mean "to a 3 disk raid5 (2+1s)" ...
> And off it went. I had a power failure a good ways in and now when the system boots, it runs the array, attempts to restart the reshape then disk IO locks up hard.
>
> I figure it is disk IO rather than a complete system lockup as I have about 10 seconds of interactive shell before it locks. If I enter a dmesg command then that command will succeed. After the lockup, subsequent dmesg commands succeed (dmesg is in cache) but another command such as ls will lock up. Cat /proc/mdstat shows the reshape in progress but it doesn't actually progress beyond the point at which the system locks.
>
> Booting from a USB stick, I tried:
> Mdadm --assemble --readonly
> But that too hangs the system.
>
> Do I have any recourse beyond just recreating the array and restoring from backup? Kernel is 3.16.
Probably it can be fixed...
Can you include the output of "mdadm -D" instead of just mentioning the
command :-)
Also "mdadm --examine" of each device.
And "mdadm --version".
And
cat /proc/PID/stack
for and PID which is in "D" state in 'ps'.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.
From: NeilBrown @ 2014-10-28 22:38 UTC (permalink / raw)
To: Jason Keltz; +Cc: linux-raid
In-Reply-To: <5445799A.8020205@cse.yorku.ca>
[-- Attachment #1: Type: text/plain, Size: 2825 bytes --]
On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz <jas@cse.yorku.ca> wrote:
> On 10/20/2014 12:19 PM, Jason Keltz wrote:
> > Hi.
> >
> > I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system.
> > I've experimented with setting "speed_limit_min" and "speed_limit_max"
> > kernel variables so that I get the best balance of performance during
> > a RAID rebuild of one of the RAID1 pairs. If, for example, I set
> > speed_limit_min AND speed_limit_max to 80000 then fail a disk when
> > there is no other disk activity, then I do get a rebuild rate of
> > around 80 MB/s. However, if I then start up a write intensive
> > operation on the MD array (eg. a dd, or a mkfs on an LVM logical
> > volume that is created on that MD), then, my write operation seems to
> > get "full power", and my rebuild drops to around 25 MB/s. This means
> > that the rebuild of my RAID10 disk is going to take a huge amount of
> > time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to
> > the same value, am I not guaranteeing the rebuild speed? Is this a bug
> > that I should be reporting to Red Hat, or a "feature"?
> >
> > Thanks in advance for any help that you can provide...
> >
> > Jason.
>
> I would like to add that I downloaded the latest version of Ubuntu, and
> am running it on the same server with the same MD.
> When I set speed_limit_min and speed_limit_max to 80000, I was able to
> start two large dds on the md array, and the rebuild stuck at around 71
> MB/s, which is close enough. This leads me to believe that the problem
> above is probably a RHEL6 issue. However, after I stopped the two dd
> operations, and raised both speed_limit_min and speed_limit_max to
> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes
> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher
> (even though I had 120 MB/s and above on the RHEL system without any
> load)... Hmm.
>
md certainly cannot "guarantee" any speed - it can only deliver what the
underlying devices deliver.
I know the kernels logs say something about a "guarantee". That was added
before my time and I haven't had occasion to remove it.
md will normally just try to recover as fast as it can unless that exceeds
one of the limits - then it will back-off.
What speed it actually achieved depends on other load and the behaviour of
the IO scheduler.
"RHEL6" and "Ubuntu" don't mean a lot to me. Specific kernel version might,
though in the case of Redhat I know that backport lots of stuff so even the
kernel version isn't very helpful. I'm must prefer having report against
mainline kernels.
Rotating drives do get lower transfer speeds at higher addresses. That might
explain the 120 / 100 difference.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Very small bug on assemble --force
From: Ethan Wilson @ 2014-10-28 22:40 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20141029075050.605a11a9@notabene.brown>
On 28/10/2014 21:50, NeilBrown wrote:
> On Tue, 28 Oct 2014 15:27:02 +0100 Ethan Wilson <ethan.wilson@shiftmail.org>
> wrote:
>
>> Very small bug report:
> Thanks for the bug report.
> However, please always quote version of mdadm and version of Linux.
>
> I think this might have been fixed already, in mdadm 3.3.1.
>
> NeilBrown
>
Whops
kernel 3.14.22
mdadm - v3.2.5 - 18th May 2012
mdadm is old indeed so you are probably right.
It is difficult for me to test on the new version because I don't have
degraded arrays anymore right now :-/
Sorry
EW
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox