* Why wipe crypto keys during suspend (was Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation)
From: Pavel Machek @ 2015-04-06 21:13 UTC (permalink / raw)
To: Mike Snitzer
Cc: Pali Rohár, Alasdair Kergon, Neil Brown, Rafael J. Wysocki,
Len Brown, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150406205145.GA19677@redhat.com>
On Mon 2015-04-06 16:51:45, Mike Snitzer wrote:
> On Mon, Apr 06 2015 at 9:25am -0400,
> Pavel Machek <pavel@ucw.cz> wrote:
>
> > On Mon 2015-04-06 09:00:46, Mike Snitzer wrote:
> > > On Sun, Apr 05 2015 at 1:20pm -0400,
> > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > >
> > > > This patch series increase security of suspend and hibernate actions. It allows
> > > > user to safely wipe crypto keys before suspend and hibernate actions starts
> > > > without race conditions on userspace process with heavy I/O.
> > > >
> > > > To automatically wipe cryto key for <device> before hibernate action call:
> > > > $ dmsetup message <device> 0 key wipe_on_hibernation 1
> > > >
> > > > To automatically wipe cryto key for <device> before suspend action call:
> > > > $ dmsetup message <device> 0 key wipe_on_suspend 1
> > > >
> > > > (Value 0 after wipe_* string reverts original behaviour - to not wipe key)
> > >
> > > Can you elaborate on the attack vector your changes are meant to protect
> > > against? The user already authorized access, why is it inherently
> > > dangerous to _not_ wipe the associated key across these events?
> >
> > Umm. You are using your notebook. It is unlikely to be stolen at that
> > point. You close the lid and board the airplane, stowing it in
> > overhead bin. There's much better chance of notebook being stolen now.
>
> Yes, pretty straight forward but the thief would need to then login upon
> resume (at least with most common desktop configs)... the barrier then
> is only the strength of the user's password and not the crypt
> passphrase.
Why would he want to do that? :-).
No; at that point, attacker would either wait for something remotely
exploitable to exploit, or attach JTAG debugger to the machine, or use
liquid nitrogen on RAMs and then attach them to running machine.
Yes, it is better when keys are not on your machine when it is stolen.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pali Rohár @ 2015-04-06 21:27 UTC (permalink / raw)
To: Pavel Machek
Cc: Mike Snitzer, Alasdair Kergon, Neil Brown, Rafael J. Wysocki,
Len Brown, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150406181738.GC9978@amd>
[-- Attachment #1: Type: Text/Plain, Size: 3337 bytes --]
On Monday 06 April 2015 20:17:38 Pavel Machek wrote:
> On Mon 2015-04-06 15:29:57, Pali Rohár wrote:
> > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> > > On Sun, Apr 05 2015 at 1:20pm -0400,
> > >
> > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > > This patch series increase security of suspend and
> > > > hibernate actions. It allows user to safely wipe crypto
> > > > keys before suspend and hibernate actions starts
> > > > without race conditions on userspace process with heavy
> > > > I/O.
> > > >
> > > > To automatically wipe cryto key for <device> before
> > > > hibernate action call: $ dmsetup message <device> 0 key
> > > > wipe_on_hibernation 1
> > > >
> > > > To automatically wipe cryto key for <device> before
> > > > suspend action call: $ dmsetup message <device> 0 key
> > > > wipe_on_suspend 1
> > > >
> > > > (Value 0 after wipe_* string reverts original behaviour
> > > > - to not wipe key)
> > >
> > > Can you elaborate on the attack vector your changes are
> > > meant to protect against? The user already authorized
> > > access, why is it inherently dangerous to _not_ wipe the
> > > associated key across these events?
> >
> > Hi,
> >
> > yes, I will try to explain current problems with cryptsetup
> > luksSuspend command and hibernation.
> >
> > First, sometimes it is needed to put machine into other
> > hands. You can still watch other person what is doing with
> > machine, but once if you let machine unlocked (e.g opened
> > luks disk), she/he can access encrypted data.
> >
> > If you turn off machine, it could be safe, because luks disk
> > devices are locked. But if you enter machine into suspend or
> > hibernate state luks devices are still open. And my patches
> > try to achieve similar security as when machine is off (=
> > no crypto keys in RAM or on swap).
> >
> > When doing hibernate on unencrypted swap it is to prevent
> > leaking crypto keys to hibernate image (which is stored in
> > swap).
> >
> > When doing suspend action it is again to prevent leaking
> > crypto keys. E.g when you suspend laptop and put it off
> > (somebody can remove RAMs and do some cold boot attack).
> >
> > The most common situation is:
> > You have mounted partition from dm-crypt device (e.g.
> > /home/), some userspace processes access it (e.g opened
> > firefox which still reads/writes to cache ~/.firefox/) and
> > you want to drop crypto keys from kernel for some time.
> >
> > For that operation there is command cryptsetup luksSuspend,
> > which suspend dm device and then tell kernel to wipe crypto
> > keys. All I/O operations are then stopped and userspace
> > processes which want to do some those I/O operations are
> > stopped too (until you call cryptsetup luksResume and enter
> > correct key).
>
> Actually... is the list of sites where the process wait small
> enough? Could we modify them to be freezeable? Suspend should
> work even if user stopped the his crypto partitions...
>
> Pavel
If you suspend dm device and then you want to read file from fs
which is on that device, then process freeze and you even cannot
kill it with SIGKILL. Before entering suspend kernel tries to do
sync and that operation also fails...
--
Pali Rohár
pali.rohar@gmail.com
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Recovery of RAID 6 after three rapid disk events (disks are still working)
From: Mike @ 2015-04-06 23:28 UTC (permalink / raw)
To: linux-raid
Hi guys,
mdadm 3.2.5, 14 disk RAID 6
Had a drive failure Friday. Two more today in rapid succession. All
drives seem fully operational.
This is roughly 40TB array, I could use your help please to rescue the
array.
All members are on partition 2 of each physical disk.
# mdadm --assemble --scan
mdadm: /dev/md/media:0 assembled from 11 drives - not enough to start
the array.
mdadm: No arrays found in config file or automatically
# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/loop0
mdadm: Cannot assemble mbr metadata on /dev/sdo1
mdadm: Cannot assemble mbr metadata on /dev/sdo
mdadm: no RAID superblock on /dev/sdm1
mdadm: no RAID superblock on /dev/sdm
mdadm: no RAID superblock on /dev/sdn1
mdadm: no RAID superblock on /dev/sdn
mdadm: no RAID superblock on /dev/sdj1
mdadm: no RAID superblock on /dev/sdj
mdadm: no RAID superblock on /dev/sdk1
mdadm: no RAID superblock on /dev/sdk
mdadm: no RAID superblock on /dev/sdl1
mdadm: no RAID superblock on /dev/sdl
mdadm: no RAID superblock on /dev/sdi1
mdadm: no RAID superblock on /dev/sdi
mdadm: no RAID superblock on /dev/sdh1
mdadm: no RAID superblock on /dev/sdh
mdadm: no RAID superblock on /dev/sdb1
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: no RAID superblock on /dev/sdf1
mdadm: no RAID superblock on /dev/sdf
mdadm: no RAID superblock on /dev/sdd1
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdg1
mdadm: no RAID superblock on /dev/sdg
mdadm: no RAID superblock on /dev/sde1
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sdm2 is identified as a member of /dev/md/media:0, slot 3.
mdadm: /dev/sdn2 is identified as a member of /dev/md/media:0, slot 8.
mdadm: /dev/sdj2 is identified as a member of /dev/md/media:0, slot 5.
mdadm: /dev/sdk2 is identified as a member of /dev/md/media:0, slot 11.
mdadm: /dev/sdl2 is identified as a member of /dev/md/media:0, slot 4.
mdadm: /dev/sdi2 is identified as a member of /dev/md/media:0, slot 7.
mdadm: /dev/sdh2 is identified as a member of /dev/md/media:0, slot 1.
mdadm: /dev/sdb2 is identified as a member of /dev/md/media:0, slot 6.
mdadm: /dev/sda2 is identified as a member of /dev/md/media:0, slot 13.
mdadm: /dev/sdf2 is identified as a member of /dev/md/media:0, slot 2.
mdadm: /dev/sdd2 is identified as a member of /dev/md/media:0, slot 9.
mdadm: /dev/sdc2 is identified as a member of /dev/md/media:0, slot 6.
mdadm: /dev/sdg2 is identified as a member of /dev/md/media:0, slot 12.
mdadm: /dev/sde2 is identified as a member of /dev/md/media:0, slot 0.
mdadm: added /dev/sdh2 to /dev/md/media:0 as 1
mdadm: added /dev/sdf2 to /dev/md/media:0 as 2
mdadm: added /dev/sdm2 to /dev/md/media:0 as 3
mdadm: added /dev/sdl2 to /dev/md/media:0 as 4
mdadm: added /dev/sdj2 to /dev/md/media:0 as 5
mdadm: added /dev/sdc2 to /dev/md/media:0 as 6 (possibly out of date)
mdadm: added /dev/sdi2 to /dev/md/media:0 as 7
mdadm: added /dev/sdn2 to /dev/md/media:0 as 8
mdadm: added /dev/sdd2 to /dev/md/media:0 as 9 (possibly out of date)
mdadm: no uptodate device for slot 10 of /dev/md/media:0
mdadm: added /dev/sdk2 to /dev/md/media:0 as 11
mdadm: added /dev/sdg2 to /dev/md/media:0 as 12
mdadm: added /dev/sda2 to /dev/md/media:0 as 13
mdadm: added /dev/sde2 to /dev/md/media:0 as 0
mdadm: /dev/md/media:0 assembled from 11 drives - not enough to start
the array.
mdadm: looking for devices for further assembly
mdadm: No arrays found in config file or automatically
# mdadm --examine /dev/sd?2 --verbose
/dev/sda2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 822fd02b:454da94c:57f68535:964996b1
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 8e7a48ab - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 13
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdb2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x2
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Recovery Offset : 0 sectors
State : clean
Device UUID : 5c4a1046:c232402c:45f1c604:569f6896
Update Time : Mon Apr 6 16:12:05 2015
Checksum : 3c7663c2 - correct
Events : 2382358
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAAAA.AAA ('A' == active, '.' == missing)
/dev/sdc2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x2
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Recovery Offset : 0 sectors
State : clean
Device UUID : e211f88d:5c355522:fbeb7134:16b17684
Update Time : Mon Apr 6 16:28:09 2015
Checksum : 982c3e96 - correct
Events : 2382480
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAAAA.AAA ('A' == active, '.' == missing)
/dev/sdd2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : de3f41b8:3016870c:344f2a92:c08e1085
Update Time : Mon Apr 6 19:19:34 2015
Checksum : af2bfdb - correct
Events : 2384965
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAA.AAA.AAA ('A' == active, '.' == missing)
/dev/sde2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 071afb12:f78f4f15:f65aa629:8eadcfa7
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 28a53e70 - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdf2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e319bdaa:22bc1153:c43b4878:8a9c1832
Update Time : Mon Apr 6 19:19:53 2015
Checksum : d7143a42 - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdg2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f5477580:94357948:6e97fe82:c8805bcd
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 482d21b8 - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdh2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3dd1df1b:203c6453:0964ebad:245b1670
Update Time : Mon Apr 6 19:19:53 2015
Checksum : bc2a587a - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdi2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4a013330:37c565f9:cb761d35:7ce4ddb4
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 4278adbb - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdj2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : d5521994:6c4f04f9:f7ca0dd9:dff3c6cd
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 62d6ed06 - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdk2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c1514b9f:24616fed:324a50fb:9469043a
Update Time : Mon Apr 6 19:19:53 2015
Checksum : f0f3f2a5 - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdl2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 6c33c472:af1ffd8f:22d10ea3:9edc75bb
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 902a8ccd - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdm2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b750b4e4:2b1bac5f:cbd3bde5:eab657e7
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 405a828f - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
/dev/sdn2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1c40379:914e5d18:dddb893b:4dc5a28f
Name : media:0
Creation Time : Wed Nov 7 22:06:02 2012
Raid Level : raid6
Raid Devices : 14
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 35160446976 (33531.62 GiB 36004.30 GB)
Used Dev Size : 5860074496 (2794.30 GiB 3000.36 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4670b36c:07cbe661:20e3d314:f7c3fd42
Update Time : Mon Apr 6 19:19:53 2015
Checksum : 55506e59 - correct
Events : 2384973
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAA.AA..AAA ('A' == active, '.' == missing)
--
Mike
^ permalink raw reply
* Re: Recovery of RAID 6 after three rapid disk events (disks are still working)
From: NeilBrown @ 2015-04-06 23:33 UTC (permalink / raw)
To: Mike; +Cc: linux-raid
In-Reply-To: <104310d53affd63c6e13088daa8dd5f2@zeus.bigmiketrading.com>
[-- Attachment #1: Type: text/plain, Size: 912 bytes --]
On Mon, 06 Apr 2015 18:28:55 -0500 Mike <mboulter@gmail.com> wrote:
> Hi guys,
>
> mdadm 3.2.5, 14 disk RAID 6
>
> Had a drive failure Friday. Two more today in rapid succession. All
> drives seem fully operational.
>
> This is roughly 40TB array, I could use your help please to rescue the
> array.
>
> All members are on partition 2 of each physical disk.
>
> # mdadm --assemble --scan
Add "--force" to that line, and it will work.
In this context, "--force" means "yes, I know that some devices are a bit out
of date and that that if there was a write happening while one of the devices
appeared to fail then there could be some corruption of a few blocks in the
array, but I want all the rest of the data and I promise to be careful and I
won't assume that everything is perfect and I won't blame you if a few files
are lost - just do what you can. Please."
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Can't mount partitions after "mdadm --zero-superblock"
From: Eduard Rozenberg @ 2015-04-06 23:45 UTC (permalink / raw)
To: linux-raid
Hello folks,
I previously had the following setup:
sda & sdb partitioned w/ GPT, 7 partitions each (usr, opt, var etc...)
7 raid1’s with 2 devices for each pair of partitions (/dev/sda1 & /dev/sdab1, etc)
They’d been created under Slackware 13.37.
I was trying to clean out mdadm from those partitions but keep the data so I ran
"mdadm --zero-superblock” on each of those previously RAID1 mdadm 1.2 ext4
partitions.
As a result I am now currently unable to mount any partition after the first one on either
disk. The first partition does mount. The partition table is visible and looks fine in gdisk.
mount -t ext4 /dev/sdac2 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/sdac2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
I did try superblock recovery with each backup superblock that ext4 normally creates,
but none of the superblock locations worked.
For example:
fsck.ext4 -b 4096000 /dev/sdac2
e2fsck 1.42.8 (20-Jun-2013)
/sbin/e2fsck: Invalid argument while trying to open /dev/sdac2
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
Would be grateful for any advice on anything else I can try.
Regards,
—Ed--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-06 23:46 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AFDEijn-7gNmnWNNabzcuPRMabNwr8xfRFt0xacsvZfog@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 7552 bytes --]
On Fri, 3 Apr 2015 19:22:15 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> I have a raid0 array whose component devices are raid1 arrays. In an
> attempt to expand the pre-existing raid0 array, I created a new raid1
> device and then added it, growing the raid0 array.
> But then the system lost power shortly after the reshaping began.
>
> After rebooting the original two components are listed as spares in an
> inactive raid4 array and the new component is not listed in
> /proc/mdstat:
>
>
> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> [linear] [multipath]
> md124 : inactive md126[0](S) md127[1](S)
> 3907022200 blocks super 1.2
>
> md0 : active raid1 sda5[0] sdb2[1]
> 107652416 blocks [2/2] [UU]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md125 : active raid1 sdh1[0] sdg1[1]
> 2930134016 blocks super 1.2 [2/2] [UU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdc1[0] sdd1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sde1[2] sdf1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
>
>
> Looking at the details of the inactive array shows that it is in a
> reshape between raid0 and raid4:
>
>
> /dev/md124:
> Version : 1.2
> Raid Level : raid0
> Total Devices : 2
> Persistence : Superblock is persistent
>
> State : inactive
>
> Delta Devices : -1, (1->0)
> New Level : raid4
> New Chunksize : 512K
>
> Name : hordern:hordern1 (local to host hordern)
> UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Events : 14013
>
> Number Major Minor RaidDevice
>
> - 9 126 - /dev/md/beta
> - 9 127 - /dev/md/alpha
>
>
> And examining each component shows that they have a consistent view of
> where in that reshape they are (based on the reshape position), but
> not consistent in the size of the array:
>
> /dev/md/alpha:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 3
>
> Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Array Size : 3907021824 (3726.03 GiB 4000.79 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=752 sectors
> State : active
> Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : -1 (4->3)
>
> Update Time : Sun Mar 29 15:11:35 2015
> Checksum : 8be5e0e6 - correct
> Events : 14013
>
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/md/beta:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 3
>
> Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
> Array Size : 3907021824 (3726.03 GiB 4000.79 GB)
> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=752 sectors
> State : clean
> Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : -1 (4->3)
>
> Update Time : Sun Mar 29 15:11:35 2015
> Checksum : f7526add - correct
> Events : 14013
>
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/md/gamma:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x6
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 4
>
> Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Recovery Offset : 86403072 sectors
> Unused Space : before=1960 sectors, after=1953244160 sectors
> State : active
> Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : 1 (3->4)
>
> Update Time : Sun Mar 29 00:05:29 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 710dc078 - correct
> Events : 673
>
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
>
>
> When I stop the inactive array and try to assemble it from all three
> components, I get an error about the superblock on the third component
> not matching the other two components (which makes sense since the
> array sizes are different):
>
> hordern ~ # mdadm --verbose --verbose --assemble /dev/md/hordern1
> /dev/md/alpha /dev/md/beta /dev/md/gamma
> mdadm: looking for devices for /dev/md/hordern1
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: superblock on /dev/md/gamma doesn't match others - assembly aborted
>
>
> First, what could cause the initial two components to have a different
> superblock than the newly added component? And can I convince them to
> be the same?
Good question. The important difference is this line:
> Delta Devices : -1 (4->3)
> Delta Devices : 1 (3->4)
Also "Raid Devices" are different. This is what leads to the array size
being reported differently.
I don't know why this happened. I'll have to hunt through the code.
>
> Second, is there documentation anywhere about the internal process of
> growing a raid0 array? Why does it convert to a raid4 array? And what
> do the Delta Devices lines mean?
- no documentation like that
- the "raid0" module does support reshape. The "raid5" module does.
So the array is converted to a RAID4 with a missing parity device, then
reshaped, then converted back. It is just an internal detail really.
- "Delta devices" means that there is reshape happening that is changing
then number of devices - "delta" is a difference.
It is clearly confused.
>
> Third, is it possible to resume the reshape? If not, can it be reverted?
>
I'll have to get back to you on that. Hopefully you can be patient a little
longer.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Can't mount partitions after "mdadm --zero-superblock"
From: NeilBrown @ 2015-04-07 0:52 UTC (permalink / raw)
To: Eduard Rozenberg; +Cc: linux-raid
In-Reply-To: <F271C907-A2A7-4132-BA11-D33353AE9FFF@pobox.com>
[-- Attachment #1: Type: text/plain, Size: 2697 bytes --]
On Mon, 6 Apr 2015 16:45:58 -0700 Eduard Rozenberg <eduardr@pobox.com> wrote:
> Hello folks,
>
> I previously had the following setup:
>
> sda & sdb partitioned w/ GPT, 7 partitions each (usr, opt, var etc...)
> 7 raid1’s with 2 devices for each pair of partitions (/dev/sda1 & /dev/sdab1, etc)
> They’d been created under Slackware 13.37.
>
> I was trying to clean out mdadm from those partitions but keep the data so I ran
> "mdadm --zero-superblock” on each of those previously RAID1 mdadm 1.2 ext4
> partitions.
The "1.2" metadata is stored 4k from the start of the device. The actual
data is some megabytes further in. I don't suppose you still have the output
of "mdadm --examine" from before you destroyed the superblocks??
>
> As a result I am now currently unable to mount any partition after the first one on either
> disk. The first partition does mount. The partition table is visible and looks fine in gdisk.
>
> mount -t ext4 /dev/sdac2 /mnt
> mount: wrong fs type, bad option, bad superblock on /dev/sdac2,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> I did try superblock recovery with each backup superblock that ext4 normally creates,
> but none of the superblock locations worked.
>
> For example:
>
> fsck.ext4 -b 4096000 /dev/sdac2
> e2fsck 1.42.8 (20-Jun-2013)
> /sbin/e2fsck: Invalid argument while trying to open /dev/sdac2
>
> The superblock could not be read or does not describe a correct ext2
> filesystem. If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
> e2fsck -b 8193 <device>
>
>
> Would be grateful for any advice on anything else I can try.
You need to find where the filesystem actually starts, then you need to
create some way to access it as a block device, then it should "just work".
An ext4 filesystem superblock has 0xef53 at an offset of 0x38, and the
superblock is typically 1K from the start of the partition.
So you could:
od -x /dev/sdac2 | awk '$6 == "ef53"'
Then subtrace 0002060 (octal) from the leading number, and that might be the
start of the partition.
Then
losetup -o "start in decimal" /dev/loop0 /dev/sdac2
and try 'fsck' on /dev/loop0
Good luck.
NeilBrown
>
> Regards,
> —Ed--
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: multi-threaded raid5
From: NeilBrown @ 2015-04-07 1:11 UTC (permalink / raw)
To: stkoupr; +Cc: linux-raid
In-Reply-To: <551AD6AE.4030607@cra.cz>
[-- Attachment #1: Type: text/plain, Size: 879 bytes --]
On Tue, 31 Mar 2015 19:17:34 +0200 stkoupr <p.kouril@cra.cz> wrote:
> Hi,
>
> We are testing write performance of md raid5 with SSD array and we are
> hitting limit which I understand is caused by writes being handled in
> single thread (md0_raid5 process is eating 100% of 1 CPU core when the
> test is running). I understood that there is some work in progress which
> should make write handling multi-threaded. I'd like to ask about the
> status of this work, is it still in progress or planned?
>
> Regards,
> Prema
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
See /sys/block/mdXXX/md/group_thread_cnt
Set it to a bigger number, get more threads.
Requires 3.12 or later.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: openSUSE 13.2: md - Need advice which raid1 super format (1.0/1.2) is preferred
From: NeilBrown @ 2015-04-07 1:14 UTC (permalink / raw)
To: Dieter Nützel; +Cc: linux-raid
In-Reply-To: <309b2c5704621339acbe84a19d84f766@mail.ud03.udmedia.de>
[-- Attachment #1: Type: text/plain, Size: 1839 bytes --]
On Wed, 01 Apr 2015 01:55:18 +0200 Dieter Nützel <Dieter@nuetzel-hh.de> wrote:
> Hello List,
>
> I need advice from the 'gurus' which raid1 md super format (1.0 or 1.2)
> is preferred for SSD/'old' SATA disks.
It depends on what you want.
If you want the filesystem in the partition to be visible without configuring
RAID, then 1.0.
If you explicitly don't. i.e. you don't see the filesystem until you
assemble the raid1, then 1.2.
Personally, I prefer 1.2, but your needs might be different.
If you boot off the raid1 array, then your boot loader might have a
preference.
NeilBrown
>
> Currently I have this on my gfx devel machine (Xeon X3430):
>
> cat /proc/mdstat
> Personalities : [raid1]
> md2 : active raid1 sdc1[0] sdd1[1]
> 156288896 blocks super 1.0 [2/2] [UU]
> bitmap: 0/2 pages [0KB], 65536KB chunk
>
> md1 : active raid1 sdb2[1] sda2[0]
> 60391744 blocks super 1.2 [2/2] [UU]
>
> md0 : active raid1 sdb1[1] sda1[0]
> 2095104 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
>
> md0 and md1
> are on my SSD's which I've created by 'hand' (with mdadm) under openSUSE
> 13.2.
>
> md2
> is on my SATAs which I've created 'automatically' under yast2.
>
> My question to you is:
> Looks this sane (chunk only for super 1.0 on real SATAs and super 1.2 on
> SSDs) or should I change anything, here.
>
> I'll nuke md0 (swap) and merge it with md1 (root) 'cause I've upgraded
> my RAM from 4 GB to 16 GB (soon to 24/32 GB max) and do not need swap
> anylonger.
>
> Any hints?
>
> -Dieter
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: write barrier in RAID5
From: NeilBrown @ 2015-04-07 1:25 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <551C37E6.6090201@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]
On Wed, 01 Apr 2015 12:24:38 -0600 Eric Mei <meijia@gmail.com> wrote:
> I can see how FUA is handled, but REQ_FLUSH seems problematic. My
> understanding of REQ_FLUSH semantics: all requests before the flush
> request must *complete* before the flush request complete. For IO remap
> layer like MD, we need to guarantee all previous requests are
> *submitted* to component devices before submitting the flush request. Is
> that correct?
No.
See Documentation/block/writeback_cache_control.txt
Only "previously completed" requests are affect by REQ_FLUSH.
So any code using REQ_FLUSH must first wait for previous requests to
complete, and then submit REQ_FLUSH. The REQ_FLUSH will not return until the
data written by those requests is actually in stable storage, rather than
write-behind cache.
NeilBrown
>
> If so why I don't see the code which make sure the pending write
> requests in stripe cache are processed & submitted before
> submit_flushes() get called? Thanks
>
> Eric
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Can't mount partitions after "mdadm --zero-superblock"
From: Eduard Rozenberg @ 2015-04-07 1:33 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150407105230.4ff9a97a@notabene.brown>
Thanks very much Neil!
I never do anything forward thinking like mdadm —examine before
doing destructive things :). Unfortunately.
I couldn’t figure out how to make the offset work so I used
losetup -f --show -P /dev/sdac
which automatically creates:
/dev/loop0
/dev/loop0p1
/dev/loop0p2
/dev/loop0p3
/dev/loop0p4
/dev/loop0p5
/dev/loop0p6
/dev/loop0p7
/dev/loop0p7
/dev/loop0p8
But the errors are the same (below). No idea why it thinks
NTFS is involved. I’ll probably have to give up on
this and figure out how to get way better at backups because
my history of success with recovery once things get hosed
in some way is very poor.
[root@repo1 ~]# fsck.ext4 /dev/loop0p2
e2fsck 1.42.8 (20-Jun-2013)
ext2fs_open2: Bad magic number in super-block
/sbin/e2fsck: Superblock invalid, trying backup blocks...
/sbin/e2fsck: Bad magic number in super-block while trying to open /dev/loop0p2
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
======
mount -o ro /dev/loop0p2 /mnt
Failed to mount '/dev/loop0p2': Invalid argument
The device '/dev/loop0p2' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
mount -o ro /dev/loop0p3 /mnt
NTFS signature is missing.
Failed to mount '/dev/loop0p3': Invalid argument
The device '/dev/loop0p3' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
Regards,
—Ed
> On Apr 6, 2015, at 17:52, NeilBrown <neilb@suse.de> wrote:
>
> On Mon, 6 Apr 2015 16:45:58 -0700 Eduard Rozenberg <eduardr@pobox.com> wrote:
>
>> Hello folks,
>>
>> I previously had the following setup:
>>
>> sda & sdb partitioned w/ GPT, 7 partitions each (usr, opt, var etc...)
>> 7 raid1’s with 2 devices for each pair of partitions (/dev/sda1 & /dev/sdab1, etc)
>> They’d been created under Slackware 13.37.
>>
>> I was trying to clean out mdadm from those partitions but keep the data so I ran
>> "mdadm --zero-superblock” on each of those previously RAID1 mdadm 1.2 ext4
>> partitions.
>
> The "1.2" metadata is stored 4k from the start of the device. The actual
> data is some megabytes further in. I don't suppose you still have the output
> of "mdadm --examine" from before you destroyed the superblocks??
>
>>
>> As a result I am now currently unable to mount any partition after the first one on either
>> disk. The first partition does mount. The partition table is visible and looks fine in gdisk.
>>
>> mount -t ext4 /dev/sdac2 /mnt
>> mount: wrong fs type, bad option, bad superblock on /dev/sdac2,
>> missing codepage or helper program, or other error
>> In some cases useful info is found in syslog - try
>> dmesg | tail or so
>>
>> I did try superblock recovery with each backup superblock that ext4 normally creates,
>> but none of the superblock locations worked.
>>
>> For example:
>>
>> fsck.ext4 -b 4096000 /dev/sdac2
>> e2fsck 1.42.8 (20-Jun-2013)
>> /sbin/e2fsck: Invalid argument while trying to open /dev/sdac2
>>
>> The superblock could not be read or does not describe a correct ext2
>> filesystem. If the device is valid and it really contains an ext2
>> filesystem (and not swap or ufs or something else), then the superblock
>> is corrupt, and you might try running e2fsck with an alternate superblock:
>> e2fsck -b 8193 <device>
>>
>>
>> Would be grateful for any advice on anything else I can try.
>
> You need to find where the filesystem actually starts, then you need to
> create some way to access it as a block device, then it should "just work".
>
> An ext4 filesystem superblock has 0xef53 at an offset of 0x38, and the
> superblock is typically 1K from the start of the partition.
>
> So you could:
> od -x /dev/sdac2 | awk '$6 == "ef53"'
>
> Then subtrace 0002060 (octal) from the leading number, and that might be the
> start of the partition.
>
> Then
> losetup -o "start in decimal" /dev/loop0 /dev/sdac2
>
> and try 'fsck' on /dev/loop0
>
> Good luck.
>
> NeilBrown
>
>
>
>>
>> Regards,
>> —Ed--
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 1:50 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <20150407094608.4a9dd142@notabene.brown>
[-- Attachment #1: Type: text/plain, Size: 642 bytes --]
On Tue, 7 Apr 2015 09:46:08 +1000 NeilBrown <neilb@suse.de> wrote:
> On Fri, 3 Apr 2015 19:22:15 -0700 "Jonathan Harker (Jesusaurus)"
> <jesusaurus@gentlydownthe.net> wrote:
>
> >
> > Third, is it possible to resume the reshape? If not, can it be reverted?
> >
>
> I'll have to get back to you on that. Hopefully you can be patient a little
> longer.
>
> NeilBrown
>
>
Hi again Jonathan,
please report what version of mdadm and of Linux that you are running.
Also is there any chance that you tried
mdadm --assemble .... --update=revert-reshape
??
And if you did, why didn't you say so :-)
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Can't mount partitions after "mdadm --zero-superblock"
From: NeilBrown @ 2015-04-07 1:52 UTC (permalink / raw)
To: Eduard Rozenberg; +Cc: linux-raid
In-Reply-To: <CE6323D1-485D-4951-9E99-CBE5A09AE7BD@pobox.com>
[-- Attachment #1: Type: text/plain, Size: 621 bytes --]
On Mon, 6 Apr 2015 18:33:36 -0700 Eduard Rozenberg <eduardr@pobox.com> wrote:
> Thanks very much Neil!
>
> I never do anything forward thinking like mdadm —examine before
> doing destructive things :). Unfortunately.
>
> I couldn’t figure out how to make the offset work so I used
>
> losetup -f --show -P /dev/sdac
Sounds like you are saying "I couldn't make the drill work, so I tried using a
hammer"... and then were surprised that you couldn't make any holes.
Try harder to make the offset work. if/when it doesn't, report exactly what
you did and exactly what the result was.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Can't mount partitions after "mdadm --zero-superblock"
From: Eduard Rozenberg @ 2015-04-07 3:31 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150407105230.4ff9a97a@notabene.brown>
Hello Neil,
Success! This is stronger voodoo magic than I’ve ever
had to perform so really didn’t have the faith to continue
without the extra encouragement :). My initial mistake
had been to use the octal value with losetup instead of
getting the decimal value.
Documenting my steps below for anybody else who
might come here later.
In the examples below, we are using partition "/dev/sdac2"
Replace this with the appropriate partition you’re recovering.
Step 1: find the decimal value for the start of the partition
----------------------------------------------------------------------------
Note:
ext4 partitions have a “magic" octal value of “ef53"
to indicate the start of the partition. Note that "ef53"
may show more than once as you read further into
the partition. We are interested in the location the
very first occurrence of "ef53". Other types of
partitions (ext2, ext3, etc) probably have other magic
values to look for so this “ef53" may not apply there.
The "od" hex viewer command to search for “ef53":
od -x /dev/sdac2 | awk '$6 == "ef53"'
The results will look something like:
4002060 f3fd 5521 0004 0025 ef53 0001 0001 0000
1004000060 64be 4ec9 0000 0025 ef53 0000 0001 0000
1042630400 17f8 a7dd bb6e ee40 ef53 000d 3cfb 9e22
We are only interested in the first line. So we now have
the octal address of the ef53 magic value: it’s the first
long number on the line: "4002060" (octal value!)
According to Neil’s instructions then we have to subtract
the octal value “0002060" from this number we found.
We then have to convert the octal result into decimal.
Luckily an online calculator makes this easy:
http://www.csgnetwork.com/octaddsubcalc.html
"Enter a octal value" - Enter “4002060" here
"Enter Second Octal Value" - Enter “0002060" here
Then take the value from the line:
"Calculated Decimal Subtraction" - 1048576
This is the decimal value for the start of our partition.
Step 2: use the decimal start value to mount partition
———————————————————————————————————
First create a loop device loop0 as a handle to the
partition. We tell losetup where the start of the partition is:
losetup -o 1048576 /dev/loop0 /dev/sdac2
Next, try to mount loop0 read-only (hopefully it will work!)
mount -o ro /dev/loop0 /mnt
If the partition is unclean and needs to be fscked:
fsck.ext4 /dev/loop0
Thanks again Neil! Maybe a few years from now I’ll
understand why this worked when nothing else did
( linux tools still have some ways to go to being
intelligent enough to do this kind of recovery).
Regards,
—Ed
> On Apr 6, 2015, at 17:52, NeilBrown <neilb@suse.de> wrote:
>
> On Mon, 6 Apr 2015 16:45:58 -0700 Eduard Rozenberg <eduardr@pobox.com> wrote:
>
>> Hello folks,
>>
>> I previously had the following setup:
>>
>> sda & sdb partitioned w/ GPT, 7 partitions each (usr, opt, var etc...)
>> 7 raid1’s with 2 devices for each pair of partitions (/dev/sda1 & /dev/sdab1, etc)
>> They’d been created under Slackware 13.37.
>>
>> I was trying to clean out mdadm from those partitions but keep the data so I ran
>> "mdadm --zero-superblock” on each of those previously RAID1 mdadm 1.2 ext4
>> partitions.
>
> The "1.2" metadata is stored 4k from the start of the device. The actual
> data is some megabytes further in. I don't suppose you still have the output
> of "mdadm --examine" from before you destroyed the superblocks??
>
>>
>> As a result I am now currently unable to mount any partition after the first one on either
>> disk. The first partition does mount. The partition table is visible and looks fine in gdisk.
>>
>> mount -t ext4 /dev/sdac2 /mnt
>> mount: wrong fs type, bad option, bad superblock on /dev/sdac2,
>> missing codepage or helper program, or other error
>> In some cases useful info is found in syslog - try
>> dmesg | tail or so
>>
>> I did try superblock recovery with each backup superblock that ext4 normally creates,
>> but none of the superblock locations worked.
>>
>> For example:
>>
>> fsck.ext4 -b 4096000 /dev/sdac2
>> e2fsck 1.42.8 (20-Jun-2013)
>> /sbin/e2fsck: Invalid argument while trying to open /dev/sdac2
>>
>> The superblock could not be read or does not describe a correct ext2
>> filesystem. If the device is valid and it really contains an ext2
>> filesystem (and not swap or ufs or something else), then the superblock
>> is corrupt, and you might try running e2fsck with an alternate superblock:
>> e2fsck -b 8193 <device>
>>
>>
>> Would be grateful for any advice on anything else I can try.
>
> You need to find where the filesystem actually starts, then you need to
> create some way to access it as a block device, then it should "just work".
>
> An ext4 filesystem superblock has 0xef53 at an offset of 0x38, and the
> superblock is typically 1K from the start of the partition.
>
> So you could:
> od -x /dev/sdac2 | awk '$6 == "ef53"'
>
> Then subtrace 0002060 (octal) from the leading number, and that might be the
> start of the partition.
>
> Then
> losetup -o "start in decimal" /dev/loop0 /dev/sdac2
>
> and try 'fsck' on /dev/loop0
>
> Good luck.
>
> NeilBrown
>
>
>
>>
>> Regards,
>> —Ed--
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: multi-threaded raid5
From: Thomas Fjellstrom @ 2015-04-07 4:34 UTC (permalink / raw)
To: NeilBrown, linux-raid
In-Reply-To: <20150407111157.2200fb0b@notabene.brown>
On Tue 07 Apr 2015 11:11:57 AM NeilBrown wrote:
> On Tue, 31 Mar 2015 19:17:34 +0200 stkoupr <p.kouril@cra.cz> wrote:
> > Hi,
> >
> > We are testing write performance of md raid5 with SSD array and we are
> > hitting limit which I understand is caused by writes being handled in
> > single thread (md0_raid5 process is eating 100% of 1 CPU core when the
> > test is running). I understood that there is some work in progress which
> > should make write handling multi-threaded. I'd like to ask about the
> > status of this work, is it still in progress or planned?
> >
> > Regards,
> > Prema
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> See /sys/block/mdXXX/md/group_thread_cnt
>
> Set it to a bigger number, get more threads.
>
> Requires 3.12 or later.
What does it mean when that property is set to 0?
> NeilBrown
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-07 5:14 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150407115033.1d63b65c@notabene.brown>
On Mon, Apr 6, 2015 at 6:50 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 7 Apr 2015 09:46:08 +1000 NeilBrown <neilb@suse.de> wrote:
>
>> On Fri, 3 Apr 2015 19:22:15 -0700 "Jonathan Harker (Jesusaurus)"
>> <jesusaurus@gentlydownthe.net> wrote:
>>
>> >
>> > Third, is it possible to resume the reshape? If not, can it be reverted?
>> >
>>
>> I'll have to get back to you on that. Hopefully you can be patient a little
>> longer.
>>
>> NeilBrown
>>
Thanks for taking the time to look into this. I've managed to be
patient with the problem for about a week now (I was cautiously poking
and prodding the array for a few days before reaching out to this
list) so I think I can manage a little more patience.
>>
>
> Hi again Jonathan,
> please report what version of mdadm and of Linux that you are running.
My mdadm version is "mdadm - v3.3.1 - 5th June 2014" and my linux
version (according to `uname -r`) is 3.16.2-3.
>
> Also is there any chance that you tried
> mdadm --assemble .... --update=revert-reshape
> ??
> And if you did, why didn't you say so :-)
During those first couple days I tried a couple assemble and
incremental commands to try to get the reshape to restart, though I
don't remember exactly what they were. It's possible I tried something
as scary and irreversible as --update=revert-reshape in some desperate
haze, so let's assume I did, but I was trying hard to avoid any advice
that included --force in the command.
>
> NeilBrown
Thanks again for looking into this, and let me know if there is any
other useful information I can provide.
--
Jonathan
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 6:30 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AFo47Zfe7rMqro7_k8dR-xpMcfiwQA7a9DTKpKPuHr=fA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]
On Mon, 6 Apr 2015 22:14:56 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> On Mon, Apr 6, 2015 at 6:50 PM, NeilBrown <neilb@suse.de> wrote:
> > On Tue, 7 Apr 2015 09:46:08 +1000 NeilBrown <neilb@suse.de> wrote:
> >
> >> On Fri, 3 Apr 2015 19:22:15 -0700 "Jonathan Harker (Jesusaurus)"
> >> <jesusaurus@gentlydownthe.net> wrote:
> >>
> >> >
> >> > Third, is it possible to resume the reshape? If not, can it be reverted?
> >> >
> >>
> >> I'll have to get back to you on that. Hopefully you can be patient a little
> >> longer.
> >>
> >> NeilBrown
> >>
>
> Thanks for taking the time to look into this. I've managed to be
> patient with the problem for about a week now (I was cautiously poking
> and prodding the array for a few days before reaching out to this
> list) so I think I can manage a little more patience.
>
> >>
> >
> > Hi again Jonathan,
> > please report what version of mdadm and of Linux that you are running.
>
> My mdadm version is "mdadm - v3.3.1 - 5th June 2014" and my linux
> version (according to `uname -r`) is 3.16.2-3.
>
> >
> > Also is there any chance that you tried
> > mdadm --assemble .... --update=revert-reshape
> > ??
> > And if you did, why didn't you say so :-)
>
> During those first couple days I tried a couple assemble and
> incremental commands to try to get the reshape to restart, though I
> don't remember exactly what they were. It's possible I tried something
> as scary and irreversible as --update=revert-reshape in some desperate
> haze, so let's assume I did, but I was trying hard to avoid any advice
> that included --force in the command.
Try:
mdadm -S /dev/md124
mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
mdadm -S /dev/md124
mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
What does that report?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [dm-devel] [PATCH 3/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Alasdair G Kergon @ 2015-04-07 13:55 UTC (permalink / raw)
To: Pali Rohár
Cc: Alasdair Kergon, Mike Snitzer, Neil Brown, Rafael J. Wysocki,
Len Brown, Pavel Machek, linux-raid, dm-devel, linux-kernel,
linux-pm
In-Reply-To: <1428254419-7334-4-git-send-email-pali.rohar@gmail.com>
On Sun, Apr 05, 2015 at 07:20:19PM +0200, Pali Rohár wrote:
> This patch adds dm message commands and option strings to optionally wipe key
> from dm-crypt device before entering suspend or hibernate state.
Try to avoid 0/1 - use descriptive options instead.
E.g. key wipe_on_hibernation / key retain_on_hibernation (message)
wipe_key_on_hiberation ('dmsetup table' - don't forget the reporting interface!)
Have you tested against every state the driver might be in at the time of
suspend/hibernation?
> +static void crypt_suspend_and_wipe_key(struct crypt_config *cc)
> +{
> + dm_suspend_md(dm_table_get_md(cc->ti->table));
I'm not particularly keen on this - silently ignoring expected error states
like -EINVAL rather than checking first and not calling the function at all
when it's known not to be needed.
Alasdair
^ permalink raw reply
* Creating volume in IMSM container
From: Ian Pilcher @ 2015-04-07 15:29 UTC (permalink / raw)
To: linux-raid
I am trying to create an IMSM container and volume in a KVM virtual
machine (so that I can create a reproducer for a Fedora problem that
seems to only/mostly affect IMSM RAID). I've managed to create a
container with the following command:
IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/imsm \
/dev/vd{a,b} --raid-devices 2 --metadata=imsm
/proc/mdstat now contains:
Personalities: [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] ...
md127 : inactive vdb[1](S) vda[0](S)
2210 blocks super external:imsm
I haven't, however, been able to create a volume within the container:
# IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
/dev/md/imsm --raid-devices 2 --level 1 --metadata=imsm
mdmon: not enough devices with space to create array.
# IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
/dev/vd{a,b} --raid-devices 2 --level 1 --metadata=imsm
mdadm: /dev/vda appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 00:00:00 1970
mdmon: /dev/vdb is not in the same imsm set
mdadm: /dev/vdb is not suitable for this array.
mdadm: create aborted
Am I doing something wrong, or is this simply not possible?
--
========================================================================
Ian Pilcher arequipeno@gmail.com
-------- "I grew up before Mark Zuckerberg invented friendship" --------
========================================================================
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-07 17:02 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150407163004.7550da77@notabene.brown>
On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
>
> Try:
> mdadm -S /dev/md124
> mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> mdadm -S /dev/md124
> mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>
> What does that report?
>
> NeilBrown
>
# mdadm --stop /dev/md124
mdadm: stopped /dev/md124
# mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
[linear] [multipath]
md124 : inactive md126[0](S) md127[1](S)
3907022200 blocks super 1.2
md0 : active raid1 sda5[0] sdb2[1]
107652416 blocks [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md125 : active raid1 sdh1[0] sdg1[1]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md126 : active raid1 sdc1[0] sdd1[1]
1953512312 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[2] sdf1[1]
1953512312 blocks super 1.2 [2/2] [UU]
unused devices: <none>
# mdadm --stop /dev/md124
mdadm: stopped /dev/md124
# mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
mdadm: looking for devices for /dev/md124
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
mdadm: :/dev/md124 has an active reshape - checking if critical
section needs to be restored
mdadm: added /dev/md/alpha to /dev/md124 as 1
mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
mdadm: no uptodate device for slot 6 of /dev/md124
mdadm: added /dev/md/beta to /dev/md124 as 0
mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
[linear] [multipath]
md124 : inactive md125[3](S) md127[1](S) md126[0](S)
6837155192 blocks super 1.2
md0 : active raid1 sda5[0] sdb2[1]
107652416 blocks [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md125 : active raid1 sdh1[0] sdg1[1]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md126 : active raid1 sdc1[0] sdd1[1]
1953512312 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[2] sdf1[1]
1953512312 blocks super 1.2 [2/2] [UU]
unused devices: <none>
# mdadm --examine /dev/md/alpha
/dev/md/alpha:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 4
Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=752 sectors
State : active
Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : 1 (3->4)
Update Time : Sun Mar 29 15:11:35 2015
Checksum : 8be5e0e8 - correct
Events : 14013
Chunk Size : 512K
Device Role : Active device 1
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm --examine /dev/md/beta
/dev/md/beta:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 4
Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=752 sectors
State : clean
Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : 1 (3->4)
Update Time : Sun Mar 29 15:11:35 2015
Checksum : f7526adf - correct
Events : 14013
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm --examine /dev/md/gamma
/dev/md/gamma:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x6
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 4
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Recovery Offset : 86403072 sectors
Unused Space : before=1960 sectors, after=1953244160 sectors
State : active
Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : 1 (3->4)
Update Time : Sun Mar 29 00:05:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 710dc078 - correct
Events : 673
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm --detail /dev/md124
/dev/md124:
Version : 1.2
Raid Level : raid0
Total Devices : 3
Persistence : Superblock is persistent
State : inactive
Delta Devices : 1, (-1->0)
New Level : raid4
New Chunksize : 512K
Name : hordern:hordern1 (local to host hordern)
UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Events : 673
Number Major Minor RaidDevice
- 9 125 - /dev/md/gamma
- 9 126 - /dev/md/beta
- 9 127 - /dev/md/alpha
So it looks like all three component devices have consistent
superblocks now, awesome! But the raid0 array is still inactive with
all three components listed as spares. It looks like /dev/md/gamma has
a much lower event count, I'm guessing that is what causes the disk to
be marked as possibly out of date.
Is an "uptodate device" a specific thing, or does that simply mean
that some component devices are out of date? The lack of spaces makes
me think that uptodate is some keyword I'm not recognizing.
--
Jonathan
^ permalink raw reply
* [PATCH 2/2][v2] blk-plug: don't flush nested plug lists
From: Jeff Moyer @ 2015-04-07 18:55 UTC (permalink / raw)
To: Jens Axboe, Ming Lei
Cc: Konrad Rzeszutek Wilk, Roger Pau Monn??, Alasdair Kergon,
Mike Snitzer, Neil Brown, Nicholas A. Bellinger, Alexander Viro,
Chris Mason, Josef Bacik, David Sterba, Theodore Ts'o,
Andreas Dilger, Jaegeuk Kim, Changman Lee, Steven Whitehouse,
Mikulas Patocka, Andrew Morton, Rik van Riel, Johannes Weiner,
Mel Gorman, Trond Myklebust <trond.myk>
In-Reply-To: <1428347694-17704-2-git-send-email-jmoyer@redhat.com>
The way the on-stack plugging currently works, each nesting level
flushes its own list of I/Os. This can be less than optimal (read
awful) for certain workloads. For example, consider an application
that issues asynchronous O_DIRECT I/Os. It can send down a bunch of
I/Os together in a single io_submit call, only to have each of them
dispatched individually down in the bowels of the dirct I/O code.
The reason is that there are blk_plug-s instantiated both at the upper
call site in do_io_submit and down in do_direct_IO. The latter will
submit as little as 1 I/O at a time (if you have a small enough I/O
size) instead of performing the batching that the plugging
infrastructure is supposed to provide.
Now, for the case where there is an elevator involved, this doesn't
really matter too much. The elevator will keep the I/O around long
enough for it to be merged. However, in cases where there is no
elevator (like blk-mq), I/Os are simply dispatched immediately.
Try this, for example:
fio --rw=read --bs=4k --iodepth=128 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=10s --direct=1 --filename=/dev/vdd --name=job1 --ioengine=libaio --time_based
If you run that on a current kernel, you will get zero merges. Zero!
After this patch, you will get many merges (the actual number depends
on how fast your storage is, obviously), and much better throughput.
Here are results from my test systems:
First, I tested in a VM using a virtio-blk device:
Unpatched kernel:
Throughput: 280,262 KB/s
avg latency: 14,587.72 usec
Patched kernel:
throughput: 832,158 KB/s
avg latency: 4,901.95 usec
Next, I tesetd using a micron p320h on bare metal:
Unpatched kernel:
Throughput: 688,967 KB/s
avg latency: 5,933.92 usec
Patched kernel:
Throughput: 1,160.6 MB/s
avg latency: 3,437.01 usec
As you can see, both throughput and latency improved dramatically.
I've included the full fio output below, so you can also see the
marked improvement in standard deviation as well.
I considered several approaches to solving the problem:
1) get rid of the inner-most plugs
2) handle nesting by using only one on-stack plug
2a) #2, except use a per-cpu blk_plug struct, which may clean up the
code a bit at the expense of memory footprint
Option 1 will be tricky or impossible to do, since inner most plug
lists are sometimes the only plug lists, depending on the call path.
Option 2 is what this patch implements. Option 2a may add unneeded
complexity.
Much of the patch involves modifying call sites to blk_finish_plug,
since its signature is changed. The meat of the patch is actually
pretty simple and constrained to block/blk-core.c and
include/linux/blkdev.h. The only tricky bits were places where plugs
were finished and then restarted to flush out I/O. There, I left things
as-is. So long as they are the outer-most plugs, they should continue
to function as before.
NOTE TO SUBSYSTEM MAINTAINERS: Before this patch, blk_finish_plug
would always flush the plug list. After this patch, this is only the
case for the outer-most plug. If you require the plug list to be
flushed, you should be calling blk_flush_plug(current). Btrfs and dm
maintainers should take a close look at this patch and ensure they get
the right behavior in the end.
Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
---
Changelog:
v1->v2: Keep the blk_start_plug interface the same, suggested by Ming Lei.
Test results
------------
Virtio-blk:
unpatched:
job1: (groupid=0, jobs=1): err= 0: pid=8032: Tue Apr 7 13:33:53 2015
read : io=2736.1MB, bw=280262KB/s, iops=70065, runt= 10000msec
slat (usec): min=40, max=10472, avg=207.82, stdev=364.02
clat (usec): min=211, max=35883, avg=14379.83, stdev=2213.95
lat (usec): min=862, max=36000, avg=14587.72, stdev=2223.80
clat percentiles (usec):
| 1.00th=[11328], 5.00th=[12096], 10.00th=[12480], 20.00th=[12992],
| 30.00th=[13376], 40.00th=[13760], 50.00th=[14144], 60.00th=[14400],
| 70.00th=[14784], 80.00th=[15168], 90.00th=[15936], 95.00th=[16768],
| 99.00th=[24448], 99.50th=[25216], 99.90th=[28544], 99.95th=[35072],
| 99.99th=[36096]
bw (KB /s): min=265984, max=302720, per=100.00%, avg=280549.84, stdev=10264.36
lat (usec) : 250=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%, 10=0.05%, 20=96.57%, 50=3.34%
cpu : usr=7.56%, sys=55.57%, ctx=6174, majf=0, minf=523
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=700656/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=2736.1MB, aggrb=280262KB/s, minb=280262KB/s, maxb=280262KB/s, mint=10000msec, maxt=10000msec
Disk stats (read/write):
vdd: ios=695490/0, merge=0/0, ticks=785741/0, in_queue=785442, util=90.69%
patched:
job1: (groupid=0, jobs=1): err= 0: pid=7743: Tue Apr 7 13:19:07 2015
read : io=8126.6MB, bw=832158KB/s, iops=208039, runt= 10000msec
slat (usec): min=20, max=14351, avg=55.08, stdev=143.47
clat (usec): min=283, max=20003, avg=4846.77, stdev=1355.35
lat (usec): min=609, max=20074, avg=4901.95, stdev=1362.40
clat percentiles (usec):
| 1.00th=[ 4016], 5.00th=[ 4048], 10.00th=[ 4080], 20.00th=[ 4128],
| 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4512],
| 70.00th=[ 4896], 80.00th=[ 5664], 90.00th=[ 5920], 95.00th=[ 6752],
| 99.00th=[11968], 99.50th=[13632], 99.90th=[15552], 99.95th=[17024],
| 99.99th=[19840]
bw (KB /s): min=740992, max=896640, per=100.00%, avg=836978.95, stdev=51034.87
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 4=0.50%, 10=97.79%, 20=1.70%, 50=0.01%
cpu : usr=20.28%, sys=69.11%, ctx=879, majf=0, minf=522
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=2080396/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=8126.6MB, aggrb=832158KB/s, minb=832158KB/s, maxb=832158KB/s, mint=10000msec, maxt=10000msec
Disk stats (read/write):
vdd: ios=127877/0, merge=1918166/0, ticks=23118/0, in_queue=23047, util=94.08%
micron p320h:
unpatched:
job1: (groupid=0, jobs=1): err= 0: pid=3244: Tue Apr 7 13:29:14 2015
read : io=6728.9MB, bw=688968KB/s, iops=172241, runt= 10001msec
slat (usec): min=43, max=6273, avg=81.79, stdev=125.96
clat (usec): min=78, max=12485, avg=5852.06, stdev=1154.76
lat (usec): min=146, max=12572, avg=5933.92, stdev=1163.75
clat percentiles (usec):
| 1.00th=[ 4192], 5.00th=[ 4384], 10.00th=[ 4576], 20.00th=[ 5600],
| 30.00th=[ 5664], 40.00th=[ 5728], 50.00th=[ 5792], 60.00th=[ 5856],
| 70.00th=[ 6112], 80.00th=[ 6176], 90.00th=[ 6240], 95.00th=[ 6368],
| 99.00th=[11840], 99.50th=[11968], 99.90th=[12096], 99.95th=[12096],
| 99.99th=[12224]
bw (KB /s): min=648328, max=859264, per=98.80%, avg=680711.16, stdev=62016.70
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.04%, 10=97.07%, 20=2.87%
cpu : usr=10.28%, sys=73.61%, ctx=104436, majf=0, minf=6217
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1722592/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=6728.9MB, aggrb=688967KB/s, minb=688967KB/s, maxb=688967KB/s, mint=10001msec, maxt=10001msec
Disk stats (read/write):
rssda: ios=1688772/0, merge=0/0, ticks=188820/0, in_queue=188678, util=96.61%
patched:
job1: (groupid=0, jobs=1): err= 0: pid=9531: Tue Apr 7 13:22:28 2015
read : io=11607MB, bw=1160.6MB/s, iops=297104, runt= 10001msec
slat (usec): min=21, max=6376, avg=43.05, stdev=81.82
clat (usec): min=116, max=9844, avg=3393.90, stdev=752.57
lat (usec): min=167, max=9889, avg=3437.01, stdev=757.02
clat percentiles (usec):
| 1.00th=[ 2832], 5.00th=[ 2992], 10.00th=[ 3056], 20.00th=[ 3120],
| 30.00th=[ 3152], 40.00th=[ 3248], 50.00th=[ 3280], 60.00th=[ 3344],
| 70.00th=[ 3376], 80.00th=[ 3504], 90.00th=[ 3728], 95.00th=[ 3824],
| 99.00th=[ 9152], 99.50th=[ 9408], 99.90th=[ 9664], 99.95th=[ 9664],
| 99.99th=[ 9792]
bw (MB /s): min= 1139, max= 1183, per=100.00%, avg=1161.07, stdev=10.58
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=98.31%, 10=1.67%
cpu : usr=18.59%, sys=66.65%, ctx=55655, majf=0, minf=6218
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=2971338/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=11607MB, aggrb=1160.6MB/s, minb=1160.6MB/s, maxb=1160.6MB/s, mint=10001msec, maxt=10001msec
Disk stats (read/write):
rssda: ios=183005/0, merge=2745105/0, ticks=31972/0, in_queue=31948, util=97.63%
---
block/blk-core.c | 29 ++++++++++++++++-------------
block/blk-lib.c | 2 +-
block/blk-throttle.c | 2 +-
drivers/block/xen-blkback/blkback.c | 2 +-
drivers/md/dm-bufio.c | 6 +++---
drivers/md/dm-crypt.c | 2 +-
drivers/md/dm-kcopyd.c | 2 +-
drivers/md/dm-thin.c | 2 +-
drivers/md/md.c | 2 +-
drivers/md/raid1.c | 2 +-
drivers/md/raid10.c | 2 +-
drivers/md/raid5.c | 4 ++--
drivers/target/target_core_iblock.c | 2 +-
fs/aio.c | 2 +-
fs/block_dev.c | 2 +-
fs/btrfs/scrub.c | 2 +-
fs/btrfs/transaction.c | 2 +-
fs/btrfs/tree-log.c | 12 ++++++------
fs/btrfs/volumes.c | 6 +++---
fs/buffer.c | 2 +-
fs/direct-io.c | 2 +-
fs/ext4/file.c | 2 +-
fs/ext4/inode.c | 4 ++--
fs/f2fs/checkpoint.c | 2 +-
fs/f2fs/gc.c | 2 +-
fs/f2fs/node.c | 2 +-
fs/gfs2/log.c | 2 +-
fs/hpfs/buffer.c | 2 +-
fs/jbd/checkpoint.c | 2 +-
fs/jbd/commit.c | 4 ++--
fs/jbd2/checkpoint.c | 2 +-
fs/jbd2/commit.c | 2 +-
fs/mpage.c | 2 +-
fs/nfs/blocklayout/blocklayout.c | 4 ++--
fs/xfs/xfs_buf.c | 4 ++--
fs/xfs/xfs_dir2_readdir.c | 2 +-
fs/xfs/xfs_itable.c | 2 +-
include/linux/blkdev.h | 5 +++--
mm/madvise.c | 2 +-
mm/page-writeback.c | 2 +-
mm/readahead.c | 2 +-
mm/swap_state.c | 2 +-
mm/vmscan.c | 2 +-
43 files changed, 74 insertions(+), 70 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 794c3e7..fcd9c2f 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -3018,21 +3018,21 @@ void blk_start_plug(struct blk_plug *plug)
{
struct task_struct *tsk = current;
+ if (tsk->plug) {
+ tsk->plug->depth++;
+ return;
+ }
+
+ plug->depth = 1;
INIT_LIST_HEAD(&plug->list);
INIT_LIST_HEAD(&plug->mq_list);
INIT_LIST_HEAD(&plug->cb_list);
/*
- * If this is a nested plug, don't actually assign it. It will be
- * flushed on its own.
+ * Store ordering should not be needed here, since a potential
+ * preempt will imply a full memory barrier
*/
- if (!tsk->plug) {
- /*
- * Store ordering should not be needed here, since a potential
- * preempt will imply a full memory barrier
- */
- tsk->plug = plug;
- }
+ tsk->plug = plug;
}
EXPORT_SYMBOL(blk_start_plug);
@@ -3177,12 +3177,15 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule)
local_irq_restore(flags);
}
-void blk_finish_plug(struct blk_plug *plug)
+void blk_finish_plug(void)
{
- blk_flush_plug_list(plug, false);
+ struct blk_plug *plug = current->plug;
- if (plug == current->plug)
- current->plug = NULL;
+ if (--plug->depth > 0)
+ return;
+
+ blk_flush_plug_list(plug, false);
+ current->plug = NULL;
}
EXPORT_SYMBOL(blk_finish_plug);
diff --git a/block/blk-lib.c b/block/blk-lib.c
index 7688ee3..ac347d3 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -128,7 +128,7 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector,
*/
cond_resched();
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
/* Wait for bios in-flight */
if (!atomic_dec_and_test(&bb.done))
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 5b9c6d5..222a77a 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1281,7 +1281,7 @@ static void blk_throtl_dispatch_work_fn(struct work_struct *work)
blk_start_plug(&plug);
while((bio = bio_list_pop(&bio_list_on_stack)))
generic_make_request(bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
}
diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
index 2a04d34..74bea21 100644
--- a/drivers/block/xen-blkback/blkback.c
+++ b/drivers/block/xen-blkback/blkback.c
@@ -1374,7 +1374,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
submit_bio(operation, biolist[i]);
/* Let the I/Os go.. */
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (operation == READ)
blkif->st_rd_sect += preq.nr_sects;
diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
index 86dbbc7..502c63b 100644
--- a/drivers/md/dm-bufio.c
+++ b/drivers/md/dm-bufio.c
@@ -715,7 +715,7 @@ static void __flush_write_list(struct list_head *write_list)
submit_io(b, WRITE, b->block, write_endio);
dm_bufio_cond_resched();
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
@@ -1126,7 +1126,7 @@ void dm_bufio_prefetch(struct dm_bufio_client *c,
&write_list);
if (unlikely(!list_empty(&write_list))) {
dm_bufio_unlock(c);
- blk_finish_plug(&plug);
+ blk_finish_plug();
__flush_write_list(&write_list);
blk_start_plug(&plug);
dm_bufio_lock(c);
@@ -1149,7 +1149,7 @@ void dm_bufio_prefetch(struct dm_bufio_client *c,
dm_bufio_unlock(c);
flush_plug:
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
EXPORT_SYMBOL_GPL(dm_bufio_prefetch);
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 713a962..65d7b72 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -1224,7 +1224,7 @@ pop_from_list:
rb_erase(&io->rb_node, &write_tree);
kcryptd_io_write(io);
} while (!RB_EMPTY_ROOT(&write_tree));
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
return 0;
}
diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c
index 3a7cade..4a76e42 100644
--- a/drivers/md/dm-kcopyd.c
+++ b/drivers/md/dm-kcopyd.c
@@ -593,7 +593,7 @@ static void do_work(struct work_struct *work)
process_jobs(&kc->complete_jobs, kc, run_complete_job);
process_jobs(&kc->pages_jobs, kc, run_pages_job);
process_jobs(&kc->io_jobs, kc, run_io_job);
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
index 921aafd..be42bf5 100644
--- a/drivers/md/dm-thin.c
+++ b/drivers/md/dm-thin.c
@@ -1824,7 +1824,7 @@ static void process_thin_deferred_bios(struct thin_c *tc)
dm_pool_issue_prefetches(pool->pmd);
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static int cmp_cells(const void *lhs, const void *rhs)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 717daad..c4ec179 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7686,7 +7686,7 @@ void md_do_sync(struct md_thread *thread)
/*
* this also signals 'finished resyncing' to md_stop
*/
- blk_finish_plug(&plug);
+ blk_finish_plug();
wait_event(mddev->recovery_wait, !atomic_read(&mddev->recovery_active));
/* tell personality that we are finished */
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index d34e238..4f8fad4 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2441,7 +2441,7 @@ static void raid1d(struct md_thread *thread)
if (mddev->flags & ~(1<<MD_CHANGE_PENDING))
md_check_recovery(mddev);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static int init_resync(struct r1conf *conf)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index a7196c4..92bb5dd 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2835,7 +2835,7 @@ static void raid10d(struct md_thread *thread)
if (mddev->flags & ~(1<<MD_CHANGE_PENDING))
md_check_recovery(mddev);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static int init_resync(struct r10conf *conf)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cd2f96b..695bf0f 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5281,7 +5281,7 @@ static void raid5_do_work(struct work_struct *work)
pr_debug("%d stripes handled\n", handled);
spin_unlock_irq(&conf->device_lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
pr_debug("--- raid5worker inactive\n");
}
@@ -5352,7 +5352,7 @@ static void raid5d(struct md_thread *thread)
spin_unlock_irq(&conf->device_lock);
async_tx_issue_pending_all();
- blk_finish_plug(&plug);
+ blk_finish_plug();
pr_debug("--- raid5d inactive\n");
}
diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c
index d4a4b0f..17d8730 100644
--- a/drivers/target/target_core_iblock.c
+++ b/drivers/target/target_core_iblock.c
@@ -367,7 +367,7 @@ static void iblock_submit_bios(struct bio_list *list, int rw)
blk_start_plug(&plug);
while ((bio = bio_list_pop(list)))
submit_bio(rw, bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static void iblock_end_io_flush(struct bio *bio, int err)
diff --git a/fs/aio.c b/fs/aio.c
index f8e52a1..b873698 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -1616,7 +1616,7 @@ long do_io_submit(aio_context_t ctx_id, long nr,
if (ret)
break;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
percpu_ref_put(&ctx->users);
return i ? i : ret;
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 975266b..f5848de 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -1609,7 +1609,7 @@ ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (err < 0)
ret = err;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
EXPORT_SYMBOL_GPL(blkdev_write_iter);
diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index ec57687..f314cfb8 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -3316,7 +3316,7 @@ out:
scrub_wr_submit(sctx);
mutex_unlock(&sctx->wr_ctx.wr_lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_free_path(path);
btrfs_free_path(ppath);
return ret < 0 ? ret : 0;
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 8be4278..fee10af 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -983,7 +983,7 @@ static int btrfs_write_and_wait_marked_extents(struct btrfs_root *root,
blk_start_plug(&plug);
ret = btrfs_write_marked_extents(root, dirty_pages, mark);
- blk_finish_plug(&plug);
+ blk_finish_plug();
ret2 = btrfs_wait_marked_extents(root, dirty_pages, mark);
if (ret)
diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index c5b8ba3..879c7fd 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -2574,7 +2574,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
blk_start_plug(&plug);
ret = btrfs_write_marked_extents(log, &log->dirty_log_pages, mark);
if (ret) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_abort_transaction(trans, root, ret);
btrfs_free_logged_extents(log, log_transid);
btrfs_set_log_full_commit(root->fs_info, trans);
@@ -2619,7 +2619,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
if (!list_empty(&root_log_ctx.list))
list_del_init(&root_log_ctx.list);
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_set_log_full_commit(root->fs_info, trans);
if (ret != -ENOSPC) {
@@ -2635,7 +2635,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
}
if (log_root_tree->log_transid_committed >= root_log_ctx.log_transid) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
mutex_unlock(&log_root_tree->log_mutex);
ret = root_log_ctx.log_ret;
goto out;
@@ -2643,7 +2643,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
index2 = root_log_ctx.log_transid % 2;
if (atomic_read(&log_root_tree->log_commit[index2])) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
ret = btrfs_wait_marked_extents(log, &log->dirty_log_pages,
mark);
btrfs_wait_logged_extents(trans, log, log_transid);
@@ -2669,7 +2669,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
* check the full commit flag again
*/
if (btrfs_need_log_full_commit(root->fs_info, trans)) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_wait_marked_extents(log, &log->dirty_log_pages, mark);
btrfs_free_logged_extents(log, log_transid);
mutex_unlock(&log_root_tree->log_mutex);
@@ -2680,7 +2680,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
ret = btrfs_write_marked_extents(log_root_tree,
&log_root_tree->dirty_log_pages,
EXTENT_DIRTY | EXTENT_NEW);
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (ret) {
btrfs_set_log_full_commit(root->fs_info, trans);
btrfs_abort_transaction(trans, root, ret);
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 8222f6f..16db068 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -358,7 +358,7 @@ loop_lock:
if (pending_bios == &device->pending_sync_bios) {
sync_pending = 1;
} else if (sync_pending) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
blk_start_plug(&plug);
sync_pending = 0;
}
@@ -415,7 +415,7 @@ loop_lock:
}
/* unplug every 64 requests just for good measure */
if (batch_run % 64 == 0) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
blk_start_plug(&plug);
sync_pending = 0;
}
@@ -431,7 +431,7 @@ loop_lock:
spin_unlock(&device->io_lock);
done:
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static void pending_bios_fn(struct btrfs_work *work)
diff --git a/fs/buffer.c b/fs/buffer.c
index 20805db..8181c44 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -758,7 +758,7 @@ static int fsync_buffers_list(spinlock_t *lock, struct list_head *list)
}
spin_unlock(lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
spin_lock(lock);
while (!list_empty(&tmp)) {
diff --git a/fs/direct-io.c b/fs/direct-io.c
index e181b6b..16f16ed 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -1262,7 +1262,7 @@ do_blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
if (sdio.bio)
dio_bio_submit(dio, &sdio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
/*
* It is possible that, we return short IO due to end of file.
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index 33a09da..3a293eb 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -183,7 +183,7 @@ ext4_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
ret = err;
}
if (o_direct)
- blk_finish_plug(&plug);
+ blk_finish_plug();
errout:
if (aio_mutex)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 5cb9a21..90ce0cb 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -2302,7 +2302,7 @@ static int ext4_writepages(struct address_space *mapping,
blk_start_plug(&plug);
ret = write_cache_pages(mapping, wbc, __writepage, mapping);
- blk_finish_plug(&plug);
+ blk_finish_plug();
goto out_writepages;
}
@@ -2438,7 +2438,7 @@ retry:
if (ret)
break;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (!ret && !cycled && wbc->nr_to_write > 0) {
cycled = 1;
mpd.last_page = writeback_index - 1;
diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 7f794b7..86ba453 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -846,7 +846,7 @@ retry_flush_nodes:
goto retry_flush_nodes;
}
out:
- blk_finish_plug(&plug);
+ blk_finish_plug();
return err;
}
diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
index 76adbc3..abeef77 100644
--- a/fs/f2fs/gc.c
+++ b/fs/f2fs/gc.c
@@ -678,7 +678,7 @@ static void do_garbage_collect(struct f2fs_sb_info *sbi, unsigned int segno,
gc_data_segment(sbi, sum->entries, gc_list, segno, gc_type);
break;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
stat_inc_seg_count(sbi, GET_SUM_TYPE((&sum->footer)));
stat_inc_call_count(sbi->stat_info);
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index 97bd9d3..c4aa9e2 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -1098,7 +1098,7 @@ repeat:
ra_node_page(sbi, nid);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
lock_page(page);
if (unlikely(page->mapping != NODE_MAPPING(sbi))) {
diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c
index 536e7a6..06f25d17 100644
--- a/fs/gfs2/log.c
+++ b/fs/gfs2/log.c
@@ -159,7 +159,7 @@ restart:
goto restart;
}
spin_unlock(&sdp->sd_ail_lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
trace_gfs2_ail_flush(sdp, wbc, 0);
}
diff --git a/fs/hpfs/buffer.c b/fs/hpfs/buffer.c
index 8057fe4..138462d 100644
--- a/fs/hpfs/buffer.c
+++ b/fs/hpfs/buffer.c
@@ -35,7 +35,7 @@ void hpfs_prefetch_sectors(struct super_block *s, unsigned secno, int n)
secno++;
n--;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/* Map a sector into a buffer and return pointers to it and to the buffer. */
diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
index 08c0304..cd6b09f 100644
--- a/fs/jbd/checkpoint.c
+++ b/fs/jbd/checkpoint.c
@@ -263,7 +263,7 @@ __flush_batch(journal_t *journal, struct buffer_head **bhs, int *batch_count)
blk_start_plug(&plug);
for (i = 0; i < *batch_count; i++)
write_dirty_buffer(bhs[i], WRITE_SYNC);
- blk_finish_plug(&plug);
+ blk_finish_plug();
for (i = 0; i < *batch_count; i++) {
struct buffer_head *bh = bhs[i];
diff --git a/fs/jbd/commit.c b/fs/jbd/commit.c
index bb217dc..e1046c3 100644
--- a/fs/jbd/commit.c
+++ b/fs/jbd/commit.c
@@ -447,7 +447,7 @@ void journal_commit_transaction(journal_t *journal)
blk_start_plug(&plug);
err = journal_submit_data_buffers(journal, commit_transaction,
write_op);
- blk_finish_plug(&plug);
+ blk_finish_plug();
/*
* Wait for all previously submitted IO to complete.
@@ -697,7 +697,7 @@ start_journal_io:
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
/* Lo and behold: we have just managed to send a transaction to
the log. Before we can commit it, wait for the IO so far to
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index 988b32e..6aa0039 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -187,7 +187,7 @@ __flush_batch(journal_t *journal, int *batch_count)
blk_start_plug(&plug);
for (i = 0; i < *batch_count; i++)
write_dirty_buffer(journal->j_chkpt_bhs[i], WRITE_SYNC);
- blk_finish_plug(&plug);
+ blk_finish_plug();
for (i = 0; i < *batch_count; i++) {
struct buffer_head *bh = journal->j_chkpt_bhs[i];
diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
index b73e021..8f532c8 100644
--- a/fs/jbd2/commit.c
+++ b/fs/jbd2/commit.c
@@ -805,7 +805,7 @@ start_journal_io:
__jbd2_journal_abort_hard(journal);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
/* Lo and behold: we have just managed to send a transaction to
the log. Before we can commit it, wait for the IO so far to
diff --git a/fs/mpage.c b/fs/mpage.c
index 3e79220..bf7d6c3 100644
--- a/fs/mpage.c
+++ b/fs/mpage.c
@@ -695,7 +695,7 @@ mpage_writepages(struct address_space *mapping,
if (mpd.bio)
mpage_bio_submit(WRITE, mpd.bio);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
EXPORT_SYMBOL(mpage_writepages);
diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c
index 1cac3c1..e93b6a8 100644
--- a/fs/nfs/blocklayout/blocklayout.c
+++ b/fs/nfs/blocklayout/blocklayout.c
@@ -311,7 +311,7 @@ bl_read_pagelist(struct nfs_pgio_header *header)
}
out:
bl_submit_bio(READ, bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
put_parallel(par);
return PNFS_ATTEMPTED;
}
@@ -433,7 +433,7 @@ bl_write_pagelist(struct nfs_pgio_header *header, int sync)
header->res.count = header->args.count;
out:
bl_submit_bio(WRITE, bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
put_parallel(par);
return PNFS_ATTEMPTED;
}
diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 1790b00..2f89ca2 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -1289,7 +1289,7 @@ _xfs_buf_ioapply(
if (size <= 0)
break; /* all done */
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
@@ -1823,7 +1823,7 @@ __xfs_buf_delwri_submit(
xfs_buf_submit(bp);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
return pinned;
}
diff --git a/fs/xfs/xfs_dir2_readdir.c b/fs/xfs/xfs_dir2_readdir.c
index 098cd78..7e8fa3f 100644
--- a/fs/xfs/xfs_dir2_readdir.c
+++ b/fs/xfs/xfs_dir2_readdir.c
@@ -455,7 +455,7 @@ xfs_dir2_leaf_readbuf(
}
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
out:
*bpp = bp;
diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
index 82e3142..c3ac5ec 100644
--- a/fs/xfs/xfs_itable.c
+++ b/fs/xfs/xfs_itable.c
@@ -196,7 +196,7 @@ xfs_bulkstat_ichunk_ra(
&xfs_inode_buf_ops);
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 7f9a516..188133f 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1091,6 +1091,7 @@ static inline void blk_post_runtime_resume(struct request_queue *q, int err) {}
* schedule() where blk_schedule_flush_plug() is called.
*/
struct blk_plug {
+ int depth; /* number of nested plugs */
struct list_head list; /* requests */
struct list_head mq_list; /* blk-mq requests */
struct list_head cb_list; /* md requires an unplug callback */
@@ -1107,7 +1108,7 @@ struct blk_plug_cb {
extern struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug,
void *data, int size);
extern void blk_start_plug(struct blk_plug *);
-extern void blk_finish_plug(struct blk_plug *);
+extern void blk_finish_plug(void);
extern void blk_flush_plug_list(struct blk_plug *, bool);
static inline void blk_flush_plug(struct task_struct *tsk)
@@ -1646,7 +1647,7 @@ static inline void blk_start_plug(struct blk_plug *plug)
{
}
-static inline void blk_finish_plug(struct blk_plug *plug)
+static inline void blk_finish_plug(void)
{
}
diff --git a/mm/madvise.c b/mm/madvise.c
index d551475..18a34ee 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -539,7 +539,7 @@ SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior)
vma = find_vma(current->mm, start);
}
out:
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (write)
up_write(¤t->mm->mmap_sem);
else
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 644bcb6..4570f6e 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2020,7 +2020,7 @@ int generic_writepages(struct address_space *mapping,
blk_start_plug(&plug);
ret = write_cache_pages(mapping, wbc, __writepage, mapping);
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
diff --git a/mm/readahead.c b/mm/readahead.c
index 9356758..64182a2 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -136,7 +136,7 @@ static int read_pages(struct address_space *mapping, struct file *filp,
ret = 0;
out:
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 405923f..5721f64 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -478,7 +478,7 @@ struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask,
SetPageReadahead(page);
page_cache_release(page);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
lru_add_drain(); /* Push any new pages onto the LRU now */
skip:
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 5e8eadd..56bb274 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2222,7 +2222,7 @@ static void shrink_lruvec(struct lruvec *lruvec, int swappiness,
scan_adjusted = true;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
sc->nr_reclaimed += nr_reclaimed;
/*
--
1.8.3.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 21:13 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AEtgtiN-WD-U8OG_nzsoRWQmrn6mxKMhKk6cxK4ZD5yaQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 8564 bytes --]
On Tue, 7 Apr 2015 10:02:13 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
> >
> > Try:
> > mdadm -S /dev/md124
> > mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> > mdadm -S /dev/md124
> > mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >
> > What does that report?
> >
> > NeilBrown
> >
>
> # mdadm --stop /dev/md124
> mdadm: stopped /dev/md124
> # mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> [linear] [multipath]
> md124 : inactive md126[0](S) md127[1](S)
> 3907022200 blocks super 1.2
>
> md0 : active raid1 sda5[0] sdb2[1]
> 107652416 blocks [2/2] [UU]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md125 : active raid1 sdh1[0] sdg1[1]
> 2930134016 blocks super 1.2 [2/2] [UU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdc1[0] sdd1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sde1[2] sdf1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
> # mdadm --stop /dev/md124
> mdadm: stopped /dev/md124
> # mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> mdadm: looking for devices for /dev/md124
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
> mdadm: :/dev/md124 has an active reshape - checking if critical
> section needs to be restored
> mdadm: added /dev/md/alpha to /dev/md124 as 1
> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
> mdadm: no uptodate device for slot 6 of /dev/md124
> mdadm: added /dev/md/beta to /dev/md124 as 0
> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> [linear] [multipath]
> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
> 6837155192 blocks super 1.2
>
> md0 : active raid1 sda5[0] sdb2[1]
> 107652416 blocks [2/2] [UU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> md125 : active raid1 sdh1[0] sdg1[1]
> 2930134016 blocks super 1.2 [2/2] [UU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdc1[0] sdd1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sde1[2] sdf1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
>
> # mdadm --examine /dev/md/alpha
> /dev/md/alpha:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 4
>
> Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=752 sectors
> State : active
> Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : 1 (3->4)
>
> Update Time : Sun Mar 29 15:11:35 2015
> Checksum : 8be5e0e8 - correct
> Events : 14013
>
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>
> # mdadm --examine /dev/md/beta
> /dev/md/beta:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 4
>
> Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=752 sectors
> State : clean
> Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : 1 (3->4)
>
> Update Time : Sun Mar 29 15:11:35 2015
> Checksum : f7526adf - correct
> Events : 14013
>
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>
> # mdadm --examine /dev/md/gamma
> /dev/md/gamma:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x6
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 4
>
> Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Recovery Offset : 86403072 sectors
> Unused Space : before=1960 sectors, after=1953244160 sectors
> State : active
> Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : 1 (3->4)
>
> Update Time : Sun Mar 29 00:05:29 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 710dc078 - correct
> Events : 673
>
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> # mdadm --detail /dev/md124
> /dev/md124:
> Version : 1.2
> Raid Level : raid0
> Total Devices : 3
> Persistence : Superblock is persistent
>
> State : inactive
>
> Delta Devices : 1, (-1->0)
> New Level : raid4
> New Chunksize : 512K
>
> Name : hordern:hordern1 (local to host hordern)
> UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Events : 673
>
> Number Major Minor RaidDevice
>
> - 9 125 - /dev/md/gamma
> - 9 126 - /dev/md/beta
> - 9 127 - /dev/md/alpha
>
> So it looks like all three component devices have consistent
> superblocks now, awesome! But the raid0 array is still inactive with
> all three components listed as spares. It looks like /dev/md/gamma has
> a much lower event count, I'm guessing that is what causes the disk to
> be marked as possibly out of date.
>
> Is an "uptodate device" a specific thing, or does that simply mean
> that some component devices are out of date? The lack of spaces makes
> me think that uptodate is some keyword I'm not recognizing.
>
Looks good. Nearly there.
The difference in event counts is probably due to you trying lots of things
out, and them only affecting two devices.
If you
# mdadm --stop /dev/md124
# mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
i.e. just add --force, it should ignored the difference in event count and
assemble the array.
For RAID0, the event count isn't really relevant to the data as there is no
possibility for inconsistency between data and parity on different devices.
As the reshape position is the same on all devices, I don't think there is
any risk at all in just using --force.
Of course, perform an fsck afterwards just to build confidence.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Creating volume in IMSM container
From: NeilBrown @ 2015-04-07 21:28 UTC (permalink / raw)
To: Ian Pilcher; +Cc: linux-raid
In-Reply-To: <mg0t3u$ol$1@ger.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]
On Tue, 07 Apr 2015 10:29:02 -0500 Ian Pilcher <arequipeno@gmail.com> wrote:
> I am trying to create an IMSM container and volume in a KVM virtual
> machine (so that I can create a reproducer for a Fedora problem that
> seems to only/mostly affect IMSM RAID). I've managed to create a
> container with the following command:
>
> IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/imsm \
> /dev/vd{a,b} --raid-devices 2 --metadata=imsm
>
> /proc/mdstat now contains:
>
> Personalities: [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] ...
> md127 : inactive vdb[1](S) vda[0](S)
> 2210 blocks super external:imsm
2210 blocks is tiny.
I think each device needs at least 6306 sectors for IMSM.
I suggest going for at least 20,000.
>
> I haven't, however, been able to create a volume within the container:
>
> # IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
> /dev/md/imsm --raid-devices 2 --level 1 --metadata=imsm
> mdmon: not enough devices with space to create array.
This should work once you have devices with enough space.
>
> # IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
> /dev/vd{a,b} --raid-devices 2 --level 1 --metadata=imsm
> mdadm: /dev/vda appears to be part of a raid array:
> level=container devices=0 ctime=Thu Jan 1 00:00:00 1970
> mdmon: /dev/vdb is not in the same imsm set
> mdadm: /dev/vdb is not suitable for this array.
> mdadm: create aborted
>
This looks like a bug. What version of mdadm.
NeilBrown
> Am I doing something wrong, or is this simply not possible?
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-07 22:31 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150408071339.3295567b@notabene.brown>
On Tue, Apr 7, 2015 at 2:13 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 7 Apr 2015 10:02:13 -0700 "Jonathan Harker (Jesusaurus)"
> <jesusaurus@gentlydownthe.net> wrote:
>
>> On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
>> >
>> > Try:
>> > mdadm -S /dev/md124
>> > mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
>> > mdadm -S /dev/md124
>> > mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>> >
>> > What does that report?
>> >
>> > NeilBrown
>> >
>>
>> # mdadm --stop /dev/md124
>> mdadm: stopped /dev/md124
>> # mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
>> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
>> # cat /proc/mdstat
>> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
>> [linear] [multipath]
>> md124 : inactive md126[0](S) md127[1](S)
>> 3907022200 blocks super 1.2
>>
>> md0 : active raid1 sda5[0] sdb2[1]
>> 107652416 blocks [2/2] [UU]
>> bitmap: 1/1 pages [4KB], 65536KB chunk
>>
>> md125 : active raid1 sdh1[0] sdg1[1]
>> 2930134016 blocks super 1.2 [2/2] [UU]
>> bitmap: 0/22 pages [0KB], 65536KB chunk
>>
>> md126 : active raid1 sdc1[0] sdd1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> md127 : active raid1 sde1[2] sdf1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> unused devices: <none>
>> # mdadm --stop /dev/md124
>> mdadm: stopped /dev/md124
>> # mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>> mdadm: looking for devices for /dev/md124
>> mdadm: UUID differs from /dev/md0.
>> mdadm: UUID differs from /dev/md/alpha.
>> mdadm: UUID differs from /dev/md/beta.
>> mdadm: UUID differs from /dev/md/gamma.
>> mdadm: UUID differs from /dev/md0.
>> mdadm: UUID differs from /dev/md/alpha.
>> mdadm: UUID differs from /dev/md/beta.
>> mdadm: UUID differs from /dev/md/gamma.
>> mdadm: UUID differs from /dev/md0.
>> mdadm: UUID differs from /dev/md/alpha.
>> mdadm: UUID differs from /dev/md/beta.
>> mdadm: UUID differs from /dev/md/gamma.
>> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
>> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
>> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
>> mdadm: :/dev/md124 has an active reshape - checking if critical
>> section needs to be restored
>> mdadm: added /dev/md/alpha to /dev/md124 as 1
>> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
>> mdadm: no uptodate device for slot 6 of /dev/md124
>> mdadm: added /dev/md/beta to /dev/md124 as 0
>> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
>> # cat /proc/mdstat
>> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
>> [linear] [multipath]
>> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
>> 6837155192 blocks super 1.2
>>
>> md0 : active raid1 sda5[0] sdb2[1]
>> 107652416 blocks [2/2] [UU]
>> bitmap: 0/1 pages [0KB], 65536KB chunk
>>
>> md125 : active raid1 sdh1[0] sdg1[1]
>> 2930134016 blocks super 1.2 [2/2] [UU]
>> bitmap: 0/22 pages [0KB], 65536KB chunk
>>
>> md126 : active raid1 sdc1[0] sdd1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> md127 : active raid1 sde1[2] sdf1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> unused devices: <none>
>>
>> # mdadm --examine /dev/md/alpha
>> /dev/md/alpha:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Name : hordern:hordern1 (local to host hordern)
>> Creation Time : Fri Jan 2 09:59:40 2009
>> Raid Level : raid4
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=752 sectors
>> State : active
>> Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
>>
>> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
>> Delta Devices : 1 (3->4)
>>
>> Update Time : Sun Mar 29 15:11:35 2015
>> Checksum : 8be5e0e8 - correct
>> Events : 14013
>>
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> # mdadm --examine /dev/md/beta
>> /dev/md/beta:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Name : hordern:hordern1 (local to host hordern)
>> Creation Time : Fri Jan 2 09:59:40 2009
>> Raid Level : raid4
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=752 sectors
>> State : clean
>> Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
>>
>> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
>> Delta Devices : 1 (3->4)
>>
>> Update Time : Sun Mar 29 15:11:35 2015
>> Checksum : f7526adf - correct
>> Events : 14013
>>
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> # mdadm --examine /dev/md/gamma
>> /dev/md/gamma:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x6
>> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Name : hordern:hordern1 (local to host hordern)
>> Creation Time : Fri Jan 2 09:59:40 2009
>> Raid Level : raid4
>> Raid Devices : 4
>>
>> Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
>> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Recovery Offset : 86403072 sectors
>> Unused Space : before=1960 sectors, after=1953244160 sectors
>> State : active
>> Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
>>
>> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
>> Delta Devices : 1 (3->4)
>>
>> Update Time : Sun Mar 29 00:05:29 2015
>> Bad Block Log : 512 entries available at offset 72 sectors
>> Checksum : 710dc078 - correct
>> Events : 673
>>
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> # mdadm --detail /dev/md124
>> /dev/md124:
>> Version : 1.2
>> Raid Level : raid0
>> Total Devices : 3
>> Persistence : Superblock is persistent
>>
>> State : inactive
>>
>> Delta Devices : 1, (-1->0)
>> New Level : raid4
>> New Chunksize : 512K
>>
>> Name : hordern:hordern1 (local to host hordern)
>> UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Events : 673
>>
>> Number Major Minor RaidDevice
>>
>> - 9 125 - /dev/md/gamma
>> - 9 126 - /dev/md/beta
>> - 9 127 - /dev/md/alpha
>>
>> So it looks like all three component devices have consistent
>> superblocks now, awesome! But the raid0 array is still inactive with
>> all three components listed as spares. It looks like /dev/md/gamma has
>> a much lower event count, I'm guessing that is what causes the disk to
>> be marked as possibly out of date.
>>
>> Is an "uptodate device" a specific thing, or does that simply mean
>> that some component devices are out of date? The lack of spaces makes
>> me think that uptodate is some keyword I'm not recognizing.
>>
>
> Looks good. Nearly there.
>
> The difference in event counts is probably due to you trying lots of things
> out, and them only affecting two devices.
>
> If you
> # mdadm --stop /dev/md124
> # mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>
> i.e. just add --force, it should ignored the difference in event count and
> assemble the array.
> For RAID0, the event count isn't really relevant to the data as there is no
> possibility for inconsistency between data and parity on different devices.
> As the reshape position is the same on all devices, I don't think there is
> any risk at all in just using --force.
> Of course, perform an fsck afterwards just to build confidence.
>
> NeilBrown
>
Unfortunately, adding --force didn't seem to make any difference:
# mdadm --stop /dev/md124
mdadm: stopped /dev/md124
# mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
mdadm: looking for devices for /dev/md124
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
mdadm: :/dev/md124 has an active reshape - checking if critical
section needs to be restored
mdadm: added /dev/md/alpha to /dev/md124 as 1
mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
mdadm: no uptodate device for slot 6 of /dev/md124
mdadm: added /dev/md/beta to /dev/md124 as 0
mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
[linear] [multipath]
md124 : inactive md125[3](S) md127[1](S) md126[0](S)
6837155192 blocks super 1.2
md0 : active raid1 sda5[0] sdb2[1]
107652416 blocks [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md125 : active raid1 sdh1[0] sdg1[1]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md126 : active raid1 sdc1[0] sdd1[1]
1953512312 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[2] sdf1[1]
1953512312 blocks super 1.2 [2/2] [UU]
unused devices: <none>
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 22:56 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AGk2MK8=qy2CL-WN-ewfVzgA2D-WRkefpriXrdHSEjU-Q@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 12018 bytes --]
On Tue, 7 Apr 2015 15:31:32 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> On Tue, Apr 7, 2015 at 2:13 PM, NeilBrown <neilb@suse.de> wrote:
> > On Tue, 7 Apr 2015 10:02:13 -0700 "Jonathan Harker (Jesusaurus)"
> > <jesusaurus@gentlydownthe.net> wrote:
> >
> >> On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
> >> >
> >> > Try:
> >> > mdadm -S /dev/md124
> >> > mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> >> > mdadm -S /dev/md124
> >> > mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >> >
> >> > What does that report?
> >> >
> >> > NeilBrown
> >> >
> >>
> >> # mdadm --stop /dev/md124
> >> mdadm: stopped /dev/md124
> >> # mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> >> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> >> # cat /proc/mdstat
> >> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> >> [linear] [multipath]
> >> md124 : inactive md126[0](S) md127[1](S)
> >> 3907022200 blocks super 1.2
> >>
> >> md0 : active raid1 sda5[0] sdb2[1]
> >> 107652416 blocks [2/2] [UU]
> >> bitmap: 1/1 pages [4KB], 65536KB chunk
> >>
> >> md125 : active raid1 sdh1[0] sdg1[1]
> >> 2930134016 blocks super 1.2 [2/2] [UU]
> >> bitmap: 0/22 pages [0KB], 65536KB chunk
> >>
> >> md126 : active raid1 sdc1[0] sdd1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> md127 : active raid1 sde1[2] sdf1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> unused devices: <none>
> >> # mdadm --stop /dev/md124
> >> mdadm: stopped /dev/md124
> >> # mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >> mdadm: looking for devices for /dev/md124
> >> mdadm: UUID differs from /dev/md0.
> >> mdadm: UUID differs from /dev/md/alpha.
> >> mdadm: UUID differs from /dev/md/beta.
> >> mdadm: UUID differs from /dev/md/gamma.
> >> mdadm: UUID differs from /dev/md0.
> >> mdadm: UUID differs from /dev/md/alpha.
> >> mdadm: UUID differs from /dev/md/beta.
> >> mdadm: UUID differs from /dev/md/gamma.
> >> mdadm: UUID differs from /dev/md0.
> >> mdadm: UUID differs from /dev/md/alpha.
> >> mdadm: UUID differs from /dev/md/beta.
> >> mdadm: UUID differs from /dev/md/gamma.
> >> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
> >> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
> >> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
> >> mdadm: :/dev/md124 has an active reshape - checking if critical
> >> section needs to be restored
> >> mdadm: added /dev/md/alpha to /dev/md124 as 1
> >> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
> >> mdadm: no uptodate device for slot 6 of /dev/md124
> >> mdadm: added /dev/md/beta to /dev/md124 as 0
> >> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> >> # cat /proc/mdstat
> >> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> >> [linear] [multipath]
> >> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
> >> 6837155192 blocks super 1.2
> >>
> >> md0 : active raid1 sda5[0] sdb2[1]
> >> 107652416 blocks [2/2] [UU]
> >> bitmap: 0/1 pages [0KB], 65536KB chunk
> >>
> >> md125 : active raid1 sdh1[0] sdg1[1]
> >> 2930134016 blocks super 1.2 [2/2] [UU]
> >> bitmap: 0/22 pages [0KB], 65536KB chunk
> >>
> >> md126 : active raid1 sdc1[0] sdd1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> md127 : active raid1 sde1[2] sdf1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> unused devices: <none>
> >>
> >> # mdadm --examine /dev/md/alpha
> >> /dev/md/alpha:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x4
> >> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Name : hordern:hordern1 (local to host hordern)
> >> Creation Time : Fri Jan 2 09:59:40 2009
> >> Raid Level : raid4
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> >> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> Unused Space : before=1968 sectors, after=752 sectors
> >> State : active
> >> Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
> >>
> >> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> >> Delta Devices : 1 (3->4)
> >>
> >> Update Time : Sun Mar 29 15:11:35 2015
> >> Checksum : 8be5e0e8 - correct
> >> Events : 14013
> >>
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 1
> >> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> >>
> >> # mdadm --examine /dev/md/beta
> >> /dev/md/beta:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x4
> >> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Name : hordern:hordern1 (local to host hordern)
> >> Creation Time : Fri Jan 2 09:59:40 2009
> >> Raid Level : raid4
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
> >> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> >> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> Unused Space : before=1968 sectors, after=752 sectors
> >> State : clean
> >> Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
> >>
> >> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> >> Delta Devices : 1 (3->4)
> >>
> >> Update Time : Sun Mar 29 15:11:35 2015
> >> Checksum : f7526adf - correct
> >> Events : 14013
> >>
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 0
> >> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> >>
> >> # mdadm --examine /dev/md/gamma
> >> /dev/md/gamma:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x6
> >> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Name : hordern:hordern1 (local to host hordern)
> >> Creation Time : Fri Jan 2 09:59:40 2009
> >> Raid Level : raid4
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
> >> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> >> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> Recovery Offset : 86403072 sectors
> >> Unused Space : before=1960 sectors, after=1953244160 sectors
> >> State : active
> >> Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
> >>
> >> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> >> Delta Devices : 1 (3->4)
> >>
> >> Update Time : Sun Mar 29 00:05:29 2015
> >> Bad Block Log : 512 entries available at offset 72 sectors
> >> Checksum : 710dc078 - correct
> >> Events : 673
> >>
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 2
> >> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
> >>
> >> # mdadm --detail /dev/md124
> >> /dev/md124:
> >> Version : 1.2
> >> Raid Level : raid0
> >> Total Devices : 3
> >> Persistence : Superblock is persistent
> >>
> >> State : inactive
> >>
> >> Delta Devices : 1, (-1->0)
> >> New Level : raid4
> >> New Chunksize : 512K
> >>
> >> Name : hordern:hordern1 (local to host hordern)
> >> UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Events : 673
> >>
> >> Number Major Minor RaidDevice
> >>
> >> - 9 125 - /dev/md/gamma
> >> - 9 126 - /dev/md/beta
> >> - 9 127 - /dev/md/alpha
> >>
> >> So it looks like all three component devices have consistent
> >> superblocks now, awesome! But the raid0 array is still inactive with
> >> all three components listed as spares. It looks like /dev/md/gamma has
> >> a much lower event count, I'm guessing that is what causes the disk to
> >> be marked as possibly out of date.
> >>
> >> Is an "uptodate device" a specific thing, or does that simply mean
> >> that some component devices are out of date? The lack of spaces makes
> >> me think that uptodate is some keyword I'm not recognizing.
> >>
> >
> > Looks good. Nearly there.
> >
> > The difference in event counts is probably due to you trying lots of things
> > out, and them only affecting two devices.
> >
> > If you
> > # mdadm --stop /dev/md124
> > # mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >
> > i.e. just add --force, it should ignored the difference in event count and
> > assemble the array.
> > For RAID0, the event count isn't really relevant to the data as there is no
> > possibility for inconsistency between data and parity on different devices.
> > As the reshape position is the same on all devices, I don't think there is
> > any risk at all in just using --force.
> > Of course, perform an fsck afterwards just to build confidence.
> >
> > NeilBrown
> >
>
> Unfortunately, adding --force didn't seem to make any difference:
>
> # mdadm --stop /dev/md124
> mdadm: stopped /dev/md124
> # mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> mdadm: looking for devices for /dev/md124
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
> mdadm: :/dev/md124 has an active reshape - checking if critical
> section needs to be restored
> mdadm: added /dev/md/alpha to /dev/md124 as 1
> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
> mdadm: no uptodate device for slot 6 of /dev/md124
> mdadm: added /dev/md/beta to /dev/md124 as 0
> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> [linear] [multipath]
> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
> 6837155192 blocks super 1.2
>
> md0 : active raid1 sda5[0] sdb2[1]
> 107652416 blocks [2/2] [UU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> md125 : active raid1 sdh1[0] sdg1[1]
> 2930134016 blocks super 1.2 [2/2] [UU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdc1[0] sdd1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sde1[2] sdf1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
Hmm... I think I see the bug. It should be easy enough to fix, but I'd like
to be able to test it.
Could you please:
mkdir /tmp/md.metadata
mdadm --dump /tmp/md.metadata /dev/md/alpha /dev/md/beta /dev/md/gamma
tar czSf /tmp/md.tgz /tmp/md.metadata
and then send me /tmp/md.tgz, which should be tiny and contain just the
metadata from the array.
[[the patch which introduced the problem has a description which starts
"This is a bit of a hack and ..."
Never accept hacks!
]]
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox