* Re: clustered MD - beyond RAID1
From: NeilBrown @ 2015-12-21 20:47 UTC (permalink / raw)
To: Tejas Rao, Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <567850C4.30108@bnl.gov>
[-- Attachment #1: Type: text/plain, Size: 646 bytes --]
On Tue, Dec 22 2015, Tejas Rao wrote:
> What if the application is doing the locking and making sure that only 1
> node writes to a md device at a time? Will this work? How are rebuilds
> handled? This would be helpful with distributed filesystems like
> GPFS/lustre etc.
>
You would also need to make sure that the filesystem only wrote from a
single node at a time (or access the block device directly). I doubt
GPFS/lustre make any promise like that, but I'm happy to be educated.
rebuilds are handled by using a cluster-wide lock to block all writes to
a range of addresses while those stripes are repaired.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* [PATCH 3/3] Add new journal to array that does not have journal
From: Song Liu @ 2015-12-21 19:23 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1450725823-1832511-1-git-send-email-songliubraving@fb.com>
This patch enables adding journal to an array that does not have journal before.
To add the journal, the array should finish resync. Otherwise, mdadm complains:
[root] mdadm --add-journal /dev/md0 /dev/sdb8 -vvv
mdadm: /dev/md0 has active resync, please retry after resync is done.
The array need to be restarted for the journal to work:
[root] mdadm --add-journal /dev/md0 /dev/sdb8
mdadm: Making /dev/md0 readonly before adding journal...
mdadm: Added new journal to /dev/md0.
mdadm: Please restart the array to make it live.
[root] mdadm --stop /dev/md*
mdadm: stopped /dev/md0
[root] mdadm -A /dev/md0 /dev/sdb[12358]
mdadm: device 8 in /dev/md0 has wrong state in superblock, but /dev/sdb8 seems ok
mdadm: /dev/md0 has been started with 4 drives and 1 journal.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
Assemble.c | 5 +----
Manage.c | 32 ++++++++++++++++++++++++++------
2 files changed, 27 insertions(+), 10 deletions(-)
diff --git a/Assemble.c b/Assemble.c
index a7cd163..4ddd650 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -1556,10 +1556,7 @@ try_again:
*/
if (content->array.level != LEVEL_MULTIPATH) {
if (devices[j].i.disk.state & (1<<MD_DISK_JOURNAL)) {
- if (content->journal_device_required)
- journalcnt++;
- else /* unexpected journal, mark as faulty */
- devices[j].i.disk.state |= (1<<MD_DISK_FAULTY);
+ journalcnt++;
} else if (!(devices[j].i.disk.state & (1<<MD_DISK_ACTIVE))) {
if (!(devices[j].i.disk.state
& (1<<MD_DISK_FAULTY))) {
diff --git a/Manage.c b/Manage.c
index 7e1b94b..da14b99 100644
--- a/Manage.c
+++ b/Manage.c
@@ -740,6 +740,7 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
struct supertype *dev_st = NULL;
int j;
mdu_disk_info_t disc;
+ int new_journal = 0;
if (!get_dev_size(tfd, dv->devname, &ldsize)) {
if (dv->disposition == 'M')
@@ -935,19 +936,33 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
if (dv->disposition == 'j') {
struct mdinfo mdi;
struct mdinfo *mdp;
+ struct mdstat_ent *mds, *m;
+ int percent = -1;
+
+ mds = mdstat_read(0, 0);
+ for (m = mds; m; m = m->next)
+ if (strcmp(m->devnm, fd2devnm(fd)) == 0)
+ percent = m->percent;
+ free_mdstat(mds);
+
+ if (percent > 0) {
+ pr_err("%s has active resync, please retry after resync is done.\n", devname);
+ return -1;
+ }
mdp = sysfs_read(fd, NULL, GET_ARRAY_STATE);
if (strncmp(mdp->sysfs_array_state, "readonly", 8) != 0) {
- pr_err("%s is not readonly, cannot add journal.\n", devname);
- return -1;
+ pr_err("Making %s readonly before adding journal...\n", devname);
+ if (Manage_ro(devname, fd, 1)) {
+ pr_err("Please retry.\n");
+ return -1;
+ }
}
tst->ss->getinfo_super(tst, &mdi, NULL);
- if (mdi.journal_device_required == 0) {
- pr_err("%s does not support journal device.\n", devname);
- return -1;
- }
+ if (mdi.journal_device_required == 0)
+ new_journal = 1;
disc.raid_disk = 0;
}
@@ -1064,6 +1079,11 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
close(container_fd);
} else {
tst->ss->free_super(tst);
+ if (new_journal) {
+ pr_err("Added new journal to %s. \n", devname);
+ pr_err("Please restart the array to make it live.\n");
+ return 1;
+ }
if (ioctl(fd, ADD_NEW_DISK, &disc)) {
if (dv->disposition == 'j')
pr_err("Failed to hot add %s as journal, "
--
2.4.6
^ permalink raw reply related
* [PATCH 2/3] [mdadm] in --add assign raid_disk of 0 to journal
From: Song Liu @ 2015-12-21 19:23 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1450725823-1832511-1-git-send-email-songliubraving@fb.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
Manage.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Manage.c b/Manage.c
index 4540fac..7e1b94b 100644
--- a/Manage.c
+++ b/Manage.c
@@ -948,7 +948,7 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
pr_err("%s does not support journal device.\n", devname);
return -1;
}
- disc.raid_disk = array->raid_disks;
+ disc.raid_disk = 0;
}
if (array->not_persistent==0) {
--
2.4.6
^ permalink raw reply related
* [PATCH 1/3] [mdadm] move journal to end of --detail list
From: Song Liu @ 2015-12-21 19:23 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1450725823-1832511-1-git-send-email-songliubraving@fb.com>
As we give journal device raid_disk of 0, the output of --detail is:
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
5 8 24 0 journal /dev/sdb8
1 8 18 1 active sync /dev/sdb2
2 8 19 2 active sync /dev/sdb3
3 8 21 3 active sync /dev/sdb5
4 8 23 - spare /dev/sdb7
This patch makes it back to:
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 19 2 active sync /dev/sdb3
3 8 21 3 active sync /dev/sdb5
4 8 23 - spare /dev/sdb7
5 8 24 - journal /dev/sdb8
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
Detail.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/Detail.c b/Detail.c
index 5bd2dc6..37403d6 100644
--- a/Detail.c
+++ b/Detail.c
@@ -326,7 +326,8 @@ int Detail(char *dev, struct context *c)
&& disks[disk.raid_disk*2].state == (1<<MD_DISK_REMOVED))
disks[disk.raid_disk*2] = disk;
else if (disk.raid_disk >= 0 && disk.raid_disk < array.raid_disks
- && disks[disk.raid_disk*2+1].state == (1<<MD_DISK_REMOVED))
+ && disks[disk.raid_disk*2+1].state == (1<<MD_DISK_REMOVED)
+ && !(disk.state & (1<<MD_DISK_JOURNAL)))
disks[disk.raid_disk*2+1] = disk;
else if (next < max_disks*2)
disks[next++] = disk;
@@ -621,7 +622,7 @@ This is pretty boring
if (disk.number < 0)
printf(" - %5d %5d - ",
disk.major, disk.minor);
- else if (disk.raid_disk < 0)
+ else if (disk.raid_disk < 0 || disk.state & (1<<MD_DISK_JOURNAL))
printf(" %5d %5d %5d - ",
disk.number, disk.major, disk.minor);
else if (disk.number < 0)
--
2.4.6
^ permalink raw reply related
* [PATCH 0/3] [mdadm] fixes and feature for journal device
From: Song Liu @ 2015-12-21 19:23 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
As we give journal device disk role of 0
(http://marc.info/?l=linux-raid&m=145030044008753), we need 2 fixes
in mdadm (patch 1 and 2).
Patch 3 tries to add journal to an existing array that does not have
journal before.
Song Liu (3):
[mdadm] move journal to end of --detail list
[mdadm] in --add assign raid_disk of 0 to journal
Add new journal to array that does not have journal
Assemble.c | 5 +----
Detail.c | 5 +++--
Manage.c | 34 +++++++++++++++++++++++++++-------
3 files changed, 31 insertions(+), 13 deletions(-)
--
2.4.6
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: Tejas Rao @ 2015-12-21 19:19 UTC (permalink / raw)
To: NeilBrown, Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <87si2w66tm.fsf@notabene.neil.brown.name>
What if the application is doing the locking and making sure that only 1
node writes to a md device at a time? Will this work? How are rebuilds
handled? This would be helpful with distributed filesystems like
GPFS/lustre etc.
Tejas.
On 12/20/2015 18:25, NeilBrown wrote:
> On Sat, Dec 19 2015, Scott Sinno wrote:
>
>> Neil(or anyone well informed in mdadm development roadmaps),
>>
>> Aaron and myself are engineers at NASA Goddard with strong interest in
>> MDADM. We currently host 6PB(raw) of live JBOD storage leveraging MDADM
>> exclusively for RAID functionality.
>>
>> We're very interested in Clustered MDADM to improve data-availability
>> in the environment, but note that only RAID1 is currently supported.
>> Are there plans in the nearish-term(say over the next year) to expound
>> clustered bitmap functionality to RAID5/6, or anything else you can
>> divulge on that front? Thanks in advance for any guidance.
> We don't talk about plans that are not backed by code - you can't trust
> them.
>
> However I cannot imagine how you could make RAID5 work efficiently in a
> cluster.
> RAID1 works because we assume that the file system will have its own
> locking to ensure that only one node writes to a given block at a given
> time. So while node-A is writing to a block, RAID1 knows that no other
> node is writing there so it can update all copies and be sure no race
> will result in the copies being inconsistent.
>
> For this to work with RAID5 we would need to assume the filesystem will
> ensure only one node is writing to a given stripe at a time, and that is
> not realistic.
>
> So to make it work we would need the md layer to lock each stripe during
> an update. I have trouble imagining that running with much speed. Hard
> to know without testing of course.
> I know of no-one with plans to do that testing.
>
> NeilBrown
^ permalink raw reply
* Re: WANTED new maintainer for Linux/md (and mdadm)
From: Phil Turmel @ 2015-12-21 12:28 UTC (permalink / raw)
To: NeilBrown, linux-raid, dm-devel, LKML; +Cc: Linus Torvalds, J. Bruce Fields
In-Reply-To: <87k2o849j5.fsf@notabene.neil.brown.name>
On 12/21/2015 01:10 AM, NeilBrown wrote:
> In recent times I've been doing less of this and have been absolutely
> thrilled that the gap has been more than filled by other very competent
> community members. Not developers particular but a number of md users
> have been providing excellent support. I'd particularly like to
> high-light Phil Turmel who is very forthcoming with excellent advice,
> but he is certainly not the only one who deserves a lot of thanks.
> So "Thank you" to everyone who answers questions on linux-raid.
You are very welcome. I may just have to polish my kernel coding skills
to earn a place in a maintainership team.
Phil Turmel
^ permalink raw reply
* Re: Reconstruct a RAID 6 that has failed in a non typical manner
From: Phil Turmel @ 2015-12-21 12:20 UTC (permalink / raw)
To: NeilBrown, Clement Parisot; +Cc: linux-raid
In-Reply-To: <87vb7s4gfv.fsf@notabene.neil.brown.name>
Good morning Neil,
On 12/20/2015 10:40 PM, NeilBrown wrote:
> On Fri, Nov 06 2015, Phil Turmel wrote:
>>
>> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
>>
>
> Would it make sense for mdadm to automagically do something like this?
> i.e. whenever it adds a device to an array (with redundancy) it write
> 180 (or something configurable) to the 'timeout' file if there is one?
Yes, I've been thinking this should be automagic, but I'm not sure if it
really belongs at the MD layer.
> Why do we pick 180?
I empirically determined that 120 was sufficient on the Seagate drives
that kicked my tail when I first figured this out. Someone else (I'm
afraid I don't remember) found that to be not quite enough and suggested
180.
> Can this cause problems on some drives?
Not that I'm aware of, but it does make for rather troublesome
*application* stalls.
Considering that this aggressively long error recovery behavior is
*intended* for desktop drives or any non-redundant usage, I believe
linux shouldn't time out at 30 seconds by default. It cuts off any
opportunity for these drives to report a good sector that is
reconstructed in more than 30 seconds.
Meanwhile, any device that *does* support scterc and/or has scterc
enabled out of the gate arguably should have a timeout just a few
seconds longer than the larger of the two error recovery settings.
I propose:
1) The kernel default timeout be set to 180 (or some number
cooperatively established with the drive manufacturers.)
2) the initial probe sequence that retrieves the drive's parameter pages
also pick up the SCT page and if ERC is enabled, adjust the timeout
downward. I believe these capabilities should be reflected in sysfs for
use by udev.
3) mdadm should inspect member device ERC capabilities during creation
and assembly and enable it for drives that have it available but disabled.
In light of your maintainership notice, I will pursue this directly.
Phil
^ permalink raw reply
* [MDADM PATCH] Check and remove bitmap first when reshape to raid0
From: Xiao Ni @ 2015-12-21 8:23 UTC (permalink / raw)
To: neilb; +Cc: Jes.Sorensen, linux-raid
If reshape one raid device with bitmap to raid0, the reshape progress will
start. But it'll fail and lose some components. So it should remove bitmap
first.
And it shouldn't close fd when try to call open_dev_excl. Because it's an
input argument. It should be closed in the function who open it.
There are some places that return(1) directly but don't free subarray and
close cfd in Grow_reshape. So add fallback: to do free(subarray) and
close(cfd).
Signed-off-by: Xiao Ni <xni@redhat.com>
---
Grow.c | 37 ++++++++++++++++++++++++++++---------
1 file changed, 28 insertions(+), 9 deletions(-)
diff --git a/Grow.c b/Grow.c
index 6dfb9c9..f892023 100755
--- a/Grow.c
+++ b/Grow.c
@@ -1591,6 +1591,17 @@ int Grow_reshape(char *devname, int fd,
return 1;
}
+ if (s->level == 0 && array.state & (1<<MD_SB_BITMAP_PRESENT)) {
+ array.state &= ~(1<<MD_SB_BITMAP_PRESENT);
+ if (ioctl(fd, SET_ARRAY_INFO, &array)!= 0) {
+ if (array.state & (1<<MD_SB_CLUSTERED))
+ pr_err("failed to remove clustered bitmap.\n");
+ else
+ pr_err("failed to remove internal bitmap.\n");
+ return 1;
+ }
+ }
+
/* in the external case we need to check that the requested reshape is
* supported, and perform an initial check that the container holds the
* pre-requisite spare devices (mdmon owns final validation)
@@ -1603,7 +1614,6 @@ int Grow_reshape(char *devname, int fd,
cfd = open_dev_excl(st->container_devnm);
} else {
container = st->devnm;
- close(fd);
cfd = open_dev_excl(st->devnm);
fd = cfd;
}
@@ -1619,8 +1629,8 @@ int Grow_reshape(char *devname, int fd,
if (rv) {
pr_err("Cannot read superblock for %s\n",
devname);
- free(subarray);
- return 1;
+ rv = 1;
+ goto fallback;
}
/* check if operation is supported for metadata handler */
@@ -1644,8 +1654,8 @@ int Grow_reshape(char *devname, int fd,
pr_err("cannot reshape arrays in container with unsupported metadata: %s(%s)\n",
devname, container);
sysfs_free(cc);
- free(subarray);
- return 1;
+ rv = 1;
+ goto fallback;
}
}
sysfs_free(cc);
@@ -1665,7 +1675,8 @@ int Grow_reshape(char *devname, int fd,
s->raiddisks - array.raid_disks,
s->raiddisks - array.raid_disks == 1 ? "" : "s",
array.spare_disks + added_disks);
- return 1;
+ rv = 1;
+ goto fallback;
}
sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
@@ -1678,17 +1689,20 @@ int Grow_reshape(char *devname, int fd,
} else {
pr_err("failed to read sysfs parameters for %s\n",
devname);
- return 1;
+ rv = 1;
+ goto fallback;
}
frozen = freeze(st);
if (frozen < -1) {
/* freeze() already spewed the reason */
sysfs_free(sra);
- return 1;
+ rv = 1;
+ goto fallback;
} else if (frozen < 0) {
pr_err("%s is performing resync/recovery and cannot be reshaped\n", devname);
sysfs_free(sra);
- return 1;
+ rv = 1;
+ goto fallback;
}
/* ========= set size =============== */
@@ -2084,6 +2098,11 @@ release:
sysfs_free(sra);
if (frozen > 0)
unfreeze(st);
+fallback:
+ if (cfd > -1)
+ close(cfd);
+ if (subarray)
+ free(subarray);
return rv;
}
--
2.4.3
^ permalink raw reply related
* WANTED new maintainer for Linux/md (and mdadm)
From: NeilBrown @ 2015-12-21 6:10 UTC (permalink / raw)
To: linux-raid, dm-devel, LKML; +Cc: Linus Torvalds, J. Bruce Fields, Phil Turmel
[-- Attachment #1: Type: text/plain, Size: 4527 bytes --]
hi,
I became maintainer for md (Linux Software RAID) in late 2001 and on
the whole it has been fun and a valuable experience. But I have been
losing interest in recent years (https://lwn.net/Articles/511073/) and
as was mentioned at the kernel summit, I would like to resign. Some
years ago I managed to hand over nfsd to the excellent Bruce Fields,
but I do not seem to have the gift that Linus has of attracting
maintainers. While there are a number of people who know quite a bit
about md and/or have contributed to development, there is no obvious
candidate for replacement maintainer - no one who has already been
doing significant parts of the maintainer role.
So I have decided to fall back on the mechanism by which I ended up
being maintainer in the first place. I will create a vacuum and hope
someone fills it (yes: I was sucked-in....). So as of 1st February
2016 I will be resigning.
At the kernel summit in October Linus talked about the value of
maintainership teams (https://lwn.net/Articles/662979/). I think it
would be great if a (small) team formed to continue to oversee md
rather than just a single individual (or maybe the dm team could extend
to include md??). If I had managed to be part of a team rather than
"going it alone" for so long, I might feel less tired of the whole
thing now.
I don't see it as my place to appoint that team or any individuals, or
even to nominate any candidates. A very important attribute of a
maintainer is that they need to care about the code and the subsystem
and I cannot tell other people to care (or even know if they do). It
is really up to individuals to volunteer. A few people have been
mentioned to me in earlier less-public conversations. Any of them may
well be suitable, but I would rather they named themselves if
interested.
So I'm hoping to get one or more volunteers to be maintainer:
- to gather and manage patches and outstanding issues,
- to review patches or get them reviewed
- to follow up bug reports and get them resolved
- to feed patches upstream, maybe directly to Linus,
maybe through some other maintainer, depending on what
relationships already exist or can be formed,
- to guide the longer term direction (saying "no" is important
sometimes),
- to care,
but also to be aware that maintainership takes real effort and time, as
does anything that is really worthwhile.
This all applies to mdadm as well as md (except you would ultimately
*be* upstream for mdadm, not needing to send it anywhere). Even if a
clear team doesn't form it would be great if different people
maintained mdadm and md.
One part of the job that I have put a lot of time in to is following
the linux-raid@vger.kernel.org list and providing support. This makes
people feel good about md and so more adventurous in using it.
Consequently I tend to hear about bugs and usability issues nice and
early (well before paying customers hit them in most cases) and that is
a big win.
In recent times I've been doing less of this and have been absolutely
thrilled that the gap has been more than filled by other very competent
community members. Not developers particular but a number of md users
have been providing excellent support. I'd particularly like to
high-light Phil Turmel who is very forthcoming with excellent advice,
but he is certainly not the only one who deserves a lot of thanks.
So "Thank you" to everyone who answers questions on linux-raid.
This would be a good place for any future maintainer to hang out to
receive wisdom as well as to provide support.
I will still be around. I can certainly help out in some sort of
mentor role, and can probably be convinced to review patches and
comment on designs. But I really want to head towards spending less
time on md (there are so many other interesting things to learn about).
So: if anyone is interested - please announce yourself, ask questions
and start doing things. I have no clear idea about how a transition
will happen. That is really up to you (plural). Take the bull by the
horns and start *being* a maintainer(team). I won't get in your way
and I'll help where I can.
Thanks,
NeilBrown
P.S. I'm committed to continue to work with the raid5-journal effort
From Facebook and the raid1-cluster effort from SUSE and the
line-in-the-sand of 1st February won't affect my support for those.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH] mdadm: check bitmap first when reshape raid1 to raid0
From: Xiao Ni @ 2015-12-21 6:00 UTC (permalink / raw)
To: NeilBrown, linux-raid
In-Reply-To: <87y4co4ixe.fsf@notabene.neil.brown.name>
On 12/21/2015 10:47 AM, NeilBrown wrote:
> On Tue, Nov 03 2015, Xiao Ni wrote:
>
>> One raid1 with bitmap is composed by 4 disks. It'll fail when rashape
>> to raid0 and lose 3 disks. It should check bitmap first when reshape
>> raid1 to raid0.
>>
>> And need to free subarray, close cfd in Grow_reshape.
>>
>> It can't remove bitmap in Grow_reshape, because it's already frozen. I
>> think it should not unfreeze in the process of the function. So I just
>> test the bitmap.
> I don't find this acceptable, sorry.
> If mdadm it asked the change the level from raid1 to raid0 it should do
> that, unless it is dangerous or impossible.
> Removing a bitmap is neither dangerous nor impossible. To just do it.
>
> I would probably put a test in before freezing the array.
> if array has a bitmap and a request was made to change the level to
> raid0, then remove the bitmap.
>
> It that so hard?
I focused my attention on the codes nearby the place where I modified.
It's easy to do this and I'll re-send it.
>
> Also you patch does multiple different things, some of which weren't
> explained at all in the comment above.
I'll give more comments for them.
>
> In particlar, the change to mdadm.c is completely pointless.
> It seems to just be adding a 'close' call before 'exit'.
> As 'exit' implicitly closes all file descriptors there is nothing to be
> gained by closing anything immediately before calling exit.
Hmm, I know this. I just thought it maybe better to call close in pairs
with open. Yes, as you said, it's really pointless.
>
> Thanks,
> NeilBrown
>
>
>> Signed-off-by: Xiao Ni <xni@redhat.com>
>> ---
>> Grow.c | 45 ++++++++++++++++++++++++++++-----------------
>> mdadm.c | 16 ++++++++++++----
>> 2 files changed, 40 insertions(+), 21 deletions(-)
>>
>> diff --git a/Grow.c b/Grow.c
>> index 80d7b22..c4ea2a5 100644
>> --- a/Grow.c
>> +++ b/Grow.c
>> @@ -1600,15 +1600,14 @@ int Grow_reshape(char *devname, int fd,
>> cfd = open_dev_excl(st->container_devnm);
>> } else {
>> container = st->devnm;
>> - close(fd);
>> cfd = open_dev_excl(st->devnm);
>> fd = cfd;
>> }
>> if (cfd < 0) {
>> pr_err("Unable to open container for %s\n",
>> devname);
>> - free(subarray);
>> - return 1;
>> + rv = 1;
>> + goto release;
>> }
>>
>> rv = st->ss->load_container(st, cfd, NULL);
>> @@ -1616,8 +1615,8 @@ int Grow_reshape(char *devname, int fd,
>> if (rv) {
>> pr_err("Cannot read superblock for %s\n",
>> devname);
>> - free(subarray);
>> - return 1;
>> + rv = 1;
>> + goto release;
>> }
>>
>> /* check if operation is supported for metadata handler */
>> @@ -1641,8 +1640,8 @@ int Grow_reshape(char *devname, int fd,
>> pr_err("cannot reshape arrays in container with unsupported metadata: %s(%s)\n",
>> devname, container);
>> sysfs_free(cc);
>> - free(subarray);
>> - return 1;
>> + rv = 1;
>> + goto release;
>> }
>> }
>> sysfs_free(cc);
>> @@ -1662,7 +1661,8 @@ int Grow_reshape(char *devname, int fd,
>> s->raiddisks - array.raid_disks,
>> s->raiddisks - array.raid_disks == 1 ? "" : "s",
>> array.spare_disks + added_disks);
>> - return 1;
>> + rv = 1;
>> + goto release;
>> }
>>
>> sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
>> @@ -1675,17 +1675,18 @@ int Grow_reshape(char *devname, int fd,
>> } else {
>> pr_err("failed to read sysfs parameters for %s\n",
>> devname);
>> - return 1;
>> + rv = 1;
>> + goto release;
>> }
>> frozen = freeze(st);
>> if (frozen < -1) {
>> /* freeze() already spewed the reason */
>> - sysfs_free(sra);
>> - return 1;
>> + rv = 1;
>> + goto release;
>> } else if (frozen < 0) {
>> pr_err("%s is performing resync/recovery and cannot be reshaped\n", devname);
>> - sysfs_free(sra);
>> - return 1;
>> + rv = 1;
>> + goto release;
>> }
>>
>> /* ========= set size =============== */
>> @@ -1898,11 +1899,16 @@ size_change_error:
>> array.layout == ((1 << 8) + 2) && !(array.raid_disks & 1)) ||
>> (s->level == 0 && array.level == 1 && sra)) {
>> int err;
>> +
>> + if (array.state & (1<<MD_SB_BITMAP_PRESENT)) {
>> + pr_err("Bitmap must be removed before level can be changed\n");
>> + rv = 1;
>> + goto release;
>> + }
>> +
>> err = remove_disks_for_takeover(st, sra, array.layout);
>> if (err) {
>> - dprintf("Array cannot be reshaped\n");
>> - if (cfd > -1)
>> - close(cfd);
>> + pr_err("Array cannot be reshaped\n");
>> rv = 1;
>> goto release;
>> }
>> @@ -2078,9 +2084,14 @@ size_change_error:
>> frozen = 0;
>> }
>> release:
>> - sysfs_free(sra);
>> if (frozen > 0)
>> unfreeze(st);
>> + if (sra)
>> + sysfs_free(sra);
>> + if (cfd > -1)
>> + close(cfd);
>> + if (subarray)
>> + free(subarray);
>> return rv;
>> }
>>
>> diff --git a/mdadm.c b/mdadm.c
>> index f56a8cf..95db2a0 100644
>> --- a/mdadm.c
>> +++ b/mdadm.c
>> @@ -1299,7 +1299,8 @@ int main(int argc, char *argv[])
>> pr_err("'1' is an unusual number of drives for an array, so it is probably\n"
>> " a mistake. If you really mean it you will need to specify --force before\n"
>> " setting the number of drives.\n");
>> - exit(2);
>> + rv = 2;
>> + goto release;
>> }
>> }
>>
>> @@ -1326,13 +1327,15 @@ int main(int argc, char *argv[])
>> rv = get_cluster_name(&c.homecluster);
>> if (rv) {
>> pr_err("The md can't get cluster name\n");
>> - exit(1);
>> + rv = 1;
>> + goto release;
>> }
>> }
>>
>> if (c.backup_file && data_offset != INVALID_SECTORS) {
>> pr_err("--backup-file and --data-offset are incompatible\n");
>> - exit(2);
>> + rv = 2;
>> + goto release;
>> }
>>
>> if ((mode == MISC && devmode == 'E')
>> @@ -1340,7 +1343,8 @@ int main(int argc, char *argv[])
>> /* Anyone may try this */;
>> else if (geteuid() != 0) {
>> pr_err("must be super-user to perform this action\n");
>> - exit(1);
>> + rv = 1;
>> + goto release;
>> }
>>
>> ident.autof = c.autof;
>> @@ -1640,6 +1644,10 @@ int main(int argc, char *argv[])
>> autodetect();
>> break;
>> }
>> +
>> +release:
>> + if (mdfd > -1)
>> + close(mdfd);
>> exit(rv);
>> }
>>
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Stop resync operation
From: NeilBrown @ 2015-12-21 4:22 UTC (permalink / raw)
To: d4lamar, linux-raid
In-Reply-To: <CAF8SPVLYv4fq1Wf80RGHuiO0bLwCK7EU+yp3M3bBXk2_23Nf4Q@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 853 bytes --]
On Fri, Nov 06 2015, d4lamar wrote:
> Dear Developers,
>
> I have a question for you.
>
> I have a raid10 md array:
>
> Is it theoretically possible to stop a running resync operation and
> mark the array as clean without stopping the array ?
Firstly stop the resync with:
echo frozen > /sys/block/mdXXX/md/sync_action
Then mark the resync as complete with
echo none > /sys/block/mdXXX/md/resync_start
then re-enable sync with
echo idle > /sys/block/mdXXX/md/sync_action
But maybe 6 weeks was too long to wait for a reply :-(
NeilBrown
>
> I do not want the resync to be respawned automatically.
>
> Thanks and Best Regards,
> d4lamar
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: RESYNC Bug ? [was Re: Stop resync operation]
From: NeilBrown @ 2015-12-21 4:09 UTC (permalink / raw)
To: d4lamar, Anugraha Sinha; +Cc: linux-raid
In-Reply-To: <CAF8SPVK_S0QGETNvZacmnN7Y=T0d0MFYWZsEWEke1sHf=YHX-A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2781 bytes --]
On Tue, Nov 10 2015, d4lamar wrote:
> 2015-11-10 5:26 GMT+01:00 Anugraha Sinha <asinha.mailinglist@gmail.com>:
>> Hi,
>>
>> Could you share your kernel version and mdadm package version, please?
>
> Of course.
>
> 2.6.32-220.7.1.el6.lustre.4033.x86_64
This is a Redhat kernel based on a very old mainline kernel.
You are experiencing a kernel bug (which seems vaguely familiar by I
don't recall the details).
To get support for a Redhat kernel, you need to talk to Redhat.
If you want help here, you need to be running a kernel must closer to
current mainline. It doesn't have to be the very latest, but something
released in the last year would be best.
NeilBrown
> mdadm-3.2.2-9.xrtx.5.x86_64
>
> Thanks,
> d4lamar
>
>> On 11/10/2015 1:13 PM, d4lamar wrote:
>>>
>>> Dear Developers,
>>>
>>> I am here again to ask you support about RESYNC operation.
>>> In my first mail I asked if there were any method to stop a running
>>> RESYNC.
>>>
>>> Now I am experiencing a very strange behaviour... RESYNC operation
>>> exceeded 100% completion percentage...
>>>
>>> # cat /proc/mdstat
>>> Tue Nov 10 05:02:41 CET 2015
>>> Personalities : [raid1] [raid10] [linear]
>>> md66 : active linear md0[0]
>>> 2939452080 blocks super 1.2 4k rounding
>>>
>>> md0 : active raid10 sdab[17](S) sdt[16](S) sdaj[0] sdx[12] sdai[11]
>>> sdah[9] sdam[15] sdae[7] sdas[6] sdaa[5] sdap[4] sdao[3] sdaf[14]
>>> sdz[1]
>>> 2939453104 blocks super 1.2 4K chunks 2 near-copies [14/12]
>>> [UUUUUUUUUU_UU_]
>>> [=========================>] resync =129.9% (545894292/419921872)
>>> finish=20192933978332.0min speed=7612K/sec
>>> bitmap: 22/22 pages [88KB], 65536KB chunk
>>>
>>> md65 : active raid1 sdal[0] sdar[1]
>>> 419921875 blocks super 1.2 [2/2] [UU]
>>> bitmap: 0/4 pages [0KB], 65536KB chunk
>>>
>>> unused devices: <none>
>>>
>>> Are you aware of such possible behaviour ? Do you think I have just hit a
>>> BUG ?
>>>
>>> Thanks and Best Regards,
>>> d4lamar
>>>
>>> 2015-11-06 7:44 GMT+01:00 d4lamar <d4lamar@gmail.com>:
>>>>
>>>> Dear Developers,
>>>>
>>>> I have a question for you.
>>>>
>>>> I have a raid10 md array:
>>>>
>>>> Is it theoretically possible to stop a running resync operation and
>>>> mark the array as clean without stopping the array ?
>>>>
>>>> I do not want the resync to be respawned automatically.
>>>>
>>>> Thanks and Best Regards,
>>>> d4lamar
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: badblocks seem to be causing problems with raid6 - badblocks list replicating all all drives
From: NeilBrown @ 2015-12-21 4:00 UTC (permalink / raw)
To: matt, linux-raid
In-Reply-To: <3fafa3e9267b4ba0b5f9d61d3a416cf5@digitallyhosted.com>
[-- Attachment #1: Type: text/plain, Size: 3972 bytes --]
On Thu, Nov 12 2015, matt@digitallyhosted.com wrote:
> Hello,
>
> I posted a while back about getting buffer i/o errors in my dmesg logs
> to my raid array, something along the lines of this:
>
> [158219.456484] EXT4-fs warning (device md4): ext4_end_bio:329: I/O
> error -5 writing to inode 125274714 (offset 176160768 size 8388608
> starting block 4955235712)
> [158219.456487] Buffer I/O error on device md4, logical block 4955235584
> [158219.456490] Buffer I/O error on device md4, logical block 4955235585
> [158219.456491] Buffer I/O error on device md4, logical block 4955235586
> [158219.456491] Buffer I/O error on device md4, logical block 4955235587
> [158219.456492] Buffer I/O error on device md4, logical block 4955235588
> [158219.456493] Buffer I/O error on device md4, logical block 4955235589
> [158219.456494] Buffer I/O error on device md4, logical block 4955235590
> [158219.456495] Buffer I/O error on device md4, logical block 4955235591
> [158219.456496] Buffer I/O error on device md4, logical block 4955235592
> [158219.456497] Buffer I/O error on device md4, logical block 4955235593
> [158219.456580] EXT4-fs warning (device md4): ext4_end_bio:329: I/O
> error -5 writing to inode 125274714 (offset 176160768 size 8388608
> starting block 4955235456)
> [158219.456663] EXT4-fs warning (device md4): ext4_end_bio:329: I/O
> error -5 writing to inode 125274714 (offset 176160768 size 8388608
> starting block 4955235200)
> [158219.456747] EXT4-fs warning (device md4): ext4_end_bio:329: I/O
> error -5 writing to inode 125274714 (offset 176160768 size 8388608
> starting block 4955234944)
> [158219.456829] EXT4-fs warning (device md4): ext4_end_bio:329: I/O
> error -5 writing to inode 125274714 (offset 176160768 size 8388608
> starting block 4955234688)
> [158219.456912] EXT4-fs warning (device md4): ext4_end_bio:329: I/O
> error -5 writing to inode 125274714 (offset 176160768 size 8388608
> starting block 4955234432)
> [158469.158278] EXT4-fs warning (device md4): ext4_end_bio:329: I/O
> error -5 writing to inode 123995503 (offset 0 size 8388608 starting
> block 4970080384)
> [158469.158281] buffer_io_error: 1526 callbacks suppressed
>
> I am now using the latest mainline kernel, 4.3.0 and I believe something
> is going wrong with the badblocks implementation.
>
> I originally had 3 drives, all with the same badblocks list. This array
> has been running a while so I have no idea how these 3 discs all ended
> up with the same list of badblocks.
>
> Now, if I remove any drive, which has no badblock entries, and re-add
> it. Once the sync is complete I end up with another drive with the same
> badblocks list.
An entry in the bad-blocks list means that the data at that location is
not available, possibly because the block is bad.
If you have a degraded RAID6 where any appears in 2 or more bad-blocks
lists, then it is not possible to recover the data at that address when
a spare is recovered. So the same address will be added to the bad
block log on the spare.
You could remove he bad block from all the device by writing to all of
the affected blocks at once, but that is admittedly a little difficult
to manage.
I probably need to make it possible to clear the bad block log by a
successful write to just a single data block (and the matching parity
blocks). I've added that to by to-do list.
I've just push out a modification to mdadm so you can run
mdadm --assemble --update=force-no-bbl /dev/md/whatver list of devices
and it will remove the bad-block lists even though they are not empty.
So if you
git clone git://neil.brown.name/mdadm
cd mdadm
make
./mdadm --stop /dev/md4
./mdadm --assemble /dev/md4 --update=force-no-bblk list-of-devices
it should get rid of your problem.
However, as your mail is 6 weeks old (I was on leave...) maybe you have
already found another solution.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Reconstruct a RAID 6 that has failed in a non typical manner
From: NeilBrown @ 2015-12-21 3:40 UTC (permalink / raw)
To: Phil Turmel, Clement Parisot; +Cc: linux-raid
In-Reply-To: <563B5AEB.5020006@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 402 bytes --]
On Fri, Nov 06 2015, Phil Turmel wrote:
>
> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
>
Would it make sense for mdadm to automagically do something like this?
i.e. whenever it adds a device to an array (with redundancy) it write
180 (or something configurable) to the 'timeout' file if there is one?
Why do we pick 180?
Can this cause problems on some drives?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH] mdadm: check bitmap first when reshape raid1 to raid0
From: NeilBrown @ 2015-12-21 2:47 UTC (permalink / raw)
To: Xiao Ni, linux-raid
In-Reply-To: <1446510328-22917-1-git-send-email-xni@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 6183 bytes --]
On Tue, Nov 03 2015, Xiao Ni wrote:
> One raid1 with bitmap is composed by 4 disks. It'll fail when rashape
> to raid0 and lose 3 disks. It should check bitmap first when reshape
> raid1 to raid0.
>
> And need to free subarray, close cfd in Grow_reshape.
>
> It can't remove bitmap in Grow_reshape, because it's already frozen. I
> think it should not unfreeze in the process of the function. So I just
> test the bitmap.
I don't find this acceptable, sorry.
If mdadm it asked the change the level from raid1 to raid0 it should do
that, unless it is dangerous or impossible.
Removing a bitmap is neither dangerous nor impossible. To just do it.
I would probably put a test in before freezing the array.
if array has a bitmap and a request was made to change the level to
raid0, then remove the bitmap.
It that so hard?
Also you patch does multiple different things, some of which weren't
explained at all in the comment above.
In particlar, the change to mdadm.c is completely pointless.
It seems to just be adding a 'close' call before 'exit'.
As 'exit' implicitly closes all file descriptors there is nothing to be
gained by closing anything immediately before calling exit.
Thanks,
NeilBrown
>
> Signed-off-by: Xiao Ni <xni@redhat.com>
> ---
> Grow.c | 45 ++++++++++++++++++++++++++++-----------------
> mdadm.c | 16 ++++++++++++----
> 2 files changed, 40 insertions(+), 21 deletions(-)
>
> diff --git a/Grow.c b/Grow.c
> index 80d7b22..c4ea2a5 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1600,15 +1600,14 @@ int Grow_reshape(char *devname, int fd,
> cfd = open_dev_excl(st->container_devnm);
> } else {
> container = st->devnm;
> - close(fd);
> cfd = open_dev_excl(st->devnm);
> fd = cfd;
> }
> if (cfd < 0) {
> pr_err("Unable to open container for %s\n",
> devname);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> rv = st->ss->load_container(st, cfd, NULL);
> @@ -1616,8 +1615,8 @@ int Grow_reshape(char *devname, int fd,
> if (rv) {
> pr_err("Cannot read superblock for %s\n",
> devname);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> /* check if operation is supported for metadata handler */
> @@ -1641,8 +1640,8 @@ int Grow_reshape(char *devname, int fd,
> pr_err("cannot reshape arrays in container with unsupported metadata: %s(%s)\n",
> devname, container);
> sysfs_free(cc);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
> }
> sysfs_free(cc);
> @@ -1662,7 +1661,8 @@ int Grow_reshape(char *devname, int fd,
> s->raiddisks - array.raid_disks,
> s->raiddisks - array.raid_disks == 1 ? "" : "s",
> array.spare_disks + added_disks);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
> @@ -1675,17 +1675,18 @@ int Grow_reshape(char *devname, int fd,
> } else {
> pr_err("failed to read sysfs parameters for %s\n",
> devname);
> - return 1;
> + rv = 1;
> + goto release;
> }
> frozen = freeze(st);
> if (frozen < -1) {
> /* freeze() already spewed the reason */
> - sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto release;
> } else if (frozen < 0) {
> pr_err("%s is performing resync/recovery and cannot be reshaped\n", devname);
> - sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> /* ========= set size =============== */
> @@ -1898,11 +1899,16 @@ size_change_error:
> array.layout == ((1 << 8) + 2) && !(array.raid_disks & 1)) ||
> (s->level == 0 && array.level == 1 && sra)) {
> int err;
> +
> + if (array.state & (1<<MD_SB_BITMAP_PRESENT)) {
> + pr_err("Bitmap must be removed before level can be changed\n");
> + rv = 1;
> + goto release;
> + }
> +
> err = remove_disks_for_takeover(st, sra, array.layout);
> if (err) {
> - dprintf("Array cannot be reshaped\n");
> - if (cfd > -1)
> - close(cfd);
> + pr_err("Array cannot be reshaped\n");
> rv = 1;
> goto release;
> }
> @@ -2078,9 +2084,14 @@ size_change_error:
> frozen = 0;
> }
> release:
> - sysfs_free(sra);
> if (frozen > 0)
> unfreeze(st);
> + if (sra)
> + sysfs_free(sra);
> + if (cfd > -1)
> + close(cfd);
> + if (subarray)
> + free(subarray);
> return rv;
> }
>
> diff --git a/mdadm.c b/mdadm.c
> index f56a8cf..95db2a0 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -1299,7 +1299,8 @@ int main(int argc, char *argv[])
> pr_err("'1' is an unusual number of drives for an array, so it is probably\n"
> " a mistake. If you really mean it you will need to specify --force before\n"
> " setting the number of drives.\n");
> - exit(2);
> + rv = 2;
> + goto release;
> }
> }
>
> @@ -1326,13 +1327,15 @@ int main(int argc, char *argv[])
> rv = get_cluster_name(&c.homecluster);
> if (rv) {
> pr_err("The md can't get cluster name\n");
> - exit(1);
> + rv = 1;
> + goto release;
> }
> }
>
> if (c.backup_file && data_offset != INVALID_SECTORS) {
> pr_err("--backup-file and --data-offset are incompatible\n");
> - exit(2);
> + rv = 2;
> + goto release;
> }
>
> if ((mode == MISC && devmode == 'E')
> @@ -1340,7 +1343,8 @@ int main(int argc, char *argv[])
> /* Anyone may try this */;
> else if (geteuid() != 0) {
> pr_err("must be super-user to perform this action\n");
> - exit(1);
> + rv = 1;
> + goto release;
> }
>
> ident.autof = c.autof;
> @@ -1640,6 +1644,10 @@ int main(int argc, char *argv[])
> autodetect();
> break;
> }
> +
> +release:
> + if (mdfd > -1)
> + close(mdfd);
> exit(rv);
> }
>
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: mdadm and size differences
From: NeilBrown @ 2015-12-21 2:32 UTC (permalink / raw)
To: Marco De Vitis, linux-raid
In-Reply-To: <n31ec1$ebi$1@ger.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 3131 bytes --]
On Tue, Nov 24 2015, Marco De Vitis wrote:
> Hi,
> I cannot understand a difference in size I can see on an old Debian 5.0
> machine, maybe I'm just missing something obvious.
...
>
> You can see that sdd6 is twice the size of sdd7.
> cfdisk reports sizes of around 20 GB and 10 GB:
> sdd6 -> 19329.41
> sdd7 -> 10010.17
>
> All partitions in these two identical drives are configured as RAID 1
> using mdadm, but the md devices associated to sdd6/sdf6 and sdd7/sdf7
> are both reported to be 10 GB in size, as can be seen from the mdadm
> output below.
>
> Why?
Probably because the arrays aren't using the whole partition.
e.g. if you have a RAID! comprised of 2 1GB partitions,
then you fail one partition and had a 2GB partition as a spare and let
it rebuild. Then fail the other 1GB partition and add a 3GB partition
as a spare and let it rebuild.
Then you will have a 1GB RAID made from 2 partitions, a 2GB partition
and a 3GB partition.
If you want to resize the RAID! to use all the available space, then use
the command
mdadm --grow /dev/md2 --size=max
You might then need to resize and filesystem you have in that array.
NeilBrown
>
> Thanks for any info.
>
>> #>mdadm -vv --detail /dev/md2
>> /dev/md2:
>> Version : 00.90
>> Creation Time : Thu Feb 22 16:18:11 2007
>> Raid Level : raid1
>> Array Size : 9767424 (9.31 GiB 10.00 GB)
>> Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
>> Raid Devices : 2
>> Total Devices : 2
>> Preferred Minor : 2
>> Persistence : Superblock is persistent
>>
>> Update Time : Mon Nov 23 14:19:23 2015
>> State : clean
>> Active Devices : 2
>> Working Devices : 2
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> UUID : 5582dad7:7ac89883:0c2b9e1d:5009dd35
>> Events : 0.69228
>>
>> Number Major Minor RaidDevice State
>> 0 8 54 0 active sync /dev/sdd6
>> 1 8 86 1 active sync /dev/sdf6
>>
>> #>mdadm -vv --detail /dev/md3
>> /dev/md3:
>> Version : 00.90
>> Creation Time : Thu Feb 22 16:18:54 2007
>> Raid Level : raid1
>> Array Size : 9767424 (9.31 GiB 10.00 GB)
>> Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
>> Raid Devices : 2
>> Total Devices : 2
>> Preferred Minor : 3
>> Persistence : Superblock is persistent
>>
>> Update Time : Mon Nov 23 14:19:27 2015
>> State : clean
>> Active Devices : 2
>> Working Devices : 2
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> UUID : e44836cf:6340f7fa:52fb3562:a78d4dbb
>> Events : 0.187630
>>
>> Number Major Minor RaidDevice State
>> 0 8 55 0 active sync /dev/sdd7
>> 1 8 87 1 active sync /dev/sdf7
>
> --
> Ciao,
> Marco.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: I/O errors without erros from underlying device
From: NeilBrown @ 2015-12-21 2:25 UTC (permalink / raw)
To: arekm, linux-raid
In-Reply-To: <201512081205.48777.a.miskiewicz@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]
On Tue, Dec 08 2015, Arkadiusz Miskiewicz wrote:
> On Monday 07 of December 2015, Arkadiusz Miskiewicz wrote:
>
>> Anyway I would expect my problem to be related to badblock lists which
>> numbers are close to dmesg error message: [ 848.988518] Buffer I/O error
>> on dev md7, logical block 3907148544, async page read
>>
>> > >> http://sprunge.us/XSWI
>>
>> But how to repair these if write() also fails and
>> http://www.spinics.net/lists/raid/msg49325.html suggests that write should
>> "fix" these (by using replacement blocks I guess) ?
>
> Tried to get rid of badblock lists (well, corruption in that area is better
> than no access at all):
>
> mdadm --assemble /dev/md7 --force --update=no-bbl
> mdadm: Cannot remove active bbl from /dev/sdae1
> mdadm: Cannot remove active bbl from /dev/sdag1
> mdadm: Cannot remove active bbl from /dev/sdai1
> mdadm: Cannot remove active bbl from /dev/sdn1
> mdadm: Cannot remove active bbl from /dev/sdg
> mdadm: Cannot remove active bbl from /dev/sdad1
> mdadm: /dev/md7 has been started with 10 drives.
>
> Is there a way to archieve that anyway?
>
You probably have bad blocks in multiple disks in the one stripe
(look in /sys/block/md7/md/dev-*/badblocks or something like that to
see).
To get rid of these you would need to write to every block in the
stripe. I guess I should try to find a way to make that easier.
If you like you could hack mdadm to allow you to remove the bbl even
though they aren't empty.
In super1.c look for:
} else if (strcmp(update, "no-bbl") == 0) {
if (sb->feature_map & __cpu_to_le32(MD_FEATURE_BAD_BLOCKS))
pr_err("Cannot remove active bbl from %s\n",devname);
else {
sb->bblog_size = 0;
sb->bblog_shift = 0;
sb->bblog_offset = 0;
}
and change it to be unconditional and also to clear
MD_FEATURE_BAD_BLOCKS.
No warranty expressed or implied.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Reshape stalls and fails when SELinux is enabled
From: NeilBrown @ 2015-12-21 1:50 UTC (permalink / raw)
To: Edward Kuns, Enrico Tagliavini; +Cc: Linux-RAID
In-Reply-To: <CACsGCyQX4DKfT-6CYd7pa8P-Rq12HhP2tA79F27rw1PbNHe_QQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]
On Fri, Dec 11 2015, Edward Kuns wrote:
> I ran into this as well. I unwisely worked around it by placing the
> backup file on /tmp. I don't recommend that choice! I didn't see
> your AVC on your EMail, but I recommend you reach out to the SELinux
> folks for labeling help. I've reach out to them in the past and have
> found them very helpful (as are the folks here). Getting the labeling
> corrected will not only help you but everyone else in the long term.
>
> I agree that it would be good for mdadm to handle this kind of failure
> in a more productive fashion as well. But I'm only a lurker here. I
> don't have enough knowledge to suggest what would be better. One
> thing that really helped me debug this was commenting out the lines
>
> #StandardOutput=null
> #StandardError=null
>
> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
> -- note you have to run systemctl daemon-reload after changing the
> file for the change to be noticed.
>
> Once I did that, the system logs had much more useful information.
> I'm not sure where that file is sourced, whether it's from the
> linux-raid people or from the distributions or somewhere in between.
> Does anyone here know if there's a reason why standard out and
> standard error are sent to /dev/null for this service?
That mdadm-grow-continue@.service file is part of mdadm (in the systemd
directory).
As for why stdout/err is set to 'null', you could try finding out who
did that and ask them ....
http://git.neil.brown.name/?p=mdadm.git;a=commitdiff;h=5e76dce1acd906e8fc8af04973c3a129cdc77fd6
but I'm pretty sure that he has no idea.
You could try sending me a patch which changes those lines and gives some
sort of explanation why it is a good change to make.
There is a very good chance that I would apply that patch.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Seeking advice to convert RAID5 to RAID10 adding a drive in the process
From: NeilBrown @ 2015-12-21 1:44 UTC (permalink / raw)
To: Micheal Blue, linux-raid
In-Reply-To: <trinity-e29048a1-862f-4ae7-a1a2-e4e826b37f95-1450018043035@3capp-mailcom-bs10>
[-- Attachment #1: Type: text/plain, Size: 1487 bytes --]
On Mon, Dec 14 2015, Micheal Blue wrote:
> I have a 3 disk RAID5 currently that uses LUKS. I would like to add a new disk and covert the array to RAID10 but cannot find any walk-through of this process. I am grateful for any input from the list be it links or a personal summary.
>
>
> Below is a graphic of how my current 3 drive array is setup:
> ---------------------------------------
> [ ext4 file system ]
> ---------------------------------------
> [ LUKS cryptdevice ]
> ---------------------------------------
> [ /dev/dm0 ]
> ---------------------------------------
> [ /dev/sdb1 ] [ /dev/sdc1 ] [ /dev/sdd1 ]
> ---------------------------------------
I assume that is /dev/md0, not /dev/dm0 ....
You would need to convert to RAID0 first.
Something like
mdadm --grow /dev/md0 --level=0 --raid-disks=2
# wait for that to complete
mdadm --grow /dev/md0 --level=10 --raid-disks=4 --add /dev/sdd1 /dev/sde1
You should experiment first by creating some largish file (100Meg),
using losetup to make block devices /dev/loop0, /dev/loop1
... /dev/loop3
then create a raid 5 of some of those devices, creaet a LUKS and ext4 on
that, and then perform the reshape and make sure it all works as you
expect.
I just tested and it seemed to work on the kernel and mdadm that I have.
Of course, if you can create a backup - at least of the most important
files - that is always a good idea.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: RAID 6 "Failed to restore critical section for reshape, sorry." - recovery advice?
From: NeilBrown @ 2015-12-21 1:35 UTC (permalink / raw)
To: George Rapp, Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <CAF-KpgYe9Vxgcy2E5T1_9HdEM0YMfATMzN7WcjHra_EE+sOOTg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]
On Fri, Dec 11 2015, George Rapp wrote:
>
> I appear to be too early in the reshape for auto-recovery, but too far
> along to just say "never mind on that whole reshape business". Any
> other thoughts?
>
What this means is that you've hit a corner case that was never thought
through properly and isn't handled correctly.
The current state of the array is (I think) that it looks like a reshape
to reduce the number of devices in the array has very nearly completed.
Only the first stripe needs to be completed. Whether that first stripe
is still in the old "N+1" device layout or the new "N" device layout is
unknown to the kernel - this information is only in the backup file
(which doesn't exist).
By telling mdadm --invalid-backup, you effectively tell mdadm that there
is nothing useful in the backup file so it should know that the reshape
has actually completed. But it has no way to tell the kernel that.
What it should do in this case is (I think) rewrite the metadata to
record that the reshape is complete. But it doesn't.
I shouldn't be too hard to fix, but it isn't trivial either and I'm
unlikely to get anywhere before the Christmas break.
If you can get reshape to work at all (disable selinux?) you could try
--update=revert-reshape and let the reshape to more devices progress for
a while, and then revert it.
If you cannot get anywhere, then use
"mdadm --dump=/tmp/whatever /dev/mdthing"
to create a copy of the metadata in some spares files.
Then tar those up (a compressed tarchive should be tiny) and email them.
Then I can try and see if I can make something work on exactly the array
you have.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Start reshape failed
From: NeilBrown @ 2015-12-21 1:09 UTC (permalink / raw)
To: Kev Dorman, linux-raid
In-Reply-To: <CAN4XXPD2bmrJKRBMm+3Rz4_pegBK4Qp37cn2O-TvEZZ8ww1bFw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 865 bytes --]
On Wed, Dec 16 2015, Kev Dorman wrote:
> Hi,
>
> Xiao Ni posted back in September about a problem where mdadm --grow fails with:
>
> mdadm: Failed to initiate reshape!
>
> I'm seeing the same problem, with mdadm 3.3.4, Kernel 4.3.0. I can
> reproduce this reliably
> when adding a device to a 2-device raid0 array. Instead of switching
> to raid4 for the
> reshape, then switching back to raid0 when done, it ends up in raid4
> in recovery mode.
>
> I tried the suggested change of adding a call to md_check_recovery()
> in action_store(),
> and that does seem to help most of the time.
>
> I was curious if there is a better solution, and/or if this or another
> change will show up in
> 4.3.x soon.
I have a patch queued in my for-next branch which I'll send to Linus
before Christmas. It should then get into 4.3.x in early January (at a
guess).
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Problem w/ commit ac8fa4196d20 on older, slower hardware
From: NeilBrown @ 2015-12-21 0:43 UTC (permalink / raw)
To: Andreas Klauer, Joshua Kinard; +Cc: linux-raid
In-Reply-To: <20151113000306.GA3563@EIS>
[-- Attachment #1: Type: text/plain, Size: 3429 bytes --]
On Fri, Nov 13 2015, Andreas Klauer wrote:
> On Thu, Nov 12, 2015 at 05:28:41PM -0500, Joshua Kinard wrote:
>> running MD RAID5 and the XFS filesystem. I have /, /home, /usr, /var,
>> and /tmp on separate partitions, each a RAID5 setup.
>
> Hi, sorry for butting in,
>
> I have the same issue, on a regular consumer Haswell i5 box,
> with a setup very very similar to yours:
>
> 7x2TB disks, multiple partitions, for each: RAID-5, LUKS, LVM, XFS.
>
> The issue occurs during regular RAID check which I run daily
> (different partition/RAID each day, so it's more like a
> evenly distributed weekly check).
>
> I have an application that uses `find -size +100M` on a directory
> tree with ~3k subdirs and ~6k files in total. It doesn't do anything
> with the find result, it's purely informal. So no big data involved,
> even though the files themselves aren't small.
>
> Yet, it's slooow. The following tests were on a completely idle box,
> apart from a running RAID check on the same /dev/mdX device.
>
> Kernel 4.2.3, unpatched:
>
> real 0m53.555s
> user 0m0.013s
> sys 0m0.037s
>
> real 1m3.777s
> user 0m0.013s
> sys 0m0.037s
>
> real 1m3.453s
> user 0m0.014s
> sys 0m0.036s
>
> Kernel 4.2.3, reverted ac8fa4196d20:
>
> real 0m3.206s
> user 0m0.010s
> sys 0m0.030s
>
> real 0m0.450s
> user 0m0.003s
> sys 0m0.014s
>
> real 0m0.375s
> user 0m0.003s
> sys 0m0.012s
>
> I did echo 3 > /proc/sys/vm/drop_caches between each find.
> For some reason, subsequent calls in the reverted kernel are
> considerably faster regardless. In the original kernel it
> stays slow... if I don't drop_caches, the time is 0.006s.
>
> I don't normally reboot (while a RAID sync or check is
> running) but while switching between kernels I noticed
> the shutdown was very slow also in the original kernel.
>
> Are small requests getting delayed a lot or something?
Thanks for all the details and sorry for the delay.
Are (either of) you able to test with this small incremental patch?
When the md resync notices there is other IO pending, the old code would
cause the resync to wait at least 500msec and possibly longer to get the
overall resync speed below a threshold.
Having the threshold fixed doesn't make sense when devices have such a
wide range of speeds.
The problem patch changes it to only wait until pending resync requests
have finished. These means the wait is proportional to the speed of the
devices, which makes more sense. The hope was that this would allow
quite a few regular IO request to slip in the gap between resync requests
so that regular IO would proceed reasonably quickly. Sometimes that
worked, but obviously not for you.
This patch adds an extra delay, still proportional to the speed of the
devices, but with (hopefully) a lot more room for regular IO requests to
get queued and handled.
Thanks,
NeilBrown
diff --git a/drivers/md/md.c b/drivers/md/md.c
index c0c3e6dec248..8a25cf6087ed 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -8070,8 +8070,10 @@ void md_do_sync(struct md_thread *thread)
* Give other IO more of a chance.
* The faster the devices, the less we wait.
*/
+ unsigned long start = jiffies;
wait_event(mddev->recovery_wait,
!atomic_read(&mddev->recovery_active));
+ msleep(jiffies_to_msecs(jiffies - start));
}
}
}
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply related
* Re: clustered MD - beyond RAID1
From: NeilBrown @ 2015-12-20 23:25 UTC (permalink / raw)
To: Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <56742652.5040304@nasa.gov>
[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]
On Sat, Dec 19 2015, Scott Sinno wrote:
> Neil(or anyone well informed in mdadm development roadmaps),
>
> Aaron and myself are engineers at NASA Goddard with strong interest in
> MDADM. We currently host 6PB(raw) of live JBOD storage leveraging MDADM
> exclusively for RAID functionality.
>
> We're very interested in Clustered MDADM to improve data-availability
> in the environment, but note that only RAID1 is currently supported.
> Are there plans in the nearish-term(say over the next year) to expound
> clustered bitmap functionality to RAID5/6, or anything else you can
> divulge on that front? Thanks in advance for any guidance.
We don't talk about plans that are not backed by code - you can't trust
them.
However I cannot imagine how you could make RAID5 work efficiently in a
cluster.
RAID1 works because we assume that the file system will have its own
locking to ensure that only one node writes to a given block at a given
time. So while node-A is writing to a block, RAID1 knows that no other
node is writing there so it can update all copies and be sure no race
will result in the copies being inconsistent.
For this to work with RAID5 we would need to assume the filesystem will
ensure only one node is writing to a given stripe at a time, and that is
not realistic.
So to make it work we would need the md layer to lock each stripe during
an update. I have trouble imagining that running with much speed. Hard
to know without testing of course.
I know of no-one with plans to do that testing.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 3/3] raid5: allow r5l_io_unit allocations to fail
From: NeilBrown @ 2015-12-20 22:59 UTC (permalink / raw)
To: Shaohua Li, Christoph Hellwig; +Cc: linux-raid
In-Reply-To: <20151218230657.GA342953@devbig084.prn1.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 1950 bytes --]
On Sat, Dec 19 2015, Shaohua Li wrote:
> On Fri, Dec 18, 2015 at 12:25:35PM +0100, Christoph Hellwig wrote:
>> [can you trim your replies please? I had to trim almost 150 lines of
>> junk before getting to your reply!]
>>
>> > > spin_unlock_irqrestore(&log->io_list_lock, flags);
>> > >
>> > > or is that too simplistic?
>> >
>> > maybe add a new list and run the list at the begining of r5l_do_reclaim().
>>
>> What's the advantage of another list?
>
> I mean simply waking up reclaim thread doesn't work as r5l_do_reclaim()
> will return if reclaimable == 0. There is no guarantee reclaimable space
> is available in the allocation failure. So we'd better move the retry at
> the begining of r5l_do_reclaim(). If yes, we can't reuse the
> no_space_stripes list.
They certainly are conceptually different lists. Whether the same
linked list can be used for both is an engineering question.
One list is for updates that cannot be handled yet because the log is
full. Once the head of the log is cleared and the start pointer
updated, those requests can be handled (or some of them can).
The other list is for updates that cannot be handled yet because a
kmalloc failed. There is no clear trigger for when to handle those
again so we would need to retry quite frequently. It would be easy to
miss retrying in rare circumstances...
I wonder if we should have a mempool for these io units too.
We would allocate with GFP_ATOMIC (or similar) so the allocation woult
fail instead of blocking, but we would then know that an allocation
could only fail if there was another request in flight. So the place
where we free an io_unit would be the obviously correct place to trigger
a retry of the delayed-due-to-mem-allocation-failure stripes.
So I think I would prefer two lists, another mempool, and very well
defined places to retry the two lists. Is that over-engineering?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox