* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-21 16:05 UTC (permalink / raw)
To: Wols Lists, andras, Linux-RAID
In-Reply-To: <5627BAB4.7010402@youngman.org.uk>
Hi Wols,
I glad you've got the big picture correct, but some details need to be
addressed:
On 10/21/2015 12:17 PM, Wols Lists wrote:
> tl;dr summary ...
>
> Desktop drives are spec'd as being okay with one soft error per 10TB
> read - that's where a read fails, you try again, and everything's okay.
No, this isn't correct.
That spec is for *unrecoverable* read errors. For desktop drives,
typically spec'd as one such error every 1e14 bits read, on average.
These are failures where you really have lost the sector contents. Such
sectors are marked as "Pending Relocations" in drive firmware. But the
recording surface might still be good, so the drive waits for a write to
that pending sector, which it then verifies, before deciding to relocate
or not.
When MD raid receives a read error, whether in normal operation or a
scrub, it will reconstruct the missing data and write it back, closing
this loop immediately. Where "normal operation" means "read errors are
reported by the drive before the driver times out".
> A resync will scan the array from start to finish - if you have 10TB's
> worth of disk, you MUST be prepared to handle these errors.
>
> By default, mdadm will assume a disk is faulty and kick it after about
> 10secs, but a desktop drive will hang for maybe several minutes before
> reporting a problem.
MD raid has no timeout, and does not kick drives out for occassional
read errors. The timeout is in the per-device drivers (SCSI, SATA,
whatever). Which defaults to 30 seconds. Desktop drives typically keep
trying to read a bad sector for 120 seconds or more, ignoring the world
while they do so. Drives with default SCTERC support typically report a
read error within four to seven seconds.
With a desktop drive, the linux device driver bails after 30 seconds and
resets the link to the drive -- which gets ignored. And keeps getting
ignored until the original read retry cycle finishes. During this time,
MD has reconstructed the data and told the driver to write the fixed
sector. That *write* also fails (because the driver is failing to
reset) and that *write error* kicks the drive out of the array.
Anyways, please consider reading the threads I pointed Andras at :-)
Phil
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Wols Lists @ 2015-10-21 16:17 UTC (permalink / raw)
To: andras, Linux-RAID
In-Reply-To: <562660EE.9020504@turmel.org>
On 20/10/15 16:42, Phil Turmel wrote:
> Don't. You have another problem: green & desktop drives in a raid
> array. They aren't built for it and will give you grief of one form or
> another. Anyways, their problem with timeout mismatch can be worked
> around with long driver timeouts. Before you do anything else, you
> *MUST* run this command:
>
> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
>
> (Arrange for this to happen on every boot, and keep doing it manually
> until your boot scripts are fixed.)
tl;dr summary ...
Desktop drives are spec'd as being okay with one soft error per 10TB
read - that's where a read fails, you try again, and everything's okay.
A resync will scan the array from start to finish - if you have 10TB's
worth of disk, you MUST be prepared to handle these errors.
By default, mdadm will assume a disk is faulty and kick it after about
10secs, but a desktop drive will hang for maybe several minutes before
reporting a problem.
In other words, your drives can meet manufacturer's specs, but, with
default settings, your array will never be able to rebuild after a
problem! (Note that many people will say "I've never had a problem", but
most drives are better than spec. You just don't want to be the unlucky
one ...)
Not that I have any (yet), but I'd second the recommendation for WD
Reds. I've got Seagate Barracudas (not raid-compliant), and the Reds are
not much more expensive, and are also the only drives I've found that
support the raid features - mostly that by default they will fail and
report a problem very quickly. (Plus they're spec'd at reading about
40TB per soft error :-)
Cheers,
Wol
^ permalink raw reply
* [PATCH 0/3] improve mdadm support of write journal
From: Song Liu @ 2015-10-21 18:35 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, shli, hch, dan.j.williams, kernel-team, Song Liu
Per discussion in
http://marc.info/?l=linux-raid&m=144522071120149
and related threads, the following patches add more support
for write journal (test script, man page, small refactoring).
Song Liu (3):
mdadm: refactor write journal code in Assemble and Incremental
mdadm: Add description of write journal to md.4
mdadm: add test script for raid456 journal
Assemble.c | 24 +++++---------------
Incremental.c | 25 ++++++--------------
md.4 | 20 ++++++++++++++++
mdadm.h | 6 ++---
super1.c | 19 +++-------------
test | 9 ++++++++
tests/20raid5journal | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
7 files changed, 111 insertions(+), 56 deletions(-)
create mode 100644 tests/20raid5journal
--
2.4.6
^ permalink raw reply
* [PATCH 1/3] mdadm: refactor write journal code in Assemble and Incremental
From: Song Liu @ 2015-10-21 18:35 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, shli, hch, dan.j.williams, kernel-team, Song Liu
In-Reply-To: <1445452516-2816184-1-git-send-email-songliubraving@fb.com>
As discussed, standalone require_journal() in struct superswitch
is not a very good idea. Instead, journal related information
fits well in struct mdinfo.
This patch simplifies journal support code in Assemble and
Incremental as:
- Add journal_device_required and journal_clean to struct mdinfo;
- Remove function require_journal from struct superswitch;
- Update Assemble and Incremental to use journal_device_required
and journal_clean from struct mdinfo (instead of separate var).
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
Assemble.c | 24 ++++++------------------
Incremental.c | 25 +++++++------------------
mdadm.h | 6 ++----
super1.c | 19 +++----------------
4 files changed, 18 insertions(+), 56 deletions(-)
diff --git a/Assemble.c b/Assemble.c
index 0661e8d..90c9918 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -948,16 +948,14 @@ static int start_array(int mdfd,
int clean, char *avail,
int start_partial_ok,
int err_ok,
- int was_forced,
- int expect_journal,
- int journal_clean
+ int was_forced
)
{
int rv;
int i;
unsigned int req_cnt;
- if (expect_journal && (journal_clean == 0)) {
+ if (content->journal_device_required && (content->journal_clean == 0)) {
if (!c->force) {
pr_err("Not safe to assemble with missing or stale journal device, consider --force.\n");
return 1;
@@ -1130,7 +1128,7 @@ static int start_array(int mdfd,
fprintf(stderr, "%s %d rebuilding", sparecnt?",":" and", rebuilding_cnt);
if (sparecnt)
fprintf(stderr, " and %d spare%s", sparecnt, sparecnt==1?"":"s");
- if (journal_clean)
+ if (content->journal_clean)
fprintf(stderr, " and %d journal", journalcnt);
fprintf(stderr, ".\n");
}
@@ -1307,8 +1305,6 @@ int Assemble(struct supertype *st, char *mddev,
int i;
int was_forced = 0;
int most_recent = 0;
- int expect_journal = 0;
- int journal_clean = 0;
int chosen_drive;
int change = 0;
int inargv = 0;
@@ -1371,14 +1367,6 @@ try_again:
if (!st || !st->sb || !content)
return 2;
- if (st->ss->require_journal) {
- expect_journal = st->ss->require_journal(st);
- if (expect_journal == 2) {
- pr_err("BUG: Superblock not loaded in Assemble.c:Assemble\n");
- return 1;
- }
- }
-
/* We have a full set of devices - we now need to find the
* array device.
* However there is a risk that we are racing with "mdadm -I"
@@ -1567,7 +1555,7 @@ try_again:
*/
if (content->array.level != LEVEL_MULTIPATH) {
if (devices[j].i.disk.state & (1<<MD_DISK_JOURNAL)) {
- if (expect_journal)
+ if (content->journal_device_required)
journalcnt++;
else /* unexpected journal, mark as faulty */
devices[j].i.disk.state |= (1<<MD_DISK_FAULTY);
@@ -1604,7 +1592,7 @@ try_again:
) {
devices[j].uptodate = 1;
if (devices[j].i.disk.state & (1<<MD_DISK_JOURNAL))
- journal_clean = 1;
+ content->journal_clean = 1;
if (i < content->array.raid_disks * 2) {
if (devices[j].i.recovery_start == MaxSector ||
(content->reshape_active &&
@@ -1833,7 +1821,7 @@ try_again:
c,
clean, avail, start_partial_ok,
pre_exist != NULL,
- was_forced, expect_journal, journal_clean);
+ was_forced);
if (rv == 1 && !pre_exist)
ioctl(mdfd, STOP_ARRAY, NULL);
free(devices);
diff --git a/Incremental.c b/Incremental.c
index 5b2974c..781d27d 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -35,7 +35,7 @@
static int count_active(struct supertype *st, struct mdinfo *sra,
int mdfd, char **availp,
- struct mdinfo *info, int *journal_device_missing);
+ struct mdinfo *info);
static void find_reject(int mdfd, struct supertype *st, struct mdinfo *sra,
int number, __u64 events, int verbose,
char *array_name);
@@ -520,7 +520,10 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
sysfs_free(sra);
sra = sysfs_read(mdfd, NULL, (GET_DEVS | GET_STATE |
GET_OFFSET | GET_SIZE));
- active_disks = count_active(st, sra, mdfd, &avail, &info, &journal_device_missing);
+ active_disks = count_active(st, sra, mdfd, &avail, &info);
+
+ journal_device_missing = (info.journal_device_required) && (info.journal_clean == 0);
+
if (enough(info.array.level, info.array.raid_disks,
info.array.layout, info.array.state & 1,
avail) == 0) {
@@ -690,8 +693,7 @@ static void find_reject(int mdfd, struct supertype *st, struct mdinfo *sra,
static int count_active(struct supertype *st, struct mdinfo *sra,
int mdfd, char **availp,
- struct mdinfo *bestinfo,
- int *journal_device_missing)
+ struct mdinfo *bestinfo)
{
/* count how many devices in sra think they are active */
struct mdinfo *d;
@@ -705,8 +707,6 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
int devnum;
int b, i;
int raid_disks = 0;
- int require_journal_dev = 0;
- int has_journal_dev = 0;
if (!sra)
return 0;
@@ -728,18 +728,10 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
if (ok != 0)
continue;
- if (st->ss->require_journal) {
- require_journal_dev = st->ss->require_journal(st);
- if (require_journal_dev == 2) {
- pr_err("BUG: Superblock not loaded in Incremental.c:count_active\n");
- return 0;
- }
- }
-
info.array.raid_disks = raid_disks;
st->ss->getinfo_super(st, &info, devmap + raid_disks * devnum);
if (info.disk.raid_disk == MD_DISK_ROLE_JOURNAL)
- has_journal_dev = 1;
+ bestinfo->journal_clean = 1;
if (!avail) {
raid_disks = info.array.raid_disks;
avail = xcalloc(raid_disks, 1);
@@ -790,9 +782,6 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
st->ss->free_super(st);
}
- if (require_journal_dev && !has_journal_dev)
- *journal_device_missing = 1;
-
if (!avail)
return 0;
/* We need to reject any device that thinks the best device is
diff --git a/mdadm.h b/mdadm.h
index b1028be..b4f2011 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -250,7 +250,8 @@ struct mdinfo {
#define DS_REMOVE 1024
#define DS_UNBLOCK 2048
int prev_state, curr_state, next_state;
-
+ int journal_device_required;
+ int journal_clean;
};
struct createinfo {
@@ -970,9 +971,6 @@ extern struct superswitch {
/* validate container after assemble */
int (*validate_container)(struct mdinfo *info);
- /* whether the array require a journal device */
- int (*require_journal)(struct supertype *st);
-
int swapuuid; /* true if uuid is bigending rather than hostendian */
int external;
const char *name; /* canonical metadata name */
diff --git a/super1.c b/super1.c
index 47acdec..2799529 100644
--- a/super1.c
+++ b/super1.c
@@ -140,21 +140,6 @@ struct misc_dev_info {
|MD_FEATURE_BITMAP_VERSIONED \
|MD_FEATURE_JOURNAL \
)
-/* return value:
- * 0, jouranl not required
- * 1, journal required
- * 2, no superblock loated (st->sb == NULL)
- */
-static int require_journal1(struct supertype *st)
-{
- struct mdp_superblock_1 *sb = st->sb;
-
- if (sb->feature_map & MD_FEATURE_JOURNAL)
- return 1;
- else if (!sb)
- return 2; /* no sb loaded */
- return 0;
-}
static int role_from_sb(struct mdp_superblock_1 *sb)
{
@@ -1086,6 +1071,9 @@ static void getinfo_super1(struct supertype *st, struct mdinfo *info, char *map)
}
info->array.working_disks = working;
+ if (sb->feature_map & __le32_to_cpu(MD_FEATURE_JOURNAL))
+ info->journal_device_required = 1;
+ info->journal_clean = 0;
}
static struct mdinfo *container_content1(struct supertype *st, char *subarray)
@@ -2586,7 +2574,6 @@ struct superswitch super1 = {
.locate_bitmap = locate_bitmap1,
.write_bitmap = write_bitmap1,
.free_super = free_super1,
- .require_journal = require_journal1,
#if __BYTE_ORDER == BIG_ENDIAN
.swapuuid = 0,
#else
--
2.4.6
^ permalink raw reply related
* [PATCH 2/3] mdadm: Add description of write journal to md.4
From: Song Liu @ 2015-10-21 18:35 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, shli, hch, dan.j.williams, kernel-team, Song Liu
In-Reply-To: <1445452516-2816184-1-git-send-email-songliubraving@fb.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
md.4 | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/md.4 b/md.4
index e955c3b..f1b88ee 100644
--- a/md.4
+++ b/md.4
@@ -874,6 +874,26 @@ The list is particularly useful when recovering to a spare. If a few blocks
cannot be read from the other devices, the bulk of the recovery can
complete and those few bad blocks will be recorded in the bad block list.
+.SS RAID456 WRITE JOURNAL
+
+Due to non-atomicity nature of RAID write operations, interruption of
+write operations (system crash, etc.) to RAID456 array can lead to
+inconsistent parity and data loss (so called RAID-5 write hole).
+
+To plug the write hole, from Linux 4.4 (to be confirmed),
+.I md
+supports write ahead journal for RAID456. When the array is created,
+an additional journal device can be added to the array through
+.IR write-journal
+option. The RAID write journal works similar to file system journals.
+Before writing to the data disks, md persists data AND parity of the
+stripe to the journal device. After crashes, md searches the journal
+device for incomplete write operations, and replay them to the data
+disks.
+
+When the journal device fails, the RAID array is forced to run in
+read-only mode.
+
.SS WRITE-BEHIND
From Linux 2.6.14,
--
2.4.6
^ permalink raw reply related
* [PATCH 3/3] mdadm: add test script for raid456 journal
From: Song Liu @ 2015-10-21 18:35 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, shli, hch, dan.j.williams, kernel-team, Song Liu
In-Reply-To: <1445452516-2816184-1-git-send-email-songliubraving@fb.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
test | 9 ++++++++
tests/20raid5journal | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 73 insertions(+)
create mode 100644 tests/20raid5journal
diff --git a/test b/test
index d0a6cb8..13f1bda 100755
--- a/test
+++ b/test
@@ -246,6 +246,15 @@ check() {
fi
;;
+ readonly )
+ grep -s "read-only" > /dev/null /proc/mdstat || {
+ echo >&2 "ERROR array is not read-only!"; cat /proc/mdstat ; exit 1; }
+ ;;
+
+ inactive )
+ grep -s "inactive" > /dev/null /proc/mdstat || {
+ echo >&2 "ERROR array is not inactive!"; cat /proc/mdstat ; exit 1; }
+ ;;
* ) echo >&2 ERROR unknown check $1 ; exit 1;
esac
}
diff --git a/tests/20raid5journal b/tests/20raid5journal
new file mode 100644
index 0000000..f751ace
--- /dev/null
+++ b/tests/20raid5journal
@@ -0,0 +1,64 @@
+# check write journal of raid456
+
+# test --detail
+test_detail_shows_journal() {
+ mdadm -D $1 | grep journal || {
+ echo >&2 "ERROR --detail does show journal device!"; mdadm -D $1 ; exit 1; }
+}
+
+# test --examine
+test_examine_shows_journal() {
+ mdadm -E $1 | grep Journal || {
+ echo >&2 "ERROR --examine does show Journal device!"; mdadm -E $1 ; exit 1; }
+}
+
+# test --create
+create_with_journal_and_stop() {
+ mdadm -CR $md0 -l5 -n4 $dev0 $dev1 $dev2 $dev3 --write-journal $dev4
+ check wait
+ tar cf - /etc > $md0
+ ./raid6check $md0 0 0 | grep 'Error detected' && exit 1
+ test_detail_shows_journal $md0
+ test_examine_shows_journal $dev4
+ mdadm -S $md0
+}
+
+# test --assemble
+test_assemble() {
+ create_with_journal_and_stop
+ if mdadm -A $md0 $dev0 $dev1 $dev2 $dev3
+ then
+ echo >&2 "ERROR should return 1 when journal is missing!"; cat /proc/mdstat ; exit 1;
+ fi
+ mdadm -S $md0
+
+ mdadm -A $md0 $dev0 $dev1 $dev2 $dev3 --force
+ check readonly
+ mdadm -S $md0
+}
+
+# test --incremental
+test_incremental() {
+ create_with_journal_and_stop
+ for d in $dev0 $dev1 $dev2 $dev3
+ do
+ mdadm -I $d
+ done
+ check inactive
+ mdadm -I $dev4
+ check raid5
+ mdadm -S $md0
+
+ # test --incremental with journal missing
+ for d in $dev0 $dev1 $dev2 $dev3
+ do
+ mdadm -I $d
+ done
+ mdadm -R $md0
+ check readonly
+ mdadm -S $md0
+}
+
+create_with_journal_and_stop
+test_assemble
+test_incremental
--
2.4.6
^ permalink raw reply related
* Re: How to recover after md crash during reshape?
From: Neil Brown @ 2015-10-21 20:26 UTC (permalink / raw)
To: Phil Turmel, andras, linux-raid
In-Reply-To: <56278284.5020306@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 1909 bytes --]
Phil Turmel <philip@turmel.org> writes:
> Good morning Neil,
>
> On 10/20/2015 09:35 PM, Neil Brown wrote:
>
>> Nothing dumb about that - you don't need a --backup option.
>> If you did, mdadm would have complained.
>>
>> You only need --backup when the size of the array is unchanged or
>> decreasing.
>
>> mdadm reads the first few stripes and stores them somewhere in each of
>> the spares. md (in the kernel) then reads those stripes again and
>> writes them out in the new configuration. It appears that one of the
>> writes failed, others might have succeeded. This may not have corrupted
>> anything (the first few blocks are in the same position for both the old
>> and new layout) but it might have done.
>
>> If you do want to look for the backup, it is around about the middle of
>> the device and has some metadata which contains the string
>> "md_backup_data-1". If you find that, you are close to getting the
>> backup data back.
>
> Hmmm. This feature has advanced beyond my last look at the code. I was
> under the impression the backup option was only optional when mdadm
> could move the data offset. Does this new algorithm apply to v0.90
> metadata, a v3.2 kernel, and v3.2.5 mdadm?
>
It isn't a new algorithm, it is the original algorithm.
In mdadm-2.4-pre1 (march 2006), you couldn't specify a backup file, but
you could grow a raid5 to more devices.
That was changed by a patch with comment:
Allow resize to backup to a file.
To support resizing an array without a spare, mdadm now understands
--backup-file=
which should point to a file for storing a backup of critical data.
This can be given to --grow which will create the file, or
--assemble which will restore from the file if needed.
The backup-file was subsequently used to support in-place reshapes and
array shrinking.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-21 20:37 UTC (permalink / raw)
To: Neil Brown, andras, linux-raid
In-Reply-To: <87k2qg0xz6.fsf@notabene.neil.brown.name>
On 10/21/2015 04:26 PM, Neil Brown wrote:
> Phil Turmel <philip@turmel.org> writes:
>> Hmmm. This feature has advanced beyond my last look at the code. I was
>> under the impression the backup option was only optional when mdadm
>> could move the data offset. Does this new algorithm apply to v0.90
>> metadata, a v3.2 kernel, and v3.2.5 mdadm?
>>
>
> It isn't a new algorithm, it is the original algorithm.
>
> In mdadm-2.4-pre1 (march 2006), you couldn't specify a backup file, but
> you could grow a raid5 to more devices.
> That was changed by a patch with comment:
>
> Allow resize to backup to a file.
>
> To support resizing an array without a spare, mdadm now understands
> --backup-file=
> which should point to a file for storing a backup of critical data.
> This can be given to --grow which will create the file, or
> --assemble which will restore from the file if needed.
>
> The backup-file was subsequently used to support in-place reshapes and
> array shrinking.
Ah, ok. I wasn't using parity raid that far back, and never noticed
that growing to more devices worked that way.
Thanks for clarifying.
Phil
^ permalink raw reply
* Re: [PATCH 0/3] improve mdadm support of write journal
From: Neil Brown @ 2015-10-22 1:22 UTC (permalink / raw)
To: linux-raid; +Cc: shli, hch, dan.j.williams, kernel-team, Song Liu
In-Reply-To: <1445452516-2816184-1-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]
Song Liu <songliubraving@fb.com> writes:
> Per discussion in
>
> http://marc.info/?l=linux-raid&m=144522071120149
>
> and related threads, the following patches add more support
> for write journal (test script, man page, small refactoring).
>
> Song Liu (3):
> mdadm: refactor write journal code in Assemble and Incremental
> mdadm: Add description of write journal to md.4
> mdadm: add test script for raid456 journal
>
> Assemble.c | 24 +++++---------------
> Incremental.c | 25 ++++++--------------
> md.4 | 20 ++++++++++++++++
> mdadm.h | 6 ++---
> super1.c | 19 +++-------------
> test | 9 ++++++++
> tests/20raid5journal | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> 7 files changed, 111 insertions(+), 56 deletions(-)
> create mode 100644 tests/20raid5journal
>
> --
> 2.4.6
Thanks for these. I have applied these patches and pushed them out.
I did make one small change.
The two entries you added to 'struct mdinfo' you put at the end, which
is in a section labeled:
/* Device info for mdmon: */
They don't really fit there. I moved them earlier.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* [PATCH] md-cluster: Only one thread should request DLM lock
From: rgoldwyn @ 2015-10-22 13:31 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: gqjiang, Goldwyn Rodrigues
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
If a DLM lock is in progress, requesting the same DLM lock will
result in -EBUSY. Use a mutex to make sure only one thread requests
for dlm_lock() function at a time.
This will fix the error -EBUSY returned from DLM's validate_lock_args().
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md-cluster.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 35ac2e8..9b977a2 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -29,6 +29,7 @@ struct dlm_lock_resource {
void (*bast)(void *arg, int mode); /* blocking AST function pointer*/
struct mddev *mddev; /* pointing back to mddev. */
int mode;
+ struct mutex res_lock;
};
struct suspend_info {
@@ -102,14 +103,19 @@ static int dlm_lock_sync(struct dlm_lock_resource *res, int mode)
{
int ret = 0;
+ mutex_lock(&res->res_lock);
+
ret = dlm_lock(res->ls, mode, &res->lksb,
res->flags, res->name, strlen(res->name),
0, sync_ast, res, res->bast);
- if (ret)
+ if (ret) {
+ mutex_unlock(&res->res_lock);
return ret;
+ }
wait_for_completion(&res->completion);
if (res->lksb.sb_status == 0)
res->mode = mode;
+ mutex_unlock(&res->res_lock);
return res->lksb.sb_status;
}
@@ -134,6 +140,7 @@ static struct dlm_lock_resource *lockres_init(struct mddev *mddev,
res->mode = DLM_LOCK_IV;
namelen = strlen(name);
res->name = kzalloc(namelen + 1, GFP_KERNEL);
+ mutex_init(&res->res_lock);
if (!res->name) {
pr_err("md-cluster: Unable to allocate resource name for resource %s\n", name);
goto out_err;
--
1.8.5.6
^ permalink raw reply related
* [PATCH] md-cluster: Call update_raid_disks() if another node --grow's raid_disks
From: rgoldwyn @ 2015-10-22 13:31 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: gqjiang, Goldwyn Rodrigues
In-Reply-To: <1445520669-4406-1-git-send-email-rgoldwyn@suse.de>
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
To incorporate --grow feature executed on one node, other nodes need to
acknowledge the change in number of disks. Call update_raid_disks()
to update internal data structures.
This leads to call check_reshape() -> md_allow_write() -> md_update_sb(),
this results in a deadlock. This is done so it can safely allocate memory
(which might trigger writeback which might write to raid1). This is
not required for md with a bitmap.
In the clustered case, we don't perform md_update_sb() in md_allow_write(),
but in do_md_run(). Also we disable safemode for clustered mode.
mddev->recovery_cp need not be set in check_sb_changes() because this
is required only when a node reads another node's bitmap. mddev->recovery_cp
(which is read from sb->resync_offset), is set only if mddev is in_sync.
Since we disabled safemode, in_sync is set to zero.
In a clustered environment, the MD may not be in sync because another
node could be writing to it. So make sure that in_sync is not set in
case of clustered node in __md_stop_writes().
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md.c | 22 ++++++++++++++++------
drivers/md/raid1.c | 8 +++++---
2 files changed, 21 insertions(+), 9 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index a71b36f..0c70856 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2230,7 +2230,6 @@ static bool does_sb_need_changing(struct mddev *mddev)
/* Check if any mddev parameters have changed */
if ((mddev->dev_sectors != le64_to_cpu(sb->size)) ||
(mddev->reshape_position != le64_to_cpu(sb->reshape_position)) ||
- (mddev->recovery_cp != le64_to_cpu(sb->resync_offset)) ||
(mddev->layout != le64_to_cpu(sb->layout)) ||
(mddev->raid_disks != le32_to_cpu(sb->raid_disks)) ||
(mddev->chunk_sectors != le32_to_cpu(sb->chunksize)))
@@ -3314,6 +3313,11 @@ safe_delay_store(struct mddev *mddev, const char *cbuf, size_t len)
{
unsigned long msec;
+ if (mddev_is_clustered(mddev)) {
+ pr_info("md: Safemode is disabled for clustered mode\n");
+ return -EINVAL;
+ }
+
if (strict_strtoul_scaled(cbuf, &msec, 3) < 0)
return -EINVAL;
if (msec == 0)
@@ -5224,7 +5228,10 @@ int md_run(struct mddev *mddev)
atomic_set(&mddev->max_corr_read_errors,
MD_DEFAULT_MAX_CORRECTED_READ_ERRORS);
mddev->safemode = 0;
- mddev->safemode_delay = (200 * HZ)/1000 +1; /* 200 msec delay */
+ if (mddev_is_clustered(mddev))
+ mddev->safemode_delay = 0;
+ else
+ mddev->safemode_delay = (200 * HZ)/1000 +1; /* 200 msec delay */
mddev->in_sync = 1;
smp_wmb();
spin_lock(&mddev->lock);
@@ -5267,6 +5274,9 @@ static int do_md_run(struct mddev *mddev)
goto out;
}
+ if (mddev_is_clustered(mddev))
+ md_allow_write(mddev);
+
md_wakeup_thread(mddev->thread);
md_wakeup_thread(mddev->sync_thread); /* possibly kick off a reshape */
@@ -5363,7 +5373,8 @@ static void __md_stop_writes(struct mddev *mddev)
md_super_wait(mddev);
if (mddev->ro == 0 &&
- (!mddev->in_sync || (mddev->flags & MD_UPDATE_SB_FLAGS))) {
+ ((!mddev->in_sync && !mddev_is_clustered(mddev)) ||
+ (mddev->flags & MD_UPDATE_SB_FLAGS))) {
/* mark array as shutdown cleanly */
mddev->in_sync = 1;
md_update_sb(mddev, 1);
@@ -9007,9 +9018,8 @@ static void check_sb_changes(struct mddev *mddev, struct md_rdev *rdev)
}
}
- /* recovery_cp changed */
- if (le64_to_cpu(sb->resync_offset) != mddev->recovery_cp)
- mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
+ if (mddev->raid_disks != le32_to_cpu(sb->raid_disks))
+ update_raid_disks(mddev, le32_to_cpu(sb->raid_disks));
/* Finally set the event to be up to date */
mddev->events = le64_to_cpu(sb->events);
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index a2d813c..5f38430 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -3044,9 +3044,11 @@ static int raid1_reshape(struct mddev *mddev)
return -EINVAL;
}
- err = md_allow_write(mddev);
- if (err)
- return err;
+ if (!mddev_is_clustered(mddev)) {
+ err = md_allow_write(mddev);
+ if (err)
+ return err;
+ }
raid_disks = mddev->raid_disks + mddev->delta_disks;
--
1.8.5.6
^ permalink raw reply related
* Re: [PATCH 0/2] raid1/10: Handle write errors correctly in narrow_write_error()
From: Jes Sorensen @ 2015-10-22 15:59 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, William.Kuzeja, xni, nate.dailey
In-Reply-To: <87pp092sid.fsf@notabene.neil.brown.name>
Neil Brown <neilb@suse.de> writes:
> Jes.Sorensen@redhat.com writes:
>
>> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>>
>> Hi,
>>
>> Bill Kuzeja reported a problem to me about data corruption when
>> repeatedly removing and re-adding devices in raid1 arrays. It showed
>> up to be caused by the return value of submit_bio_wait() being handled
>> incorrectly. Tracking this down is credit of Bill!
>>
>> Looks like commit 9e882242c6193ae6f416f2d8d8db0d9126bd996b changed the
>> return of submit_bio_wait() to return != 0 on error, whereas before it
>> returned 0 on error.
>>
>> This fix should be suitable for -stable as far back as 3.9
>
> 3.10?
>
> Thanks to both of you!
>
> I took the liberty of changing the patches a little so they are now:
>
> - if (submit_bio_wait(WRITE, wbio) == 0)
> + if (submit_bio_wait(WRITE, wbio) < 0)
>
> because when there is no explicit test I tend to expect a Bool but these
> values are not Bool.
>
> Patches are in my for-linus branch and will be forwarded sometime this
> week.
>
> This bug only causes a problem when bad-block logs are active, so
> hopefully it won't have caused too much corruption yet -- you would need
> to be using a newish mdadm.
Neil,
An additional twist on this one - Nate ran more tests on this, but was
still able to hit data corruption. He suggests the it is a mistake to
set 'ok = rdev_set_badblocks()' and it should instead be set to 0 if
submit_bio_wait() fails. Like this:
--- raid1.c
+++ raid1.c
@@ -2234,11 +2234,12 @@
bio_trim(wbio, sector - r1_bio->sector, sectors);
wbio->bi_sector += rdev->data_offset;
wbio->bi_bdev = rdev->bdev;
if (submit_bio_wait(WRITE, wbio) < 0) {
/* failure! */
- ok = rdev_set_badblocks(rdev, sector,
- sectors, 0)
- && ok;
+ ok = 0;
+ rdev_set_badblocks(rdev, sector,
+ sectors, 0);
+ }
Question is whether this change has any negative impact in case of a
real write failure?
I have actual patches, I'll send as a reply to this one.
Cheers,
Jes
^ permalink raw reply
* [PATCH 1/2] md/raid1: Do not clear bitmap bit if submit_bio_wait() fails
From: Jes.Sorensen @ 2015-10-22 16:01 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, xni, William.Kuzeja, nate.dailey
In-Reply-To: <wrfjpp06lws6.fsf@redhat.com>
From: Jes Sorensen <Jes.Sorensen@redhat.com>
If submit_bio_wait fails(), we should always set 'ok' to 0, as this
means the write failed. This way the bitmap bit won't be cleared and
the chunk will eventually be resynced.
This is a slightly modified version of a patch provided by Nate
Dailey.
Reported-by: Nate Dailey <nate.dailey@stratus.com>
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid1.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index cfca6ed..80a61a2 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2195,11 +2195,11 @@ static int narrow_write_error(struct r1bio *r1_bio, int i)
bio_trim(wbio, sector - r1_bio->sector, sectors);
wbio->bi_iter.bi_sector += rdev->data_offset;
wbio->bi_bdev = rdev->bdev;
- if (submit_bio_wait(WRITE, wbio) < 0)
+ if (submit_bio_wait(WRITE, wbio) < 0) {
/* failure! */
- ok = rdev_set_badblocks(rdev, sector,
- sectors, 0)
- && ok;
+ ok = 0;
+ rdev_set_badblocks(rdev, sector, sectors, 0);
+ }
bio_put(wbio);
sect_to_write -= sectors;
--
2.4.3
^ permalink raw reply related
* [PATCH 2/2] md/raid10: Do not clear bitmap bit if submit_bio_wait() fails
From: Jes.Sorensen @ 2015-10-22 16:01 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, xni, William.Kuzeja, nate.dailey
In-Reply-To: <wrfjpp06lws6.fsf@redhat.com>
From: Jes Sorensen <Jes.Sorensen@redhat.com>
If submit_bio_wait fails(), we should always set 'ok' to 0, as this
means the write failed. This way the bitmap bit won't be cleared and
the chunk will eventually be resynced.
This is a slightly modified version of a patch provided by Nate
Dailey for drivers/md/raid1.c.
Reported-by: Nate Dailey <nate.dailey@stratus.com>
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid10.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 3afce75..c1adcd3 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2470,11 +2470,11 @@ static int narrow_write_error(struct r10bio *r10_bio, int i)
choose_data_offset(r10_bio, rdev) +
(sector - r10_bio->sector));
wbio->bi_bdev = rdev->bdev;
- if (submit_bio_wait(WRITE, wbio) < 0)
+ if (submit_bio_wait(WRITE, wbio) < 0) {
/* Failure! */
- ok = rdev_set_badblocks(rdev, sector,
- sectors, 0)
- && ok;
+ ok = 0;
+ rdev_set_badblocks(rdev, sector, sectors, 0);
+ }
bio_put(wbio);
sect_to_write -= sectors;
--
2.4.3
^ permalink raw reply related
* Re: Hung RAID5 array with discard
From: Peter Kieser @ 2015-10-22 16:07 UTC (permalink / raw)
To: NeilBrown, Terry Hardie; +Cc: linux-raid
In-Reply-To: <20150323135754.02afa72e@notabene.brown>
[-- Attachment #1: Type: text/plain, Size: 14710 bytes --]
FYI I ran into this problem in 3.18.22 when forcing RAID5 TRIM support
w/ raid456.devices_handle_discard_safely on Intel DC S3500 SSDs.
-Peter
On 2015-03-22 7:57 PM, NeilBrown wrote:
> On Wed, 4 Mar 2015 13:47:08 -0800 Terry Hardie <thardie@instartlogic.com>
> wrote:
>
>> Well, I'm dissapointed no one responded to this. This basically means
>> linux RAID 4/5/6 and discard is fundamentally broken, and no one wants
>> to acknowledge it.
> It might just mean that no-one noticed your email, or that they were busy, or
> were just about to leave on Christmas holidays or ......
>
> If you don't get a response, resending after a reasonable period (couple of
> weeks) is perfectly acceptable.
>> I hope someone finds this post while I still have my lab available and
>> I can help them troubleshoot this issue.
>>
>> I tried this again today on 3.13.0-44-generic (Ubuntu) and was easily
>> able to reproduce it.
> Can you try with a more recent kernel? 3.13.0 is over year old and there is
> at least one raid5 bugfix that went into the 3.13-stable series.
>
> If you can reproduce with 3.19, I'll definitely look into it.
>
> Thanks for the report,
>
> NeilBrown
>
>> On Wed, Dec 17, 2014 at 7:08 PM, Terry Hardie <thardie@instartlogic.com> wrote:
>>> Hi,
>>>
>>> I am testing 3 SSDs (1TB Crucial M550 with DRZAT, and I tested they do
>>> return zeros after discard) with RAID5 and discard. I create the array
>>> with a 64k chunk size, and it starts to sync. During it's initial
>>> reconstruction, I do a mkfs.ext4, which starts to do the "Discarding
>>> device blocks". After a short period (I believe when the mkfs reaches
>>> the point where the reconstruction is at, all IO to the disks freezes,
>>> and mkfs does not advance. iostat shows 2 of the 3 drives at 100%
>>> utilization with no data read or written. After 2 minutes, I get the
>>> hung task dump. Most CPUs are idle, and here are a few which are not,
>>> which look like a deadlock to me:
>>>
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154399] INFO:
>>> rcu_sched detected stalls on CPUs/tasks: { 4 5} (detected by 3,
>>> t=285032 jiffies, g=1160, c=1159, q=0)
>>>
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154902] NMI
>>> backtrace for cpu 4
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154904] CPU: 4 PID:
>>> 2146 Comm: md3_raid5 Tainted: G W IOX 3.13.0-43-generic
>>> #72~precise1
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154905] Hardware
>>> name: Supermicro SYS-2028TP-HC1R/X10DRT-P, BIOS 1.0a 08/28/2014
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154906] task:
>>> ffff88202594c800 ti: ffff8810245a0000 task.ti: ffff8810245a0000
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154907] RIP:
>>> 0010:[<ffffffff817644c1>] [<ffffffff817644c1>]
>>> _raw_spin_lock_irqsave+0x41/0x60
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154910] RSP:
>>> 0018:ffff8810245a1cc8 EFLAGS: 00000006
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154911] RAX:
>>> 0000000000002ec5 RBX: ffff882028a6ec00 RCX: 0000000000007b78
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154912] RDX:
>>> 0000000000000202 RSI: 0000000000007b78 RDI: ffff882028a6ec10
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154913] RBP:
>>> ffff8810245a1cc8 R08: 0000000000007b76 R09: ffff882023629170
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154914] R10:
>>> 0000000000000000 R11: ffff882028a6ec00 R12: ffff882028a6ee68
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154915] R13:
>>> 0000000000000003 R14: 0000000000000002 R15: ffff882028a6ec10
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154916] FS:
>>> 0000000000000000(0000) GS:ffff88103fc80000(0000)
>>> knlGS:0000000000000000
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154917] CS: 0010
>>> DS: 0000 ES: 0000 CR0: 0000000080050033
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154918] CR2:
>>> 00007f208c2d0000 CR3: 0000000001c0d000 CR4: 00000000001407e0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154919] Stack:
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154920]
>>> ffff8810245a1d18 ffffffffa0149890 0000000000000002 ffff882028a6ee88
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154923]
>>> ffff882028a6ee68 ffff882028a6ec00 0000000000000008 ffff882028a6ee68
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154926]
>>> 0000000000000000 ffff882028a6ee50 ffff8810245a1d98 ffffffffa015212f
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154929] Call Trace:
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154933]
>>> [<ffffffffa0149890>] release_inactive_stripe_list+0x50/0x160 [raid456]
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154937]
>>> [<ffffffffa015212f>] handle_active_stripes.isra.38+0x7f/0x190
>>> [raid456]
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154940]
>>> [<ffffffffa0152758>] raid5d+0x198/0x2f0 [raid456]
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154942]
>>> [<ffffffff815d30a7>] md_thread+0x117/0x150
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154945]
>>> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154947]
>>> [<ffffffff815d2f90>] ? md_rdev_init+0x110/0x110
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154949]
>>> [<ffffffff8108fb59>] kthread+0xc9/0xe0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154952]
>>> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154954]
>>> [<ffffffff8176d5bc>] ret_from_fork+0x7c/0xb0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154956]
>>> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154957] Code: 1f 44
>>> 00 00 b8 00 00 02 00 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 75 05 48 89
>>> d0 5d c3 83 e1 fe 0f b7 f1 b8 00 80 00 00 44 0f b7 07 <66> 44 39 c1 74
>>> e6 f3 90 83 e8 01 75 ef 0f 1f 80 00 00 00 00 eb
>>>
>>>
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155001] NMI
>>> backtrace for cpu 5
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155003] CPU: 5 PID:
>>> 2147 Comm: md3_resync Tainted: G W IOX 3.13.0-43-generic
>>> #72~precise1
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155004] Hardware
>>> name: Supermicro SYS-2028TP-HC1R/X10DRT-P, BIOS 1.0a 08/28/2014
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155005] task:
>>> ffff88202594b000 ti: ffff8810274a0000 task.ti: ffff8810274a0000
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155006] RIP:
>>> 0010:[<ffffffffa01483b7>] [<ffffffffa01483b7>]
>>> __find_stripe+0x57/0xa0 [raid456]
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155010] RSP:
>>> 0018:ffff8810274a1b68 EFLAGS: 00000006
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155011] RAX:
>>> ffff882027092da0 RBX: 0000000000a30c10 RCX: 0000000000000001
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155012] RDX:
>>> 0000000000000c10 RSI: 0000000000a30c10 RDI: ffff882028a6ec00
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155013] RBP:
>>> ffff8810274a1b88 R08: 0000000000000000 R09: 0000000000000000
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155014] R10:
>>> 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155015] R13:
>>> ffff882028a6ec00 R14: 0000000000000000 R15: ffff882028a6eda8
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155016] FS:
>>> 0000000000000000(0000) GS:ffff88103fca0000(0000)
>>> knlGS:0000000000000000
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155017] CS: 0010
>>> DS: 0000 ES: 0000 CR0: 0000000080050033
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155018] CR2:
>>> 00000000006e1dc8 CR3: 0000000001c0d000 CR4: 00000000001407e0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155019] Stack:
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155020]
>>> ffff8810274a1ba8 ffff882028a6ec00 000000007b767b00 ffff882028a6ec10
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155023]
>>> ffff8810274a1c28 ffffffffa0150555 ffff882023773b50 ffff882028a6eda8
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155026]
>>> 0000000200000001 ffff882028a6ec08 0000000000000000 0000000000a30c10
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155029] Call Trace:
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155033]
>>> [<ffffffffa0150555>] get_active_stripe+0x115/0x3e0 [raid456]
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155036]
>>> [<ffffffffa014aea8>] ? release_stripe+0x68/0x100 [raid456]
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155040]
>>> [<ffffffffa0154f3b>] sync_request+0x11b/0x2a0 [raid456]
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155042]
>>> [<ffffffff815d5ccf>] md_do_sync+0x84f/0xdb0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155046]
>>> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155048]
>>> [<ffffffff815d30a7>] md_thread+0x117/0x150
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155050]
>>> [<ffffffff815d2f90>] ? md_rdev_init+0x110/0x110
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155052]
>>> [<ffffffff8108fb59>] kthread+0xc9/0xe0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155054]
>>> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155057]
>>> [<ffffffff8176d5bc>] ret_from_fork+0x7c/0xb0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155059]
>>> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
>>> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155060] Code: e2 f8
>>> 0f 00 00 48 8b 04 02 48 85 c0 75 25 f6 05 29 25 01 00 04 75 3e 31 c0
>>> 48 83 c4 08 5b 41 5c 41 5d 5d c3 66 44 39 60 30 74 ee <48> 8b 00 48 85
>>> c0 74 db 48 39 58 38 75 f2 eb e9 48 89 f2 48 c7
>>>
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670082] INFO: task
>>> mkfs.ext4:2235 blocked for more than 120 seconds.
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670109]
>>> Tainted: G W IOX 3.13.0-43-generic #72~precise1
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670130] "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670155] mkfs.ext4
>>> D ffff881024fe39e0 0 2235 2080 0x00000000
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670158]
>>> ffff882026eafa68 0000000000000082 ffff88103fc73480 ffff882026eaffd8
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670162]
>>> 0000000000013480 0000000000013480 ffff8820293e8000 ffff88202208b000
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670165]
>>> ffff882026eafa78 ffff882028a6ec00 ffff882028a6ed98 ffff882028a6ec0c
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670169] Call Trace:
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670175]
>>> [<ffffffff81760ae9>] schedule+0x29/0x70
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670181]
>>> [<ffffffffa01506e3>] get_active_stripe+0x2a3/0x3e0 [raid456]
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670185]
>>> [<ffffffff8134c152>] ? blk_check_plugged+0x72/0xb0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670189]
>>> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670193]
>>> [<ffffffffa0155e44>] make_discard_request+0x108/0x12c4 [raid456]
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670196]
>>> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670201]
>>> [<ffffffffa0155c91>] make_request+0x581/0x590 [raid456]
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670204]
>>> [<ffffffff8109cfd6>] ? ttwu_do_activate.constprop.82+0x66/0x70
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670207]
>>> [<ffffffff8109d097>] ? ttwu_queue+0xb7/0xd0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670210]
>>> [<ffffffff8109f950>] ? try_to_wake_up+0x190/0x210
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670212]
>>> [<ffffffff815d2c53>] md_make_request+0xd3/0x230
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670216]
>>> [<ffffffff8115b085>] ? mempool_alloc_slab+0x15/0x20
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670219]
>>> [<ffffffff8134ceb7>] generic_make_request.part.62+0x77/0xb0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670222]
>>> [<ffffffff8134d428>] generic_make_request+0x68/0x70
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670225]
>>> [<ffffffff8134d4a8>] submit_bio+0x78/0x160
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670228]
>>> [<ffffffff81202f80>] ? bio_alloc_bioset+0xa0/0x1d0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670232]
>>> [<ffffffff813578c0>] blkdev_issue_discard+0x1f0/0x2a0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670235]
>>> [<ffffffff8135c1f4>] blkdev_ioctl+0x354/0x810
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670238]
>>> [<ffffffff8101361d>] ? __switch_to+0x16d/0x4d0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670241]
>>> [<ffffffff81204370>] block_ioctl+0x40/0x50
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670244]
>>> [<ffffffff811dd5c5>] do_vfs_ioctl+0x75/0x2c0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670247]
>>> [<ffffffff817606be>] ? __schedule+0x38e/0x700
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670249]
>>> [<ffffffff811dd8a1>] SyS_ioctl+0x91/0xb0
>>> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670252]
>>> [<ffffffff8176d66d>] system_call_fastpath+0x1a/0x1f
>>>
>>>
>>>
>>>
>>> If I do the mkfs.ext4 after the initial reconstruction is done, is
>>> gets all the way through. I don't want to put this system into
>>> production, since this could mean this condition could show up in the
>>> future if the array needs to reconstruct again at a future point while
>>> in service.
>>>
>>> This is a test system in a lab, so I'd be happy to try some tests.
>>>
>>> Terry
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4311 bytes --]
^ permalink raw reply
* Remove inactive array created by open
From: Simon Guinot @ 2015-10-22 16:24 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, Remi Reroller, Vincent Donnefort, Yoann Sculo
[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]
Hi Neil,
I'd like to have your advice about destroying an array created by open
at close time if not configured, rather than waiting for a ioctl or a
sysfs configuration. This would allow to get rid of the inactive md
devices created by an "accidental" open.
On the Linux distribution embedded in LaCie NAS, we are able to
observe the following scenario:
1. A RAID array is stopped with a command such as mdadm --stop /dev/mdx.
2. The node /dev/mdx is still available because not removed by mdadm at
stop.
3. /dev/mdx is opened by a process such as udev or mdadm --monitor.
4. An inactive RAID array mdx is created and a "add" uevent is
broadcasted to userland. It is let to userland to understand that
this event must be discarded.
You have to admit that this behaviour is at best awkward :)
I read the commit d3374825ce57
"md: make devices disappear when they are no longer needed" in which
you express some concerns about an infinite loop due to udev always
opening newly created devices. Is that still actual ?
In your opinion, how could we get rid of an inactive RAID array created
by open ? Maybe we could switch the hold_active flag from UNTIL_IOCTL to
0 after some delay (enough to prevent udev from looping) ? In addition,
maybe we could remove the device node from mdadm --stop ? Or maybe
something else :)
If you are interested by any of this solutions or one of yours, I'll
be happy to work on it.
Thanks in advance for your thoughts,
Simon
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* RAID6 grow unexpected results
From: Liwei @ 2015-10-22 19:47 UTC (permalink / raw)
To: linux-raid
Hi list,
I tried growing a RAID6 array by executing the mdadm --add and
mdadm --grow --raid-devices commands. Usually that should run fine and
I can continue with pvresize.
This time however, the actual usable raid array size did not
change even though reshape begins. In my confusion, I decided to
reboot. That didn't solve the problem (and apparently the reshape was
interrupted, but mdadm shows the grown array as clean?)
I then executed mdadm --grow --size=max, which seemed to have
increased the array size, but apparently by too much. Now I'm getting
"attempt to access beyond end of device" errors for each of the
component drives. A new reshape operation has also begun, which I
expect would fail as the array size is now too big.
What did I do wrong and how can I remedy this?
Liwei
^ permalink raw reply
* Re: Remove inactive array created by open
From: Neil Brown @ 2015-10-22 21:23 UTC (permalink / raw)
To: Simon Guinot; +Cc: linux-raid, Remi Reroller, Vincent Donnefort, Yoann Sculo
In-Reply-To: <20151022162445.GA11904@kw.sim.vm.gnt>
[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]
Simon Guinot <simon.guinot@sequanux.org> writes:
> Hi Neil,
>
> I'd like to have your advice about destroying an array created by open
> at close time if not configured, rather than waiting for a ioctl or a
> sysfs configuration. This would allow to get rid of the inactive md
> devices created by an "accidental" open.
>
> On the Linux distribution embedded in LaCie NAS, we are able to
> observe the following scenario:
>
> 1. A RAID array is stopped with a command such as mdadm --stop /dev/mdx.
> 2. The node /dev/mdx is still available because not removed by mdadm at
> stop.
> 3. /dev/mdx is opened by a process such as udev or mdadm --monitor.
> 4. An inactive RAID array mdx is created and a "add" uevent is
> broadcasted to userland. It is let to userland to understand that
> this event must be discarded.
>
> You have to admit that this behaviour is at best awkward :)
No argument there.
>
> I read the commit d3374825ce57
> "md: make devices disappear when they are no longer needed" in which
> you express some concerns about an infinite loop due to udev always
> opening newly created devices. Is that still actual ?
>
> In your opinion, how could we get rid of an inactive RAID array created
> by open ? Maybe we could switch the hold_active flag from UNTIL_IOCTL to
> 0 after some delay (enough to prevent udev from looping) ? In addition,
> maybe we could remove the device node from mdadm --stop ? Or maybe
> something else :)
>
> If you are interested by any of this solutions or one of yours, I'll
> be happy to work on it.
By far the best solution here is to used named md devices. These are
relatively recent and I wouldn't be surprised if you weren't aware of
them.
md devices 9,0 to 9,511 (those are major,minor numbers) are "numeric" md
devices. They have in-kernel names md%d which appear in /proc/mdstat
and /sys/block/
If you create a block-special-device node with these numbers, that will
create the md device if it doesn't already exist.
md devices 9,512 to 9,$BIGNUM are "named" md devices. These have
in-kernel names like md_whatever-you-like.
If you create a block-special-device with device number 9,512 and try to
open it you will get -ENODEV.
To create these you
echo whatever-you-like > /sys/module/md_mod/parameters/new_array
A number 512 or greater will be allocated as the minor number.
These arrays behave as you would want them to. They are only created
when explicitly requested and they disappear when stopped.
mdadm will create this sort of array if you add
CREATE names=yes
to mdadm.conf and don't use numeric device names.
i.e. if you ask for /dev/md0, you will still get 9,0.
But if you ask for /dev/md/home, you will get 9,512 where as
with names=no (the default) you would probably get 9,127.
A timeout for dropping idle numeric md devices might make sense but it
would need to be several seconds at least as udev can sometimes get very
backlogged and would wouldn't want to add to that. Would 5 minutes be
soon enough to meet your need?
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 0/2] raid1/10: Handle write errors correctly in narrow_write_error()
From: Neil Brown @ 2015-10-22 21:36 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid, William.Kuzeja, xni, nate.dailey
In-Reply-To: <wrfjpp06lws6.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Neil Brown <neilb@suse.de> writes:
>> Jes.Sorensen@redhat.com writes:
>>
>>> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>>>
>>> Hi,
>>>
>>> Bill Kuzeja reported a problem to me about data corruption when
>>> repeatedly removing and re-adding devices in raid1 arrays. It showed
>>> up to be caused by the return value of submit_bio_wait() being handled
>>> incorrectly. Tracking this down is credit of Bill!
>>>
>>> Looks like commit 9e882242c6193ae6f416f2d8d8db0d9126bd996b changed the
>>> return of submit_bio_wait() to return != 0 on error, whereas before it
>>> returned 0 on error.
>>>
>>> This fix should be suitable for -stable as far back as 3.9
>>
>> 3.10?
>>
>> Thanks to both of you!
>>
>> I took the liberty of changing the patches a little so they are now:
>>
>> - if (submit_bio_wait(WRITE, wbio) == 0)
>> + if (submit_bio_wait(WRITE, wbio) < 0)
>>
>> because when there is no explicit test I tend to expect a Bool but these
>> values are not Bool.
>>
>> Patches are in my for-linus branch and will be forwarded sometime this
>> week.
>>
>> This bug only causes a problem when bad-block logs are active, so
>> hopefully it won't have caused too much corruption yet -- you would need
>> to be using a newish mdadm.
>
> Neil,
>
> An additional twist on this one - Nate ran more tests on this, but was
> still able to hit data corruption. He suggests the it is a mistake to
> set 'ok = rdev_set_badblocks()' and it should instead be set to 0 if
> submit_bio_wait() fails. Like this:
>
> --- raid1.c
> +++ raid1.c
> @@ -2234,11 +2234,12 @@
> bio_trim(wbio, sector - r1_bio->sector, sectors);
> wbio->bi_sector += rdev->data_offset;
> wbio->bi_bdev = rdev->bdev;
> if (submit_bio_wait(WRITE, wbio) < 0) {
> /* failure! */
> - ok = rdev_set_badblocks(rdev, sector,
> - sectors, 0)
> - && ok;
> + ok = 0;
> + rdev_set_badblocks(rdev, sector,
> + sectors, 0);
> + }
>
> Question is whether this change has any negative impact in case of a
> real write failure?
>
> I have actual patches, I'll send as a reply to this one.
>
If we unconditionally set ok to 0 on a write error, then
narrow_write_error() will return 0 and handle_write finished() will call
md_error() to kick the device out of the array.
And given that we only call narrow_write_error() when we got a write
error, we strongly expect at least one sector to give an error.
So it seems to me that the net result of this patch is to make
bad-block-lists completely ineffective.
What sort of tests are you running, and what sort of corruption do you
see?
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 0/2] raid1/10: Handle write errors correctly in narrow_write_error()
From: Nate Dailey @ 2015-10-22 22:37 UTC (permalink / raw)
To: Neil Brown, Jes Sorensen; +Cc: linux-raid, William.Kuzeja, xni
In-Reply-To: <87r3kmziux.fsf@notabene.neil.brown.name>
The problem is that we aren't getting true write (medium) errors.
In this case we're testing device removals. The write errors happen because the
disk goes away. Narrow_write_error returns 1, the bitmap bit is cleared, and
then when the device is re-added the resync might not include the sectors in
that chunk (there's some luck involved; if other writes to that chunk happen
while the disk is removed, we're okay--bug is easier to hit with smaller bitmap
chunks because of this).
On 10/22/2015 05:36 PM, Neil Brown wrote:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>
>> Neil Brown <neilb@suse.de> writes:
>>> Jes.Sorensen@redhat.com writes:
>>>
>>>> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>>>>
>>>> Hi,
>>>>
>>>> Bill Kuzeja reported a problem to me about data corruption when
>>>> repeatedly removing and re-adding devices in raid1 arrays. It showed
>>>> up to be caused by the return value of submit_bio_wait() being handled
>>>> incorrectly. Tracking this down is credit of Bill!
>>>>
>>>> Looks like commit 9e882242c6193ae6f416f2d8d8db0d9126bd996b changed the
>>>> return of submit_bio_wait() to return != 0 on error, whereas before it
>>>> returned 0 on error.
>>>>
>>>> This fix should be suitable for -stable as far back as 3.9
>>> 3.10?
>>>
>>> Thanks to both of you!
>>>
>>> I took the liberty of changing the patches a little so they are now:
>>>
>>> - if (submit_bio_wait(WRITE, wbio) == 0)
>>> + if (submit_bio_wait(WRITE, wbio) < 0)
>>>
>>> because when there is no explicit test I tend to expect a Bool but these
>>> values are not Bool.
>>>
>>> Patches are in my for-linus branch and will be forwarded sometime this
>>> week.
>>>
>>> This bug only causes a problem when bad-block logs are active, so
>>> hopefully it won't have caused too much corruption yet -- you would need
>>> to be using a newish mdadm.
>> Neil,
>>
>> An additional twist on this one - Nate ran more tests on this, but was
>> still able to hit data corruption. He suggests the it is a mistake to
>> set 'ok = rdev_set_badblocks()' and it should instead be set to 0 if
>> submit_bio_wait() fails. Like this:
>>
>> --- raid1.c
>> +++ raid1.c
>> @@ -2234,11 +2234,12 @@
>> bio_trim(wbio, sector - r1_bio->sector, sectors);
>> wbio->bi_sector += rdev->data_offset;
>> wbio->bi_bdev = rdev->bdev;
>> if (submit_bio_wait(WRITE, wbio) < 0) {
>> /* failure! */
>> - ok = rdev_set_badblocks(rdev, sector,
>> - sectors, 0)
>> - && ok;
>> + ok = 0;
>> + rdev_set_badblocks(rdev, sector,
>> + sectors, 0);
>> + }
>>
>> Question is whether this change has any negative impact in case of a
>> real write failure?
>>
>> I have actual patches, I'll send as a reply to this one.
>>
> If we unconditionally set ok to 0 on a write error, then
> narrow_write_error() will return 0 and handle_write finished() will call
> md_error() to kick the device out of the array.
>
> And given that we only call narrow_write_error() when we got a write
> error, we strongly expect at least one sector to give an error.
>
> So it seems to me that the net result of this patch is to make
> bad-block-lists completely ineffective.
>
> What sort of tests are you running, and what sort of corruption do you
> see?
>
> NeilBrown
^ permalink raw reply
* Re: RAID6 grow unexpected results
From: Neil Brown @ 2015-10-22 22:37 UTC (permalink / raw)
To: Liwei, linux-raid
In-Reply-To: <CAPE0SYz97qwHaQjgguZbQ3rdxKZjDdS9sg5F5dy10W3GP=U3+w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1833 bytes --]
Liwei <xieliwei@gmail.com> writes:
> Hi list,
> I tried growing a RAID6 array by executing the mdadm --add and
> mdadm --grow --raid-devices commands. Usually that should run fine and
> I can continue with pvresize.
That is incorrect. You cannot run "pvresize" until the reshape
completes.
>
> This time however, the actual usable raid array size did not
> change even though reshape begins. In my confusion, I decided to
> reboot. That didn't solve the problem (and apparently the reshape was
> interrupted, but mdadm shows the grown array as clean?)
Clean is expected. It will be in an intermediate state, part has
reshaped to extra devices, part is still the original.
>
> I then executed mdadm --grow --size=max, which seemed to have
> increased the array size, but apparently by too much. Now I'm getting
> "attempt to access beyond end of device" errors for each of the
> component drives. A new reshape operation has also begun, which I
> expect would fail as the array size is now too big.
"--size" isn't the same as "--array-size".
"--size" is the size of the component devices to use.
The "attempt to access beyond end of device" errors are unexpected but
presumably related o the "--size=max". Maybe the 'max' chosen was too
large. That suggests a bug somewhere.
You probably want to reduce that size back to where it was.
>
> What did I do wrong and how can I remedy this?
Could we start with "mdadm --examine" status of all devices, "mdadm
--detail" of the array, and all kernel messages that could possibly
relate to the array since you first started this exercise?
NeilBrown
>
> Liwei
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Remove inactive array created by open
From: Neil Brown @ 2015-10-22 23:38 UTC (permalink / raw)
To: Simon Guinot; +Cc: linux-raid, Remi Reroller, Vincent Donnefort, Yoann Sculo
In-Reply-To: <87vb9yzjfp.fsf@notabene.neil.brown.name>
[-- Attachment #1: Type: text/plain, Size: 303 bytes --]
Neil Brown <neilb@suse.com> writes:
> To create these you
> echo whatever-you-like > /sys/module/md_mod/parameters/new_array
correction. It should be
echo md_whatever-you-like > /sys/module/md_mod/parameters/new_array
strings must start with "md_" and length must not exceed 31.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 0/2] raid1/10: Handle write errors correctly in narrow_write_error()
From: Neil Brown @ 2015-10-23 0:09 UTC (permalink / raw)
To: Nate Dailey, Jes Sorensen; +Cc: linux-raid, William.Kuzeja, xni
In-Reply-To: <56296510.4030702@stratus.com>
[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]
Nate Dailey <nate.dailey@stratus.com> writes:
> The problem is that we aren't getting true write (medium) errors.
>
> In this case we're testing device removals. The write errors happen because the
> disk goes away. Narrow_write_error returns 1, the bitmap bit is cleared, and
> then when the device is re-added the resync might not include the sectors in
> that chunk (there's some luck involved; if other writes to that chunk happen
> while the disk is removed, we're okay--bug is easier to hit with smaller bitmap
> chunks because of this).
>
>
OK, that makes sense.
The device removal will be noticed when the bad block log is written
out.
When a bad-block is recorded we make sure to write that out promptly
before bio_endio() gets called. But not before close_write() has called
bitmap_end_write().
So I guess we need to delay the close_write() call until the
bad-block-log has been written.
I think this patch should do it. Can you test?
Thanks,
NeilBrown
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index c1ad0b075807..1a1c5160c930 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2269,8 +2269,6 @@ static void handle_write_finished(struct r1conf *conf, struct r1bio *r1_bio)
rdev_dec_pending(conf->mirrors[m].rdev,
conf->mddev);
}
- if (test_bit(R1BIO_WriteError, &r1_bio->state))
- close_write(r1_bio);
if (fail) {
spin_lock_irq(&conf->device_lock);
list_add(&r1_bio->retry_list, &conf->bio_end_io_list);
@@ -2396,6 +2394,9 @@ static void raid1d(struct md_thread *thread)
r1_bio = list_first_entry(&tmp, struct r1bio,
retry_list);
list_del(&r1_bio->retry_list);
+ if (mddev->degraded)
+ set_bit(R1BIO_Degraded, &r1_bio->state);
+ close_write(r1_bio);
raid_end_bio_io(r1_bio);
}
}
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply related
* Re: [PATCH] md-cluster: Only one thread should request DLM lock
From: Neil Brown @ 2015-10-23 2:11 UTC (permalink / raw)
To: rgoldwyn, linux-raid; +Cc: gqjiang, Goldwyn Rodrigues
In-Reply-To: <1445520669-4406-1-git-send-email-rgoldwyn@suse.de>
[-- Attachment #1: Type: text/plain, Size: 2811 bytes --]
rgoldwyn@suse.de writes:
> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>
> If a DLM lock is in progress, requesting the same DLM lock will
> result in -EBUSY. Use a mutex to make sure only one thread requests
> for dlm_lock() function at a time.
>
> This will fix the error -EBUSY returned from DLM's
> validate_lock_args().
I can see that we only want one thread calling dlm_lock() with a given
'struct dlm_lock_resource' at a time, otherwise nasty things could
happen.
However if such a race is possible, then aren't there other possibly
complications.
Suppose two threads try to lock the same resource.
Presumably one will try to lock the resource, then the next one (when it
gets the mutex) will discover that it already has the resource, but will
think it has exclusive access - maybe?
Then both threads will eventually try to unlock, and the second one will
unlock something that doesn't have locked.
I'm not certain, but that doesn't sound entirely safe.
Which resources to we actually see races with?
Thanks,
NeilBrown
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md-cluster.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index 35ac2e8..9b977a2 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -29,6 +29,7 @@ struct dlm_lock_resource {
> void (*bast)(void *arg, int mode); /* blocking AST function pointer*/
> struct mddev *mddev; /* pointing back to mddev. */
> int mode;
> + struct mutex res_lock;
> };
>
> struct suspend_info {
> @@ -102,14 +103,19 @@ static int dlm_lock_sync(struct dlm_lock_resource *res, int mode)
> {
> int ret = 0;
>
> + mutex_lock(&res->res_lock);
> +
> ret = dlm_lock(res->ls, mode, &res->lksb,
> res->flags, res->name, strlen(res->name),
> 0, sync_ast, res, res->bast);
> - if (ret)
> + if (ret) {
> + mutex_unlock(&res->res_lock);
> return ret;
> + }
> wait_for_completion(&res->completion);
> if (res->lksb.sb_status == 0)
> res->mode = mode;
> + mutex_unlock(&res->res_lock);
> return res->lksb.sb_status;
> }
>
> @@ -134,6 +140,7 @@ static struct dlm_lock_resource *lockres_init(struct mddev *mddev,
> res->mode = DLM_LOCK_IV;
> namelen = strlen(name);
> res->name = kzalloc(namelen + 1, GFP_KERNEL);
> + mutex_init(&res->res_lock);
> if (!res->name) {
> pr_err("md-cluster: Unable to allocate resource name for resource %s\n", name);
> goto out_err;
> --
> 1.8.5.6
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Hung RAID5 array with discard
From: Neil Brown @ 2015-10-23 4:42 UTC (permalink / raw)
To: Peter Kieser, Terry Hardie; +Cc: linux-raid
In-Reply-To: <562909AC.7040003@kieser.ca>
[-- Attachment #1: Type: text/plain, Size: 390 bytes --]
Peter Kieser <peter@kieser.ca> writes:
> FYI I ran into this problem in 3.18.22 when forcing RAID5 TRIM support
> w/ raid456.devices_handle_discard_safely on Intel DC S3500 SSDs.
Can you please be precise about the problem you experienced.
Complete kernel messages are a minimum.
Anything interesting that might have been happening at the time might
help.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox