* Re: md resync ignoring unreadable sectors
From: Phil Turmel @ 2015-02-07 23:42 UTC (permalink / raw)
To: Roman Mamedov, Eyal Lebedinsky; +Cc: linux-raid
In-Reply-To: <20150208040438.156a8554@natsu>
Hi Roman,
On 02/07/2015 06:04 PM, Roman Mamedov wrote:
> On Sun, 08 Feb 2015 09:39:47 +1100
> Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
>> I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
>> around the bad area also showed it to be all zeroes.
>
> I wouldn't expect mdadm to have any headers or unused areas as far as 133 MB
> into a RAID member.
Look at mdadm -E for that drive and your partition start sector. I bet
Eyal is right. Latest mdadm gives me a 128MB data offset.
Phil
^ permalink raw reply
* Re: md resync ignoring unreadable sectors
From: Roman Mamedov @ 2015-02-07 23:04 UTC (permalink / raw)
To: Eyal Lebedinsky; +Cc: linux-raid
In-Reply-To: <54D69433.9060605@eyal.emu.id.au>
On Sun, 08 Feb 2015 09:39:47 +1100
Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
> The error did not show up during normal operation (or during scrub), only during the smartctl long test.
> What triggered the error for you?
Just appeared during boot-up after a reboot (after 50 days uptime) which was
performed for some hardware upgrades (RAM, SATA controller). The error doesn't
go away after swapping the SATA controller for different one.
> I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
> around the bad area also showed it to be all zeroes.
I wouldn't expect mdadm to have any headers or unused areas as far as 133 MB
into a RAID member.
--
With respect,
Roman
^ permalink raw reply
* Re: md resync ignoring unreadable sectors
From: Eyal Lebedinsky @ 2015-02-07 22:39 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150208024745.01b9d3ff@natsu>
On 08/02/15 08:47, Roman Mamedov wrote:
> Hello,
>
> I've got some bad sectors on one drive:
>
> dd: reading `/dev/sdh1': Input/output error
> 260200+0 records in
> 260200+0 records out
> 133222400 bytes (133 MB) copied, 2.97188 s, 44.8 MB/s
>
> [ 3908.350331] ata9.00: exception Emask 0x0 SAct 0x40000 SErr 0x0 action 0x0
> [ 3908.350385] ata9.00: irq_stat 0x40000008
> [ 3908.350427] ata9.00: failed command: READ FPDMA QUEUED
> [ 3908.350474] ata9.00: cmd 60/06:90:6a:00:04/00:00:00:00:00/40 tag 18 ncq 3072 in
> [ 3908.350474] res 51/40:06:6a:00:04/00:00:00:00:00/40 Emask 0x409 (media error) <F>
> [ 3908.350628] ata9.00: status: { DRDY ERR }
> [ 3908.350669] ata9.00: error: { UNC }
> [ 3908.354643] ata9.00: configured for UDMA/133
> [ 3908.354664] sd 8:0:0:0: [sdh] Unhandled sense code
> [ 3908.354668] sd 8:0:0:0: [sdh]
> [ 3908.354671] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 3908.354674] sd 8:0:0:0: [sdh]
> [ 3908.354677] Sense Key : Medium Error [current] [descriptor]
> [ 3908.354681] Descriptor sense data with sense descriptors (in hex):
> [ 3908.354683] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
> [ 3908.354695] 00 04 00 6a
> [ 3908.354701] sd 8:0:0:0: [sdh]
> [ 3908.354705] Add. Sense: Unrecovered read error - auto reallocate failed
> [ 3908.354708] sd 8:0:0:0: [sdh] CDB:
> [ 3908.354710] Read(10): 28 00 00 04 00 6a 00 00 06 00
> [ 3908.354721] end_request: I/O error, dev sdh, sector 262250
> [ 3908.354773] Buffer I/O error on device sdh1, logical block 260202
> [ 3908.354825] Buffer I/O error on device sdh1, logical block 260203
> [ 3908.354891] Buffer I/O error on device sdh1, logical block 260204
> [ 3908.354942] Buffer I/O error on device sdh1, logical block 260205
> [ 3908.354992] Buffer I/O error on device sdh1, logical block 260206
> [ 3908.355042] Buffer I/O error on device sdh1, logical block 260207
> [ 3908.355125] ata9: EH complete
>
> Generally I believe these should go away when overwritten, but how do I
> overwrite them? The drive is an md RAID1 member:
>
> /dev/md4:
> Version : 1.2
> Creation Time : Mon May 26 13:40:18 2014
> Raid Level : raid1
> Array Size : 1953379936 (1862.89 GiB 2000.26 GB)
> Used Dev Size : 1953379936 (1862.89 GiB 2000.26 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Sun Feb 8 02:39:58 2015
> State : active
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> Name : natsu.romanrm.net:4 (local to host natsu.romanrm.net)
> UUID : 3b8c3166:073249b5:e1384bd6:4611df90
> Events : 50426
>
> Number Major Minor RaidDevice State
> 0 8 49 0 active sync /dev/sdd1
> 1 8 113 1 active sync /dev/sdh1
>
> I thought I would run a 'check' or 'repair', this will read from both drives,
> fail to read from sdh, then try to overwrite the affected areas on sdh. But
> nope:
>
> # echo 0 > /sys/block/md4/md/sync_min
> # echo check > /sys/block/md4/md/sync_action
>
> [ 4059.451036] md: data-check of RAID array md4
> [ 4059.451040] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 4059.451042] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> [ 4059.451046] md: using 128k window, over a total of 1953379936k.
>
> This happily proceeds through the supposedly unreadable area:
>
> md4 : active raid1 sdd1[0] sdh1[1]
> 1953379936 blocks super 1.2 [2/2] [UU]
> [>....................] check = 0.0% (1479680/1953379936) finish=1116.8min speed=29128K/sec
> bitmap: 2/8 pages [8KB], 131072KB chunk
>
> at 1.5GB already, while the unreadable sectors are at ~133MB. And no new ATA
> errors in dmesg. How is this possible?
>
> If I retry the 'dd' command right now, it fails exactly in the same way as
> before (and ATA errors do indeed appear).
Hi,
I had a similar situation. In my case the bad sectors fell in an unused control area, part of the header,
which is not read (or written) by the md normally or by the sync.
The error did not show up during normal operation (or during scrub), only during the smartctl long test.
What triggered the error for you?
I looked up the size of the different parts of the RAID to arrive at that conclusion. Dumping the sectors
around the bad area also showed it to be all zeroes.
I ended up directly zeroing the bad sectors (hdparm --repair-sector ...).
YMMV
--
Eyal Lebedinsky (eyal@eyal.emu.id.au)
^ permalink raw reply
* md resync ignoring unreadable sectors
From: Roman Mamedov @ 2015-02-07 21:47 UTC (permalink / raw)
To: linux-raid
Hello,
I've got some bad sectors on one drive:
dd: reading `/dev/sdh1': Input/output error
260200+0 records in
260200+0 records out
133222400 bytes (133 MB) copied, 2.97188 s, 44.8 MB/s
[ 3908.350331] ata9.00: exception Emask 0x0 SAct 0x40000 SErr 0x0 action 0x0
[ 3908.350385] ata9.00: irq_stat 0x40000008
[ 3908.350427] ata9.00: failed command: READ FPDMA QUEUED
[ 3908.350474] ata9.00: cmd 60/06:90:6a:00:04/00:00:00:00:00/40 tag 18 ncq 3072 in
[ 3908.350474] res 51/40:06:6a:00:04/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[ 3908.350628] ata9.00: status: { DRDY ERR }
[ 3908.350669] ata9.00: error: { UNC }
[ 3908.354643] ata9.00: configured for UDMA/133
[ 3908.354664] sd 8:0:0:0: [sdh] Unhandled sense code
[ 3908.354668] sd 8:0:0:0: [sdh]
[ 3908.354671] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 3908.354674] sd 8:0:0:0: [sdh]
[ 3908.354677] Sense Key : Medium Error [current] [descriptor]
[ 3908.354681] Descriptor sense data with sense descriptors (in hex):
[ 3908.354683] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[ 3908.354695] 00 04 00 6a
[ 3908.354701] sd 8:0:0:0: [sdh]
[ 3908.354705] Add. Sense: Unrecovered read error - auto reallocate failed
[ 3908.354708] sd 8:0:0:0: [sdh] CDB:
[ 3908.354710] Read(10): 28 00 00 04 00 6a 00 00 06 00
[ 3908.354721] end_request: I/O error, dev sdh, sector 262250
[ 3908.354773] Buffer I/O error on device sdh1, logical block 260202
[ 3908.354825] Buffer I/O error on device sdh1, logical block 260203
[ 3908.354891] Buffer I/O error on device sdh1, logical block 260204
[ 3908.354942] Buffer I/O error on device sdh1, logical block 260205
[ 3908.354992] Buffer I/O error on device sdh1, logical block 260206
[ 3908.355042] Buffer I/O error on device sdh1, logical block 260207
[ 3908.355125] ata9: EH complete
Generally I believe these should go away when overwritten, but how do I
overwrite them? The drive is an md RAID1 member:
/dev/md4:
Version : 1.2
Creation Time : Mon May 26 13:40:18 2014
Raid Level : raid1
Array Size : 1953379936 (1862.89 GiB 2000.26 GB)
Used Dev Size : 1953379936 (1862.89 GiB 2000.26 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Feb 8 02:39:58 2015
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : natsu.romanrm.net:4 (local to host natsu.romanrm.net)
UUID : 3b8c3166:073249b5:e1384bd6:4611df90
Events : 50426
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 113 1 active sync /dev/sdh1
I thought I would run a 'check' or 'repair', this will read from both drives,
fail to read from sdh, then try to overwrite the affected areas on sdh. But
nope:
# echo 0 > /sys/block/md4/md/sync_min
# echo check > /sys/block/md4/md/sync_action
[ 4059.451036] md: data-check of RAID array md4
[ 4059.451040] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 4059.451042] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[ 4059.451046] md: using 128k window, over a total of 1953379936k.
This happily proceeds through the supposedly unreadable area:
md4 : active raid1 sdd1[0] sdh1[1]
1953379936 blocks super 1.2 [2/2] [UU]
[>....................] check = 0.0% (1479680/1953379936) finish=1116.8min speed=29128K/sec
bitmap: 2/8 pages [8KB], 131072KB chunk
at 1.5GB already, while the unreadable sectors are at ~133MB. And no new ATA
errors in dmesg. How is this possible?
If I retry the 'dd' command right now, it fails exactly in the same way as
before (and ATA errors do indeed appear).
--
With respect,
Roman
^ permalink raw reply
* Re: [PATCH 3/3] dm ioctl: Delete an unnecessary check before the function call "dm_table_destroy"
From: Mike Snitzer @ 2015-02-06 21:12 UTC (permalink / raw)
To: SF Markus Elfring
Cc: Alasdair Kergon, Neil Brown, dm-devel, linux-raid, Julia Lawall,
kernel-janitors, LKML
In-Reply-To: <54CF9666.5060407@users.sourceforge.net>
On Mon, Feb 02 2015 at 10:23am -0500,
SF Markus Elfring <elfring@users.sourceforge.net> wrote:
> From: Markus Elfring <elfring@users.sourceforge.net>
> Date: Mon, 2 Feb 2015 15:30:37 +0100
>
> The dm_table_destroy() function tests whether its argument is NULL and then
> returns immediately. Thus the test around the call is not needed.
>
> This issue was detected by using the Coccinelle software.
Your proposed patch (while technically correct) hurts code clarity.
Nack.
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: Peter Zijlstra @ 2015-02-06 11:39 UTC (permalink / raw)
To: NeilBrown; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150206085133.2c1ab892@notabene.brown>
On Fri, Feb 06, 2015 at 08:51:33AM +1100, NeilBrown wrote:
> That is exactly what is happening here. However I don't think that is an
> "observed problem" but rather an "observed false-positive".
>
> If nothing inside the outer loop blocks, then in particular
> generic_make_request will not be called, so nothing will be added to the
> queue that blk_schedule_flush_plug flushes.
> So the first time through the loop, a call the 'schedule()' may not actually
> block, but every subsequent time it will.
> So there is no actual problem here.
>
> So I'd be included to add sched_annotate_sleep() in blk_flush_plug_list().
>
> Peter: what do you think is the best way to silence this warning.
> > Call Trace:
> > [<ffffffff8027ee62>] __might_sleep+0x82/0x90
> > [<ffffffff803bee06>] generic_make_request_checks+0x36/0x2d0
> > [<ffffffff803bf0b3>] generic_make_request+0x13/0x100
> > [<ffffffff8054983b>] raid1_unplug+0x12b/0x170
> > [<ffffffff803c1302>] blk_flush_plug_list+0xa2/0x230
> > [<ffffffff80646383>] io_schedule+0x43/0x80
> > [<ffffffff80646787>] bit_wait_io+0x27/0x50
Well, I don't know. I don't particularly like the whole blk_flush_plug()
thing scheduling while on its way to schedule. If you ever end up
calling io_schedule() from it there's 'fun'.
Also, how likely is it to actually schedule when doing all that? This
block layer stuff is somewhat impenetrable for me, too many callbacks.
You have some words on how its unlikely, but I can't even find _where_
it would schedule :/ All I see is a loop calling ->make_request_fn() and
god only knows where that ends up.
So there appear to be two blk_flush_plug() variants, one with an
@from_schedule = true, which seems to really try not to schedule, which
seems to suggest the 'false' one (the one above) is meant to schedule?
If scheduling is the rule rather than the exception, the above is
properly broken.
But again, I don't know.
If you're confident that scheduling is rare for _ALL_ (current and
future) block device implementations, not just the raid one, then you
can annotate blk_flush_plug_list() I suppose.
Otherwise I would suggest adding them one at a time in whatever blk
device thing likes to go schedule on us. Also, add a comment that
explains why its rare for the future us who need to look at it again.
^ permalink raw reply
* Re: [PATCH 00/24] Clustered MD RAID1
From: NeilBrown @ 2015-02-06 2:39 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: lzhong, linux-raid
In-Reply-To: <20141218161456.GA29504@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 3977 bytes --]
On Thu, 18 Dec 2014 10:14:57 -0600 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> Hello,
>
> This is an attempt to make MD-RAID cluster-aware. The advantage of
> redundancy can help highly available systems to improve uptime.
> Currently, the implementation is limited to RAID1 but with further work
> (and some positive feedback), we could extend this to other compatible
> RAID scenarios.
>
> The design document (first patch) is pretty descriptive of how
> the md has been made cluster-aware and how DLM is used to safeguard data
> and communication.
>
> This work requires some patches to the mdadm tool [1]
>
> A quick howto:
>
> 1. With your corosync/pacemaker based cluster running execute:
> # mdadm --create md0 --bitmap=clustered --raid-devices=2 --level=mirror --assume-clean /dev/sda /dev/sdb
>
> 2. On other nodes, issue:
> # mdadm --assemble md0 /dev/sda /dev/sdb
>
> References:
> [1] mdadm tool changes: https://github.com/goldwynr/mdadm branch:cluster-md
> [2] Patches against stable 3.14: https://github.com/goldwynr/linux branch: cluster-md-devel
>
> Regards,
>
hi Goldwyn,
thanks for these - and sorry for the long delay. Lots of leave over
southern summer, and the lots of email etc to deal with.
This patch set is very close and I am tempted to just apply it and then
fix things up with subsequent patches. In order to allow that, could you
please:
- rebase against current upstream
- fix the checkpatch.pl errors and warnings.
The "WARNING: line over 80 characters" are often a judgement call
so I'm not particularly worried about those. Most, if not all, of
the others should be followed just to have consistent layout.
Then I'll queue them up for 3.21, providing I don't find anything that would
hurt non-cluster usage ....
On that topic: why initialise rv to -EINVAL in "metadata_update sends
message...". That looks wrong.
I noticed that a number of times a patch will revert something that a
previous patch added. It would be much nicer to fold these changes back into
the original patch. Often this is just extra blank lines, but occasionally
variable names are changed (md -> mddev). It should be given the final name
when introduced. Every chunk in every patch should be directly relevant to
that patch.
Some other issues, that could possibly be fixed up afterwards:
- Is a clustername 64 bytes or 63 bytes? I would have thought 64,
but the use of strlcpy make is 63 plus a nul. Is that really what is
wanted?
- Based on https://lkml.org/lkml/2012/10/23/580 it might be good to add
"default n" to Kconfig, and possible add a WARN() if anyone tries to use
the code.
- I'm a bit concerned about the behaviour on node failure.
When a node fails, two things must happen w.r.t the bits in that node's
bitmap.
1/ The corresponding regions of the array need to be resynced. You do have
code to do this.
2/ Other nodes must avoid read-balancing on those regions until the
resync has completed.
You do have code for this second bit, but it looks wrong. It avoids
read-balancing if ->area_resyncing(). That isn't sufficient.
The "area_resyncing" is always (I assume) a relatively small region of
the array which will be completely resynced quite quickly. It must be
because writes are blocked to this area. However the region in which
we must disable re-balancing can be much larger. It covers *all* bits
that are set in any unsynced bitmap. So it isn't just the area that is
currently being synced, but all areas that will be synced.
- I think md_reload_sb() might be too simple. It probably should check that
nothing serious has changed. The "mddev->raid_disks = 0" look suspicious.
I'll have to think about this a bit more.
That's all I can see for now. I'll have another look once I have it all in my tree.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: md: 'array_size' sysfs attribute
From: NeilBrown @ 2015-02-05 22:34 UTC (permalink / raw)
To: Dan Carpenter; +Cc: dan.j.williams, linux-raid
In-Reply-To: <20150205110237.GA7933@mwanda>
[-- Attachment #1: Type: text/plain, Size: 4419 bytes --]
On Thu, 5 Feb 2015 14:02:37 +0300 Dan Carpenter <dan.carpenter@oracle.com>
wrote:
> Hello Dan Williams,
>
> The patch b522adcde9c4: "md: 'array_size' sysfs attribute" from Mar
> 31, 2009, leads to the following static checker warning:
>
> drivers/md/md.c:5069 md_run()
> error: we previously assumed 'mddev->pers' could be null (see line 4936)
>
> This code is really old. I don't know why my stupid scripts are marking
> it as a new warning.
Probably because it really is a new warning. The problem is that your
"stupid scripts" (and we need more like them!!) are identifying the wrong
commit.
commit 516253a32c7abf3bc5754b1210106173ed191f7c
Author: NeilBrown <neilb@suse.de>
Date: Mon Dec 15 12:56:58 2014 +1100
md: protect ->pers changes with mddev->lock
is the guilty party (only in -next at present). It delays the setting of
mddev->pers, but doesn't change all intermediate uses for mddev->pers into
pers.
I've just merged:
- (unsigned long long)mddev->pers->size(mddev, 0, 0) / 2);
+ (unsigned long long)pers->size(mddev, 0, 0) / 2);
into that patch.
> We don't set "mddev->pers" to non-NULL until the
> end of the function so it looks like a real bug and that "pers->size"
> was intended. When I fix that bug then it un-silences this warning:
>
> drivers/md/md.c:5080 md_run()
> error: we previously assumed 'mddev->pers' could be null (see line 4936)
>
> And that also is a real bug, but I'm not sure the right fix for that.
>
> Basically, it's bugs all the way down from the code, to the fix, to the
> static checker. *sigh*.
>
> drivers/md/md.c
> 5060 err = pers->run(mddev);
> 5061 if (err)
> 5062 printk(KERN_ERR "md: pers->run() failed ...\n");
> 5063 else if (pers->size(mddev, 0, 0) < mddev->array_sectors) {
> 5064 WARN_ONCE(!mddev->external_size, "%s: default size too small,"
> 5065 " but 'external_size' not in effect?\n", __func__);
> 5066 printk(KERN_ERR
> 5067 "md: invalid array_size %llu > default size %llu\n",
> 5068 (unsigned long long)mddev->array_sectors / 2,
> 5069 (unsigned long long)mddev->pers->size(mddev, 0, 0) / 2);
> ^^^^^^^^^^^^^^^^^^
> This should be "pers->size()".
>
> 5070 err = -EINVAL;
> 5071 }
> 5072 if (err == 0 && pers->sync_request &&
> 5073 (mddev->bitmap_info.file || mddev->bitmap_info.offset)) {
> 5074 err = bitmap_create(mddev);
> 5075 if (err)
> 5076 printk(KERN_ERR "%s: failed to create bitmap (%d)\n",
> 5077 mdname(mddev), err);
> 5078 }
> 5079 if (err) {
> 5080 mddev_detach(mddev);
> ^^^^^^^^^^^^^^^^^^^
> mddev_detach() will oops if mddev->pers() is not set. We could set
> mdev->pers earlier, I suppose.
mddev_detach doesn't really need ->pers in this case. I've merged:
@@ -5134,7 +5134,7 @@ static void mddev_detach(struct mddev *mddev)
wait_event(bitmap->behind_wait,
atomic_read(&bitmap->behind_writes) == 0);
}
- if (mddev->pers->quiesce) {
+ if (mddev->pers && mddev->pers->quiesce) {
mddev->pers->quiesce(mddev, 1);
mddev->pers->quiesce(mddev, 0);
}
Thanks a lot to you and your static checker!!
NeilBrown
>
> 5081 pers->free(mddev, mddev->private);
> 5082 module_put(pers->owner);
> 5083 bitmap_destroy(mddev);
> 5084 return err;
> 5085 }
> 5086 if (mddev->queue) {
> 5087 mddev->queue->backing_dev_info.congested_data = mddev;
> 5088 mddev->queue->backing_dev_info.congested_fn = md_congested;
> 5089 blk_queue_merge_bvec(mddev->queue, md_mergeable_bvec);
> 5090 }
>
> regards,
> dan carpenter
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [PATCH] drivers: md: md-thin: remove duplication with time_after
From: Pierre Chevalier @ 2015-02-05 21:58 UTC (permalink / raw)
To: agk; +Cc: linux-raid, linux-kernel, Pierre Chevalier
The logic that was expressed previously was equivalent of calling
time_after.
Remove duplication.
Signed-off-by: Pierre Chevalier <pierrechevalier83@gmail.com>
---
drivers/md/dm-thin.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
index 07705ee..c4d319b 100644
--- a/drivers/md/dm-thin.c
+++ b/drivers/md/dm-thin.c
@@ -11,6 +11,7 @@
#include <linux/device-mapper.h>
#include <linux/dm-io.h>
#include <linux/dm-kcopyd.h>
+#include <linux/jiffies.h>
#include <linux/log2.h>
#include <linux/list.h>
#include <linux/rculist.h>
@@ -1700,8 +1701,7 @@ static void process_cell_fail(struct thin_c *tc, struct dm_bio_prison_cell *cell
*/
static int need_commit_due_to_time(struct pool *pool)
{
- return jiffies < pool->last_commit_jiffies ||
- jiffies > pool->last_commit_jiffies + COMMIT_PERIOD;
+ return time_after(jiffies, pool->last_commit_jiffies + COMMIT_PERIOD);
}
#define thin_pbd(node) rb_entry((node), struct dm_thin_endio_hook, rb_node)
--
2.2.2
^ permalink raw reply related
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: NeilBrown @ 2015-02-05 21:51 UTC (permalink / raw)
To: Tony Battersby; +Cc: linux-raid, Peter Zijlstra, lkml, axboe
In-Reply-To: <54D3D24E.5060303@cybernetics.com>
[-- Attachment #1: Type: text/plain, Size: 4996 bytes --]
On Thu, 05 Feb 2015 15:27:58 -0500 Tony Battersby <tonyb@cybernetics.com>
wrote:
> I get the might_sleep() warning below when writing some data to an ext3
> filesystem on a RAID1. But everything works OK, so there is no actual
> problem, just a warning.
>
> I see that there has been a fix for a might_sleep() warning in md/bitmap
> since 3.19-rc7, but this is a different warning.
Hi Tony,
this is another false positive caused by
commit 8eb23b9f35aae413140d3fda766a98092c21e9b0
Author: Peter Zijlstra <peterz@infradead.org>
Date: Wed Sep 24 10:18:55 2014 +0200
sched: Debug nested sleeps
It is even described in that commit:
Another observed problem is calling a blocking function from
schedule()->sched_submit_work()->blk_schedule_flush_plug() which will
then destroy the task state for the actual __schedule() call that
comes after it.
That is exactly what is happening here. However I don't think that is an
"observed problem" but rather an "observed false-positive".
If nothing inside the outer loop blocks, then in particular
generic_make_request will not be called, so nothing will be added to the
queue that blk_schedule_flush_plug flushes.
So the first time through the loop, a call the 'schedule()' may not actually
block, but every subsequent time it will.
So there is no actual problem here.
So I'd be included to add sched_annotate_sleep() in blk_flush_plug_list().
Peter: what do you think is the best way to silence this warning.
Thanks,
NeilBrown
>
> ---
>
> > cat /proc/mdstat
> Personalities : [raid1]
> md0 : active raid1 sda1[0] sdb1[1]
> 1959884 blocks super 1.0 [2/2] [UU]
>
> unused devices: <none>
>
> ---
>
> > grep md0 /proc/mounts
> /dev/md0 / ext3 rw,noatime,errors=continue,barrier=1,data=journal 0 0
>
> ---
>
> WARNING: CPU: 3 PID: 1069 at kernel/sched/core.c:7300 __might_sleep+0x82/0x90()
> do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff8028faa1>] prepare_to_wait+0x31/0xa0
> Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi igb i2c_algo_bit ptp pps_core mptsas mptscsih mptbase pm80xx libsas mpt2sas scsi_transport_sas raid_class sg coretemp eeprom w83795 i2c_i801
> CPU: 3 PID: 1069 Comm: kjournald Not tainted 3.19.0-rc7 #1
> Hardware name: Supermicro X8DTH-i/6/iF/6F/X8DTH, BIOS 2.1b 05/04/12
> 0000000000001c84 ffff88032f1df608 ffffffff80645918 0000000000001c84
> ffff88032f1df658 ffff88032f1df648 ffffffff8025ea6b ffff8800bb0b4d58
> 0000000000000000 00000000000006f6 ffffffff80942b6f ffff8803317b8a00
> Call Trace:
> [<ffffffff80645918>] dump_stack+0x4f/0x6f
> [<ffffffff8025ea6b>] warn_slowpath_common+0x8b/0xd0
> [<ffffffff8025eb51>] warn_slowpath_fmt+0x41/0x50
> [<ffffffff8028faa1>] ? prepare_to_wait+0x31/0xa0
> [<ffffffff8028faa1>] ? prepare_to_wait+0x31/0xa0
> [<ffffffff8027ee62>] __might_sleep+0x82/0x90
> [<ffffffff803bee06>] generic_make_request_checks+0x36/0x2d0
> [<ffffffff802943ed>] ? trace_hardirqs_on+0xd/0x10
> [<ffffffff803bf0b3>] generic_make_request+0x13/0x100
> [<ffffffff8054983b>] raid1_unplug+0x12b/0x170
> [<ffffffff803c1302>] blk_flush_plug_list+0xa2/0x230
> [<ffffffff80294315>] ? trace_hardirqs_on_caller+0x105/0x1d0
> [<ffffffff80646760>] ? bit_wait_timeout+0x70/0x70
> [<ffffffff80646383>] io_schedule+0x43/0x80
> [<ffffffff80646787>] bit_wait_io+0x27/0x50
> [<ffffffff80646a7d>] __wait_on_bit+0x5d/0x90
> [<ffffffff803bf160>] ? generic_make_request+0xc0/0x100
> [<ffffffff80646760>] ? bit_wait_timeout+0x70/0x70
> [<ffffffff80646bc3>] out_of_line_wait_on_bit+0x73/0x90
> [<ffffffff8028f680>] ? wake_atomic_t_function+0x40/0x40
> [<ffffffff8034b60f>] __wait_on_buffer+0x3f/0x50
> [<ffffffff8034df18>] __bread_gfp+0xa8/0xd0
> [<ffffffff80388d45>] ext3_get_branch+0x95/0x140
> [<ffffffff80389716>] ext3_get_blocks_handle+0xb6/0xca0
> [<ffffffff8029760c>] ? __lock_acquire+0x50c/0xc30
> [<ffffffff803114b2>] ? __slab_alloc+0x212/0x560
> [<ffffffff80294315>] ? trace_hardirqs_on_caller+0x105/0x1d0
> [<ffffffff8038a3a8>] ext3_get_block+0xa8/0x100
> [<ffffffff80349bba>] generic_block_bmap+0x3a/0x40
> [<ffffffff8038956d>] ext3_bmap+0x7d/0x90
> [<ffffffff80333e2c>] bmap+0x1c/0x20
> [<ffffffff8039ee70>] journal_bmap+0x30/0xa0
> [<ffffffff8039f238>] journal_next_log_block+0x78/0xa0
> [<ffffffff8039a637>] journal_commit_transaction+0x657/0x13e0
> [<ffffffff802aaa87>] ? lock_timer_base+0x37/0x70
> [<ffffffff802ab0c0>] ? get_next_timer_interrupt+0x240/0x240
> [<ffffffff8039e632>] kjournald+0xf2/0x210
> [<ffffffff8028f600>] ? woken_wake_function+0x10/0x10
> [<ffffffff8039e540>] ? commit_timeout+0x10/0x10
> [<ffffffff80279e2e>] kthread+0xee/0x120
> [<ffffffff80279d40>] ? __init_kthread_worker+0x70/0x70
> [<ffffffff8064b56c>] ret_from_fork+0x7c/0xb0
> [<ffffffff80279d40>] ? __init_kthread_worker+0x70/0x70
> ---[ end trace 27f081e879dfbb12 ]---
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [PATCH] drivers: md: remove duplication with time_after
From: Pierre Chevalier @ 2015-02-05 21:39 UTC (permalink / raw)
To: agk; +Cc: linux-raid, linux-kernel, Pierre Chevalier
The logic that was expressed previously was equivalent of calling
time_after.
Remove duplication.
Signed-off-by: Pierre Chevalier <pierrechevalier83@gmail.com>
---
drivers/md/dm-cache-target.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index e165053..855e3b3 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -14,6 +14,7 @@
#include <linux/init.h>
#include <linux/mempool.h>
#include <linux/module.h>
+#include <linux/jiffies.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
@@ -1562,8 +1563,7 @@ static void process_bio(struct cache *cache, struct prealloc *structs,
static int need_commit_due_to_time(struct cache *cache)
{
- return jiffies < cache->last_commit_jiffies ||
- jiffies > cache->last_commit_jiffies + COMMIT_PERIOD;
+ return time_after(jiffies, cache->last_commit_jiffies + COMMIT_PERIOD);
}
static int commit_if_needed(struct cache *cache)
--
2.2.2
^ permalink raw reply related
* RAID1 might_sleep() warning on 3.19-rc7
From: Tony Battersby @ 2015-02-05 20:27 UTC (permalink / raw)
To: Neil Brown, linux-raid
I get the might_sleep() warning below when writing some data to an ext3
filesystem on a RAID1. But everything works OK, so there is no actual
problem, just a warning.
I see that there has been a fix for a might_sleep() warning in md/bitmap
since 3.19-rc7, but this is a different warning.
---
> cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[0] sdb1[1]
1959884 blocks super 1.0 [2/2] [UU]
unused devices: <none>
---
> grep md0 /proc/mounts
/dev/md0 / ext3 rw,noatime,errors=continue,barrier=1,data=journal 0 0
---
WARNING: CPU: 3 PID: 1069 at kernel/sched/core.c:7300 __might_sleep+0x82/0x90()
do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff8028faa1>] prepare_to_wait+0x31/0xa0
Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi igb i2c_algo_bit ptp pps_core mptsas mptscsih mptbase pm80xx libsas mpt2sas scsi_transport_sas raid_class sg coretemp eeprom w83795 i2c_i801
CPU: 3 PID: 1069 Comm: kjournald Not tainted 3.19.0-rc7 #1
Hardware name: Supermicro X8DTH-i/6/iF/6F/X8DTH, BIOS 2.1b 05/04/12
0000000000001c84 ffff88032f1df608 ffffffff80645918 0000000000001c84
ffff88032f1df658 ffff88032f1df648 ffffffff8025ea6b ffff8800bb0b4d58
0000000000000000 00000000000006f6 ffffffff80942b6f ffff8803317b8a00
Call Trace:
[<ffffffff80645918>] dump_stack+0x4f/0x6f
[<ffffffff8025ea6b>] warn_slowpath_common+0x8b/0xd0
[<ffffffff8025eb51>] warn_slowpath_fmt+0x41/0x50
[<ffffffff8028faa1>] ? prepare_to_wait+0x31/0xa0
[<ffffffff8028faa1>] ? prepare_to_wait+0x31/0xa0
[<ffffffff8027ee62>] __might_sleep+0x82/0x90
[<ffffffff803bee06>] generic_make_request_checks+0x36/0x2d0
[<ffffffff802943ed>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff803bf0b3>] generic_make_request+0x13/0x100
[<ffffffff8054983b>] raid1_unplug+0x12b/0x170
[<ffffffff803c1302>] blk_flush_plug_list+0xa2/0x230
[<ffffffff80294315>] ? trace_hardirqs_on_caller+0x105/0x1d0
[<ffffffff80646760>] ? bit_wait_timeout+0x70/0x70
[<ffffffff80646383>] io_schedule+0x43/0x80
[<ffffffff80646787>] bit_wait_io+0x27/0x50
[<ffffffff80646a7d>] __wait_on_bit+0x5d/0x90
[<ffffffff803bf160>] ? generic_make_request+0xc0/0x100
[<ffffffff80646760>] ? bit_wait_timeout+0x70/0x70
[<ffffffff80646bc3>] out_of_line_wait_on_bit+0x73/0x90
[<ffffffff8028f680>] ? wake_atomic_t_function+0x40/0x40
[<ffffffff8034b60f>] __wait_on_buffer+0x3f/0x50
[<ffffffff8034df18>] __bread_gfp+0xa8/0xd0
[<ffffffff80388d45>] ext3_get_branch+0x95/0x140
[<ffffffff80389716>] ext3_get_blocks_handle+0xb6/0xca0
[<ffffffff8029760c>] ? __lock_acquire+0x50c/0xc30
[<ffffffff803114b2>] ? __slab_alloc+0x212/0x560
[<ffffffff80294315>] ? trace_hardirqs_on_caller+0x105/0x1d0
[<ffffffff8038a3a8>] ext3_get_block+0xa8/0x100
[<ffffffff80349bba>] generic_block_bmap+0x3a/0x40
[<ffffffff8038956d>] ext3_bmap+0x7d/0x90
[<ffffffff80333e2c>] bmap+0x1c/0x20
[<ffffffff8039ee70>] journal_bmap+0x30/0xa0
[<ffffffff8039f238>] journal_next_log_block+0x78/0xa0
[<ffffffff8039a637>] journal_commit_transaction+0x657/0x13e0
[<ffffffff802aaa87>] ? lock_timer_base+0x37/0x70
[<ffffffff802ab0c0>] ? get_next_timer_interrupt+0x240/0x240
[<ffffffff8039e632>] kjournald+0xf2/0x210
[<ffffffff8028f600>] ? woken_wake_function+0x10/0x10
[<ffffffff8039e540>] ? commit_timeout+0x10/0x10
[<ffffffff80279e2e>] kthread+0xee/0x120
[<ffffffff80279d40>] ? __init_kthread_worker+0x70/0x70
[<ffffffff8064b56c>] ret_from_fork+0x7c/0xb0
[<ffffffff80279d40>] ? __init_kthread_worker+0x70/0x70
---[ end trace 27f081e879dfbb12 ]---
^ permalink raw reply
* re: md: 'array_size' sysfs attribute
From: Dan Carpenter @ 2015-02-05 11:02 UTC (permalink / raw)
To: dan.j.williams; +Cc: linux-raid
Hello Dan Williams,
The patch b522adcde9c4: "md: 'array_size' sysfs attribute" from Mar
31, 2009, leads to the following static checker warning:
drivers/md/md.c:5069 md_run()
error: we previously assumed 'mddev->pers' could be null (see line 4936)
This code is really old. I don't know why my stupid scripts are marking
it as a new warning. We don't set "mddev->pers" to non-NULL until the
end of the function so it looks like a real bug and that "pers->size"
was intended. When I fix that bug then it un-silences this warning:
drivers/md/md.c:5080 md_run()
error: we previously assumed 'mddev->pers' could be null (see line 4936)
And that also is a real bug, but I'm not sure the right fix for that.
Basically, it's bugs all the way down from the code, to the fix, to the
static checker. *sigh*.
drivers/md/md.c
5060 err = pers->run(mddev);
5061 if (err)
5062 printk(KERN_ERR "md: pers->run() failed ...\n");
5063 else if (pers->size(mddev, 0, 0) < mddev->array_sectors) {
5064 WARN_ONCE(!mddev->external_size, "%s: default size too small,"
5065 " but 'external_size' not in effect?\n", __func__);
5066 printk(KERN_ERR
5067 "md: invalid array_size %llu > default size %llu\n",
5068 (unsigned long long)mddev->array_sectors / 2,
5069 (unsigned long long)mddev->pers->size(mddev, 0, 0) / 2);
^^^^^^^^^^^^^^^^^^
This should be "pers->size()".
5070 err = -EINVAL;
5071 }
5072 if (err == 0 && pers->sync_request &&
5073 (mddev->bitmap_info.file || mddev->bitmap_info.offset)) {
5074 err = bitmap_create(mddev);
5075 if (err)
5076 printk(KERN_ERR "%s: failed to create bitmap (%d)\n",
5077 mdname(mddev), err);
5078 }
5079 if (err) {
5080 mddev_detach(mddev);
^^^^^^^^^^^^^^^^^^^
mddev_detach() will oops if mddev->pers() is not set. We could set
mdev->pers earlier, I suppose.
5081 pers->free(mddev, mddev->private);
5082 module_put(pers->owner);
5083 bitmap_destroy(mddev);
5084 return err;
5085 }
5086 if (mddev->queue) {
5087 mddev->queue->backing_dev_info.congested_data = mddev;
5088 mddev->queue->backing_dev_info.congested_fn = md_congested;
5089 blk_queue_merge_bvec(mddev->queue, md_mergeable_bvec);
5090 }
regards,
dan carpenter
^ permalink raw reply
* Re: Can't reshape raid0 to raid10
From: Xiao Ni @ 2015-02-05 8:44 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150204173921.23df9d79@notabene.brown>
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, February 4, 2015 2:39:21 PM
> Subject: Re: Can't reshape raid0 to raid10
>
> On Mon, 29 Dec 2014 22:13:41 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
>
> > Hi Neil
> >
> > When I try to reshape a raid0 to raid10, it'll fail like this:
> >
> > [root@dhcp-12-133 mdadm-3.3.2]# lsblk
> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > sda 8:0 0 111.8G 0 disk
> > ├─sda1 8:1 0 1000M 0 part /boot
> > ├─sda2 8:2 0 29.3G 0 part /
> > ├─sda3 8:3 0 512M 0 part [SWAP]
> > ├─sda4 8:4 0 1K 0 part
> > ├─sda5 8:5 0 102M 0 part
> > └─sda6 8:6 0 10.1G 0 part
> > └─VolGroup00-LogVol00 254:0 0 9.9G 0 lvm
> > sdb 8:16 0 111.8G 0 disk
> > ├─sdb1 8:17 0 2G 0 part
> > └─sdb2 8:18 0 10G 0 part
> > sdc 8:32 0 186.3G 0 disk
> > ├─sdc1 8:33 0 2G 0 part
> > └─sdc2 8:34 0 10G 0 part
> > sdd 8:48 0 111.8G 0 disk
> > ├─sdd1 8:49 0 2G 0 part
> > └─sdd2 8:50 0 10G 0 part
> > [root@dhcp-12-133 mdadm-3.3.2]# mdadm -CR /dev/md0 -l0 -n3 /dev/sdb1
> > /dev/sdc1 /dev/sdd1
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: array /dev/md0 started.
> > [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l10 -a /dev/sdb2
> > /dev/sdc2 /dev/sdd2
> > mdadm: level of /dev/md0 changed to raid10
> > mdadm: add new device failed for /dev/sdb2 as 6: No space left on device
> >
> >
> > But if I reshape the raid0 to raid5, reshape raid5 to raid0, then
> > reshape raid0 to raid10 use
> > the same command it'll succeed.
> >
> > [root@dhcp-12-133 mdadm-3.3.2]# mdadm -CR /dev/md0 -l0 -n3 /dev/sdb1
> > /dev/sdc1 /dev/sdd1
> > [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l5
> > [root@dhcp-12-133 mdadm-3.3.2]# cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
> > md0 : active raid5 sdd1[2] sdc1[1] sdb1[0]
> > 6285312 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3]
> > [UUU_]
> >
> > unused devices: <none>
> >
> > [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l0
> > [root@dhcp-12-133 mdadm-3.3.2]# cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
> > md0 : active raid0 sdd1[2] sdc1[1] sdb1[0]
> > 6285312 blocks super 1.2 512k chunks
> >
> > unused devices: <none>
> > [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l10 -a /dev/sdb2
> > /dev/sdc2 /dev/sdd2
> > mdadm: level of /dev/md0 changed to raid10
> > mdadm: added /dev/sdb2
> > mdadm: added /dev/sdc2
> > mdadm: added /dev/sdd2
> >
> > So I guess it's the problem add the disk to raid10 after the reshaping.
> > In the function
> > super_1_validate, it'll set the mddev->dev_sectors using the superblock
> > read from disks.
> > If it's raid0, the le64_to_cpu(sb-size) is 0. So when add disk to raid10
> > bind_rdev_to_array
> > return -ENOSPC.
> >
> > When create raid0, it doesn't write give the value to s->size. So the
> > sb-size is 0.
> > I modify the code about Create.c. I'm not sure whether it's right to do so.
> > But it can resolve
> > the problem.
>
> Thanks for the report.
>
> I think this need to be fixing in the md driver, when the reshape happens.
>
> Something like the following.
>
> NeilBrown
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index d1203cddb024..b8d76b1fba64 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -3872,7 +3872,7 @@ static int raid10_resize(struct mddev *mddev, sector_t
> sectors)
> return 0;
> }
>
> -static void *raid10_takeover_raid0(struct mddev *mddev)
> +static void *raid10_takeover_raid0(struct mddev *mddev, sector_t size, int
> devs)
> {
> struct md_rdev *rdev;
> struct r10conf *conf;
> @@ -3882,6 +3882,7 @@ static void *raid10_takeover_raid0(struct mddev *mddev)
> mdname(mddev));
> return ERR_PTR(-EINVAL);
> }
> + sector_div(size, devs);
>
> /* Set new parameters */
> mddev->new_level = 10;
> @@ -3892,12 +3893,15 @@ static void *raid10_takeover_raid0(struct mddev
> *mddev)
> mddev->raid_disks *= 2;
> /* make sure it will be not marked as dirty */
> mddev->recovery_cp = MaxSector;
> + mddev->dev_sectors = size;
>
> conf = setup_conf(mddev);
> if (!IS_ERR(conf)) {
> rdev_for_each(rdev, mddev)
> - if (rdev->raid_disk >= 0)
> + if (rdev->raid_disk >= 0) {
> rdev->new_raid_disk = rdev->raid_disk * 2;
> + rdev->sectors = size;
> + }
> conf->barrier = 1;
> }
>
> @@ -3920,7 +3924,9 @@ static void *raid10_takeover(struct mddev *mddev)
> mdname(mddev));
> return ERR_PTR(-EINVAL);
> }
> - return raid10_takeover_raid0(mddev);
> + return raid10_takeover_raid0(mddev,
> + raid0_conf->strip_zone->zone_end,
> + raid0_conf->strip_zone->nb_dev);
> }
> return ERR_PTR(-EINVAL);
> }
>
Thanks, the problem is fixed.
Best Regards
Xiao
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: BUG drivers/md/md.c: data-offset reshape renders array unloadable
From: Wesley W. Terpstra @ 2015-02-05 8:17 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150205171402.7dc2f394@notabene.brown>
On Thu, Feb 5, 2015 at 7:14 AM, NeilBrown <neilb@suse.de> wrote:
> A v1.2 array normally has data from the data_offset all the way to the end of
> the device (maybe rounded down to chunk size).
Right. So in my case there were 5120 sectors unused at the start.
Unfortunately, I do not know whether the array initially had a 'size'
that included the trailing 3072 sectors or not.
What I do know: before reshape I had a chunk size of 512K and 4 disks
in raid5. When I created the array, tools at the time did not support
--data-offset, but I probably specified all the options I could find
saying I wanted 4M alignment. I reshaped the array from 1 disk to 2 to
3 to 4 as I added disks.
Unfortunately, I do not have a record of what 'mdadm -E' said before I
started the reshape.
Before I fixed the superblock:
Avail Dev Size : 5842897920 (2786.11 GiB 2991.56 GB)
Array Size : 8764342272 (8358.33 GiB 8974.69 GB)
Used Dev Size : 5842894848 (2786.11 GiB 2991.56 GB)
Data Offset : 8192 sectors
After I fixed the superblock:
Avail Dev Size : 5842894848 (2786.11 GiB 2991.56 GB)
Array Size : 8764342272 (8358.33 GiB 8974.69 GB)
Data Offset : 8192 sectors
> So you normally wouldn't be able to increase the data_offset from 5120 to
> 8192 unless you first reduced the --size of the array (having reduced the
> size of any filesystem first).
Well, it let me do it.
> Did you do that? That should have reduced sb->data_size appropriately.
No. I just went ahead and increased the data offset.
> If you did --grow the --size first, that should have reduced sb->data_size.
I did not do this step.
> If you didn't the --grow --data-offset should have failed.
I did this directly, and it let me. I had a slightly older kernel when
I started the reshape. 3.17.7 and mdadm 3.3.2.
^ permalink raw reply
* Re: BUG drivers/md/md.c: data-offset reshape renders array unloadable
From: NeilBrown @ 2015-02-05 6:14 UTC (permalink / raw)
To: Wesley W. Terpstra; +Cc: linux-raid
In-Reply-To: <CAA-O0Xjw8H1nPsi+MsQnrLxv=gC2tj3=W=t3aMnCE6ESUHMF8w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]
On Sun, 25 Jan 2015 17:46:20 +0100 "Wesley W. Terpstra" <wesley@terpstra.ca>
wrote:
> On Sat, Jan 24, 2015 at 11:23 PM, Wesley W. Terpstra <wesley@terpstra.ca> wrote:
> >> First, it is obviously the last test in super_1_load that is rejecting
> >> the array. The superblock reports more sectors than are calculated, so
> >> the check
> >> if (sectors < le64_to_cpu(sb->data_size)) {
> >> fails.
> >
> > Thus, an alternative explanation could be that the the sb->data_size
> > was not updated after the reshape completed.
>
> I can confirm that this was the problem.
>
> I manually modified my super block using the attached quick hack.
> Thereafter I was able to reassemble the array and fsck everything
> successfully.
>
> I will try and see if I can reproduce the problem tomorrow. It's a
> pretty nasty bug to have a reshape complete and render your array
> unassemblable.
Thanks for all the details and analysis!!!
A v1.2 array normally has data from the data_offset all the way to the end of
the device (maybe rounded down to chunk size).
So you normally wouldn't be able to increase the data_offset from 5120 to
8192 unless you first reduced the --size of the array (having reduced the
size of any filesystem first).
Did you do that? That should have reduced sb->data_size appropriately.
If you did --grow the --size first, that should have reduced sb->data_size.
If you didn't the --grow --data-offset should have failed.
Obviously one of those 'should's didn't...
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: raid5 Unplug the disk
From: NeilBrown @ 2015-02-05 5:03 UTC (permalink / raw)
To: John Drescher; +Cc: lilofile, stan, Linux RAID
In-Reply-To: <CAEhu1-4krpTbF-0mLdgoTKrkdrWo8J2k80BC0t48rv3WkG3UBA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 934 bytes --]
On Wed, 28 Jan 2015 11:59:30 -0500 John Drescher <drescherjm@gmail.com> wrote:
> On Wed, Jan 28, 2015 at 1:44 AM, lilofile <lilofile@aliyun.com> wrote:
> > kernel linux 3.10, centos
> > when I use 5 disk to create raid5,after sync complete, I unplug a disk,and wait a moment, then replug the same disk.
> >
> > the original 4 disk are still a raid5 which is degraded.
> > I find the single disk become a new raid5,but inactive,what the reason?
>
> stop whatever new array that shows up then use
>
> mdadm --manage /dev/mymddevice --re-add /dev/disk5
>
Also, you probably need a newer version of mdadm. Creating a new inactive
raid5 was a bug.
If you have udev configured to run "mdadm -I" on newly added devices (which I
think most distros do), then adding
POLICY action=re-add
to /etc/mdadm.conf should cause the device to automatically be added back to
the array when you plug it in.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: raid1 narrow_write_error with 4K disks, sd "bad block number requested" messages
From: NeilBrown @ 2015-02-05 4:59 UTC (permalink / raw)
To: Nate Dailey; +Cc: linux-raid, linux-scsi
In-Reply-To: <54C9006A.2030807@stratus.com>
[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]
On Wed, 28 Jan 2015 10:29:46 -0500 Nate Dailey <nate.dailey@stratus.com>
wrote:
> I'm writing about something that appears to be an issue with raid1's
> narrow_write_error, particular to non-512-byte-sector disks. Here's what
> I'm doing:
>
> - 2 disk raid1, 4K disks, each connected to a different SAS HBA
> - mount a filesystem on the raid1, run a test that writes to it
> - remove one of the SAS HBAs (echo 1 >
> /sys/bus/pci/devices/0000\:45\:00.0/remove)
>
> At this point, writes fail and narrow_write_error breaks them up and
> retries, one sector at a time. But these are 512-byte sectors, and sd
> doesn't like it:
>
> [ 2645.310517] sd 3:0:1:0: [sde] Bad block number requested
> [ 2645.310610] sd 3:0:1:0: [sde] Bad block number requested
> [ 2645.310690] sd 3:0:1:0: [sde] Bad block number requested
> ...
>
> There appears to be no real harm done, but there can be a huge number of
> these messages in the log.
>
> I can avoid this by disabling bad block tracking, but it looks like
> maybe the superblock's bblog_shift is intended to address this exact
> issue. However, I don't see a way to change it. Presumably this is
> something mdadm should be setting up? I don't see bblog_shift ever set
> to anything other than 0.
>
> This is on a RHEL 7.1 kernel, version 3.10.0-221.el7. I took a look at
> upstream sd and md changes and nothing jumps out at me that would have
> affected this (but I have not tested to see if the bad block messages do
> or do not happen on an upstream kernel).
>
> I'd appreciate any advice re: how to handle this. Thanks!
Thanks for the report.
narrow_write_error() should use bdev_logical_block_size() and round up to
that.
Possibly mdadm should get the same information and set bblog_shift
accordingly when creating a bad block log.
I've made a note to fix that, but I'm happy to review patches too :-)
thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: unsynchronized raid10 with different events count
From: NeilBrown @ 2015-02-05 4:52 UTC (permalink / raw)
To: LuVar; +Cc: linux-raid
In-Reply-To: <850189010.36671422720846038.JavaMail.root@shiva>
[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]
On Sat, 31 Jan 2015 17:14:06 +0100 (GMT+01:00) LuVar <luvar@plaintext.sk>
wrote:
> Hi,
> finally,
> I have run --re-add thing. It seems that it is what I wanted. My array is now OK.
>
> So only one question remains. Is there any possibility to make improvements and make array autoassemble if there are clear that in raid10, there is consistent highest events count for at least one drive for raid1 subparts? I have layout=n3 for situations when one drive fails, to be able to correctly recover from others two same copies and to know that given data are probably ok if they are same.
It appears from the mdstat you posted:
Personalities : [raid10]
md3 : active raid10 sdj[9] sdl[13] sdm[6] sdi[3] sdb[10]
937709952 blocks super 1.2 128K chunks 3 near-copies [9/5] [U_UU__UU_]
bitmap: 4/7 pages [16KB], 65536KB chunk
unused devices: <none>
That it *did* autoassemble and provide you with a working array. It just
didn't include all the devices - because some of them looked old and mdadm
couldn't be sure they were reliable.
You did the correct thing to --re-add them. That is how you tell mdadm
"these devices are reliable, even though someone went wrong before and they
didn't get updated with the latest event count".
So it appears to me that everything is working correctly.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Off-Topic Write cache disabling?
From: NeilBrown @ 2015-02-05 4:47 UTC (permalink / raw)
To: Weedy; +Cc: linux-raid
In-Reply-To: <CAFE24U1G9+bqTzK4eD-cenyc5CGQQdrcu7S9Ks4CsHPS9wJ4zA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2389 bytes --]
On Tue, 3 Feb 2015 12:45:44 -0500 Weedy <weedy2887@gmail.com> wrote:
> Is there a kernel option or sysfs toggle that disables write caching?
> Or forces the kernel to commit everything constantly.
>
> ------
> I don't really want to join another ML, especially a higher traffic
> one just to ask this when it only bugs me sometimes. But I'll shut up
> if this is unwanted.
> ------
>
> I use a similar kernel config with respect to selected options on all
> my systems but this only effect my laptop.
Maybe it has something to do with "laptop_mode"
/proc/sys/vm/laptop_mode
https://www.kernel.org/doc/Documentation/laptops/laptop-mode.txt
> On any of my personal system or system I have remote access too
> nr_dirty drifts up and down and nr_writeback stays around 0 (assuming
> the system isn't working hard).
> On my laptop both nr_dirty and nr_writeback stay at 0. I can make them
> go up to 10ish if I untar something but then almost immediately go
> back to 0. If I didn't know better I would swear dirty_*_centisecs or
> something was set to a near instant commit interval but I haven't
> found evidence of that. The hard drive light blinks almost constantly
> once a second, even if I'm at a X login screen.
That doesn't quite sound like laptop mode...
laptop mode syncs things more aggressively when the disk is spinning, and
more lazily when it isn't ... or something like that.
So I can imagine nr_dirty staying low - any read will spin the disk, and that
will quickly flush out any dirty pages.
The once-a-second blink is harder to explain.
Still, it would be worth checking if laptop mode is enabled, and if so
disable it and see if anything changes.
NeilBrown
> As I said this doesn't bug me most of the time but if I let my FF
> session get too large or start multiple VMs anything that might make
> me swap a little, the machine pretty much dies from IOWAIT. Which I'm
> guessing is because it's trying to flush (syncfs?) imediately and
> constantly.
>
> You guys spend all day in the IO subsystem, any idea where I can keep
> looking? It has persisted across reboots and kernel updates.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Array died during grow; now resync stopped
From: NeilBrown @ 2015-02-05 4:40 UTC (permalink / raw)
To: Jörg Habenicht; +Cc: linux-raid
In-Reply-To: <trinity-85824e10-6eeb-490e-91e9-29cac6bb884e-1423051086472@3capp-gmx-bs24>
[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]
On Wed, 4 Feb 2015 12:58:06 +0100 "Jörg Habenicht" <j.habenicht@gmx.de> wrote:
> It works! :-)) The array syncs!
Excellent!
>
>
> > > >What does
> > > >cat /proc/1671/stack
> > > >cat /proc/1672/stack
> > > >show?
> > >
> > > $ cat /proc/1671/stack
> > > cat: /proc/1671/stack: No such file or directory
> >
> > I guess you don't have that feature compiled into your kernel.
>
> Guess so. I'm going to look to the missing CONFIG_ line.
CONFIG_STACKTRACE ... in case you haven't found it already.
> >
> > And this is stuck in the same place.... what what is consuming all the
> > stripes I wonder....
>
> Do you like me to collect more information?
No thanks, the required information was further down. They were all consumed
by the reshape (which needs at least twice the largest chunk size).
> > If you double it, the problem should go away.
> >
> > mdadm should do that for you .... I wonder why it didn't.
> >
>
> Do you like to have more test results?
I had a look at the code and it was fairly obvious. mdadm adjusts the stripe
cache based on the 'old' chunk size. It never includes the 'new' chunk size
in those calculations. I've made a note to fix that.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Off-Topic Write cache disabling?
From: Bernd Schubert @ 2015-02-04 12:46 UTC (permalink / raw)
To: Weedy, linux-raid
In-Reply-To: <CAFE24U1G9+bqTzK4eD-cenyc5CGQQdrcu7S9Ks4CsHPS9wJ4zA@mail.gmail.com>
On 02/03/2015 06:45 PM, Weedy wrote:
> Is there a kernel option or sysfs toggle that disables write caching?
> Or forces the kernel to commit everything constantly.
>
> ------
> I don't really want to join another ML, especially a higher traffic
> one just to ask this when it only bugs me sometimes. But I'll shut up
> if this is unwanted.
> ------
>
> I use a similar kernel config with respect to selected options on all
> my systems but this only effect my laptop.
> On any of my personal system or system I have remote access too
> nr_dirty drifts up and down and nr_writeback stays around 0 (assuming
> the system isn't working hard).
> On my laptop both nr_dirty and nr_writeback stay at 0. I can make them
> go up to 10ish if I untar something but then almost immediately go
> back to 0. If I didn't know better I would swear dirty_*_centisecs or
> something was set to a near instant commit interval but I haven't
> found evidence of that. The hard drive light blinks almost constantly
> once a second, even if I'm at a X login screen.
> As I said this doesn't bug me most of the time but if I let my FF
> session get too large or start multiple VMs anything that might make
> me swap a little, the machine pretty much dies from IOWAIT. Which I'm
> guessing is because it's trying to flush (syncfs?) imediately and
> constantly.
>
> You guys spend all day in the IO subsystem, any idea where I can keep
> looking? It has persisted across reboots and kernel updates.
Try mount -osync or force your application to write with direct IO.
Cheers,
Bernd
^ permalink raw reply
* Re: Off-Topic Write cache disabling?
From: Bernd Schubert @ 2015-02-04 12:45 UTC (permalink / raw)
To: Roger Heflin, Weedy; +Cc: Linux RAID
In-Reply-To: <CAAMCDedvF+Nv9LYqM7Ddgg2xMQKtgQDntOsG0krxWCV4H+zQmA@mail.gmail.com>
On 02/03/2015 07:49 PM, Roger Heflin wrote:
> In general if you completely disable write caching things will likely
> be unusable. When I have benchmarked things attempting to turn off
> write cache performance has been mostly unusable.
>
> I usually change /etc/sysctl.conf and change these entries to control
> the amount of dirty cache.
>
> vm.dirty_background_bytes = 25000000
> vm.dirty_bytes = 50000000
>
> That says max dirty allowed is 50mb, and it syncs down to 25mb when it syncs.
Er, where do you have this from?
dirty_background_bytes and dirty_bytes are more about non-blocking and
blocking dirty writes. And about the amount of dirty data that triggers
it (i.e. there is no need to flush temporary file data to disk if files
get immediately deleted after writing them)
linux/Documentation/sysctl/vm.txt
==============================================================
dirty_background_bytes
Contains the amount of dirty memory at which the background kernel
flusher threads will start writeback.
Note: dirty_background_bytes is the counterpart of dirty_background_ratio. Only
one of them may be specified at a time. When one sysctl is written it is
immediately taken into account to evaluate the dirty memory limits and the
other appears as 0 when read.
==============================================================
==============================================================
dirty_bytes
Contains the amount of dirty memory at which a process generating disk writes
will itself start writeback.
Note: dirty_bytes is the counterpart of dirty_ratio. Only one of them may be
specified at a time. When one sysctl is written it is immediately taken into
account to evaluate the dirty memory limits and the other appears as 0 when
read.
Note: the minimum value allowed for dirty_bytes is two pages (in bytes); any
value lower than this limit will be ignored and the old configuration will be
retained.
==============================================================
Bernd
^ permalink raw reply
* Aw: Re: Array died during grow; now resync stopped
From: "Jörg Habenicht" @ 2015-02-04 11:58 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150204174556.426eb865@notabene.brown>
It works! :-)) The array syncs!
> > >What does
> > >cat /proc/1671/stack
> > >cat /proc/1672/stack
> > >show?
> >
> > $ cat /proc/1671/stack
> > cat: /proc/1671/stack: No such file or directory
>
> I guess you don't have that feature compiled into your kernel.
Guess so. I'm going to look to the missing CONFIG_ line.
> > >Alternatively,
> > >echo w > /proc/sysrq-trigger
> > >and see what appears in 'dmesg'.
> >
> > No good:
>
> Quite the reverse, this is exactly what I wanted. It shows the stack trace
> of pid 1671 and 1672..
Ahh, OK. I thought it being a crash dump.
> >
> > [99166.625796] SysRq : Show Blocked State
> > [99166.625829] task PC stack pid father
> > [99166.625845] md0_reshape D ffff88006cb81e08 0 1671 2 0x00000000
> > [99166.625854] ffff88006a17fb30 0000000000000046 000000000000a000 ffff88006cc9b7e0
> > [99166.625861] ffff88006a17ffd8 ffff88006cc9b7e0 ffff88006fc11830 ffff88006fc11830
> > [99166.625866] 0000000000000001 ffffffff81068670 ffff88006ca56848 ffff88006fc11830
> > [99166.625871] Call Trace:
> > [99166.625884] [<ffffffff81068670>] ? __dequeue_entity+0x40/0x50
> > [99166.625891] [<ffffffff8106b966>] ? pick_next_task_fair+0x56/0x1b0
> > [99166.625898] [<ffffffff813f4a50>] ? __schedule+0x2a0/0x820
> > [99166.625905] [<ffffffff8106273d>] ? ttwu_do_wakeup+0xd/0x80
> > [99166.625914] [<ffffffffa027b4c5>] ? get_active_stripe+0x185/0x5c0 [raid456]
> > [99166.625922] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> > [99166.625929] [<ffffffffa027e83a>] ? reshape_request+0x21a/0x860 [raid456]
> > [99166.625935] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> > [99166.625942] [<ffffffffa02744f6>] ? sync_request+0x236/0x380 [raid456]
> > [99166.625955] [<ffffffffa01557ad>] ? md_do_sync+0x82d/0xd00 [md_mod]
> > [99166.625961] [<ffffffff810684b4>] ? update_curr+0x64/0xe0
> > [99166.625971] [<ffffffffa0152197>] ? md_thread+0xf7/0x110 [md_mod]
> > [99166.625977] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> > [99166.625985] [<ffffffffa01520a0>] ? md_register_thread+0xf0/0xf0 [md_mod]
> > [99166.625991] [<ffffffff81059de8>] ? kthread+0xb8/0xd0
> > [99166.625997] [<ffffffff81059d30>] ? kthread_create_on_node+0x180/0x180
> > [99166.626003] [<ffffffff813f837c>] ? ret_from_fork+0x7c/0xb0
> > [99166.626008] [<ffffffff81059d30>] ? kthread_create_on_node+0x180/0x180
>
> That's not surprise. Whenever anything goes wrong in raid5, something gets
> stuck in get_active_stripe()...
>
>
> > [99166.626012] udevd D ffff88006cb81e08 0 1672 1289 0x00000004
> > [99166.626017] ffff88006a1819e8 0000000000000086 000000000000a000 ffff88006c4967a0
> > [99166.626022] ffff88006a181fd8 ffff88006c4967a0 0000000000000000 0000000000000000
> > [99166.626027] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > [99166.626032] Call Trace:
> > [99166.626039] [<ffffffff810c24ed>] ? zone_statistics+0x9d/0xa0
> > [99166.626044] [<ffffffff810c24ed>] ? zone_statistics+0x9d/0xa0
> > [99166.626050] [<ffffffff810b13e7>] ? get_page_from_freelist+0x507/0x850
> > [99166.626057] [<ffffffffa027b4c5>] ? get_active_stripe+0x185/0x5c0 [raid456]
> > [99166.626063] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> > [99166.626069] [<ffffffffa027f627>] ? make_request+0x7a7/0xa00 [raid456]
> > [99166.626075] [<ffffffff81080afd>] ? ktime_get_ts+0x3d/0xd0
> > [99166.626080] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> > [99166.626089] [<ffffffffa014ea12>] ? md_make_request+0xd2/0x210 [md_mod]
> > [99166.626096] [<ffffffff811e649d>] ? generic_make_request_checks+0x23d/0x270
> > [99166.626100] [<ffffffff810acc68>] ? mempool_alloc+0x58/0x140
> > [99166.626106] [<ffffffff811e7238>] ? generic_make_request+0xa8/0xf0
> > [99166.626111] [<ffffffff811e72e7>] ? submit_bio+0x67/0x130
> > [99166.626117] [<ffffffff8112a638>] ? bio_alloc_bioset+0x1b8/0x2a0
> > [99166.626123] [<ffffffff81126a57>] ? _submit_bh+0x127/0x200
> > [99166.626129] [<ffffffff8112815d>] ? block_read_full_page+0x1fd/0x290
> > [99166.626133] [<ffffffff8112b680>] ? I_BDEV+0x10/0x10
> > [99166.626140] [<ffffffff810aad2b>] ? add_to_page_cache_locked+0x6b/0xc0
> > [99166.626146] [<ffffffff810b5520>] ? __do_page_cache_readahead+0x1b0/0x220
> > [99166.626152] [<ffffffff810b5812>] ? force_page_cache_readahead+0x62/0xa0
> > [99166.626159] [<ffffffff810ac936>] ? generic_file_aio_read+0x4b6/0x6c0
> > [99166.626166] [<ffffffff810f9f87>] ? do_sync_read+0x57/0x90
> > [99166.626172] [<ffffffff810fa571>] ? vfs_read+0xa1/0x180
> > [99166.626178] [<ffffffff810fb0ab>] ? SyS_read+0x4b/0xc0
> > [99166.626183] [<ffffffff813f7f72>] ? page_fault+0x22/0x30
> > [99166.626190] [<ffffffff813f8422>] ? system_call_fastpath+0x16/0x1b
>
> And this is stuck in the same place.... what what is consuming all the
> stripes I wonder....
Do you like me to collect more information?
> > >
> > > The system got 2G RAM and 2G swap. Is this sufficient to complete?
> >
> > >Memory shouldn't be a problem.
> > >However it wouldn't hurt to see what value is in
> > >/sys/block/md0/md/stripe_cache_size
> > >and double it.
> >
> > $ cat /sys/block/md0/md/stripe_cache_size
> > 256
>
>
> You are setting the chunk size to 1M, which is 256 4K pages.
> So this stripe_cache only just has enough space to store one full stripe at
> the new chunk size. That isn't enough.
>
> If you double it, the problem should go away.
>
> mdadm should do that for you .... I wonder why it didn't.
>
Do you like to have more test results?
> >
> > I did not change it due to the crash in md_reshape
>
> What crash is that? The above stack traces that you said "No good" about?
> That isn't a crash. That is the kernel showing you stack traces because you
> asked for them.
Ok, Thanks, learned something more :-)
>
> echo 1024 > /sys/block/md0/md/stripe_cache_size
>
> should make it work.
Yes it did!
$ echo 1024 > /sys/block/md0/md/stripe_cache_size
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1[0] sdg1[9] sdc1[6] sdb1[7] sdd1[8] sdf1[5]
5860548608 blocks super 1.0 level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]
[=====>...............] reshape = 28.0% (410341376/1465137152) finish=29602912.5min speed=0K/sec
unused devices: <none>
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1[0] sdg1[9] sdc1[6] sdb1[7] sdd1[8] sdf1[5]
5860548608 blocks super 1.0 level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]
[=====>...............] reshape = 28.0% (410656252/1465137152) finish=7746625.6min speed=2K/sec
unused devices: <none>
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1[0] sdg1[9] sdc1[6] sdb1[7] sdd1[8] sdf1[5]
5860548608 blocks super 1.0 level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]
[=====>...............] reshape = 28.0% (410851328/1465137152) finish=5314609.8min speed=3K/sec
unused devices: <none>
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1[0] sdg1[9] sdc1[6] sdb1[7] sdd1[8] sdf1[5]
5860548608 blocks super 1.0 level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]
[=====>...............] reshape = 28.4% (416577276/1465137152) finish=870.3min speed=20079K/sec
unused devices: <none>
Immediatly it starts to sync.
I wonder why it got stuck at 27% and not at 0%? Shouldn't it get stuck at the beginning if the cache size is not sufficient?
Or is it because of a reboot which happened at 27% sync status?
Anyways,
Thank you for your help.
>
> NeilBrown
>
>
cu,
Joerg
^ permalink raw reply
* Re: Array died during grow; now resync stopped
From: NeilBrown @ 2015-02-04 6:45 UTC (permalink / raw)
To: Jörg Habenicht; +Cc: linux-raid
In-Reply-To: <trinity-d6d8d6b9-e19b-4ff6-a39f-3748b5fb2e13-1422959753855@3capp-gmx-bs42>
[-- Attachment #1: Type: text/plain, Size: 7305 bytes --]
On Tue, 3 Feb 2015 11:35:53 +0100 "Jörg Habenicht" <j.habenicht@gmx.de> wrote:
> Hello Neil,
>
> thank you for caring.
>
> (And sorry for the malformed structure, I have to use webmail.. )
>
>
> On Mon, 2 Feb 2015 09:41:02 +0000 (UTC) Jörg Habenicht <j.habenicht@gmx.de>
> wrote:
>
> > Hi all,
> >
> > I had a server crash during an array grow.
> > Commandline was "mdadm --grow /dev/md0 --raid-devices=6 --chunk=1M"
> >
> >
> >
> > Could this be caused by a software lock?
>
> >Some sort of software problem I suspect.
> >What does
> >cat /proc/1671/stack
> >cat /proc/1672/stack
> >show?
>
> $ cat /proc/1671/stack
> cat: /proc/1671/stack: No such file or directory
I guess you don't have that feature compiled into your kernel.
>
> Huch?
> $ ls /proc/1671
> ls: cannot read symbolic link /proc/1671/exe: No such file or directory
> attr comm fdinfo mounts oom_score stat
> autogroup coredump_filter io mountstats oom_score_adj statm
> auxv cwd limits net pagemap status
> cgroup environ maps ns personality syscall
> clear_refs exe mem numa_maps root task
> cmdline fd mountinfo oom_adj smaps wchan
> $ id
> uid=0(root) gid=0(root) groups=0(root), ...
>
> $ cat /proc/1672/stack
> cat: /proc/1672/stack: No such file or directory
>
>
> >Alternatively,
> >echo w > /proc/sysrq-trigger
> >and see what appears in 'dmesg'.
>
> No good:
Quite the reverse, this is exactly what I wanted. It shows the stack trace
of pid 1671 and 1672..
>
> [99166.625796] SysRq : Show Blocked State
> [99166.625829] task PC stack pid father
> [99166.625845] md0_reshape D ffff88006cb81e08 0 1671 2 0x00000000
> [99166.625854] ffff88006a17fb30 0000000000000046 000000000000a000 ffff88006cc9b7e0
> [99166.625861] ffff88006a17ffd8 ffff88006cc9b7e0 ffff88006fc11830 ffff88006fc11830
> [99166.625866] 0000000000000001 ffffffff81068670 ffff88006ca56848 ffff88006fc11830
> [99166.625871] Call Trace:
> [99166.625884] [<ffffffff81068670>] ? __dequeue_entity+0x40/0x50
> [99166.625891] [<ffffffff8106b966>] ? pick_next_task_fair+0x56/0x1b0
> [99166.625898] [<ffffffff813f4a50>] ? __schedule+0x2a0/0x820
> [99166.625905] [<ffffffff8106273d>] ? ttwu_do_wakeup+0xd/0x80
> [99166.625914] [<ffffffffa027b4c5>] ? get_active_stripe+0x185/0x5c0 [raid456]
> [99166.625922] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> [99166.625929] [<ffffffffa027e83a>] ? reshape_request+0x21a/0x860 [raid456]
> [99166.625935] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> [99166.625942] [<ffffffffa02744f6>] ? sync_request+0x236/0x380 [raid456]
> [99166.625955] [<ffffffffa01557ad>] ? md_do_sync+0x82d/0xd00 [md_mod]
> [99166.625961] [<ffffffff810684b4>] ? update_curr+0x64/0xe0
> [99166.625971] [<ffffffffa0152197>] ? md_thread+0xf7/0x110 [md_mod]
> [99166.625977] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> [99166.625985] [<ffffffffa01520a0>] ? md_register_thread+0xf0/0xf0 [md_mod]
> [99166.625991] [<ffffffff81059de8>] ? kthread+0xb8/0xd0
> [99166.625997] [<ffffffff81059d30>] ? kthread_create_on_node+0x180/0x180
> [99166.626003] [<ffffffff813f837c>] ? ret_from_fork+0x7c/0xb0
> [99166.626008] [<ffffffff81059d30>] ? kthread_create_on_node+0x180/0x180
That's not surprise. Whenever anything goes wrong in raid5, something gets
stuck in get_active_stripe()...
> [99166.626012] udevd D ffff88006cb81e08 0 1672 1289 0x00000004
> [99166.626017] ffff88006a1819e8 0000000000000086 000000000000a000 ffff88006c4967a0
> [99166.626022] ffff88006a181fd8 ffff88006c4967a0 0000000000000000 0000000000000000
> [99166.626027] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [99166.626032] Call Trace:
> [99166.626039] [<ffffffff810c24ed>] ? zone_statistics+0x9d/0xa0
> [99166.626044] [<ffffffff810c24ed>] ? zone_statistics+0x9d/0xa0
> [99166.626050] [<ffffffff810b13e7>] ? get_page_from_freelist+0x507/0x850
> [99166.626057] [<ffffffffa027b4c5>] ? get_active_stripe+0x185/0x5c0 [raid456]
> [99166.626063] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> [99166.626069] [<ffffffffa027f627>] ? make_request+0x7a7/0xa00 [raid456]
> [99166.626075] [<ffffffff81080afd>] ? ktime_get_ts+0x3d/0xd0
> [99166.626080] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
> [99166.626089] [<ffffffffa014ea12>] ? md_make_request+0xd2/0x210 [md_mod]
> [99166.626096] [<ffffffff811e649d>] ? generic_make_request_checks+0x23d/0x270
> [99166.626100] [<ffffffff810acc68>] ? mempool_alloc+0x58/0x140
> [99166.626106] [<ffffffff811e7238>] ? generic_make_request+0xa8/0xf0
> [99166.626111] [<ffffffff811e72e7>] ? submit_bio+0x67/0x130
> [99166.626117] [<ffffffff8112a638>] ? bio_alloc_bioset+0x1b8/0x2a0
> [99166.626123] [<ffffffff81126a57>] ? _submit_bh+0x127/0x200
> [99166.626129] [<ffffffff8112815d>] ? block_read_full_page+0x1fd/0x290
> [99166.626133] [<ffffffff8112b680>] ? I_BDEV+0x10/0x10
> [99166.626140] [<ffffffff810aad2b>] ? add_to_page_cache_locked+0x6b/0xc0
> [99166.626146] [<ffffffff810b5520>] ? __do_page_cache_readahead+0x1b0/0x220
> [99166.626152] [<ffffffff810b5812>] ? force_page_cache_readahead+0x62/0xa0
> [99166.626159] [<ffffffff810ac936>] ? generic_file_aio_read+0x4b6/0x6c0
> [99166.626166] [<ffffffff810f9f87>] ? do_sync_read+0x57/0x90
> [99166.626172] [<ffffffff810fa571>] ? vfs_read+0xa1/0x180
> [99166.626178] [<ffffffff810fb0ab>] ? SyS_read+0x4b/0xc0
> [99166.626183] [<ffffffff813f7f72>] ? page_fault+0x22/0x30
> [99166.626190] [<ffffffff813f8422>] ? system_call_fastpath+0x16/0x1b
And this is stuck in the same place.... what what is consuming all the
stripes I wonder....
>
>
> >
> > The system got 2G RAM and 2G swap. Is this sufficient to complete?
>
> >Memory shouldn't be a problem.
> >However it wouldn't hurt to see what value is in
> >/sys/block/md0/md/stripe_cache_size
> >and double it.
>
> $ cat /sys/block/md0/md/stripe_cache_size
> 256
You are setting the chunk size to 1M, which is 256 4K pages.
So this stripe_cache only just has enough space to store one full stripe at
the new chunk size. That isn't enough.
If you double it, the problem should go away.
mdadm should do that for you .... I wonder why it didn't.
>
> I did not change it due to the crash in md_reshape
What crash is that? The above stack traces that you said "No good" about?
That isn't a crash. That is the kernel showing you stack traces because you
asked for them.
echo 1024 > /sys/block/md0/md/stripe_cache_size
should make it work.
NeilBrown
>
>
> >If all else fails a reboot should be safe and will probably start the reshape
> >properly. md is very careful about surviving reboots.
>
> I already did reboot twice before I wrote to the list. Same result.
>
>
> >NeilBrown
>
>
> cu,
> Joerg
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox