* Re: deprecating /proc/mdstat (was: Re: mdadm bad blocks list)
From: Shaohua Li @ 2016-01-28 18:21 UTC (permalink / raw)
To: Jens-U. Mozdzen; +Cc: NeilBrown, linux-raid
In-Reply-To: <20160128124121.Horde.YKtbZL0CUYKTuEkxLcmmhQ4@www3.nde.ag>
On Thu, Jan 28, 2016 at 12:41:21PM +0100, Jens-U. Mozdzen wrote:
> Hello Neil & *,
>
> Zitat von NeilBrown <nfbrown@novell.com>:
> >[...]
> >I'd like to deprecate /proc/mdstat. It is not really easy to extend.
>
> while I understand that /proc/mdstat's format might be considered "frozen"
> as in "do not confuse old scripts by new formats", I'd hate to see
> /proc/mdstat go away without a similar replacement: calling "mdadm" (or any
> other CLI) to gather that information is unruly expensive when all you have
> to do is "watch cat /proc/mdstat" to manually monitor critical operations.
That will not happen soon. Deprecating an interface takes years.
> >I'd recommend using "mdadm" to get status of an array, or examine file
> >in /sys.
>
> If there's to be a "new mdstat" in /sys, I'd be fine with that. That would
> help migration for those "old scripts grep'ing /proc/mdstat" you rightfully
> care about.
>
> I suggest to include a file format version information on line 1
> "/sys/.../mdstat", that way any client parsing such an interface could
> verify the file format first, and bail out if it doesn't support the
> currently presented format.
All the info you can get from /proc/mdstat can be found in /sys/xxx. There
isn't a central mdstat file in sysfs entry, each sysfs entry only export single
type info. Version info is uncessary, if we need add new info, we'd just add a
new sysfs entry.
though the /proc/mdstat will not be deprecated soon, it's highly encouraged app
switches to /sys. sysfs entry is easy to parse. And as Neil said, /proc/mdstat
is hard to extend, so new info will likely only appear in sysfs.
Thanks,
Shaohua
^ permalink raw reply
* Re: reinstate dm target local ioctl support
From: Christoph Hellwig @ 2016-01-28 14:00 UTC (permalink / raw)
To: Andy Whitcroft
Cc: Alasdair Kergon, Mike Snitzer, Christoph Hellwig,
mohan_srinivasan, dm-devel, Neil Brown, linux-raid, linux-kernel,
Tim Gardner
In-Reply-To: <1453989019-24300-1-git-send-email-apw@canonical.com>
On Thu, Jan 28, 2016 at 01:50:19PM +0000, Andy Whitcroft wrote:
> However this also removed the possibility of a dm target having target
> specific ioctls. Currently this is not used by any in-tree targets but
> was utilised by the flashcache out-of-tree module.
>
> How would we feel about carrying something like the patch below in
> mainline to allow for such target local ioctls.
Bad. Because a) targets shouldn't have their own ioctls and b) your out
of tree modules are your problem, don't burden us wіth upstream
workarounds for it.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* reinstate dm target local ioctl support
From: Andy Whitcroft @ 2016-01-28 13:50 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Christoph Hellwig
Cc: mohan_srinivasan, dm-devel, Neil Brown, linux-raid, linux-kernel,
Andy Whitcroft, Tim Gardner
With the commit below the dm ioctl support was reordered so that the
target only had to identify the underlying device, the dm_blk_ioctl()
itself then applied those ioctls to the appropriate device:
commit e56f81e0b01ef4e45292d8c1e19edd4d09724e14
Author: Christoph Hellwig <hch@lst.de>
Date: Thu Oct 15 14:10:50 2015 +0200
dm: refactor ioctl handling
However this also removed the possibility of a dm target having target
specific ioctls. Currently this is not used by any in-tree targets but
was utilised by the flashcache out-of-tree module.
How would we feel about carrying something like the patch below in
mainline to allow for such target local ioctls.
-apw
From 3fa0480193b944dd97565364efe4df89414c250b Mon Sep 17 00:00:00 2001
From: Andy Whitcroft <apw@canonical.com>
Date: Wed, 27 Jan 2016 16:05:32 +0000
Subject: dm: introduce a target_ioctl op to allow target specific ioctls
In e56f81e0b01e "dm: refactor ioctl handling" the target specific ioctl
operation was removed in favour of providing a mapping to the underlying
device, to which ioctls are all assumed to apply. This prevents targets
from having target specific ioctls.
Introduce a new target_ioctl callback which (if present) is offered the
command and arguments for processing. This callback can return -ENOTTY
to indicate the ioctl should be passed on to the underlying device as
normal.
BugLink: http://bugs.launchpad.net/bugs/1538618
Signed-off-by: Andy Whitcroft <apw@canonical.com>
---
drivers/md/dm.c | 9 +++++++++
include/linux/device-mapper.h | 3 +++
2 files changed, 12 insertions(+)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 5df4048..9014d46 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -610,6 +610,15 @@ static int dm_blk_ioctl(struct block_device *bdev, fmode_t mode,
if (r < 0)
return r;
+ if (tgt->type->target_ioctl) {
+ int res = tgt->type->target_ioctl(tgt, cmd, arg);
+
+ if (res != -ENOTTY) {
+ r = res;
+ goto out;
+ }
+ }
+
if (r > 0) {
/*
* Target determined this ioctl is being issued against
diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
index ec1c61c..d770bf0 100644
--- a/include/linux/device-mapper.h
+++ b/include/linux/device-mapper.h
@@ -81,6 +81,8 @@ typedef int (*dm_message_fn) (struct dm_target *ti, unsigned argc, char **argv);
typedef int (*dm_prepare_ioctl_fn) (struct dm_target *ti,
struct block_device **bdev, fmode_t *mode);
+typedef int (*dm_target_ioctl_fn) (struct dm_target *ti, unsigned int cmd,
+ unsigned long arg);
/*
* These iteration functions are typically used to check (and combine)
@@ -157,6 +159,7 @@ struct target_type {
dm_status_fn status;
dm_message_fn message;
dm_prepare_ioctl_fn prepare_ioctl;
+ dm_target_ioctl_fn target_ioctl;
dm_busy_fn busy;
dm_iterate_devices_fn iterate_devices;
dm_io_hints_fn io_hints;
--
2.7.0.rc3
^ permalink raw reply related
* deprecating /proc/mdstat (was: Re: mdadm bad blocks list)
From: Jens-U. Mozdzen @ 2016-01-28 11:41 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <87twly1jc7.fsf@notabene.neil.brown.name>
Hello Neil & *,
Zitat von NeilBrown <nfbrown@novell.com>:
> [...]
> I'd like to deprecate /proc/mdstat. It is not really easy to extend.
while I understand that /proc/mdstat's format might be considered
"frozen" as in "do not confuse old scripts by new formats", I'd hate
to see /proc/mdstat go away without a similar replacement: calling
"mdadm" (or any other CLI) to gather that information is unruly
expensive when all you have to do is "watch cat /proc/mdstat" to
manually monitor critical operations.
> I'd recommend using "mdadm" to get status of an array, or examine file
> in /sys.
If there's to be a "new mdstat" in /sys, I'd be fine with that. That
would help migration for those "old scripts grep'ing /proc/mdstat" you
rightfully care about.
I suggest to include a file format version information on line 1
"/sys/.../mdstat", that way any client parsing such an interface could
verify the file format first, and bail out if it doesn't support the
currently presented format.
With regards
Jens
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: Joshua Kinard @ 2016-01-28 9:58 UTC (permalink / raw)
To: NeilBrown, Chien Lee, linux-raid, shli, owner-linux-raid
In-Reply-To: <87wpqu1jrl.fsf@notabene.neil.brown.name>
On 01/27/2016 22:10, NeilBrown wrote:
> On Wed, Jan 27 2016, Chien Lee wrote:
>
>> 2016-01-27 6:12 GMT+08:00 NeilBrown <neilb@suse.com>:
>>> On Tue, Jan 26 2016, Chien Lee wrote:
>>>
>>>> Hello,
>>>>
>>>> Recently we find a bug about this patch (commit No. is
>>>> ac8fa4196d205ac8fff3f8932bddbad4f16e4110 ).
>>>>
>>>> We know that this patch committed after Linux kernel 4.1.x is intended
>>>> to allowing resync to go faster when there is competing IO. However,
>>>> we find the performance of random read on syncing Raid6 will come up
>>>> with a huge drop in this case. The following is our testing detail.
>>>>
>>>> The OS what we choose in our test is CentOS Linux release 7.1.1503
>>>> (Core) and the kernel image will be replaced for testing. In our
>>>> testing result, the 4K random read performance on syncing raid6 in
>>>> Kernel 4.2.8 is much lower than in Kernel 3.19.8. In order to find out
>>>> the root cause, we try to rollback this patch in Kernel 4.2.8, and we
>>>> find the 4K random read performance on syncing Raid6 will be improved
>>>> and go back to as what it should be in Kernel 3.19.8.
>>>>
>>>> Nevertheless, it seems that it will not affect some other read/write
>>>> patterns. In our testing result, the 1M sequential read/write, 4K
>>>> random write performance in Kernel 4.2.8 is performed almost the same
>>>> as in Kernel 3.19.8.
>>>>
>>>> It seems that although this patch increases the resync speed, the
>>>> logic of !is_mddev_idle() cause the sync request wait too short and
>>>> reduce the chance for raid5d to handle the random read I/O.
>>>
>>> This has been raised before.
>>> Can you please try the patch at the end of
>>>
>>> http://permalink.gmane.org/gmane.linux.raid/51002
>>>
>>> and let me know if it makes any difference. If it isn't sufficient I
>>> will explore further.
>>>
>>> Thanks,
>>> NeilBrown
>>
>>
>> Hello Neil,
>>
>> I try the patch (http://permalink.gmane.org/gmane.linux.raid/51002) in
>> Kernel 4.2.8. Here are the test results:
>>
>>
>> Part I. SSD (4 x 240GB Intel SSD create Raid6(syncing))
>>
>> a. 4K Random Read, numjobs=64
>>
>> Average Throughput Average IOPS
>>
>> Kernel 4.2.8 Patch 601249KB/s 150312
>>
>>
>> b. 4K Random Read, numjobs=1
>>
>> Average Throughput Average IOPS
>>
>> Kernel 4.2.8 Patch 1166.4KB/s 291
>>
>>
>>
>> Part II. HDD (4 x 1TB TOSHIBA HDD create Raid6(syncing))
>>
>> a. 4K Random Read, numjobs=64
>>
>> Average Throughput Average IOPS
>>
>> Kernel 4.2.8 Patch 2946.4KB/s 736
>>
>>
>> b. 4K Random Read, numjobs=1
>>
>> Average Throughput Average IOPS
>>
>> Kernel 4.2.8 Patch 119199 B/s 28
>>
>>
>> Although the performance that compare to the original Kernel 4.2.8
>> test results is increased, the patch
>> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ac8fa4196d205ac8fff3f8932bddbad4f16e4110)
>> rollback still has the best performance. I also observe the sync speed
>> at numjobs=64 almost drop to the sync_speed_min, but sync speed at
>> numjobs=1 almost keep in the original speed.
>>
>> >From my test results, I think this patch isn't sufficient that maybe
>> Neil can explore further and give me some advice.
>>
>>
>> Thanks,
>> Chien Lee
>>
>>
>>>>
>>>>
>>>> Following is our test environment and some testing results:
>>>>
>>>>
>>>> OS: CentOS Linux release 7.1.1503 (Core)
>>>>
>>>> CPU: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
>>>>
>>>> Processor number: 8
>>>>
>>>> Memory: 12GB
>>>>
>>>> fio command:
>>>>
>>>> 1. (for numjobs=64):
>>>>
>>>> fio --filename=/dev/md2 --sync=0 --direct=0 --rw=randread --bs=4K
>>>> --runtime=180 --size=50G --name=test-read --ioengine=libaio
>>>> --numjobs=64 --iodepth=1 --group_reporting
>>>>
>>>> 2. (for numjobs=1):
>>>>
>>>> fio --filename=/dev/md2 --sync=0 --direct=0 --rw=randread --bs=4K
>>>> --runtime=180 --size=50G --name=test-read --ioengine=libaio
>>>> --numjobs=1 --iodepth=1 --group_reporting
>>>>
>>>>
>>>>
>>>> Here are test results:
>>>>
>>>>
>>>> Part I. SSD (4 x 240GB Intel SSD create Raid6(syncing))
>>>>
>>>>
>>>> a. 4K Random Read, numjobs=64
>>>>
>>>> Average Throughput Average IOPS
>>>>
>>>> Kernel 3.19.8 715937KB/s 178984
>>>>
>>>> Kernel 4.2.8 489874KB/s 122462
>>>>
>>>> Kernel 4.2.8 Patch Rollback 717377KB/s 179344
>>>>
>>>>
>>>>
>>>> b. 4K Random Read, numjobs=1
>>>>
>>>> Average Throughput Average IOPS
>>>>
>>>> Kernel 3.19.8 32203KB/s 8051
>>>>
>>>> Kernel 4.2.8 2535.7KB/s 633
>>>>
>>>> Kernel 4.2.8 Patch Rollback 31861KB/s 7965
>>>>
>>>>
>>>>
>>>>
>>>> Part II. HDD (4 x 1TB TOSHIBA HDD create Raid6(syncing))
>>>>
>>>>
>>>> a. 4K Random Read, numjobs=64
>>>>
>>>> Average Throughput Average IOPS
>>>>
>>>> Kernel 3.19.8 2976.6KB/s 744
>>>>
>>>> Kernel 4.2.8 2915.8KB/s 728
>>>>
>>>> Kernel 4.2.8 Patch Rollback 2973.3KB/s 743
>>>>
>>>>
>>>>
>>>> b. 4K Random Read, numjobs=1
>>>>
>>>> Average Throughput Average IOPS
>>>>
>>>> Kernel 3.19.8 481844 B/s 117
>>>>
>>>> Kernel 4.2.8 24718 B/s 5
>>>>
>>>> Kernel 4.2.8 Patch Rollback 460090 B/s 112
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>>
>>>> Chien Lee
>
> Thanks for testing.
>
> I'd like to suggest that these results are fairly reasonable for the
> numjobs=64 case. Certainly read-speed is reduced by presumably resync
> speed is increased.
> The numbers for numjob=1 are appalling though. That would generally
> affect any synchronous load. As the synchronous load doesn't interfere
> much with the resync load, the delays that are inserted won't be very
> long.
>
> I feel there must be an answer here - I just cannot find it.
> I'd like to be able to dynamically estimate the bandwidth of the array
> and use (say) 10% of that, but I cannot think of a way to do that at all
> reliably.
>
> I'll ponder it a bit longer. We may need to ultimately revert that
> patch, but not yet.
>
> Thanks,
> NeilBrown
>
So I was one of the original reporters who noticed the problem on some old SGI
hardware that uses a QL1040B chipset. Per hdparm -tT, the upper-end of the
speed to an MD device on this machine (an SGI Octane) is ~18.5MB/s.
I've been testing other kernel changes on this system, and finally managed to
scramble one of the disks enough that MD kicked off a resync on my largest
partition when booting and slowed the userland bringup down. But, I also
recently enabled the bitmaps feature, and while it took about ~20mins to boot
to runlevel 3, by the time it got there, the resync had completed. Usually, if
MD forces a resync, it'd resync that entire partition, which usually took 2+ hours.
So, a win for bitmaps, but the resync issue does need to be dealt with at some
point. I suspect I noticed it first because this isn't exactly fast hardware
for this day and age (dual 600MHz CPUs), and the modified resync algorithm is
more aggressive in grabbing resources to complete its job (which I don't blame
it, you're skating on thin ice during the small resync window).
As far as a solution, can MD, when it needs to resync, run a test similar to
hdparm to check the speed to one of the member disks and use that value as a
basis to calculate the I/O it needs? I.e., if it can determine that the upper
bound is ~18.5MB/s, it can then work out how much to use when the system is
idle and when it's not idle?
--
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
"The past tempts us, the present confuses us, the future frightens us. And our
lives slip away, moment by moment, lost in that vast, terrible in-between."
--Emperor Turhan, Centauri Republic
^ permalink raw reply
* ANNOUNCE: mdadm 3.4 - A tool for managing md Soft RAID under Linux
From: NeilBrown @ 2016-01-28 6:18 UTC (permalink / raw)
To: linux-raid, LKML; +Cc: Jes Sorensen
[-- Attachment #1: Type: text/plain, Size: 812 bytes --]
I am pleased to announce the availability of
mdadm version 3.4
It is available at the usual places:
http://www.kernel.org/pub/linux/utils/raid/mdadm/
and via git at
git://github.com/neilbrown/mdadm
git://neil.brown.name/mdadm
http://git.neil.brown.name/git/mdadm
The new second-level version number reflects significant new
functionality, particular support for journalled RAID5/6 and clustered
RAID1. This new support is probably still buggy. Please report bugs.
There are also a number of fixes for Intel's IMSM metadata support,
and an assortment of minor bug fixes.
I plan for this to be the last release of mdadm that I provide as I am
retiring from MD and mdadm maintenance. Jes Sorensen has volunteered
to oversee mdadm for the next while. Thanks Jes!
NeilBrown 28th January 2016
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: mdadm bad blocks list
From: NeilBrown @ 2016-01-28 4:45 UTC (permalink / raw)
To: Sarah Newman, linux-raid
In-Reply-To: <56A99118.2080508@prgmr.com>
[-- Attachment #1: Type: text/plain, Size: 3633 bytes --]
On Thu, Jan 28 2016, Sarah Newman wrote:
> On 01/27/2016 07:19 PM, NeilBrown wrote:
>> On Thu, Jan 28 2016, Sarah Newman wrote:
>>
>>> I experienced the following problems with the mdadm bad blocks list:
>>>
>>> 1. Additions to the bad block list do not cause an email to be sent by the mdadm monitor. Expected behavior is for an email to be sent as soon as the
>>> bad blocks list becomes non-empty.
>>
>> Yes, that would be a good idea. If you do develop patches, please post
>> them.
>
> Will do, but I don't have a definite time frame for it.
>
>>
>>> 2. /proc/mdstat does not show any indication that there are bad blocks present on an md member. Specifically, the status for the raid personality
>>> should show something other than "U" if the badblocks list is not empty for that member (maybe "B"?)
>>
>> I'd like to deprecate /proc/mdstat. It is not really easy to extend.
>> People might have programs that parse it which could break if you change
>> 'U' to 'B'.
>> I'd recommend using "mdadm" to get status of an array, or examine file
>> in /sys.
>
> If /proc/mdstat isn't going to be updated, is it going to be removed? If not and changing 'U' to 'B' isn't acceptable, then what about adding a flag
> to the device? Example
Removing is not better than changing. Legacy is a problem...
>
> md0 : active raid1 sda1[1] sdb1[2](B)
That might be acceptable. There is precedent for that sort of change.
>
> Where is the bad blocks list in /sys?
/sys/block/mdXXX/md/dev-YYY/bad_blocks
>
>>
>>> 3. Adding a device when there is an md member with bad blocks does not appear to trigger a rebuild, meaning there could be at least one good copy of
>>> all the data but no way to get all good data on a single device without expanding the entire array.
>>
>> Good point. That would be quite easy to change. Just set
>> WantReplacement if the bad block list is ever empty.
>> Not sure it is always a good idea though. You can have a bad block on a
>> perfectly good device if the device it was recovered from has a bad
>> block.
>> You only really want to set WantReplacement automatically if a write
>> fails. We do do that, but if you stop and restart an array the fact
>> that a write failed can be forgotten.
>
> Yes, I am quite aware there can be a bad block on a perfectly good device. But in a mirror if there are multiple perfectly good devices that each have
> bad blocks marked for whatever reason, the only way to get back to a single good device is to rebuild off of all of them. Speaking as a user, this is
> what I would want to happen.
Performing a "check" - e.g.
echo check > /sys/block/mdXXX/md/sync_action
should do that. I'm not certain that it does but it is an avenue worth
exploring and possibly fixing.
Running "check" on a regular basis is something everyone should do
(there is a script in mdadm to help with this).
>
>> I'm not convinced that it is harmful, though I accept that it is not perfect.
>
> Yes. You both know the current behavior of mdadm perfectly and probably didn't just experience data loss.
Fair comment.
>
> The old behavior was to fail immediately and alert if there was a problem rather than silently accepting errors. I expect there are some people who
> think they have a good RAID, but don't, based on /proc/mdstat and lack of errors from mdadm monitor.
>
> Thanks, Sarah
Getting feed back like this is an important part of making MD better!
I'm unlikely to be coding any changes myself in the immediate future
but I'm very happy to discuss them.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: Chien Lee @ 2016-01-28 4:42 UTC (permalink / raw)
To: NeilBrown, linux-raid, shli, owner-linux-raid
In-Reply-To: <87wpqu1jrl.fsf@notabene.neil.brown.name>
>
> Thanks for testing.
>
> I'd like to suggest that these results are fairly reasonable for the
> numjobs=64 case. Certainly read-speed is reduced by presumably resync
> speed is increased.
> The numbers for numjob=1 are appalling though. That would generally
> affect any synchronous load. As the synchronous load doesn't interfere
> much with the resync load, the delays that are inserted won't be very
> long.
>
> I feel there must be an answer here - I just cannot find it.
> I'd like to be able to dynamically estimate the bandwidth of the array
> and use (say) 10% of that, but I cannot think of a way to do that at all
> reliably.
>
> I'll ponder it a bit longer. We may need to ultimately revert that
> patch, but not yet.
>
> Thanks,
> NeilBrown
Hello Neil,
This issue about competing between sync IO and non-sync IO is a
trade-off question
and need to consider deeply and widely. Thanks for your advice.
If any progress about this, please let me know first. I'm happy that I
can provide
my test results for community.
Thanks again,
Chien Lee
^ permalink raw reply
* Re: mdadm bad blocks list
From: Sarah Newman @ 2016-01-28 3:55 UTC (permalink / raw)
To: NeilBrown, linux-raid
In-Reply-To: <87twly1jc7.fsf@notabene.neil.brown.name>
On 01/27/2016 07:19 PM, NeilBrown wrote:
> On Thu, Jan 28 2016, Sarah Newman wrote:
>
>> I experienced the following problems with the mdadm bad blocks list:
>>
>> 1. Additions to the bad block list do not cause an email to be sent by the mdadm monitor. Expected behavior is for an email to be sent as soon as the
>> bad blocks list becomes non-empty.
>
> Yes, that would be a good idea. If you do develop patches, please post
> them.
Will do, but I don't have a definite time frame for it.
>
>> 2. /proc/mdstat does not show any indication that there are bad blocks present on an md member. Specifically, the status for the raid personality
>> should show something other than "U" if the badblocks list is not empty for that member (maybe "B"?)
>
> I'd like to deprecate /proc/mdstat. It is not really easy to extend.
> People might have programs that parse it which could break if you change
> 'U' to 'B'.
> I'd recommend using "mdadm" to get status of an array, or examine file
> in /sys.
If /proc/mdstat isn't going to be updated, is it going to be removed? If not and changing 'U' to 'B' isn't acceptable, then what about adding a flag
to the device? Example
md0 : active raid1 sda1[1] sdb1[2](B)
Where is the bad blocks list in /sys?
>
>> 3. Adding a device when there is an md member with bad blocks does not appear to trigger a rebuild, meaning there could be at least one good copy of
>> all the data but no way to get all good data on a single device without expanding the entire array.
>
> Good point. That would be quite easy to change. Just set
> WantReplacement if the bad block list is ever empty.
> Not sure it is always a good idea though. You can have a bad block on a
> perfectly good device if the device it was recovered from has a bad
> block.
> You only really want to set WantReplacement automatically if a write
> fails. We do do that, but if you stop and restart an array the fact
> that a write failed can be forgotten.
Yes, I am quite aware there can be a bad block on a perfectly good device. But in a mirror if there are multiple perfectly good devices that each have
bad blocks marked for whatever reason, the only way to get back to a single good device is to rebuild off of all of them. Speaking as a user, this is
what I would want to happen.
> I'm not convinced that it is harmful, though I accept that it is not perfect.
Yes. You both know the current behavior of mdadm perfectly and probably didn't just experience data loss.
The old behavior was to fail immediately and alert if there was a problem rather than silently accepting errors. I expect there are some people who
think they have a good RAID, but don't, based on /proc/mdstat and lack of errors from mdadm monitor.
Thanks, Sarah
^ permalink raw reply
* Re: mdadm bad blocks list
From: NeilBrown @ 2016-01-28 3:19 UTC (permalink / raw)
To: Sarah Newman, linux-raid
In-Reply-To: <56A9102D.4030304@prgmr.com>
[-- Attachment #1: Type: text/plain, Size: 2410 bytes --]
On Thu, Jan 28 2016, Sarah Newman wrote:
> I experienced the following problems with the mdadm bad blocks list:
>
> 1. Additions to the bad block list do not cause an email to be sent by the mdadm monitor. Expected behavior is for an email to be sent as soon as the
> bad blocks list becomes non-empty.
Yes, that would be a good idea. If you do develop patches, please post
them.
> 2. /proc/mdstat does not show any indication that there are bad blocks present on an md member. Specifically, the status for the raid personality
> should show something other than "U" if the badblocks list is not empty for that member (maybe "B"?)
I'd like to deprecate /proc/mdstat. It is not really easy to extend.
People might have programs that parse it which could break if you change
'U' to 'B'.
I'd recommend using "mdadm" to get status of an array, or examine file
in /sys.
> 3. Adding a device when there is an md member with bad blocks does not appear to trigger a rebuild, meaning there could be at least one good copy of
> all the data but no way to get all good data on a single device without expanding the entire array.
Good point. That would be quite easy to change. Just set
WantReplacement if the bad block list is ever empty.
Not sure it is always a good idea though. You can have a bad block on a
perfectly good device if the device it was recovered from has a bad
block.
You only really want to set WantReplacement automatically if a write
fails. We do do that, but if you stop and restart an array the fact
that a write failed can be forgotten.
>
> Kernel: CentOS 6 Xen4CentOS 3.18.21-17
> mdadm: CentOS 6 v3.3.2
>
> With the above behavior, I consider the bad blocks list to be actively harmful. If it's expected behavior in the current version, please consider
> disabling the bad blocks list by default.
You can do this yourself by putting
CREATE bbl=no
in /etc/mdadm.conf. That doesn't help others though.
I'm not convinced that it is harmful, though I accept that it is not perfect.
> We might be able to provide some patches to correct 1. and 2. but we don't have anything ready right now.
That would be great if you could.
Thanks for your thoughts.
NeilBrown
>
> --Sarah
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: NeilBrown @ 2016-01-28 3:10 UTC (permalink / raw)
To: Chien Lee, linux-raid, shli, owner-linux-raid
In-Reply-To: <CAByoP04se9yQv_u1Kc1pXk5shtzT_udTB4v4HYaJRfc0Eh1RqA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6494 bytes --]
On Wed, Jan 27 2016, Chien Lee wrote:
> 2016-01-27 6:12 GMT+08:00 NeilBrown <neilb@suse.com>:
>> On Tue, Jan 26 2016, Chien Lee wrote:
>>
>>> Hello,
>>>
>>> Recently we find a bug about this patch (commit No. is
>>> ac8fa4196d205ac8fff3f8932bddbad4f16e4110 ).
>>>
>>> We know that this patch committed after Linux kernel 4.1.x is intended
>>> to allowing resync to go faster when there is competing IO. However,
>>> we find the performance of random read on syncing Raid6 will come up
>>> with a huge drop in this case. The following is our testing detail.
>>>
>>> The OS what we choose in our test is CentOS Linux release 7.1.1503
>>> (Core) and the kernel image will be replaced for testing. In our
>>> testing result, the 4K random read performance on syncing raid6 in
>>> Kernel 4.2.8 is much lower than in Kernel 3.19.8. In order to find out
>>> the root cause, we try to rollback this patch in Kernel 4.2.8, and we
>>> find the 4K random read performance on syncing Raid6 will be improved
>>> and go back to as what it should be in Kernel 3.19.8.
>>>
>>> Nevertheless, it seems that it will not affect some other read/write
>>> patterns. In our testing result, the 1M sequential read/write, 4K
>>> random write performance in Kernel 4.2.8 is performed almost the same
>>> as in Kernel 3.19.8.
>>>
>>> It seems that although this patch increases the resync speed, the
>>> logic of !is_mddev_idle() cause the sync request wait too short and
>>> reduce the chance for raid5d to handle the random read I/O.
>>
>> This has been raised before.
>> Can you please try the patch at the end of
>>
>> http://permalink.gmane.org/gmane.linux.raid/51002
>>
>> and let me know if it makes any difference. If it isn't sufficient I
>> will explore further.
>>
>> Thanks,
>> NeilBrown
>
>
> Hello Neil,
>
> I try the patch (http://permalink.gmane.org/gmane.linux.raid/51002) in
> Kernel 4.2.8. Here are the test results:
>
>
> Part I. SSD (4 x 240GB Intel SSD create Raid6(syncing))
>
> a. 4K Random Read, numjobs=64
>
> Average Throughput Average IOPS
>
> Kernel 4.2.8 Patch 601249KB/s 150312
>
>
> b. 4K Random Read, numjobs=1
>
> Average Throughput Average IOPS
>
> Kernel 4.2.8 Patch 1166.4KB/s 291
>
>
>
> Part II. HDD (4 x 1TB TOSHIBA HDD create Raid6(syncing))
>
> a. 4K Random Read, numjobs=64
>
> Average Throughput Average IOPS
>
> Kernel 4.2.8 Patch 2946.4KB/s 736
>
>
> b. 4K Random Read, numjobs=1
>
> Average Throughput Average IOPS
>
> Kernel 4.2.8 Patch 119199 B/s 28
>
>
> Although the performance that compare to the original Kernel 4.2.8
> test results is increased, the patch
> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ac8fa4196d205ac8fff3f8932bddbad4f16e4110)
> rollback still has the best performance. I also observe the sync speed
> at numjobs=64 almost drop to the sync_speed_min, but sync speed at
> numjobs=1 almost keep in the original speed.
>
>>From my test results, I think this patch isn't sufficient that maybe
> Neil can explore further and give me some advice.
>
>
> Thanks,
> Chien Lee
>
>
>>>
>>>
>>> Following is our test environment and some testing results:
>>>
>>>
>>> OS: CentOS Linux release 7.1.1503 (Core)
>>>
>>> CPU: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
>>>
>>> Processor number: 8
>>>
>>> Memory: 12GB
>>>
>>> fio command:
>>>
>>> 1. (for numjobs=64):
>>>
>>> fio --filename=/dev/md2 --sync=0 --direct=0 --rw=randread --bs=4K
>>> --runtime=180 --size=50G --name=test-read --ioengine=libaio
>>> --numjobs=64 --iodepth=1 --group_reporting
>>>
>>> 2. (for numjobs=1):
>>>
>>> fio --filename=/dev/md2 --sync=0 --direct=0 --rw=randread --bs=4K
>>> --runtime=180 --size=50G --name=test-read --ioengine=libaio
>>> --numjobs=1 --iodepth=1 --group_reporting
>>>
>>>
>>>
>>> Here are test results:
>>>
>>>
>>> Part I. SSD (4 x 240GB Intel SSD create Raid6(syncing))
>>>
>>>
>>> a. 4K Random Read, numjobs=64
>>>
>>> Average Throughput Average IOPS
>>>
>>> Kernel 3.19.8 715937KB/s 178984
>>>
>>> Kernel 4.2.8 489874KB/s 122462
>>>
>>> Kernel 4.2.8 Patch Rollback 717377KB/s 179344
>>>
>>>
>>>
>>> b. 4K Random Read, numjobs=1
>>>
>>> Average Throughput Average IOPS
>>>
>>> Kernel 3.19.8 32203KB/s 8051
>>>
>>> Kernel 4.2.8 2535.7KB/s 633
>>>
>>> Kernel 4.2.8 Patch Rollback 31861KB/s 7965
>>>
>>>
>>>
>>>
>>> Part II. HDD (4 x 1TB TOSHIBA HDD create Raid6(syncing))
>>>
>>>
>>> a. 4K Random Read, numjobs=64
>>>
>>> Average Throughput Average IOPS
>>>
>>> Kernel 3.19.8 2976.6KB/s 744
>>>
>>> Kernel 4.2.8 2915.8KB/s 728
>>>
>>> Kernel 4.2.8 Patch Rollback 2973.3KB/s 743
>>>
>>>
>>>
>>> b. 4K Random Read, numjobs=1
>>>
>>> Average Throughput Average IOPS
>>>
>>> Kernel 3.19.8 481844 B/s 117
>>>
>>> Kernel 4.2.8 24718 B/s 5
>>>
>>> Kernel 4.2.8 Patch Rollback 460090 B/s 112
>>>
>>>
>>>
>>> Thanks,
>>>
>>> --
>>>
>>> Chien Lee
Thanks for testing.
I'd like to suggest that these results are fairly reasonable for the
numjobs=64 case. Certainly read-speed is reduced by presumably resync
speed is increased.
The numbers for numjob=1 are appalling though. That would generally
affect any synchronous load. As the synchronous load doesn't interfere
much with the resync load, the delays that are inserted won't be very
long.
I feel there must be an answer here - I just cannot find it.
I'd like to be able to dynamically estimate the bandwidth of the array
and use (say) 10% of that, but I cannot think of a way to do that at all
reliably.
I'll ponder it a bit longer. We may need to ultimately revert that
patch, but not yet.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* mdadm bad blocks list
From: Sarah Newman @ 2016-01-27 18:45 UTC (permalink / raw)
To: linux-raid
I experienced the following problems with the mdadm bad blocks list:
1. Additions to the bad block list do not cause an email to be sent by the mdadm monitor. Expected behavior is for an email to be sent as soon as the
bad blocks list becomes non-empty.
2. /proc/mdstat does not show any indication that there are bad blocks present on an md member. Specifically, the status for the raid personality
should show something other than "U" if the badblocks list is not empty for that member (maybe "B"?)
3. Adding a device when there is an md member with bad blocks does not appear to trigger a rebuild, meaning there could be at least one good copy of
all the data but no way to get all good data on a single device without expanding the entire array.
Kernel: CentOS 6 Xen4CentOS 3.18.21-17
mdadm: CentOS 6 v3.3.2
With the above behavior, I consider the bad blocks list to be actively harmful. If it's expected behavior in the current version, please consider
disabling the bad blocks list by default. We might be able to provide some patches to correct 1. and 2. but we don't have anything ready right now.
--Sarah
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: Chien Lee @ 2016-01-27 9:49 UTC (permalink / raw)
To: NeilBrown, linux-raid, shli, owner-linux-raid
In-Reply-To: <87si1k2do8.fsf@notabene.neil.brown.name>
2016-01-27 6:12 GMT+08:00 NeilBrown <neilb@suse.com>:
> On Tue, Jan 26 2016, Chien Lee wrote:
>
>> Hello,
>>
>> Recently we find a bug about this patch (commit No. is
>> ac8fa4196d205ac8fff3f8932bddbad4f16e4110 ).
>>
>> We know that this patch committed after Linux kernel 4.1.x is intended
>> to allowing resync to go faster when there is competing IO. However,
>> we find the performance of random read on syncing Raid6 will come up
>> with a huge drop in this case. The following is our testing detail.
>>
>> The OS what we choose in our test is CentOS Linux release 7.1.1503
>> (Core) and the kernel image will be replaced for testing. In our
>> testing result, the 4K random read performance on syncing raid6 in
>> Kernel 4.2.8 is much lower than in Kernel 3.19.8. In order to find out
>> the root cause, we try to rollback this patch in Kernel 4.2.8, and we
>> find the 4K random read performance on syncing Raid6 will be improved
>> and go back to as what it should be in Kernel 3.19.8.
>>
>> Nevertheless, it seems that it will not affect some other read/write
>> patterns. In our testing result, the 1M sequential read/write, 4K
>> random write performance in Kernel 4.2.8 is performed almost the same
>> as in Kernel 3.19.8.
>>
>> It seems that although this patch increases the resync speed, the
>> logic of !is_mddev_idle() cause the sync request wait too short and
>> reduce the chance for raid5d to handle the random read I/O.
>
> This has been raised before.
> Can you please try the patch at the end of
>
> http://permalink.gmane.org/gmane.linux.raid/51002
>
> and let me know if it makes any difference. If it isn't sufficient I
> will explore further.
>
> Thanks,
> NeilBrown
Hello Neil,
I try the patch (http://permalink.gmane.org/gmane.linux.raid/51002) in
Kernel 4.2.8. Here are the test results:
Part I. SSD (4 x 240GB Intel SSD create Raid6(syncing))
a. 4K Random Read, numjobs=64
Average Throughput Average IOPS
Kernel 4.2.8 Patch 601249KB/s 150312
b. 4K Random Read, numjobs=1
Average Throughput Average IOPS
Kernel 4.2.8 Patch 1166.4KB/s 291
Part II. HDD (4 x 1TB TOSHIBA HDD create Raid6(syncing))
a. 4K Random Read, numjobs=64
Average Throughput Average IOPS
Kernel 4.2.8 Patch 2946.4KB/s 736
b. 4K Random Read, numjobs=1
Average Throughput Average IOPS
Kernel 4.2.8 Patch 119199 B/s 28
Although the performance that compare to the original Kernel 4.2.8
test results is increased, the patch
(http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ac8fa4196d205ac8fff3f8932bddbad4f16e4110)
rollback still has the best performance. I also observe the sync speed
at numjobs=64 almost drop to the sync_speed_min, but sync speed at
numjobs=1 almost keep in the original speed.
From my test results, I think this patch isn't sufficient that maybe
Neil can explore further and give me some advice.
Thanks,
Chien Lee
>>
>>
>> Following is our test environment and some testing results:
>>
>>
>> OS: CentOS Linux release 7.1.1503 (Core)
>>
>> CPU: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
>>
>> Processor number: 8
>>
>> Memory: 12GB
>>
>> fio command:
>>
>> 1. (for numjobs=64):
>>
>> fio --filename=/dev/md2 --sync=0 --direct=0 --rw=randread --bs=4K
>> --runtime=180 --size=50G --name=test-read --ioengine=libaio
>> --numjobs=64 --iodepth=1 --group_reporting
>>
>> 2. (for numjobs=1):
>>
>> fio --filename=/dev/md2 --sync=0 --direct=0 --rw=randread --bs=4K
>> --runtime=180 --size=50G --name=test-read --ioengine=libaio
>> --numjobs=1 --iodepth=1 --group_reporting
>>
>>
>>
>> Here are test results:
>>
>>
>> Part I. SSD (4 x 240GB Intel SSD create Raid6(syncing))
>>
>>
>> a. 4K Random Read, numjobs=64
>>
>> Average Throughput Average IOPS
>>
>> Kernel 3.19.8 715937KB/s 178984
>>
>> Kernel 4.2.8 489874KB/s 122462
>>
>> Kernel 4.2.8 Patch Rollback 717377KB/s 179344
>>
>>
>>
>> b. 4K Random Read, numjobs=1
>>
>> Average Throughput Average IOPS
>>
>> Kernel 3.19.8 32203KB/s 8051
>>
>> Kernel 4.2.8 2535.7KB/s 633
>>
>> Kernel 4.2.8 Patch Rollback 31861KB/s 7965
>>
>>
>>
>>
>> Part II. HDD (4 x 1TB TOSHIBA HDD create Raid6(syncing))
>>
>>
>> a. 4K Random Read, numjobs=64
>>
>> Average Throughput Average IOPS
>>
>> Kernel 3.19.8 2976.6KB/s 744
>>
>> Kernel 4.2.8 2915.8KB/s 728
>>
>> Kernel 4.2.8 Patch Rollback 2973.3KB/s 743
>>
>>
>>
>> b. 4K Random Read, numjobs=1
>>
>> Average Throughput Average IOPS
>>
>> Kernel 3.19.8 481844 B/s 117
>>
>> Kernel 4.2.8 24718 B/s 5
>>
>> Kernel 4.2.8 Patch Rollback 460090 B/s 112
>>
>>
>>
>> Thanks,
>>
>> --
>>
>> Chien Lee
^ permalink raw reply
* Re: md127 auto created when use "-B" to build a legacy array without superblocks
From: yizhan @ 2016-01-27 9:18 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <87y4bc2e2m.fsf@notabene.neil.brown.name>
On 01/27/2016 06:03 AM, NeilBrown wrote:
> On Tue, Jan 26 2016, yizhan wrote:
>
>> On 10/02/2015 03:55 PM, Neil Brown wrote:
>>> Yi Zhang <yizhan@redhat.com> writes:
>>>
>>>> Hi Neil
>>>>
>>>> When testing 00raid1, found the md127 auto created when use "-B" to build a legacy array without superblocks, is it reasonable?
>>> This happens because udev notices a new device has appeared, looks
>>> inside it, sees that it could be part of an md array, and so runs
>>> "mdadm --incremental" on it.
>> Hi Neil
>> I tried the mdadm --incremental, but it doesn't work, below is the
>> operation.
>> # mdadm -D /dev/md0
>> /dev/md0:
>> Version :
>> Creation Time : Tue Jan 26 03:16:17 2016
>> Raid Level : raid1
>> Array Size : 2097152 (2.00 GiB 2.15 GB)
>> Used Dev Size : 2097152 (2.00 GiB 2.15 GB)
>> Raid Devices : 2
>> Total Devices : 2
>>
>> State : clean
>> Active Devices : 2
>> Working Devices : 2
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Number Major Minor RaidDevice State
>> 0 7 0 0 active sync /dev/loop0
>> 1 7 1 1 active sync /dev/loop1
>> # mdadm -D /dev/md127
>> /dev/md127:
>> Version : 1.2
>> Raid Level : raid0
>> Total Devices : 1
>> Persistence : Superblock is persistent
>>
>> State : inactive
>>
>> Name : 0
>> UUID : 13ada118:129135ff:686de7b7:4cb71d6d
>> Events : 17
>>
>> Number Major Minor RaidDevice
>>
>> - 9 0 - /dev/md0
>> # mdadm -I /dev/md127
>> mdadm: /dev/md127 is not part of an md array.
>> # mdadm -I /dev/md0
>> mdadm: cannot reopen /dev/md0: Device or resource busy.
>>
>> After I stopped md127, the md0 can be used again.
> If, after stopping md127, you try
>
> mdadm -I /dev/md0
>
> again it will start /dev/md127 again. This is what udev does which
> causes md127 to appear in the first place.
>
> NeilBrown
Thanks Neil
I have tried 'mdadm -I /dev/md0', the /dev/md127 start again
Yi
>
>> Thanks
>> Yi
>>> Sometimes we want udev to do that. Sometimes we don't.
>>> There is no easy way for udev to know what we want.
>>>
>>> NeilBrown
>>>
>>>
>>>> pls check below detailed info:
>>>>
>>>> + mdadm -CR /dev/md0 --level=raid1 -n3 /dev/loop0 /dev/loop1 /dev/loop2
>>>> mdadm: /dev/loop0 appears to contain an ext2fs file system
>>>> size=58368K mtime=Thu Jan 1 08:00:00 1970
>>>> mdadm: Note: this array has metadata at the start and
>>>> may not be suitable as a boot device. If you plan to
>>>> store '/boot' on this device please ensure that
>>>> your boot-loader understands md/v1.x metadata, or use
>>>> --metadata=0.90
>>>> mdadm: /dev/loop1 appears to contain an ext2fs file system
>>>> size=38912K mtime=Thu Jan 1 08:00:00 1970
>>>> mdadm: Defaulting to version 1.2 metadata
>>>> mdadm: array /dev/md0 started.
>>>> + mdadm --wait /dev/md0
>>>> + cat /proc/mdstat
>>>> Personalities : [raid6] [raid5] [raid4] [raid1]
>>>> md0 : active raid1 loop2[2] loop1[1] loop0[0]
>>>> 19968 blocks super 1.2 [3/3] [UUU]
>>>>
>>>> unused devices: <none>
>>>> + mdadm -S /dev/md0
>>>> mdadm: stopped /dev/md0
>>>> + mdadm -B /dev/md0 -l raid1 -n2 /dev/loop0 /dev/loop1
>>>> mdadm: array /dev/md0 built and started.
>>>> + sleep 2
>>>> + cat /proc/mdstat
>>>> Personalities : [raid6] [raid5] [raid4] [raid1]
>>>> md127 : inactive md0[0](S)
>>>> 19968 blocks super 1.2
>>>>
>>>> md0 : active raid1 loop1[1] loop0[0]
>>>> 20000 blocks super non-persistent [2/2] [UU]
>>>>
>>>> unused devices: <none>
>>>> [root@dhcp-12-171 bug]# uname -r
>>>> 4.2.0
>>>> [root@dhcp-12-171 bug]# mdadm -D /dev/md0
>>>> /dev/md0:
>>>> Version :
>>>> Creation Time : Mon Sep 7 20:21:20 2015
>>>> Raid Level : raid1
>>>> Array Size : 20000 (19.53 MiB 20.48 MB)
>>>> Used Dev Size : 20000 (19.53 MiB 20.48 MB)
>>>> Raid Devices : 2
>>>> Total Devices : 2
>>>>
>>>> State : clean
>>>> Active Devices : 2
>>>> Working Devices : 2
>>>> Failed Devices : 0
>>>> Spare Devices : 0
>>>>
>>>> Number Major Minor RaidDevice State
>>>> 0 7 0 0 active sync /dev/loop0
>>>> 1 7 1 1 active sync /dev/loop1
>>>> [root@dhcp-12-171 bug]# mdadm -D /dev/md127
>>>> /dev/md127:
>>>> Version : 1.2
>>>> Raid Level : raid0
>>>> Total Devices : 1
>>>> Persistence : Superblock is persistent
>>>>
>>>> State : inactive
>>>>
>>>> Name : dhcp-12-171.nay.redhat.com:0 (local to host dhcp-12-171.nay.redhat.com)
>>>> UUID : 40ace956:a9dd0793:f4984d2b:8431b92b
>>>> Events : 17
>>>>
>>>> Number Major Minor RaidDevice
>>>>
>>>> - 9 0 - /dev/md0
>>>>
>>>> Best Regards,
>>>> Yi Zhang
^ permalink raw reply
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
From: Eric Valette @ 2016-01-27 7:24 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid
In-Reply-To: <20160126233133.GB12721@kernel.org>
On 27/01/2016 00:31, Shaohua Li wrote:
> On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
>> Hi,
>>
>> My raid 10 array (5 disk with one spare) was doing a resync after an
>> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
>> the /proc/mdstat was apparently complete but when rebooting, it started
>> resycing over and over. I noticed my dmesg was totally filled with raid10
>> conf printout message so it was impossible to trace anything else.
>>
>> Did a resync test with 3.14.58 (because I knew it had worked for resync
>> before and was still available as a boot option) and the array was
>> correctly rebuild.
>> Runs fine with 4.1.16 now.
>>
>> Please CC me as I'm not subscribed.
>
> Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
> a stop/reassemble, does the resync start?
I'm not going to retry kernel 4.4.0 on this device as I'm no more
confident about raid10 support with this 4.4 version.
So on 4.1.16 :
mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Sun Jan 24 17:15:58 2016
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Layout : near=2
Chunk Size : 512K
Name : nas2:0 (local to host nas2)
UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Events : 480
Number Major Minor RaidDevice State
0 8 17 0 active sync set-A /dev/sdb1
1 8 33 1 active sync set-B /dev/sdc1
2 8 49 2 active sync set-A /dev/sdd1
3 8 65 3 active sync set-B /dev/sde1
4 8 81 - spare /dev/sdf1
uname -a
Linux nas2 4.1.16 #1 SMP Sat Jan 23 19:29:59 CET 2016 x86_64 GNU/Linux
And if I stop the array manually and reboot, its no more resynced with
this 4.1.16 kernel.
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: NeilBrown @ 2016-01-27 1:12 UTC (permalink / raw)
To: Shaohua Li; +Cc: Chien Lee, linux-raid, owner-linux-raid
In-Reply-To: <20160126232731.GA12721@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 4317 bytes --]
On Wed, Jan 27 2016, Shaohua Li wrote:
> On Wed, Jan 27, 2016 at 10:08:45AM +1100, Neil Brown wrote:
>> On Wed, Jan 27 2016, Shaohua Li wrote:
>>
>> > On Wed, Jan 27, 2016 at 09:12:23AM +1100, Neil Brown wrote:
>> >> On Tue, Jan 26 2016, Chien Lee wrote:
>> >>
>> >> > Hello,
>> >> >
>> >> > Recently we find a bug about this patch (commit No. is
>> >> > ac8fa4196d205ac8fff3f8932bddbad4f16e4110 ).
>> >> >
>> >> > We know that this patch committed after Linux kernel 4.1.x is intended
>> >> > to allowing resync to go faster when there is competing IO. However,
>> >> > we find the performance of random read on syncing Raid6 will come up
>> >> > with a huge drop in this case. The following is our testing detail.
>> >> >
>> >> > The OS what we choose in our test is CentOS Linux release 7.1.1503
>> >> > (Core) and the kernel image will be replaced for testing. In our
>> >> > testing result, the 4K random read performance on syncing raid6 in
>> >> > Kernel 4.2.8 is much lower than in Kernel 3.19.8. In order to find out
>> >> > the root cause, we try to rollback this patch in Kernel 4.2.8, and we
>> >> > find the 4K random read performance on syncing Raid6 will be improved
>> >> > and go back to as what it should be in Kernel 3.19.8.
>> >> >
>> >> > Nevertheless, it seems that it will not affect some other read/write
>> >> > patterns. In our testing result, the 1M sequential read/write, 4K
>> >> > random write performance in Kernel 4.2.8 is performed almost the same
>> >> > as in Kernel 3.19.8.
>> >> >
>> >> > It seems that although this patch increases the resync speed, the
>> >> > logic of !is_mddev_idle() cause the sync request wait too short and
>> >> > reduce the chance for raid5d to handle the random read I/O.
>> >>
>> >> This has been raised before.
>> >> Can you please try the patch at the end of
>> >>
>> >> http://permalink.gmane.org/gmane.linux.raid/51002
>> >>
>> >> and let me know if it makes any difference. If it isn't sufficient I
>> >> will explore further.
>> >
>> > I'm curious why we don't calculate the wait time. Say the target resync speed
>> > is speed_min. The wait time should be:
>> >
>> > (currspeed * SYNC_MARK_STEP - speed_min * SYNC_MARK_STEP) / speed_min
>> > = (currspeed / speed_min - 1) * SYNC_MARK_STEP
>> >
>> > if SYNC_MARK_STEP is too big and sync speed has drift, we can make it smaller.
>>
>> What do you hope this would achieve?
>
> The whole point is to throttle sync speed to specific speed. If we know the
> target speed, for any given time interval, we can calculate the sync
> IO size.
Actually, no. The main point is to not interfere with filesystem IO too
much. Limiting to a low target is a fairly poor way to do that (but is
all we have) and as there is such a wide range of device speeds it is no
longer possible to choose a sensible default.
>
>> If I understand correctly, this might allow the thread to sleep for
>> longer instead of looping around every 500ms or so. But we don't really
>> want to do that. As soon as filesystem IO pauses, we want resync IO to
>> go back to full speed.
>>
>> The "speed_min" isn't really a "target". It is only a "target" for
>> those times when there is no filesystem IO.
>
> Yep, target is a little bit hard to determine. I think we can do:
> if (curspeed > min) {
> if (!is_mddev_idle())
> targetspeed = minspeed;
> if (curspeed > max)
> targetspeed = maxspeed;
> sleep(max((currspeed / targetspeed - 1), 0) * SYNC_MARK_STEP)
> }
>
> This way we don't throttle if there is no filesystem IO. would this
> work?
But if there is filesystem IO, then we throttle for at least 3 seconds
(SYNC_MARK_STEP is 3*HZ). The filesystem could go idle in 1 second and
then we would spend 2 seconds pointlessly doing nothing. We could make
SYNC_MARK_STEP smaller, but if recent speed had been high we could still
throttle for a lot longer than needed.
Before the patch that caused the regression we would throttle for 500ms,
so if the filesystem went idle we would waste at most 500ms.
After the patch, we throttle until pending requests have completed.
This causes the delay to scale with the speed of the device, but doesn't
seem to be enough of a delay in some cases.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH] util: fix wrong return value of cluster_get_dlmlock
From: NeilBrown @ 2016-01-27 0:44 UTC (permalink / raw)
Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1453278085-19883-1-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]
On Wed, Jan 20 2016, Guoqing Jiang wrote:
> Actually lksb.sb_status means that a node get the lock
> or not instead of the return value of dlm_lock.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> util.c | 14 ++++----------
> 1 file changed, 4 insertions(+), 10 deletions(-)
>
> diff --git a/util.c b/util.c
> index f1b0b95..cf9572b 100644
> --- a/util.c
> +++ b/util.c
> @@ -142,7 +142,7 @@ int cluster_get_dlmlock(int *lockid)
> dlm_lock_res->ls = dlm_hooks->create_lockspace(cluster_name, O_RDWR);
> if (!dlm_lock_res->ls) {
> pr_err("%s failed to create lockspace\n", cluster_name);
> - goto out;
> + return -ENOMEM;
> }
>
> /* Conversions need the lockid in the LKSB */
> @@ -157,21 +157,15 @@ int cluster_get_dlmlock(int *lockid)
> dlm_lock_res, NULL, NULL);
> if (ret) {
> pr_err("error %d when get PW mode on lock %s\n", errno, str);
> - goto out;
> + dlm_hooks->release_lockspace(cluster_name, dlm_lock_res->ls, 1);
> + return ret;
> }
>
> /* Wait for it to complete */
> poll_for_ast(dlm_lock_res->ls);
> *lockid = dlm_lock_res->lksb.sb_lkid;
>
> - errno = dlm_lock_res->lksb.sb_status;
> - if (errno) {
> - pr_err("error %d happened in ast with lock %s\n", errno, str);
> - goto out;
> - }
> -
> -out:
> - return ret;
> + return dlm_lock_res->lksb.sb_status;
> }
>
> int cluster_release_dlmlock(int lockid)
> --
Applied, thanks.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Uneven wear on raid1 devices
From: Adam Goryachev @ 2016-01-27 0:18 UTC (permalink / raw)
To: Phil Turmel, Roman Mamedov; +Cc: linux-raid
In-Reply-To: <56A662A3.6000402@turmel.org>
On 26/01/16 05:00, Phil Turmel wrote:
> On 01/25/2016 12:40 AM, Roman Mamedov wrote:
>> On Mon, 25 Jan 2016 16:29:02 +1100
>> Adam Goryachev <adam@websitemanagers.com.au> wrote:
>>
>>> 9 Power_On_Hours -O--CK 098 098 000 - 6435
>>> 9 Power_On_Hours -O--CK 095 095 000 - 23178
>> 2nd drive has almost 4x as much power-on time than the first one. My guess
>> would be that it accumulated all that write usage back before you put it into
>> this RAID1.
> Or you are doing "repair" scrubs when you should be doing "check"
> scrubs. Any operation that requires resynchronization will read from
> the first mirror and write to the others.
No, definitely not doing a repair scrub, so sounds like the issue was
just the power on hours problem.
>> If you want to ensure the RAID1 usage is even, record the SMART data you have
>> now, and compare to the readings you will have a month later.
> By the way, your devices show scterc disabled. That's bad. You should
> definitely use boot scripts or udev rules to enable it.
Ooops, thank you for catching this, well understood how bad it is, I
just assumed (badly) that I was using decent drives (I always used the
WD enterprise black), and clearly forgot to check for this on an SSD.
I've enabled it now from /etc/rc.local, as well as fixing the scheduler
to noop on the SSD instead of cfq.
> Has sdd been getting bumped out of the array lately, and resyncing when
> you put it back?
No, but I wonder if this might have explained the drop from around 9
months ago, when I re-plugged it and it tested fine, but I asked them to
replace it anyway (which they did). So, I can't complain about it too much.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* RAID journal, requirements?
From: Linus Lüssing @ 2016-01-27 0:15 UTC (permalink / raw)
To: linux-raid
Hi,
I am very excited about the new journaling feature for RAID in md
and want to try it out soon.
Neil's article on LWN already gave a great overview and some hints
about it, but I am still a little unsure about the technical
requirements for the journal device. E.g. speed, reliability and
size-wise. And what the implications of failing them means.
I want to setup a small home NAS. My personal priorities are:
Firstly, low power-per-GB as it will be solar powered and battery
backed. Second priority is reliability. Performance is not that
important to me.
My current plan: Tiny ARM-based computer (Odroid C1+: 4x USB (2.0,
unfortunately) ports), 4x 2TB SATA HDDs (@0.7W idle, 1.7W max. each)
+ SATA-to-USB -> RAID5, microSD for OS + applications, an eMMC for
a RAID journal. (later, once they hopefully get available in April,
switch to the Turris Omnia which is ARM-based again but has
mini-PCIe, too, where I would attach mini-PCIe-to-SATA adapters)
Is it a stupid idea to use an eMMC for the journal in the first
place? I am unsure how quickly it would wear out. And seems like I
do not have SMART for it either... So it would probably fail badly
at some point. Is that an issue, can I simply replace the eMMC
then without any data loss? Will the current code at least try a
write-through directly to the HDDs once a journal disk throws
errors?
Does MD assume that an underlying device performs some sort of wear
leveling if necessary like SSDs do for instance - or does MD
itself distribute writes evenly over the journaling disk?
(Does anyone know whether eMMCs usually do some kind of wear leveling?)
Thanks for all your work on this awesome new feature! (and special
thanks to Neil for all the years of maintenance - it is a pitty you
have to step down :( )
Regards, Linus
^ permalink raw reply
* Re: Uneven wear on raid1 devices
From: Adam Goryachev @ 2016-01-27 0:05 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-raid
In-Reply-To: <20160125104027.597041a3@natsu>
On 25/01/16 16:40, Roman Mamedov wrote:
> On Mon, 25 Jan 2016 16:29:02 +1100
> Adam Goryachev <adam@websitemanagers.com.au> wrote:
>
>> 9 Power_On_Hours -O--CK 098 098 000 - 6435
>> 9 Power_On_Hours -O--CK 095 095 000 - 23178
> 2nd drive has almost 4x as much power-on time than the first one. My guess
> would be that it accumulated all that write usage back before you put it into
> this RAID1.
>
> If you want to ensure the RAID1 usage is even, record the SMART data you have
> now, and compare to the readings you will have a month later.
Hmmm, oops, I should have looked at that. Now of course, I realise one
drive was replaced under warranty when it failed around 6 months ago
(well, the value says just under 9 months actually), and that would
explain the difference in wear and power on hours (aside from the
initial full sync when the new device was first installed).
So, 2.5 years old, and 74% life remaining, and 9 months with 93% life
remaining.
Sounds good to me, I should expect these to last at least the 5 years I
was hoping for, and probably before that happens I will want to upgrade
them to increase capacity anyway.
Thanks for your help
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
From: Shaohua Li @ 2016-01-26 23:31 UTC (permalink / raw)
To: Eric Valette; +Cc: linux-raid
In-Reply-To: <56A78786.10103@Free.fr>
On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
> Hi,
>
> My raid 10 array (5 disk with one spare) was doing a resync after an
> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
> the /proc/mdstat was apparently complete but when rebooting, it started
> resycing over and over. I noticed my dmesg was totally filled with raid10
> conf printout message so it was impossible to trace anything else.
>
> Did a resync test with 3.14.58 (because I knew it had worked for resync
> before and was still available as a boot option) and the array was
> correctly rebuild.
> Runs fine with 4.1.16 now.
>
> Please CC me as I'm not subscribed.
Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
a stop/reassemble, does the resync start?
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: Shaohua Li @ 2016-01-26 23:27 UTC (permalink / raw)
To: NeilBrown; +Cc: Chien Lee, linux-raid, owner-linux-raid
In-Reply-To: <87mvrs2b2a.fsf@notabene.neil.brown.name>
On Wed, Jan 27, 2016 at 10:08:45AM +1100, Neil Brown wrote:
> On Wed, Jan 27 2016, Shaohua Li wrote:
>
> > On Wed, Jan 27, 2016 at 09:12:23AM +1100, Neil Brown wrote:
> >> On Tue, Jan 26 2016, Chien Lee wrote:
> >>
> >> > Hello,
> >> >
> >> > Recently we find a bug about this patch (commit No. is
> >> > ac8fa4196d205ac8fff3f8932bddbad4f16e4110 ).
> >> >
> >> > We know that this patch committed after Linux kernel 4.1.x is intended
> >> > to allowing resync to go faster when there is competing IO. However,
> >> > we find the performance of random read on syncing Raid6 will come up
> >> > with a huge drop in this case. The following is our testing detail.
> >> >
> >> > The OS what we choose in our test is CentOS Linux release 7.1.1503
> >> > (Core) and the kernel image will be replaced for testing. In our
> >> > testing result, the 4K random read performance on syncing raid6 in
> >> > Kernel 4.2.8 is much lower than in Kernel 3.19.8. In order to find out
> >> > the root cause, we try to rollback this patch in Kernel 4.2.8, and we
> >> > find the 4K random read performance on syncing Raid6 will be improved
> >> > and go back to as what it should be in Kernel 3.19.8.
> >> >
> >> > Nevertheless, it seems that it will not affect some other read/write
> >> > patterns. In our testing result, the 1M sequential read/write, 4K
> >> > random write performance in Kernel 4.2.8 is performed almost the same
> >> > as in Kernel 3.19.8.
> >> >
> >> > It seems that although this patch increases the resync speed, the
> >> > logic of !is_mddev_idle() cause the sync request wait too short and
> >> > reduce the chance for raid5d to handle the random read I/O.
> >>
> >> This has been raised before.
> >> Can you please try the patch at the end of
> >>
> >> http://permalink.gmane.org/gmane.linux.raid/51002
> >>
> >> and let me know if it makes any difference. If it isn't sufficient I
> >> will explore further.
> >
> > I'm curious why we don't calculate the wait time. Say the target resync speed
> > is speed_min. The wait time should be:
> >
> > (currspeed * SYNC_MARK_STEP - speed_min * SYNC_MARK_STEP) / speed_min
> > = (currspeed / speed_min - 1) * SYNC_MARK_STEP
> >
> > if SYNC_MARK_STEP is too big and sync speed has drift, we can make it smaller.
>
> What do you hope this would achieve?
The whole point is to throttle sync speed to specific speed. If we know the
target speed, for any given time interval, we can calculate the sync IO size.
> If I understand correctly, this might allow the thread to sleep for
> longer instead of looping around every 500ms or so. But we don't really
> want to do that. As soon as filesystem IO pauses, we want resync IO to
> go back to full speed.
>
> The "speed_min" isn't really a "target". It is only a "target" for
> those times when there is no filesystem IO.
Yep, target is a little bit hard to determine. I think we can do:
if (curspeed > min) {
if (!is_mddev_idle())
targetspeed = minspeed;
if (curspeed > max)
targetspeed = maxspeed;
sleep(max((currspeed / targetspeed - 1), 0) * SYNC_MARK_STEP)
}
This way we don't throttle if there is no filesystem IO. would this work?
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: NeilBrown @ 2016-01-26 23:08 UTC (permalink / raw)
To: Shaohua Li; +Cc: Chien Lee, linux-raid, owner-linux-raid
In-Reply-To: <20160126225222.GA29409@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 2684 bytes --]
On Wed, Jan 27 2016, Shaohua Li wrote:
> On Wed, Jan 27, 2016 at 09:12:23AM +1100, Neil Brown wrote:
>> On Tue, Jan 26 2016, Chien Lee wrote:
>>
>> > Hello,
>> >
>> > Recently we find a bug about this patch (commit No. is
>> > ac8fa4196d205ac8fff3f8932bddbad4f16e4110 ).
>> >
>> > We know that this patch committed after Linux kernel 4.1.x is intended
>> > to allowing resync to go faster when there is competing IO. However,
>> > we find the performance of random read on syncing Raid6 will come up
>> > with a huge drop in this case. The following is our testing detail.
>> >
>> > The OS what we choose in our test is CentOS Linux release 7.1.1503
>> > (Core) and the kernel image will be replaced for testing. In our
>> > testing result, the 4K random read performance on syncing raid6 in
>> > Kernel 4.2.8 is much lower than in Kernel 3.19.8. In order to find out
>> > the root cause, we try to rollback this patch in Kernel 4.2.8, and we
>> > find the 4K random read performance on syncing Raid6 will be improved
>> > and go back to as what it should be in Kernel 3.19.8.
>> >
>> > Nevertheless, it seems that it will not affect some other read/write
>> > patterns. In our testing result, the 1M sequential read/write, 4K
>> > random write performance in Kernel 4.2.8 is performed almost the same
>> > as in Kernel 3.19.8.
>> >
>> > It seems that although this patch increases the resync speed, the
>> > logic of !is_mddev_idle() cause the sync request wait too short and
>> > reduce the chance for raid5d to handle the random read I/O.
>>
>> This has been raised before.
>> Can you please try the patch at the end of
>>
>> http://permalink.gmane.org/gmane.linux.raid/51002
>>
>> and let me know if it makes any difference. If it isn't sufficient I
>> will explore further.
>
> I'm curious why we don't calculate the wait time. Say the target resync speed
> is speed_min. The wait time should be:
>
> (currspeed * SYNC_MARK_STEP - speed_min * SYNC_MARK_STEP) / speed_min
> = (currspeed / speed_min - 1) * SYNC_MARK_STEP
>
> if SYNC_MARK_STEP is too big and sync speed has drift, we can make it smaller.
What do you hope this would achieve?
If I understand correctly, this might allow the thread to sleep for
longer instead of looping around every 500ms or so. But we don't really
want to do that. As soon as filesystem IO pauses, we want resync IO to
go back to full speed.
The "speed_min" isn't really a "target". It is only a "target" for
those times when there is no filesystem IO.
I'd certainly be happy to discuss alternate approaches to this. Details
are important through.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: WANTED new maintainer for Linux/md (and mdadm)
From: NeilBrown @ 2016-01-26 23:01 UTC (permalink / raw)
To: Artur Paszkiewicz, linux-raid, dm-devel, LKML; +Cc: Jes Sorensen, Shaohua Li
In-Reply-To: <569F4505.6060308@intel.com>
[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]
On Wed, Jan 20 2016, Artur Paszkiewicz wrote:
>
> Hi Neil,
>
> Thank you for your work and time spent on maintaining MD/mdadm. I would also
> like to offer help for the emerging maintainership team. I've been working with
> MD RAID for more than 4 years, mostly testing and developing the IMSM-related
> parts on behalf of my employer - Intel. I realize that I was not very visible
> on this mailing list, but I think I have a pretty good knowledge about mdadm
> and the MD drivers. Now I have Intel's approval to take on maintaining MD RAID
> as part of my job, not focusing primarily on IMSM. I definitely feel more
> confident with maintaining mdadm, but I would certainly like to learn more
> about the kernel MD stack and help with it as much as I can.
Hi Arthur,
thanks for all your IMSM work over the years! It is great that you can
continue contributing and do so more broadly.
For the moment Jes Serensen will be co-ordinating mdadm (once I make a
release ... tomorrow?) and Shaohua Li will be looking after the
kernel/md side. I'm sure there is plenty of room for you too though.
Only one person (at a time) can queue patches, but several can
collaborate at development and support and bug fixing and testing and
....
I suggest you find ways to co-ordinate with them.
This:
https://bugzilla.kernel.org/show_bug.cgi?id=108741
is currently most important in my mind, but there are other things to
do.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: Shaohua Li @ 2016-01-26 22:52 UTC (permalink / raw)
To: NeilBrown; +Cc: Chien Lee, linux-raid, owner-linux-raid
In-Reply-To: <87si1k2do8.fsf@notabene.neil.brown.name>
On Wed, Jan 27, 2016 at 09:12:23AM +1100, Neil Brown wrote:
> On Tue, Jan 26 2016, Chien Lee wrote:
>
> > Hello,
> >
> > Recently we find a bug about this patch (commit No. is
> > ac8fa4196d205ac8fff3f8932bddbad4f16e4110 ).
> >
> > We know that this patch committed after Linux kernel 4.1.x is intended
> > to allowing resync to go faster when there is competing IO. However,
> > we find the performance of random read on syncing Raid6 will come up
> > with a huge drop in this case. The following is our testing detail.
> >
> > The OS what we choose in our test is CentOS Linux release 7.1.1503
> > (Core) and the kernel image will be replaced for testing. In our
> > testing result, the 4K random read performance on syncing raid6 in
> > Kernel 4.2.8 is much lower than in Kernel 3.19.8. In order to find out
> > the root cause, we try to rollback this patch in Kernel 4.2.8, and we
> > find the 4K random read performance on syncing Raid6 will be improved
> > and go back to as what it should be in Kernel 3.19.8.
> >
> > Nevertheless, it seems that it will not affect some other read/write
> > patterns. In our testing result, the 1M sequential read/write, 4K
> > random write performance in Kernel 4.2.8 is performed almost the same
> > as in Kernel 3.19.8.
> >
> > It seems that although this patch increases the resync speed, the
> > logic of !is_mddev_idle() cause the sync request wait too short and
> > reduce the chance for raid5d to handle the random read I/O.
>
> This has been raised before.
> Can you please try the patch at the end of
>
> http://permalink.gmane.org/gmane.linux.raid/51002
>
> and let me know if it makes any difference. If it isn't sufficient I
> will explore further.
I'm curious why we don't calculate the wait time. Say the target resync speed
is speed_min. The wait time should be:
(currspeed * SYNC_MARK_STEP - speed_min * SYNC_MARK_STEP) / speed_min
= (currspeed / speed_min - 1) * SYNC_MARK_STEP
if SYNC_MARK_STEP is too big and sync speed has drift, we can make it smaller.
Thanks,
Shaohua
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox