* Re: [PATCH 1/3] mdadm/test: Convert code format to use Tab
From: Jes Sorensen @ 2017-06-09 15:06 UTC (permalink / raw)
To: Zhilong Liu; +Cc: linux-raid
In-Reply-To: <1496827888-17118-2-git-send-email-zlliu@suse.com>
On 06/07/2017 05:31 AM, Zhilong Liu wrote:
> In case to make codes neat, this commit didn't change
> any codes, just tidy up and use Tab as code format.
>
> Signed-off-by: Zhilong Liu <zlliu@suse.com>
Tabs are good :)
Applied!
Jes
^ permalink raw reply
* Re: [PATCH 0/3] mdadm/test: Refactor the codes of 'test' script
From: Jes Sorensen @ 2017-06-09 15:08 UTC (permalink / raw)
To: Zhilong Liu; +Cc: linux-raid
In-Reply-To: <1496827888-17118-1-git-send-email-zlliu@suse.com>
On 06/07/2017 05:31 AM, Zhilong Liu wrote:
> Hi, Jes;
> This patchset mainly focus on improving the 'test' script, rip it
> apart and put it together in a more coherent form.
> Over 3 months passed, the RFC patch of improving 'test' hadn't received
> any ideas, thus I post this sample version that I have tested many times.
> It's just the beginning to improve the test part of mdadm, any questions
> is very welcome.
>
> Sorry for that, I haven't added the function suggested from Guoqing, cut
> the words from the email tree of
> "ANNOUNCE: mdadm 4.0 - A tool for managing md Soft RAID under Linux":
>
> Guoqing asked:
> ...
> Is current test can run the test for different raid level, say,
> "./test --raidtype=raid1"
> could execute all the *r1* tests, does it make sense to do it if we don't
> support it now.
> ...
>
> Just list this suggestion here, waiting for ideas.
>
> RFC patch mail-tree:
> [PATCH RFC] test: revise 'test' and make it easier to understand
> Acked-by: Neil Brown
> And suggest to split those changing into smaller patches from Guoqing.
The whole test infrastructure is something I would like to see rewamped
and modernized, so you starting to work on this is excellent.
With regard to your question, then having an option to say 'run all
raid1 tests' is probably good, we should have that for all raid levels,
imsm, and ddf.
I also would like to see a more flexible way to configure the tests,
like it should be possible to specify a bunch of physical devices for
the tests, instead of relying on loop devices.
Cheers,
Jes
^ permalink raw reply
* [GIT PULL] MD update for 4.12-rc5
From: Shaohua Li @ 2017-06-09 16:32 UTC (permalink / raw)
To: torvalds; +Cc: linux-kernel, linux-raid
Hi,
One bug fix from Neil Brown for MD. The bug is introduced in this cycle. Please
pull!
Thanks,
Shaohua
The following changes since commit 3c2993b8c6143d8a5793746a54eba8f86f95240f:
Linux 4.12-rc4 (2017-06-04 16:47:43 -0700)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/shli/md.git tags/md/4.12-rc5
for you to fetch changes up to a415c0f10627913793709ddb75add09d2ea334dc:
md: initialise ->writes_pending in personality modules. (2017-06-05 16:04:35 -0700)
----------------------------------------------------------------
NeilBrown (1):
md: initialise ->writes_pending in personality modules.
drivers/md/md.c | 16 ++++++++++++----
drivers/md/md.h | 1 +
drivers/md/raid1.c | 2 ++
drivers/md/raid10.c | 3 +++
drivers/md/raid5.c | 3 +++
5 files changed, 21 insertions(+), 4 deletions(-)
^ permalink raw reply
* [PATCH] md: make suspend range wait timed out
From: Shaohua Li @ 2017-06-09 19:41 UTC (permalink / raw)
To: linux-raid; +Cc: Shaohua Li, NeilBrown, Mikulas Patocka
From: Shaohua Li <shli@fb.com>
suspend range is controlled by userspace. If userspace doesn't clear suspend
range, it's possible a thread will wait for the range forever, we can't even
kill it. This is bad behavior. Add a timeout in the wait. If timeout happens,
we return IO error. The app controlling suspend range looks like part of disk
firmware, if disk isn't responded for a long time, timed out IO error is
returned.
A simple search in SCSI code shows maximum IO timeout is 120s, so I use this
value here too.
Cc: NeilBrown <neilb@suse.com>
Cc: Mikulas Patocka <mpatocka@redhat.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.h | 1 +
drivers/md/raid1.c | 12 +++++++++++-
drivers/md/raid5.c | 10 +++++++++-
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 63d342d560b8..11a0ec33e79b 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -29,6 +29,7 @@
#define MaxSector (~(sector_t)0)
+#define MD_SUSPEND_TIMEOUT (120 * HZ)
/*
* These flags should really be called "NO_RETRY" rather than
* "FAILFAST" because they don't make any promise about time lapse,
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index d9e5373444d2..bc6dee0259df 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1326,6 +1326,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
(mddev_is_clustered(mddev) &&
md_cluster_ops->area_resyncing(mddev, WRITE,
bio->bi_iter.bi_sector, bio_end_sector(bio)))) {
+ long remaining = -1;
/*
* As the suspend_* range is controlled by userspace, we want
@@ -1345,10 +1346,19 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
break;
sigfillset(&full);
sigprocmask(SIG_BLOCK, &full, &old);
- schedule();
+ remaining = schedule_timeout(MD_SUSPEND_TIMEOUT);
sigprocmask(SIG_SETMASK, &old, NULL);
+ if (remaining == 0)
+ break;
}
finish_wait(&conf->wait_barrier, &w);
+ if (remaining == 0) {
+ pr_err("md/raid1:%s: suspend range is locked\n",
+ mdname(mddev));
+ bio->bi_error = -ETIMEDOUT;
+ bio_endio(bio);
+ return;
+ }
}
wait_barrier(conf, bio->bi_iter.bi_sector);
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cf1ac2e0f4c8..24297f1530d1 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5685,6 +5685,7 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
if (rw == WRITE &&
logical_sector >= mddev->suspend_lo &&
logical_sector < mddev->suspend_hi) {
+ long remaining = -1;
raid5_release_stripe(sh);
/* As the suspend_* range is controlled by
* userspace, we want an interruptible
@@ -5697,10 +5698,17 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
sigset_t full, old;
sigfillset(&full);
sigprocmask(SIG_BLOCK, &full, &old);
- schedule();
+ remaining = schedule_timeout(
+ MD_SUSPEND_TIMEOUT);
sigprocmask(SIG_SETMASK, &old, NULL);
do_prepare = true;
}
+ if (remaining == 0) {
+ pr_err("md/raid5:%s: suspend range is locked\n",
+ mdname(mddev));
+ bi->bi_error = -ETIMEDOUT;
+ break;
+ }
goto retry;
}
--
2.11.0
^ permalink raw reply related
* Re: My md won't assemble even when disks are in clean state.
From: Wols Lists @ 2017-06-09 20:55 UTC (permalink / raw)
To: Linux Raid
In-Reply-To: <4353246d-a092-e342-6ef7-091d8e9d77c5@gmail.com>
On 09/06/17 06:44, Ram Ramesh wrote:
> Thanks. That did it. I was able to assemble. It assembled degraded and I
> --re-added the remaining drives and it accepted without any issue. There
> was no rebuild after --re-add which is consistent with the examine
> output above that all disks are clean. Did fsck a couple of times for a
> good measure and things seem normal now.
Just added a note to that effect to the "When things go wrogn" section
of the front page of the wiki. Arrays left partially assembled on boot
seem such a common problem that a one-liner there seems a fairly good idea.
Cheers,
Wol
^ permalink raw reply
* Re: My md won't assemble even when disks are in clean state.
From: Ram Ramesh @ 2017-06-09 23:38 UTC (permalink / raw)
To: Wols Lists, Linux Raid
In-Reply-To: <593B0B49.5000102@youngman.org.uk>
On 06/09/2017 03:55 PM, Wols Lists wrote:
> On 09/06/17 06:44, Ram Ramesh wrote:
>> Thanks. That did it. I was able to assemble. It assembled degraded and I
>> --re-added the remaining drives and it accepted without any issue. There
>> was no rebuild after --re-add which is consistent with the examine
>> output above that all disks are clean. Did fsck a couple of times for a
>> good measure and things seem normal now.
> Just added a note to that effect to the "When things go wrogn" section
> of the front page of the wiki. Arrays left partially assembled on boot
> seem such a common problem that a one-liner there seems a fairly good idea.
>
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Wol,
In fact, that is where I checked first to see if it is safe to stop
the array. Since it did not have or I missed it, I got worried. Most of
the trouble reported here seem to be created by user error in issuing
wrong commands. So, I waited and posted the lengthy details even when my
hunch was to stop the array and (re) assemble.
Thanks for adding the note. Google many times takes us to this page:
https://raid.wiki.kernel.org/index.php/Assemble_Run.
It may be good idea to repeat it there, if this is not too much of
redundancy.
Ramesh
^ permalink raw reply
* Re: My md won't assemble even when disks are in clean state.
From: Anthony Youngman @ 2017-06-10 0:07 UTC (permalink / raw)
To: Ram Ramesh, Linux Raid
In-Reply-To: <95dc6a23-167b-4dd8-2436-46428715e14f@gmail.com>
On 10/06/17 00:38, Ram Ramesh wrote:
> On 06/09/2017 03:55 PM, Wols Lists wrote:
>> On 09/06/17 06:44, Ram Ramesh wrote:
>>> Thanks. That did it. I was able to assemble. It assembled degraded and I
>>> --re-added the remaining drives and it accepted without any issue. There
>>> was no rebuild after --re-add which is consistent with the examine
>>> output above that all disks are clean. Did fsck a couple of times for a
>>> good measure and things seem normal now.
>> Just added a note to that effect to the "When things go wrogn" section
>> of the front page of the wiki. Arrays left partially assembled on boot
>> seem such a common problem that a one-liner there seems a fairly good
>> idea.
>>
> Wol,
>
> In fact, that is where I checked first to see if it is safe to stop
> the array. Since it did not have or I missed it, I got worried. Most of
> the trouble reported here seem to be created by user error in issuing
> wrong commands. So, I waited and posted the lengthy details even when my
> hunch was to stop the array and (re) assemble.
>
> Thanks for adding the note. Google many times takes us to this page:
> https://raid.wiki.kernel.org/index.php/Assemble_Run.
> It may be good idea to repeat it there, if this is not too much of
> redundancy.
>
Good idea. I've added a section at the start of that page, with a bit
more detail. I don't want to repeat stuff too often, but that's a good
place to put it. (Too much repetition, and people can't see the wood for
the trees :-)
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH v2 01/11] crypto: introduce crypto wait for async op
From: Herbert Xu @ 2017-06-10 3:43 UTC (permalink / raw)
To: Gilad Ben-Yossef
Cc: David S. Miller, Jonathan Corbet, David Howells, Alasdair Kergon,
Mike Snitzer, dm-devel, Shaohua Li, Steve French,
Theodore Y. Ts'o, Jaegeuk Kim, Mimi Zohar, Dmitry Kasatkin,
James Morris, Serge E. Hallyn, Eric Biggers, Ofir Drang,
Pavel Shilovsky, linux-crypto, linux-doc, linux-kernel, keyrings,
linux-raid, linux-cif
In-Reply-To: <1496046180-21962-2-git-send-email-gilad@benyossef.com>
On Mon, May 29, 2017 at 11:22:48AM +0300, Gilad Ben-Yossef wrote:
>
> +static inline int crypto_wait_req(int err, struct crypto_wait *wait)
> +{
> + switch (err) {
> + case -EINPROGRESS:
> + case -EBUSY:
> + wait_for_completion(&wait->completion);
> + reinit_completion(&wait->completion);
> + err = wait->err;
> + break;
> + };
> +
> + return err;
> +}
This assumes that the request is used with backlog. For non-backlog
requests this would result in a memory leak as EBUSY in that case is
a fatal error.
So this API can't be used without backlog.
We could introduce a flag to indicate whether we want backlog or not,
or maybe we should change our API so that in the non-backlog case we
return something other than EBUSY.
Opinions?
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v2 01/11] crypto: introduce crypto wait for async op
From: Gilad Ben-Yossef @ 2017-06-10 8:05 UTC (permalink / raw)
To: Herbert Xu
Cc: David S. Miller, Jonathan Corbet, David Howells, Alasdair Kergon,
Mike Snitzer, device-mapper development, Shaohua Li, Steve French,
Theodore Y. Ts'o, Jaegeuk Kim, Mimi Zohar, Dmitry Kasatkin,
James Morris, Serge E. Hallyn, Eric Biggers, Ofir Drang,
Pavel Shilovsky, linux-crypto, linux-doc,
Linux kernel mailing list
In-Reply-To: <20170610034336.GB14206@gondor.apana.org.au>
On Sat, Jun 10, 2017 at 6:43 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Mon, May 29, 2017 at 11:22:48AM +0300, Gilad Ben-Yossef wrote:
>>
>> +static inline int crypto_wait_req(int err, struct crypto_wait *wait)
>> +{
>> + switch (err) {
>> + case -EINPROGRESS:
>> + case -EBUSY:
>> + wait_for_completion(&wait->completion);
>> + reinit_completion(&wait->completion);
>> + err = wait->err;
>> + break;
>> + };
>> +
>> + return err;
>> +}
>
> This assumes that the request is used with backlog. For non-backlog
> requests this would result in a memory leak as EBUSY in that case is
> a fatal error.
>
> So this API can't be used without backlog.
You are right, of course. I did not take that into account.
>
> We could introduce a flag to indicate whether we want backlog or not,
> or maybe we should change our API so that in the non-backlog case we
> return something other than EBUSY.
>
> Opinions?
I guess there is a question if it really is important to know that
your request ended up
on the backlog, rather than being handled.I can imagine it can be used
as back pressure
indication but I wonder if someone is using that.
If not, maybe we can simplify things and use EINPROGRESS asindication
of a request
being accepted by the next layer (either being processed or queued in
the back log), whereas
EBUSY would indicate failure.
It does have a potential to make things simpler, I think.
Gilad
>
> Thanks,
> --
> Email: Herbert Xu <herbert@gondor.apana.org.au>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
Gilad Ben-Yossef
Chief Coffee Drinker
"If you take a class in large-scale robotics, can you end up in a
situation where the homework eats your dog?"
-- Jean-Baptiste Queru
^ permalink raw reply
* Re: [PATCH v2 01/11] crypto: introduce crypto wait for async op
From: Herbert Xu @ 2017-06-10 9:01 UTC (permalink / raw)
To: Gilad Ben-Yossef
Cc: Mike Snitzer, linux-doc, David Howells, device-mapper development,
keyrings, linux-ima-devel, Pavel Shilovsky, Alasdair Kergon,
linux-cifs, Jonathan Corbet, Mimi Zohar, Serge E. Hallyn,
linux-ima-user, Eric Biggers, linux-raid, linux-fscrypt,
James Morris, Jaegeuk Kim, Ofir Drang, Theodore Y. Ts'o,
Dmitry Kasatkin, samba-technical, Linux kernel mailing list,
Steve French
In-Reply-To: <CAOtvUMeRtcVcp8HE9oJAkJO35yF9XBsry3tdNQYj-e31DnDFQQ@mail.gmail.com>
On Sat, Jun 10, 2017 at 11:05:39AM +0300, Gilad Ben-Yossef wrote:
>
> I guess there is a question if it really is important to know that
> your request ended up
> on the backlog, rather than being handled.I can imagine it can be used
> as back pressure
> indication but I wonder if someone is using that.
Oh yes we do want it to return EBUSY if we put it on the backlog
because in that case we want the user to stop sending us new
requests.
It's the other case where we dropped the request and returned
EBUSY where I think we could return something other than EBUSY
and get rid of the ambiguity.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Paul Tonelli @ 2017-06-10 20:04 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170605235619.GA8478@metamorpher.de>
Hello Everybody,
tl dr: I found the --data-offset to allow a e2fsck, but things are not
as they should be. I still manage to get back data (pictures which are
>= 4Mb in size).
On 06/06/2017 01:56 AM, Andreas Klauer wrote:
> On Tue, Jun 06, 2017 at 01:24:41AM +0200, Paul Tonelli wrote:
>> mdadm --create /dev/md0 --level=5 --assume-clean --raid-devices=3 missing /dev/mapper/sdd /dev/mapper/sdb
> You did not specify the --data-offset here?
> Check mdadm --examine to make sure which offset it's using.
The objective was just to reverse the /dev/sdc destruction using the xor
on /dev/sdb and /dev/sdc, I believe the default offset is earlier than
the one I specify by hand.
>> xxd -u /dev/mapper/sdc | grep -C 3 'LABELONE'
>> >7f001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> >7f001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> >7f00200: 4C41 4245 4C4F 4E45 0100 0000 0000 0000 LABELONE........
> If correct this should appear at the start of /dev/md0 (even w/o adding sdc).
>
> LABELONE should appear on the first drive so this should not be wrong,
> however the sdd sdb could still have switched order, and of course
> the chunk size could be different (although unlikely according to your log).
Even after rebuilding the disks, I find the LABELONE at 7f00200. But it
does not provide a recoverable LVM / ext4 filesystem (the backup
superblock offset is incorrect for this partition size)
>> I believe you are right, the issue is still the raid: I have tried
>> photorec and most files I have opened look like they have been
>> truncated.
> Well, files could be fragmented, there's a certain sweet spot
> (like - megapixel JPEGs of few megs size) where it's sufficiently
> unlikely to be a problem.
>
> I don't know what files you have, if it's movies it would be okay
> too if the first few megs of the file were playable.
I have multiple assemblies with a custom data-offset in which a e2fsck
agrees to run from a backup superblock (I have brute forced the tests)
the possible assemblies are (offset, disk order):
261120 missing /dev/mapper/sdc /dev/mapper/sdd
261120 missing /dev/mapper/sdd /dev/mapper/sdc
261120 /dev/mapper/sdb missing /dev/mapper/sdd
261120 /dev/mapper/sdb /dev/mapper/sdd missing
I have already found bmp/png which are > 4MB by using photorec, so I
think I am going forward.
Right now, I am running e2fsck -n to see which order returns the least
errors, I will then try to get back as many files as I can, still using
overlay.
>> - apart from the raid superblock, the disks I use (sdd and sdb) have not
>> been erased (as sdc is rebuilt from xor)
> It only rebuilds starting from offset. So it should not have covered that
> offset if you did not specify it. Check it's not there before you --add.
> If it's there then this is after all not the drive you overwrite with dd?
I believe the offset we manually specify is after the default one on a
raid assembly with 3 disks, so it should have rebuilt the previous LVM
superblock, or am I missing something (again).
> I am confused now.
So am I, and I am afraid I have used all the time I could spend to get
this data back. Thanks to your help, I can still recover many files,
even if it is not a full filesystem :-). Thank you !
Paul
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Andreas Klauer @ 2017-06-10 20:41 UTC (permalink / raw)
To: Paul Tonelli; +Cc: linux-raid
In-Reply-To: <5f9ed8be-ec90-17bc-3e45-0721689a62a8@tonel.li>
On Sat, Jun 10, 2017 at 10:04:16PM +0200, Paul Tonelli wrote:
> I believe the offset we manually specify is after the default one on a
> raid assembly with 3 disks, so it should have rebuilt the previous LVM
> superblock, or am I missing something (again).
It should be the reverse. When you grow the raid, the offset shrinks.
Not able to provide further insights. You have a LVM2 header which,
given the correct offset, should appear on the /dev/md device and
in that case everything else should appear too.
If that is not the case then things will be a lot more complicated.
Regards
Andreas Klauer
^ permalink raw reply
* Re: [PATCH 0/3] mdadm/test: Refactor the codes of 'test' script
From: Zhilong Liu @ 2017-06-12 2:32 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <d92b0a65-198b-3637-83a1-a71b78cdfd43@gmail.com>
On 06/09/2017 11:08 PM, Jes Sorensen wrote:
> On 06/07/2017 05:31 AM, Zhilong Liu wrote:
>> Hi, Jes;
>> This patchset mainly focus on improving the 'test' script, rip it
>> apart and put it together in a more coherent form.
>> Over 3 months passed, the RFC patch of improving 'test' hadn't
>> received
>> any ideas, thus I post this sample version that I have tested many
>> times.
>> It's just the beginning to improve the test part of mdadm, any
>> questions
>> is very welcome.
>>
>> Sorry for that, I haven't added the function suggested from
>> Guoqing, cut
>> the words from the email tree of
>> "ANNOUNCE: mdadm 4.0 - A tool for managing md Soft RAID under Linux":
>>
>> Guoqing asked:
>> ...
>> Is current test can run the test for different raid level, say,
>> "./test --raidtype=raid1"
>> could execute all the *r1* tests, does it make sense to do it if we
>> don't
>> support it now.
>> ...
>>
>> Just list this suggestion here, waiting for ideas.
>>
>> RFC patch mail-tree:
>> [PATCH RFC] test: revise 'test' and make it easier to understand
>> Acked-by: Neil Brown
>> And suggest to split those changing into smaller patches from Guoqing.
>
> The whole test infrastructure is something I would like to see
> rewamped and modernized, so you starting to work on this is excellent.
>
> With regard to your question, then having an option to say 'run all
> raid1 tests' is probably good, we should have that for all raid
> levels, imsm, and ddf.
>
> I also would like to see a more flexible way to configure the tests,
> like it should be possible to specify a bunch of physical devices for
> the tests, instead of relying on loop devices.
I would send another sample version once complete your suggestions.
Thanks for your ideas.
Welcome various requirements during I'm doing this.
Thanks,
-Zhilong
>
> Cheers,
> Jes
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* [PATCH] md: change the initialization value for a spare device spot to MD_DISK_ROLE_SPARE
From: Lidong Zhong @ 2017-06-12 2:45 UTC (permalink / raw)
To: linux-raid
The value for spare spot of sb->dev_roles is changed from
MD_DISK_ROLE_FAULTY to MD_DISK_ROLE_SPARE to keep align
with the value when the superblock is firstly created in
userspace.
Signed-off-by: Lidong Zhong <lzhong@suse.com>
---
drivers/md/md.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 87edc342ccb3..a9aacedefbca 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -1852,7 +1852,7 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
max_dev = le32_to_cpu(sb->max_dev);
for (i=0; i<max_dev;i++)
- sb->dev_roles[i] = cpu_to_le16(MD_DISK_ROLE_FAULTY);
+ sb->dev_roles[i] = cpu_to_le16(MD_DISK_ROLE_SPARE);
if (test_bit(MD_HAS_JOURNAL, &mddev->flags))
sb->feature_map |= cpu_to_le32(MD_FEATURE_JOURNAL);
--
2.12.0
^ permalink raw reply related
* Re: internal write-intent bitmap is horribly slow with RAID10 over 20 drives
From: CoolCold @ 2017-06-12 6:12 UTC (permalink / raw)
To: NeilBrown; +Cc: Linux RAID
In-Reply-To: <87fufcdbak.fsf@notabene.neil.brown.name>
Hello!
i've started doing testing as proposed by you and found other strange
behavior with _4_ drives ~ 44 iops as well:
mdadm --create --assume-clean -c $((64*1)) -b internal
--bitmap-chunk=$((128*1024)) -n 4 -l 10 /dev/md1 /dev/sde /dev/sdf
/dev/sdg /dev/sdh
mdstat:
[root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid10 sdh[3] sdg[2] sdf[1] sde[0]
3516066176 blocks super 1.2 64K chunks 2 near-copies [4/4] [UUUU]
bitmap: 0/14 pages [0KB], 131072KB chunk
fio:
[root@spare-a17484327407661 tests]# fio --runtime 60 randwrite.conf
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
iodepth=512
fio-2.2.8
Starting 1 process
Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/179KB/0KB /s] [0/44/0 iops]
[eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=35048: Mon Jun 12 06:03:28 2017
write: io=35728KB, bw=609574B/s, iops=148, runt= 60018msec
slat (usec): min=6, max=3006.6K, avg=6714.32, stdev=33548.39
clat (usec): min=137, max=14323K, avg=3430245.54, stdev=4822029.01
lat (msec): min=22, max=14323, avg=3436.96, stdev=4830.87
clat percentiles (msec):
| 1.00th=[ 40], 5.00th=[ 76], 10.00th=[ 87], 20.00th=[ 115],
| 30.00th=[ 437], 40.00th=[ 510], 50.00th=[ 553], 60.00th=[ 619],
| 70.00th=[ 2376], 80.00th=[11600], 90.00th=[11731], 95.00th=[11863],
| 99.00th=[12387], 99.50th=[13435], 99.90th=[14091], 99.95th=[14222],
| 99.99th=[14353]
bw (KB /s): min= 111, max=14285, per=95.41%, avg=567.70, stdev=1623.95
lat (usec) : 250=0.01%
lat (msec) : 50=2.02%, 100=12.52%, 250=7.01%, 500=17.02%, 750=30.62%
lat (msec) : 1000=0.12%, 2000=0.50%, >=2000=30.18%
cpu : usr=0.06%, sys=0.34%, ctx=2607, majf=0, minf=30
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=0/w=8932/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=512
Run status group 0 (all jobs):
WRITE: io=35728KB, aggrb=595KB/s, minb=595KB/s, maxb=595KB/s,
mint=60018msec, maxt=60018msec
Disk stats (read/write):
md1: ios=61/8928, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=16/6693, aggrmerge=0/0, aggrticks=13/512251,
aggrin_queue=512265, aggrutil=83.63%
sde: ios=40/6787, merge=0/0, ticks=15/724812, in_queue=724824, util=83.63%
sdf: ios=2/6787, merge=0/0, ticks=5/694057, in_queue=694061, util=82.20%
sdg: ios=24/6599, merge=0/0, ticks=27/154988, in_queue=155022, util=80.72%
sdh: ios=1/6599, merge=0/0, ticks=6/475150, in_queue=475155, util=82.29%
comparing to the same drives on RAID5 fio shows ~ 142 iops:
[root@spare-a17484327407661 tests]# fio --runtime 60 randwrite.conf
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
iodepth=512
fio-2.2.8
Starting 1 process
Jobs: 1 (f=1): [w(1)] [0.0% done] [0KB/571KB/0KB /s] [0/142/0 iops]
[eta 93d:11h:20m:52s]
randwrite: (groupid=0, jobs=1): err= 0: pid=34914: Mon Jun 12 05:59:23 2017
write: io=41880KB, bw=707115B/s, iops=172, runt= 60648msec
raid5 created basically the same as for RAID10
mdadm --create --assume-clean -c $((64*1)) -b internal
--bitmap-chunk=$((128*1024)) -n 4 -l 5 /dev/md1 /dev/sde /dev/sdf
/dev/sdg /dev/sdh
mdstat output for raid5:
[root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid5 sdh[3] sdg[2] sdf[1] sde[0]
5274099264 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 7/7 pages [28KB], 131072KB chunk
for both cases, the same fio config:
[root@spare-a17484327407661 tests]# cat randwrite.conf
[randwrite]
blocksize=4k
#blocksize=64k
filename=/dev/md1
#filename=/dev/md2
readwrite=randwrite
#rwmixread=75
direct=1
buffered=0
ioengine=libaio
iodepth=512
#numjobs=4
group_reporting=1
from iostat, hard drives are having more requests than md1 (compare
40-43 on md1 and ~ 60 per device)
[root@spare-a17484327407661 rovchinnikov]# iostat -d -xk 1 /dev/md1
/dev/sde /dev/sdf /dev/sdg /dev/sdh
Linux 3.10.0-327.el7.x86_64 (spare-a17484327407661.sgdc)
06/12/2017 _x86_64_ (40 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sde 0.00 0.00 0.00 59.00 0.00 236.00
8.00 0.76 12.76 0.00 12.76 12.75 75.20
sdh 0.00 0.00 0.00 59.00 0.00 236.00
8.00 0.76 12.85 0.00 12.85 12.88 76.00
sdf 0.00 0.00 0.00 62.00 0.00 248.00
8.00 0.78 12.89 0.00 12.89 12.58 78.00
sdg 0.00 0.00 0.00 62.00 0.00 248.00
8.00 0.77 12.71 0.00 12.71 12.45 77.20
md1 0.00 0.00 0.00 40.00 0.00 160.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sde 0.00 0.00 0.00 66.00 0.00 264.00
8.00 0.80 12.39 0.00 12.39 12.09 79.80
sdh 0.00 0.00 0.00 62.00 0.00 248.00
8.00 0.78 12.87 0.00 12.87 12.58 78.00
sdf 0.00 0.00 0.00 66.00 0.00 264.00
8.00 0.78 11.82 0.00 11.82 11.82 78.00
sdg 0.00 0.00 0.00 62.00 0.00 248.00
8.00 0.80 12.82 0.00 12.82 12.85 79.70
md1 0.00 0.00 0.00 43.00 0.00 172.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sde 0.00 0.00 0.00 65.00 0.00 260.00
8.00 0.81 12.43 0.00 12.43 12.40 80.60
sdh 0.00 0.00 0.00 58.00 0.00 232.00
8.00 0.74 12.81 0.00 12.81 12.78 74.10
sdf 0.00 0.00 0.00 71.00 0.00 284.00
8.00 0.81 11.38 0.00 11.38 11.34 80.50
sdg 0.00 0.00 0.00 64.00 0.00 256.00
8.00 0.82 12.77 0.00 12.77 12.73 81.50
md1 0.00 0.00 0.00 43.00 0.00 172.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
I don't see any good explanation for this, kindly waiting for your advice.
On Wed, Jun 7, 2017 at 5:02 AM, NeilBrown <neilb@suse.com> wrote:
> On Tue, Jun 06 2017, CoolCold wrote:
>
>> Hello!
>> Neil, thanks for reply, further inline
>>
>> On Tue, Jun 6, 2017 at 10:40 AM, NeilBrown <neilb@suse.com> wrote:
>>> On Mon, Jun 05 2017, CoolCold wrote:
>>>
>>>> Hello!
>>>> Keep testing the new box and while having not the best sync speed,
>>>> it's not the worst thing I found.
>>>>
>>>> Doing FIO testing, for RAID10 over 20 10k RPM drives, I have very bad
>>>> performance, like _45_ iops only.
>>>
>>> ...
>>>>
>>>>
>>>> Output from fio with internal write-intent bitmap:
>>>> Jobs: 1 (f=1): [w(1)] [28.3% done] [0KB/183KB/0KB /s] [0/45/0 iops]
>>>> [eta 07m:11s]
>>>>
>>>> array definition:
>>>> [root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
>>>> Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
>>>> md1 : active raid10 sdx[19] sdw[18] sdv[17] sdu[16] sdt[15] sds[14]
>>>> sdr[13] sdq[12] sdp[11] sdo[10] sdn[9] sdm[8] sdl[7] sdk[6] sdj[5]
>>>> sdi[4] sdh[3] sdg[2] sdf[1] sde[0]
>>>> 17580330880 blocks super 1.2 64K chunks 2 near-copies [20/20]
>>>> [UUUUUUUUUUUUUUUUUUUU]
>>>> bitmap: 0/66 pages [0KB], 131072KB chunk
>>>>
>>>> Setting journal to be
>>>> 1) on SSD (separate drives), shows
>>>> Jobs: 1 (f=1): [w(1)] [5.0% done] [0KB/18783KB/0KB /s] [0/4695/0 iops]
>>>> [eta 09m:31s]
>>>> 2) to 'none' (disabling) shows
>>>> Jobs: 1 (f=1): [w(1)] [14.0% done] [0KB/18504KB/0KB /s] [0/4626/0
>>>> iops] [eta 08m:36s]
>>>
>>> These numbers suggest that the write intent bitmap causes a 100-fold slow
>>> down.
>>> i.e. 45 iops instead of 4500 iops (roughly).
>>>
>>> That is certainly more than I would expect, so maybe there is a bug.
>> I suppose noone is using raid10 over more than 4 drives then, i can't
>> believe i'm the one who hit this problem.
>
> We have customers who use RAID10 will many more than 4 drives, but I
> haven't had reports like this. Presumably whatever problem is affecting
> you is not affecting them. We cannot know until we drill down to
> understand the problem.
>
>>
>>>
>>> Large RAID10 is a worst-base for bitmap updates as the bitmap is written
>>> to all devices instead of just those devices that contain the data which
>>> the bit corresponds to. So every bitmap update goes to 10 device.
>>>
>>> Your bitmap chunk size of 128M is nice and large, but making it larger
>>> might help - maybe 1GB.
>> Tried that already, wasn't any much difference, but will gather more statistics.
>>
>>>
>>> Still 100-fold ... that's a lot..
>>>
>>> A potentially useful exercise would be to run a series of tests,
>>> changing the number of devices in the array from 2 to 10, changing the
>>> RAID chunk size from 64K to 64M, and changing the bitmap chunk size from
>>> 64M to 4G.
>> Changing chunk size to up to 64M just to gather statistics or you
>> suppose it may be some practical usage for this?
>
> I don't have any particular reason to expect this to have an effect.
> But it is easy to change, and changing it might show provide some hints.
> So probably "just to gather statistics".
>
> NeilBrown
>
>
>>> In each configuration, run the same test and record the iops.
>>> (You don't need to wait for a resync each time, just use
>>> --assume-clean).
>> This helps, thanks
>>> Then graph all this data (or just provide the table and I'll graph it).
>>> That might provide an insight into where to start looking for the
>>> slowdown.
>>>
>>> NeilBrown
>>
>>
>>
>> --
>> Best regards,
>> [COOLCOLD-RIPN]
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Best regards,
[COOLCOLD-RIPN]
^ permalink raw reply
* [PATCH] md/raid10: fix FailFast test for wrong device
From: Guoqing Jiang @ 2017-06-13 3:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
We need to test FailFast flag for replacement device here
since the set up for writing is for the replacement, so we
need fix it like:
- if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))
+ if (test_bit(FailFast, &conf->mirrors[d].replacement->flags))
Since commit f90145f317ef ("md/raid10: add rcu protection
to rdev access in raid10_sync_request.") had added the rcu
protection for the part, so let's extend the range protected
by rcu and use rdev directly.
Fixes: 1919cbb ("md/raid10: add failfast handling for writes.")
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/raid10.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 3273887..f25a2ae 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -3294,7 +3294,7 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
biolist = bio;
bio->bi_end_io = end_sync_read;
bio_set_op_attrs(bio, REQ_OP_READ, 0);
- if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))
+ if (test_bit(FailFast, &rdev->flags))
bio->bi_opf |= MD_FAILFAST;
bio->bi_iter.bi_sector = sector + rdev->data_offset;
bio->bi_bdev = rdev->bdev;
@@ -3306,7 +3306,6 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
continue;
}
atomic_inc(&rdev->nr_pending);
- rcu_read_unlock();
/* Need to set up for writing to the replacement */
bio = r10_bio->devs[i].repl_bio;
@@ -3317,11 +3316,12 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
biolist = bio;
bio->bi_end_io = end_sync_write;
bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
- if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))
+ if (test_bit(FailFast, &rdev->flags))
bio->bi_opf |= MD_FAILFAST;
bio->bi_iter.bi_sector = sector + rdev->data_offset;
bio->bi_bdev = rdev->bdev;
count++;
+ rcu_read_unlock();
}
if (count < 2) {
--
2.10.0
^ permalink raw reply related
* Re: [PATCH] md/raid10: fix FailFast test for wrong device
From: Shaohua Li @ 2017-06-13 17:11 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: shli, linux-raid
In-Reply-To: <20170613031608.25923-1-gqjiang@suse.com>
On Tue, Jun 13, 2017 at 11:16:08AM +0800, Guoqing Jiang wrote:
> We need to test FailFast flag for replacement device here
> since the set up for writing is for the replacement, so we
> need fix it like:
>
> - if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))
> + if (test_bit(FailFast, &conf->mirrors[d].replacement->flags))
good catch!
> Since commit f90145f317ef ("md/raid10: add rcu protection
> to rdev access in raid10_sync_request.") had added the rcu
> protection for the part, so let's extend the range protected
> by rcu and use rdev directly.
it's unnecessary, since we already increase the nr_pending for the rdev, but
this isn't harmful either.
Applied, thanks!
> Fixes: 1919cbb ("md/raid10: add failfast handling for writes.")
> Reviewed-by: NeilBrown <neilb@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> drivers/md/raid10.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 3273887..f25a2ae 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -3294,7 +3294,7 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
> biolist = bio;
> bio->bi_end_io = end_sync_read;
> bio_set_op_attrs(bio, REQ_OP_READ, 0);
> - if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))
> + if (test_bit(FailFast, &rdev->flags))
> bio->bi_opf |= MD_FAILFAST;
> bio->bi_iter.bi_sector = sector + rdev->data_offset;
> bio->bi_bdev = rdev->bdev;
> @@ -3306,7 +3306,6 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
> continue;
> }
> atomic_inc(&rdev->nr_pending);
> - rcu_read_unlock();
>
> /* Need to set up for writing to the replacement */
> bio = r10_bio->devs[i].repl_bio;
> @@ -3317,11 +3316,12 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
> biolist = bio;
> bio->bi_end_io = end_sync_write;
> bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
> - if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))
> + if (test_bit(FailFast, &rdev->flags))
> bio->bi_opf |= MD_FAILFAST;
> bio->bi_iter.bi_sector = sector + rdev->data_offset;
> bio->bi_bdev = rdev->bdev;
> count++;
> + rcu_read_unlock();
> }
>
> if (count < 2) {
> --
> 2.10.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] md: change the initialization value for a spare device spot to MD_DISK_ROLE_SPARE
From: Shaohua Li @ 2017-06-13 17:17 UTC (permalink / raw)
To: Lidong Zhong, jes.sorensen; +Cc: linux-raid
In-Reply-To: <20170612024555.1120-1-lzhong@suse.com>
On Mon, Jun 12, 2017 at 10:45:55AM +0800, Lidong Zhong wrote:
> The value for spare spot of sb->dev_roles is changed from
> MD_DISK_ROLE_FAULTY to MD_DISK_ROLE_SPARE to keep align
> with the value when the superblock is firstly created in
> userspace.
Could you point out where mdadm does this?
I'm wondering what's the impact here. If there is a disk, we will set correct
role at the end of super_1_sync(), otherwise faulty sounds more reasonable.
Jes,
Could you help check this?
Thanks,
Shaohua
> Signed-off-by: Lidong Zhong <lzhong@suse.com>
> ---
> drivers/md/md.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 87edc342ccb3..a9aacedefbca 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -1852,7 +1852,7 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
> max_dev = le32_to_cpu(sb->max_dev);
>
> for (i=0; i<max_dev;i++)
> - sb->dev_roles[i] = cpu_to_le16(MD_DISK_ROLE_FAULTY);
> + sb->dev_roles[i] = cpu_to_le16(MD_DISK_ROLE_SPARE);
>
> if (test_bit(MD_HAS_JOURNAL, &mddev->flags))
> sb->feature_map |= cpu_to_le32(MD_FEATURE_JOURNAL);
> --
> 2.12.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: internal write-intent bitmap is horribly slow with RAID10 over 20 drives
From: NeilBrown @ 2017-06-14 1:40 UTC (permalink / raw)
To: CoolCold; +Cc: Linux RAID
In-Reply-To: <CAGqmV7p+Ww0Yand0wpZqFmAwwTeghDB7N3+qh8Jwe2gqpnJoGg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 7994 bytes --]
On Mon, Jun 12 2017, CoolCold wrote:
> Hello!
> i've started doing testing as proposed by you and found other strange
> behavior with _4_ drives ~ 44 iops as well:
> mdadm --create --assume-clean -c $((64*1)) -b internal
> --bitmap-chunk=$((128*1024)) -n 4 -l 10 /dev/md1 /dev/sde /dev/sdf
> /dev/sdg /dev/sdh
>
> mdstat:
> [root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
> Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
> md1 : active raid10 sdh[3] sdg[2] sdf[1] sde[0]
> 3516066176 blocks super 1.2 64K chunks 2 near-copies [4/4] [UUUU]
> bitmap: 0/14 pages [0KB], 131072KB chunk
>
>
> fio:
> [root@spare-a17484327407661 tests]# fio --runtime 60 randwrite.conf
> randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
> iodepth=512
> fio-2.2.8
> Starting 1 process
> Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/179KB/0KB /s] [0/44/0 iops]
> [eta 00m:00s]
> randwrite: (groupid=0, jobs=1): err= 0: pid=35048: Mon Jun 12 06:03:28 2017
> write: io=35728KB, bw=609574B/s, iops=148, runt= 60018msec
> slat (usec): min=6, max=3006.6K, avg=6714.32, stdev=33548.39
> clat (usec): min=137, max=14323K, avg=3430245.54, stdev=4822029.01
> lat (msec): min=22, max=14323, avg=3436.96, stdev=4830.87
> clat percentiles (msec):
> | 1.00th=[ 40], 5.00th=[ 76], 10.00th=[ 87], 20.00th=[ 115],
> | 30.00th=[ 437], 40.00th=[ 510], 50.00th=[ 553], 60.00th=[ 619],
> | 70.00th=[ 2376], 80.00th=[11600], 90.00th=[11731], 95.00th=[11863],
> | 99.00th=[12387], 99.50th=[13435], 99.90th=[14091], 99.95th=[14222],
> | 99.99th=[14353]
> bw (KB /s): min= 111, max=14285, per=95.41%, avg=567.70, stdev=1623.95
> lat (usec) : 250=0.01%
> lat (msec) : 50=2.02%, 100=12.52%, 250=7.01%, 500=17.02%, 750=30.62%
> lat (msec) : 1000=0.12%, 2000=0.50%, >=2000=30.18%
> cpu : usr=0.06%, sys=0.34%, ctx=2607, majf=0, minf=30
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
> issued : total=r=0/w=8932/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=512
>
> Run status group 0 (all jobs):
> WRITE: io=35728KB, aggrb=595KB/s, minb=595KB/s, maxb=595KB/s,
> mint=60018msec, maxt=60018msec
>
> Disk stats (read/write):
> md1: ios=61/8928, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
> aggrios=16/6693, aggrmerge=0/0, aggrticks=13/512251,
> aggrin_queue=512265, aggrutil=83.63%
> sde: ios=40/6787, merge=0/0, ticks=15/724812, in_queue=724824, util=83.63%
> sdf: ios=2/6787, merge=0/0, ticks=5/694057, in_queue=694061, util=82.20%
> sdg: ios=24/6599, merge=0/0, ticks=27/154988, in_queue=155022, util=80.72%
> sdh: ios=1/6599, merge=0/0, ticks=6/475150, in_queue=475155, util=82.29%
>
>
>
>
>
> comparing to the same drives on RAID5 fio shows ~ 142 iops:
> [root@spare-a17484327407661 tests]# fio --runtime 60 randwrite.conf
> randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
> iodepth=512
> fio-2.2.8
> Starting 1 process
> Jobs: 1 (f=1): [w(1)] [0.0% done] [0KB/571KB/0KB /s] [0/142/0 iops]
> [eta 93d:11h:20m:52s]
> randwrite: (groupid=0, jobs=1): err= 0: pid=34914: Mon Jun 12 05:59:23 2017
> write: io=41880KB, bw=707115B/s, iops=172, runt= 60648msec
>
> raid5 created basically the same as for RAID10
> mdadm --create --assume-clean -c $((64*1)) -b internal
> --bitmap-chunk=$((128*1024)) -n 4 -l 5 /dev/md1 /dev/sde /dev/sdf
> /dev/sdg /dev/sdh
>
> mdstat output for raid5:
> [root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
> Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
> md1 : active raid5 sdh[3] sdg[2] sdf[1] sde[0]
> 5274099264 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
> bitmap: 7/7 pages [28KB], 131072KB chunk
>
>
> for both cases, the same fio config:
> [root@spare-a17484327407661 tests]# cat randwrite.conf
> [randwrite]
> blocksize=4k
> #blocksize=64k
> filename=/dev/md1
> #filename=/dev/md2
> readwrite=randwrite
> #rwmixread=75
> direct=1
> buffered=0
> ioengine=libaio
> iodepth=512
> #numjobs=4
> group_reporting=1
>
> from iostat, hard drives are having more requests than md1 (compare
> 40-43 on md1 and ~ 60 per device)
> [root@spare-a17484327407661 rovchinnikov]# iostat -d -xk 1 /dev/md1
> /dev/sde /dev/sdf /dev/sdg /dev/sdh
> Linux 3.10.0-327.el7.x86_64 (spare-a17484327407661.sgdc)
> 06/12/2017 _x86_64_ (40 CPU)
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> avgrq-sz avgqu-sz await r_await w_await svctm %util
> sde 0.00 0.00 0.00 59.00 0.00 236.00
> 8.00 0.76 12.76 0.00 12.76 12.75 75.20
> sdh 0.00 0.00 0.00 59.00 0.00 236.00
> 8.00 0.76 12.85 0.00 12.85 12.88 76.00
> sdf 0.00 0.00 0.00 62.00 0.00 248.00
> 8.00 0.78 12.89 0.00 12.89 12.58 78.00
> sdg 0.00 0.00 0.00 62.00 0.00 248.00
> 8.00 0.77 12.71 0.00 12.71 12.45 77.20
> md1 0.00 0.00 0.00 40.00 0.00 160.00
> 8.00 0.00 0.00 0.00 0.00 0.00 0.00
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> avgrq-sz avgqu-sz await r_await w_await svctm %util
> sde 0.00 0.00 0.00 66.00 0.00 264.00
> 8.00 0.80 12.39 0.00 12.39 12.09 79.80
> sdh 0.00 0.00 0.00 62.00 0.00 248.00
> 8.00 0.78 12.87 0.00 12.87 12.58 78.00
> sdf 0.00 0.00 0.00 66.00 0.00 264.00
> 8.00 0.78 11.82 0.00 11.82 11.82 78.00
> sdg 0.00 0.00 0.00 62.00 0.00 248.00
> 8.00 0.80 12.82 0.00 12.82 12.85 79.70
> md1 0.00 0.00 0.00 43.00 0.00 172.00
> 8.00 0.00 0.00 0.00 0.00 0.00 0.00
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> avgrq-sz avgqu-sz await r_await w_await svctm %util
> sde 0.00 0.00 0.00 65.00 0.00 260.00
> 8.00 0.81 12.43 0.00 12.43 12.40 80.60
> sdh 0.00 0.00 0.00 58.00 0.00 232.00
> 8.00 0.74 12.81 0.00 12.81 12.78 74.10
> sdf 0.00 0.00 0.00 71.00 0.00 284.00
> 8.00 0.81 11.38 0.00 11.38 11.34 80.50
> sdg 0.00 0.00 0.00 64.00 0.00 256.00
> 8.00 0.82 12.77 0.00 12.77 12.73 81.50
> md1 0.00 0.00 0.00 43.00 0.00 172.00
> 8.00 0.00 0.00 0.00 0.00 0.00 0.00
>
> I don't see any good explanation for this, kindly waiting for your advice.
>
I really did want to see the multi-dimensional collection of data
points, rather than just one. It is hard to see patterns in a single
number.
RAID5 and RAID10 are not directly comparable.
For every block written to the array, RAID10 writes 2 block, and RAID5
writes 1.33 (on average). So you would expect 50% more writes to blocks
in a (4 device) RAID10.
Also each bit in the bitmap for RAID10 covers less space, so you get
more bitmap updates.
I don't think this quite covers the difference though.
40 writes to the array, 60 writes to each device is a little high.
I think that is worst case.
Every write to the array updates the bitmap on all devices, and the data
on 2 devices.
So it seems like every write is being handled synchronously with no
write combining. Normally multiple bitmap updates are handled with a
single write.
Having only one job doing direct IO would quite possibly cause this
worst-case performance (though I don't know details about how fio
works).
Try using buffered io (that easily allows more parallelism), and try
multiple concurrent threads.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: [PATCH] md: change the initialization value for a spare device spot to MD_DISK_ROLE_SPARE
From: Lidong Zhong @ 2017-06-14 3:16 UTC (permalink / raw)
To: Shaohua Li, jes.sorensen; +Cc: linux-raid
In-Reply-To: <20170613171708.roims6ja6vtuwwlo@kernel.org>
On 06/14/2017 01:17 AM, Shaohua Li wrote:
> On Mon, Jun 12, 2017 at 10:45:55AM +0800, Lidong Zhong wrote:
>> The value for spare spot of sb->dev_roles is changed from
>> MD_DISK_ROLE_FAULTY to MD_DISK_ROLE_SPARE to keep align
>> with the value when the superblock is firstly created in
>> userspace.
>
> Could you point out where mdadm does this?
>
Hi Shaohua,
It's done in init_super1() during initialization.
1615memset(sb->dev_roles,0xff,
1616 MAX_SB_SIZE - sizeof(struct mdp_superblock_1));
> I'm wondering what's the impact here. If there is a disk, we will set correct
> role at the end of super_1_sync(), otherwise faulty sounds more reasonable.
>
Actually there is no impact if it works fine. I found the value for
spare slot in sb->dev_roles is changed while debugging a problem.
That's the reason I submit this patch.
Thanks,
Lidong
> Jes,
> Could you help check this?
>
> Thanks,
> Shaohua
>
>> Signed-off-by: Lidong Zhong <lzhong@suse.com>
>> ---
>> drivers/md/md.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 87edc342ccb3..a9aacedefbca 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -1852,7 +1852,7 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
>> max_dev = le32_to_cpu(sb->max_dev);
>>
>> for (i=0; i<max_dev;i++)
>> - sb->dev_roles[i] = cpu_to_le16(MD_DISK_ROLE_FAULTY);
>> + sb->dev_roles[i] = cpu_to_le16(MD_DISK_ROLE_SPARE);
>>
>> if (test_bit(MD_HAS_JOURNAL, &mddev->flags))
>> sb->feature_map |= cpu_to_le32(MD_FEATURE_JOURNAL);
>> --
>> 2.12.0
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* [*Working with you |18.1*]
From: VS - .41 @ 2017-06-14 4:03 UTC (permalink / raw)
To: mspway
[*Hello and good day.
I am looking to work with a reputable individual/firm to engage in a profit oriented ventures in your country and perhaps with your assistance, we could get low tax rates.
I have the directive of Mr. Mikhail Khodorkovsky to source for partner abroad who can accommodate and manage 150M & 350M USD respectively. The sums are derived from an executed project with Yukos Oil Company before the company was merged with Rosneft Oil Corporation in Russia.
We shall apply for the necessary paper work required to re-profile your name as the receipient and also ensure payment is carried out by Rosneft into a bank account in your name. I guaranty we would execute this business under a legitimate arrangement without breach of the law.
Further details will follow upon your positive reply.
Regards,
Vladimir Stankov*].
--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
^ permalink raw reply
* [PATCH] block: missing break in process_queued_bios()
From: Dan Carpenter @ 2017-06-14 9:04 UTC (permalink / raw)
To: Alasdair Kergon, Christoph Hellwig
Cc: linux-raid, kernel-janitors, dm-devel, Shaohua Li, Mike Snitzer
This used to be a fall through case, but we shifted code around and I
think we want a break here now.
Fixes: 4e4cbee93d56 ("block: switch bios to blk_status_t")
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index a7d2e0840cc5..0e8ab5bb3575 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -625,6 +625,7 @@ static void process_queued_bios(struct work_struct *work)
case DM_MAPIO_KILL:
bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
+ break;
case DM_MAPIO_REQUEUE:
bio->bi_status = BLK_STS_DM_REQUEUE;
bio_endio(bio);
^ permalink raw reply related
* Re: [PATCH] block: missing break in process_queued_bios()
From: Christoph Hellwig @ 2017-06-14 9:07 UTC (permalink / raw)
To: Dan Carpenter
Cc: Mike Snitzer, kernel-janitors, linux-raid, dm-devel, Shaohua Li,
Christoph Hellwig, Alasdair Kergon
In-Reply-To: <20170614090437.GA31017@elgon.mountain>
On Wed, Jun 14, 2017 at 12:04:37PM +0300, Dan Carpenter wrote:
> This used to be a fall through case, but we shifted code around and I
> think we want a break here now.
>
> Fixes: 4e4cbee93d56 ("block: switch bios to blk_status_t")
> Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Thanks Dan, this looks coreect:
Acked-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* [PATCH v2 0/3] Refactor and revamp the 'test' script
From: Zhilong Liu @ 2017-06-14 13:02 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, Zhilong Liu
Hi, Jes;
Mainly added the suggestions from v1. I'm awared that these patches
maybe still to improve, draft them just from my understanding. I would
improve for any defect, and thanks for your patience to point out.
Still need ideas about testdev():
- Now the 'test' has supported new arguments like --disks= to test physical
devices, but testdev() in all cases was designed by loop mode, we should
decide whether or not the 'disk' mode also supports the testdev() testing.
Currently, I have ignored the testdev() testing and make it 'return 0' if
has set --dev=disk mode in this patch version.
For --raidtype:
- I have devided the raid levels to
raid0|linear|raid1|raid456|raid10|ddf|imsm
For --disks:
- It supports user to specify physical devices as follow.
Example: ./test --dev=disk --disks=/dev/sda{5..16} or
./test --dev=disk --disks=/dev/sd{a..m}
Changes from v1:
- Refactor and improve all functions of test 'script'.
- Add --raidtype argument to support testing different raid
level cases.
- Add --disks argument to support testing physical devices
v1 mail-tree:
[PATCH 0/3] mdadm/test: Refactor the codes of 'test' script
RFC patch mail-tree:
[PATCH RFC] test: revise 'test' and make it easier to understand
Thanks very much,
-Zhilong
Zhilong Liu (3):
mdadm/test: Refactor and revamp 'test' script
mdadm/test: Add '--raidtype=' to run different raidlevel cases
mdadm/test: Add '--disks=' to support testing phsical devices
test | 464 +++++++++++++++++++++++++++++++++++++++++--------------------------
1 file changed, 285 insertions(+), 179 deletions(-)
--
2.6.6
^ permalink raw reply
* [PATCH v2 1/3] mdadm/test: Refactor and revamp 'test' script
From: Zhilong Liu @ 2017-06-14 13:02 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, Zhilong Liu
In-Reply-To: <1497445373-27985-1-git-send-email-zlliu@suse.com>
Adding functions:
die()
uniform the abnormal situations that have to abort.
check_env()
do various basic checking before running test suite.
save_log()
collect array infos, include of dmesg, superblock,
bitmap and /proc/mdstat.
main()
the core function of this script.
Improve functions:
cleanup()
clear dmesg and remove the /vat/tmp/mdtest* files.
mdadm()
clear superblock once creating or building arrays
every time, because it's always creating arrays
many times in a test case.
check()
just tidy up with die(), didn't change code meanings.
testdev()
add checking $1 must be a block device, add 'return 0'
in final because this function exists in last line of
test case, such as tests/05r6tor0.
do_test()
add checking abnormal dmesg and changing log management.
do_help()
just recommend a better way to print Usage.
parse_args()
revamp and improve.
Delete function:
fast_sync()
It's no longer used, so get rid of it.
Signed-off-by: Zhilong Liu <zlliu@suse.com>
---
test | 408 +++++++++++++++++++++++++++++++++++++------------------------------
1 file changed, 228 insertions(+), 180 deletions(-)
diff --git a/test b/test
index 7942d6e..7ee523b 100755
--- a/test
+++ b/test
@@ -1,37 +1,20 @@
#!/bin/bash
#
# run test suite for mdadm
-user=`id -un`
-if [ " $user" != " root" ]
-then
- echo >&2 "test: testing can only be done as 'root'."
- exit 1
-fi
-
-prefix='[0-9][0-9]'
-
-dir=`pwd`
+dir=$(pwd)
mdadm=$dir/mdadm
-if [ \! -x $mdadm ]
-then
- echo >&2 "test: $mdadm isn't usable."
-fi
-
testdir="tests"
-logdir="$testdir/logs"
-logsave=0
-exitonerror=1
+targetdir="/var/tmp"
+logdir="$targetdir"
+config=/tmp/mdadm.conf
-echo "Testing on linux-$(uname -r) kernel"
+savelogs=0
+exitonerror=1
+prefix='[0-9][0-9]'
-# Check whether to run multipath tests
-modprobe multipath 2> /dev/null
-if grep -s 'Personalities : .*multipath' > /dev/null /proc/mdstat
-then
- MULTIPATH="yes"
-fi
-INTEGRITY=yes
+# use loop devices by default if doesn't specify --dev
DEVTYPE=loop
+INTEGRITY=yes
LVM_VOLGROUP=mdtest
# make sure to test local mdmon, not system one
@@ -46,7 +29,6 @@ mdp1=/dev/md_d1
# We test mdadm on loop-back block devices.
# dir for storing files should be settable by command line maybe
-targetdir=/var/tmp
size=20000
# super0, round down to multiple of 64 and substract 64
mdsize0=19904
@@ -68,20 +50,65 @@ mdsize12=19988
# ddf needs bigger devices as 32Meg is reserved!
ddfsize=65536
-config=/tmp/mdadm.conf
+# $1 is optional parameter, it shows why to save log
+save_log() {
+ status=$1
+ logfile="$status""$_basename".log
+
+ cat $targetdir/stderr >> $targetdir/log
+ cp $targetdir/log $logdir/$_basename.log
+ echo "## $HOSTNAME: saving dmesg." >> $logdir/$logfile
+ dmesg -c >> $logdir/$logfile
+ $mdadm -As 2> /dev/null
+ echo "## $HOSTNAME: saving proc mdstat." >> $logdir/$logfile
+ cat /proc/mdstat >> $logdir/$logfile
+ array=($(mdadm -Ds | cut -d' ' -f2))
+ echo "## $HOSTNAME: mdadm -D ${array[@]}" >> $logdir/$logfile
+ $mdadm -D ${array[@]} >> $logdir/$logfile
+ [ "$1" == "fail" ] &&
+ echo "FAILED - see $logdir/$_basename.log and $logdir/$logfile for details"
+ # ignore saving external(external file, imsm...) bitmap
+ cat /proc/mdstat | grep -q "linear\|external" && return 0
+ if [ $DEVTYPE == 'lvm' ]
+ then
+ # not supported lvm type yet
+ echo
+ elif [ $DEVTYPE == 'loop' ]
+ then
+ if [ ! -z ${array[@]} -a ${#array[@]} -ge 1 ]
+ then
+ md_disks=($($mdadm -D -Y ${array[@]} | grep "/dev/$DEVTYPE" | cut -d'=' -f2))
+ cat /proc/mdstat | grep -q "bitmap"
+ if [ $? -eq 0 ]
+ then
+ echo "## $HOSTNAME: mdadm -X ${md_disks[@]}" >> $logdir/$logfile
+ $mdadm -X ${md_disks[@]} >> $logdir/$logfile
+ fi
+ else
+ echo "## $HOSTNAME: no array assembled!" >> $logdir/$logfile
+ fi
+ fi
+}
+
+die() {
+ echo -e "\n\tERROR: $* \n"
+ save_log fail
+ exit 2
+}
cleanup() {
udevadm settle
$mdadm -Ssq 2> /dev/null
case $DEVTYPE in
- loop)
+ loop )
for d in 0 1 2 3 4 5 6 7 8 9 10 11 12 13
do
- losetup -d /dev/loop$d # rm -f $targetdir/mdtest$d
+ losetup -d /dev/loop$d
rm -f /dev/disk/by-path/loop*
+ rm -f /var/tmp/mdtest$d
done
;;
- lvm)
+ lvm )
for d in 0 1 2 3 4 5 6 7 8 9 10 11 12 13
do
eval "lvremove --quiet -f \$dev$d"
@@ -98,23 +125,26 @@ do_setup() {
trap cleanup 0 1 3 15
trap ctrl_c 2
- # make sure there are no loop devices remaining.
- # udev started things can sometimes prevent them being stopped
- # immediately
- while grep loop /proc/partitions > /dev/null 2>&1
- do
- mdadm -Ss
- losetup -d /dev/loop[0-9]* 2> /dev/null
- sleep 1
- done
+ [ -d $logdir ] || mkdir -p $logdir
+ dmesg -c > /dev/null
+
+ if [ "$DEVTYPE" == "loop" ]
+ then
+ # make sure there are no loop devices remaining.
+ # udev started things can sometimes prevent them being stopped
+ # immediately
+ while grep loop /proc/partitions > /dev/null 2>&1
+ do
+ $mdadm -Ssq
+ losetup -d /dev/loop[0-9]* 2> /dev/null
+ sleep 0.2
+ done
+ fi
devlist=
for d in 0 1 2 3 4 5 6 7 8 9 10 11 12 13
do
sz=$size
- if [ $d -gt 7 ]
- then
- sz=$ddfsize
- fi
+ [ $d -gt 7 ] && sz=$ddfsize
case $DEVTYPE in
loop)
[ -f $targetdir/mdtest$d ] ||
@@ -169,7 +199,17 @@ mdadm() {
;;
esac
case $* in
- *-C* )
+ *-C* | *--create* | *-B* | *--build* )
+ # clear superblock every time once creating or
+ # building arrays, because it's always creating
+ # and building array many times in a test case.
+ for args in $*
+ do
+ [[ $args =~ "/dev/" ]] && {
+ [[ $args =~ "md" ]] ||
+ $mdadm --zero $args > /dev/null
+ }
+ done
$mdadm 2> $targetdir/stderr --quiet "$@" --auto=yes
;;
* )
@@ -191,39 +231,28 @@ mdadm() {
check() {
case $1 in
spares )
- spares=`tr '] ' '\012\012' < /proc/mdstat | grep -c '(S)' || exit 0`
- if [ $spares -ne $2 ]
- then
- echo >&2 "ERROR expected $2 spares, found $spares"
- exit 1
- fi
+ spares=$(tr '] ' '\012\012' < /proc/mdstat | grep -c '(S)' || exit 0)
+ [ $spares -ne $2 ] &&
+ die "expected $2 spares, found $spares"
;;
raid* | linear )
- grep -s "active $1 " /proc/mdstat > /dev/null || {
- echo >&2 "ERROR active $1 not found"
- cat /proc/mdstat
- exit 1
- }
+ grep -sq "active $1 " /proc/mdstat ||
+ die "active $1 not found"
;;
algorithm )
- grep -s " algorithm $2 " /proc/mdstat > /dev/null || {
- echo >&2 "ERROR algorithm $2 not found"
- cat /proc/mdstat
- exit 1
- }
+ grep -sq " algorithm $2 " /proc/mdstat ||
+ die "algorithm $2 not found"
;;
resync | recovery | reshape )
cnt=5
- while ! grep -s $1 /proc/mdstat > /dev/null
+ while ! grep -sq $1 /proc/mdstat
do
if [ $cnt -gt 0 ] && grep -v idle /sys/block/md*/md/sync_action > /dev/null
then # Something isn't idle - wait a bit
sleep 0.5
cnt=$[cnt-1]
else
- echo >&2 ERROR no $1 happening
- cat /proc/mdstat
- exit 1
+ die "no $1 happening"
fi
done
;;
@@ -234,22 +263,18 @@ check() {
# to do can still take a little longer than expected.
# add an extra check: is sync_completed shows the end is reached, assume
# there is no recovery.
- if grep -s -E '(resync|recovery|reshape) *=' > /dev/null /proc/mdstat
+ if grep -sq -E '(resync|recovery|reshape) *=' /proc/mdstat
then
incomplete=`grep / /sys/block/md*/md/sync_completed 2> /dev/null | sed '/^ *\([0-9]*\) \/ \1/d'`
- if [ -n "$incomplete" ]
- then
- echo >&2 "ERROR resync or recovery is happening!"
- cat /proc/mdstat
- exit 1
- fi
+ [ -n "$incomplete" ] &&
+ die "resync or recovery is happening!"
fi
;;
wait )
p=`cat /proc/sys/dev/raid/speed_limit_max`
echo 2000000 > /proc/sys/dev/raid/speed_limit_max
sleep 0.1
- while grep -E '(resync|recovery|reshape|check|repair) *=' > /dev/null /proc/mdstat ||
+ while grep -Eq '(resync|recovery|reshape|check|repair) *=' /proc/mdstat ||
grep -v idle > /dev/null /sys/block/md*/md/sync_action
do
sleep 0.5
@@ -257,45 +282,28 @@ check() {
echo $p > /proc/sys/dev/raid/speed_limit_max
;;
state )
- grep -s "blocks.*\[$2\]\$" /proc/mdstat > /dev/null || {
- echo >&2 "ERROR state $2 not found!"
- cat /proc/mdstat
- exit 1
- }
+ grep -sq "blocks.*\[$2\]\$" /proc/mdstat ||
+ die "state $2 not found!"
sleep 0.5
;;
bitmap )
- grep -s bitmap > /dev/null /proc/mdstat || {
- echo >&2 ERROR no bitmap
- cat /proc/mdstat
- exit 1
- }
+ grep -sq bitmap /proc/mdstat ||
+ die "no bitmap"
;;
nobitmap )
- if grep -s "bitmap" > /dev/null /proc/mdstat
- then
- echo >&2 ERROR bitmap present
- cat /proc/mdstat
- exit 1
- fi
+ grep -sq "bitmap" /proc/mdstat &&
+ die "bitmap present"
;;
readonly )
- grep -s "read-only" > /dev/null /proc/mdstat || {
- echo >&2 "ERROR array is not read-only!"
- cat /proc/mdstat
- exit 1
- }
+ grep -sq "read-only" /proc/mdstat ||
+ die "array is not read-only!"
;;
inactive )
- grep -s "inactive" > /dev/null /proc/mdstat || {
- echo >&2 "ERROR array is not inactive!"
- cat /proc/mdstat
- exit 1
- }
+ grep -sq "inactive" /proc/mdstat ||
+ die "array is not inactive!"
;;
* )
- echo >&2 ERROR unknown check $1
- exit 1
+ die "unknown check $1"
;;
esac
}
@@ -311,6 +319,7 @@ no_errors() {
# basic device test
testdev() {
+ [ -b $1 ] || die "$1 isn't a block device."
udevadm settle
dev=$1
cnt=$2
@@ -329,20 +338,11 @@ testdev() {
rasize=$[rasize/DEV_ROUND_K/2]
rasize=$[rasize*DEV_ROUND_K*2]
fi
- if [ `/sbin/blockdev --getsize $dev` -eq 0 ]
- then
- sleep 2
- fi
+ [ `/sbin/blockdev --getsize $dev` -eq 0 ] && sleep 2
_sz=`/sbin/blockdev --getsize $dev`
- if [ $rasize -lt $_sz -o $[rasize*4/5] -gt $_sz ]
- then
- echo "ERROR: size is wrong for $dev: $cnt * $dvsize (chunk=$chunk) = $rasize, not $_sz"
- exit 1
- fi
-}
-
-fast_sync() {
- echo 200000 > /proc/sys/dev/raid/speed_limit_max
+ [ $rasize -lt $_sz -o $[rasize*4/5] -gt $_sz ] &&
+ die "size is wrong for $dev: $cnt * $dvsize (chunk=$chunk) = $rasize, not $_sz"
+ return 0
}
rotest() {
@@ -359,7 +359,6 @@ do_test() {
# stop all arrays, just incase some script left an array active.
$mdadm -Ssq 2> /dev/null
mdadm --zero $devlist 2> /dev/null
- mdadm --zero $devlist 2> /dev/null
# this might have been reset: restore the default.
echo 2000 > /proc/sys/dev/raid/speed_limit_max
# source script in a subshell, so it has access to our
@@ -367,52 +366,44 @@ do_test() {
echo -ne "$_script... "
if ( set -ex ; . $_script ) &> $targetdir/log
then
+ dmesg | grep -iq "error\|call trace\|segfault" &&
+ die "dmesg prints errors when testing $_basename!"
echo "succeeded"
_fail=0
else
- log=log
- cat $targetdir/stderr >> $targetdir/log
- echo "=======================dmesg=================" >> $targetdir/log
- dmesg | tail -n 200 >> $targetdir/log
- if [ $exitonerror == 0 ]; then
- log=log-`basename $_script`
- mv $targetdir/log $logdir/$log
- fi
- echo "FAILED - see $logdir/$log for details"
+ save_log fail
_fail=1
fi
- if [ "$savelogs" == "1" ]
- then
- cp $targetdir/log $logdir/$_basename.log
- fi
- if [ "$_fail" == "1" -a "$exitonerror" == "1" ]
- then
- exit 1
- fi
+ [ "$savelogs" == "1" ] &&
+ mv -f $targetdir/log $logdir/$_basename.log
+ [ "$_fail" == "1" -a "$exitonerror" == "1" ] && exit 1
fi
}
do_help() {
- echo "Usage: $0 [options]"
- echo " Options:"
- echo " --tests=<test1,test2,..> Comma separated list of tests to run"
- echo " --disable-multipath Disable any tests involving multipath"
- echo " --disable-integrity Disable slow tests of RAID[56] consistency"
- echo " --logdir=<directory> Directory to save logfiles in"
- echo " --save-logs Save all logs in <logdir>"
- echo " --keep-going Don't stop on error, ie. run all tests"
- echo " --dev=[loop|lvm|ram] Use loop devices (default), LVM, or RAM disk"
- echo " --volgroup=<name> LVM volume group for LVM test"
- echo " setup Setup test environment and exit"
- echo " cleanup Cleanup test environment"
- echo " <prefix> Run tests with <prefix>"
+ cat <<-EOF
+ Usage: $0 [options]
+ Options:
+ --tests=test1,test2,... Comma separated list of tests to run
+ --disable-multipath Disable any tests involving multipath
+ --disable-integrity Disable slow tests of RAID[56] consistency
+ --logdir=directory Directory to save all logfiles in
+ --save-logs Usually use with --logdir together
+ --keep-going | --no-error Don't stop on error, ie. run all tests
+ --dev=loop|lvm|ram Use loop devices (default), LVM, or RAM disk
+ --volgroup=name LVM volume group for LVM test
+ setup Setup test environment and exit
+ cleanup Cleanup test environment
+ prefix Run tests with <prefix>
+ --help | -h Print this usage
+ EOF
}
parse_args() {
for i in $*
do
case $i in
- [0-9]* )
+ [0-9][0-9] )
prefix=$i
;;
setup )
@@ -426,10 +417,10 @@ parse_args() {
exit 0
;;
--tests=* )
- TESTLIST=`expr "x$i" : 'x[^=]*=\(.*\)' | sed -e 's/,/ /g'`
+ TESTLIST=($(echo ${i##*=} | sed -e 's/,/ /g'))
;;
--logdir=* )
- logdir=`expr "x$i" : 'x[^=]*=\(.*\)'`
+ logdir="${i##*=}"
;;
--save-logs )
savelogs=1
@@ -443,52 +434,109 @@ parse_args() {
--disable-integrity )
unset INTEGRITY
;;
- --dev=loop )
- DEVTYPE=loop
- ;;
- --dev=lvm )
- DEVTYPE=lvm
- ;;
- --dev=ram )
- DEVTYPE=ram
+ --dev=* )
+ case ${i##*=} in
+ loop )
+ DEVTYPE=loop
+ ;;
+ lvm )
+ DEVTYPE=lvm
+ ;;
+ ram )
+ DEVTYPE=ram
+ ;;
+ * )
+ echo "Unknown argument: $i"
+ do_help
+ exit 1
+ ;;
+ esac
;;
--volgroup=* )
LVM_VOLGROUP=`expr "x$i" : 'x[^=]*=\(.*\)'`
;;
- --help )
+ --help | -h )
do_help
exit 0
;;
- -* )
+ * )
echo " $0: Unknown argument: $i"
do_help
- exit 0
+ exit 1
;;
esac
done
}
-logdir=$targetdir
-parse_args $@
+check_env() {
+ user=$(id -un)
+ [ "X$user" != "Xroot" ] && {
+ echo "test: testing can only be done as 'root'."
+ exit 1
+ }
+ [ -x "raid6check" -a -x $mdadm ] || {
+ echo "test: please run 'make everything' before perform testing."
+ exit 1
+ }
+ cmds=(mdadm lsblk df udevadm losetup mkfs.ext3 fsck)
+ for cmd in ${cmds[@]}
+ do
+ which $cmd > /dev/null || {
+ echo "$cmd command not found!"
+ exit 1
+ }
+ done
+ mdadm_src_ver="$($mdadm -V 2>&1)"
+ mdadm_sbin_ver="$($(which mdadm) -V 2>&1)"
+ if [ "$mdadm_src_ver" != "$mdadm_sbin_ver" ]
+ then
+ # it's nessesary to 'make install' mdadm to /SBIN/DIR,
+ # such as systemd/mdadm-grow-continue@.service, would
+ # run as an instance by systemd when reshape happens,
+ # thus ensure that the correct mdadm is in testing.
+ echo "test: please run 'make install' before testing."
+ exit 1
+ fi
+ if ! $(df -T . | grep -iq ext)
+ then
+ # 'external file' bitmap only supports with ext[2-4] file system
+ echo "test: please run test suite with ext[2-4] file system."
+ exit 1
+ fi
+ if $(lsblk -a | grep -iq raid)
+ then
+ # donot run mdadm -Ss directly if there are RAIDs working.
+ echo "test: please run test suite without running RAIDs environment."
+ exit 1
+ fi
+ # Check whether to run multipath tests
+ modprobe multipath 2> /dev/null
+ grep -sq 'Personalities : .*multipath' /proc/mdstat &&
+ MULTIPATH="yes"
+}
-do_setup
-mkdir -p $logdir
+main() {
+ check_env
+ do_setup
-if [ "$savelogs" == "1" ]
-then
- echo "Saving logs to $logdir"
-fi
+ echo "Testing on linux-$(uname -r) kernel"
+ [ "$savelogs" == "1" ] &&
+ echo "Saving logs to $logdir"
+ if [ "x$TESTLIST" != "x" ]
+ then
+ for script in ${TESTLIST[@]}
+ do
+ do_test $testdir/$script
+ done
+ else
+ for script in $testdir/$prefix $testdir/$prefix*[^~]
+ do
+ do_test $script
+ done
+ fi
-if [ "x$TESTLIST" != "x" ]
-then
- for script in $TESTLIST
- do
- do_test $testdir/$script
- done
-else
- for script in $testdir/$prefix $testdir/$prefix*[^~]
- do
- do_test $script
- done
-fi
-exit 0
+ exit 0
+}
+
+parse_args $@
+main
--
2.6.6
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox