* Re: not enough operational mirrors
From: NeilBrown @ 2014-09-22 5:47 UTC (permalink / raw)
To: Ian Young; +Cc: linux-raid
In-Reply-To: <CANs+QMzWWqh-5MemOrbguaq7fNzJPDaYUH66K0rgGrnaXY7FNA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]
On Sun, 21 Sep 2014 22:32:19 -0700 Ian Young <ian@duffrecords.com> wrote:
> My 6-drive software RAID 10 array failed. The individual drives
> failed one at a time over the past few months but it's been an
> extremely busy summer and I didn't have the free time to RMA the
> drives and rebuild the array. Now I'm wishing I had acted sooner
> because three of the drives are marked as removed and the array
> doesn't have enough mirrors to start. I followed the recovery
> instructions at raid.wiki.kernel.org and, before making things any
> worse, saved the status using mdadm --examine and consulted this
> mailing list. Here's the status:
>
> http://pastebin.com/KkV8e8Gq
>
> I can see that the event counts on sdd2 and sdf2 are significantly far
> behind, so we can consider that data too old. sdc2 is only behind by
> two events, so any data loss there should be minimal. If I can make
> the array start with sd[abce]2 I think that will be enough to mount
> the filesystem, back up my data, and start replacing drives. How do I
> do that?
Use the "--force" option with "--assemble".
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* not enough operational mirrors
From: Ian Young @ 2014-09-22 5:32 UTC (permalink / raw)
To: linux-raid
My 6-drive software RAID 10 array failed. The individual drives
failed one at a time over the past few months but it's been an
extremely busy summer and I didn't have the free time to RMA the
drives and rebuild the array. Now I'm wishing I had acted sooner
because three of the drives are marked as removed and the array
doesn't have enough mirrors to start. I followed the recovery
instructions at raid.wiki.kernel.org and, before making things any
worse, saved the status using mdadm --examine and consulted this
mailing list. Here's the status:
http://pastebin.com/KkV8e8Gq
I can see that the event counts on sdd2 and sdf2 are significantly far
behind, so we can consider that data too old. sdc2 is only behind by
two events, so any data loss there should be minimal. If I can make
the array start with sd[abce]2 I think that will be enough to mount
the filesystem, back up my data, and start replacing drives. How do I
do that?
^ permalink raw reply
* Re: What happens with array when external write-intent bitmap fails?
From: NeilBrown @ 2014-09-22 3:37 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: linux-raid
In-Reply-To: <CAA91j0X2WSyF_QuZdoWWNgDCZV7TKJbhQmq_vOrygqzkg3yK8g@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 992 bytes --]
On Tue, 16 Sep 2014 17:42:13 +0400 Andrei Borzenkov <arvidjaar@gmail.com>
wrote:
> Theoretically write-intent is optional, so if it fails it would be
> possible to simply discard it and continue. But I have seen statement
> that if external write-intent bitmap file is not available at time of
> assembly, starting of array fails.
>
> Thank you for clarification.
What exactly did you want clarified?
I suggest the best thing to do is to use "losetup" to create some loop-back
devices. Then use mdadm to create an array using those devices.
Then simulate the failure mode that you are concerned about and see what
happens.
If you find that the documentation does not agree with reality, please
consider posting details.
Thanks,
NeilBrown
>
> -andrei
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: read error recovery threshold
From: NeilBrown @ 2014-09-22 3:35 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <CAK=nhWO75WOJNCOXG+_ynug3ta6939b-8fyz0R1D-ENXF7jEGg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]
On Mon, 15 Sep 2014 10:56:11 -0600 Eric Mei <meijia@gmail.com> wrote:
> Hi,
>
> After a read error detected, RAID6 will initiate a recovery procedure
> try to correct it, until the number of read error exceeds a threshold,
> which is "conf->max_nr_stripes" (see raid5_end_read_request()), I'm
> wondering the reasoning behind this. To me the threshold seems a drive
> property, but max_nr_stripes is a array-wide cache setting and can be
> changed at runtime. In our specific case, we observed a drive emitting
> lots of read errors without being marked as faulty because the larger
> max_nr_stripes
> setting.
>
> Look at other part of MD code, there is "mddev::max_corr_read_errors"
> which is set to 20, but only RAID10 makes use of it. Also the comment
> above MD_DEFAULT_MAX_CORRECTED_READ_ERRORS says "...We divide the read
> error count by 2 for every hour elapsed between read errors", but I
> don't see any code matching this description.
>
> Any thoughts? Thanks
Yes, it is inconsistent.
It wasn't designed to be inconsistent, it just happened.
Patch with good justification will be looked on kindly.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Please advise, strange "not enough to start the array while not clean"
From: NeilBrown @ 2014-09-22 3:19 UTC (permalink / raw)
To: Patrik Horník; +Cc: linux-raid
In-Reply-To: <CAAOsTSkaFdum4LD8aOkOsG-f45fTjRSUPb=EqguJi4yaNjcSdQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3292 bytes --]
On Mon, 22 Sep 2014 04:11:20 +0200 Patrik Horník <patrik@dsl.sk> wrote:
> Hello Neil,
>
> I've got this situation unfamiliar to me on RAID6 array md1 with important data.
>
> - It is RAID6 with 6 devices, 5 are partitions and 1 is another RAID0
> array md101 from two smaller drives. One of the smaller drives froze,
> so md101 got kicked out from md1 and marked as faulty in md1. After
> while I've stopped md1 without removing md101 from it first. Then I
> rebooted and assembled md101.
>
> - First I tried mdadm -A --no-degraded -u UUID /dev/md1 but got
> "mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started."
> so I stopped the md1.
>
> - Second time I started it with -v and got:
>
> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
> mdadm: added /dev/sdi1 to /dev/md1 as 1
> mdadm: added /dev/sdh1 to /dev/md1 as 2
> mdadm: added /dev/sde1 to /dev/md1 as 3
> mdadm: added /dev/sdk1 to /dev/md1 as 4
> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
> mdadm: added /dev/sdg1 to /dev/md1 as 0
> mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started.
>
> - On third time I tried without --nodegraded with mdadm -A -v -u UUID
> /dev/md1. This is what I've got:
>
> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
> mdadm: added /dev/sdi1 to /dev/md1 as 1
> mdadm: added /dev/sdh1 to /dev/md1 as 2
> mdadm: added /dev/sde1 to /dev/md1 as 3
> mdadm: added /dev/sdk1 to /dev/md1 as 4
> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
> mdadm: added /dev/sdg1 to /dev/md1 as 0
> mdadm: /dev/md1 assembled from 5 drives - not enough to start the
> array while not clean - consider --force.
>
> Array md1 has bitmap. All drive devices have all same Events, their
> state is clean and Device Role is Active device. md101 has active
> state and lower Events.
>
> Is this expected behavior? My theory is that it is caused by md101 and
> I should start array md1 without it (by for example stopping md101)
> and then re-add it. Is that a case or is it something else?
>
> Thanks.
>
> Best regards,
>
> Patrik
The array is clearly degraded as one of the devices failed and hasn't been
recovered yet, so using --nodegraded is counter productive, as you
discovered.
It appears that the array is also marked as 'dirty'. That suggests that it
wasn't shut down cleanly.
What does "mdadm --examine" of some device show?
You probably need to re-assemble the array with --force like it suggests,
then add the failed device and let it recover.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Please advise, strange "not enough to start the array while not clean"
From: Patrik Horník @ 2014-09-22 2:11 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
Hello Neil,
I've got this situation unfamiliar to me on RAID6 array md1 with important data.
- It is RAID6 with 6 devices, 5 are partitions and 1 is another RAID0
array md101 from two smaller drives. One of the smaller drives froze,
so md101 got kicked out from md1 and marked as faulty in md1. After
while I've stopped md1 without removing md101 from it first. Then I
rebooted and assembled md101.
- First I tried mdadm -A --no-degraded -u UUID /dev/md1 but got
"mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started."
so I stopped the md1.
- Second time I started it with -v and got:
mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
mdadm: added /dev/sdi1 to /dev/md1 as 1
mdadm: added /dev/sdh1 to /dev/md1 as 2
mdadm: added /dev/sde1 to /dev/md1 as 3
mdadm: added /dev/sdk1 to /dev/md1 as 4
mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
mdadm: added /dev/sdg1 to /dev/md1 as 0
mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started.
- On third time I tried without --nodegraded with mdadm -A -v -u UUID
/dev/md1. This is what I've got:
mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
mdadm: added /dev/sdi1 to /dev/md1 as 1
mdadm: added /dev/sdh1 to /dev/md1 as 2
mdadm: added /dev/sde1 to /dev/md1 as 3
mdadm: added /dev/sdk1 to /dev/md1 as 4
mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
mdadm: added /dev/sdg1 to /dev/md1 as 0
mdadm: /dev/md1 assembled from 5 drives - not enough to start the
array while not clean - consider --force.
Array md1 has bitmap. All drive devices have all same Events, their
state is clean and Device Role is Active device. md101 has active
state and lower Events.
Is this expected behavior? My theory is that it is caused by md101 and
I should start array md1 without it (by for example stopping md101)
and then re-add it. Is that a case or is it something else?
Thanks.
Best regards,
Patrik
^ permalink raw reply
* HOW ARE YOU?
From: Benjamin Siaka @ 2014-09-22 1:20 UTC (permalink / raw)
To: linux-raid
Hello my Dear,
I will greatly appreciate my correspondence meets you in good health condition.
My name is Mr. Benjamin Siaka. I am seeking for your co-operation for investment partnership in your Country. I shall provide the FUND for the investment. When you acknowledged the receipt of this correspondence, thereafter I will give you the Full Details of my investment proposal.
I await your response in earliest.
My regards,
Mr. Benjamin Siaka.
^ permalink raw reply
* Re: Raid 1 vs Raid 10 single thread performance
From: keld @ 2014-09-18 13:19 UTC (permalink / raw)
To: Bostjan Skufca; +Cc: David Brown, linux-raid
In-Reply-To: <CAEp_DRDBQQmBHe7uYdOWWnUD084RtTrnbZe3jUrG3b6c6w=ivQ@mail.gmail.com>
Hi Bostjan
The raid.wiki.kernel.org is not mine, but it is the official wiki for this email list
and kernel group. I am one of the more active people on the wiki:
Some of the benchmarks ar provided by me, but most are provided by
others. The source of the benchmark is reported in every case.
I wrote many years ago when "far" layout was originally implemented that F2
should be the default raid10 layout, as I think it has the best
overall performance, but that has not happened (yet!).
There are some shortcomings, tho, as F2 is the only raid10 layout
that is not possible to grow. This could be solved by implementing it.
Also a better allocation of the disk partitions is not fully implemented,
which gives better redundancy. The fully supported
implementation of "far" layout gives the redundancy of raid 0+1,
while the partly implemented "far" layout (implemented partly in the kernel)
gives raid 1+0 redundancy.
Best regards
keld
On Tue, Sep 16, 2014 at 05:19:59PM +0200, Bostjan Skufca wrote:
> I expected "optimized" result, but not by that much. Positively surprised.
>
> Looking over at results shown on the wiki (yours, I presume), my results
> for n2 could be even higher. Yours are within 2% range (for sequential
> writes), mine 10%.
>
> Do you think f2 should be made default for 2-device RAID 10 arrays?
>
> b.
>
> PS: Judging by the results it would benefit almost everyone (trade 2-10%
> write penalty for 100% read throughput increase). But this is just my
> personal opinion. Heck, the best would be to replace raid1 with 10
> altogether, so users would not be surprised by this unexpected
> single-client RAID 1 non-performance.
>
> PPS: BTW it seems you guys did a great job here, like David stated in his
> last response ("way ahead":).
>
> PPPS: David: enthusiasm came from finally being enough p...ed off about why
> can't linux raid 1 behave like a raid 1 should, even for single client. And
> that 1(0)Gbps connection is not saturated when it could/should be! :)
>
>
> On 16 September 2014 12:19, <keld@keldix.com> wrote:
>
> > On Tue, Sep 16, 2014 at 09:48:28AM +0200, Bostjan Skufca wrote:
> > > David and Neil, thanks for hints!
> > >
> > > (I was busy with other things lately, but believe it or not I got the
> > > "why not try raid 10 with only 2 partitions" idea just last night,
> > > tested it a couple of minutes ago with fascination, and now here I am
> > > reading your emails - please do not remind me again of time wasted :)
> > >
> > > The write performance is curious though:
> > > - f2: 147 MB/s
> > > - n2: 162 MB/s
> > > I was expecting greater difference (bu I must admit this was not
> > > tested on the whole 3TB disk, just 400GB partition on it).
> >
> >
> > This is as expected, and also as reported in other benchmarks.
> >
> > Many expect that writing is considerably slower in F2 than n2,
> > because the blocks are distributed much more apart in f2 than in n2,
> > but the elevator algorithm for IO sceduling collects writing blocks
> > in the cache and does almost equalize the time used for about all mirrored
> > raid types.
> >
> > See also https://raid.wiki.kernel.org/index.php/Performance
> > for more benchmarks.
> >
> > Best regards
> > Keld
> >
> > > b.
> > >
> > >
> > > On 12 September 2014 10:49, David Brown <david.brown@hesbynett.no>
> > wrote:
> > > > On 10/09/14 23:24, Bostjan Skufca wrote:
> > > >> Hi,
> > > >>
> > > >> I have a simple question:
> > > >> - Where is the code that is used for actual RAID 10 creation? In
> > > >> kernel or in mdadm?
> > > >>
> > > >>
> > > >> Explanation:
> > > >>
> > > >> I was dissatisfied with single-threaded RAID 1 sequential read
> > > >> performance (basically boils down to the speed of one disk). I figured
> > > >> that instead of using level 1 I could create RAID level 10 and use two
> > > >> equally-sized partitions on each drive (instead of one).
> > > >>
> > > >> It turns out that if array is created properly, it is capable of
> > > >> sequential reads at almost 2x single device speed, as expected (on
> > > >> SSD!) and what would anyone expect from ordinary RAID 1.
> > > >>
> > > >> What does "properly" actually mean?
> > > >> I was doing some benchmarks with various raid configurations and
> > > >> figured out that the order of devices submitted to creation command is
> > > >> significant. It also makes raid10 created in such mode reliable or
> > > >> unreliable to a device failure (not partition failure, device failure,
> > > >> which means that two raid underlying devices fail at once).
> > > >>
> > > >> Sum:
> > > >> - if such array is created properly, it has redundancy in place and
> > > >> performs as expected
> > > >> - if not, it performs as raid1 and fails with one physical disk
> > failure
> > > >>
> > > >> I am trying to find the code responsible for creation of RAID 10 in
> > > >> order to try and make it more inteligent about where to place RAID 10
> > > >> parts if it gets a list of devices to use, and some of those devices
> > > >> are on the same physical disks.
> > > >>
> > > >> Thanks for hints,
> > > >> b.
> > > >>
> > > >>
> > > >>
> > > >> PS: More details about testing is available here, but be warned, it is
> > > >> still a bit hectic to read:
> > > >>
> > http://blog.a2o.si/2014/09/07/linux-software-raid-why-you-should-always-use-raid-10-instead-of-raid-1/
> > > >
> > > >
> > > > Hi,
> > > >
> > > > First let me applaud your enthusiasm for trying to inform people about
> > > > raid in your blog, your interest in investigating different ideas in
> > the
> > > > hope of making md raid faster and/or easier and/or safer.
> > > >
> > > > Then let me tell you your entire blog post is wasted, because md
> > already
> > > > has a solution that is faster, easier and safer than anything you have
> > > > come up with so far.
> > > >
> > > > You are absolutely correct about the single-threaded read performance
> > of
> > > > raid1 pairs - for a number of reasons, a single thread read will get
> > > > reads from only one disk. This is not a problem in many cases, because
> > > > you often have multiple simultaneous reads on "typical" systems with
> > > > raid1. But for some cases, such as a high performance desktop, it can
> > > > be a limitation.
> > > >
> > > > You are also correct that the solution is basically to split the drives
> > > > into two parts, pair up halves from each disk as raid1 mirrors, and
> > > > stripe the two mirrors as raid0.
> > > >
> > > > And you are correct that you have to get the sets right, or you will
> > may
> > > > lose redundancy and/or speed.
> > > >
> > > > Fortunately, Neil and the other md raid developers are way ahead of
> > you.
> > > >
> > > > Neil gave you the pointers in one of his replies, but I suspect you did
> > > > not understand that Linux raid10 is not limited to the arrangement of
> > > > traditional raid10, and thus did not see his point.
> > > >
> > > > md raid and mdadmin already support a very flexible form of raid10.
> > > > Unlike traditional raid10 that requires a multiple of 4 disks, Linux
> > > > raid10 can work with /any/ number of disks greater than 1. There are
> > > > various layouts that can be used for this - the Wikipedia entry gives
> > > > some useful diagrams:
> > > >
> > > > <
> > http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10>
> > > >
> > > > You can also read about it in the mdadm manual page, and various
> > > > documents and resources around the web.
> > > >
> > > >
> > > > In your particular case, what you want is to use "--layout raid10,f2"
> > on
> > > > your two disks. This asks md to split each disk (or the partitions you
> > > > use) into two parts, without creating any new partitions. The first
> > > > half of disk 1 is mirrored with the second half of disk 2, and vice
> > > > versa, then these mirrors are striped. This is very similar to the
> > > > layout you are trying to achieve, except for four points:
> > > >
> > > > The mirrors are crossed-over, so that a first half is mirrored with a
> > > > second half. This makes no difference on an SSD, but makes a huge
> > > > difference on a hard disk.
> > > >
> > > > mdadm and md raid get the ordering right every time - there is no need
> > > > to worry about the ordering of the two disks.
> > > >
> > > > You don't have to have extra partitions, automatic detection works, and
> > > > the layout has one less layer, meaning less complexity and lower
> > latency
> > > > and overheads.
> > > >
> > > > md raid knows more about the layout, and can use it to optimise the
> > speed.
> > > >
> > > >
> > > > In particular, md will (almost) always read from the outer halves of
> > the
> > > > disks. On a hard disk, this can be twice the speed of the inner
> > layers.
> > > >
> > > > Obviously you pay a penalty in writing when you have such an
> > arrangement
> > > > - writes need to go to both disks, and involve significant head
> > > > movement. There are other raid10 layouts that have lower streamed read
> > > > speeds but also lower write latencies (choose the balance you want).
> > > >
> > > >
> > > > With this in mind, I hope you can try out raid10,f2 layout on your
> > > > system and then change your blog to show how easy this all is with md
> > > > raid, how practical it is for a fast workstation or desktop, and how
> > > > much faster such a setup is than anything that can be achieved with
> > > > hardware raid cards or anything other than md raid.
> > > >
> > > > mvh.,
> > > >
> > > > David
> > > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
^ permalink raw reply
* Re: frequent disk activity with mdadm-3.3
From: Marco Schindler @ 2014-09-18 11:22 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20140918210942.113acb17@notabene.brown>
On 18.09.2014, at 13:09, NeilBrown <neilb@suse.de> wrote:
> Thanks. There is nothing there which point to the device being spun down.
> I tried spinning down disks on a couple of machines and no udev events were
> created. So I'm a bit suspicious that there is something I'm missing.
>
> How exactly do you spin down the devices?
> I use "hdparm -Y /dev/sda" or "hdparm -S 1 /dev/sda".
I’m using spindown (https://code.google.com/p/spindown) which in turn uses sg3-utils (http://sg.danny.cz/sg/sg3_utils.html), issuing "sg_start —stop DEVICE"
I can confirm there’s no change events with hdparm -Y here either. but with sg_start —stop there are.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: frequent disk activity with mdadm-3.3
From: NeilBrown @ 2014-09-18 11:09 UTC (permalink / raw)
To: Marco Schindler; +Cc: linux-raid
In-Reply-To: <D28F9C01-C75C-4B69-932F-FA2425DEDDB4@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6946 bytes --]
On Thu, 18 Sep 2014 12:38:55 +0200 Marco Schindler
<marco.schindler@gmail.com> wrote:
>
> On 18.09.2014, at 12:03, NeilBrown <neilb@suse.de> wrote:
>
> > Getting a "change" even on spindown is causing the problem I suspect.
> > A change in 3.3.1 causes "mdadm -I" to be run on a device when it 'changes'.
> > That will read from the device which will wake it up.
> > (commit 25392f5fc59f96fb76 - revert it and the symptom will probably go away).
> >
> > I really think the "bug" here is that the change event is emitted on
> > 'spindown', but maybe the bug is that the exact meaning of 'change' isn't
> > well documented.
> >
> > I can probably get "mdadm -I" to use O_EXCL which will fail on devices
> > already in an array, but I'm not sure that is a complete solution. You could
> > still get wakeups on other devices.
> >
> > Can you rung the 'udevadm monitor' again, but this time with '--property'.
> > Maybe there is some property associated with spindown events which we can use
> > to ignore them.
> >
> > NeilBrown
>
> sure. I also took separate logs for standby and wakeup.
>
> change events only occur when the drive goes standby (see below).
> strangely enough, there are no events when the drive wakes up.
>
> monitor will print the received events for:
> UDEV - the event which udev sends out after rule processing
> KERNEL - the kernel uevent
...
Thanks. There is nothing there which point to the device being spun down.
I tried spinning down disks on a couple of machines and no udev events were
created. So I'm a bit suspicious that there is something I'm missing.
How exactly do you spin down the devices?
I use "hdparm -Y /dev/sda" or "hdparm -S 1 /dev/sda".
NeilBrown
>
> KERNEL[558130.868218] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda (block)
> ACTION=change
> DEVNAME=/dev/sda
> DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda
> DEVTYPE=disk
> MAJOR=8
> MINOR=0
> SEQNUM=3239
> SUBSYSTEM=block
>
> KERNEL[558132.028485] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
> ACTION=change
> DEVNAME=/dev/sdb
> DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb
> DEVTYPE=disk
> MAJOR=8
> MINOR=16
> SEQNUM=3240
> SUBSYSTEM=block
>
> UDEV [558139.263973] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda (block)
> ACTION=change
> DEVLINKS=/dev/disk/by-id/ata-WDC_WD20EARS-00S8B1_WD-WCAVY1872131 /dev/disk/by-id/wwn-0x50014ee203dca984 /dev/disk/by-path/pci-0000:09:00.0-sas-0x4433221103000000-lun-0
> DEVNAME=/dev/sda
> DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda
> DEVTYPE=disk
> ID_ATA=1
> ID_ATA_DOWNLOAD_MICROCODE=1
> ID_ATA_FEATURE_SET_AAM=1
> ID_ATA_FEATURE_SET_AAM_CURRENT_VALUE=254
> ID_ATA_FEATURE_SET_AAM_ENABLED=0
> ID_ATA_FEATURE_SET_AAM_VENDOR_RECOMMENDED_VALUE=128
> ID_ATA_FEATURE_SET_HPA=1
> ID_ATA_FEATURE_SET_HPA_ENABLED=1
> ID_ATA_FEATURE_SET_PM=1
> ID_ATA_FEATURE_SET_PM_ENABLED=1
> ID_ATA_FEATURE_SET_PUIS=1
> ID_ATA_FEATURE_SET_PUIS_ENABLED=0
> ID_ATA_FEATURE_SET_SECURITY=1
> ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
> ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=408
> ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=408
> ID_ATA_FEATURE_SET_SMART=1
> ID_ATA_FEATURE_SET_SMART_ENABLED=1
> ID_ATA_SATA=1
> ID_ATA_SATA_SIGNAL_RATE_GEN1=1
> ID_ATA_SATA_SIGNAL_RATE_GEN2=1
> ID_ATA_WRITE_CACHE=1
> ID_ATA_WRITE_CACHE_ENABLED=1
> ID_BUS=ata
> ID_FS_LABEL=alina.o81.5:media3
> ID_FS_LABEL_ENC=alina.o81.5:media3
> ID_FS_TYPE=linux_raid_member
> ID_FS_USAGE=raid
> ID_FS_UUID=058bd7b0-455c-0d7e-6de0-a845ea05ee38
> ID_FS_UUID_ENC=058bd7b0-455c-0d7e-6de0-a845ea05ee38
> ID_FS_UUID_SUB=da1c1cad-6661-174c-b0b9-4bc1a712902d
> ID_FS_UUID_SUB_ENC=da1c1cad-6661-174c-b0b9-4bc1a712902d
> ID_FS_VERSION=1.2
> ID_MODEL=WDC_WD20EARS-00S8B1
> ID_MODEL_ENC=WDC\x20WD20EARS-00S8B1\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
> ID_PATH=pci-0000:09:00.0-sas-0x4433221103000000-lun-0
> ID_PATH_TAG=pci-0000_09_00_0-sas-0x4433221103000000-lun-0
> ID_REVISION=80.00A80
> ID_SERIAL=WDC_WD20EARS-00S8B1_WD-WCAVY1872131
> ID_SERIAL_SHORT=WD-WCAVY1872131
> ID_TYPE=disk
> ID_WWN=0x50014ee203dca984
> ID_WWN_WITH_EXTENSION=0x50014ee203dca984
> MAJOR=8
> MINOR=0
> SEQNUM=3239
> SUBSYSTEM=block
> USEC_INITIALIZED=2003
>
> UDEV [558139.293742] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
> ACTION=change
> DEVLINKS=/dev/disk/by-id/ata-WDC_WD20EARS-00S8B1_WD-WCAVY1879365 /dev/disk/by-id/wwn-0x50014ee25931e63a /dev/disk/by-path/pci-0000:09:00.0-sas-0x4433221102000000-lun-0
> DEVNAME=/dev/sdb
> DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb
> DEVTYPE=disk
> ID_ATA=1
> ID_ATA_DOWNLOAD_MICROCODE=1
> ID_ATA_FEATURE_SET_AAM=1
> ID_ATA_FEATURE_SET_AAM_CURRENT_VALUE=254
> ID_ATA_FEATURE_SET_AAM_ENABLED=0
> ID_ATA_FEATURE_SET_AAM_VENDOR_RECOMMENDED_VALUE=128
> ID_ATA_FEATURE_SET_HPA=1
> ID_ATA_FEATURE_SET_HPA_ENABLED=1
> ID_ATA_FEATURE_SET_PM=1
> ID_ATA_FEATURE_SET_PM_ENABLED=1
> ID_ATA_FEATURE_SET_PUIS=1
> ID_ATA_FEATURE_SET_PUIS_ENABLED=0
> ID_ATA_FEATURE_SET_SECURITY=1
> ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
> ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=408
> ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=408
> ID_ATA_FEATURE_SET_SMART=1
> ID_ATA_FEATURE_SET_SMART_ENABLED=1
> ID_ATA_SATA=1
> ID_ATA_SATA_SIGNAL_RATE_GEN1=1
> ID_ATA_SATA_SIGNAL_RATE_GEN2=1
> ID_ATA_WRITE_CACHE=1
> ID_ATA_WRITE_CACHE_ENABLED=1
> ID_BUS=ata
> ID_FS_LABEL=alina.o81.5:media3
> ID_FS_LABEL_ENC=alina.o81.5:media3
> ID_FS_TYPE=linux_raid_member
> ID_FS_USAGE=raid
> ID_FS_UUID=058bd7b0-455c-0d7e-6de0-a845ea05ee38
> ID_FS_UUID_ENC=058bd7b0-455c-0d7e-6de0-a845ea05ee38
> ID_FS_UUID_SUB=ade4cbb9-501c-cd2c-00b2-607f1699133b
> ID_FS_UUID_SUB_ENC=ade4cbb9-501c-cd2c-00b2-607f1699133b
> ID_FS_VERSION=1.2
> ID_MODEL=WDC_WD20EARS-00S8B1
> ID_MODEL_ENC=WDC\x20WD20EARS-00S8B1\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
> ID_PATH=pci-0000:09:00.0-sas-0x4433221102000000-lun-0
> ID_PATH_TAG=pci-0000_09_00_0-sas-0x4433221102000000-lun-0
> ID_REVISION=80.00A80
> ID_SERIAL=WDC_WD20EARS-00S8B1_WD-WCAVY1879365
> ID_SERIAL_SHORT=WD-WCAVY1879365
> ID_TYPE=disk
> ID_WWN=0x50014ee25931e63a
> ID_WWN_WITH_EXTENSION=0x50014ee25931e63a
> MAJOR=8
> MINOR=16
> SEQNUM=3240
> SUBSYSTEM=block
> USEC_INITIALIZED=2361
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: frequent disk activity with mdadm-3.3
From: Marco Schindler @ 2014-09-18 10:38 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20140918200347.762845bb@notabene.brown>
On 18.09.2014, at 12:03, NeilBrown <neilb@suse.de> wrote:
> Getting a "change" even on spindown is causing the problem I suspect.
> A change in 3.3.1 causes "mdadm -I" to be run on a device when it 'changes'.
> That will read from the device which will wake it up.
> (commit 25392f5fc59f96fb76 - revert it and the symptom will probably go away).
>
> I really think the "bug" here is that the change event is emitted on
> 'spindown', but maybe the bug is that the exact meaning of 'change' isn't
> well documented.
>
> I can probably get "mdadm -I" to use O_EXCL which will fail on devices
> already in an array, but I'm not sure that is a complete solution. You could
> still get wakeups on other devices.
>
> Can you rung the 'udevadm monitor' again, but this time with '--property'.
> Maybe there is some property associated with spindown events which we can use
> to ignore them.
>
> NeilBrown
sure. I also took separate logs for standby and wakeup.
change events only occur when the drive goes standby (see below).
strangely enough, there are no events when the drive wakes up.
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
KERNEL[558130.868218] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda (block)
ACTION=change
DEVNAME=/dev/sda
DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda
DEVTYPE=disk
MAJOR=8
MINOR=0
SEQNUM=3239
SUBSYSTEM=block
KERNEL[558132.028485] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
ACTION=change
DEVNAME=/dev/sdb
DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb
DEVTYPE=disk
MAJOR=8
MINOR=16
SEQNUM=3240
SUBSYSTEM=block
UDEV [558139.263973] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda (block)
ACTION=change
DEVLINKS=/dev/disk/by-id/ata-WDC_WD20EARS-00S8B1_WD-WCAVY1872131 /dev/disk/by-id/wwn-0x50014ee203dca984 /dev/disk/by-path/pci-0000:09:00.0-sas-0x4433221103000000-lun-0
DEVNAME=/dev/sda
DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda
DEVTYPE=disk
ID_ATA=1
ID_ATA_DOWNLOAD_MICROCODE=1
ID_ATA_FEATURE_SET_AAM=1
ID_ATA_FEATURE_SET_AAM_CURRENT_VALUE=254
ID_ATA_FEATURE_SET_AAM_ENABLED=0
ID_ATA_FEATURE_SET_AAM_VENDOR_RECOMMENDED_VALUE=128
ID_ATA_FEATURE_SET_HPA=1
ID_ATA_FEATURE_SET_HPA_ENABLED=1
ID_ATA_FEATURE_SET_PM=1
ID_ATA_FEATURE_SET_PM_ENABLED=1
ID_ATA_FEATURE_SET_PUIS=1
ID_ATA_FEATURE_SET_PUIS_ENABLED=0
ID_ATA_FEATURE_SET_SECURITY=1
ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=408
ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=408
ID_ATA_FEATURE_SET_SMART=1
ID_ATA_FEATURE_SET_SMART_ENABLED=1
ID_ATA_SATA=1
ID_ATA_SATA_SIGNAL_RATE_GEN1=1
ID_ATA_SATA_SIGNAL_RATE_GEN2=1
ID_ATA_WRITE_CACHE=1
ID_ATA_WRITE_CACHE_ENABLED=1
ID_BUS=ata
ID_FS_LABEL=alina.o81.5:media3
ID_FS_LABEL_ENC=alina.o81.5:media3
ID_FS_TYPE=linux_raid_member
ID_FS_USAGE=raid
ID_FS_UUID=058bd7b0-455c-0d7e-6de0-a845ea05ee38
ID_FS_UUID_ENC=058bd7b0-455c-0d7e-6de0-a845ea05ee38
ID_FS_UUID_SUB=da1c1cad-6661-174c-b0b9-4bc1a712902d
ID_FS_UUID_SUB_ENC=da1c1cad-6661-174c-b0b9-4bc1a712902d
ID_FS_VERSION=1.2
ID_MODEL=WDC_WD20EARS-00S8B1
ID_MODEL_ENC=WDC\x20WD20EARS-00S8B1\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
ID_PATH=pci-0000:09:00.0-sas-0x4433221103000000-lun-0
ID_PATH_TAG=pci-0000_09_00_0-sas-0x4433221103000000-lun-0
ID_REVISION=80.00A80
ID_SERIAL=WDC_WD20EARS-00S8B1_WD-WCAVY1872131
ID_SERIAL_SHORT=WD-WCAVY1872131
ID_TYPE=disk
ID_WWN=0x50014ee203dca984
ID_WWN_WITH_EXTENSION=0x50014ee203dca984
MAJOR=8
MINOR=0
SEQNUM=3239
SUBSYSTEM=block
USEC_INITIALIZED=2003
UDEV [558139.293742] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
ACTION=change
DEVLINKS=/dev/disk/by-id/ata-WDC_WD20EARS-00S8B1_WD-WCAVY1879365 /dev/disk/by-id/wwn-0x50014ee25931e63a /dev/disk/by-path/pci-0000:09:00.0-sas-0x4433221102000000-lun-0
DEVNAME=/dev/sdb
DEVPATH=/devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb
DEVTYPE=disk
ID_ATA=1
ID_ATA_DOWNLOAD_MICROCODE=1
ID_ATA_FEATURE_SET_AAM=1
ID_ATA_FEATURE_SET_AAM_CURRENT_VALUE=254
ID_ATA_FEATURE_SET_AAM_ENABLED=0
ID_ATA_FEATURE_SET_AAM_VENDOR_RECOMMENDED_VALUE=128
ID_ATA_FEATURE_SET_HPA=1
ID_ATA_FEATURE_SET_HPA_ENABLED=1
ID_ATA_FEATURE_SET_PM=1
ID_ATA_FEATURE_SET_PM_ENABLED=1
ID_ATA_FEATURE_SET_PUIS=1
ID_ATA_FEATURE_SET_PUIS_ENABLED=0
ID_ATA_FEATURE_SET_SECURITY=1
ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=408
ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=408
ID_ATA_FEATURE_SET_SMART=1
ID_ATA_FEATURE_SET_SMART_ENABLED=1
ID_ATA_SATA=1
ID_ATA_SATA_SIGNAL_RATE_GEN1=1
ID_ATA_SATA_SIGNAL_RATE_GEN2=1
ID_ATA_WRITE_CACHE=1
ID_ATA_WRITE_CACHE_ENABLED=1
ID_BUS=ata
ID_FS_LABEL=alina.o81.5:media3
ID_FS_LABEL_ENC=alina.o81.5:media3
ID_FS_TYPE=linux_raid_member
ID_FS_USAGE=raid
ID_FS_UUID=058bd7b0-455c-0d7e-6de0-a845ea05ee38
ID_FS_UUID_ENC=058bd7b0-455c-0d7e-6de0-a845ea05ee38
ID_FS_UUID_SUB=ade4cbb9-501c-cd2c-00b2-607f1699133b
ID_FS_UUID_SUB_ENC=ade4cbb9-501c-cd2c-00b2-607f1699133b
ID_FS_VERSION=1.2
ID_MODEL=WDC_WD20EARS-00S8B1
ID_MODEL_ENC=WDC\x20WD20EARS-00S8B1\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
ID_PATH=pci-0000:09:00.0-sas-0x4433221102000000-lun-0
ID_PATH_TAG=pci-0000_09_00_0-sas-0x4433221102000000-lun-0
ID_REVISION=80.00A80
ID_SERIAL=WDC_WD20EARS-00S8B1_WD-WCAVY1879365
ID_SERIAL_SHORT=WD-WCAVY1879365
ID_TYPE=disk
ID_WWN=0x50014ee25931e63a
ID_WWN_WITH_EXTENSION=0x50014ee25931e63a
MAJOR=8
MINOR=16
SEQNUM=3240
SUBSYSTEM=block
USEC_INITIALIZED=2361
^ permalink raw reply
* Re: frequent disk activity with mdadm-3.3
From: NeilBrown @ 2014-09-18 10:03 UTC (permalink / raw)
To: Marco Schindler; +Cc: linux-raid
In-Reply-To: <DDAF4EEF-EFA9-4B0E-BB97-E5D8B9788A70@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2938 bytes --]
On Mon, 15 Sep 2014 12:52:07 +0200 Marco Schindler
<marco.schindler@gmail.com> wrote:
>
> On 15.09.2014, at 02:18, NeilBrown <neilb@suse.de> wrote:
>
> > It would help to get "udevadm monitor" info to correlate with this.
> > Presumably some uevent is generated when the spindown happens. udev might
> > respond to this by reading from the device, which defeats the purpose...
> >
> >
> >>
> >> here’s the output of blktrace -d /dev/sda during that time.
> >> https://dl.dropboxusercontent.com/u/3464720/blktrace.tar.bz2
> >
> > That suggest that something is reading the metadata from the device almost
> > constantly. Mostly a 'kworker' thread. I don't know what would cause that.
> >
> > Let's look at the 'udevadm monitor' trace first and see what that shows.
>
> here’s the output of udevadm monitor during the spindown cycle while mdadm-3.3 is installed.
>
> monitor will print the received events for:
> UDEV - the event which udev sends out after rule processing
> KERNEL - the kernel uevent
>
> KERNEL[299719.336261] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
> UDEV [299720.646760] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
> KERNEL[299780.202901] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda (block)
> UDEV [299781.567308] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda (block)
> KERNEL[299841.090818] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
> UDEV [299842.407035] change /devices/pci0000:00/0000:00:06.0/0000:09:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb (block)
>
> please note that the issue immediately disappears when downgrading to mdadm-3.2 without touching anything else.
> I see the udev rules have been updated in mdadm-3.3..
Getting a "change" even on spindown is causing the problem I suspect.
A change in 3.3.1 causes "mdadm -I" to be run on a device when it 'changes'.
That will read from the device which will wake it up.
(commit 25392f5fc59f96fb76 - revert it and the symptom will probably go away).
I really think the "bug" here is that the change event is emitted on
'spindown', but maybe the bug is that the exact meaning of 'change' isn't
well documented.
I can probably get "mdadm -I" to use O_EXCL which will fail on devices
already in an array, but I'm not sure that is a complete solution. You could
still get wakeups on other devices.
Can you rung the 'udevadm monitor' again, but this time with '--property'.
Maybe there is some property associated with spindown events which we can use
to ignore them.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: [PATCH 0/5] Fixes for RAID1 resync
From: NeilBrown @ 2014-09-18 7:48 UTC (permalink / raw)
To: Brassow Jonathan; +Cc: Eivind Sarto, linux-raid, majianpeng
In-Reply-To: <2C41CCF8-8B5C-486F-AE43-42D10EBAA0A5@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 4707 bytes --]
On Tue, 16 Sep 2014 11:31:26 -0500 Brassow Jonathan <jbrassow@redhat.com>
wrote:
>
> On Sep 14, 2014, at 10:30 PM, NeilBrown wrote:
>
> > On Thu, 11 Sep 2014 12:12:01 -0500 Brassow Jonathan <jbrassow@redhat.com>
> > wrote:
> >
> >>
> >> On Sep 10, 2014, at 10:45 PM, Brassow Jonathan wrote:
> >>
> >>>
> >>> On Sep 10, 2014, at 1:20 AM, NeilBrown wrote:
> >>>
> >>>>
> >>>> Jon: could you test with these patches on top of what you
> >>>> have just in case something happens to fix the problem without
> >>>> me realising it?
> >>>
> >>> I'm on it. The test is running. I'll know later tomorrow.
> >>>
> >>> brassow
> >>
> >> The test is still failing from here. I grabbed 3.17.0-rc4, added the 5 patches, and got the attached backtraces when testing. As I said, the hangs are not exactly the same. This set shows the mdX_raid1 thread in the middle of handling a read failure.
> >
> > Thanks.
> > mdX_raid1 is blocked in freeze_array.
> > That could be caused by conf->nr_pending nor aligning properly with
> > conf->nr_queued.
> >
> > Both normal IO and resync IO can be retried with reschedule_retry()
> > and so be counted into ->nr_queued, but only normal IO gets counted in
> > ->nr_pending.
> >
> > Previously could could only possibly have on or the other and when handling
> > a read failure it could only be normal IO. But now that they two types can
> > interleave, we can have both normal and resync IO requests queued, so we need
> > to count them both in nr_pending.
> >
> > So the following patch might help.
> >
> > How complicated are your test scripts? Could you send them to me so I can
> > try too?
> >
> > Thanks,
> > NeilBrown
> >
> > diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> > index 888dbdfb6986..6a9c73435eb8 100644
> > --- a/drivers/md/raid1.c
> > +++ b/drivers/md/raid1.c
> > @@ -856,6 +856,7 @@ static void raise_barrier(struct r1conf *conf, sector_t sector_nr)
> > conf->next_resync + RESYNC_SECTORS),
> > conf->resync_lock);
> >
> > + conf->nr_pending++;
> > spin_unlock_irq(&conf->resync_lock);
> > }
> >
> > @@ -865,6 +866,7 @@ static void lower_barrier(struct r1conf *conf)
> > BUG_ON(conf->barrier <= 0);
> > spin_lock_irqsave(&conf->resync_lock, flags);
> > conf->barrier--;
> > + conf->nr_pending--;
> > spin_unlock_irqrestore(&conf->resync_lock, flags);
> > wake_up(&conf->wait_barrier);
> > }
>
> No luck, it is failing faster than before.
>
> I haven't looked into this myself, but the dm-raid1.c code makes use of dm-region-hash.c which coordinates recovery and nominal I/O in a way that allows them to both occur in a simple, non-overlapping way. I'm not sure it would make sense to use that instead of this new approach. I have no idea how much effort that would be, but I could have someone look into it at some point if you think it might be interesting.
>
Hi Jon,
I can see the appeal of using known-working code, but there is every chance
that we would break it when plugging it into md ;-)
I've found another bug.... it is a very subtle one and it has been around
since before the patch you bisected to so it probably isn't your bug.
It also only affects array with bad-blocks listed. The patch is below
but I very much doubt testing will show any change...
I'll keep looking..... oh, found one. This one looks more convincing.
If memory is short, make_request() will allocate an r1bio from the mempool
rather than from the slab. That r1bio won't have just been zeroed.
This is mostly OK as we initialise all the fields that aren't left in
a clean state ... except ->start_next_window.
We initialise that for write requests, but not for read.
So when we use a mempool-allocated r1bio that was previously used for
write and had ->start_next_window set, and is now used for read,
then things will go wrong.
So this patch definitely is worth testing.
Thanks for your continued patience in testing!!!
Thanks,
NeilBrown
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index a95f9e179e6f..7187d9b8431f 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1185,6 +1185,7 @@ read_again:
atomic_read(&bitmap->behind_writes) == 0);
}
r1_bio->read_disk = rdisk;
+ r1_bio->start_next_window = 0;
read_bio = bio_clone_mddev(bio, GFP_NOIO, mddev);
bio_trim(read_bio, r1_bio->sector - bio->bi_iter.bi_sector,
@@ -1444,6 +1445,7 @@ read_again:
r1_bio->state = 0;
r1_bio->mddev = mddev;
r1_bio->sector = bio->bi_iter.bi_sector + sectors_handled;
+ start_next_window = wait_barrier(conf, bio);
goto retry_write;
}
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: grub-probe: "found two disks with the index 3 for RAID md2" after changing superblock 0.9 to 1.0
From: NeilBrown @ 2014-09-18 6:49 UTC (permalink / raw)
To: Andy Smith; +Cc: linux-raid
In-Reply-To: <20140916040023.GG27492@bitfolk.com>
[-- Attachment #1: Type: text/plain, Size: 1954 bytes --]
On Tue, 16 Sep 2014 04:00:24 +0000 Andy Smith <andy@strugglers.net> wrote:
> Hi Neil,
>
> On Tue, Sep 16, 2014 at 07:54:58AM +1000, NeilBrown wrote:
> > On Mon, 15 Sep 2014 17:12:52 +0000 Andy Smith <andy@strugglers.net> wrote:
> > > On Mon, Sep 15, 2014 at 05:00:37PM +1000, NeilBrown wrote:
> > > > mdadm --zero-super --metadata=0.90 /dev/sda3
> > > >
> > > > will remove any 0.90 metadata.
> > >
> > > Trying that whole the array is running:
> > >
> > > mdadm: Couldn't open /dev/sda3 for write - not zeroing
> > >
> > > This needs to be done with the array stopped, presumably?
> >
> > It's preferred, but if you are really confident (and I think you are), then
> > adding --force is justified and should make it work.
>
> Yep, that worked, thank you! No more complaints from grub-probe.
>
> Do you consider it a bug that mdadm does not zero the old metadata
> when upgrading it to a new version?
No, but it might be excessive caution. I think it was a deliberate decision
to make it (relatively) easy to revert if anything when wrong.
mdadm always prefers the more recent metadata, so the old metadata is
effectively invisible to mdadm. Not to grub though.
>
> Or do you consider it a documentation issue to explain that it may
> be necessary to manually do it?
Maybe. Though I really don't want to impose that sort of burden on users.
Things should "just work". Maybe I should bite the bullet and just erase
the old metadata.... if I do it *after* the array is assembled it should be
safe.
The code might be a bit clumsy but that isn't the end of the world...
I've made a not on my to-do list :-)
Thanks,
NeilBrown
>
> I think I would've liked grub-probe at least in verbose mode to say
> which version of metadata it was reading from where. That would've
> been immediately enlightening, so I'll see if they would accept a
> patch for that…
>
> Cheers,
> Andy
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: raid1_end_read_request does not retry failed READ from a recovering drive
From: NeilBrown @ 2014-09-18 1:05 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: linux-raid
In-Reply-To: <CAGRgLy54ByqUG8bTyPr_rDrNMbyFYRU+rx4LEWVqMHOw2=k=Dg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 5028 bytes --]
On Wed, 17 Sep 2014 20:57:13 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
wrote:
> Hi Neil,
>
> On Mon, Sep 8, 2014 at 10:17 AM, NeilBrown <neilb@suse.de> wrote:
> > On Sun, 7 Sep 2014 17:18:16 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
> > wrote:
> >
> >> Hi Neil,
> >> we see the following issue:
> >>
> >> # RAID1 has 2 drives A and B, drive B is recovering
> >> # READ request arrives
> >> # read_balanace selects drive B to read from, because READ sector
> >> comes before B->recovery_offset
> >> # READ is issued to drive B, but fails (drive B fails again)
> >>
> >> Now raid1_end_read_request() has the following code:
> >>
> >> if (uptodate)
> >> set_bit(R1BIO_Uptodate, &r1_bio->state);
> >> else {
> >> /* If all other devices have failed, we want to return
> >> * the error upwards rather than fail the last device.
> >> * Here we redefine "uptodate" to mean "Don't want to retry"
> >> */
> >> unsigned long flags;
> >> spin_lock_irqsave(&conf->device_lock, flags);
> >> if (r1_bio->mddev->degraded == conf->raid_disks ||
> >> (r1_bio->mddev->degraded == conf->raid_disks-1 &&
> >> !test_bit(Faulty, &conf->mirrors[mirror].rdev->flags)))
> >> uptodate = 1;
> >> spin_unlock_irqrestore(&conf->device_lock, flags);
> >> }
> >>
> >> According to this code uptodate wrongly becomes 1, because:
> >> r1_bio->mddev->degraded == conf->raid_disks-1 is TRUE
> >> and
> >> !test_bit(Faulty, &conf->mirrors[mirror].rdev->flags) is also TRUE
> >>
> >> Indeed, drive B is not marked as Faulty, but also not marked as In_sync.
> >> However, this function treats !Faulty being equal to In_Sync, so it
> >> decides that the last good drive failed, so it does not retry the
> >> READ.
> >>
> >> As a result, there is IO error, while we should have retried the READ
> >> from the healthy drive.
> >>
> >> This is happening in 3.8.13, but your master branch seems to have the
> >> same issue.
> >>
> >> What is a reasonable fix?
> >> 1) Do not read from drives which are !In_sync (a bit scary to read
> >> from such drive)
> >
> > It is perfectly safe to read from a !In_sync device providing you are before
> > ->recovery_offset.
> >
> >
> >> 2) replace !Faulty to In_sync check
> >
> > That probably makes sense... though that could race with raid1_spare_active().
> > If a read-error returned just after raid1_spare_active() set In_sync, and
> > before 'count' was subtracted from ->degraded, we would still set uptodate
> > when we shouldn't.
> > It probably make sense to put all of raid1_spare_active inside the spinlock -
> > it doesn't get call often enough that performance is an issue (I hope).
> >
> > So:
> > 1/ change !Faulty to In_sync
> > 2/ extend the spinlock in raid1_spare_active to cover the whole function.
>
>
> I made these fixes and reproduced the issue. However, the result is
> not what we expect:
>
> # raid1_end_read_request() now indeed adds the r1_bio into retry_list,
> as we wanted
> # raid1d calls fix_read_error()
> # fix_read_error() searches for an In_sync drive to read the data
> from. It finds such drive (this is our good drive A)
> # now fix_read_error() wants to rewrite the bad area. But it rewrites
> only on those drives that are In_sync (except the drive it got the
> data from). In our case, it never tries to rewrite the data on drive B
> (drive B is not marked Faulty and not marked In_sync). As a result,
> md_error() is not called, so drive B is still not marked as Failed
> when fix_read_error() completes
> # so handle_read_error() retries the original READ by calling
> read_balance(), which again in my case selects the recovering drive
> B...
>
> And then the whole flow repeats itself again and again...and READ
> never completes.
>
> Maybe we should not allow selecting recovering drives for READ? Or
> some other approach?
>
Thanks for the testing and analysis.
Presumably we just want handle_read_error() to write to all non-faulty
devices, not just the InSync ones.
i.e. the following patch.
Thanks,
NeilBrown
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 6a9c73435eb8..a95f9e179e6f 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2153,7 +2153,7 @@ static void fix_read_error(struct r1conf *conf, int read_disk,
d--;
rdev = conf->mirrors[d].rdev;
if (rdev &&
- test_bit(In_sync, &rdev->flags))
+ !test_bit(Faulty, &rdev->flags))
r1_sync_page_io(rdev, sect, s,
conf->tmppage, WRITE);
}
@@ -2165,7 +2165,7 @@ static void fix_read_error(struct r1conf *conf, int read_disk,
d--;
rdev = conf->mirrors[d].rdev;
if (rdev &&
- test_bit(In_sync, &rdev->flags)) {
+ !test_bit(Faulty, &rdev->flags)) {
if (r1_sync_page_io(rdev, sect, s,
conf->tmppage, READ)) {
atomic_add(s, &rdev->corrected_errors);
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: raid1_end_read_request does not retry failed READ from a recovering drive
From: Alexander Lyakas @ 2014-09-17 17:57 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20140908171702.5b2853fc@notabene.brown>
Hi Neil,
On Mon, Sep 8, 2014 at 10:17 AM, NeilBrown <neilb@suse.de> wrote:
> On Sun, 7 Sep 2014 17:18:16 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
> wrote:
>
>> Hi Neil,
>> we see the following issue:
>>
>> # RAID1 has 2 drives A and B, drive B is recovering
>> # READ request arrives
>> # read_balanace selects drive B to read from, because READ sector
>> comes before B->recovery_offset
>> # READ is issued to drive B, but fails (drive B fails again)
>>
>> Now raid1_end_read_request() has the following code:
>>
>> if (uptodate)
>> set_bit(R1BIO_Uptodate, &r1_bio->state);
>> else {
>> /* If all other devices have failed, we want to return
>> * the error upwards rather than fail the last device.
>> * Here we redefine "uptodate" to mean "Don't want to retry"
>> */
>> unsigned long flags;
>> spin_lock_irqsave(&conf->device_lock, flags);
>> if (r1_bio->mddev->degraded == conf->raid_disks ||
>> (r1_bio->mddev->degraded == conf->raid_disks-1 &&
>> !test_bit(Faulty, &conf->mirrors[mirror].rdev->flags)))
>> uptodate = 1;
>> spin_unlock_irqrestore(&conf->device_lock, flags);
>> }
>>
>> According to this code uptodate wrongly becomes 1, because:
>> r1_bio->mddev->degraded == conf->raid_disks-1 is TRUE
>> and
>> !test_bit(Faulty, &conf->mirrors[mirror].rdev->flags) is also TRUE
>>
>> Indeed, drive B is not marked as Faulty, but also not marked as In_sync.
>> However, this function treats !Faulty being equal to In_Sync, so it
>> decides that the last good drive failed, so it does not retry the
>> READ.
>>
>> As a result, there is IO error, while we should have retried the READ
>> from the healthy drive.
>>
>> This is happening in 3.8.13, but your master branch seems to have the
>> same issue.
>>
>> What is a reasonable fix?
>> 1) Do not read from drives which are !In_sync (a bit scary to read
>> from such drive)
>
> It is perfectly safe to read from a !In_sync device providing you are before
> ->recovery_offset.
>
>
>> 2) replace !Faulty to In_sync check
>
> That probably makes sense... though that could race with raid1_spare_active().
> If a read-error returned just after raid1_spare_active() set In_sync, and
> before 'count' was subtracted from ->degraded, we would still set uptodate
> when we shouldn't.
> It probably make sense to put all of raid1_spare_active inside the spinlock -
> it doesn't get call often enough that performance is an issue (I hope).
>
> So:
> 1/ change !Faulty to In_sync
> 2/ extend the spinlock in raid1_spare_active to cover the whole function.
I made these fixes and reproduced the issue. However, the result is
not what we expect:
# raid1_end_read_request() now indeed adds the r1_bio into retry_list,
as we wanted
# raid1d calls fix_read_error()
# fix_read_error() searches for an In_sync drive to read the data
from. It finds such drive (this is our good drive A)
# now fix_read_error() wants to rewrite the bad area. But it rewrites
only on those drives that are In_sync (except the drive it got the
data from). In our case, it never tries to rewrite the data on drive B
(drive B is not marked Faulty and not marked In_sync). As a result,
md_error() is not called, so drive B is still not marked as Failed
when fix_read_error() completes
# so handle_read_error() retries the original READ by calling
read_balance(), which again in my case selects the recovering drive
B...
And then the whole flow repeats itself again and again...and READ
never completes.
Maybe we should not allow selecting recovering drives for READ? Or
some other approach?
Thanks,
Alex.
>
> Thanks,
> NeilBrown
^ permalink raw reply
* Re: [PATCH v3 01/12] crypto: LLVMLinux: Add macro to remove use of VLAIS in crypto code
From: Behan Webster @ 2014-09-17 15:26 UTC (permalink / raw)
To: Herbert Xu, Dmitry Kasatkin
Cc: agk, clm, davem, dm-devel, fabf, jbacik, snitzer, tadeusz.struk,
akpm, bruce.w.allan, james.l.morris, john.griffin, linux-btrfs,
linux-crypto, linux-ima-devel, linux-ima-user, linux-kernel,
linux-raid, linux-security-module, neilb, qat-linux, serge,
thomas.lendacky, zohar, torvalds
In-Reply-To: <20140917113035.GB2129@gondor.apana.org.au>
On 09/17/14 04:30, Herbert Xu wrote:
> On Wed, Sep 17, 2014 at 02:15:40PM +0300, Dmitry Kasatkin wrote:
>> On 17/09/14 12:22, Herbert Xu wrote:
>>> On Mon, Sep 15, 2014 at 12:30:23AM -0700, behanw@converseincode.com wrote:
>>>> From: Behan Webster <behanw@converseincode.com>
>>>>
>>>> Add a macro which replaces the use of a Variable Length Array In Struct (VLAIS)
>>>> with a C99 compliant equivalent. This macro instead allocates the appropriate
>>>> amount of memory using an char array.
>>>>
>>>> The new code can be compiled with both gcc and clang.
>>>>
>>>> struct shash_desc contains a flexible array member member ctx declared with
>>>> CRYPTO_MINALIGN_ATTR, so sizeof(struct shash_desc) aligns the beginning
>>>> of the array declared after struct shash_desc with long long.
>>>>
>>>> No trailing padding is required because it is not a struct type that can
>>>> be used in an array.
>>>>
>>>> The CRYPTO_MINALIGN_ATTR is required so that desc is aligned with long long
>>>> as would be the case for a struct containing a member with
>>>> CRYPTO_MINALIGN_ATTR.
>>>>
>>>> Signed-off-by: Behan Webster <behanw@converseincode.com>
>>> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
>>>
>>> Thanks,
>> Just in case.
>> I would still follow advice from "Michał Mirosław" to use shash##__desc[]
Absolutely. I will be posting a v4 patchset . Just waiting a bit more
for more comments on v3.
The macro from v4 will look like this which I believe will satisfy the
concern and indeed be safer than my previous version.
+#define SHASH_DESC_ON_STACK(shash, tfm) \
+ char __##shash##_desc[sizeof(struct shash_desc) + \
+ crypto_shash_descsize(tfm)] CRYPTO_MINALIGN_ATTR; \
+ struct shash_desc *shash = (struct shash_desc *)__##shash##_desc
Hmm. Is it worth adding a comment with this macro explaining the reason this works? Essentially much of what is in the commit message?
> Oh yes of course. My ack is more about the approach.
Wonderful!
Indeed. I would have asked for you to wait for v4 anyways. :)
Thank you,
Behan
--
Behan Webster
behanw@converseincode.com
--
To unsubscribe from this list: send the line "unsubscribe linux-security-module" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Add disks and convert level 0 to level 5
From: Robin Hill @ 2014-09-17 15:15 UTC (permalink / raw)
To: Michael Muratet; +Cc: NeilBrown, linux-raid
In-Reply-To: <535A7DC1-0953-41D6-B149-DB038D1631D2@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3360 bytes --]
On Wed Sep 17, 2014 at 09:59:00am -0500, Michael Muratet wrote:
>
> On Sep 10, 2014, at 7:27 PM, NeilBrown wrote:
>
> > On Wed, 10 Sep 2014 11:15:24 -0500 Michael Muratet <muratetm@gmail.com> wrote:
> >
> >> I have a two-disk RAID0 system that is working splendidly, thanks
> >> to the list for the help.
> >>
> >> I managed to get my hands on more identical disks and since I have
> >> the disks and because I'm adding precious data, I'd like to add two
> >> more disks and grow to RAID5.
> >>
> >> I have partitioned the two new drives to type 'fd', /dev/sde and /dev/sdf
> >>
> >> I believe the command to accomplish the change is this:
> >>
> >> mdadm /dev/md0 --grow --level=5 --add /dev/sde /dev/sdf
> >>
> >> Following the old adage "measure twice, cut once", is this syntax
> >> correct? Is there any danger of data loss in such a conversion?
> >
> > I recommend creating a few loop-back devices and experimenting.
> > i.e.:
> > create some 100M files.
> > use "losetup" to turn them into block devices.
> > create an 2-device raid0
> > try converting it as you suggest.
> >
> > You find it doesn't do quite what you expected, but should be easy to fix.
>
> In case anyone wants to do a similar thing...
>
> As predicted, it did not do as I expected. I have another identical
> server and disks and so I did the experiment there.
>
> 980 sudo fdisk /dev/sdc
> 981 sudo fdisk /dev/sdd
> 982 sudo fdisk /dev/sde
> 983 sudo fdisk /dev/sdf
> 988 sudo partprobe /dev/sdc
> 989 sudo partprobe /dev/sdd
> 990 sudo partprobe /dev/sde
> 991 sudo partprobe /dev/sdf
> 995 sudo mdadm --create /dev/md0 --level=0 --raid-disk=2 /dev/sdc1 /dev/sdd1
> 996 sudo mkfs -t ext3 /dev/md0
> 997 sudo mount -t ext3 /dev/md0 /db
>
> Same RAID0 system I created before. Now to grow
>
> 1004 sudo mdadm /dev/md0 --grow --level=5 --add /dev/sde1 /dev/sdf1
>
> Makes a RAID5 system, but sde1 was added as a spare
>
> 1022 sudo mdadm --grow /dev/md0 --raid-devices=4
>
You should be able to combine the above into a single command (it worked
for me on loopback devices anyway):
sudo mdadm --grow /dev/md0 --level=5 --raid-devices=4 \
--add /dev/sde1 /dev/sdf1
> It's now a four disk RAID, but smaller than expected. It spread the
> original space over four drives, which I think is the documented
> behavior.
>
As md has no knowledge of the filesystem or how to change the size of
it, then all it can do it redistribute the original data over the
additional disks. The md device size should have increased though, even
if the filesystem size hadn't.
> 1040 sudo umount /dev/md0
> 1045 sudo e2fsck -f /dev/md0
> 1046 sudo resize2fs -p /dev/md0
> 1048 sudo mount -t ext3 /dev/md0 /db
> 1049 df -h
>
> /dev/md0 2.7T 202M 2.6T 1% /db
>
> which is what I wanted. I can't take my original system offline now,
> but when I can I will apply the same steps.
>
You shouldn't need to take it offline - resize2fs works perfectly well
online.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: Add disks and convert level 0 to level 5
From: Michael Muratet @ 2014-09-17 14:59 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20140911102719.053a5d5c@notabene.brown>
On Sep 10, 2014, at 7:27 PM, NeilBrown wrote:
> On Wed, 10 Sep 2014 11:15:24 -0500 Michael Muratet <muratetm@gmail.com> wrote:
>
>> I have a two-disk RAID0 system that is working splendidly, thanks to the list for the help.
>>
>> I managed to get my hands on more identical disks and since I have the disks and because I'm adding precious data, I'd like to add two more disks and grow to RAID5.
>>
>> I have partitioned the two new drives to type 'fd', /dev/sde and /dev/sdf
>>
>> I believe the command to accomplish the change is this:
>>
>> mdadm /dev/md0 --grow --level=5 --add /dev/sde /dev/sdf
>>
>> Following the old adage "measure twice, cut once", is this syntax correct? Is there any danger of data loss in such a conversion?
>
> I recommend creating a few loop-back devices and experimenting.
> i.e.:
> create some 100M files.
> use "losetup" to turn them into block devices.
> create an 2-device raid0
> try converting it as you suggest.
>
> You find it doesn't do quite what you expected, but should be easy to fix.
In case anyone wants to do a similar thing...
As predicted, it did not do as I expected. I have another identical server and disks and so I did the experiment there.
980 sudo fdisk /dev/sdc
981 sudo fdisk /dev/sdd
982 sudo fdisk /dev/sde
983 sudo fdisk /dev/sdf
988 sudo partprobe /dev/sdc
989 sudo partprobe /dev/sdd
990 sudo partprobe /dev/sde
991 sudo partprobe /dev/sdf
995 sudo mdadm --create /dev/md0 --level=0 --raid-disk=2 /dev/sdc1 /dev/sdd1
996 sudo mkfs -t ext3 /dev/md0
997 sudo mount -t ext3 /dev/md0 /db
Same RAID0 system I created before. Now to grow
1004 sudo mdadm /dev/md0 --grow --level=5 --add /dev/sde1 /dev/sdf1
Makes a RAID5 system, but sde1 was added as a spare
1022 sudo mdadm --grow /dev/md0 --raid-devices=4
It's now a four disk RAID, but smaller than expected. It spread the original space over four drives, which I think is the documented behavior.
1040 sudo umount /dev/md0
1045 sudo e2fsck -f /dev/md0
1046 sudo resize2fs -p /dev/md0
1048 sudo mount -t ext3 /dev/md0 /db
1049 df -h
/dev/md0 2.7T 202M 2.6T 1% /db
which is what I wanted. I can't take my original system offline now, but when I can I will apply the same steps.
Thanks for the help
Cheers
Mike
>
> Providing your new devices are reliable (as least read/write the entire drive
> once if you feel at all cautious) there is no particular danger of data loss.
>
> NeilBrown
>
>
>>
>> Thanks
>>
>> Mike--
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: [PATCH v3 02/12] btrfs: LLVMLinux: Remove VLAIS
From: Chris Mason @ 2014-09-17 13:49 UTC (permalink / raw)
To: behanw, agk, davem, dm-devel, fabf, herbert, jbacik, snitzer,
tadeusz.struk
Cc: akpm, bruce.w.allan, d.kasatkin, james.l.morris, john.griffin,
linux-btrfs, linux-crypto, linux-ima-devel, linux-ima-user,
linux-kernel, linux-raid, linux-security-module, neilb, qat-linux,
serge, thomas.lendacky, zohar, torvalds, Vinícius Tinti
In-Reply-To: <1410766234-1634-3-git-send-email-behanw@converseincode.com>
On 09/15/2014 03:30 AM, behanw@converseincode.com wrote:
> From: Vinícius Tinti <viniciustinti@gmail.com>
>
> Replaced the use of a Variable Length Array In Struct (VLAIS) with a C99
> compliant equivalent. This is the original VLAIS struct.
>
> struct {
> struct shash_desc shash;
> char ctx[crypto_shash_descsize(tfm)];
> } desc;
>
> This patch instead allocates the appropriate amount of memory using a
> char array using the SHASH_DESC_ON_STACK macro.
>
> The new code can be compiled with both gcc and clang.
>
> Signed-off-by: Vinícius Tinti <viniciustinti@gmail.com>
> Reviewed-by: Jan-Simon Möller <dl9pf@gmx.de>
> Reviewed-by: Mark Charlebois <charlebm@gmail.com>
> Signed-off-by: Behan Webster <behanw@converseincode.com>
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
Acked-by: Chris Mason <clm@fb.com>
On the btrfs bits. Thanks for the v3.
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-security-module" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 01/12] crypto: LLVMLinux: Add macro to remove use of VLAIS in crypto code
From: Herbert Xu @ 2014-09-17 11:30 UTC (permalink / raw)
To: Dmitry Kasatkin
Cc: behanw, agk, clm, davem, dm-devel, fabf, jbacik, snitzer,
tadeusz.struk, akpm, bruce.w.allan, james.l.morris, john.griffin,
linux-btrfs, linux-crypto, linux-ima-devel, linux-ima-user,
linux-kernel, linux-raid, linux-security-module, neilb, qat-linux,
serge, thomas.lendacky, zohar, torvalds
In-Reply-To: <54196D5C.6040104@samsung.com>
On Wed, Sep 17, 2014 at 02:15:40PM +0300, Dmitry Kasatkin wrote:
> On 17/09/14 12:22, Herbert Xu wrote:
> > On Mon, Sep 15, 2014 at 12:30:23AM -0700, behanw@converseincode.com wrote:
> >> From: Behan Webster <behanw@converseincode.com>
> >>
> >> Add a macro which replaces the use of a Variable Length Array In Struct (VLAIS)
> >> with a C99 compliant equivalent. This macro instead allocates the appropriate
> >> amount of memory using an char array.
> >>
> >> The new code can be compiled with both gcc and clang.
> >>
> >> struct shash_desc contains a flexible array member member ctx declared with
> >> CRYPTO_MINALIGN_ATTR, so sizeof(struct shash_desc) aligns the beginning
> >> of the array declared after struct shash_desc with long long.
> >>
> >> No trailing padding is required because it is not a struct type that can
> >> be used in an array.
> >>
> >> The CRYPTO_MINALIGN_ATTR is required so that desc is aligned with long long
> >> as would be the case for a struct containing a member with
> >> CRYPTO_MINALIGN_ATTR.
> >>
> >> Signed-off-by: Behan Webster <behanw@converseincode.com>
> > Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
> >
> > Thanks,
>
> Just in case.
> I would still follow advice from "Michał Mirosław" to use shash##__desc[]
Oh yes of course. My ack is more about the approach.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v3 01/12] crypto: LLVMLinux: Add macro to remove use of VLAIS in crypto code
From: Dmitry Kasatkin @ 2014-09-17 11:15 UTC (permalink / raw)
To: Herbert Xu, behanw
Cc: agk, clm, davem, dm-devel, fabf, jbacik, snitzer, tadeusz.struk,
akpm, bruce.w.allan, james.l.morris, john.griffin, linux-btrfs,
linux-crypto, linux-ima-devel, linux-ima-user, linux-kernel,
linux-raid, linux-security-module, neilb, qat-linux, serge,
thomas.lendacky, zohar, torvalds
In-Reply-To: <20140917092212.GA1213@gondor.apana.org.au>
On 17/09/14 12:22, Herbert Xu wrote:
> On Mon, Sep 15, 2014 at 12:30:23AM -0700, behanw@converseincode.com wrote:
>> From: Behan Webster <behanw@converseincode.com>
>>
>> Add a macro which replaces the use of a Variable Length Array In Struct (VLAIS)
>> with a C99 compliant equivalent. This macro instead allocates the appropriate
>> amount of memory using an char array.
>>
>> The new code can be compiled with both gcc and clang.
>>
>> struct shash_desc contains a flexible array member member ctx declared with
>> CRYPTO_MINALIGN_ATTR, so sizeof(struct shash_desc) aligns the beginning
>> of the array declared after struct shash_desc with long long.
>>
>> No trailing padding is required because it is not a struct type that can
>> be used in an array.
>>
>> The CRYPTO_MINALIGN_ATTR is required so that desc is aligned with long long
>> as would be the case for a struct containing a member with
>> CRYPTO_MINALIGN_ATTR.
>>
>> Signed-off-by: Behan Webster <behanw@converseincode.com>
> Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
>
> Thanks,
Just in case.
I would still follow advice from "Michał Mirosław" to use shash##__desc[]
- Dmitry
^ permalink raw reply
* Re: [PATCH v3 01/12] crypto: LLVMLinux: Add macro to remove use of VLAIS in crypto code
From: Herbert Xu @ 2014-09-17 9:22 UTC (permalink / raw)
To: behanw
Cc: agk, clm, davem, dm-devel, fabf, jbacik, snitzer, tadeusz.struk,
akpm, bruce.w.allan, d.kasatkin, james.l.morris, john.griffin,
linux-btrfs, linux-crypto, linux-ima-devel, linux-ima-user,
linux-kernel, linux-raid, linux-security-module, neilb, qat-linux,
serge, thomas.lendacky, zohar, torvalds
In-Reply-To: <1410766234-1634-2-git-send-email-behanw@converseincode.com>
On Mon, Sep 15, 2014 at 12:30:23AM -0700, behanw@converseincode.com wrote:
> From: Behan Webster <behanw@converseincode.com>
>
> Add a macro which replaces the use of a Variable Length Array In Struct (VLAIS)
> with a C99 compliant equivalent. This macro instead allocates the appropriate
> amount of memory using an char array.
>
> The new code can be compiled with both gcc and clang.
>
> struct shash_desc contains a flexible array member member ctx declared with
> CRYPTO_MINALIGN_ATTR, so sizeof(struct shash_desc) aligns the beginning
> of the array declared after struct shash_desc with long long.
>
> No trailing padding is required because it is not a struct type that can
> be used in an array.
>
> The CRYPTO_MINALIGN_ATTR is required so that desc is aligned with long long
> as would be the case for a struct containing a member with
> CRYPTO_MINALIGN_ATTR.
>
> Signed-off-by: Behan Webster <behanw@converseincode.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH 0/5] Fixes for RAID1 resync
From: Brassow Jonathan @ 2014-09-16 16:31 UTC (permalink / raw)
To: NeilBrown; +Cc: Eivind Sarto, linux-raid, majianpeng
In-Reply-To: <20140915133006.14e57085@notabene.brown>
On Sep 14, 2014, at 10:30 PM, NeilBrown wrote:
> On Thu, 11 Sep 2014 12:12:01 -0500 Brassow Jonathan <jbrassow@redhat.com>
> wrote:
>
>>
>> On Sep 10, 2014, at 10:45 PM, Brassow Jonathan wrote:
>>
>>>
>>> On Sep 10, 2014, at 1:20 AM, NeilBrown wrote:
>>>
>>>>
>>>> Jon: could you test with these patches on top of what you
>>>> have just in case something happens to fix the problem without
>>>> me realising it?
>>>
>>> I'm on it. The test is running. I'll know later tomorrow.
>>>
>>> brassow
>>
>> The test is still failing from here. I grabbed 3.17.0-rc4, added the 5 patches, and got the attached backtraces when testing. As I said, the hangs are not exactly the same. This set shows the mdX_raid1 thread in the middle of handling a read failure.
>
> Thanks.
> mdX_raid1 is blocked in freeze_array.
> That could be caused by conf->nr_pending nor aligning properly with
> conf->nr_queued.
>
> Both normal IO and resync IO can be retried with reschedule_retry()
> and so be counted into ->nr_queued, but only normal IO gets counted in
> ->nr_pending.
>
> Previously could could only possibly have on or the other and when handling
> a read failure it could only be normal IO. But now that they two types can
> interleave, we can have both normal and resync IO requests queued, so we need
> to count them both in nr_pending.
>
> So the following patch might help.
>
> How complicated are your test scripts? Could you send them to me so I can
> try too?
>
> Thanks,
> NeilBrown
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 888dbdfb6986..6a9c73435eb8 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -856,6 +856,7 @@ static void raise_barrier(struct r1conf *conf, sector_t sector_nr)
> conf->next_resync + RESYNC_SECTORS),
> conf->resync_lock);
>
> + conf->nr_pending++;
> spin_unlock_irq(&conf->resync_lock);
> }
>
> @@ -865,6 +866,7 @@ static void lower_barrier(struct r1conf *conf)
> BUG_ON(conf->barrier <= 0);
> spin_lock_irqsave(&conf->resync_lock, flags);
> conf->barrier--;
> + conf->nr_pending--;
> spin_unlock_irqrestore(&conf->resync_lock, flags);
> wake_up(&conf->wait_barrier);
> }
No luck, it is failing faster than before.
I haven't looked into this myself, but the dm-raid1.c code makes use of dm-region-hash.c which coordinates recovery and nominal I/O in a way that allows them to both occur in a simple, non-overlapping way. I'm not sure it would make sense to use that instead of this new approach. I have no idea how much effort that would be, but I could have someone look into it at some point if you think it might be interesting.
brassow
[-rc5 kernel with previous 5 patches, plus the one above]
Sep 15 16:52:35 bp-01 kernel: INFO: task kworker/u129:2:21621 blocked for more than 120 seconds.
Sep 15 16:52:35 bp-01 kernel: Tainted: G E 3.17.0-rc5 #1
Sep 15 16:52:35 bp-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 15 16:52:35 bp-01 kernel: kworker/u129:2 D 0000000000000001 0 21621 2 0x00000080
Sep 15 16:52:35 bp-01 kernel: Workqueue: writeback bdi_writeback_workfn (flush-253:11)
Sep 15 16:52:35 bp-01 kernel: ffff8802040538c8 0000000000000046 0000000000000000 ffff880217254150
Sep 15 16:52:35 bp-01 kernel: ffff880204050010 0000000000012bc0 0000000000012bc0 ffff8802173c2490
Sep 15 16:52:35 bp-01 kernel: ffff880204053898 ffff88021fa32bc0 ffff8802173c2490 ffffffff81580a60
Sep 15 16:52:35 bp-01 kernel: Call Trace:
Sep 15 16:52:35 bp-01 kernel: [<ffffffff81580a60>] ? yield_to+0x180/0x180
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580779>] schedule+0x29/0x70
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8158084c>] io_schedule+0x8c/0xd0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580a8c>] bit_wait_io+0x2c/0x50
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580d75>] __wait_on_bit+0x65/0x90
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811350a4>] wait_on_page_bit+0xc4/0xd0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8108eb00>] ? wake_atomic_t_function+0x40/0x40
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8114126a>] write_cache_pages+0x33a/0x510
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8113fda0>] ? set_page_dirty+0x60/0x60
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81141491>] generic_writepages+0x51/0x80
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811414f5>] do_writepages+0x35/0x40
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811bfbe9>] __writeback_single_inode+0x49/0x230
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811c3029>] writeback_sb_inodes+0x249/0x360
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811c3309>] wb_writeback+0xf9/0x2c0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811c3552>] wb_do_writeback+0x82/0x1f0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff810790c6>] ? ttwu_queue+0x136/0x150
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811c3730>] bdi_writeback_workfn+0x70/0x210
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8106b5fe>] process_one_work+0x14e/0x430
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8106b9ff>] worker_thread+0x11f/0x3c0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8106b8e0>] ? process_one_work+0x430/0x430
Sep 15 16:52:36 bp-01 kernel: [<ffffffff810707de>] kthread+0xce/0xf0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81070710>] ? kthread_freezable_should_stop+0x70/0x70
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8158432c>] ret_from_fork+0x7c/0xb0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81070710>] ? kthread_freezable_should_stop+0x70/0x70
Sep 15 16:52:36 bp-01 kernel: INFO: task kjournald:26375 blocked for more than 120 seconds.
Sep 15 16:52:36 bp-01 kernel: Tainted: G E 3.17.0-rc5 #1
Sep 15 16:52:36 bp-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 15 16:52:36 bp-01 kernel: kjournald D 0000000000000004 0 26375 2 0x00000080
Sep 15 16:52:36 bp-01 kernel: ffff8804019dfb98 0000000000000046 000000000000003c ffff88021726cf40
Sep 15 16:52:36 bp-01 kernel: ffff8804019dc010 0000000000012bc0 0000000000012bc0 ffff880415632f00
Sep 15 16:52:36 bp-01 kernel: ffff8804019dfb68 ffff88021fa92bc0 ffff880415632f00 ffff8804019dfc50
Sep 15 16:52:36 bp-01 kernel: Call Trace:
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580a60>] ? yield_to+0x180/0x180
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580779>] schedule+0x29/0x70
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8158084c>] io_schedule+0x8c/0xd0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580a8c>] bit_wait_io+0x2c/0x50
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580b76>] __wait_on_bit_lock+0x76/0xb0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580a60>] ? yield_to+0x180/0x180
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81580c28>] out_of_line_wait_on_bit_lock+0x78/0x90
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8108eb00>] ? wake_atomic_t_function+0x40/0x40
Sep 15 16:52:36 bp-01 kernel: [<ffffffff811ca27e>] __lock_buffer+0x2e/0x30
Sep 15 16:52:36 bp-01 kernel: [<ffffffffa042eac0>] journal_submit_data_buffers+0x2b0/0x2f0 [jbd]
Sep 15 16:52:36 bp-01 kernel: [<ffffffffa042eda6>] journal_commit_transaction+0x2a6/0xf80 [jbd]
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8108841f>] ? put_prev_entity+0x2f/0x400
Sep 15 16:52:36 bp-01 kernel: [<ffffffff810b211b>] ? try_to_del_timer_sync+0x5b/0x70
Sep 15 16:52:36 bp-01 kernel: [<ffffffffa0432ae1>] kjournald+0xf1/0x270 [jbd]
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8108ea70>] ? bit_waitqueue+0xb0/0xb0
Sep 15 16:52:36 bp-01 kernel: [<ffffffffa04329f0>] ? commit_timeout+0x10/0x10 [jbd]
Sep 15 16:52:36 bp-01 kernel: [<ffffffff810707de>] kthread+0xce/0xf0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81070710>] ? kthread_freezable_should_stop+0x70/0x70
Sep 15 16:52:36 bp-01 kernel: [<ffffffff8158432c>] ret_from_fork+0x7c/0xb0
Sep 15 16:52:36 bp-01 kernel: [<ffffffff81070710>] ? kthread_freezable_should_stop+0x70/0x70
^ permalink raw reply
* What happens with array when external write-intent bitmap fails?
From: Andrei Borzenkov @ 2014-09-16 13:42 UTC (permalink / raw)
To: linux-raid
Theoretically write-intent is optional, so if it fails it would be
possible to simply discard it and continue. But I have seen statement
that if external write-intent bitmap file is not available at time of
assembly, starting of array fails.
Thank you for clarification.
-andrei
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox