* Re: [PATCH 00/24] Clustered MD RAID1
From: Goldwyn Rodrigues @ 2015-02-10 17:00 UTC (permalink / raw)
To: NeilBrown; +Cc: lzhong, linux-raid
In-Reply-To: <20150206133952.173f1975@notabene.brown>
Hi Neil,
>
>
> hi Goldwyn,
> thanks for these - and sorry for the long delay. Lots of leave over
> southern summer, and the lots of email etc to deal with.
>
> This patch set is very close and I am tempted to just apply it and then
> fix things up with subsequent patches. In order to allow that, could you
> please:
> - rebase against current upstream
> - fix the checkpatch.pl errors and warnings.
> The "WARNING: line over 80 characters" are often a judgement call
> so I'm not particularly worried about those. Most, if not all, of
> the others should be followed just to have consistent layout.
Done.
>
> Then I'll queue them up for 3.21, providing I don't find anything that would
> hurt non-cluster usage ....
> On that topic: why initialise rv to -EINVAL in "metadata_update sends
> message...". That looks wrong.
Yes, this is fixed.
>
> I noticed that a number of times a patch will revert something that a
> previous patch added. It would be much nicer to fold these changes back into
> the original patch. Often this is just extra blank lines, but occasionally
> variable names are changed (md -> mddev). It should be given the final name
> when introduced. Every chunk in every patch should be directly relevant to
> that patch.
I have cross-checked this and I did not find anything with respect to
variable names. I did some cleanup with respect to the code though.
There is one instance where I have used a variable: cluster_setup_done
and then removed it. I think this is required to understand the patch
and a smooth transition to subsequent patches. However, if you want me
to aggressively remove that part, I should be able to do that.
>
> Some other issues, that could possibly be fixed up afterwards:
>
> - Is a clustername 64 bytes or 63 bytes? I would have thought 64,
> but the use of strlcpy make is 63 plus a nul. Is that really what is
> wanted?
Yes, it is 64 bytes. I haven't fixed this as yet.
>
> - Based on https://lkml.org/lkml/2012/10/23/580 it might be good to add
> "default n" to Kconfig, and possible add a WARN() if anyone tries to use
> the code.
Done. Added pr_warn while loading the module.
>
> - I'm a bit concerned about the behaviour on node failure.
> When a node fails, two things must happen w.r.t the bits in that node's
> bitmap.
> 1/ The corresponding regions of the array need to be resynced. You do have
> code to do this.
> 2/ Other nodes must avoid read-balancing on those regions until the
> resync has completed.
>
> You do have code for this second bit, but it looks wrong. It avoids
> read-balancing if ->area_resyncing(). That isn't sufficient.
> The "area_resyncing" is always (I assume) a relatively small region of
> the array which will be completely resynced quite quickly. It must be
> because writes are blocked to this area. However the region in which
> we must disable re-balancing can be much larger. It covers *all* bits
> that are set in any unsynced bitmap. So it isn't just the area that is
> currently being synced, but all areas that will be synced.
What are unsynced bitmaps? Are they bitmaps which are associated with an
active node or dirty bitmaps with dead nodes? If it is the former, I
agree this is not enough. If it is latter, all nodes maintain a linked
list of all the nodes which are currently performing resync (probably
because of multiple nodes died simultaneously). One node performs the
recovery (aka bitmap resync) of exactly one "dead" node at a time.
area_resyncing goes through all the nodes which are performing resync.
>
> - I think md_reload_sb() might be too simple. It probably should check that
> nothing serious has changed. The "mddev->raid_disks = 0" look suspicious.
> I'll have to think about this a bit more.
Yes, I get that feeling as well. However, I am not sure how to perform
an exact comparison to understand what has changed. Perhaps it needs a
new flag?
>
> That's all I can see for now. I'll have another look once I have it all in my tree.
>
I have put all the changes in my git:
https://github.com/goldwynr/linux
The branch cluster-md is against the latest upstream. I also performed a
small sanity test to check everything is working properly.
Let me know if you would want me to repost the entire patchset to the
mailing list.
--
Goldwyn
^ permalink raw reply
* ignoring %s as it reports %s as failed
From: Valentijn @ 2015-02-10 15:54 UTC (permalink / raw)
To: linux-raid
Hello list,
This is a repost - I sent it to linux-raid back in January, but you were
all still having holidays - or hangovers or anything.
I was wondering why "--assemble --force" didn't work for a particular
(failed) raid-device. I got it to work and the data is safe now, but the
question about the non-working assembly remains.
I was trying to resurrect an array from a Synology disk station. As
follows: raid5 array with 4 disks, one disk utterly broken
(click-clack-click), one disk thrown out of the array.
Please note, that I used dmsetup to create an overlay disk, so the
"/dev/mapper/diskXp5" actually points to a /dev/sdN5 which is the
original array; I'm trying to not harm the original data. Here is the
result of mdadm v3.2.5 running assemble+force:
~# mdadm --verbose --assemble --force /dev/md99 /dev/mapper/disk1p5
/dev/mapper/disk2p5 /dev/mapper/disk3p5
mdadm: looking for devices for /dev/md99
mdadm: /dev/mapper/disk1p5 is identified as a member of /dev/md99, slot
32768.
mdadm: /dev/mapper/disk2p5 is identified as a member of /dev/md99, slot 1.
mdadm: /dev/mapper/disk3p5 is identified as a member of /dev/md99, slot 2.
mdadm: ignoring /dev/mapper/disk2p5 as it reports /dev/mapper/disk1p5 as
failed
mdadm: ignoring /dev/mapper/disk3p5 as it reports /dev/mapper/disk1p5 as
failed
mdadm: No suitable drives found for /dev/md99
OK. http://permalink.gmane.org/gmane.linux.raid/45064 says I should use
mdadm 3.3+, and luckily I had v3.3.2 - 21st August lying around, so
there we go:
~# ./mdadm --verbose --assemble --force /dev/md99 /dev/mapper/disk1p5
/dev/mapper/disk2p5 /dev/mapper/disk3p5
mdadm: looking for devices for /dev/md99
mdadm: /dev/mapper/disk1p5 is identified as a member of /dev/md99, slot
32768.
mdadm: /dev/mapper/disk2p5 is identified as a member of /dev/md99, slot 1.
mdadm: /dev/mapper/disk3p5 is identified as a member of /dev/md99, slot 2.
mdadm: no uptodate device for slot 0 of /dev/md99
mdadm: added /dev/mapper/disk3p5 to /dev/md99 as 2
mdadm: no uptodate device for slot 6 of /dev/md99
mdadm: added /dev/mapper/disk2p5 to /dev/md99 as 1
mdadm: /dev/md99 assembled from 2 drives - not enough to start the array.
Hmm, that doesn't help us either. (Slight difference between the two
versions, 3.2.5 deploys a broken array named md99 - which you can stop
afterwards - and 3.3.2 seems to auto-remove the array after trying to
start it - not sure if I describe the inner workings correctly)
Disk1 reports:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 8ad57063:cf592975:9fabcc2b:13922e19
Name : 2
Creation Time : Sun May 13 08:23:21 2012
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1944057456 (927.00 GiB 995.36 GB)
Array Size : 2916086016 (2781.00 GiB 2986.07 GB)
Used Dev Size : 1944057344 (927.00 GiB 995.36 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4ae571dc:3c458e25:d219b536:c8b9a50e
Update Time : Sun Dec 28 10:41:55 2014
Checksum : 9a35a9ae - correct
Events : 744446
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 32768
Array State : .AA. ('A' == active, '.' == missing)
Disk2:
[...]
State : clean
Device UUID : b381cfd4:d2474cc4:e4c0ad21:8565b2a5
Update Time : Sun Dec 28 10:41:55 2014
Checksum : b3569b7e - correct
Events : 744446
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : .AA. ('A' == active, '.' == missing)
Disk3:
[...]
State : clean
Device UUID : c00d4418:c8313607:fb3a2018:d267a693
Update Time : Sun Dec 28 10:41:55 2014
Checksum : 1e1b8de5 - correct
Events : 744446
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : .AA. ('A' == active, '.' == missing)
So the "Active device 32768" seems to be the culprit here, is that correct?
What would be the correct way to reassemble this array?
I managed to get it *working* again by recreating it with:
./mdadm --create --assume-clean --data-offset=136 -e 1.2 --level=5
--bitmap=none --raid-devices=4 --chunk=64 /dev/md99 /dev/mapper/disk1p5
/dev/mapper/disk2p5 /dev/mapper/disk3p5 missing
... but that feels a bit rude.
So, is there a "proper" way to reassemble this array?
For the record: the original array was most likely created on the
Synology disk station itself, which currently contains mdadm - v2.6.7 (I
don't know if that is the version that created the array). Also, I'm not
sure about the kernel version that's running there, but /lib/modules
contains modules that have a string "vermagic=2.6.15 ARMv5 gcc-3.4"
inside, so that might be the kernel version that was running when things
went wrong.
Best regards,
Valentijn
^ permalink raw reply
* Re: Wierd: Degrading while recovering raid5
From: Phil Turmel @ 2015-02-10 13:51 UTC (permalink / raw)
To: Adam Goryachev, Kyle Logue, linux-raid
In-Reply-To: <54D9B4AD.8010204@websitemanagers.com.au>
Hi Kyle,
Your symptoms look like classic timeout mismatch. Details interleaved.
On 02/10/2015 02:35 AM, Adam Goryachev wrote:
> There are other people who will jump in and help you with your problem,
> but I'll add a couple of pointers while you are waiting. See below.
> On 10/02/15 15:20, Kyle Logue wrote:
>> Hey all:
>>
>> I have a 5 disk software raid5 that was working fine until I decided
>> to swap out an old disk with a new one.
>>
>> mdadm /dev/md0 --add /dev/sda1
>> mdadm /dev/md0 --fail /dev/sde1
As Adam pointed out, you should have used --replace, but you probably
wouldn't have made it through the replace function anyways.
>> At this point it started automatically rebuilding the array.
>> About 60%? of the way in it stops and I see a lot of this repeated in
>> my dmesg:
>>
>> [Mon Feb 9 18:06:48 2015] ata5.00: exception Emask 0x0 SAct 0x0 SErr
>> 0x0 action 0x6 frozen
>> [Mon Feb 9 18:06:48 2015] ata5.00: failed command: SMART
>> [Mon Feb 9 18:06:48 2015] ata5.00: cmd
>> b0/da:00:00:4f:c2/00:00:00:00:00/00 tag 7
>> [Mon Feb 9 18:06:48 2015] res
>> 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
^^^^^^^^^
Smoking gun.
>> [Mon Feb 9 18:06:48 2015] ata5.00: status: { DRDY }
>> [Mon Feb 9 18:06:48 2015] ata5: hard resetting link
>> [Mon Feb 9 18:06:58 2015] ata5: softreset failed (1st FIS failed)
>> [Mon Feb 9 18:06:58 2015] ata5: hard resetting link
>> [Mon Feb 9 18:07:08 2015] ata5: softreset failed (1st FIS failed)
>> [Mon Feb 9 18:07:08 2015] ata5: hard resetting link
>> [Mon Feb 9 18:07:12 2015] ata5: SATA link up 1.5 Gbps (SStatus 113
>> SControl 310)
>> [Mon Feb 9 18:07:12 2015] ata5.00: configured for UDMA/33
>> [Mon Feb 9 18:07:12 2015] ata5: EH complete
Notice that after a timeout error, the drive is unresponsive for several
more seconds -- about 24 in your case.
> .... read about timing mismatches
> between the kernel and the hard drive, and how to solve that. There was
> another post earlier today with some links to specific posts that will
> be helpful (check the online archive).
That would have been me. Start with this link for a description of what
you are experiencing:
http://marc.info/?l=linux-raid&m=135811522817345&w=1
First, you need to protect yourself from timeout mismatch due to the use
of desktop-grade drives. (Enterprise and raid-rated drives don't have
this problem.)
{ If you were stuck in the middle of a replace a you had just
worked-around your timeout problem, it would likely continue and
complete. You've lost that opportunity. }
Show us the output of "smartctl -x" for all of your drives if you'd like
advice on your particular drives. (Pasted inline is preferred.)
Second, you need to find and overwrite (with zeros) the bad sectors on
your drives. Or ddrescue to a complete set of replacement drives and
assemble those.
Third, you need to set up a cron job to scrub your array regularly to
clean out UREs before they accumulate beyond MD's ability to handle it
(20 read errors in an hour, 10 per hour sustained).
Phil
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: Peter Zijlstra @ 2015-02-10 11:01 UTC (permalink / raw)
To: NeilBrown; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150210092936.GW21418@twins.programming.kicks-ass.net>
On Tue, Feb 10, 2015 at 10:29:36AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 10, 2015 at 01:50:17PM +1100, NeilBrown wrote:
> > On Mon, 9 Feb 2015 10:10:00 +0100 Peter Zijlstra <peterz@infradead.org> wrote:
> > > > However, when io_schedule() explicitly calls blk_flush_plug(), then
> > > > @from_schedule=false variant is used, and the unplug functions are allowed to
> > > > allocate memory and block and maybe even call mempool_alloc() which might
> > > > call io_schedule().
Note that as it stands recursively calling io_schedule() is already
broken. Things like delayacct_blkio_{start,end}() do not nest properly.
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: Peter Zijlstra @ 2015-02-10 9:29 UTC (permalink / raw)
To: NeilBrown; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150210135017.7659e49c@notabene.brown>
On Tue, Feb 10, 2015 at 01:50:17PM +1100, NeilBrown wrote:
> On Mon, 9 Feb 2015 10:10:00 +0100 Peter Zijlstra <peterz@infradead.org> wrote:
> > > However, when io_schedule() explicitly calls blk_flush_plug(), then
> > > @from_schedule=false variant is used, and the unplug functions are allowed to
> > > allocate memory and block and maybe even call mempool_alloc() which might
> > > call io_schedule().
> > >
> > > This shouldn't be a problem as blk_flush_plug() spliced out the plug list, so
> > > any recursive call will find an empty list and do nothing.
> >
> > Unless, something along the way stuck something back on, right? So
> > should we stick an:
> >
> > WARN_ON(current->in_iowait);
> >
> > somewhere near where things are added to this plug list? (and move the
> > blk_flush_plug() call inside of where that's actually true of course).
>
> No, I don't think so.
>
> It is certainly possible that some request on plug->cb_list could add
> something to plug->list - which is processed after ->cb_list.
>
> I think the best way to think about this is that the *problem* was that a
> wait_event loop could spin without making any progress. So any time that
> clear forward progress is made it is safe sleep without necessitating the
> warning. Hence sched_annotate_sleep() is reasonable.
> blk_flush_plug() with definitely have dispatched some requests if it
> might_sleep(), so the sleep is OK.
Well, yes, but you forget that this gets us back into recursion land.
io_schedule() calling io_schedule() calling io_schedule() and *boom*
stack overflow -> dead machine.
We must either guarantee io_schedule() will never call io_schedule() or
that io_schedule() itself will not add new work to the current plug such
that calling io_schedule() itself will not recurse on the blk stuff.
Pick either option, but pick one.
Without providing such a guarantee I'm not comfortable making this warn
go away.
^ permalink raw reply
* Re: Wierd: Degrading while recovering raid5
From: Adam Goryachev @ 2015-02-10 7:35 UTC (permalink / raw)
To: Kyle Logue, linux-raid
In-Reply-To: <CAP7a4UQCB=jdf7=sz8MoYL+WGbMbT_09_xL460DLX-epLAS0Sw@mail.gmail.com>
Hi Kyle,
There are other people who will jump in and help you with your problem,
but I'll add a couple of pointers while you are waiting. See below.
On 10/02/15 15:20, Kyle Logue wrote:
> Hey all:
>
> I have a 5 disk software raid5 that was working fine until I decided
> to swap out an old disk with a new one.
>
> mdadm /dev/md0 --add /dev/sda1
> mdadm /dev/md0 --fail /dev/sde1
>
> At this point it started automatically rebuilding the array.
> About 60%? of the way in it stops and I see a lot of this repeated in my dmesg:
>
> [Mon Feb 9 18:06:48 2015] ata5.00: exception Emask 0x0 SAct 0x0 SErr
> 0x0 action 0x6 frozen
> [Mon Feb 9 18:06:48 2015] ata5.00: failed command: SMART
> [Mon Feb 9 18:06:48 2015] ata5.00: cmd
> b0/da:00:00:4f:c2/00:00:00:00:00/00 tag 7
> [Mon Feb 9 18:06:48 2015] res
> 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
> [Mon Feb 9 18:06:48 2015] ata5.00: status: { DRDY }
> [Mon Feb 9 18:06:48 2015] ata5: hard resetting link
> [Mon Feb 9 18:06:58 2015] ata5: softreset failed (1st FIS failed)
> [Mon Feb 9 18:06:58 2015] ata5: hard resetting link
> [Mon Feb 9 18:07:08 2015] ata5: softreset failed (1st FIS failed)
> [Mon Feb 9 18:07:08 2015] ata5: hard resetting link
> [Mon Feb 9 18:07:12 2015] ata5: SATA link up 1.5 Gbps (SStatus 113
> SControl 310)
> [Mon Feb 9 18:07:12 2015] ata5.00: configured for UDMA/33
> [Mon Feb 9 18:07:12 2015] ata5: EH complete
>
> ata5 corresponds to my /dev/sdc drive.
First, check if the drive is faulty.
dd if=/dev/sdc of=/dev/null bs=10M
If that completes without any errors from dd, then the drive can be read
OK. Now check the logs, was there any errors there? Especially if there
were errors in the logs, (or even if not) read about timing mismatches
between the kernel and the hard drive, and how to solve that. There was
another post earlier today with some links to specific posts that will
be helpful (check the online archive).
Finally, I think your first mistake was to fail the drive. You should
have replaced it which will stop you from losing protection from a
failed drive.
See the second answer to this question:
http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: Upgrading storage server
From: Adam Goryachev @ 2015-02-10 7:22 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <21721.30729.509462.974269@quad.stoffel.home>
On 10/02/15 14:16, John Stoffel wrote:
> Adam> After making a whole string of mistakes in building a iSCSI
> Adam> server about 2 years ago, I'm now looking to replace it without
> Adam> all the wrong turns/mistakes. I was hoping you could all offer
> Adam> some advice on hardware selection/choices.
>
> I remember those discussions, they were quite informative and it was
> interesting seeing Stan help you out. Now that you've got this system
> working well, or at least well enough, what is the biggest remaining
> problem you have?
The only current requirement is to get some sort of DR
configuration/setup in place, that doesn't involve restoring from
backups. Performance is satisfactory right now, so I don't want to hit
any new performance issues in the process.
> I've become a big fan of supermicro FatTwin systems, and they might be
> what you want here for your setup. But I'd also think about maybe you
> want to go to fewer larger PCIe SSD cards in mirrored pairs instead
> for better performance. Or is performance a problem still?
The users are satisfied with the current performance level, though I
suspect if performance could be improved without drastically increasing
the cost, then it would make sense as well.
Those FatTwin systems look pretty awesome, but since I only need two
systems (nodes) and ideally I want one in each rack, then it doesn't
quite work out. I tend to prefer white box systems, due to it being
easier to find replacement parts, and I am avoiding too much redundancy
within each system (eg dual power, raid6, etc) as I am relying on the
second node to take over, allowing the primary to be repaired and added
back in later.
> There's also *alot* to be said for simply replicating what you have,
> but with larger SSDs, say 1Tb each, and keeping the rest of the system
> and config exactly the same. Limit the changes, esp since you went
> through so much pain before.
That was my thoughts, although I assume motherboards, CPU's, and perhaps
SATA controller cards have changed a lot over the past 3 years (although
I note that Intel suggests the motherboard isn't EOL until this year).
I'd prefer to get current models of hardware so that they will be well
supported (ie, replacements are easy to get) for the next few years.
Basically, at the same time as adding DR capability, I will be
refreshing the model of hardware. I suppose repeating this process every
3 years means that the DR hardware will be up to 6 years old, which is
probably still satisfactory (unless I see a lot of failures there),
considering that there is still a replicated pair (as long as they don't
both fail at the same time, or lose 2 "disks" each at the same time).
> Now I might also think about upgrading all the clients to 10Gb as
> well, and just moving to a completely 10G network if possible. I seem
> to remember that you didn't have any way to throttle or setup Quality
> of Service limits on your iSCSI vs. other network traffic, which is
> why you ended up splitting up the traffic like this, so that a single
> VM couldn't bring the rest to their knees when a user did something
> silly.
Well, I split the iSCSI SAN and the user LAN partly to satisfy "best
practice", improve security, as well as obviously
performance/reliability. I don't think I'll upgrade all the VM servers
to 10G at this stage (aren't planning to replace them all for another 6
months or more). At that stage, it might be something to consider, but I
would still be concerned about one VM "hogging" all the disk bandwidth.
Perhaps in practice, it wouldn't be an issue, since it is more IOPS that
is the limiting factor, and you can steal all available IOPS without
using very much bandwidth. This will likely depend on the cost/ability
to get a 16 port (or minimum of 10port) 10Gbps switch. Maybe something
like this:
http://www.netgear.com.au/business/products/switches/smart/10g-smart-switch.aspx#tab-overview
at approx AUD$1800
> So again, if it's working well now, don't chage your architecture at
> all, just change some of the components for higher capacity or
> performance. This will also let you stress test the new cluster pair
> next to your production setup before you migrate the VMs over to the
> new setup and then move the old offsite.
I'm haven't properly thought about how to do the migration, but I would
think I can bring up one of the new servers, and replace the current
"secondary" in the DRBD. Then, when that has settled in (for a week or
so), I can flip it to become the primary. Again, allow to test for a
week or so (any issues I can easily flip it back to secondary and so
revert back to the known good status), and then remove the second old
server, and replace with the second new server. Finally, reconfigure
both old servers onsite with the DRBD proxy config. Once that is working
well, (and obviously all the data is synced up to date) I can move them
offsite.
> One warning is that you will need to make sure that the link between
> the two sites has enough bandwidth and low enough RTT so that you can
> properly replicate between them, esp if the end users will be
> generating a bunch of data that changes alot.
Yep, that is something I'm looking into at the moment. Supposedly with
drbdproxy, as long as the changed data per day is less than the
bandwidth per day, then it should work. Also, this will relate to how
much RAM is available on the drbdproxy node to cache the changes.
I'm actually struggling a little with getting the "right" data for this.
Currently, I'm pulling all the data from /proc/drbd into an RRD file
(for each drbd device). Hopefully I'm a little crazy, but if I do this:
rrdtool fetch ${i} AVERAGE -s -25h |grep -v nan|tail -288| cut -d' '
-f3| awk '{s+=$1}END{print s}'
Which should select the past 25 hours worth of 5 minute averages, then
remove the unknowns at the end (because the rrd file is only updated
every 30 minutes, the values are cached), and then pick only the last 24
hours of reports (288), pick out the nr (network read) value, sum all
those to get the total of the 5 minute average data read over the
network (by the secondary). Finally, I multiply this by 300 to get the
actual data transferred. (Assuming that a 5 minute average is 3MB/s,
therefore the original amount of data transferred is 3 x 300 = 900MB in
5minutes).
The problem is I got an answer of over 200GB, which isn't going to fit
on my WAN (max 10Mbps, or 1MB/s), unless I upgrade the WAN, or
compression works really well, or my calculations are entirely wrong.....
Anyway, this section is somewhat off-topic for here. I'll follow up the
DRBD side elsewhere.
Thanks for your comments/suggestions.
Regards,
Adam
> Adam> The target usage as above is an iSCSI server as the backend to a bunch
> Adam> of VM's. Currently I have two identical storage servers, using 7 x SSD
> Adam> with Linux MD Raid, then using LVM to divide it up for each VM, and then
> Adam> DRBD on top to sync the two servers together, on the top is ietd to
> Adam> share the multiple DRBD devices out. The two servers have a single
> Adam> 10Gbps connection between them for DRBD to sync the data. They also have
> Adam> a second 10Gbps ethernet for iscsi to use, with a pair of 1Gbps for
> Adam> management (on board). I have 8 x PC's running Xen with 2 x 1Gbps
> Adam> ethernet for iSCSI and one 1Gbps ethernet for the "user"/management LAN.
>
> Adam> Current hardware of the storage servers are:
> Adam> 7 x Intel 480GB SSD Model SSDSC2CW480A3
> Adam> 1 x Intel 180GB SSD Model SSDSC2CT180A4 (for the OS)
> Adam> 1 x LSI Logic SAS2308 PCI-Express (8 x SATA connections)
> Adam> 1 x Intel Dual port 10Gbps 82599EB SFI/SFP+ Ethernet
> Adam> 1 x Intel Xeon CPU E3-1230 V2 @ 3.30GHz
> Adam> Motherboard Intel S1200
> Adam> http://ark.intel.com/products/67494/Intel-Server-Board-S1200BTLR
>
> Adam> What I'm hoping to achieve is to purchase two new (identical) servers,
> Adam> using current recommended (and well supported for the new few years)
> Adam> parts, and then move the two existing servers to a remote site,
> Adam> combining with DRBD proxy to give a full, "live" off-site backup
> Adam> solution. (Note, by backup I mean Disaster Recovery, not backup).
>
> Adam> I would also like to be able to grow the total size of the data further
> Adam> if needed, currently I have 7 x 480G in RAID5, which is likely somewhat
> Adam> sub-optimal. Options include moving to larger size SSD, or at perhaps
> Adam> splitting into 2 x RAID5 arrays. The advantage of larger SSD's would be
> Adam> a smaller "system", with lower complexity, while using more smaller
> Adam> drives would provide (potentially) better performance, since each drive
> Adam> (regardless of size) has the same overall performance (both throughput
> Adam> and IOPS).
>
> Adam> I would appreciate any advise or suggestions you can make to help me
> Adam> avoid the many mistakes I made last time.
>
--
Adam Goryachev
Website Managers
P: +61 2 8304 0000 adam@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.com.au
^ permalink raw reply
* Wierd: Degrading while recovering raid5
From: Kyle Logue @ 2015-02-10 4:20 UTC (permalink / raw)
To: linux-raid
Hey all:
I have a 5 disk software raid5 that was working fine until I decided
to swap out an old disk with a new one.
mdadm /dev/md0 --add /dev/sda1
mdadm /dev/md0 --fail /dev/sde1
At this point it started automatically rebuilding the array.
About 60%? of the way in it stops and I see a lot of this repeated in my dmesg:
[Mon Feb 9 18:06:48 2015] ata5.00: exception Emask 0x0 SAct 0x0 SErr
0x0 action 0x6 frozen
[Mon Feb 9 18:06:48 2015] ata5.00: failed command: SMART
[Mon Feb 9 18:06:48 2015] ata5.00: cmd
b0/da:00:00:4f:c2/00:00:00:00:00/00 tag 7
[Mon Feb 9 18:06:48 2015] res
40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[Mon Feb 9 18:06:48 2015] ata5.00: status: { DRDY }
[Mon Feb 9 18:06:48 2015] ata5: hard resetting link
[Mon Feb 9 18:06:58 2015] ata5: softreset failed (1st FIS failed)
[Mon Feb 9 18:06:58 2015] ata5: hard resetting link
[Mon Feb 9 18:07:08 2015] ata5: softreset failed (1st FIS failed)
[Mon Feb 9 18:07:08 2015] ata5: hard resetting link
[Mon Feb 9 18:07:12 2015] ata5: SATA link up 1.5 Gbps (SStatus 113
SControl 310)
[Mon Feb 9 18:07:12 2015] ata5.00: configured for UDMA/33
[Mon Feb 9 18:07:12 2015] ata5: EH complete
ata5 corresponds to my /dev/sdc drive.
So I was worried but it didn't look so terrible when i did examine:
sudo mdadm --examine /dev/sd[dabfec]1 | egrep 'dev|Update|Role|State|Events'
/dev/sda1:
State : clean
Update Time : Sun Feb 8 20:43:27 2015
Device Role : spare
Array State : .A.AA ('A' == active, '.' == missing)
Events : 27009
/dev/sdb1:
State : clean
Update Time : Sun Feb 8 20:43:27 2015
Device Role : Active device 4
Array State : .A.AA ('A' == active, '.' == missing)
Events : 27009
/dev/sdc1:
State : clean
Update Time : Sun Feb 8 20:21:13 2015
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing)
Events : 26995
/dev/sdd1:
State : clean
Update Time : Sun Feb 8 20:43:27 2015
Device Role : Active device 1
Array State : .A.AA ('A' == active, '.' == missing)
Events : 27009
/dev/sde1:
State : clean
Update Time : Sun Feb 8 12:17:10 2015
Device Role : Active device 2
Array State : AAAAA ('A' == active, '.' == missing)
Events : 21977
/dev/sdf1:
State : clean
Update Time : Sun Feb 8 20:43:27 2015
Device Role : Active device 3
Array State : .A.AA ('A' == active, '.' == missing)
Events : 27009
So the event counts looked pretty close on the drives I was updating, so I did:
mdadm --stop /dev/md0
mdadm --assemble --force /dev/md0 /dev/sd[dabfec]1
But it stopped again during recovery at some point while at work with
the same ATA errors in the dmesg.
Searching the web for these errors show lots of people having this
issue with various linux distros and laying the blame on everything
from faulty SATA cables to BIOS to NVIDIA drivers - nothing
definitive. I powered off my box and reconnected all my SATA cables as
a sanity check.
I tried --assemble --force again and it got to 70%:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid5 sdc1[7] sda1[8] sdb1[6] sdf1[4] sdd1[5]
7814047744 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UU_UU]
[=============>.......] recovery = 68.9%
(1347855508/1953511936) finish=306.1min speed=32967K/sec
...but died again. I was monitoring dmesg like a hawk this time and
saw those ata5 errors every 3-15 minutes with different cmd and res
values. At the very end I got this:
[Mon Feb 9 23:11:01 2015] ata5.00: configured for UDMA/33
[Mon Feb 9 23:11:01 2015] sd 4:0:0:0: [sdc] Unhandled sense code
[Mon Feb 9 23:11:01 2015] sd 4:0:0:0: [sdc]
[Mon Feb 9 23:11:01 2015] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon Feb 9 23:11:01 2015] sd 4:0:0:0: [sdc]
[Mon Feb 9 23:11:01 2015] Sense Key : Medium Error [current] [descriptor]
[Mon Feb 9 23:11:01 2015] Descriptor sense data with sense
descriptors (in hex):
[Mon Feb 9 23:11:01 2015] 72 03 11 04 00 00 00 0c 00 0a 80 00
00 00 00 00
[Mon Feb 9 23:11:01 2015] a4 1c 1d e8
[Mon Feb 9 23:11:01 2015] sd 4:0:0:0: [sdc]
[Mon Feb 9 23:11:01 2015] Add. Sense: Unrecovered read error - auto
reallocate failed
[Mon Feb 9 23:11:01 2015] sd 4:0:0:0: [sdc] CDB:
[Mon Feb 9 23:11:01 2015] Read(10): 28 00 a4 1c 1d e8 00 00 80 00
[Mon Feb 9 23:11:01 2015] end_request: I/O error, dev sdc, sector 2753306088
[Mon Feb 9 23:11:01 2015] md/raid:md0: Disk failure on sdc1, disabling device.
[Mon Feb 9 23:11:01 2015] md/raid:md0: Operation continuing on 3 devices.
[Mon Feb 9 23:11:01 2015] ata5: EH complete
[Mon Feb 9 23:11:01 2015] md: md0: recovery interrupted.
[Mon Feb 9 23:11:01 2015] RAID conf printout:
[Mon Feb 9 23:11:01 2015] --- level:5 rd:5 wd:3
[Mon Feb 9 23:11:01 2015] disk 0, o:0, dev:sdc1
[Mon Feb 9 23:11:01 2015] disk 1, o:1, dev:sdd1
[Mon Feb 9 23:11:01 2015] disk 2, o:1, dev:sda1
[Mon Feb 9 23:11:01 2015] disk 3, o:1, dev:sdf1
[Mon Feb 9 23:11:01 2015] disk 4, o:1, dev:sdb1
[Mon Feb 9 23:11:01 2015] RAID conf printout:
[Mon Feb 9 23:11:01 2015] --- level:5 rd:5 wd:3
[Mon Feb 9 23:11:01 2015] disk 1, o:1, dev:sdd1
[Mon Feb 9 23:11:01 2015] disk 2, o:1, dev:sda1
[Mon Feb 9 23:11:01 2015] disk 3, o:1, dev:sdf1
[Mon Feb 9 23:11:01 2015] disk 4, o:1, dev:sdb1
[Mon Feb 9 23:11:01 2015] RAID conf printout:
[Mon Feb 9 23:11:01 2015] --- level:5 rd:5 wd:3
[Mon Feb 9 23:11:01 2015] disk 1, o:1, dev:sdd1
[Mon Feb 9 23:11:01 2015] disk 2, o:1, dev:sda1
[Mon Feb 9 23:11:01 2015] disk 3, o:1, dev:sdf1
[Mon Feb 9 23:11:01 2015] disk 4, o:1, dev:sdb1
[Mon Feb 9 23:11:01 2015] RAID conf printout:
[Mon Feb 9 23:11:01 2015] --- level:5 rd:5 wd:3
[Mon Feb 9 23:11:01 2015] disk 1, o:1, dev:sdd1
[Mon Feb 9 23:11:01 2015] disk 3, o:1, dev:sdf1
[Mon Feb 9 23:11:01 2015] disk 4, o:1, dev:sdb1
and mdstat now has:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid5 sdc1[7](F) sda1[8](S) sdb1[6] sdf1[4] sdd1[5]
7814047744 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/3] [_U_UU]
And now I am out of ideas. Any thoughts on correcting those ata5
errors? or skipping those sectors maybe? While sde1 is the disk i
manually failed, it hasn't been touched yet. The event count is way
off now, but maybe I can use that somehow? Should i replace the sata
cable for sdc and retry?
Anybody in DC want a beer on me for helping figure this out? I have
more log files stored, but was trying to keep it short.
Thanks for looking,
Kyle L
PS. mdadm v3.2.5 on Ubuntu 14.04 running linux 3.13.0-45
PPS. Last full backup was six months ago. Hmm.
^ permalink raw reply
* Re: Upgrading storage server
From: John Stoffel @ 2015-02-10 3:16 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid
In-Reply-To: <54D8A9AD.3060700@websitemanagers.com.au>
Adam> After making a whole string of mistakes in building a iSCSI
Adam> server about 2 years ago, I'm now looking to replace it without
Adam> all the wrong turns/mistakes. I was hoping you could all offer
Adam> some advice on hardware selection/choices.
I remember those discussions, they were quite informative and it was
interesting seeing Stan help you out. Now that you've got this system
working well, or at least well enough, what is the biggest remaining
problem you have?
I've become a big fan of supermicro FatTwin systems, and they might be
what you want here for your setup. But I'd also think about maybe you
want to go to fewer larger PCIe SSD cards in mirrored pairs instead
for better performance. Or is performance a problem still?
There's also *alot* to be said for simply replicating what you have,
but with larger SSDs, say 1Tb each, and keeping the rest of the system
and config exactly the same. Limit the changes, esp since you went
through so much pain before.
Now I might also think about upgrading all the clients to 10Gb as
well, and just moving to a completely 10G network if possible. I seem
to remember that you didn't have any way to throttle or setup Quality
of Service limits on your iSCSI vs. other network traffic, which is
why you ended up splitting up the traffic like this, so that a single
VM couldn't bring the rest to their knees when a user did something
silly.
So again, if it's working well now, don't chage your architecture at
all, just change some of the components for higher capacity or
performance. This will also let you stress test the new cluster pair
next to your production setup before you migrate the VMs over to the
new setup and then move the old offsite.
One warning is that you will need to make sure that the link between
the two sites has enough bandwidth and low enough RTT so that you can
properly replicate between them, esp if the end users will be
generating a bunch of data that changes alot.
Adam> The target usage as above is an iSCSI server as the backend to a bunch
Adam> of VM's. Currently I have two identical storage servers, using 7 x SSD
Adam> with Linux MD Raid, then using LVM to divide it up for each VM, and then
Adam> DRBD on top to sync the two servers together, on the top is ietd to
Adam> share the multiple DRBD devices out. The two servers have a single
Adam> 10Gbps connection between them for DRBD to sync the data. They also have
Adam> a second 10Gbps ethernet for iscsi to use, with a pair of 1Gbps for
Adam> management (on board). I have 8 x PC's running Xen with 2 x 1Gbps
Adam> ethernet for iSCSI and one 1Gbps ethernet for the "user"/management LAN.
Adam> Current hardware of the storage servers are:
Adam> 7 x Intel 480GB SSD Model SSDSC2CW480A3
Adam> 1 x Intel 180GB SSD Model SSDSC2CT180A4 (for the OS)
Adam> 1 x LSI Logic SAS2308 PCI-Express (8 x SATA connections)
Adam> 1 x Intel Dual port 10Gbps 82599EB SFI/SFP+ Ethernet
Adam> 1 x Intel Xeon CPU E3-1230 V2 @ 3.30GHz
Adam> Motherboard Intel S1200
Adam> http://ark.intel.com/products/67494/Intel-Server-Board-S1200BTLR
Adam> What I'm hoping to achieve is to purchase two new (identical) servers,
Adam> using current recommended (and well supported for the new few years)
Adam> parts, and then move the two existing servers to a remote site,
Adam> combining with DRBD proxy to give a full, "live" off-site backup
Adam> solution. (Note, by backup I mean Disaster Recovery, not backup).
Adam> I would also like to be able to grow the total size of the data further
Adam> if needed, currently I have 7 x 480G in RAID5, which is likely somewhat
Adam> sub-optimal. Options include moving to larger size SSD, or at perhaps
Adam> splitting into 2 x RAID5 arrays. The advantage of larger SSD's would be
Adam> a smaller "system", with lower complexity, while using more smaller
Adam> drives would provide (potentially) better performance, since each drive
Adam> (regardless of size) has the same overall performance (both throughput
Adam> and IOPS).
Adam> I would appreciate any advise or suggestions you can make to help me
Adam> avoid the many mistakes I made last time.
Adam> Regards,
Adam> Adam
Adam> --
Adam> Adam Goryachev
Adam> Website Managers
Adam> www.websitemanagers.com.au
Adam> --
Adam> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
Adam> the body of a message to majordomo@vger.kernel.org
Adam> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: NeilBrown @ 2015-02-10 2:50 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150209091000.GN5029@twins.programming.kicks-ass.net>
[-- Attachment #1: Type: text/plain, Size: 2464 bytes --]
On Mon, 9 Feb 2015 10:10:00 +0100 Peter Zijlstra <peterz@infradead.org> wrote:
> On Mon, Feb 09, 2015 at 12:13:57PM +1100, NeilBrown wrote:
> > I had to re-read the code (And your analysis) a couple of times to be sure ...
>
> Sorry :-)
My point was that actually reading it (rather than assuming I knew what it
said) actually helped!
>
> > However, when io_schedule() explicitly calls blk_flush_plug(), then
> > @from_schedule=false variant is used, and the unplug functions are allowed to
> > allocate memory and block and maybe even call mempool_alloc() which might
> > call io_schedule().
> >
> > This shouldn't be a problem as blk_flush_plug() spliced out the plug list, so
> > any recursive call will find an empty list and do nothing.
>
> Unless, something along the way stuck something back on, right? So
> should we stick an:
>
> WARN_ON(current->in_iowait);
>
> somewhere near where things are added to this plug list? (and move the
> blk_flush_plug() call inside of where that's actually true of course).
No, I don't think so.
It is certainly possible that some request on plug->cb_list could add
something to plug->list - which is processed after ->cb_list.
I think the best way to think about this is that the *problem* was that a
wait_event loop could spin without making any progress. So any time that
clear forward progress is made it is safe sleep without necessitating the
warning. Hence sched_annotate_sleep() is reasonable.
blk_flush_plug() with definitely have dispatched some requests if it
might_sleep(), so the sleep is OK.
>
> The only thing that really goes wrong then is if people 'forget' to put
> a loop around io_schedule().
You mean like in congestion_wait() ??
Though that is mostly called inside a loop...
NeilBrown
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 92f4b4b288dd..7334be27823d 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1111,6 +1111,14 @@ static inline void blk_flush_plug(struct task_struct *tsk)
{
struct blk_plug *plug = tsk->plug;
+ /*
+ * Any sleeping in blk_flush_plug() should not
+ * trigger the "do not call blocking ops" warning
+ * as it makes clear forward process (requests are
+ * dispatched) and so it will not cause indefinite
+ * looping in a higher level wait loop.
+ */
+ sched_annotate_sleep();
if (plug)
blk_flush_plug_list(plug, false);
}
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: mdsadm -A won't assemble my array
From: G. Michael Carter @ 2015-02-10 2:48 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <54D9672A.7040104@turmel.org>
Well I'm pouring myself an ice wine and raising a glass to you.
Downloaded fedora 21, ran the force assemble and it fixed my problem.
Glad I did the sensible thing this time and waited for a response
*wink*
Thanks for your help.
It's running a consistency check on those drives and then upgrading
this server to fedora 21... was next on my list anyways.
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: Phil Turmel @ 2015-02-10 2:04 UTC (permalink / raw)
To: G. Michael Carter; +Cc: linux-raid
In-Reply-To: <CANjusZdgzDxN4ZxsXFc20OU1Y9_LoScYQcZmffyRGHWYUviY0Q@mail.gmail.com>
Hi Michael,
On 02/09/2015 08:36 PM, G. Michael Carter wrote:
> Was actually dreading the complete info question, as the machine has
> no CD/DVD, it is my netboot station and holds all my ISO mirrors. But
> turns out starting it's network interface from the emergency shell was
> easier than I thought.
Almost all good livecds can be put on a thumb drive to boot from,
instead of using a real CD. I highly recommend sysrescuecd.org, FWIW.
> ---- uname
> Linux andromeda 3.16.6-203.fc20.x86_64 #1 SMP Sat Oct 25 12:44:32 UTC
> 2014 x86_64 x86_64 x86_64 GNU/Linux
Not too old, good. There been a steady stream of small bugfixes since
3.16. I'm not sure what fedora's been backporting.
> ---- mdadm -V
> mdadm - v3.3 - 3rd September 2013
Bugfixes to this, too.
> ---- disk-by-id
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W3009GE3
> -> ../../sdp
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W300E08A
> -> ../../sdk
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300PYF2
> -> ../../sdb
> lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300Q4YS
> -> ../../sdo
I think you missed the 'excerpt' part, but no harm done. Anyways, if I
recall Seagate model numbering (past misery), these are green drives.
Very bad for raid service. After we revive your array, you'll want to
do some reading on 'timeout mismatch'.[1] (You are dangerously close to
option "D" there, and option "C" is your only choice w/ green drives.)
> ---- mdadm -E
> /dev/sdb:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : clean
> Device UUID : e9e7af60:e1bc1c7f:107157b4:4099c48a
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:13:20 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 57eaf19 - correct
> Events : 158964
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdk:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : active
> Device UUID : ed09357e:36655c6d:b7430500:63d5e540
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:10:53 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 131587ad - correct
> Events : 158964
matching events.
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdo:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : clean
> Device UUID : de849807:80d7f071:9909f3f2:78022d94
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:11:55 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : e73f681f - correct
> Events : 158962
Events off by two. Pretty minor.
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdp:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
> Name : andromeda:3 (local to host andromeda)
> Creation Time : Tue Jul 22 16:02:30 2014
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
> Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
> Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
> Data Offset : 259072 sectors
> Super Offset : 8 sectors
> Unused Space : before=258984 sectors, after=3760 sectors
> State : clean
> Device UUID : 67fcd9a9:432b0c8b:178cc556:67b003b3
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Mon Feb 9 05:11:55 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 254563d8 - correct
> Events : 158962
Also off by two. Again, minor.
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
> ---- dmesg
> [ 576.890380] md: md3 stopped.
> [ 576.891629] md: unbind<sdb>
> [ 576.900841] md: export_rdev(sdb)
> [ 576.902147] md: unbind<sdo>
> [ 576.906824] md: export_rdev(sdo)
> [ 576.908029] md: unbind<sdk>
> [ 576.911845] md: export_rdev(sdk)
> [ 576.913030] md: unbind<sdp>
> [ 576.916862] md: export_rdev(sdp)
> [ 585.164936] md: md3 stopped.
> [ 585.360571] md: bind<sdk>
> [ 585.360833] md: bind<sdp>
> [ 585.361050] md: bind<sdo>
> [ 585.361261] md: bind<sdb>
> [ 585.361294] md: md3 stopped.
> [ 585.361298] md: unbind<sdb>
> [ 585.386616] md: export_rdev(sdb)
> [ 585.387767] md: unbind<sdo>
> [ 585.398644] md: export_rdev(sdo)
> [ 585.399633] md: unbind<sdp>
> [ 585.404662] md: export_rdev(sdp)
> [ 585.405681] md: unbind<sdk>
> [ 585.410647] md: export_rdev(sdk)
> ---- mdadm -A output
> mdadm: looking for devices for /dev/md3
> mdadm: /dev/sdb is identified as a member of /dev/md3, slot 0.
> mdadm: /dev/sdk is identified as a member of /dev/md3, slot 1.
> mdadm: /dev/sdo is identified as a member of /dev/md3, slot 3.
> mdadm: /dev/sdp is identified as a member of /dev/md3, slot 2.
> mdadm: added /dev/sdk to /dev/md3 as 1
> mdadm: added /dev/sdp to /dev/md3 as 2 (possibly out of date)
> mdadm: added /dev/sdo to /dev/md3 as 3 (possibly out of date)
> mdadm: added /dev/sdb to /dev/md3 as 0
> mdadm: /dev/md3 assembled from 2 drives - not enough to start the array.
Please redo this with an explicit command line so we can see what's
going on:
mdadm --assemble --force --verbose /dev/md3 /dev/sd[bkop]
> ---- /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1]
> md1 : active raid1 sda[0] sde[1]
> 1953383488 blocks super 1.2 [2/2] [UU]
> bitmap: 0/15 pages [0KB], 65536KB chunk
>
> md0 : active raid5 sdl[0] sdn[3] sdd[5] sdm[1]
> 8790405120 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md2 : active raid6 sdj[0] sdi[1] sdg[3] sdh[2]
> 5860270080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> unused devices: <none>
>
> ** md3 is shutdown which is why I'm assuming it's not there.
Yup. If --assemble --force doesn't work with your installed OS,
temporarily boot from a recent system rescue cd and do the above over
again (especially the /dev/disk/by-id excerpt).
Phil
[1] http://marc.info/?l=linux-raid&m=135811522817345&w=1
More history:
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=139050322510249&w=2
You might want to read more from those threads than just the mails I've
pointed out...
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: G. Michael Carter @ 2015-02-10 1:36 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <54D94D3C.3060808@turmel.org>
Was actually dreading the complete info question, as the machine has
no CD/DVD, it is my netboot station and holds all my ISO mirrors. But
turns out starting it's network interface from the emergency shell was
easier than I thought.
---- uname
Linux andromeda 3.16.6-203.fc20.x86_64 #1 SMP Sat Oct 25 12:44:32 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
---- mdadm -V
mdadm - v3.3 - 3rd September 2013
---- disk-by-id
total 0
lrwxrwxrwx 1 root root 9 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part3 -> ../../sdc3
lrwxrwxrwx 1 root root 10 Feb 9 20:16
ata-OCZ-AGILITY2_OCZ-12ENW740X6E8681U-part4 -> ../../sdc4
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST2000DL003-9VT166_5YD5QSG3
-> ../../sda
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST2000DL003-9VT166_5YD604E0
-> ../../sde
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-1CH166_Z1F2H9YC
-> ../../sdm
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-9YN166_S1F026CS
-> ../../sdf
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-9YN166_W1F0GD7Y
-> ../../sdn
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST3000DM001-9YN166_W1F0JSVP
-> ../../sdh
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_9XK0A9AD -> ../../sdd
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_9XK0AV1G -> ../../sdl
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_9XK0N7GY -> ../../sdg
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_Z291009B -> ../../sdi
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST33000651AS_Z2911DKS -> ../../sdj
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W3009GE3
-> ../../sdp
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_W300E08A
-> ../../sdk
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300PYF2
-> ../../sdb
lrwxrwxrwx 1 root root 9 Feb 9 20:16 ata-ST4000DM000-1F2168_Z300Q4YS
-> ../../sdo
lrwxrwxrwx 1 root root 9 Feb 9 20:16
lvm-pv-uuid-fIZX8P-yR3l-KR3t-b206-KiDM-PhpK-JwLevw -> ../../md0
lrwxrwxrwx 1 root root 9 Feb 9 20:16
lvm-pv-uuid-Uvrg7m-X8Hr-JJn9-CAlu-Lno6-K3rp-A2p6fj -> ../../md1
lrwxrwxrwx 1 root root 9 Feb 9 20:16
lvm-pv-uuid-XetEWl-bxcb-WOHF-jNIg-M10Q-xxz1-HDeCwH -> ../../md2
lrwxrwxrwx 1 root root 9 Feb 9 20:16 md-name-andromeda:0 -> ../../md0
lrwxrwxrwx 1 root root 9 Feb 9 20:16 md-name-andromeda:1 -> ../../md1
lrwxrwxrwx 1 root root 9 Feb 9 20:16 md-name-andromeda:2 -> ../../md2
lrwxrwxrwx 1 root root 9 Feb 9 20:16
md-uuid-569d52c7:91ba146a:2dc88abf:1dbd4f12 -> ../../md0
lrwxrwxrwx 1 root root 9 Feb 9 20:16
md-uuid-8b1dbda6:fc378fa5:774dcb4f:c273dca5 -> ../../md1
lrwxrwxrwx 1 root root 9 Feb 9 20:16
md-uuid-fb065d4d:c906243c:945b8291:73539d13 -> ../../md2
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5002da066d5 -> ../../sdd
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5002dad7e24 -> ../../sdl
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50036281493 -> ../../sdi
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50036406c91 -> ../../sdj
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50044338d40 -> ../../sde
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50045add261 -> ../../sda
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5004a1255d8 -> ../../sdf
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5004ffa2772 -> ../../sdm
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50050dd4721 -> ../../sdn
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c500510652c5 -> ../../sdh
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c500608925d1 -> ../../sdp
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c5006434f070 -> ../../sdb
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5000c50064360fff -> ../../sdo
lrwxrwxrwx 1 root root 9 Feb 9 20:16 wwn-0x5e83a97f4233045f -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part1 ->
../../sdc1
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part2 ->
../../sdc2
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part3 ->
../../sdc3
lrwxrwxrwx 1 root root 10 Feb 9 20:16 wwn-0x5e83a97f4233045f-part4 ->
../../sdc4
---- mdadm -E
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : clean
Device UUID : e9e7af60:e1bc1c7f:107157b4:4099c48a
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:13:20 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 57eaf19 - correct
Events : 158964
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : active
Device UUID : ed09357e:36655c6d:b7430500:63d5e540
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:10:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 131587ad - correct
Events : 158964
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : clean
Device UUID : de849807:80d7f071:9909f3f2:78022d94
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:11:55 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e73f681f - correct
Events : 158962
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ce6de916:62aeda9c:b5688f54:f5a9249d
Name : andromeda:3 (local to host andromeda)
Creation Time : Tue Jul 22 16:02:30 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 11720661504 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258984 sectors, after=3760 sectors
State : clean
Device UUID : 67fcd9a9:432b0c8b:178cc556:67b003b3
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 9 05:11:55 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 254563d8 - correct
Events : 158962
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
---- blkid
/dev/sda: UUID="8b1dbda6-fc37-8fa5-774d-cb4fc273dca5"
UUID_SUB="82a7750b-74d9-c901-766e-7802b2737d3e" LABEL="andromeda:1"
TYPE="linux_raid_member"
/dev/sdb: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="e9e7af60-e1bc-1c7f-1071-57b44099c48a" LABEL="andromeda:3"
TYPE="linux_raid_member"
/dev/sdc: PTUUID="99e8af68-ba3a-4343-9751-3217c4e3d9a1" PTTYPE="gpt"
/dev/sdc1: PARTUUID="69de1013-cf8a-4131-bdab-03705decd42e"
/dev/sdc2: UUID="944c019a-4fb0-434d-b998-361d90352230" TYPE="ext4"
PARTLABEL="ext4" PARTUUID="cf7f5556-567e-450a-9712-e3d2bc735537"
/dev/sdc3: UUID="9dbf5b08-afdd-47c5-b92b-5350dc26524c" TYPE="ext4"
PARTUUID="46335bee-eea2-441a-8422-207b5c5fa44c"
/dev/sdc4: UUID="7407b6b8-517f-49ca-aac0-01120d98fcdc" TYPE="swap"
PARTUUID="f14bdff5-23f9-42f0-a6e3-b859076ca0bc"
/dev/sdd: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="9786712b-834a-47c1-4402-a4eba529e89e" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sde: UUID="8b1dbda6-fc37-8fa5-774d-cb4fc273dca5"
UUID_SUB="44280e97-717e-09d5-92f5-3af95a5c7364" LABEL="andromeda:1"
TYPE="linux_raid_member"
/dev/sdf: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="fdb635cc-f399-943e-bbb0-67809f0ac896" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdg: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="08104eec-720d-4cae-ed0e-3ef06d4938ff" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdh: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="49b3eafa-f4ab-2bb3-7faa-6996dfc4bb00" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdi: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="a70e7fa1-6909-b030-e899-0f85555f0094" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdj: UUID="fb065d4d-c906-243c-945b-829173539d13"
UUID_SUB="c74f4f4f-0b4b-65d4-b8e9-90ce689678c8" LABEL="andromeda:2"
TYPE="linux_raid_member"
/dev/sdk: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="ed09357e-3665-5c6d-b743-050063d5e540" LABEL="andromeda:3"
TYPE="linux_raid_member"
/dev/sdl: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="bd04495d-ce3b-09c7-0cac-37b98f74b3a7" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdm: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="5c6e4d63-3f4d-a05e-d64e-62b339f4f767" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdn: UUID="569d52c7-91ba-146a-2dc8-8abf1dbd4f12"
UUID_SUB="1b2773d1-391b-3c78-42e0-fd405416d9e7" LABEL="andromeda:0"
TYPE="linux_raid_member"
/dev/sdo: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="de849807-80d7-f071-9909-f3f278022d94" LABEL="andromeda:3"
TYPE="linux_raid_member"
/dev/sdp: UUID="ce6de916-62ae-da9c-b568-8f54f5a9249d"
UUID_SUB="67fcd9a9-432b-0c8b-178c-c55667b003b3" LABEL="andromeda:3"
TYPE="linux_raid_member"
---- dmesg
[ 576.890380] md: md3 stopped.
[ 576.891629] md: unbind<sdb>
[ 576.900841] md: export_rdev(sdb)
[ 576.902147] md: unbind<sdo>
[ 576.906824] md: export_rdev(sdo)
[ 576.908029] md: unbind<sdk>
[ 576.911845] md: export_rdev(sdk)
[ 576.913030] md: unbind<sdp>
[ 576.916862] md: export_rdev(sdp)
[ 585.164936] md: md3 stopped.
[ 585.360571] md: bind<sdk>
[ 585.360833] md: bind<sdp>
[ 585.361050] md: bind<sdo>
[ 585.361261] md: bind<sdb>
[ 585.361294] md: md3 stopped.
[ 585.361298] md: unbind<sdb>
[ 585.386616] md: export_rdev(sdb)
[ 585.387767] md: unbind<sdo>
[ 585.398644] md: export_rdev(sdo)
[ 585.399633] md: unbind<sdp>
[ 585.404662] md: export_rdev(sdp)
[ 585.405681] md: unbind<sdk>
[ 585.410647] md: export_rdev(sdk)
---- mdadm -A output
mdadm: looking for devices for /dev/md3
mdadm: /dev/sdb is identified as a member of /dev/md3, slot 0.
mdadm: /dev/sdk is identified as a member of /dev/md3, slot 1.
mdadm: /dev/sdo is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdp is identified as a member of /dev/md3, slot 2.
mdadm: added /dev/sdk to /dev/md3 as 1
mdadm: added /dev/sdp to /dev/md3 as 2 (possibly out of date)
mdadm: added /dev/sdo to /dev/md3 as 3 (possibly out of date)
mdadm: added /dev/sdb to /dev/md3 as 0
mdadm: /dev/md3 assembled from 2 drives - not enough to start the array.
---- /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid1 sda[0] sde[1]
1953383488 blocks super 1.2 [2/2] [UU]
bitmap: 0/15 pages [0KB], 65536KB chunk
md0 : active raid5 sdl[0] sdn[3] sdd[5] sdm[1]
8790405120 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md2 : active raid6 sdj[0] sdi[1] sdg[3] sdh[2]
5860270080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
** md3 is shutdown which is why I'm assuming it's not there.
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: Phil Turmel @ 2015-02-10 0:13 UTC (permalink / raw)
To: G. Michael Carter, linux-raid
In-Reply-To: <CANjusZcnX=BArJJLMiBN4hfzRRLdUtD6mSq-mDZ6OhPB=KMX3w@mail.gmail.com>
Hi Michael,
[Convention on kernel.org is reply-to-all, to trim replies and to
bottom-post, or interleave your reply]
On 02/09/2015 06:05 PM, G. Michael Carter wrote:
> On Mon, Feb 9, 2015 at 3:21 PM, G. Michael Carter <mikey@carterfamily.ca> wrote:
>> Some time last night my machine had a kernel panic. Two of the arrays
>> didn't start up.
>>
>> One I managed to fix as a mdadm -E clued me in that three of the
>> drives were ok. So I just reassembled the three and added the fourth.
>> Then it just started no problem.
>>
>> My big array however I'm not so lucky.
>>
>> I've got a state of
>>
>>
>> Raid level: 5
>> /dev/sdb: AA.. (state: clean)
>> /dev/sdk: AAAA (state: active)
>> /dev/sdo: A.AA (state: clean)
>> /dev/sdp: A.AA (state: clean)
Please show us *all* of your mdadm -E output for this array. Pasted
inline is preferred. Also show a map of your device names versus drive
serial numbers. An excerpt from "ls -l /dev/disk/by-id/" will do. You
have many drives, and the kernel doesn't guarantee consistent naming.
>> Thus can only get two drives to match in any config. How do I get out
>> of this mess?
> After doing a lot more reading... I think I'm getting down to running
> something like this. assemble force isn't doing much.
This throwaway line is critical. --assemble --force is the right answer
to this situation, and if its not working, something else should be
investigated. Do *not* use --create.
Show your kernel and mdadm versions. Show the content of /proc/mdstat.
Show the output of:
mdadm --assemble --force --verbose /dev/mdX /dev/sd[bkop]
and the tail of "dmesg" that corresponds to the above.
> mdadm --create --assume-clean --level=5 --verbose --chunk 512K
> --raid-devices=4 /dev/md3 /dev/sdb /dev/sdk /dev/sdo /dev/sdp
>
> But as per the big warning... says to check with you guys first, also
> need help writing the command (since it seems to be a one shot type
> thing)
>
> Here's the key information I think I need from the examine:
>
> Raid level: 5
> Chunk Size: 512K
> Used Dev Size: 7813774336
> /dev/sdb: AA.. (state: clean - active device 0)
> /dev/sdk: AAAA (state: active - active device 1)
> /dev/sdo: A.AA (state: clean - active device 2)
> /dev/sdp: A.AA (state: clean - active device 3)
Oh, and this isn't nearly enough information to advise on --create, in
the remote chance it turns out to be the right answer.
Phil
^ permalink raw reply
* Re: mdsadm -A won't assemble my array
From: G. Michael Carter @ 2015-02-09 23:05 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CANjusZesi-8zxpZ9rCOEToSXwFoiApeN-ivO51qY=80R6_xb_g@mail.gmail.com>
After doing a lot more reading... I think I'm getting down to running
something like this. assemble force isn't doing much.
mdadm --create --assume-clean --level=5 --verbose --chunk 512K
--raid-devices=4 /dev/md3 /dev/sdb /dev/sdk /dev/sdo /dev/sdp
But as per the big warning... says to check with you guys first, also
need help writing the command (since it seems to be a one shot type
thing)
Here's the key information I think I need from the examine:
Raid level: 5
Chunk Size: 512K
Used Dev Size: 7813774336
/dev/sdb: AA.. (state: clean - active device 0)
/dev/sdk: AAAA (state: active - active device 1)
/dev/sdo: A.AA (state: clean - active device 2)
/dev/sdp: A.AA (state: clean - active device 3)
thanks
On Mon, Feb 9, 2015 at 3:21 PM, G. Michael Carter <mikey@carterfamily.ca> wrote:
> Some time last night my machine had a kernel panic. Two of the arrays
> didn't start up.
>
> One I managed to fix as a mdadm -E clued me in that three of the
> drives were ok. So I just reassembled the three and added the fourth.
> Then it just started no problem.
>
> My big array however I'm not so lucky.
>
> I've got a state of
>
>
> Raid level: 5
> /dev/sdb: AA.. (state: clean)
> /dev/sdk: AAAA (state: active)
> /dev/sdo: A.AA (state: clean)
> /dev/sdp: A.AA (state: clean)
>
> Thus can only get two drives to match in any config. How do I get out
> of this mess?
--
G. Michael Carter
Contact: H: 1-519-940-8935 | W: 1-905-267-8494 | M: 1-519-215-1869 |
F: 1-519-941-0009
Google Talk: xmpp:mikeycarter1974@gmail.com
^ permalink raw reply
* mdsadm -A won't assemble my array
From: G. Michael Carter @ 2015-02-09 20:21 UTC (permalink / raw)
To: linux-raid
Some time last night my machine had a kernel panic. Two of the arrays
didn't start up.
One I managed to fix as a mdadm -E clued me in that three of the
drives were ok. So I just reassembled the three and added the fourth.
Then it just started no problem.
My big array however I'm not so lucky.
I've got a state of
Raid level: 5
/dev/sdb: AA.. (state: clean)
/dev/sdk: AAAA (state: active)
/dev/sdo: A.AA (state: clean)
/dev/sdp: A.AA (state: clean)
Thus can only get two drives to match in any config. How do I get out
of this mess?
^ permalink raw reply
* Re: Upgrading storage server
From: Joe Landman @ 2015-02-09 14:47 UTC (permalink / raw)
To: Adam Goryachev, linux-raid
In-Reply-To: <54D8A9AD.3060700@websitemanagers.com.au>
On 02/09/2015 07:35 AM, Adam Goryachev wrote:
> Hi all,
>
> After making a whole string of mistakes in building a iSCSI server
> about 2 years ago, I'm now looking to replace it without all the wrong
> turns/mistakes. I was hoping you could all offer some advice on
> hardware selection/choices.
>
> The target usage as above is an iSCSI server as the backend to a bunch
> of VM's. Currently I have two identical storage servers, using 7 x SSD
> with Linux MD Raid, then using LVM to divide it up for each VM, and
> then DRBD on top to sync the two servers together, on the top is ietd
> to share the multiple DRBD devices out. The two servers have a single
> 10Gbps connection between them for DRBD to sync the data. They also
> have a second 10Gbps ethernet for iscsi to use, with a pair of 1Gbps
> for management (on board). I have 8 x PC's running Xen with 2 x 1Gbps
> ethernet for iSCSI and one 1Gbps ethernet for the "user"/management LAN.
>
> Current hardware of the storage servers are:
> 7 x Intel 480GB SSD Model SSDSC2CW480A3
> 1 x Intel 180GB SSD Model SSDSC2CT180A4 (for the OS)
We always use 2 drives in an MD RAID1 for OS.
> 1 x LSI Logic SAS2308 PCI-Express (8 x SATA connections)
Ok. This is a lower end card on the performance side.
> 1 x Intel Dual port 10Gbps 82599EB SFI/SFP+ Ethernet
> 1 x Intel Xeon CPU E3-1230 V2 @ 3.30GHz
> Motherboard Intel S1200
> http://ark.intel.com/products/67494/Intel-Server-Board-S1200BTLR
>
> What I'm hoping to achieve is to purchase two new (identical) servers,
> using current recommended (and well supported for the new few years)
> parts, and then move the two existing servers to a remote site,
> combining with DRBD proxy to give a full, "live" off-site backup
> solution. (Note, by backup I mean Disaster Recovery, not backup).
>
> I would also like to be able to grow the total size of the data
> further if needed, currently I have 7 x 480G in RAID5, which is likely
> somewhat sub-optimal. Options include moving to larger size SSD, or at
> perhaps splitting into 2 x RAID5 arrays.
Yes, RAIDx for x=5,6 are generally suboptimal for SSDs due to write
amplification from the RMW cycle. RAID10's are generally much gentler
on SSDs from a longevity scenario.
> The advantage of larger SSD's would be a smaller "system", with lower
> complexity, while using more smaller drives would provide
> (potentially) better performance, since each drive (regardless of
> size) has the same overall performance (both throughput and IOPS).
Are you performance limited now, or will you be shortly? If so the
performance arguments make sense.
>
> I would appreciate any advise or suggestions you can make to help me
> avoid the many mistakes I made last time.
I'm biased given what we do. If you are going to build it yourself, I'd
recommend sticking to known working elements that aren't a pain to setup
and manage. Focus on RAID10 for the primary storage, move the OS to a
completely different controller. Build the OS drives as MD RAID1.
You might want to investigate dm multipath as well as DRBD/md, and Ceph
RBD. I'm a huge fan and user of MD RAID, but you are asking much higher
level architectural questions, and MD RAID would be one of several
technologies you would use for this.
>
> Regards,
> Adam
>
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
e: landman@scalableinformatics.com
w: http://scalableinformatics.com
t: @scalableinfo
p: +1 734 786 8423 x121
c: +1 734 612 4615
^ permalink raw reply
* Upgrading storage server
From: Adam Goryachev @ 2015-02-09 12:35 UTC (permalink / raw)
To: linux-raid
Hi all,
After making a whole string of mistakes in building a iSCSI server about
2 years ago, I'm now looking to replace it without all the wrong
turns/mistakes. I was hoping you could all offer some advice on hardware
selection/choices.
The target usage as above is an iSCSI server as the backend to a bunch
of VM's. Currently I have two identical storage servers, using 7 x SSD
with Linux MD Raid, then using LVM to divide it up for each VM, and then
DRBD on top to sync the two servers together, on the top is ietd to
share the multiple DRBD devices out. The two servers have a single
10Gbps connection between them for DRBD to sync the data. They also have
a second 10Gbps ethernet for iscsi to use, with a pair of 1Gbps for
management (on board). I have 8 x PC's running Xen with 2 x 1Gbps
ethernet for iSCSI and one 1Gbps ethernet for the "user"/management LAN.
Current hardware of the storage servers are:
7 x Intel 480GB SSD Model SSDSC2CW480A3
1 x Intel 180GB SSD Model SSDSC2CT180A4 (for the OS)
1 x LSI Logic SAS2308 PCI-Express (8 x SATA connections)
1 x Intel Dual port 10Gbps 82599EB SFI/SFP+ Ethernet
1 x Intel Xeon CPU E3-1230 V2 @ 3.30GHz
Motherboard Intel S1200
http://ark.intel.com/products/67494/Intel-Server-Board-S1200BTLR
What I'm hoping to achieve is to purchase two new (identical) servers,
using current recommended (and well supported for the new few years)
parts, and then move the two existing servers to a remote site,
combining with DRBD proxy to give a full, "live" off-site backup
solution. (Note, by backup I mean Disaster Recovery, not backup).
I would also like to be able to grow the total size of the data further
if needed, currently I have 7 x 480G in RAID5, which is likely somewhat
sub-optimal. Options include moving to larger size SSD, or at perhaps
splitting into 2 x RAID5 arrays. The advantage of larger SSD's would be
a smaller "system", with lower complexity, while using more smaller
drives would provide (potentially) better performance, since each drive
(regardless of size) has the same overall performance (both throughput
and IOPS).
I would appreciate any advise or suggestions you can make to help me
avoid the many mistakes I made last time.
Regards,
Adam
--
Adam Goryachev
Website Managers
www.websitemanagers.com.au
^ permalink raw reply
* Re: raid md126, md127 problem after reboot, howto fix?
From: Sebastian Parschauer @ 2015-02-09 10:33 UTC (permalink / raw)
To: Wols Lists, Marc Widmer, linux-raid
In-Reply-To: <54D88A7A.4000207@youngman.org.uk>
On 09.02.2015 11:22, Wols Lists wrote:
> On 09/02/15 09:55, Sebastian Parschauer wrote:
>>>> Any ideas what i should do? Howto put the raid back together, possibly in
>>>> live mode, without rebooting in rescue mode and risk long downtime?
>> Just stop all arrays which aren't in use at the moment and assemble them
>> manually. If they are already running degraded, then add the now
>> unassociated disks to their respective running arrays.
>>
>> $ mdadm --stop /dev/md126
>> $ mdadm --stop /dev/md127
>> $ mdadm /dev/md1 --add /dev/sda1
>> $ mdadm /dev/md2 --add /dev/sda2
>>
>> This could require some syncing but then everything should be normal again.
>
> This isn't just a Debian/Ubuntu problem - I run gentoo.
Okay.
> And isn't there a re-add option? That will hopefully just require a
> recovery rather than a total resync.
Yes, sure, --re-add should be tried first. If it doesn't work, then the
--add option will definitely work.
Cheers,
Sebastian
^ permalink raw reply
* Re: raid md126, md127 problem after reboot, howto fix?
From: Wols Lists @ 2015-02-09 10:22 UTC (permalink / raw)
To: Sebastian Parschauer, Marc Widmer, linux-raid
In-Reply-To: <54D883FA.8080401@profitbricks.com>
On 09/02/15 09:55, Sebastian Parschauer wrote:
>> > Any ideas what i should do? Howto put the raid back together, possibly in
>> > live mode, without rebooting in rescue mode and risk long downtime?
> Just stop all arrays which aren't in use at the moment and assemble them
> manually. If they are already running degraded, then add the now
> unassociated disks to their respective running arrays.
>
> $ mdadm --stop /dev/md126
> $ mdadm --stop /dev/md127
> $ mdadm /dev/md1 --add /dev/sda1
> $ mdadm /dev/md2 --add /dev/sda2
>
> This could require some syncing but then everything should be normal again.
This isn't just a Debian/Ubuntu problem - I run gentoo.
And isn't there a re-add option? That will hopefully just require a
recovery rather than a total resync.
Cheers,
Wol
^ permalink raw reply
* [PATCH RESEND] Monitor: fix for regression with container devices
From: Artur Paszkiewicz @ 2015-02-09 10:13 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, pawel.baldysiak, Artur Paszkiewicz
This patch fixes 2 problems introduced by commit 9a518d8: not closing a
file descriptor and ignoring container devices. Array state is always
"inactive" for containers, so we make sure that the device is not a
container by reading also the "level" sysfs entry.
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-by: Pawel Baldysiak <pawel.baldysiak@intel.com>
---
Monitor.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Monitor.c b/Monitor.c
index 971d2ec..66d67ba 100644
--- a/Monitor.c
+++ b/Monitor.c
@@ -483,11 +483,17 @@ static int check_array(struct state *st, struct mdstat_ent *mdstat,
strncmp(buf,"inact",5) == 0) {
if (fd >= 0)
close(fd);
- if (!st->err)
- alert("DeviceDisappeared", dev, NULL, ainfo);
- st->err++;
- return 0;
+ fd = sysfs_open(st->devnm, NULL, "level");
+ if (fd < 0 || read(fd, buf, 10) != 0) {
+ if (fd >= 0)
+ close(fd);
+ if (!st->err)
+ alert("DeviceDisappeared", dev, NULL, ainfo);
+ st->err++;
+ return 0;
+ }
}
+ close(fd);
}
fd = open(dev, O_RDONLY);
if (fd < 0) {
--
2.1.4
^ permalink raw reply related
* Re: raid md126, md127 problem after reboot, howto fix?
From: Sebastian Parschauer @ 2015-02-09 9:55 UTC (permalink / raw)
To: Marc Widmer, linux-raid
In-Reply-To: <54D7B317.5010809@gmail.com>
On 08.02.2015 20:03, Marc Widmer wrote:
> Hi List
>
> I have no deep unterstand about raids, beside setting them up initially and
> replacing disks if needed. So this error has never happened to me before:
>
> After a reboot i have a really strange behaviour on my server. Disks are
> not marked faulty, but raid is "fallend apart".
>
> /proc/mdstat shows me:
>
> md126 : active raid1 sda1[0]
> 10485696 blocks [2/1] [U_]
>
> md127 : active raid1 sda2[0]
> 721558464 blocks [2/1] [U_]
>
> md1 : active raid1 sdb1[1]
> 10485696 blocks [2/1] [_U]
>
> md2 : active raid1 sdb2[1]
> 721558464 blocks [2/1] [_U]
>
> wished would be something similar to:
> md1 : active raid1 sdb1[1] sda1[0]
> 10238912 blocks [2/2] [UU]
>
> md2 : active raid1 sdb2[1] sda2[0]
> 1942746048 blocks [2/2] [UU]
>
> Currently only md1, md2 are running. nmon shows me, that only disks sdb is
> active, sda is not doing anything.
>
> I run debian squeeze.
>
> I am a bit concerned what to do, because at the moment i run on one disk
> only and if things go wrong i end up with a server not running (downtime)
> and possible data loss (beside backups).
>
> Any ideas what i should do? Howto put the raid back together, possibly in
> live mode, without rebooting in rescue mode and risk long downtime?
Just stop all arrays which aren't in use at the moment and assemble them
manually. If they are already running degraded, then add the now
unassociated disks to their respective running arrays.
$ mdadm --stop /dev/md126
$ mdadm --stop /dev/md127
$ mdadm /dev/md1 --add /dev/sda1
$ mdadm /dev/md2 --add /dev/sda2
This could require some syncing but then everything should be normal again.
Cheers,
Sebastian
^ permalink raw reply
* Re: raid md126, md127 problem after reboot, howto fix?
From: Sebastian Parschauer @ 2015-02-09 9:41 UTC (permalink / raw)
To: Wols Lists, Marc Widmer, linux-raid
In-Reply-To: <54D7D551.2030800@youngman.org.uk>
On 08.02.2015 22:29, Wols Lists wrote:
> On 08/02/15 19:03, Marc Widmer wrote:
>> Hi List
>>
>> I have no deep unterstand about raids, beside setting them up initially and
>> replacing disks if needed. So this error has never happened to me before:
>>
>> After a reboot i have a really strange behaviour on my server. Disks are
>> not marked faulty, but raid is "fallend apart".
>>
>> /proc/mdstat shows me:
>>
>> md126 : active raid1 sda1[0]
>> 10485696 blocks [2/1] [U_]
>>
>> md127 : active raid1 sda2[0]
>> 721558464 blocks [2/1] [U_]
>>
>> md1 : active raid1 sdb1[1]
>> 10485696 blocks [2/1] [_U]
>>
>> md2 : active raid1 sdb2[1]
>> 721558464 blocks [2/1] [_U]
>>
>> wished would be something similar to:
>> md1 : active raid1 sdb1[1] sda1[0]
>> 10238912 blocks [2/2] [UU]
>>
>> md2 : active raid1 sdb2[1] sda2[0]
>> 1942746048 blocks [2/2] [UU]
>>
>> Currently only md1, md2 are running. nmon shows me, that only disks sdb is
>> active, sda is not doing anything.
>>
>> I run debian squeeze.
>
> What version of mdadm are you running? 3.2.6 or thereabouts?
>>
>> I am a bit concerned what to do, because at the moment i run on one disk
>> only and if things go wrong i end up with a server not running (downtime)
>> and possible data loss (beside backups).
>>
>> Any ideas what i should do? Howto put the raid back together, possibly in
>> live mode, without rebooting in rescue mode and risk long downtime?
>>
>> Any help would be greatly appreciated as by now the only thing i had to do
>> was resyncing a disk after usual hd crash.
>>
> The reason I ask is this looks like a bug I had - if I'm right it's a
> known problem and you need to upgrade mdadm.
Yeah, pretty much sounds like the bad udev rules with Squeeze and old
mdadm. E.g. deactivation of the MD udev rules and assembling via init
scripts is a way to workaround this.
I have some test VMs providing MD RAID-1 on iSCSI targets and have seen
the same issue when logging in to the targets. Deactivation of the udev
rules and manual assembly helped.
Cheers,
Sebastian
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: Peter Zijlstra @ 2015-02-09 9:10 UTC (permalink / raw)
To: NeilBrown; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150209121357.29f19d36@notabene.brown>
On Mon, Feb 09, 2015 at 12:13:57PM +1100, NeilBrown wrote:
> I had to re-read the code (And your analysis) a couple of times to be sure ...
Sorry :-)
> However, when io_schedule() explicitly calls blk_flush_plug(), then
> @from_schedule=false variant is used, and the unplug functions are allowed to
> allocate memory and block and maybe even call mempool_alloc() which might
> call io_schedule().
>
> This shouldn't be a problem as blk_flush_plug() spliced out the plug list, so
> any recursive call will find an empty list and do nothing.
Unless, something along the way stuck something back on, right? So
should we stick an:
WARN_ON(current->in_iowait);
somewhere near where things are added to this plug list? (and move the
blk_flush_plug() call inside of where that's actually true of course).
> Worst case is that a wait_event loop that calls io_schedule() (i.e.
> wait_on_bit_io()) might not block in the first call to io_schedule()
> if the unplugging needed to wait. Every subsequent call will block as
> required as there is nothing else to add requests to the plug queue.
Again, assuming @cond will not actually stick something on this list.
Which if we add the above we'll get warned about.
> It isn't that scheduling is "rare" - it is that it can only occur once in a
> loop which doesn't expect it.
With the above WARN stuck in, agreed.
> So I propose the following, though I haven't tested it.
>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index e628cb11b560..b0f12ab3df23 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4374,6 +4374,11 @@ void __sched io_schedule(void)
>
> delayacct_blkio_start();
> atomic_inc(&rq->nr_iowait);
> + /* Any sleeping in blk_flush_plug() should not
> + * trigger the "do not call blocking ops" warning
> + * as it can only happen once in a wait_event loop.
> + */
Might I suggest the 'regular' multi-line comment style, and a reference
to the above WARN that makes everything actually work?
/*
* multi-line
* comments have an empty
* line at the start... As per CodingStyle ch. 8
*/
> + sched_annotate_sleep();
> blk_flush_plug(current);
Also, at this point, should we put it in blk_flush_plug()?
The only thing that really goes wrong then is if people 'forget' to put
a loop around io_schedule().
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: NeilBrown @ 2015-02-09 1:13 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150206113930.GK23123@twins.programming.kicks-ass.net>
[-- Attachment #1: Type: text/plain, Size: 4744 bytes --]
On Fri, 6 Feb 2015 12:39:30 +0100 Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, Feb 06, 2015 at 08:51:33AM +1100, NeilBrown wrote:
> > That is exactly what is happening here. However I don't think that is an
> > "observed problem" but rather an "observed false-positive".
> >
> > If nothing inside the outer loop blocks, then in particular
> > generic_make_request will not be called, so nothing will be added to the
> > queue that blk_schedule_flush_plug flushes.
> > So the first time through the loop, a call the 'schedule()' may not actually
> > block, but every subsequent time it will.
> > So there is no actual problem here.
> >
> > So I'd be included to add sched_annotate_sleep() in blk_flush_plug_list().
> >
> > Peter: what do you think is the best way to silence this warning.
>
> > > Call Trace:
>
> > > [<ffffffff8027ee62>] __might_sleep+0x82/0x90
> > > [<ffffffff803bee06>] generic_make_request_checks+0x36/0x2d0
> > > [<ffffffff803bf0b3>] generic_make_request+0x13/0x100
> > > [<ffffffff8054983b>] raid1_unplug+0x12b/0x170
> > > [<ffffffff803c1302>] blk_flush_plug_list+0xa2/0x230
> > > [<ffffffff80646383>] io_schedule+0x43/0x80
> > > [<ffffffff80646787>] bit_wait_io+0x27/0x50
>
> Well, I don't know. I don't particularly like the whole blk_flush_plug()
> thing scheduling while on its way to schedule. If you ever end up
> calling io_schedule() from it there's 'fun'.
>
> Also, how likely is it to actually schedule when doing all that? This
> block layer stuff is somewhat impenetrable for me, too many callbacks.
>
> You have some words on how its unlikely, but I can't even find _where_
> it would schedule :/ All I see is a loop calling ->make_request_fn() and
> god only knows where that ends up.
>
> So there appear to be two blk_flush_plug() variants, one with an
> @from_schedule = true, which seems to really try not to schedule, which
> seems to suggest the 'false' one (the one above) is meant to schedule?
>
> If scheduling is the rule rather than the exception, the above is
> properly broken.
>
> But again, I don't know.
I had to re-read the code (And your analysis) a couple of times to be sure ...
As you say, when schedule() calls blk_schedule_flush_plug(), the
@from_schedule=true variant is used and the unplug code doesn't block.
So there is no problem there.
However, when io_schedule() explicitly calls blk_flush_plug(), then
@from_schedule=false variant is used, and the unplug functions are allowed to
allocate memory and block and maybe even call mempool_alloc() which might
call io_schedule().
This shouldn't be a problem as blk_flush_plug() spliced out the plug list, so
any recursive call will find an empty list and do nothing.
Worst case is that a wait_event loop that calls io_schedule() (i.e.
wait_on_bit_io()) might not block in the first call to io_schedule()
if the unplugging needed to wait. Every subsequent call will block as
required as there is nothing else to add requests to the plug queue.
So as long as wait_on_bio_io() can cope with a single false wakeup (which it
can), there is no problem here.
>
> If you're confident that scheduling is rare for _ALL_ (current and
> future) block device implementations, not just the raid one, then you
> can annotate blk_flush_plug_list() I suppose.
>
> Otherwise I would suggest adding them one at a time in whatever blk
> device thing likes to go schedule on us. Also, add a comment that
> explains why its rare for the future us who need to look at it again.
It isn't that scheduling is "rare" - it is that it can only occur once in a
loop which doesn't expect it.
So I propose the following, though I haven't tested it.
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e628cb11b560..b0f12ab3df23 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4374,6 +4374,11 @@ void __sched io_schedule(void)
delayacct_blkio_start();
atomic_inc(&rq->nr_iowait);
+ /* Any sleeping in blk_flush_plug() should not
+ * trigger the "do not call blocking ops" warning
+ * as it can only happen once in a wait_event loop.
+ */
+ sched_annotate_sleep();
blk_flush_plug(current);
current->in_iowait = 1;
schedule();
@@ -4390,6 +4395,11 @@ long __sched io_schedule_timeout(long timeout)
delayacct_blkio_start();
atomic_inc(&rq->nr_iowait);
+ /* Any sleeping in blk_flush_plug() should not
+ * trigger the "do not call blocking ops" warning
+ * as it can only happen once in a wait_event loop.
+ */
+ sched_annotate_sleep();
blk_flush_plug(current);
current->in_iowait = 1;
ret = schedule_timeout(timeout);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox