* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-10 20:09 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux-RAID
In-Reply-To: <5669DB3B.30101@turmel.org>
On Thu, Dec 10, 2015 at 2:06 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/09/2015 08:34 PM, Dallas Clement wrote:
>> Hi all. I'm trying to determine best and worst case expected
>> sequential write speeds for Linux software RAID with spinning disks.
>>
>> I have been assuming on the following:
>>
>> Best case RAID 6 sequential write speed is (N-2) * X, where is is
>> number of drives and X is write speed of a single drive.
>>
>> Worst case RAID 6 sequential write speed is (N-2) * X / 2.
>>
>> Best case RAID 5 sequential write speed is (N-1) * X.
>>
>> Worst case RAID 5 sequential write speed is (N-1) * X / 2.
>>
>> Could someone please confirm whether these formulas are accurate or not?
>
> Confirm these? No. In fact, I see no theoretical basis for stating a
> worst case speed as half the best case speed. Or any other fraction.
> It's dependent on numerous variables -- block size, processor load, I/O
> bandwidth at various choke points (Northbridge, southbridge, PCI/PCIe,
> SATA/SAS channels, port mux...), I/O latency vs. queue depth vs. drive
> buffers, sector positioning at block boundaries, drive firmware
> housekeeping, etc.
>
> Where'd you get the worst case formulas?
>
> Where'd you get the worst case formulas?
Google search I'm afraid. I think the assumption for RAID 5,6 worst
case is having to read and write the parity + data every cycle.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Phil Turmel @ 2015-12-10 20:29 UTC (permalink / raw)
To: Dallas Clement; +Cc: Linux-RAID
In-Reply-To: <CAE9DZUSNkLxaKH8MtggW+5_uo38CHRTcscvh7Kk-VTb_zWPQ4g@mail.gmail.com>
On 12/10/2015 03:09 PM, Dallas Clement wrote:
> On Thu, Dec 10, 2015 at 2:06 PM, Phil Turmel <philip@turmel.org> wrote:
>> Where'd you get the worst case formulas?
>
> Google search I'm afraid. I think the assumption for RAID 5,6 worst
> case is having to read and write the parity + data every cycle.
Well, it'd be a lot worse than half, then. To use the shortcut in raid5
to write one block, you have to read it first, read the parity, compute
the change in parity, then write the block with the new parity. That's
two reads and two writes for a single upper level write. For raid6, add
read and write of the Q syndrome, assuming you have a kernel new enough
to do the raid6 shortcut at all. Three reads and three writes for a
single upper level write. In both cases, add rotational latency to
reposition for writing over sectors just read.
Those RMW operations generally happen to small random writes, which
makes the assertion for sequential writes odd. Unless you delay writes
or misalign or inhibit merging, RMW won't trigger except possibly at the
beginning or end of a stream.
That's why I questioned O_SYNC when you were using a filesystem: it
prevents merging, and forces seeking to do small metadata writes.
Basically turning a sequential workload into a random one.
Phil
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-10 21:14 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux-RAID
In-Reply-To: <5669E091.1010108@turmel.org>
On Thu, Dec 10, 2015 at 2:29 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/10/2015 03:09 PM, Dallas Clement wrote:
>> On Thu, Dec 10, 2015 at 2:06 PM, Phil Turmel <philip@turmel.org> wrote:
>
>>> Where'd you get the worst case formulas?
>>
>> Google search I'm afraid. I think the assumption for RAID 5,6 worst
>> case is having to read and write the parity + data every cycle.
>
> Well, it'd be a lot worse than half, then. To use the shortcut in raid5
> to write one block, you have to read it first, read the parity, compute
> the change in parity, then write the block with the new parity. That's
> two reads and two writes for a single upper level write. For raid6, add
> read and write of the Q syndrome, assuming you have a kernel new enough
> to do the raid6 shortcut at all. Three reads and three writes for a
> single upper level write. In both cases, add rotational latency to
> reposition for writing over sectors just read.
>
> Those RMW operations generally happen to small random writes, which
> makes the assertion for sequential writes odd. Unless you delay writes
> or misalign or inhibit merging, RMW won't trigger except possibly at the
> beginning or end of a stream.
>
> That's why I questioned O_SYNC when you were using a filesystem: it
> prevents merging, and forces seeking to do small metadata writes.
> Basically turning a sequential workload into a random one.
>
> Phil
> Those RMW operations generally happen to small random writes, which
> makes the assertion for sequential writes odd.
Exactly. I'm not expecting RMWs to be happening for large sequential
writes. But yet my RAID 5, 6 sequential write performance is still
very poor. As mentioned earlier, I'm getting around 95 MB/s on the
inner side of these disks. With 12 of them, my RAID 6 write speed
should be (12 - 2) * 95 = 950 MB/s. I'm getting about 300 MB/s less
than that for this scenario. I have the disks split up among three
different controllers. There should be plenty of bandwidth. Several
days ago I ran fio on each of the 12 disks concurrently. I was able
to see the disks at or near 100% utilization and wMB/s around 160-170
MB/s. That's why I started focusing on RAID as being the potential
bottleneck.
> That's why I questioned O_SYNC when you were using a filesystem: it
> prevents merging, and forces seeking to do small metadata writes.
> Basically turning a sequential workload into a random one.
Yes, that certainly makes sense. Not using O_SYNC anymore. Just O_DIRECT.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Phil Turmel @ 2015-12-10 21:32 UTC (permalink / raw)
To: Dallas Clement; +Cc: Linux-RAID
In-Reply-To: <CAE9DZUTHZ0+VkAQ8ixvH9m=6XXXShDwJp6_8SeEXovW+hF6OdQ@mail.gmail.com>
On 12/10/2015 04:14 PM, Dallas Clement wrote:
> Exactly. I'm not expecting RMWs to be happening for large sequential
> writes. But yet my RAID 5, 6 sequential write performance is still
> very poor. As mentioned earlier, I'm getting around 95 MB/s on the
> inner side of these disks. With 12 of them, my RAID 6 write speed
> should be (12 - 2) * 95 = 950 MB/s. I'm getting about 300 MB/s less
> than that for this scenario. I have the disks split up among three
> different controllers. There should be plenty of bandwidth. Several
> days ago I ran fio on each of the 12 disks concurrently. I was able
> to see the disks at or near 100% utilization and wMB/s around 160-170
> MB/s. That's why I started focusing on RAID as being the potential
> bottleneck.
>
>> That's why I questioned O_SYNC when you were using a filesystem: it
>> prevents merging, and forces seeking to do small metadata writes.
>> Basically turning a sequential workload into a random one.
>
> Yes, that certainly makes sense. Not using O_SYNC anymore. Just O_DIRECT.
Sounds like its time to break out blktrace to see what's really
happening between your array and its member devices.
With diffs from old kernels to new.
Phil
^ permalink raw reply
* Re: [PATCH 0/3] mdadm: re-add journal to array with bad journal
From: NeilBrown @ 2015-12-10 22:05 UTC (permalink / raw)
To: linux-raid; +Cc: dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1449018805-320020-1-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]
On Wed, Dec 02 2015, Song Liu <songliubraving@fb.com> wrote:
> These patches try to recreate missing/faulty journal in mdadm. Since
> the array is readonly with journal missing, it is necessary to
> restart the array after recreating journal.
>
> Song Liu (3):
> add sysfs_array_state to struct mdinfo
> [mdadm] recreate journal in mdadm
> make .spec file work
I didn't get that third patch???
The patches generally look good, but it bothers me that you have to stop
and restart the array for the added journal to work. Can we just teach
the kernel to accept a new journal on an active array, and then use that
functionality in mdadm?
That is certainly what I want in the longer term and I'm not sure it
makes sense to have a half-way "--add-journal" that doesn't do the whole
job.
Thanks,
NeilBrown
>
> Assemble.c | 8 +++++++-
> Manage.c | 32 +++++++++++++++++++++++++++++++-
> ReadMe.c | 3 ++-
> mdadm.c | 8 ++++++++
> mdadm.h | 5 +++++
> mdadm.spec | 8 ++++----
> super1.c | 3 ++-
> sysfs.c | 7 +++++++
> udev-md-raid-assembly.rules | 2 +-
> 9 files changed, 67 insertions(+), 9 deletions(-)
>
> --
> 2.4.6
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: RAID 6 "Failed to restore critical section for reshape, sorry." - recovery advice?
From: George Rapp @ 2015-12-10 22:05 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1512100918520.20919@uplift.swm.pp.se>
On Thu, Dec 10, 2015 at 3:22 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Wed, 9 Dec 2015, George Rapp wrote:
>
>> # mdadm --assemble --verbose --invalid-backup --force
>> --update=revert-reshape /dev/md4 $OVERLAYS
>> mdadm: looking for devices for /dev/md4
>> [snipped remainder of output]
>
> Just a shot in the dark, what happens if you add a backup file to the above
> command, but without revert-reshape? Ie state both --invalid-backup but also
> supply a backup-file. The text above could indicate that this might help.
Mikael -
First, thanks for the reply.
I tried that and got a different error message:
# mdadm --assemble --verbose --invalid-backup --force
--backup-file=/home/gwr/2015/2015-08/grow_md6.bak /dev/md4 $OVERLAYS
mdadm: looking for devices for /dev/md4
mdadm: /dev/mapper/sdd4 is identified as a member of /dev/md4, slot 0.
mdadm: /dev/mapper/sdc4 is identified as a member of /dev/md4, slot 1.
mdadm: /dev/mapper/sdg4 is identified as a member of /dev/md4, slot 2.
mdadm: /dev/mapper/sdh1 is identified as a member of /dev/md4, slot 3.
mdadm: /dev/mapper/sdi1 is identified as a member of /dev/md4, slot 5.
mdadm: /dev/mapper/sdj1 is identified as a member of /dev/md4, slot 4.
mdadm: /dev/md4 has an active reshape - checking if critical section
needs to be restored
mdadm: No backup metadata on /home/gwr/2015/2015-08/grow_md6.bak
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/mapper/sdc4 to /dev/md4 as 1
mdadm: added /dev/mapper/sdg4 to /dev/md4 as 2
mdadm: added /dev/mapper/sdh1 to /dev/md4 as 3
mdadm: added /dev/mapper/sdj1 to /dev/md4 as 4
mdadm: added /dev/mapper/sdi1 to /dev/md4 as 5
mdadm: added /dev/mapper/sdd4 to /dev/md4 as 0
mdadm: failed to RUN_ARRAY /dev/md4: Invalid argument
> If you get the array up and running again but it's still reshaping but at
> position 0, issue a --continue to it (this has worked for others). Also, I
> would get the latest git version of mdadm and try with that one if you're
> still using v3.2.2 as per the link to your august original email.
Should have updated that. I'm up to mdadm v3.2.4 now, which is the
latest version offered by the Fedora 22 update repo (and Fedora 23
doesn't offer a newer one):
# mdadm --version
mdadm - v3.3.4 - 3rd August 2015
Thanks again.
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Wols Lists @ 2015-12-10 22:19 UTC (permalink / raw)
To: John Stoffel, Dallas Clement; +Cc: Mark Knecht, Linux-RAID
In-Reply-To: <22121.54145.780249.40226@quad.stoffel.home>
On 10/12/15 19:33, John Stoffel wrote:
> Now it's simpler to just A) buy lots and lots of memory, B) bunches of
> SSDs, C) both, D) beat the developers until they learn to write better
> SQL.
>
> Sorry, D) never happens. :-)
Don't get me started ... :-)
Relational first normal form CANNOT be efficient, and it starts with the
first law - "data comes in rows and columns". In the real world, that's
not true, and the rot just gets worse from there ...
Cheers,
Wol
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Wols Lists @ 2015-12-10 22:23 UTC (permalink / raw)
To: John Stoffel, Dallas Clement; +Cc: Linux-RAID
In-Reply-To: <22121.53825.857640.201383@quad.stoffel.home>
On 10/12/15 19:28, John Stoffel wrote:
> At this point, you need to now look at your controllers and
> motherboard and how they're configured. If all those drives are on
> one controller, and if that controller is on a single lane of PCIe,
> then you will see controller bandwidth issues as well.
Very out of date ... we're talking the days of CD writers here ... but
we used a pc to write bulk CDs, and we found we simply had to buy add-in
IDE cards to get decent performance - two CD drives on the same PATA
cable and response went through the floor - massively so.
Cheers,
Wol
^ permalink raw reply
* Re: RAID 6 "Failed to restore critical section for reshape, sorry." - recovery advice?
From: George Rapp @ 2015-12-10 22:34 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <CAF-KpgaLr-qA_RJiDYstFv-=EoHH-Ea3o1q3G3tXC4nX=Wt2XA@mail.gmail.com>
On Thu, Dec 10, 2015 at 5:05 PM, George Rapp <george.rapp@gmail.com> wrote:
> On Thu, Dec 10, 2015 at 3:22 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> Just a shot in the dark, what happens if you add a backup file to the above
>> command, but without revert-reshape? Ie state both --invalid-backup but also
>> supply a backup-file. The text above could indicate that this might help.
>
> Mikael -
>
> First, thanks for the reply.
>
> I tried that and got a different error message:
>
> # mdadm --assemble --verbose --invalid-backup --force
> --backup-file=/home/gwr/2015/2015-08/grow_md6.bak /dev/md4 $OVERLAYS
> mdadm: looking for devices for /dev/md4
> mdadm: /dev/mapper/sdd4 is identified as a member of /dev/md4, slot 0.
> mdadm: /dev/mapper/sdc4 is identified as a member of /dev/md4, slot 1.
> mdadm: /dev/mapper/sdg4 is identified as a member of /dev/md4, slot 2.
> mdadm: /dev/mapper/sdh1 is identified as a member of /dev/md4, slot 3.
> mdadm: /dev/mapper/sdi1 is identified as a member of /dev/md4, slot 5.
> mdadm: /dev/mapper/sdj1 is identified as a member of /dev/md4, slot 4.
> mdadm: /dev/md4 has an active reshape - checking if critical section
> needs to be restored
> mdadm: No backup metadata on /home/gwr/2015/2015-08/grow_md6.bak
> mdadm: Failed to find backup of critical section
> mdadm: continuing without restoring backup
> mdadm: added /dev/mapper/sdc4 to /dev/md4 as 1
> mdadm: added /dev/mapper/sdg4 to /dev/md4 as 2
> mdadm: added /dev/mapper/sdh1 to /dev/md4 as 3
> mdadm: added /dev/mapper/sdj1 to /dev/md4 as 4
> mdadm: added /dev/mapper/sdi1 to /dev/md4 as 5
> mdadm: added /dev/mapper/sdd4 to /dev/md4 as 0
> mdadm: failed to RUN_ARRAY /dev/md4: Invalid argument
Forgot to include the contents of the system log:
[ 928.679299] md: bind<dm-1>
[ 928.679809] md: bind<dm-2>
[ 928.681957] md: bind<dm-3>
[ 928.693345] md: bind<dm-5>
[ 928.694155] md: bind<dm-4>
[ 928.696251] md: bind<dm-0>
[ 928.709133] md/raid:md4: reshape_position too early for
auto-recovery - aborting.
[ 928.709159] md: pers->run() failed ...
[ 928.709425] md: md4 stopped.
[ 928.709442] md: unbind<dm-0>
[ 928.709449] md: export_rdev(dm-0)
[ 928.709462] md: unbind<dm-4>
[ 928.709468] md: export_rdev(dm-4)
[ 928.709477] md: unbind<dm-5>
[ 928.709483] md: export_rdev(dm-5)
[ 928.709493] md: unbind<dm-3>
[ 928.709499] md: export_rdev(dm-3)
[ 928.709510] md: unbind<dm-2>
[ 928.709515] md: export_rdev(dm-2)
[ 928.709524] md: unbind<dm-1>
[ 928.709529] md: export_rdev(dm-1)
[ 928.831905] md: bind<dm-0>
[ 928.859783] md: bind<dm-3>
[ 928.864100] md: bind<dm-1>
[ 928.872128] md: bind<dm-4>
[ 928.878222] md: bind<dm-5>
[ 928.886799] md: bind<dm-2>
I appear to be too early in the reshape for auto-recovery, but too far
along to just say "never mind on that whole reshape business". Any
other thoughts?
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* RE: [PATCH 0/3] mdadm: re-add journal to array with bad journal
From: Song Liu @ 2015-12-10 22:37 UTC (permalink / raw)
To: NeilBrown, linux-raid@vger.kernel.org
Cc: dan.j.williams@intel.com, Shaohua Li, hch@infradead.org,
Kernel Team
In-Reply-To: <87si3ac62s.fsf@notabene.neil.brown.name>
> -----Original Message-----
> From: NeilBrown [mailto:neilb@suse.com]
> Sent: Thursday, December 10, 2015 2:06 PM
> To: Song Liu <songliubraving@fb.com>; linux-raid@vger.kernel.org
> Cc: dan.j.williams@intel.com; Shaohua Li <shli@fb.com>; hch@infradead.org;
> Kernel Team <Kernel-team@fb.com>; Song Liu <songliubraving@fb.com>
> Subject: Re: [PATCH 0/3] mdadm: re-add journal to array with bad journal
>
> On Wed, Dec 02 2015, Song Liu <songliubraving@fb.com> wrote:
>
> > These patches try to recreate missing/faulty journal in mdadm. Since
> > the array is readonly with journal missing, it is necessary to restart
> > the array after recreating journal.
> >
> > Song Liu (3):
> > add sysfs_array_state to struct mdinfo
> > [mdadm] recreate journal in mdadm
> > make .spec file work
>
> I didn't get that third patch???
That's my local hack to .spec file. Didn't mean to send it.
>
> The patches generally look good, but it bothers me that you have to stop and
> restart the array for the added journal to work. Can we just teach the kernel to
> accept a new journal on an active array, and then use that functionality in
> mdadm?
> That is certainly what I want in the longer term and I'm not sure it makes sense
> to have a half-way "--add-journal" that doesn't do the whole job.
We will look into hot_add option.
Thanks,
Song
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-10 22:44 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <5669AE9C.7040904@turmel.org>
>
> Order of device discovery is not guaranteed, but base devices will
> almost always show up before multipath devices. You have to filter out
> the base devices from mdadm consideration:
>
> Add a DEVICES statement to your mdadm.conf that only matches your
> multipath device names, not the base names. Then update your initramfs
> so it applies to early boot as well.
>
This is something which I actually already tried. I specified in my
/etc/mdadm/mdadm.conf following:
DEVICE /dev/mapper/hgst-ssd*
In my /etc/multipath.conf I instruct dm-multipath to create
/dev/mapper/hgst-ssd1 /dev/mapper/hgst-ssd2 /dev/mapper/hgst-ssd3
But the only result is that after this config, the md raid is not
started at all. My conclusion was that when md raid was starting, the
multipath devices did not yet exist so it did not start up the array.
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-10 22:50 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <22121.43026.985668.789870@quad.stoffel.home>
> But really, you need to provide details. Can you setup a small test
> MD RAID and show all the details?
Sure, thanks, I will provide all the required information tomorrow
when I am back in office
Remek
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: Shaohua Li @ 2015-12-10 23:40 UTC (permalink / raw)
To: NeilBrown; +Cc: Christoph Hellwig, linux-raid
In-Reply-To: <87vb88p1tm.fsf@notabene.neil.brown.name>
On Wed, Dec 09, 2015 at 05:34:45PM +1100, NeilBrown wrote:
> On Wed, Dec 09 2015, Shaohua Li wrote:
>
> > On Wed, Dec 09, 2015 at 11:36:30AM +1100, NeilBrown wrote:
> >> On Thu, Dec 03 2015, Christoph Hellwig wrote:
> >>
> >> > Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
> >> >
> >> > I've looked into replacing these with mempools and biosets, and for the
> >> > bio and the meta_page that's pretty trivial as they have short life times
> >> > and do make guaranteed progress. I'm massively struggling with the iounit
> >> > allocation, though. These can live on for a long time over log I/O, cache
> >> > flushing and last but not least RAID I/O, and every attempt at something
> >> > mempool-like results in reproducible deadlocks. I wonder if we need to
> >> > figure out some more efficient data structure to communicate the completion
> >> > status that doesn't rely on these fairly long living allocations from
> >> > the I/O path.
> >>
> >> Presumably the root cause of these deadlocks is that the raid5d thread
> >> has called
> >> handle_stripe -> ops_run_io ->r5l_write_stripe -> r5l_log_stripe
> >> -> r5l_get_meta -> r5l_new_meta
> >>
> >> and r5l_new_meta is blocked on memory allocation, which won't complete
> >> until some raid5 stripes get written out, which requires raid5d to do
> >> something more useful than sitting and waiting.
> >>
> >> I suspect a good direction towards a solution would be to allow the
> >> memory allocation to fail, to cleanly propagate that failure indication
> >> up through r5l_log_stripe to r5l_write_stripe which falls back to adding
> >> the stripe_head to ->no_space_stripes.
> >>
> >> Then we only release stripes from no_space_stripes when a memory
> >> allocation might succeed.
> >>
> >> There are lots of missing details, and possibly we would need a separate
> >> list rather than re-using no_space_stripes.
> >> But the key idea is that raid5d should never block (except beneath
> >> submit_bio on some other device) and when it cannot make progress
> >> without blocking, it should queue the stripe_head for later handling.
> >>
> >> Does that make sense?
> >
> > It does remove the scary __GFP_NOFAIL, but the approach is essentially
> > idential to a 'retry after allocation failure'. Why not just let the mm
> > (with __GFP_NOFAIL) to do the retry then?
> >
>
> Because deadlocks.
>
> If raid5d is waiting for the mm to allocated memory, then it cannot
> retire write requests which could free up memory.
Ok, I missed the deadlock issue. Looks raid5d can't sleep waitting for
memory. That means the mempool for metadata page/bio doesn't work too,
as raid5d might block and not dispatch previous IO. Your proposal looks
reasonable.
Would keeping NOFAIL and having a separate thread got r5l_log_stripe
work too?
Thanks,
Shaohua
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-11 0:02 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, Linux-RAID
In-Reply-To: <CAK2H+ednN7dCGzcOt8TxgNdhdDA1mN6Xr5P8vQ+Y=-uRoxRksw@mail.gmail.com>
On Thu, Dec 10, 2015 at 5:04 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Thu, Dec 10, 2015 at 12:09 PM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
>>
>> On Thu, Dec 10, 2015 at 2:06 PM, Phil Turmel <philip@turmel.org> wrote:
>> <SNIP>
>> >>
>> >> Could someone please confirm whether these formulas are accurate or
>> >> not?
>> >
>> > Confirm these? No. In fact, I see no theoretical basis for stating a
>> > worst case speed as half the best case speed. Or any other fraction.
>> > It's dependent on numerous variables -- block size, processor load, I/O
>> > bandwidth at various choke points (Northbridge, southbridge, PCI/PCIe,
>> > SATA/SAS channels, port mux...), I/O latency vs. queue depth vs. drive
>> > buffers, sector positioning at block boundaries, drive firmware
>> > housekeeping, etc.
>> >
>> > Where'd you get the worst case formulas?
>> >
>>
>> > Where'd you get the worst case formulas?
>>
>> Google search I'm afraid. I think the assumption for RAID 5,6 worst
>> case is having to read and write the parity + data every cycle.
>
> What sustained throughput do you get in this system if you skip RAID, set
> up a script and write different data to all 12 drives in parallel? I don't
> think
> you've addressed Phil's comment concerning all the other potential choke
> points in the system. You'd need to be careful and make sure all the data
> is really out to disk but it might tell you something about your assumptions
> vs what the hardware is really doing..
>
> - Mark
Hi Mark,
> What sustained throughput do you get in this system if you skip RAID, set
> up a script and write different data to all 12 drives in parallel?
Just tried this again, running fio concurrently on all 12 disks. This
time doing sequential writes, bs=2048k, direct=1 to the raw disk
device - no filesystem. The results are not encouraging. I tried to
watch the disk behavior with iostat. This 8 core xeon system was
really getting crushed. The load average during the 10 minute test
was 15.16 26.41 21.53. iostat showed %iowait varying between 40%
and 80%. Also iostat showed only about 8 of the 12 disks on average
getting CPU time. They had high near 100% utilization and pretty good
write speed ~160 - 170 MB/s. Looks like my disks are just too slow
and the CPU cores are stuck waiting for them.
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: NeilBrown @ 2015-12-11 0:09 UTC (permalink / raw)
To: Shaohua Li; +Cc: Christoph Hellwig, linux-raid
In-Reply-To: <20151210234047.GA3596673@devbig084.prn1.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 4261 bytes --]
On Fri, Dec 11 2015, Shaohua Li wrote:
> On Wed, Dec 09, 2015 at 05:34:45PM +1100, NeilBrown wrote:
>> On Wed, Dec 09 2015, Shaohua Li wrote:
>>
>> > On Wed, Dec 09, 2015 at 11:36:30AM +1100, NeilBrown wrote:
>> >> On Thu, Dec 03 2015, Christoph Hellwig wrote:
>> >>
>> >> > Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
>> >> >
>> >> > I've looked into replacing these with mempools and biosets, and for the
>> >> > bio and the meta_page that's pretty trivial as they have short life times
>> >> > and do make guaranteed progress. I'm massively struggling with the iounit
>> >> > allocation, though. These can live on for a long time over log I/O, cache
>> >> > flushing and last but not least RAID I/O, and every attempt at something
>> >> > mempool-like results in reproducible deadlocks. I wonder if we need to
>> >> > figure out some more efficient data structure to communicate the completion
>> >> > status that doesn't rely on these fairly long living allocations from
>> >> > the I/O path.
>> >>
>> >> Presumably the root cause of these deadlocks is that the raid5d thread
>> >> has called
>> >> handle_stripe -> ops_run_io ->r5l_write_stripe -> r5l_log_stripe
>> >> -> r5l_get_meta -> r5l_new_meta
>> >>
>> >> and r5l_new_meta is blocked on memory allocation, which won't complete
>> >> until some raid5 stripes get written out, which requires raid5d to do
>> >> something more useful than sitting and waiting.
>> >>
>> >> I suspect a good direction towards a solution would be to allow the
>> >> memory allocation to fail, to cleanly propagate that failure indication
>> >> up through r5l_log_stripe to r5l_write_stripe which falls back to adding
>> >> the stripe_head to ->no_space_stripes.
>> >>
>> >> Then we only release stripes from no_space_stripes when a memory
>> >> allocation might succeed.
>> >>
>> >> There are lots of missing details, and possibly we would need a separate
>> >> list rather than re-using no_space_stripes.
>> >> But the key idea is that raid5d should never block (except beneath
>> >> submit_bio on some other device) and when it cannot make progress
>> >> without blocking, it should queue the stripe_head for later handling.
>> >>
>> >> Does that make sense?
>> >
>> > It does remove the scary __GFP_NOFAIL, but the approach is essentially
>> > idential to a 'retry after allocation failure'. Why not just let the mm
>> > (with __GFP_NOFAIL) to do the retry then?
>> >
>>
>> Because deadlocks.
>>
>> If raid5d is waiting for the mm to allocated memory, then it cannot
>> retire write requests which could free up memory.
>
> Ok, I missed the deadlock issue. Looks raid5d can't sleep waitting for
> memory. That means the mempool for metadata page/bio doesn't work too,
> as raid5d might block and not dispatch previous IO. Your proposal looks
> reasonable.
raid5d mustn't sleep waiting for something that only raid5d can provide.
A bio allocated from a mempool will be returned to that mempool when the
IO request completes (if the end_io routine calls bio_put, which it
does).
So when waiting on a mempool for a bio, you are only waiting for some
underlying device to perform IO, you are not waiting for anything that
raid5d might do. So that is safe.
Similarly the meta_page is freed in the end_io function, so it is safe
to wait for that.
Note that the important feature of a mempool is not so much that it
pre-allocates some memory. The important point is that when that
pre-allocated memory is freed it can *only* be used by the owner of the
mempool.
By contrast memory allocated with NOFAIL, when freed might be used by
any other caller anywhere in the kernel.
So waiting on a mempool means waiting for certain well defined events.
Waiting in NOFAIL could end up waiting for almost anything. So it is
much harder to reason about deadlocks.
The bio and the meta_page contrast with the r5l_io_unit which isn't
freed by the first end_io, but remains until raid5d has done some more
work. So raid5d cannot wait for a r5l_io_unit to be freed.
NeilBrown
>
> Would keeping NOFAIL and having a separate thread got r5l_log_stripe
> work too?
>
> Thanks,
> Shaohua
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-11 0:41 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, Linux-RAID
In-Reply-To: <CAK2H+efF2dM1BsM7kzfTxMdQEHvbWRaVe7zJLTGcPZzafn2M6A@mail.gmail.com>
On Thu, Dec 10, 2015 at 6:22 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Thu, Dec 10, 2015 at 4:02 PM, Dallas Clement <dallas.a.clement@gmail.com>
> wrote:
>>
>> On Thu, Dec 10, 2015 at 5:04 PM, Mark Knecht <markknecht@gmail.com> wrote:
> <SNIP>
>>
>> Hi Mark,
>>
>> > What sustained throughput do you get in this system if you skip RAID,
>> > set
>> > up a script and write different data to all 12 drives in parallel?
>>
>> Just tried this again, running fio concurrently on all 12 disks. This
>> time doing sequential writes, bs=2048k, direct=1 to the raw disk
>> device - no filesystem. The results are not encouraging. I tried to
>> watch the disk behavior with iostat. This 8 core xeon system was
>> really getting crushed. The load average during the 10 minute test
>> was 15.16 26.41 21.53. iostat showed %iowait varying between 40%
>> and 80%. Also iostat showed only about 8 of the 12 disks on average
>> getting CPU time. They had high near 100% utilization and pretty good
>> write speed ~160 - 170 MB/s. Looks like my disks are just too slow
>> and the CPU cores are stuck waiting for them.
>
> Well, it was hard on the system but it might not be a total loss. I'm not
> saying this is a good test but it might give you some ideas about how to
> proceed. Fewer drives? Better controller?
>
> Was it any different at the front and back of the drive?
>
> One thing I didn't see in this thread was a check to make sure your
> alignment is on the physical sector alignment if you're using 4K sectors
> which I assume drives this large are using.
>
> Anyway, data is just data. It gives you something to think about.
>
> Good luck,
> Mark
Hi Mark. Perhaps this is normal behavior when there are more disks to
be served than there are CPUs. But it surely does seem like a waste
for the CPUs to be locked up in uninterruptible sleep waiting for I/O
on these disks. I presume this is caused by threads in the kernel
tied up in spin loops waiting for I/O. It would sure be nice if the
I/O could be handled in a more asynchronous way so that these CPUs can
go off and do other things while they are waiting for I/Os to complete
on slow disks.
> Was it any different at the front and back of the drive?
Didn't try on this particular test.
> One thing I didn't see in this thread was a check to make sure your
> alignment is on the physical sector alignment if you're using 4K sectors
> which I assume drives this large are using.
Yes, these drives surely use 4K sectors. But I haven't checked for
sector alignment issues. Any tips on how to do that?
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: Shaohua Li @ 2015-12-11 1:10 UTC (permalink / raw)
To: NeilBrown; +Cc: Christoph Hellwig, linux-raid
In-Reply-To: <87poyddew9.fsf@notabene.neil.brown.name>
On Fri, Dec 11, 2015 at 11:09:58AM +1100, NeilBrown wrote:
> On Fri, Dec 11 2015, Shaohua Li wrote:
>
> > On Wed, Dec 09, 2015 at 05:34:45PM +1100, NeilBrown wrote:
> >> On Wed, Dec 09 2015, Shaohua Li wrote:
> >>
> >> > On Wed, Dec 09, 2015 at 11:36:30AM +1100, NeilBrown wrote:
> >> >> On Thu, Dec 03 2015, Christoph Hellwig wrote:
> >> >>
> >> >> > Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
> >> >> >
> >> >> > I've looked into replacing these with mempools and biosets, and for the
> >> >> > bio and the meta_page that's pretty trivial as they have short life times
> >> >> > and do make guaranteed progress. I'm massively struggling with the iounit
> >> >> > allocation, though. These can live on for a long time over log I/O, cache
> >> >> > flushing and last but not least RAID I/O, and every attempt at something
> >> >> > mempool-like results in reproducible deadlocks. I wonder if we need to
> >> >> > figure out some more efficient data structure to communicate the completion
> >> >> > status that doesn't rely on these fairly long living allocations from
> >> >> > the I/O path.
> >> >>
> >> >> Presumably the root cause of these deadlocks is that the raid5d thread
> >> >> has called
> >> >> handle_stripe -> ops_run_io ->r5l_write_stripe -> r5l_log_stripe
> >> >> -> r5l_get_meta -> r5l_new_meta
> >> >>
> >> >> and r5l_new_meta is blocked on memory allocation, which won't complete
> >> >> until some raid5 stripes get written out, which requires raid5d to do
> >> >> something more useful than sitting and waiting.
> >> >>
> >> >> I suspect a good direction towards a solution would be to allow the
> >> >> memory allocation to fail, to cleanly propagate that failure indication
> >> >> up through r5l_log_stripe to r5l_write_stripe which falls back to adding
> >> >> the stripe_head to ->no_space_stripes.
> >> >>
> >> >> Then we only release stripes from no_space_stripes when a memory
> >> >> allocation might succeed.
> >> >>
> >> >> There are lots of missing details, and possibly we would need a separate
> >> >> list rather than re-using no_space_stripes.
> >> >> But the key idea is that raid5d should never block (except beneath
> >> >> submit_bio on some other device) and when it cannot make progress
> >> >> without blocking, it should queue the stripe_head for later handling.
> >> >>
> >> >> Does that make sense?
> >> >
> >> > It does remove the scary __GFP_NOFAIL, but the approach is essentially
> >> > idential to a 'retry after allocation failure'. Why not just let the mm
> >> > (with __GFP_NOFAIL) to do the retry then?
> >> >
> >>
> >> Because deadlocks.
> >>
> >> If raid5d is waiting for the mm to allocated memory, then it cannot
> >> retire write requests which could free up memory.
> >
> > Ok, I missed the deadlock issue. Looks raid5d can't sleep waitting for
> > memory. That means the mempool for metadata page/bio doesn't work too,
> > as raid5d might block and not dispatch previous IO. Your proposal looks
> > reasonable.
>
> raid5d mustn't sleep waiting for something that only raid5d can provide.
>
> A bio allocated from a mempool will be returned to that mempool when the
> IO request completes (if the end_io routine calls bio_put, which it
> does).
> So when waiting on a mempool for a bio, you are only waiting for some
> underlying device to perform IO, you are not waiting for anything that
> raid5d might do. So that is safe.
>
> Similarly the meta_page is freed in the end_io function, so it is safe
> to wait for that.
> Note that the important feature of a mempool is not so much that it
> pre-allocates some memory. The important point is that when that
> pre-allocated memory is freed it can *only* be used by the owner of the
> mempool.
> By contrast memory allocated with NOFAIL, when freed might be used by
> any other caller anywhere in the kernel.
> So waiting on a mempool means waiting for certain well defined events.
> Waiting in NOFAIL could end up waiting for almost anything. So it is
> much harder to reason about deadlocks.
>
> The bio and the meta_page contrast with the r5l_io_unit which isn't
> freed by the first end_io, but remains until raid5d has done some more
> work. So raid5d cannot wait for a r5l_io_unit to be freed.
I'm confused 2 bios are already enough to avoid deadlock, sorry. Can you
peek Christoph's patches? I'll work on a patch for r5l_io_unit.
Thanks,
Shaohua
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-11 1:19 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, Linux-RAID
In-Reply-To: <CAE9DZUQ+LOFWNQ2MpKoSx8j8RHVqkL15PO+jVjs7EkCQykG6VA@mail.gmail.com>
On Thu, Dec 10, 2015 at 6:41 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Thu, Dec 10, 2015 at 6:22 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>
>>
>> On Thu, Dec 10, 2015 at 4:02 PM, Dallas Clement <dallas.a.clement@gmail.com>
>> wrote:
>>>
>>> On Thu, Dec 10, 2015 at 5:04 PM, Mark Knecht <markknecht@gmail.com> wrote:
>> <SNIP>
>>>
>>> Hi Mark,
>>>
>>> > What sustained throughput do you get in this system if you skip RAID,
>>> > set
>>> > up a script and write different data to all 12 drives in parallel?
>>>
>>> Just tried this again, running fio concurrently on all 12 disks. This
>>> time doing sequential writes, bs=2048k, direct=1 to the raw disk
>>> device - no filesystem. The results are not encouraging. I tried to
>>> watch the disk behavior with iostat. This 8 core xeon system was
>>> really getting crushed. The load average during the 10 minute test
>>> was 15.16 26.41 21.53. iostat showed %iowait varying between 40%
>>> and 80%. Also iostat showed only about 8 of the 12 disks on average
>>> getting CPU time. They had high near 100% utilization and pretty good
>>> write speed ~160 - 170 MB/s. Looks like my disks are just too slow
>>> and the CPU cores are stuck waiting for them.
>>
>> Well, it was hard on the system but it might not be a total loss. I'm not
>> saying this is a good test but it might give you some ideas about how to
>> proceed. Fewer drives? Better controller?
>>
>> Was it any different at the front and back of the drive?
>>
>> One thing I didn't see in this thread was a check to make sure your
>> alignment is on the physical sector alignment if you're using 4K sectors
>> which I assume drives this large are using.
>>
>> Anyway, data is just data. It gives you something to think about.
>>
>> Good luck,
>> Mark
>
> Hi Mark. Perhaps this is normal behavior when there are more disks to
> be served than there are CPUs. But it surely does seem like a waste
> for the CPUs to be locked up in uninterruptible sleep waiting for I/O
> on these disks. I presume this is caused by threads in the kernel
> tied up in spin loops waiting for I/O. It would sure be nice if the
> I/O could be handled in a more asynchronous way so that these CPUs can
> go off and do other things while they are waiting for I/Os to complete
> on slow disks.
>
>> Was it any different at the front and back of the drive?
>
> Didn't try on this particular test.
>
>> One thing I didn't see in this thread was a check to make sure your
>> alignment is on the physical sector alignment if you're using 4K sectors
>> which I assume drives this large are using.
>
> Yes, these drives surely use 4K sectors. But I haven't checked for
> sector alignment issues. Any tips on how to do that?
According to parted, my disk partition is aligned.
(parted) align-check
alignment type(min/opt) [optimal]/minimal?
Partition number? 6
6 aligned
Partition Table: gpt
Number Start End Size File system Name Flags
1 2048s 10002431s 10000384s primary
2 10002432s 42002431s 32000000s primary
3 42002432s 42004479s 2048s primary bios_grub
4 42004480s 42006527s 2048s primary
5 42006528s 50008063s 8001536s primary
> 6 50008064s 7796883455s 7746875392s primary
50008064 / 4096 = 12209
^ permalink raw reply
* Re: Reshape stalls and fails when SELinux is enabled
From: George Rapp @ 2015-12-11 1:26 UTC (permalink / raw)
To: Wols Lists; +Cc: Edward Kuns, Enrico Tagliavini, Linux-RAID
In-Reply-To: <5669C34C.9060606@youngman.org.uk>
Indeed, SELinux appears to be very not-helpful in the RAID resize
process. I decided to put SELINUX=disabled in /etc/selinux/config,
touch /.relabel, and reboot before doing any more RAID 6 work.
The only other workaround I could see is the ability to do a "dry run"
of mdadm operations a la wodim / growisofs - a chance to test out all
the parameters before actually pulling the trigger. (Sorry, no patch,
though ... 8^)
On Thu, Dec 10, 2015 at 1:24 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 10/12/15 17:06, Edward Kuns wrote:
>> I ran into this as well. I unwisely worked around it by placing the
>> backup file on /tmp. I don't recommend that choice!
>
> Most people don't realise there are various tmp directories, with
> different behaviours. /tmp is defined by LSB or somesuch to be real
> temporary storage - no guarantees - stuff may disappear at any time.
> Which is why my /tmp is a tmpfs - stored in ram. So yes, a backup file
> on /tmp is not a good idea ...
>
> /var/tmp is defined as holding data that must survive a reboot - where
> emacs and vi and that lot are recommended to store their crash dumps,
> recovery files, etc for example.
>
> I didn't see
>> your AVC on your EMail, but I recommend you reach out to the SELinux
>> folks for labeling help. I've reach out to them in the past and have
>> found them very helpful (as are the folks here). Getting the labeling
>> corrected will not only help you but everyone else in the long term.
>>
>> I agree that it would be good for mdadm to handle this kind of failure
>> in a more productive fashion as well. But I'm only a lurker here. I
>> don't have enough knowledge to suggest what would be better. One
>> thing that really helped me debug this was commenting out the lines
>>
>> #StandardOutput=null
>> #StandardError=null
>>
>> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
>> -- note you have to run systemctl daemon-reload after changing the
>> file for the change to be noticed.
>>
>> Once I did that, the system logs had much more useful information.
>> I'm not sure where that file is sourced, whether it's from the
>> linux-raid people or from the distributions or somewhere in between.
>> Does anyone here know if there's a reason why standard out and
>> standard error are sent to /dev/null for this service?
>>
> I wouldn't know for certain, but given the amount of junk so many
> programs spew to stdout or stderr (which nobody ever sees because
> they're running in a gui), I expect it's because that's what happens by
> default anyway :-) They don't want to clutter the logs with all that
> junk and chatter.
>
> I often somehow get stdout and stderr for various programs redirected to
> konsole, because I'll use dbus-launch to start a program. And then when
> the program has exited I'll get loads of rubbish from other programs ...
> the chatter is horrendous.
>
> Cheers,
> Wol
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: NeilBrown @ 2015-12-11 1:56 UTC (permalink / raw)
To: Shaohua Li; +Cc: Christoph Hellwig, linux-raid
In-Reply-To: <20151211011049.GA3810341@devbig084.prn1.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 4949 bytes --]
On Fri, Dec 11 2015, Shaohua Li wrote:
> On Fri, Dec 11, 2015 at 11:09:58AM +1100, NeilBrown wrote:
>> On Fri, Dec 11 2015, Shaohua Li wrote:
>>
>> > On Wed, Dec 09, 2015 at 05:34:45PM +1100, NeilBrown wrote:
>> >> On Wed, Dec 09 2015, Shaohua Li wrote:
>> >>
>> >> > On Wed, Dec 09, 2015 at 11:36:30AM +1100, NeilBrown wrote:
>> >> >> On Thu, Dec 03 2015, Christoph Hellwig wrote:
>> >> >>
>> >> >> > Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
>> >> >> >
>> >> >> > I've looked into replacing these with mempools and biosets, and for the
>> >> >> > bio and the meta_page that's pretty trivial as they have short life times
>> >> >> > and do make guaranteed progress. I'm massively struggling with the iounit
>> >> >> > allocation, though. These can live on for a long time over log I/O, cache
>> >> >> > flushing and last but not least RAID I/O, and every attempt at something
>> >> >> > mempool-like results in reproducible deadlocks. I wonder if we need to
>> >> >> > figure out some more efficient data structure to communicate the completion
>> >> >> > status that doesn't rely on these fairly long living allocations from
>> >> >> > the I/O path.
>> >> >>
>> >> >> Presumably the root cause of these deadlocks is that the raid5d thread
>> >> >> has called
>> >> >> handle_stripe -> ops_run_io ->r5l_write_stripe -> r5l_log_stripe
>> >> >> -> r5l_get_meta -> r5l_new_meta
>> >> >>
>> >> >> and r5l_new_meta is blocked on memory allocation, which won't complete
>> >> >> until some raid5 stripes get written out, which requires raid5d to do
>> >> >> something more useful than sitting and waiting.
>> >> >>
>> >> >> I suspect a good direction towards a solution would be to allow the
>> >> >> memory allocation to fail, to cleanly propagate that failure indication
>> >> >> up through r5l_log_stripe to r5l_write_stripe which falls back to adding
>> >> >> the stripe_head to ->no_space_stripes.
>> >> >>
>> >> >> Then we only release stripes from no_space_stripes when a memory
>> >> >> allocation might succeed.
>> >> >>
>> >> >> There are lots of missing details, and possibly we would need a separate
>> >> >> list rather than re-using no_space_stripes.
>> >> >> But the key idea is that raid5d should never block (except beneath
>> >> >> submit_bio on some other device) and when it cannot make progress
>> >> >> without blocking, it should queue the stripe_head for later handling.
>> >> >>
>> >> >> Does that make sense?
>> >> >
>> >> > It does remove the scary __GFP_NOFAIL, but the approach is essentially
>> >> > idential to a 'retry after allocation failure'. Why not just let the mm
>> >> > (with __GFP_NOFAIL) to do the retry then?
>> >> >
>> >>
>> >> Because deadlocks.
>> >>
>> >> If raid5d is waiting for the mm to allocated memory, then it cannot
>> >> retire write requests which could free up memory.
>> >
>> > Ok, I missed the deadlock issue. Looks raid5d can't sleep waitting for
>> > memory. That means the mempool for metadata page/bio doesn't work too,
>> > as raid5d might block and not dispatch previous IO. Your proposal looks
>> > reasonable.
>>
>> raid5d mustn't sleep waiting for something that only raid5d can provide.
>>
>> A bio allocated from a mempool will be returned to that mempool when the
>> IO request completes (if the end_io routine calls bio_put, which it
>> does).
>> So when waiting on a mempool for a bio, you are only waiting for some
>> underlying device to perform IO, you are not waiting for anything that
>> raid5d might do. So that is safe.
>>
>> Similarly the meta_page is freed in the end_io function, so it is safe
>> to wait for that.
>> Note that the important feature of a mempool is not so much that it
>> pre-allocates some memory. The important point is that when that
>> pre-allocated memory is freed it can *only* be used by the owner of the
>> mempool.
>> By contrast memory allocated with NOFAIL, when freed might be used by
>> any other caller anywhere in the kernel.
>> So waiting on a mempool means waiting for certain well defined events.
>> Waiting in NOFAIL could end up waiting for almost anything. So it is
>> much harder to reason about deadlocks.
>>
>> The bio and the meta_page contrast with the r5l_io_unit which isn't
>> freed by the first end_io, but remains until raid5d has done some more
>> work. So raid5d cannot wait for a r5l_io_unit to be freed.
>
> I'm confused 2 bios are already enough to avoid deadlock, sorry. Can you
> peek Christoph's patches? I'll work on a patch for r5l_io_unit.
>
I don't understand what you are tring to say.
Yes, a bio pool of 2 bios is (I believe) enough to avoid a deadlock when
allocation a bio. It no affect on any deadlock involved in allocating
an r5l_io_unit.
I have looked at Christoph's patches. What in particular did you hope I
would see?
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH v4 0/3] Badblock tracking for gendisks
From: Dan Williams @ 2015-12-11 5:40 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-nvdimm@lists.01.org, linux-raid, linux-scsi, NeilBrown,
linux-block, James Bottomley, Vishal Verma, Christoph Hellwig
In-Reply-To: <1449602295-28781-1-git-send-email-vishal.l.verma@intel.com>
Hi Jens,
Are you on-board with this approach? Any concerns with me carrying
this through the nvdimm tree along with our other pending
error-handling enabling?
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: Phil Turmel @ 2015-12-11 14:04 UTC (permalink / raw)
To: P. Remek; +Cc: linux-raid
In-Reply-To: <CABdHLQ4rh+Rtu6U+ZX6m0CWXu3sW4boEzHaPdxD03UBf82EJcQ@mail.gmail.com>
On 12/10/2015 05:44 PM, P. Remek wrote:
>>
>> Order of device discovery is not guaranteed, but base devices will
>> almost always show up before multipath devices. You have to filter out
>> the base devices from mdadm consideration:
>>
>> Add a DEVICES statement to your mdadm.conf that only matches your
>> multipath device names, not the base names. Then update your initramfs
>> so it applies to early boot as well.
>>
>
>
> This is something which I actually already tried. I specified in my
> /etc/mdadm/mdadm.conf following:
>
> DEVICE /dev/mapper/hgst-ssd*
>
> In my /etc/multipath.conf I instruct dm-multipath to create
> /dev/mapper/hgst-ssd1 /dev/mapper/hgst-ssd2 /dev/mapper/hgst-ssd3
You haven't said that you've updated your initramfs with this info.
> But the only result is that after this config, the md raid is not
> started at all. My conclusion was that when md raid was starting, the
> multipath devices did not yet exist so it did not start up the array.
That would be an initramfs bug, most likely. Modern initramfs like
dracut use incremental assembly to start arrays and subsystems as
devices are started, regardless what order they show up.
Phil
^ permalink raw reply
* Re: Reshape stalls and fails when SELinux is enabled
From: Enrico Tagliavini @ 2015-12-11 14:26 UTC (permalink / raw)
To: George Rapp; +Cc: Wols Lists, Edward Kuns, Linux-RAID
In-Reply-To: <CAF-KpgZXbqVcanWYVMoV9+1vVo5S7DHXPs2PD8jGbO2D6wjriw@mail.gmail.com>
No need to disable SELinux. Just run
setenforce 0
before calling mdadm. When your reshape is done enable SELinux again with
setenforce 1
no reboot needed.
Also if you want a permanent workaround you can whitelist mdadm process type:
semanage permissive -a mdadm_t # but be aware I didn't tested this!
On 11 December 2015 at 02:26, George Rapp <george.rapp@gmail.com> wrote:
> Indeed, SELinux appears to be very not-helpful in the RAID resize
> process. I decided to put SELINUX=disabled in /etc/selinux/config,
> touch /.relabel, and reboot before doing any more RAID 6 work.
>
> The only other workaround I could see is the ability to do a "dry run"
> of mdadm operations a la wodim / growisofs - a chance to test out all
> the parameters before actually pulling the trigger. (Sorry, no patch,
> though ... 8^)
>
> On Thu, Dec 10, 2015 at 1:24 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 10/12/15 17:06, Edward Kuns wrote:
>>> I ran into this as well. I unwisely worked around it by placing the
>>> backup file on /tmp. I don't recommend that choice!
>>
>> Most people don't realise there are various tmp directories, with
>> different behaviours. /tmp is defined by LSB or somesuch to be real
>> temporary storage - no guarantees - stuff may disappear at any time.
>> Which is why my /tmp is a tmpfs - stored in ram. So yes, a backup file
>> on /tmp is not a good idea ...
>>
>> /var/tmp is defined as holding data that must survive a reboot - where
>> emacs and vi and that lot are recommended to store their crash dumps,
>> recovery files, etc for example.
>>
>> I didn't see
>>> your AVC on your EMail, but I recommend you reach out to the SELinux
>>> folks for labeling help. I've reach out to them in the past and have
>>> found them very helpful (as are the folks here). Getting the labeling
>>> corrected will not only help you but everyone else in the long term.
>>>
>>> I agree that it would be good for mdadm to handle this kind of failure
>>> in a more productive fashion as well. But I'm only a lurker here. I
>>> don't have enough knowledge to suggest what would be better. One
>>> thing that really helped me debug this was commenting out the lines
>>>
>>> #StandardOutput=null
>>> #StandardError=null
>>>
>>> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
>>> -- note you have to run systemctl daemon-reload after changing the
>>> file for the change to be noticed.
>>>
>>> Once I did that, the system logs had much more useful information.
>>> I'm not sure where that file is sourced, whether it's from the
>>> linux-raid people or from the distributions or somewhere in between.
>>> Does anyone here know if there's a reason why standard out and
>>> standard error are sent to /dev/null for this service?
>>>
>> I wouldn't know for certain, but given the amount of junk so many
>> programs spew to stdout or stderr (which nobody ever sees because
>> they're running in a gui), I expect it's because that's what happens by
>> default anyway :-) They don't want to clutter the logs with all that
>> junk and chatter.
>>
>> I often somehow get stdout and stderr for various programs redirected to
>> konsole, because I'll use dbus-launch to start a program. And then when
>> the program has exited I'll get loads of rubbish from other programs ...
>> the chatter is horrendous.
>>
>> Cheers,
>> Wol
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
> LinkedIn profile: https://www.linkedin.com/in/georgerapp
> Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-11 15:44 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, Linux-RAID
In-Reply-To: <CAK2H+ec-zMbhxoFyHXLkdM-z-9cYYzNbPFhn19XjTHqrOMDZKQ@mail.gmail.com>
On Thu, Dec 10, 2015 at 8:50 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Thu, Dec 10, 2015 at 5:19 PM, Dallas Clement <dallas.a.clement@gmail.com>
> wrote:
>>
>> <SNIP>
>> >> One thing I didn't see in this thread was a check to make sure your
>> >> alignment is on the physical sector alignment if you're using 4K
>> >> sectors
>> >> which I assume drives this large are using.
>> >
>> > Yes, these drives surely use 4K sectors. But I haven't checked for
>> > sector alignment issues. Any tips on how to do that?
>>
>> According to parted, my disk partition is aligned.
>>
>> (parted) align-check
>> alignment type(min/opt) [optimal]/minimal?
>> Partition number? 6
>> 6 aligned
>>
>> Partition Table: gpt
>>
>> Number Start End Size File system Name Flags
>> 1 2048s 10002431s 10000384s primary
>> 2 10002432s 42002431s 32000000s primary
>> 3 42002432s 42004479s 2048s primary
>> bios_grub
>> 4 42004480s 42006527s 2048s primary
>> 5 42006528s 50008063s 8001536s primary
>> > 6 50008064s 7796883455s 7746875392s primary
>>
>> 50008064 / 4096 = 12209
>
> I think you want to divide by 8 as 8*512 = 4096 but you're likely ok
> on all of them. Partitioning tools are better about that these days.
>
> Looking back I didn't see what sort of controller/s you have all these
> drives attached to. If you're spinning in IO loops then it could be a
> driver issue.
>
> I would suggest googling for causes of iowait cycles and post
> back additional questions.
>
> Good luck,
> Mark
Hi Mark. I have three different controllers on this motherboard. A
Marvell 9485 controls 8 of the disks. And an Intel Cougar Point
controls the 4 remaining disks.
> If you're spinning in IO loops then it could be a driver issue.
It sure is looking like that. I will try to profile the kernel
threads today and maybe use blktrace as Phil recommended to see what
is going on there.
This is pretty sad that 12 single threaded fio jobs can bring this
system to its knees.
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-11 16:01 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <566AD7EA.4070903@turmel.org>
Here is my config:
root@os-node1:~# uname -a
Linux os-node1 3.16.0-48-generic #64~14.04.1-Ubuntu SMP Thu Aug 20
23:03:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@os-node1:~# cat /etc/mdadm/mdadm.conf
DEVICE /dev/mapper/hgst-ssd*
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
ARRAY /dev/md0 metadata=0.90 UUID=4472a74f:ca206897:3b82bcad:daeae9f9
root@os-node1:~# cat /etc/multipath.conf
blacklist {
wwid 3600508b1001cbb0c5b5a2fe0a85e0053
wwid 3600508b1001c33e8fe607dfd73043d03
wwid 32020030102060804
}
multipaths {
multipath {
wwid 35000cca04f1cbdfc
alias hgst-ssd1
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1cc050
alias hgst-ssd2
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1ce930
alias hgst-ssd3
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1ce1f4
alias hgst-ssd4
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1cbe18
alias hgst-ssd5
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1cdc58
alias hgst-ssd6
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
with this config I do:
root@os-node1:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.16.0-48-generic
Then I reboot the computer but the mdadm raid doesn't start up. The
multipathed devides does start up:
root@os-node1:~# multipath -ll
Error: : Inappropriate ioctl for device
cciss TUR failed in CCISS_GETLUNINFO: Inappropriate ioctl for device
Error: : Inappropriate ioctl for device
cciss TUR failed in CCISS_GETLUNINFO: Inappropriate ioctl for device
hgst-ssd6 (35000cca04f1cdc58) dm-5 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:2:0 sde 8:64 active ready running
`- 4:0:2:0 sdk 8:160 active ready running
hgst-ssd5 (35000cca04f1cbe18) dm-7 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:4:0 sdg 8:96 active ready running
`- 4:0:4:0 sdm 8:192 active ready running
hgst-ssd4 (35000cca04f1ce1f4) dm-3 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:0 sdc 8:32 active ready running
`- 4:0:0:0 sdi 8:128 active ready running
hgst-ssd3 (35000cca04f1ce930) dm-4 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:1:0 sdd 8:48 active ready running
`- 4:0:1:0 sdj 8:144 active ready running
hgst-ssd2 (35000cca04f1cc050) dm-8 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:5:0 sdh 8:112 active ready running
`- 4:0:5:0 sdn 8:208 active ready running
hgst-ssd1 (35000cca04f1cbdfc) dm-6 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:3:0 sdf 8:80 active ready running
`- 4:0:3:0 sdl 8:176 active ready running
Regards,
Remek
On Fri, Dec 11, 2015 at 3:04 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/10/2015 05:44 PM, P. Remek wrote:
>>>
>>> Order of device discovery is not guaranteed, but base devices will
>>> almost always show up before multipath devices. You have to filter out
>>> the base devices from mdadm consideration:
>>>
>>> Add a DEVICES statement to your mdadm.conf that only matches your
>>> multipath device names, not the base names. Then update your initramfs
>>> so it applies to early boot as well.
>>>
>>
>>
>> This is something which I actually already tried. I specified in my
>> /etc/mdadm/mdadm.conf following:
>>
>> DEVICE /dev/mapper/hgst-ssd*
>>
>> In my /etc/multipath.conf I instruct dm-multipath to create
>> /dev/mapper/hgst-ssd1 /dev/mapper/hgst-ssd2 /dev/mapper/hgst-ssd3
>
> You haven't said that you've updated your initramfs with this info.
>
>> But the only result is that after this config, the md raid is not
>> started at all. My conclusion was that when md raid was starting, the
>> multipath devices did not yet exist so it did not start up the array.
>
> That would be an initramfs bug, most likely. Modern initramfs like
> dracut use incremental assembly to start arrays and subsystems as
> devices are started, regardless what order they show up.
>
> Phil
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox