* bio-based DM multipath is back from the dead [was: Re: Notes from the four separate IO track sessions at LSF/MM]
[not found] ` <1461858038.2307.16.camel@HansenPartnership.com>
@ 2016-05-26 2:38 ` Mike Snitzer
2016-05-27 8:39 ` Hannes Reinecke
0 siblings, 1 reply; 5+ messages in thread
From: Mike Snitzer @ 2016-05-26 2:38 UTC (permalink / raw)
To: James Bottomley
Cc: linux-block, lsf, device-mapper development, linux-scsi, hch
On Thu, Apr 28 2016 at 11:40am -0400,
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> On Thu, 2016-04-28 at 08:11 -0400, Mike Snitzer wrote:
> > Full disclosure: I'll be looking at reinstating bio-based DM multipath to
> > regain efficiencies that now really matter when issuing IO to extremely
> > fast devices (e.g. NVMe). bio cloning is now very cheap (due to
> > immutable biovecs), coupled with the emerging multipage biovec work that
> > will help construct larger bios, so I think it is worth pursuing to at
> > least keep our options open.
Please see the 4 topmost commits I've published here:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.8
All request-based DM multipath support/advances have been completly
preserved. I've just made it so that we can now have bio-based DM
multipath too.
All of the various modes have been tested using mptest:
https://github.com/snitm/mptest
> OK, but remember the reason we moved from bio to request was partly to
> be nearer to the device but also because at that time requests were
> accumulations of bios which had to be broken out, go back up the stack
> individually and be re-elevated, which adds to the inefficiency. In
> theory the bio splitting work will mean that we only have one or two
> split bios per request (because they were constructed from a split up
> huge bio), but when we send them back to the top to be reconstructed as
> requests there's no guarantee that the split will be correct a second
> time around and we might end up resplitting the already split bios. If
> you do reassembly into the huge bio again before resend down the next
> queue, that's starting to look like quite a lot of work as well.
I've not even delved into the level you're laser-focused on here.
But I'm struggling to grasp why multipath is any different than any
other bio-based device...
FYI, the paper I reference in my "dm mpath: reinstate bio-based support"
commit gets into what I've always thought the real justification was for
the transition from bio-based to request-based.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: bio-based DM multipath is back from the dead [was: Re: Notes from the four separate IO track sessions at LSF/MM]
2016-05-26 2:38 ` bio-based DM multipath is back from the dead [was: Re: Notes from the four separate IO track sessions at LSF/MM] Mike Snitzer
@ 2016-05-27 8:39 ` Hannes Reinecke
2016-05-27 14:44 ` Mike Snitzer
0 siblings, 1 reply; 5+ messages in thread
From: Hannes Reinecke @ 2016-05-27 8:39 UTC (permalink / raw)
To: Mike Snitzer, James Bottomley
Cc: linux-block, lsf, device-mapper development, linux-scsi, hch
On 05/26/2016 04:38 AM, Mike Snitzer wrote:
> On Thu, Apr 28 2016 at 11:40am -0400,
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
>
>> On Thu, 2016-04-28 at 08:11 -0400, Mike Snitzer wrote:
>>> Full disclosure: I'll be looking at reinstating bio-based DM multipath to
>>> regain efficiencies that now really matter when issuing IO to extremely
>>> fast devices (e.g. NVMe). bio cloning is now very cheap (due to
>>> immutable biovecs), coupled with the emerging multipage biovec work that
>>> will help construct larger bios, so I think it is worth pursuing to at
>>> least keep our options open.
>
> Please see the 4 topmost commits I've published here:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.8
>
> All request-based DM multipath support/advances have been completly
> preserved. I've just made it so that we can now have bio-based DM
> multipath too.
>
> All of the various modes have been tested using mptest:
> https://github.com/snitm/mptest
>
>> OK, but remember the reason we moved from bio to request was partly to
>> be nearer to the device but also because at that time requests were
>> accumulations of bios which had to be broken out, go back up the stack
>> individually and be re-elevated, which adds to the inefficiency. In
>> theory the bio splitting work will mean that we only have one or two
>> split bios per request (because they were constructed from a split up
>> huge bio), but when we send them back to the top to be reconstructed as
>> requests there's no guarantee that the split will be correct a second
>> time around and we might end up resplitting the already split bios. If
>> you do reassembly into the huge bio again before resend down the next
>> queue, that's starting to look like quite a lot of work as well.
>
> I've not even delved into the level you're laser-focused on here.
> But I'm struggling to grasp why multipath is any different than any
> other bio-based device...
>
Actually, _failover_ is not the primary concern. This is on a (relative)
slow path so any performance degradation during failover is acceptable.
No, the real issue is load-balancing.
If you have several paths you have to schedule I/O across all paths,
_and_ you should be feeding these paths efficiently.
With the original (bio-based) layout you had to schedule on the bio
level, causing the requests to be inefficiently assembled.
Hence the 'rr_min_io' parameter, which were changing paths after
rr_min_io _bios_. I did some experimenting a while back (I even had a
presentation on LSF at one point ...), and figuring that you would get a
performance degradation once the rr_min_io parameter went below 100.
But this means that paths will be switched after every 100 bios,
irrespective of into how many requests they'll be assembled.
It also means that we have a rather 'choppy' load-balancing behaviour,
and cannot achieve 'true' load balancing as the I/O scheduler on the bio
level doesn't have any idea when a new request will be assembled.
I was sort-of hoping that with the large bio work from Shaohua we could
build bio which would not require any merging, ie building bios which
would be assembled into a single request per bio.
Then the above problem wouldn't exist anymore and we _could_ do
scheduling on bio level.
But from what I've gathered this is not always possible (eg for btrfs
with delayed allocation).
Have you found another way of addressing this problem?
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 N�rnberg
GF: J. Hawn, J. Guild, F. Imend�rffer, HRB 16746 (AG N�rnberg)
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: bio-based DM multipath is back from the dead [was: Re: Notes from the four separate IO track sessions at LSF/MM]
2016-05-27 8:39 ` Hannes Reinecke
@ 2016-05-27 14:44 ` Mike Snitzer
2016-05-27 15:42 ` Hannes Reinecke
0 siblings, 1 reply; 5+ messages in thread
From: Mike Snitzer @ 2016-05-27 14:44 UTC (permalink / raw)
To: Hannes Reinecke
Cc: James Bottomley, linux-block, lsf, device-mapper development, hch,
linux-scsi, axboe, Ming Lei
On Fri, May 27 2016 at 4:39am -0400,
Hannes Reinecke <hare@suse.de> wrote:
> On 05/26/2016 04:38 AM, Mike Snitzer wrote:
> >On Thu, Apr 28 2016 at 11:40am -0400,
> >James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> >
> >>On Thu, 2016-04-28 at 08:11 -0400, Mike Snitzer wrote:
> >>>Full disclosure: I'll be looking at reinstating bio-based DM multipath to
> >>>regain efficiencies that now really matter when issuing IO to extremely
> >>>fast devices (e.g. NVMe). bio cloning is now very cheap (due to
> >>>immutable biovecs), coupled with the emerging multipage biovec work that
> >>>will help construct larger bios, so I think it is worth pursuing to at
> >>>least keep our options open.
> >
> >Please see the 4 topmost commits I've published here:
> >https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.8
> >
> >All request-based DM multipath support/advances have been completly
> >preserved. I've just made it so that we can now have bio-based DM
> >multipath too.
> >
> >All of the various modes have been tested using mptest:
> >https://github.com/snitm/mptest
> >
> >>OK, but remember the reason we moved from bio to request was partly to
> >>be nearer to the device but also because at that time requests were
> >>accumulations of bios which had to be broken out, go back up the stack
> >>individually and be re-elevated, which adds to the inefficiency. In
> >>theory the bio splitting work will mean that we only have one or two
> >>split bios per request (because they were constructed from a split up
> >>huge bio), but when we send them back to the top to be reconstructed as
> >>requests there's no guarantee that the split will be correct a second
> >>time around and we might end up resplitting the already split bios. If
> >>you do reassembly into the huge bio again before resend down the next
> >>queue, that's starting to look like quite a lot of work as well.
> >
> >I've not even delved into the level you're laser-focused on here.
> >But I'm struggling to grasp why multipath is any different than any
> >other bio-based device...
> >
> Actually, _failover_ is not the primary concern. This is on a
> (relative) slow path so any performance degradation during failover
> is acceptable.
>
> No, the real issue is load-balancing.
> If you have several paths you have to schedule I/O across all paths,
> _and_ you should be feeding these paths efficiently.
<snip well known limitation of bio-based mpath load balancing, also
detailed in the multipath paper I refernced>
Right, as my patch header details, this is the only limitation that
remains with the reinstated bio-based DM multipath.
> I was sort-of hoping that with the large bio work from Shaohua we
I think you mean Ming Lei and his multipage biovec work?
> could build bio which would not require any merging, ie building
> bios which would be assembled into a single request per bio.
> Then the above problem wouldn't exist anymore and we _could_ do
> scheduling on bio level.
> But from what I've gathered this is not always possible (eg for
> btrfs with delayed allocation).
I doubt many people are running btrfs over multipath in production
but...
Taking a step back: reinstating bio-based DM multipath is _not_ at the
expense of request-based DM multipath. As you can see I've made it so
that all modes (bio-based, request_fn rq-based, and blk-mq rq-based) are
supported by a single DM multipath target. When the trnasition to
request-based happened it would've been wise to preserve bio-based but I
digress...
So, the point is: there isn't any one-size-fits-all DM multipath queue
mode here. If a storage config benefits from the request_fn IO
schedulers (but isn't hurt by .request_fn's queue lock, so slower
rotational storage?) then use queue_mode=2. If the storage is connected
to a large NUMA system and there is some reason to want to use blk-mq
request_queue at the DM level: use queue_mode=3. If the storage is
_really_ fast and doesn't care about extra IO grooming (e.g. sorting and
merging) then select bio-based using queue_mode=1.
I collected some quick performance numbers against a null_blk device, on
a single NUMA node system, with various DM layers ontop -- the multipath
runs are only with a single path... fio workload is just 10 sec randread:
FIO_QUEUE_DEPTH=32
FIO_RUNTIME=10
FIO_NUMJOBS=12
{FIO} --numa_cpu_nodes=${NID} --numa_mem_policy=bind:${NID} --cpus_allowed_policy=split --group_reporting --rw=randread --bs=4k --numjobs=${FIO_NUMJOBS} \
--iodepth=${FIO_QUEUE_DEPTH} --runtime=${FIO_RUNTIME} --time_based --loops=1 --ioengine=libaio \
--direct=1 --invalidate=1 --randrepeat=1 --norandommap --exitall --name task_${TASK_NAME} --filename=${DEVICE}"
I need real hardware (NVMe over Fabrics please!) to really test this
stuff properly; but I think the following results at least approximate
the relative performance of each multipath mode.
On null_blk blk-mq
------------------
baseline:
null_blk blk-mq iops=1936.3K
dm-linear iops=1616.1K
multipath using round-robin path-selector:
bio-based iops=1579.8K
blk-mq rq-based iops=1411.6K
request_fn rq-based iops=326491
multipath using queue-length path-selector:
bio-based iops=1526.2K
blk-mq rq-based iops=1351.9K
request_fn rq-based iops=326399
On null_blk bio-based
---------------------
baseline:
null_blk blk-mq iops=2776.8K
dm-linear iops=2183.5K
multipath using round-robin path-selector:
bio-based iops=2101.5K
multipath using queue-length path-selector:
bio-based iops=2019.4K
I haven't even looked at optimizing bio-based DM yet.. not liking that
dm-linear is taking a ~15% - ~20% hit from baseline null_blk. But nice
to see bio-based multipath is very comparable to dm-linear. So any
future bio-based DM performance advances should translate to better
multipath perf.
> Have you found another way of addressing this problem?
No, bio sorting/merging really isn't a problem for DM multipath to
solve.
Though Jens did say (in the context of one of these dm-crypt bulk mode
threads) that the block core _could_ grow some additional _minimalist_
capability for bio merging:
https://www.redhat.com/archives/dm-devel/2015-November/msg00130.html
I'd like to understand a bit more about what Jens is thinking in that
area because it could benefit DM thinp as well (though that is using bio
sorting rather than merging, introduced via commit 67324ea188).
I'm not opposed to any line of future development -- but development
needs to be driven by observed limitations while testing on _real_
hardware.
Mike
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: bio-based DM multipath is back from the dead [was: Re: Notes from the four separate IO track sessions at LSF/MM]
2016-05-27 14:44 ` Mike Snitzer
@ 2016-05-27 15:42 ` Hannes Reinecke
2016-05-27 16:10 ` Mike Snitzer
0 siblings, 1 reply; 5+ messages in thread
From: Hannes Reinecke @ 2016-05-27 15:42 UTC (permalink / raw)
To: Mike Snitzer
Cc: James Bottomley, linux-block, lsf, device-mapper development, hch,
linux-scsi, axboe, Ming Lei
On 05/27/2016 04:44 PM, Mike Snitzer wrote:
> On Fri, May 27 2016 at 4:39am -0400,
> Hannes Reinecke <hare@suse.de> wrote:
>
[ .. ]
>> No, the real issue is load-balancing.
>> If you have several paths you have to schedule I/O across all paths,
>> _and_ you should be feeding these paths efficiently.
>
> <snip well known limitation of bio-based mpath load balancing, also
> detailed in the multipath paper I refernced>
>
> Right, as my patch header details, this is the only limitation that
> remains with the reinstated bio-based DM multipath.
>
:-)
And the very reason why we went into request-based multipathing in the
first place...
>> I was sort-of hoping that with the large bio work from Shaohua we
>
> I think you mean Ming Lei and his multipage biovec work?
>
Errm. Yeah, of course. Apologies.
>> could build bio which would not require any merging, ie building
>> bios which would be assembled into a single request per bio.
>> Then the above problem wouldn't exist anymore and we _could_ do
>> scheduling on bio level.
>> But from what I've gathered this is not always possible (eg for
>> btrfs with delayed allocation).
>
> I doubt many people are running btrfs over multipath in production
> but...
>
Hey. There is a company who does ...
> Taking a step back: reinstating bio-based DM multipath is _not_ at the
> expense of request-based DM multipath. As you can see I've made it so
> that all modes (bio-based, request_fn rq-based, and blk-mq rq-based) are
> supported by a single DM multipath target. When the trnasition to
> request-based happened it would've been wise to preserve bio-based but I
> digress...
>
> So, the point is: there isn't any one-size-fits-all DM multipath queue
> mode here. If a storage config benefits from the request_fn IO
> schedulers (but isn't hurt by .request_fn's queue lock, so slower
> rotational storage?) then use queue_mode=2. If the storage is connected
> to a large NUMA system and there is some reason to want to use blk-mq
> request_queue at the DM level: use queue_mode=3. If the storage is
> _really_ fast and doesn't care about extra IO grooming (e.g. sorting and
> merging) then select bio-based using queue_mode=1.
>
> I collected some quick performance numbers against a null_blk device, on
> a single NUMA node system, with various DM layers ontop -- the multipath
> runs are only with a single path... fio workload is just 10 sec randread:
>
Which is precisely the point.
Everything's nice and shiny with a single path, as then the above issue
simply doesn't apply.
Things only start getting interesting if you have _several_ paths.
So the benchmarks only prove that device-mapper doesn't add too much of
an overhead; they don't prove that the above point has been addressed...
[ .. ]
>> Have you found another way of addressing this problem?
>
> No, bio sorting/merging really isn't a problem for DM multipath to
> solve.
>
> Though Jens did say (in the context of one of these dm-crypt bulk mode
> threads) that the block core _could_ grow some additional _minimalist_
> capability for bio merging:
> https://www.redhat.com/archives/dm-devel/2015-November/msg00130.html
>
> I'd like to understand a bit more about what Jens is thinking in that
> area because it could benefit DM thinp as well (though that is using bio
> sorting rather than merging, introduced via commit 67324ea188).
>
> I'm not opposed to any line of future development -- but development
> needs to be driven by observed limitations while testing on _real_
> hardware.
>
In the end, with Ming Leis multipage bvec work we essentially already
moved some merging ability into the bios; during bio_add_page() the
block layer will already merge bios together.
(I'll probably be yelled at by hch for ignorance for the following, but
nevertheless)
From my POV there are several areas of 'merging' which currently happen:
a) bio merging: combine several consecutive bios into a larger one;
should be largely address by Ming Leis multipage bvec
b) bio sorting: reshuffle bios so that any requests on the request queue
are ordered 'best' for the underlying hardware (ie the actual I/O
scheduler). Not implemented for mq, and actually of questionable value
for fast storage. One of the points I'll be testing in the very near
future; ideally we find that it's not _that_ important (compared to the
previous point), then we could drop it altogether for mq.
c) clustering: coalescing several consecutive pages/bvecs into a single
SG element. Obviously only can happen if you have large enough requests.
But the only gain is shortening the number of SG elements for a requests.
Again of questionable value as the request itself and the amount of data
to transfer isn't changed. And another point of performance testing on
my side.
So ideally we will find that b) and c) only contribute with a small
amount to the overall performance, then we could easily drop it for MQ
and concentrate on make bio merging work well.
Then it wouldn't really matter if we were doing bio-based or
request-based multipathing as we had a 1:1 relationship, and this entire
discussion could go away.
Well. Or that's the hope, at least.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 N�rnberg
GF: J. Hawn, J. Guild, F. Imend�rffer, HRB 16746 (AG N�rnberg)
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: bio-based DM multipath is back from the dead [was: Re: Notes from the four separate IO track sessions at LSF/MM]
2016-05-27 15:42 ` Hannes Reinecke
@ 2016-05-27 16:10 ` Mike Snitzer
0 siblings, 0 replies; 5+ messages in thread
From: Mike Snitzer @ 2016-05-27 16:10 UTC (permalink / raw)
To: Hannes Reinecke
Cc: James Bottomley, linux-block, lsf, device-mapper development, hch,
linux-scsi, axboe, Ming Lei
On Fri, May 27 2016 at 11:42am -0400,
Hannes Reinecke <hare@suse.de> wrote:
> On 05/27/2016 04:44 PM, Mike Snitzer wrote:
> >On Fri, May 27 2016 at 4:39am -0400,
> >Hannes Reinecke <hare@suse.de> wrote:
> >
> [ .. ]
> >>No, the real issue is load-balancing.
> >>If you have several paths you have to schedule I/O across all paths,
> >>_and_ you should be feeding these paths efficiently.
> >
> ><snip well known limitation of bio-based mpath load balancing, also
> >detailed in the multipath paper I refernced>
> >
> >Right, as my patch header details, this is the only limitation that
> >remains with the reinstated bio-based DM multipath.
> >
>
> :-)
> And the very reason why we went into request-based multipathing in
> the first place...
>
> >>I was sort-of hoping that with the large bio work from Shaohua we
> >
> >I think you mean Ming Lei and his multipage biovec work?
> >
> Errm. Yeah, of course. Apologies.
>
> >>could build bio which would not require any merging, ie building
> >>bios which would be assembled into a single request per bio.
> >>Then the above problem wouldn't exist anymore and we _could_ do
> >>scheduling on bio level.
> >>But from what I've gathered this is not always possible (eg for
> >>btrfs with delayed allocation).
> >
> >I doubt many people are running btrfs over multipath in production
> >but...
> >
> Hey. There is a company who does ...
>
> >Taking a step back: reinstating bio-based DM multipath is _not_ at the
> >expense of request-based DM multipath. As you can see I've made it so
> >that all modes (bio-based, request_fn rq-based, and blk-mq rq-based) are
> >supported by a single DM multipath target. When the trnasition to
> >request-based happened it would've been wise to preserve bio-based but I
> >digress...
> >
> >So, the point is: there isn't any one-size-fits-all DM multipath queue
> >mode here. If a storage config benefits from the request_fn IO
> >schedulers (but isn't hurt by .request_fn's queue lock, so slower
> >rotational storage?) then use queue_mode=2. If the storage is connected
> >to a large NUMA system and there is some reason to want to use blk-mq
> >request_queue at the DM level: use queue_mode=3. If the storage is
> >_really_ fast and doesn't care about extra IO grooming (e.g. sorting and
> >merging) then select bio-based using queue_mode=1.
> >
> >I collected some quick performance numbers against a null_blk device, on
> >a single NUMA node system, with various DM layers ontop -- the multipath
> >runs are only with a single path... fio workload is just 10 sec randread:
> >
> Which is precisely the point.
> Everything's nice and shiny with a single path, as then the above
> issue simply doesn't apply.
Heh, as you can see from the request_fn results, that wasn't the case
until very recently with all the DM multipath blk-mq advances..
But my broader thesis is that for really fast storage it is looking
increasingly likely that we don't _need_ or want to have the
multipathing layer dealing with requests. Not unless there is some
inherent big win. request cloning is definitely heavier than bio
cloning.
And as you can probably infer, my work to reinstate bio-based multipath
is focused precisely at the fast storage case in the hopes of avoiding
hch's threat to pull multipathing down into the NVMe over fabrics
driver.
> Things only start getting interesting if you have _several_ paths.
> So the benchmarks only prove that device-mapper doesn't add too much
> of an overhead; they don't prove that the above point has been
> addressed...
Right, but I don't really care if it is addressed by bio-based because
we have the request_fn mode that offers the legacy IO schedulers. The
fact that request_fn multipath has been adequate for the enterprise
rotational storage arrays is somehwat surprising... the queue_lock is a
massive bottleneck.
But if bio merging (via multipage biovecs) does prove itself to be a win
for bio-based multipath for all storage (slow and fast) then yes that'll
be really good news. Nice to have options... we can dial in the option
that is best for a specific usecase/deployment and fix what isn't doing
well.
> [ .. ]
> >>Have you found another way of addressing this problem?
> >
> >No, bio sorting/merging really isn't a problem for DM multipath to
> >solve.
> >
> >Though Jens did say (in the context of one of these dm-crypt bulk mode
> >threads) that the block core _could_ grow some additional _minimalist_
> >capability for bio merging:
> >https://www.redhat.com/archives/dm-devel/2015-November/msg00130.html
> >
> >I'd like to understand a bit more about what Jens is thinking in that
> >area because it could benefit DM thinp as well (though that is using bio
> >sorting rather than merging, introduced via commit 67324ea188).
> >
> >I'm not opposed to any line of future development -- but development
> >needs to be driven by observed limitations while testing on _real_
> >hardware.
> >
> In the end, with Ming Leis multipage bvec work we essentially
> already moved some merging ability into the bios; during
> bio_add_page() the block layer will already merge bios together.
>
> (I'll probably be yelled at by hch for ignorance for the following,
> but nevertheless)
> From my POV there are several areas of 'merging' which currently happen:
> a) bio merging: combine several consecutive bios into a larger one;
> should be largely address by Ming Leis multipage bvec
> b) bio sorting: reshuffle bios so that any requests on the request
> queue are ordered 'best' for the underlying hardware (ie the actual
> I/O scheduler). Not implemented for mq, and actually of questionable
> value for fast storage. One of the points I'll be testing in the
> very near future; ideally we find that it's not _that_ important
> (compared to the previous point), then we could drop it altogether
> for mq.
> c) clustering: coalescing several consecutive pages/bvecs into a
> single SG element. Obviously only can happen if you have large
> enough requests.
> But the only gain is shortening the number of SG elements for a requests.
> Again of questionable value as the request itself and the amount of
> data to transfer isn't changed. And another point of performance
> testing on my side.
>
> So ideally we will find that b) and c) only contribute with a small
> amount to the overall performance, then we could easily drop it for
> MQ and concentrate on make bio merging work well.
> Then it wouldn't really matter if we were doing bio-based or
> request-based multipathing as we had a 1:1 relationship, and this
> entire discussion could go away.
>
> Well. Or that's the hope, at least.
Yeap, let the testing begin! ;)
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-05-27 16:10 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1461800389.2311.70.camel@HansenPartnership.com>
[not found] ` <20160428121108.GA9903@redhat.com>
[not found] ` <1461858038.2307.16.camel@HansenPartnership.com>
2016-05-26 2:38 ` bio-based DM multipath is back from the dead [was: Re: Notes from the four separate IO track sessions at LSF/MM] Mike Snitzer
2016-05-27 8:39 ` Hannes Reinecke
2016-05-27 14:44 ` Mike Snitzer
2016-05-27 15:42 ` Hannes Reinecke
2016-05-27 16:10 ` Mike Snitzer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox