[LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Hannes Reinecke]

Hi all,

I'd like to attend LSF/MM and would like to present my ideas for a 
multipath redesign.

The overall idea is to break up the centralized multipath handling 
in device-mapper (and multipath-tools) and delegate to the 
appropriate sub-systems.

Individually the plan is:
a) use the 'wwid' sysfs attribute to detect multipath devices;
    this removes the need of the current 'path_id' functionality
    in multipath-tools
b) leverage topology information from scsi_dh_alua (which we will
    have once my ALUA handler update is in) to detect the multipath
    topology. This removes the need of a 'prio' infrastructure
    in multipath-tools
c) implement block or scsi events whenever a remote port becomes
    unavailable. This removes the need of the 'path_checker'
    functionality in multipath-tools.
d) leverage these events to handle path-up/path-down events
    in-kernel
e) move the I/O redirection logic out of device-mapper proper
    and use blk-mq to redirect I/O. This is still a bit of
    hand-waving, and definitely would need discussion to figure
    out if and how it can be achieved.
    This is basically the same topic Mike Snitzer proposed, but
    coming from a different angle.

But in the end we should be able to do strip down the current 
(rather complex) multipath-tools to just handle topology changes; 
everything else will be done internally.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Sagi Grimberg]

> Hi all,
>
> I'd like to attend LSF/MM and would like to present my ideas for a
> multipath redesign.
>
> The overall idea is to break up the centralized multipath handling in
> device-mapper (and multipath-tools) and delegate to the appropriate
> sub-systems.

I agree that would be very useful. Great topic. I'd like to attend
this talk as well.

>
> Individually the plan is:
> a) use the 'wwid' sysfs attribute to detect multipath devices;
>     this removes the need of the current 'path_id' functionality
>     in multipath-tools

CC'ing Linux-nvme,

I've recently looked at multipathing support for nvme (and nvme over
fabrics) as well. For nvme the wwid equivalent is the nsid (namespace
identifier). I'm wandering if we can have better abstraction for
user-space so it won't need to change its behavior for scsi/nvme.
The same applies for the the timeout attribute for example which
assumes scsi device sysfs structure.

> b) leverage topology information from scsi_dh_alua (which we will
>     have once my ALUA handler update is in) to detect the multipath
>     topology. This removes the need of a 'prio' infrastructure
>     in multipath-tools

This would require further attention for nvme.

> c) implement block or scsi events whenever a remote port becomes
>     unavailable. This removes the need of the 'path_checker'
>     functionality in multipath-tools.

I'd prefer if we'd have it in the block layer so we can have it for all
block drivers. Also, this assumes that port events are independent of
I/O. This assumption is incorrect in SRP for example which detects port
failures only by I/O errors (which makes path sensing a must).

> d) leverage these events to handle path-up/path-down events
>     in-kernel
> e) move the I/O redirection logic out of device-mapper proper
>     and use blk-mq to redirect I/O. This is still a bit of
>     hand-waving, and definitely would need discussion to figure
>     out if and how it can be achieved.
>     This is basically the same topic Mike Snitzer proposed, but
>     coming from a different angle.

Another (adjacent) topic is multipath performance with blk-mq.

As I said, I've been looking at nvme multipathing support and
initial measurements show huge contention on the multipath lock
which really defeats the entire point of blk-mq...

I have yet to report this as my work is still in progress. I'm not sure
if it's a topic on it's own but I'd love to talk about that as well...

> But in the end we should be able to do strip down the current (rather
> complex) multipath-tools to just handle topology changes; everything
> else will be done internally.

I'd love to see that happening.

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Hannes Reinecke]

On 01/13/2016 11:50 AM, Sagi Grimberg wrote:
>
>> Hi all,
>>
>> I'd like to attend LSF/MM and would like to present my ideas for a
>> multipath redesign.
>>
>> The overall idea is to break up the centralized multipath handling in
>> device-mapper (and multipath-tools) and delegate to the appropriate
>> sub-systems.
>
> I agree that would be very useful. Great topic. I'd like to attend
> this talk as well.
>
>>
>> Individually the plan is:
>> a) use the 'wwid' sysfs attribute to detect multipath devices;
>>     this removes the need of the current 'path_id' functionality
>>     in multipath-tools
>
> CC'ing Linux-nvme,
>
> I've recently looked at multipathing support for nvme (and nvme over
> fabrics) as well. For nvme the wwid equivalent is the nsid (namespace
> identifier). I'm wandering if we can have better abstraction for
> user-space so it won't need to change its behavior for scsi/nvme.
> The same applies for the the timeout attribute for example which
> assumes scsi device sysfs structure.
>
My idea for this is to lookup the sysfs attribute directly from 
multipath-tools. As such we would need to have some transport 
information in multipath so that we know where to find it.
And with that we should easily able to accomodate NVMe, provided the 
nsid is displayed somewhere in sysfs.

>> b) leverage topology information from scsi_dh_alua (which we will
>>     have once my ALUA handler update is in) to detect the multipath
>>     topology. This removes the need of a 'prio' infrastructure
>>     in multipath-tools
>
> This would require further attention for nvme.
>
Indeed. But then I'm not sure how multipath topology would be 
represented in NVMe; we would need some way of transmitting the 
topology information.
Easiest would be to leverage VPD device information; so we only need 
the equivalent of REPORT TARGET PORT GROUPS to implement an 
ALUA-like scenario.

>> c) implement block or scsi events whenever a remote port becomes
>>     unavailable. This removes the need of the 'path_checker'
>>     functionality in multipath-tools.
>
> I'd prefer if we'd have it in the block layer so we can have it for all
> block drivers. Also, this assumes that port events are independent of
> I/O. This assumption is incorrect in SRP for example which detects port
> failures only by I/O errors (which makes path sensing a must).
>
That's what I though initially, too.
But then we're facing a layering issue:
The path events are generated at the _transport_ level.
So for SCSI we have to do a redirection
transport layer->scsi layer->scsi ULD->block device
requiring us to implement for sets of callback functions.
Which I found rather pointless (and time consuming), so I opted for 
scsi events (like we have for UNIT ATTENTION) instead.

However, even now we're having two sets of events (block events and 
scsi events) with a certain overlap, so this really could do with a 
cleanup.

>> d) leverage these events to handle path-up/path-down events
>>     in-kernel
>> e) move the I/O redirection logic out of device-mapper proper
>>     and use blk-mq to redirect I/O. This is still a bit of
>>     hand-waving, and definitely would need discussion to figure
>>     out if and how it can be achieved.
>>     This is basically the same topic Mike Snitzer proposed, but
>>     coming from a different angle.
>
> Another (adjacent) topic is multipath performance with blk-mq.
>
> As I said, I've been looking at nvme multipathing support and
> initial measurements show huge contention on the multipath lock
> which really defeats the entire point of blk-mq...
>
> I have yet to report this as my work is still in progress. I'm not sure
> if it's a topic on it's own but I'd love to talk about that as well...
>
Oh, most definitely. There are some areas in blk-mq which need to be 
covered / implemented before we can even think of that (dynamic 
queue reconfiguration and disabled queue handling being the most 
prominent).

_And_ we have the problem of queue mapping (one queue per ITL nexus?
one queue per hardware queue per ITL nexus?) which might quickly 
lead to a queue number explosion if we've not careful.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Mike Snitzer]

On Wed, Jan 13 2016 at  5:50am -0500,
Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:
 
> Another (adjacent) topic is multipath performance with blk-mq.
> 
> As I said, I've been looking at nvme multipathing support and
> initial measurements show huge contention on the multipath lock
> which really defeats the entire point of blk-mq...
> 
> I have yet to report this as my work is still in progress. I'm not sure
> if it's a topic on it's own but I'd love to talk about that as well...

This sounds like you aren't actually using blk-mq for the top-level DM
multipath queue.  And your findings contradicts what I heard from Keith
Busch when I developed request-based DM's blk-mq support, from commit 
bfebd1cdb497 ("dm: add full blk-mq support to request-based DM"):

     "Just providing a performance update. All my fio tests are getting
      roughly equal performance whether accessed through the raw block
      device or the multipath device mapper (~470k IOPS). I could only push
      ~20% of the raw iops through dm before this conversion, so this latest
      tree is looking really solid from a performance standpoint."

> >But in the end we should be able to do strip down the current (rather
> >complex) multipath-tools to just handle topology changes; everything
> >else will be done internally.
> 
> I'd love to see that happening.

Honestly, this needs to be a hardened plan that is hashed out _before_
LSF and then findings presented.  It is a complete waste of time to
debate nuance with Hannes in a one hour session.

Until I implemented the above DM core changes hch and Hannes were very
enthusiastic to throw away the existing DM multipath and multipath-tools
code (the old .request_fn queue lock bottleneck being the straw that
broke the camel's back).  Seems Hannes' enthusiasm hasn't tempered but
his hand-waving is still in full form.

Details matter.  I have no doubts aspects of what we have could be
improved but I really fail to see how moving multipathing to blk-mq is a
constructive way forward.

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Sagi Grimberg]

> This sounds like you aren't actually using blk-mq for the top-level DM
> multipath queue.

Hmm. I turned on /sys/module/dm_mod/parameters/use_blk_mq and indeed
saw a significant performance improvement. Anything else I was missing?

> And your findings contradicts what I heard from Keith
> Busch when I developed request-based DM's blk-mq support, from commit
> bfebd1cdb497 ("dm: add full blk-mq support to request-based DM"):
>
>       "Just providing a performance update. All my fio tests are getting
>        roughly equal performance whether accessed through the raw block
>        device or the multipath device mapper (~470k IOPS). I could only push
>        ~20% of the raw iops through dm before this conversion, so this latest
>        tree is looking really solid from a performance standpoint."

I too see ~500K IOPs, but my nvme can push ~1500K IOPs...
Its a simple nvme loopback [1] backed by null_blk.

[1]:
http://lists.infradead.org/pipermail/linux-nvme/2015-November/003001.html
http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/nvme-loop.2

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Mike Snitzer]

On Wed, Jan 13 2016 at 11:06am -0500,
Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:

> 
> >This sounds like you aren't actually using blk-mq for the top-level DM
> >multipath queue.
> 
> Hmm. I turned on /sys/module/dm_mod/parameters/use_blk_mq and indeed
> saw a significant performance improvement. Anything else I was missing?

You can enable CONFIG_DM_MQ_DEFAULT so you don't need to manually set
use_blk_mq.

> >And your findings contradicts what I heard from Keith
> >Busch when I developed request-based DM's blk-mq support, from commit
> >bfebd1cdb497 ("dm: add full blk-mq support to request-based DM"):
> >
> >      "Just providing a performance update. All my fio tests are getting
> >       roughly equal performance whether accessed through the raw block
> >       device or the multipath device mapper (~470k IOPS). I could only push
> >       ~20% of the raw iops through dm before this conversion, so this latest
> >       tree is looking really solid from a performance standpoint."
> 
> I too see ~500K IOPs, but my nvme can push ~1500K IOPs...
> Its a simple nvme loopback [1] backed by null_blk.
> 
> [1]:
> http://lists.infradead.org/pipermail/linux-nvme/2015-November/003001.html
> http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/nvme-loop.2

OK, so you're only getting 1/3 of the throughput.  Time for us to hunt
down the bottleneck (before real devices hit it).

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Sagi Grimberg]

>> Hmm. I turned on /sys/module/dm_mod/parameters/use_blk_mq and indeed
>> saw a significant performance improvement. Anything else I was missing?
>
> You can enable CONFIG_DM_MQ_DEFAULT so you don't need to manually set
> use_blk_mq.

I do, I started out with the manual option to test the improvements but
got tired of it very quickly :)

> OK, so you're only getting 1/3 of the throughput.  Time for us to hunt
> down the bottleneck (before real devices hit it).

I have some initial instrumentation analysis so I'll be happy if we can
work on that (probably can free this thread and move it to dm-devel).

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Hannes Reinecke]

On 01/13/2016 04:42 PM, Mike Snitzer wrote:
> On Wed, Jan 13 2016 at  5:50am -0500,
> Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:
>
>> Another (adjacent) topic is multipath performance with blk-mq.
>>
>> As I said, I've been looking at nvme multipathing support and
>> initial measurements show huge contention on the multipath lock
>> which really defeats the entire point of blk-mq...
>>
>> I have yet to report this as my work is still in progress. I'm not sure
>> if it's a topic on it's own but I'd love to talk about that as well...
>
> This sounds like you aren't actually using blk-mq for the top-level DM
> multipath queue.  And your findings contradicts what I heard from Keith
> Busch when I developed request-based DM's blk-mq support, from commit
> bfebd1cdb497 ("dm: add full blk-mq support to request-based DM"):
>
>       "Just providing a performance update. All my fio tests are getting
>        roughly equal performance whether accessed through the raw block
>        device or the multipath device mapper (~470k IOPS). I could only push
>        ~20% of the raw iops through dm before this conversion, so this latest
>        tree is looking really solid from a performance standpoint."
>
>>> But in the end we should be able to do strip down the current (rather
>>> complex) multipath-tools to just handle topology changes; everything
>>> else will be done internally.
>>
>> I'd love to see that happening.
>
> Honestly, this needs to be a hardened plan that is hashed out _before_
> LSF and then findings presented.  It is a complete waste of time to
> debate nuance with Hannes in a one hour session.
>
> Until I implemented the above DM core changes hch and Hannes were very
> enthusiastic to throw away the existing DM multipath and multipath-tools
> code (the old .request_fn queue lock bottleneck being the straw that
> broke the camel's back).  Seems Hannes' enthusiasm hasn't tempered but
> his hand-waving is still in full form.
>
> Details matter.  I have no doubts aspects of what we have could be
> improved but I really fail to see how moving multipathing to blk-mq is a
> constructive way forward.
>
So what is your plan?
Move the full blk-mq infrastructure into device-mapper?

 From my perspective, blk-mq and multipath I/O handling have a lot 
in common (the ->map_queue callback is in effect the same ->map_rq 
does), so I still think it should be possible to leverage that directly.
But for that to happen we would need to address some of the 
mentioned issues like individual queue failures and dynamic queue 
remapping; my hope is that they'll be implemented in the course of 
NVMe over fabrics.

Also note that my proposal is more with the infrastructure 
surrounding multipathing (ie topology detection and setup), so it's 
somewhat orthogonal to your proposal.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Mike Snitzer]

On Wed, Jan 13 2016 at 11:18am -0500,
Hannes Reinecke <hare@suse.de> wrote:

> On 01/13/2016 04:42 PM, Mike Snitzer wrote:
> >On Wed, Jan 13 2016 at  5:50am -0500,
> >Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:
> >
> >>Another (adjacent) topic is multipath performance with blk-mq.
> >>
> >>As I said, I've been looking at nvme multipathing support and
> >>initial measurements show huge contention on the multipath lock
> >>which really defeats the entire point of blk-mq...
> >>
> >>I have yet to report this as my work is still in progress. I'm not sure
> >>if it's a topic on it's own but I'd love to talk about that as well...
> >
> >This sounds like you aren't actually using blk-mq for the top-level DM
> >multipath queue.  And your findings contradicts what I heard from Keith
> >Busch when I developed request-based DM's blk-mq support, from commit
> >bfebd1cdb497 ("dm: add full blk-mq support to request-based DM"):
> >
> >      "Just providing a performance update. All my fio tests are getting
> >       roughly equal performance whether accessed through the raw block
> >       device or the multipath device mapper (~470k IOPS). I could only push
> >       ~20% of the raw iops through dm before this conversion, so this latest
> >       tree is looking really solid from a performance standpoint."
> >
> >>>But in the end we should be able to do strip down the current (rather
> >>>complex) multipath-tools to just handle topology changes; everything
> >>>else will be done internally.
> >>
> >>I'd love to see that happening.
> >
> >Honestly, this needs to be a hardened plan that is hashed out _before_
> >LSF and then findings presented.  It is a complete waste of time to
> >debate nuance with Hannes in a one hour session.
> >
> >Until I implemented the above DM core changes hch and Hannes were very
> >enthusiastic to throw away the existing DM multipath and multipath-tools
> >code (the old .request_fn queue lock bottleneck being the straw that
> >broke the camel's back).  Seems Hannes' enthusiasm hasn't tempered but
> >his hand-waving is still in full form.
> >
> >Details matter.  I have no doubts aspects of what we have could be
> >improved but I really fail to see how moving multipathing to blk-mq is a
> >constructive way forward.
> >
> So what is your plan?
> Move the full blk-mq infrastructure into device-mapper?

1.
Identify what the bottleneck(s) are in current request-based DM blk-mq
support (could be training top-level blk_mq request_queue capabilities
based on underlying devices). 

2.
Get blk-mq to be the primary mode of operation (scsi-mq has a role here)
and then eliminate/deprecate the old .request_fn IO path in blk-core.
- this is a secondary concern, DM can happily continue to carry all
permutations of request_fn on blk-mq path(s), blk-mq on request_fn
path(s) and, blk-mq on blk-mq path(s)... but maybe a start is to make
the top-level request-based DM queue _only_ blk-mq -- so effectively 
set CONFIG_DM_MQ_DEFAULT=Y (and eliminate code that supports
CONFIG_DM_MQ_DEFAULT=N).

IMHO, we don't yet have justification to warrant the relatively drastic
change you're floating (pushing multipathing down into blk-mq).

If/when justification is made we'll go from there.

> From my perspective, blk-mq and multipath I/O handling have a lot in
> common (the ->map_queue callback is in effect the same ->map_rq
> does), so I still think it should be possible to leverage that
> directly.
> But for that to happen we would need to address some of the
> mentioned issues like individual queue failures and dynamic queue
> remapping; my hope is that they'll be implemented in the course of
> NVMe over fabrics.
> 
> Also note that my proposal is more with the infrastructure
> surrounding multipathing (ie topology detection and setup), so it's
> somewhat orthogonal to your proposal.

Sure, it is probabky best if focus is placed on where our current
offering can be incrementally improved.  If that means pushing some
historically userspace (multipath-tools) responsibilities down to the
kernel then we can look at it.

What I want to avoid is a shotgun blast of drastic changes.  That
doesn't serve a _very_ enterprise-oriented layer well at all.

Re: [dm-devel] [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Alasdair G Kergon]

On Wed, Jan 13, 2016 at 10:10:43AM +0100, Hannes Reinecke wrote:
> The overall idea is to break up the centralized multipath handling in 
> device-mapper (and multipath-tools) and delegate to the appropriate 
> sub-systems.
>
> Individually the plan is:

Could we start to drill down into each of these and categorise them in
terms of which parts of the stack are involved in the proposed change
and prioritise them in terms of likely amount of work and
cost/benefit/risk?  Some of them probably need some prototyping and
experimentation.

Alasdair

Re: [dm-devel] [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Hannes Reinecke]

On 01/13/2016 12:08 PM, Alasdair G Kergon wrote:
> On Wed, Jan 13, 2016 at 10:10:43AM +0100, Hannes Reinecke wrote:
>> The overall idea is to break up the centralized multipath handling in
>> device-mapper (and multipath-tools) and delegate to the appropriate
>> sub-systems.
>>
>> Individually the plan is:
>
> Could we start to drill down into each of these and categorise them in
> terms of which parts of the stack are involved in the proposed change
> and prioritise them in terms of likely amount of work and
> cost/benefit/risk?  Some of them probably need some prototyping and
> experimentation.
>
Sure.
That's why I proposed it as a discussion topic :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [dm-devel] [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Alasdair G Kergon]

On Wed, Jan 13, 2016 at 12:17:27PM +0100, Hannes Reinecke wrote:
> That's why I proposed it as a discussion topic :-)

No need to wait for LSF - we have mailing lists:)

Alasdair

Re: [dm-devel] [LSF/MM ATTEND][LSF/MM TOPIC] Multipath redesign

[Benjamin Marzinski]

On Wed, Jan 13, 2016 at 10:10:43AM +0100, Hannes Reinecke wrote:
> Hi all,
> 
> I'd like to attend LSF/MM and would like to present my ideas for a multipath
> redesign.
> 
> The overall idea is to break up the centralized multipath handling in
> device-mapper (and multipath-tools) and delegate to the appropriate
> sub-systems.
> 
> Individually the plan is:
> a) use the 'wwid' sysfs attribute to detect multipath devices;
>    this removes the need of the current 'path_id' functionality
>    in multipath-tools

If all the devices that we support advertise their WWID through sysfs,
I'm all for this. Not needing to worry about callouts or udev sounds
great.

> b) leverage topology information from scsi_dh_alua (which we will
>    have once my ALUA handler update is in) to detect the multipath
>    topology. This removes the need of a 'prio' infrastructure
>    in multipath-tools

What about devices that don't use alua? Or users who want to be able to
pick a specific path to prefer? While I definitely prefer simple, we
can't drop real funtionality to get there. Have you posted your
scsi_dh_alua update somewhere?

I've recently had requests from users to
1. make a path with the TPGS pref bit set be in its own path group with
the highest priority
2. make the weighted prioritizer use persistent information to make its
choice, so its actually useful. This is to deal with the need to prefer a
specific path in a non-alua setup.

Some of the complexity with priorities is there out of necessity.

> c) implement block or scsi events whenever a remote port becomes
>    unavailable. This removes the need of the 'path_checker'
>    functionality in multipath-tools.

I'm not convinced that we will be able to find out when paths come back
online in all cases without some sort of actual polling. Again, I'd love
this to be simpler, but asking all the types of storage we plan to
support to notify us when they are up and down may not be realistic.

> d) leverage these events to handle path-up/path-down events
>    in-kernel

If polling is necessary, I'd rather it be done in userspace. Personally,
I think the checker code is probably the least obectionable part of the
multipath-tools (It's getting all the device information to set up the
devices in the first place and coordinating with uevents that's really
ugly, IMHO).

-Ben