Re: [ofa-general][PATCH 3/4] SRP fail-over faster

[David Dillow <dillowda-1Heg1YXhbW8@public.gmane.org>]

On Fri, 2009-10-23 at 12:50 -0400, Vu Pham wrote:
> David Dillow wrote:
> I don't know much about multipath in ALUA mode.
> How would multipath driver (in ALUA mode) to switch path? (ie. basing on 
> what criteria?)

ALUA sets the priority of each path, and generally multipath is set to
round-robin among all paths of the same priority. So, paths going to the
primary controller of a LUN get the best priority and are used
preferentially over the backup paths. Once no more paths in a priority
group are active, the round-robin selector will fall back to the next
highest priority path group.

The multipath core will immediately fail a path if the lower layers
propagate an error up to it, such that an I/O request completes in
error. If it has failed the path, it will start sending requests down
alternate paths without waiting for the queue to drain on the first one.

ALUA is not like RDAC -- in ALUA, all paths are valid to use, but some
paths will give better performance. You do not necessarily need to give
the array a command to move the LUN to another controller, so there's no
reason to wait for a queue to drain.

At least that is the way I understand things, having picked my way
through the block layer, multipath core, and device handlers.

> Can you switch path manually in user mode (while there are commands 
> stucked in current active path)?

I've not tried, but give the above I don't see why not.

> Without this patch, all outstanding I/Os have to go thru error recovery 
> before being returned with error code so that dm-multipath fail-over.

I think we're talking about two separate things here -- I agree that the
idea of failing IO early when we've lost our local connection, or know
the target is not in the fabric, is a good one. I want a fast failure so
that I can immediately start using my alternate paths. I'll have to deal
with the timeouts on the requests in flight at some point, but they
don't hold back independent requests.

The difference of opinion seems to be in how long to wait after being
notified of a connection loss -- or the target leaving the fabric --
before we start kicking back errors at the SRP command dispatch handler.
I agree that it makes sense to wait a moment before forcing an RDAC path
change, as they seem to be slow. But I also want it to get out of the
way for my case, when I don't incur much of a penalty to immediately
light up my backup path.

> If you want to failing requests right away, you can just set 
> device_loss_timeout=1, others don't want dm-multipath to switch path 
> right away. That's a whole idea of these patches that I submitted

The thing is, I don't want to wait even 1 second to use my backup path,
and I don't want all of those requests going into a black hole for that
time, forcing me to wait for the SCSI timeout on requests that could
have been immediately processed. On our system, 1 second is up to 1500
MB of data transferred over this one connection, and waiting around
twiddling our thumbs for a single second can potentially cost 1.3
thousand trillion operations.
-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html