why command of multipath send reinstate message to all dm's paths

why command of multipath send reinstate message to all dm's paths

[Jiaojianbing]

[-- Attachment #1.1: Type: text/plain, Size: 1333 bytes --]

Dear Christophe,
when dm-105 is in one state of below, paths of dm-105 will change to active if we run command of multipath. I check code of multipath, it sends messge "reinstate_path pathname" to kernel in routine reinstate_paths when status of pathgroup = "PGSTATE_ENABLED/PGSTATE_UNDEF" and path's state = "PSTATE_FAILED".
why command of multipath do above action to all dm devices? actually, parts of these paths are already offline or failed which can't be recovered. Maybe we can check these devices's status by sending io to these sd device at first. according to return of io, multipath send reinstate to running devices and do nothing to failed devices?

# multipath -ll
36d0d04b100b8cba665a187f0000000f9 dm-105 HUAWEI  ,XSG1
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=enabled
  |- 17:0:0:101 sdcy 70:96   failed faulty offline
  |- 18:0:0:101 sdgw 132:192 failed faulty offline
  `- 19:0:0:101 sdku 67:288  failed faulty offline

# multipath -ll
36d0d04b100b8cba665a187f0000000f9 dm-105 HUAWEI  ,XSG1
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 17:0:0:101 sdcy 70:96   failed ready running
  |- 18:0:0:101 sdgw 132:192 failed ready running
  `- 19:0:0:101 sdku 67:288  failed ready running

[-- Attachment #1.2: Type: text/html, Size: 6370 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: why command of multipath send reinstate message to all dm's paths

[Martin Wilck]

Jiaojianbing,

On Wed, 2018-06-27 at 07:58 +0000, Jiaojianbing wrote:

> Dear Christophe,
> when dm-105 is in one state of below, paths of dm-105 will change to
> active if we run command of multipath.

Could you be more specific please? What multipath command did you run?
Which version of multipath-tools are you running?

>  I check code of multipath, it sends messge "reinstate_path pathname"
> to kernel in routine reinstate_paths when status of pathgroup =
> "PGSTATE_ENABLED/PGSTATE_UNDEF" and path's state = "PSTATE_FAILED".
> why command of multipath do above action to all dm devices? actually,
> parts of these paths are already offline or failed which can't be
> recovered. Maybe we can check these devices's status by sending io to
> these sd device at first. according to return of io, multipath send
> reinstate to running devices and do nothing to failed devices?

I see this code in reinstate_paths():

		vector_foreach_slot (pgp->paths, pp, j) {
			if (pp->state != PATH_UP &&
			    (pgp->status == PGSTATE_DISABLED ||
			     pgp->status == PGSTATE_ACTIVE))
				continue;

			if (pp->dmstate == PSTATE_FAILED) {
				if (dm_reinstate_path(mpp->alias, pp->dev_t))
					condlog(0, "%s: error reinstating",
						pp->dev);
			}
		}

The reinstate command is only sent for paths which are either in
PATH_UP state, or belong to an PGSTATE_ENABLED path group. I admit I'm
unsure why all we try to reinstate paths that we know are down. This is
13-year-old code.

Interstingly, the state of your paths changes from "faulty offline" to
"ready running". So it appears that these paths are actually _not_ down
 Just the reinstate seems has failed on them.

multipathd -v3 logs and possibly kernel logs would be helpful to
understand what was going on in that situation.

Regards
Martin

-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

答复: why command of multipath send reinstate message to all dm's paths

[Jiaojianbing]

> > Dear Christophe,
> > when dm-105 is in one state of below, paths of dm-105 will change to
> > active if we run command of multipath.
> 
> Could you be more specific please? What multipath command did you run?
> Which version of multipath-tools are you running?

command is "multipath", which can run in shell as below:
#multipath

And the version:  multipath-tools v0.4.9 (05/33, 2016)

> >  I check code of multipath, it sends messge "reinstate_path pathname"
> > to kernel in routine reinstate_paths when status of pathgroup =
> > "PGSTATE_ENABLED/PGSTATE_UNDEF" and path's state = "PSTATE_FAILED".
> > why command of multipath do above action to all dm devices? actually,
> > parts of these paths are already offline or failed which can't be
> > recovered. Maybe we can check these devices's status by sending io to
> > these sd device at first. according to return of io, multipath send
> > reinstate to running devices and do nothing to failed devices?
> 
> I see this code in reinstate_paths():
> 
> 		vector_foreach_slot (pgp->paths, pp, j) {
> 			if (pp->state != PATH_UP &&
> 			    (pgp->status == PGSTATE_DISABLED ||
> 			     pgp->status == PGSTATE_ACTIVE))
> 				continue;
> 
> 			if (pp->dmstate == PSTATE_FAILED) {
> 				if (dm_reinstate_path(mpp->alias, pp->dev_t))
> 					condlog(0, "%s: error reinstating",
> 						pp->dev);
> 			}
> 		}
> 
> The reinstate command is only sent for paths which are either in PATH_UP
> state, or belong to an PGSTATE_ENABLED path group. I admit I'm unsure why
> all we try to reinstate paths that we know are down. This is 13-year-old code.
> 
> Interstingly, the state of your paths changes from "faulty offline" to "ready
> running". So it appears that these paths are actually _not_ down  Just the
> reinstate seems has failed on them.
> 
> multipathd -v3 logs and possibly kernel logs would be helpful to understand
> what was going on in that situation.

    Sorry, maybe my two multipath status sample confused you. They are just sample. Actually, I run command "rescan-scsi-bus" to
clear all mapped scsi devices by iscsid in host when all of LUNS in remote IPSAN are removed. 
In process of running rescan-scsi-bus, if command "multipath" is running, the status of dm's path will change from 
failed to active in some moment as below. If IO is sent to dm-105, the process who sends io will be in D state.
# multipath -ll
36d0d04b100b8cba665a187f0000000f9 dm-105 HUAWEI  ,XSG1
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 18:0:0:101 sdku 67:288  active faulty running

  I want to know whether command "multipath" is reasonable in reinstate_paths(). And maybe we should not call "multipath" in process of running rescan-scsi-bus ?

> Regards
> Martin
> 
> --
> Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux
> GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG
> Nürnberg)


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: 答复: why command of multipath send reinstate message to all dm's paths

[Martin Wilck]

[I've added Hannes, Ben and Douglas to the recepient list to fill in
knowledge from the past that I may lack].

tl;dr summary: We've got 3 issues:

 1) Why does multipath, in reinstate_paths(), try to reinstate paths
which are known to be down?
 2) rescan-scsi-bus.sh can call "multipath" even if "-m" switch is not
used (that looks like a bug to me).
 3) In Jiaojianbing's environment, dead paths that have been removed on
the target and were already marked "offline" may appear as "running"
after rescan-scsi-bus.sh invocation.

Furthermore,
 4) perhaps rescan-scsi-bus.sh should replace suboptimal "multipath"
calls with multipathd cli commands (or better even, we multipath-tools
people should eventually finish the "delegate to multipathd" work).


On Thu, 2018-06-28 at 06:35 +0000, Jiaojianbing wrote:
> > > Dear Christophe,
> > > when dm-105 is in one state of below, paths of dm-105 will change
> > > to
> > > active if we run command of multipath.
> > 
> > Could you be more specific please? What multipath command did you
> > run?
> > Which version of multipath-tools are you running?
> 
> command is "multipath", which can run in shell as below:
> #multipath

... and if I understand correctly, originally the problem occured while
running rescan_scsi_bus.sh. Please also state the version of sg3_utils
you are using.

> 
> And the version:  multipath-tools v0.4.9 (05/33, 2016)

Well, that's ancient. But latest multipath-tools still has the same
code.

> 
> > >  I check code of multipath, it sends messge "reinstate_path
> > > pathname"
> > > to kernel in routine reinstate_paths when status of pathgroup =
> > > "PGSTATE_ENABLED/PGSTATE_UNDEF" and path's state =
> > > "PSTATE_FAILED".
> > > why command of multipath do above action to all dm devices?
> > > actually,
> > > parts of these paths are already offline or failed which can't be
> > > recovered. Maybe we can check these devices's status by sending
> > > io to
> > > these sd device at first. according to return of io, multipath
> > > send
> > > reinstate to running devices and do nothing to failed devices?
> > 
> > I see this code in reinstate_paths():
> > 
> > 		vector_foreach_slot (pgp->paths, pp, j) {
> > 			if (pp->state != PATH_UP &&
> > 			    (pgp->status == PGSTATE_DISABLED ||
> > 			     pgp->status == PGSTATE_ACTIVE))
> > 				continue;
> > 
> > 			if (pp->dmstate == PSTATE_FAILED) {
> > 				if (dm_reinstate_path(mpp->alias, pp-
> > >dev_t))
> > 					condlog(0, "%s: error
> > reinstating",
> > 						pp->dev);
> > 			}
> > 		}
> > 
> > The reinstate command is only sent for paths which are either in
> > PATH_UP
> > state, or belong to an PGSTATE_ENABLED path group. I admit I'm
> > unsure why
> > all we try to reinstate paths that we know are down. This is 13-
> > year-old code.
> > 
> > Interstingly, the state of your paths changes from "faulty offline"
> > to "ready
> > running". So it appears that these paths are actually _not_
> > down  Just the
> > reinstate seems has failed on them.
> > 
> > multipathd -v3 logs and possibly kernel logs would be helpful to
> > understand
> > what was going on in that situation.
> 
>     Sorry, maybe my two multipath status sample confused you. They
> are just sample. Actually, I run command "rescan-scsi-bus" to
> clear all mapped scsi devices by iscsid in host when all of LUNS in
> remote IPSAN are removed. 
> In process of running rescan-scsi-bus, if command "multipath" is
> running, the status of dm's path will change from 
> failed to active in some moment as below. If IO is sent to dm-105,
> the process who sends io will be in D state.
> # multipath -ll
> 36d0d04b100b8cba665a187f0000000f9 dm-105 HUAWEI  ,XSG1
> size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
> `-+- policy='service-time 0' prio=1 status=active
>   `- 18:0:0:101 sdku 67:288  active faulty running

The strange part here is that the device is considered "running". This
is the state of the kernel device. If the LUNs are actually _removed_
as you say, the device should be gone, or at least marked "offline". 

Apparently the SCSI bus SCAN via iSCSI still showed the LUN in a
workable state. For multipath this translates to PATH_UP. Thus even if
the above code didn't have the (pgp->status == PGSTATE_DISABLED || pgp-
>status == PGSTATE_ACTIVE) clause, the reinstate would have been
attempted by multipath. This looks like a low-level problem in your
SCSI or iSCSI layer to me.

This looks like the actual problem to me. multipath aside, if the path
appears to be "running", any Linux process could try to send IO down to
it and be stuck, as you say.

> 
>   I want to know whether command "multipath" is reasonable in
> reinstate_paths().


> > And maybe we should not call "multipath" in process of running
> rescan-scsi-bus ?

Normally rescan-scsi-bus.sh should call "multipath" only if the 
"-m|--multipath" switch was used. I quickly scanned through the code
and didn't find a call to "multipath" (with no options) which wasn't
guarded by the [ -n "$mp_enable" ] condition. (FTR: there is a call to
"multipath -f" from main->flushmpaths if "-f|--flush" is set).

Again, please double-check your version of sg3_utils, and perhaps run
"bash -x rescan-scsi-bus.sh" to figure out the call chain which runs
the "multipath" command.

Thanks,
Martin

> 
> > Regards
> > Martin
> > 
> > --
> > Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE
> > Linux
> > GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284
> > (AG
> > Nürnberg)
> 
> 

-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: 答复: why command of multipath send reinstate message to all dm's paths

[Douglas Gilbert]

[-- Attachment #1: Type: text/plain, Size: 6105 bytes --]

On 2018-06-28 09:38 AM, Martin Wilck wrote:
> [I've added Hannes, Ben and Douglas to the recepient list to fill in
> knowledge from the past that I may lack].
> 
> tl;dr summary: We've got 3 issues:
> 
>   1) Why does multipath, in reinstate_paths(), try to reinstate paths
> which are known to be down?
>   2) rescan-scsi-bus.sh can call "multipath" even if "-m" switch is not
> used (that looks like a bug to me).
>   3) In Jiaojianbing's environment, dead paths that have been removed on
> the target and were already marked "offline" may appear as "running"
> after rescan-scsi-bus.sh invocation.
> 
> Furthermore,
>   4) perhaps rescan-scsi-bus.sh should replace suboptimal "multipath"
> calls with multipathd cli commands (or better even, we multipath-tools
> people should eventually finish the "delegate to multipathd" work).
> 
> 
> On Thu, 2018-06-28 at 06:35 +0000, Jiaojianbing wrote:
>>>> Dear Christophe,
>>>> when dm-105 is in one state of below, paths of dm-105 will change
>>>> to
>>>> active if we run command of multipath.
>>>
>>> Could you be more specific please? What multipath command did you
>>> run?
>>> Which version of multipath-tools are you running?
>>
>> command is "multipath", which can run in shell as below:
>> #multipath
> 
> ... and if I understand correctly, originally the problem occured while
> running rescan_scsi_bus.sh. Please also state the version of sg3_utils
> you are using.
> 
>>
>> And the version:  multipath-tools v0.4.9 (05/33, 2016)
> 
> Well, that's ancient. But latest multipath-tools still has the same
> code.
> 
>>
>>>>   I check code of multipath, it sends messge "reinstate_path
>>>> pathname"
>>>> to kernel in routine reinstate_paths when status of pathgroup =
>>>> "PGSTATE_ENABLED/PGSTATE_UNDEF" and path's state =
>>>> "PSTATE_FAILED".
>>>> why command of multipath do above action to all dm devices?
>>>> actually,
>>>> parts of these paths are already offline or failed which can't be
>>>> recovered. Maybe we can check these devices's status by sending
>>>> io to
>>>> these sd device at first. according to return of io, multipath
>>>> send
>>>> reinstate to running devices and do nothing to failed devices?
>>>
>>> I see this code in reinstate_paths():
>>>
>>> 		vector_foreach_slot (pgp->paths, pp, j) {
>>> 			if (pp->state != PATH_UP &&
>>> 			    (pgp->status == PGSTATE_DISABLED ||
>>> 			     pgp->status == PGSTATE_ACTIVE))
>>> 				continue;
>>>
>>> 			if (pp->dmstate == PSTATE_FAILED) {
>>> 				if (dm_reinstate_path(mpp->alias, pp-
>>>> dev_t))
>>> 					condlog(0, "%s: error
>>> reinstating",
>>> 						pp->dev);
>>> 			}
>>> 		}
>>>
>>> The reinstate command is only sent for paths which are either in
>>> PATH_UP
>>> state, or belong to an PGSTATE_ENABLED path group. I admit I'm
>>> unsure why
>>> all we try to reinstate paths that we know are down. This is 13-
>>> year-old code.
>>>
>>> Interstingly, the state of your paths changes from "faulty offline"
>>> to "ready
>>> running". So it appears that these paths are actually _not_
>>> down  Just the
>>> reinstate seems has failed on them.
>>>
>>> multipathd -v3 logs and possibly kernel logs would be helpful to
>>> understand
>>> what was going on in that situation.
>>
>>      Sorry, maybe my two multipath status sample confused you. They
>> are just sample. Actually, I run command "rescan-scsi-bus" to
>> clear all mapped scsi devices by iscsid in host when all of LUNS in
>> remote IPSAN are removed.
>> In process of running rescan-scsi-bus, if command "multipath" is
>> running, the status of dm's path will change from
>> failed to active in some moment as below. If IO is sent to dm-105,
>> the process who sends io will be in D state.
>> # multipath -ll
>> 36d0d04b100b8cba665a187f0000000f9 dm-105 HUAWEI  ,XSG1
>> size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
>> `-+- policy='service-time 0' prio=1 status=active
>>    `- 18:0:0:101 sdku 67:288  active faulty running
> 
> The strange part here is that the device is considered "running". This
> is the state of the kernel device. If the LUNs are actually _removed_
> as you say, the device should be gone, or at least marked "offline".
> 
> Apparently the SCSI bus SCAN via iSCSI still showed the LUN in a
> workable state. For multipath this translates to PATH_UP. Thus even if
> the above code didn't have the (pgp->status == PGSTATE_DISABLED || pgp-
>> status == PGSTATE_ACTIVE) clause, the reinstate would have been
> attempted by multipath. This looks like a low-level problem in your
> SCSI or iSCSI layer to me.
> 
> This looks like the actual problem to me. multipath aside, if the path
> appears to be "running", any Linux process could try to send IO down to
> it and be stuck, as you say.
> 
>>
>>    I want to know whether command "multipath" is reasonable in
>> reinstate_paths().
> 
> 
>>> And maybe we should not call "multipath" in process of running
>> rescan-scsi-bus ?
> 
> Normally rescan-scsi-bus.sh should call "multipath" only if the
> "-m|--multipath" switch was used. I quickly scanned through the code
> and didn't find a call to "multipath" (with no options) which wasn't
> guarded by the [ -n "$mp_enable" ] condition. (FTR: there is a call to
> "multipath -f" from main->flushmpaths if "-f|--flush" is set).
> 
> Again, please double-check your version of sg3_utils, and perhaps run
> "bash -x rescan-scsi-bus.sh" to figure out the call chain which runs
> the "multipath" command.
> 
> Thanks,
> Martin
> 

Hi,
My upstream version of rescan-scsi-bus.sh is attached. The last change was
the --ignore-rev option from Gris Ge <fge@redhat.com>. He has sent several
cleanups in the last year, usually via Hannes' github site for sg3_utils.

My ChangeLog entry to that script (since sg3_utils 1.42) is:

   - rescan-scsi-bus.sh: harden code
     - fixes from Suse; bump version
     - bump version to 20180615
     - add to install list in Makefile, hope it does
       not clash with other package providing it
     - add --ignore-rev to ignore revision change

If there are no further changes it will be like that in sg3_utils-1.43
revision 780.

Doug Gilbert


[-- Attachment #2: rescan-scsi-bus.sh --]
[-- Type: application/x-shellscript, Size: 39068 bytes --]

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

答复: 答复: why command of multipath send reinstate message to all dm's paths

[Jiaojianbing]

> [I've added Hannes, Ben and Douglas to the recepient list to fill in knowledge
> from the past that I may lack].
> 
> tl;dr summary: We've got 3 issues:
> 
>  1) Why does multipath, in reinstate_paths(), try to reinstate paths which are
> known to be down?
>  2) rescan-scsi-bus.sh can call "multipath" even if "-m" switch is not used (that
> looks like a bug to me).
>  3) In Jiaojianbing's environment, dead paths that have been removed on the
> target and were already marked "offline" may appear as "running"
> after rescan-scsi-bus.sh invocation.
> 
> Furthermore,
>  4) perhaps rescan-scsi-bus.sh should replace suboptimal "multipath"
> calls with multipathd cli commands (or better even, we multipath-tools people
> should eventually finish the "delegate to multipathd" work).

It's my negligence, command "multipath" is not the one in rescan-scsi-bus.sh, but another one called every five minutes by process "test.sh". 
It means there are two processes, one is rescan-scsi-bus.sh, another is test.sh which call multipath every five minutes.

In the scene, rescan-scsi-bus.sh will consume more larger time than the scene without calling "test.sh". The reason is that all "systemd-udevd" process 
are in D state who send io to device mapper device, such as dm-105. 

so it can be 2 issues:
 1) Why does multipath, in reinstate_paths(), try to reinstate paths which are known to be down?
 2) when run script "rescan-scsi-bus.sh", another process call command "multipath" may make mistake.

> On Thu, 2018-06-28 at 06:35 +0000, Jiaojianbing wrote:
> > > > Dear Christophe,
> > > > when dm-105 is in one state of below, paths of dm-105 will change
> > > > to active if we run command of multipath.
> > >
> > > Could you be more specific please? What multipath command did you
> > > run?
> > > Which version of multipath-tools are you running?
> >
> > command is "multipath", which can run in shell as below:
> > #multipath
> 
> ... and if I understand correctly, originally the problem occured while running
> rescan_scsi_bus.sh. Please also state the version of sg3_utils you are using.

And the version of sg3_utils: sg3_utils-libs-1.37-14.x86_64;
According to above description, the problem may be made by adding another process calling command "multipath" in period. 

> >
> > And the version:  multipath-tools v0.4.9 (05/33, 2016)
> 
> Well, that's ancient. But latest multipath-tools still has the same code.
> 
> >
> > > >  I check code of multipath, it sends messge "reinstate_path
> > > > pathname"
> > > > to kernel in routine reinstate_paths when status of pathgroup =
> > > > "PGSTATE_ENABLED/PGSTATE_UNDEF" and path's state =
> > > > "PSTATE_FAILED".
> > > > why command of multipath do above action to all dm devices?
> > > > actually,
> > > > parts of these paths are already offline or failed which can't be
> > > > recovered. Maybe we can check these devices's status by sending io
> > > > to these sd device at first. according to return of io, multipath
> > > > send reinstate to running devices and do nothing to failed
> > > > devices?
> > >
> > > I see this code in reinstate_paths():
> > >
> > > 		vector_foreach_slot (pgp->paths, pp, j) {
> > > 			if (pp->state != PATH_UP &&
> > > 			    (pgp->status == PGSTATE_DISABLED ||
> > > 			     pgp->status == PGSTATE_ACTIVE))
> > > 				continue;
> > >
> > > 			if (pp->dmstate == PSTATE_FAILED) {
> > > 				if (dm_reinstate_path(mpp->alias, pp-
> > > >dev_t))
> > > 					condlog(0, "%s: error
> > > reinstating",
> > > 						pp->dev);
> > > 			}
> > > 		}
> > >
> > > The reinstate command is only sent for paths which are either in
> > > PATH_UP state, or belong to an PGSTATE_ENABLED path group. I admit
> > > I'm unsure why all we try to reinstate paths that we know are down.
> > > This is 13- year-old code.
> > >
> > > Interstingly, the state of your paths changes from "faulty offline"
> > > to "ready
> > > running". So it appears that these paths are actually _not_ down
> > > Just the reinstate seems has failed on them.
> > >
> > > multipathd -v3 logs and possibly kernel logs would be helpful to
> > > understand what was going on in that situation.
> >
> >     Sorry, maybe my two multipath status sample confused you. They are
> > just sample. Actually, I run command "rescan-scsi-bus" to clear all
> > mapped scsi devices by iscsid in host when all of LUNS in remote IPSAN
> > are removed.
> > In process of running rescan-scsi-bus, if command "multipath" is
> > running, the status of dm's path will change from failed to active in
> > some moment as below. If IO is sent to dm-105, the process who sends
> > io will be in D state.
> > # multipath -ll
> > 36d0d04b100b8cba665a187f0000000f9 dm-105 HUAWEI  ,XSG1 size=1.0G
> > features='1 queue_if_no_path' hwhandler='0' wp=rw
> > `-+- policy='service-time 0' prio=1 status=active
> >   `- 18:0:0:101 sdku 67:288  active faulty running
> 
> The strange part here is that the device is considered "running". This is the
> state of the kernel device. If the LUNs are actually _removed_ as you say, the
> device should be gone, or at least marked "offline".
> 
> Apparently the SCSI bus SCAN via iSCSI still showed the LUN in a workable state.
> For multipath this translates to PATH_UP. Thus even if the above code didn't
> have the (pgp->status == PGSTATE_DISABLED || pgp-
> >status == PGSTATE_ACTIVE) clause, the reinstate would have been
> attempted by multipath. This looks like a low-level problem in your SCSI or iSCSI
> layer to me.
> 
> This looks like the actual problem to me. multipath aside, if the path appears to
> be "running", any Linux process could try to send IO down to it and be stuck, as
> you say.


> >
> >   I want to know whether command "multipath" is reasonable in
> > reinstate_paths().
> 
> 
> > > And maybe we should not call "multipath" in process of running
> > rescan-scsi-bus ?
> 
> Normally rescan-scsi-bus.sh should call "multipath" only if the "-m|--multipath"
> switch was used. I quickly scanned through the code and didn't find a call to
> "multipath" (with no options) which wasn't guarded by the [ -n "$mp_enable" ]
> condition. (FTR: there is a call to "multipath -f" from main->flushmpaths if
> "-f|--flush" is set).
> 
> Again, please double-check your version of sg3_utils, and perhaps run "bash -x
> rescan-scsi-bus.sh" to figure out the call chain which runs the "multipath"
> command.
> 
> Thanks,
> Martin
> 
> >
> > > Regards
> > > Martin
> > >
> > > --
> > > Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE
> > > Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB
> > > 21284 (AG
> > > Nürnberg)
> >
> >
> 
> --
> Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux
> GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG
> Nürnberg)


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: 答复: 答复: why command of multipath send reinstate message to all dm's paths

[Martin Wilck]

Jiaojianbing,

On Mon, 2018-07-02 at 01:06 +0000, Jiaojianbing wrote:
> > [I've added Hannes, Ben and Douglas to the recepient list to fill
> > in knowledge
> > from the past that I may lack].
> > 
> > tl;dr summary: We've got 3 issues:
> > 
> >  1) Why does multipath, in reinstate_paths(), try to reinstate
> > paths which are
> > known to be down?
> >  2) rescan-scsi-bus.sh can call "multipath" even if "-m" switch is
> > not used (that
> > looks like a bug to me).
> >  3) In Jiaojianbing's environment, dead paths that have been
> > removed on the
> > target and were already marked "offline" may appear as "running"
> > after rescan-scsi-bus.sh invocation.
> > 
> > Furthermore,
> >  4) perhaps rescan-scsi-bus.sh should replace suboptimal
> > "multipath"
> > calls with multipathd cli commands (or better even, we multipath-
> > tools people
> > should eventually finish the "delegate to multipathd" work).
> 
> It's my negligence, command "multipath" is not the one in rescan-
> scsi-bus.sh, but another one called every five minutes by process
> "test.sh". 
> It means there are two processes, one is rescan-scsi-bus.sh, another
> is test.sh which call multipath every five minutes.

Please describe your setup bottom-up. You have two scripts running
periodically, one calling "rescan-scsi-bus.sh" and one "multipath", and
they are (can be) running at the same time? How frequently are they
running? Be aware that "multipath" is not a monitoring command, it
basically causes a reconfiguration. It's not recommended to run it
periodically. Of course running "multipath" shouldn't cause a system
hang, but in your case I still think the root problem is that devices
that can't respond to IO are seen in "running" state by the kernel. If
that happens, other processes are allowed, actually supposed, to do
probing on these devices. But it's hard to say more without knowing
what exactly is going on.

Also, please consider updating to more recent version of the tools. dm-
devel a mailing list for discussing upstream issues, and your versions
of both multipath and sg3_utils are rather ancient. I guess you're
using some older distribution, in which case you may want to engage
with your distro's support team.

I don't think well make much progress without detailed logs of both the
kernel (please activate scsi logging with
MLCOMPLETE=1|ERROR=4|SCAN_BUS=4, run rescan_scsi_bus.sh with -d switch,
and set multipath verbosity of both  multipathd and "multipath" command
to 3 at least, and put results on a pastebin somewhere, and provide us
with links.

> In the scene, rescan-scsi-bus.sh will consume more larger time than
> the scene without calling "test.sh". The reason is that all "systemd-
> udevd" process 
> are in D state who send io to device mapper device, such as dm-105. 

If that's the case, please also run "udevadm -l debug" and provide udev
logs. We need to know which udev commands are hanging.

Martin

-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: 答复: 答复: 答复: why command of multipath send reinstate message to all dm's paths

[Martin Wilck]

On Mon, 2018-07-02 at 08:53 +0000, Jiaojianbing wrote:
> > Please describe your setup bottom-up. You have two scripts running
> > periodically,
> > one calling "rescan-scsi-bus.sh" and one "multipath", and they are
> > (can be)
> > running at the same time? How frequently are they running? Be aware
> > that
> > "multipath" is not a monitoring command, it basically causes a
> > reconfiguration.
> > It's not recommended to run it periodically. Of course running
> > "multipath"
> > shouldn't cause a system hang, but in your case I still think the
> > root problem is
> > that devices that can't respond to IO are seen in "running" state
> > by the kernel.
> > If that happens, other processes are allowed, actually supposed, to
> > do probing
> > on these devices. But it's hard to say more without knowing what
> > exactly is
> > going on.
> > 
> > Also, please consider updating to more recent version of the tools.
> > dm- devel a
> > mailing list for discussing upstream issues, and your versions of
> > both multipath
> > and sg3_utils are rather ancient. I guess you're using some older
> > distribution,
> > in which case you may want to engage with your distro's support
> > team.
> > 
> > I don't think well make much progress without detailed logs of both
> > the kernel
> > (please activate scsi logging with MLCOMPLETE=1|ERROR=4|SCAN_BUS=4,
> > run rescan_scsi_bus.sh with -d switch, and set multipath verbosity
> > of both
> > multipathd and "multipath" command to 3 at least, and put results
> > on a
> > pastebin somewhere, and provide us with links.
> > 
> 
> Dear martin and Doug Gilbert,
>    Thanks for your return. I descript all the scene and question
> below, and 6th question is the import thing I want to know.
> 
> 1、First, my scene is, there are 500 LUN in remote IPSAN, and several
> hosts connect to the IPSAN. Each host has found 500 dm devices. 
> I want to test the time for one host to rescan when we remove the 500
> LUNs map In remote IPSAN. That is the time to measure 
> for the host from 500 dm devices to zero dm devices by “rescan-scsi-
> bus.sh” script.
> 
> 2、There are two scripts running at the same time，one is “rescan-scsi-
> bus.sh”, who just run once ,but take large time
> (about serval hours) to run to complete when 500 LUNs are removed in
> remote IPSAN. The other script we called "test.sh", 
> who call "multipath" every five minutes as below: 
> 
> #! /bin/bash
> 
> while :
> do
>     multipath
> 	sleep 300
> done
> 
> 3、 if we run two scripts at the same time, “rescan-scsi-bus.sh” will
> take more larger time than running itself without "test.sh".
> And after complete of “rescan-scsi-bus.sh” , serval processes of
> "systemd-udevd" will be in D state as below. And D state of these
> processes 
> can't be recovered automatically. And also there will be leftover of
> multipath.
> 
> # ps aux |grep D
> root      2635  0.0  0.0      0     0 ?        D    Jul01   0:00
> [systemd-udevd]
> root      2682  0.0  0.0      0     0 ?        D    Jul01   0:00
> [systemd-udevd]
> root      2693  0.0  0.0      0     0 ?        D    Jul01   0:00
> [systemd-udevd]
> root      3069  0.0  0.0      0     0 ?        D    Jul01   0:00
> [systemd-udevd]
> root      3080  0.0  0.0      0     0 ?        D    Jul01   0:00
> [systemd-udevd]
> root      3517  0.0  0.0      0     0 ?        D    Jul01   0:00
> [systemd-udevd]
> root      3659  0.0  0.0      0     0 ?        D    Jul01   0:00
> [systemd-udevd]
> 
> and one of D process's stack:
> # cat /proc/3659 /stack 
> [<ffffffff8118e784>] __lock_page+0x74/0x90
> [<ffffffff811a0564>] truncate_inode_pages_range+0x704/0x740
> [<ffffffff811a05b5>] truncate_inode_pages+0x15/0x20
> [<ffffffff812519af>] kill_bdev+0x2f/0x40
> [<ffffffff81253464>] __blkdev_put+0x64/0x1a0
> [<ffffffff81253eee>] blkdev_put+0x4e/0x140
> [<ffffffff81254095>] blkdev_close+0x25/0x30
> [<ffffffff8121796c>] __fput+0xec/0x260
> [<ffffffff81217c1e>] ____fput+0xe/0x10
> [<ffffffff810b6867>] task_work_run+0xc7/0xe0
> [<ffffffff81095010>] do_exit+0x2e0/0xa50
> [<ffffffff810957ff>] do_group_exit+0x3f/0xa0
> [<ffffffff810a6d60>] get_signal_to_deliver+0x1d0/0x6e0
> [<ffffffff8102a527>] do_signal+0x57/0x6b0
> [<ffffffff8102abdf>] do_notify_resume+0x5f/0xb0
> [<ffffffff816c264f>] int_signal+0x12/0x17
> [<ffffffffffffffff>] 0xffffffffffffffff


Hmm, that stack reminds me of two situations Bart has reported earlier:

https://patchwork.kernel.org/patch/9261609/
https://patchwork.kernel.org/patch/9202331/

It doesn't seem to me that there has been agreement on Bart's proposed
fixes so far, but I may have missed it. Maybe Bart can comment on that.

It would be interesting to see full alt-sysrq-w output of that system.

What kernel have you been using?

> 
> Leftover (2 for sample, more than 2):
> 36e02861100592fcd0e1d8024000000ce dm-385 ,
> size=6.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
> `-+- policy='service-time 0' prio=0 status=enabled
>   |- #:#:#:# - #:# active faulty running
>   |- #:#:#:# - #:# active faulty running
>   |- #:#:#:# - #:# active faulty running
>   `- #:#:#:# - #:# active faulty running
> 36e02861100592fcd0e1d586400000069 dm-273 ,
> size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
> `-+- policy='service-time 0' prio=0 status=active
>   |- #:#:#:# - #:# failed faulty running
>   |- #:#:#:# - #:# failed faulty running
>   |- #:#:#:# - #:# failed faulty running
>   `- #:#:#:# - #:# failed faulty running
> 
> 4、if we run only "rescan-scsi-bus.sh " without "test.sh" script,
> there is no problem. And the time for " rescan-scsi-bus.sh " is more
> less.
>    
> 5、I'll check the last version according to you and Doug Gilbert's
> advice. And collect log.
> 
> 6、Before we do 5th step above, one conclusion maybe made that
> "multipath" called periodically in other script can't be ran in
> process of " rescan-scsi-bus.sh ", is that true?
>   I think , while " rescan-scsi-bus.sh " remove the paths and change
> to failed state, "multipath" reinstate paths' state . These two
> actions confused the state of paths ? 

I said already, I don't see the reason why multipathd reinstates paths
which are not UP, and I'm waiting for some of the people who have been
around longer than myself to comment.

Yet in your case, the path state is UP (aka "running"). At least this
seems to be the case temporarily, while rescan-scsi-bus.sh is running.
There's nothing wrong with multipathd trying to reinstate a path in UP
state, AFAICT, so we need to look elswhere for the root cause. The
above kernel stack is a high suspect.

Martin

-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: 答复: 答复: 答复: why command of multipath send reinstate message to all dm's paths

[Bart Van Assche]

On 07/02/18 02:20, Martin Wilck wrote:
> On Mon, 2018-07-02 at 08:53 +0000, Jiaojianbing wrote:
>> [ ... ]
>> and one of D process's stack:
>> # cat /proc/3659 /stack
>> [<ffffffff8118e784>] __lock_page+0x74/0x90
>> [<ffffffff811a0564>] truncate_inode_pages_range+0x704/0x740
>> [<ffffffff811a05b5>] truncate_inode_pages+0x15/0x20
>> [<ffffffff812519af>] kill_bdev+0x2f/0x40
>> [<ffffffff81253464>] __blkdev_put+0x64/0x1a0
>> [<ffffffff81253eee>] blkdev_put+0x4e/0x140
>> [<ffffffff81254095>] blkdev_close+0x25/0x30
>> [<ffffffff8121796c>] __fput+0xec/0x260
>> [<ffffffff81217c1e>] ____fput+0xe/0x10
>> [<ffffffff810b6867>] task_work_run+0xc7/0xe0
>> [<ffffffff81095010>] do_exit+0x2e0/0xa50
>> [<ffffffff810957ff>] do_group_exit+0x3f/0xa0
>> [<ffffffff810a6d60>] get_signal_to_deliver+0x1d0/0x6e0
>> [<ffffffff8102a527>] do_signal+0x57/0x6b0
>> [<ffffffff8102abdf>] do_notify_resume+0x5f/0xb0
>> [<ffffffff816c264f>] int_signal+0x12/0x17
>> [<ffffffffffffffff>] 0xffffffffffffffff
> 
> 
> Hmm, that stack reminds me of two situations Bart has reported earlier:
> 
> https://patchwork.kernel.org/patch/9261609/
> https://patchwork.kernel.org/patch/9202331/
> 
> It doesn't seem to me that there has been agreement on Bart's proposed
> fixes so far, but I may have missed it. Maybe Bart can comment on that.

Hello Martin,

The above call stack means that a block layer request got stuck. The two 
patches I posted before are not the right approach to fix this. As you 
maybe know rescan-scsi-bus.sh uses the sysfs "delete" attribute and 
writing into that attribute can trigger a deadlock. Have you noticed 
patch "Avoid that SCSI device removal through sysfs triggers a deadlock" 
(https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg74500.html)? 
If the above lockup is reproducible with a kernel v4.17 or later and 
with that patch applied, it would help if a dump of the attributes under 
/sys/kernel/debug/block could be provided.

Bart.