iscsi netboot failure in case of multiple interfaces

iscsi netboot failure in case of multiple interfaces

[Cathy Zhou]

Hi,

First I am sorry for the length of the email. Hopefully my description 
of the problem is clear. Any questions/suggestions are welcome. Please 
reply all as some of us on not on the alias.

-----

We are running into a failure with iscsi netboot with the following boot 
options:

       "... rd.luks=0 rd.lvm=0 rd.md=0 rd.dm=0 ip=dhcp 
netroot=iscsi:169.254.0.2::::iqn.2015-02.oracle.boot:uefi 
iscsi_param=node.session.timeo.replacement_timeout=6000"

On the system, there are two interfaces (say eth0 and eth1) which are 
able to get dhcp offers successfully, but only one of them (eth0) is 
able to reach the specified iscsi target. After some debugging, we 
believe we've found the root cause. Here is what happened:

1. eth0 successfully got the dhcp offer, the iscsiroot script was run 
and eventually ran "system-run" to start the "oneshot" iscsistart service.
2. Before step 1 succeeds, the iscsiroot script was also run for eth1 
and it checks the status of the first iscsistart service instance, which 
was still "activating". So the iscsistart service was restarted and that 
killed the first instance. But the second instance also fails because of 
the "existing session" error.

Here are the questions we have:

a. We found in the iscsiroot script, the iscsid service was started 
before the iscsistart service. Because of this, the creation of the mgmt 
ipc socket by iscsistart failed, and iscsi login session request was 
handled by iscsid instead. In step 2 above, the first iscsistart 
instance was killed but not the iscsid daemon, hence the "existing 
error" as the first login session still existed in iscsid.

The questions is if the iscsid service is really required by the 
iscisroot script? Because the existence of iscsid, after the second 
iscsistart instance, we saw unexpected iscsid service unavailability 
because iscsid is stopped by the iscsistart instance (iscsistart.c calls 
stop_event_loop() to stop event loop in order to exits itself, but since 
the MGMT_IPC_IMMEDIATE_STOP request was handled by iscsid, iscsid's 
event loop was stopped and iscsid exited instead).

The related error messages we saw in the log:

    "iscsistart[836]: iscsistart: Can not bind IPC socket
     iscsistart[836]: iscsistart: Could not setup mgmt ipc
     ...
     iscsiadm[970]: iscsiadm: can not connect to iSCSI daemon (111)!
     iscsiadm[970]: iscsiadm: initiator reported error (20 - could not 
connect to iscsid)
     ..."

I tried to change the iscsiroot.sh script to stop the iscsid service 
instead of restarting iscsid service, it seems it fixed our problem. But 
I am not sure if this fix has any side-effect.

b. If stopping iscsid service is not the ideal fix, I am wondering if we 
can mimic the legacy way (prior systemd) and start the iscsistart 
service independently for each interface. This means, iscsistart service 
will be started for each interface without affecting the iscsistart 
instances which are already run for other interfaces. It may mean the 
service name needs to include $netif to uniquely identify each instance.

Thanks very much! Looking forward to your suggestions!

- Cathy

Re: iscsi netboot failure in case of multiple interfaces

[Cathy Zhou]

Dracut experts,

A gentle reminder ...  Looking forward to gaining some insights on below 
questions.

Thanks
- Cathy

On 11/8/2017 10:55 AM, Cathy Zhou wrote:
> Hi,
>
> First I am sorry for the length of the email. Hopefully my description 
> of the problem is clear. Any questions/suggestions are welcome. Please 
> reply all as some of us on not on the alias.
>
> -----
>
> We are running into a failure with iscsi netboot with the following 
> boot options:
>
>       "... rd.luks=0 rd.lvm=0 rd.md=0 rd.dm=0 ip=dhcp 
> netroot=iscsi:169.254.0.2::::iqn.2015-02.oracle.boot:uefi 
> iscsi_param=node.session.timeo.replacement_timeout=6000"
>
> On the system, there are two interfaces (say eth0 and eth1) which are 
> able to get dhcp offers successfully, but only one of them (eth0) is 
> able to reach the specified iscsi target. After some debugging, we 
> believe we've found the root cause. Here is what happened:
>
> 1. eth0 successfully got the dhcp offer, the iscsiroot script was run 
> and eventually ran "system-run" to start the "oneshot" iscsistart 
> service.
> 2. Before step 1 succeeds, the iscsiroot script was also run for eth1 
> and it checks the status of the first iscsistart service instance, 
> which was still "activating". So the iscsistart service was restarted 
> and that killed the first instance. But the second instance also fails 
> because of the "existing session" error.
>
> Here are the questions we have:
>
> a. We found in the iscsiroot script, the iscsid service was started 
> before the iscsistart service. Because of this, the creation of the 
> mgmt ipc socket by iscsistart failed, and iscsi login session request 
> was handled by iscsid instead. In step 2 above, the first iscsistart 
> instance was killed but not the iscsid daemon, hence the "existing 
> error" as the first login session still existed in iscsid.
>
> The questions is if the iscsid service is really required by the 
> iscisroot script? Because the existence of iscsid, after the second 
> iscsistart instance, we saw unexpected iscsid service unavailability 
> because iscsid is stopped by the iscsistart instance (iscsistart.c 
> calls stop_event_loop() to stop event loop in order to exits itself, 
> but since the MGMT_IPC_IMMEDIATE_STOP request was handled by iscsid, 
> iscsid's event loop was stopped and iscsid exited instead).
>
> The related error messages we saw in the log:
>
>    "iscsistart[836]: iscsistart: Can not bind IPC socket
>     iscsistart[836]: iscsistart: Could not setup mgmt ipc
>     ...
>     iscsiadm[970]: iscsiadm: can not connect to iSCSI daemon (111)!
>     iscsiadm[970]: iscsiadm: initiator reported error (20 - could not 
> connect to iscsid)
>     ..."
>
> I tried to change the iscsiroot.sh script to stop the iscsid service 
> instead of restarting iscsid service, it seems it fixed our problem. 
> But I am not sure if this fix has any side-effect.
>
> b. If stopping iscsid service is not the ideal fix, I am wondering if 
> we can mimic the legacy way (prior systemd) and start the iscsistart 
> service independently for each interface. This means, iscsistart 
> service will be started for each interface without affecting the 
> iscsistart instances which are already run for other interfaces. It 
> may mean the service name needs to include $netif to uniquely identify 
> each instance.
>
> Thanks very much! Looking forward to your suggestions!
>
> - Cathy
> -- 
> To unsubscribe from this list: send the line "unsubscribe initramfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iscsi netboot failure in case of multiple interfaces

[Martin Wilck]

Hello Cathy,

On Mon, 2017-11-13 at 10:43 -0800, Cathy Zhou wrote:
> Dracut experts,
> 
> A gentle reminder ...  Looking forward to gaining some insights on
> below 
> questions.

I haven't analyzed your scenario in detail, but IMHO this is a long-
standing issue. dracut is using both iscsid and iscsistart, which is
wrong. iscsistart is meant for situations where iscsid is not
available. It starts iscsid at startup, and stops it again when done.
If proper iscsid is running, iscsiadm should be used rather than
iscsistart to activate sessions.

In my experience, iscsistart will only work reliably if you have just a
single interface.

Martin

Re: iscsi netboot failure in case of multiple interfaces

[Cathy Zhou]

Martin,

Thank you very much for your insights. See inline:

On 11/15/2017 5:12 AM, Martin Wilck wrote:
> Hello Cathy,
>
> On Mon, 2017-11-13 at 10:43 -0800, Cathy Zhou wrote:
>> Dracut experts,
>>
>> A gentle reminder ...  Looking forward to gaining some insights on
>> below
>> questions.
> I haven't analyzed your scenario in detail, but IMHO this is a long-
> standing issue. dracut is using both iscsid and iscsistart, which is
> wrong. iscsistart is meant for situations where iscsid is not
> available. It starts iscsid at startup, and stops it again when done.
> If proper iscsid is running, iscsiadm should be used rather than
> iscsistart to activate sessions.
I agree. I was confused when I saw both iscsid and iscsistart were 
started in dracut. That is why my proposal (a) was to stop the iscsid 
service first before it starts iscsistart service. It fixed our problem 
though I am not sure if it has some unexpected effect on some other 
scenario.
>
> In my experience, iscsistart will only work reliably if you have just a
> single interface.
Could you please elaborate on this? What issue you saw? It seems to me 
as long as iscsistart works without iscsid is running, multiple 
instances should also be fine?

Thanks very much again
- Cathy
>
> Martin
>

Re: iscsi netboot failure in case of multiple interfaces

[si-wei liu]

On 11/15/2017 11:17 AM, Cathy Zhou wrote:
> Martin,
>
> Thank you very much for your insights. See inline:
>
> On 11/15/2017 5:12 AM, Martin Wilck wrote:
>> Hello Cathy,
>>
>> On Mon, 2017-11-13 at 10:43 -0800, Cathy Zhou wrote:
>>> Dracut experts,
>>>
>>> A gentle reminder ...  Looking forward to gaining some insights on
>>> below
>>> questions.
>> I haven't analyzed your scenario in detail, but IMHO this is a long-
>> standing issue. dracut is using both iscsid and iscsistart, which is
>> wrong. iscsistart is meant for situations where iscsid is not
>> available. It starts iscsid at startup, and stops it again when done.
>> If proper iscsid is running, iscsiadm should be used rather than
>> iscsistart to activate sessions.
> I agree. I was confused when I saw both iscsid and iscsistart were 
> started in dracut. That is why my proposal (a) was to stop the iscsid 
> service first before it starts iscsistart service. It fixed our 
> problem though I am not sure if it has some unexpected effect on some 
> other scenario.

Indeed, we got to ask here because we thought that iscsid first landed 
on dracut with this commit:

https://github.com/dracutdevs/dracut/commit/d94050ddaea8343d8adb8f151c8f4cad591d29d9

"
parse-cmdline sets up an initial initiator-name to let iscsid start.

iscsid is started before doing any iscsistart business.
"

We are not sure what iscsid was trying to manage if iscsistart can do 
the job well. Will it lose "robustness" if we remove iscsid from dracut? 
Specifically, iscsistart itself is able to launch iscsid automatically, 
and manage the life cycle of iscsid as needed. Why do we need a separate 
iscsid service managed by our own? What is the idea behind to coordinate 
these two when there are multiple interfaces involved?

>             systemctl try-restart iscsid
>             # FIXME: iscsid is not yet ready, when the service is :-/
>             sleep 1

Please let us know if it's desirable to keep iscsid around, or we can 
post patch to make iscsistart manage iscsid instead (or leave iscsid 
disabled).

Regards,
-Siwei

>>
>> In my experience, iscsistart will only work reliably if you have just a
>> single interface.
> Could you please elaborate on this? What issue you saw? It seems to me 
> as long as iscsistart works without iscsid is running, multiple 
> instances should also be fine?
>
> Thanks very much again
> - Cathy
>>
>> Martin
>>
>

Re: iscsi netboot failure in case of multiple interfaces

[Martin Wilck]

On Wed, 2017-11-15 at 11:47 -0800, si-wei liu wrote:
> On 11/15/2017 11:17 AM, Cathy Zhou wrote:
> > 
> Please let us know if it's desirable to keep iscsid around, or we
> can 
> post patch to make iscsistart manage iscsid instead (or leave iscsid 
> disabled).

IMO dracut should ditch iscsistart and work cleanly with iscsid and
iscsiadm. We have some patches in openSUSE that unfortunately never
quite hit upstream, see e.g. here:

https://github.com/danimo/dracut/commit/5ac36324e3daa897152929e67718250
1b9d20c1f#diff-6dcf84c45218f1adf71f53a6cd80b5c9

Martin

Re: iscsi netboot failure in case of multiple interfaces

[Cathy Zhou]

Martin,

Thanks. I will take a detail look of the diff you sent. But a brief 
browse of it raised one question: assuming we solely use iscsid and 
iscsiadm instead of iscsistart, wouldn't iscsid be started outside of  
iscsiroot.sh? Otherwise, the restart/stop/start of iscsid for one 
interface could interfere with another iscsiroot instance for another 
interface?

Thanks
- Cathy

On 11/16/2017 5:03 AM, Martin Wilck wrote:
> On Wed, 2017-11-15 at 11:47 -0800, si-wei liu wrote:
>> On 11/15/2017 11:17 AM, Cathy Zhou wrote:
>> Please let us know if it's desirable to keep iscsid around, or we
>> can
>> post patch to make iscsistart manage iscsid instead (or leave iscsid
>> disabled).
> IMO dracut should ditch iscsistart and work cleanly with iscsid and
> iscsiadm. We have some patches in openSUSE that unfortunately never
> quite hit upstream, see e.g. here:
>
> https://github.com/danimo/dracut/commit/5ac36324e3daa897152929e67718250
> 1b9d20c1f#diff-6dcf84c45218f1adf71f53a6cd80b5c9
>
> Martin
>

Re: iscsi netboot failure in case of multiple interfaces

[Martin Wilck]

On Fri, 2017-11-17 at 10:34 -0800, Cathy Zhou wrote:
> Martin,
> 
> Thanks. I will take a detail look of the diff you sent. But a brief 
> browse of it raised one question: assuming we solely use iscsid and 
> iscsiadm instead of iscsistart, wouldn't iscsid be started outside
> of  
> iscsiroot.sh? Otherwise, the restart/stop/start of iscsid for one 
> interface could interfere with another iscsiroot instance for
> another 
> interface?

Yes, certainly, the iscsid service should be started only once. 

Please note that I'm not the author of the patch I linked, and that
this link I sent was also just a starting point:

https://github.com/danimo/dracut/commits/obs_from_git/modules.d/95iscsi

Note also that this branch is quite a bit behind upstream dracut in
other areas, so porting the iSCSI changes may not be trivial.

Regards,
Martin

Re: iscsi netboot failure in case of multiple interfaces

[si-wei liu]

On 11/17/2017 11:11 AM, Martin Wilck wrote:
> On Fri, 2017-11-17 at 10:34 -0800, Cathy Zhou wrote:
>> Martin,
>>
>> Thanks. I will take a detail look of the diff you sent. But a brief
>> browse of it raised one question: assuming we solely use iscsid and
>> iscsiadm instead of iscsistart, wouldn't iscsid be started outside
>> of
>> iscsiroot.sh? Otherwise, the restart/stop/start of iscsid for one
>> interface could interfere with another iscsiroot instance for
>> another
>> interface?
> Yes, certainly, the iscsid service should be started only once.
>
> Please note that I'm not the author of the patch I linked, and that
> this link I sent was also just a starting point:
>
> https://github.com/danimo/dracut/commits/obs_from_git/modules.d/95iscsi
>
> Note also that this branch is quite a bit behind upstream dracut in
> other areas, so porting the iSCSI changes may not be trivial.
Is there a plan to upstream these patches soon? I thought multipath 
booting is meant to be supported by dracut but in reality it never works 
out thanks to iscsistart. Without getting the divergence into the 
upstream it effectively prohibits other folks from contributing bug 
fixes to the long standing issue as the nature of the two approaches 
(fix iscsid v.s. overhaul with iscsiadm) are in conflict? Is there a 
tradeoff in between - we fix the iscsistart issue first then once you're 
ready to upstream you overwrite what we had done?

Thanks,
-Siwei

>
> Regards,
> Martin
>

Re: iscsi netboot failure in case of multiple interfaces

[Martin Wilck]

On Fri, 2017-11-17 at 16:09 -0800, si-wei liu wrote:
> 
> On 11/17/2017 11:11 AM, Martin Wilck wrote:
> > 
> > Please note that I'm not the author of the patch I linked, and that
> > this link I sent was also just a starting point:
> > 
> > https://github.com/danimo/dracut/commits/obs_from_git/modules.d/95i
> > scsi
> > 
> > Note also that this branch is quite a bit behind upstream dracut in
> > other areas, so porting the iSCSI changes may not be trivial.
> 
> Is there a plan to upstream these patches soon? 

I don't know. I'll try to find out.

>  Without getting the divergence into the 
> upstream it effectively prohibits other folks from contributing bug 
> fixes to the long standing issue as the nature of the two approaches 
> (fix iscsid v.s. overhaul with iscsiadm) are in conflict? Is there a 
> tradeoff in between - we fix the iscsistart issue first then once
> you're 
> ready to upstream you overwrite what we had done?

If you think you're able to fix iscsistart for good, go ahead.
I have my doubts that it'll work in complex situations.

Martin

Re: iscsi netboot failure in case of multiple interfaces

[si-wei liu]

On 11/17/2017 4:17 PM, Martin Wilck wrote:
> On Fri, 2017-11-17 at 16:09 -0800, si-wei liu wrote:
>> On 11/17/2017 11:11 AM, Martin Wilck wrote:
>>> Please note that I'm not the author of the patch I linked, and that
>>> this link I sent was also just a starting point:
>>>
>>> https://github.com/danimo/dracut/commits/obs_from_git/modules.d/95i
>>> scsi
>>>
>>> Note also that this branch is quite a bit behind upstream dracut in
>>> other areas, so porting the iSCSI changes may not be trivial.
>> Is there a plan to upstream these patches soon?
> I don't know. I'll try to find out.

OK. Thanks!
>
>>   Without getting the divergence into the
>> upstream it effectively prohibits other folks from contributing bug
>> fixes to the long standing issue as the nature of the two approaches
>> (fix iscsid v.s. overhaul with iscsiadm) are in conflict? Is there a
>> tradeoff in between - we fix the iscsistart issue first then once
>> you're
>> ready to upstream you overwrite what we had done?
> If you think you're able to fix iscsistart for good, go ahead.
> I have my doubts that it'll work in complex situations.
Yeah I understand the current mix use of iscsid and iscsistart is not 
quite good. Some may work by chance while some resorts to good timing 
thanks to various places of delay (sleep 1) in the script. I like the 
proposal of new design of replacing iscsistart, that looks clean indeed.

Thanks,
-Siwei



> Martin
>

Re: iscsi netboot failure in case of multiple interfaces

[Martin Wilck]

On Fri, 2017-11-17 at 16:42 -0800, si-wei liu wrote:
> 
> > > Is there a plan to upstream these patches soon?
> > 
> > I don't know. I'll try to find out.
> 
> OK. Thanks!

I got some information from the maintainer. It's planned, but there's
no fixed schedule. As you could see, our iscsiadm patches were created
~2y ago. The upstream and SUSE dracut code base have diverged. Don't
expect a merge of the SUSE to happen very soon.

Regards,
Martin

Re: iscsi netboot failure in case of multiple interfaces

[si-wei liu]

On 11/21/2017 01:34 PM, Martin Wilck wrote:
> On Fri, 2017-11-17 at 16:42 -0800, si-wei liu wrote:
>>>> Is there a plan to upstream these patches soon?
>>> I don't know. I'll try to find out.
>> OK. Thanks!
> I got some information from the maintainer. It's planned, but there's
> no fixed schedule. As you could see, our iscsiadm patches were created
> ~2y ago. The upstream and SUSE dracut code base have diverged. Don't
> expect a merge of the SUSE to happen very soon.
OK. Then we'll proceed with posting our patch to fix the issue in short 
term. Once you're ready to upstream please let us know so we can retest. 
Let me know if that works for you.

Thanks,
-Siwei

>
> Regards,
> Martin
>

Re: iscsi netboot failure in case of multiple interfaces

[Martin Wilck]

On Tue, 2017-11-21 at 13:58 -0800, si-wei liu wrote:
> 
> OK. Then we'll proceed with posting our patch to fix the issue in
> short 
> term. Once you're ready to upstream please let us know so we can
> retest. 
> Let me know if that works for you.

I have no objections.

Cheers,
Martin