[PATCH] sparc64: shut down to OBP correctly

[PATCH] sparc64: shut down to OBP correctly

[Larry Bassel]

Orabug: 24789774

The command "shutdown -h -H now" should shut down the system to
OBP, however the machine was incorrectly being powered off instead
(on both LDOM and bare metal).

The "exit" command to the OBP must be run and then a hard
loop to prevent return to the kernel.

Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
---
 arch/sparc/prom/misc_64.c |    7 +------
 1 files changed, 1 insertions(+), 6 deletions(-)

diff --git a/arch/sparc/prom/misc_64.c b/arch/sparc/prom/misc_64.c
index 53a696d..aa80788 100644
--- a/arch/sparc/prom/misc_64.c
+++ b/arch/sparc/prom/misc_64.c
@@ -115,16 +115,11 @@ void notrace prom_halt(void)
 {
 	unsigned long args[3];
 
-#ifdef CONFIG_SUN_LDOMS
-	if (ldom_domaining_enabled)
-		ldom_power_off();
-#endif
-again:
 	args[0] = (unsigned long) "exit";
 	args[1] = 0;
 	args[2] = 0;
 	p1275_cmd_direct(args);
-	goto again; /* PROM is out to get me -DaveM */
+	while (1);
 }
 
 void prom_halt_power_off(void)
-- 
1.7.1

Re: [PATCH] sparc64: shut down to OBP correctly

[Larry Bassel]

On 21 Dec 16 09:12, Larry Bassel wrote:
> Orabug: 24789774

My apologies -- this line likely should not be in this patch -- I'm sending
a v2.

Larry

> 
> The command "shutdown -h -H now" should shut down the system to
> OBP, however the machine was incorrectly being powered off instead
> (on both LDOM and bare metal).
> 
> The "exit" command to the OBP must be run and then a hard
> loop to prevent return to the kernel.
> 
> Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
> ---
>  arch/sparc/prom/misc_64.c |    7 +------
>  1 files changed, 1 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/sparc/prom/misc_64.c b/arch/sparc/prom/misc_64.c
> index 53a696d..aa80788 100644
> --- a/arch/sparc/prom/misc_64.c
> +++ b/arch/sparc/prom/misc_64.c
> @@ -115,16 +115,11 @@ void notrace prom_halt(void)
>  {
>  	unsigned long args[3];
>  
> -#ifdef CONFIG_SUN_LDOMS
> -	if (ldom_domaining_enabled)
> -		ldom_power_off();
> -#endif
> -again:
>  	args[0] = (unsigned long) "exit";
>  	args[1] = 0;
>  	args[2] = 0;
>  	p1275_cmd_direct(args);
> -	goto again; /* PROM is out to get me -DaveM */
> +	while (1);
>  }
>  
>  void prom_halt_power_off(void)
> -- 
> 1.7.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] sparc64: shut down to OBP correctly

[David Miller]

From: Larry Bassel <larry.bassel@oracle.com>
Date: Wed, 21 Dec 2016 09:12:13 -0800

> Orabug: 24789774
> 
> The command "shutdown -h -H now" should shut down the system to
> OBP, however the machine was incorrectly being powered off instead
> (on both LDOM and bare metal).
> 
> The "exit" command to the OBP must be run and then a hard
> loop to prevent return to the kernel.
> 
> Signed-off-by: Larry Bassel <larry.bassel@oracle.com>

On LDOM, once you boot past the early stages of the kernel, OBP is
considered volatile and thus unusable.

That's why I did a direct LDOM shutdown here.

Re: [PATCH] sparc64: shut down to OBP correctly

[Larry Bassel]

On 21 Dec 16 13:07, David Miller wrote:
> From: Larry Bassel <larry.bassel@oracle.com>
> Date: Wed, 21 Dec 2016 09:12:13 -0800
> 
> > Orabug: 24789774
> > 
> > The command "shutdown -h -H now" should shut down the system to
> > OBP, however the machine was incorrectly being powered off instead
> > (on both LDOM and bare metal).
> > 
> > The "exit" command to the OBP must be run and then a hard
> > loop to prevent return to the kernel.
> > 
> > Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
> 
> On LDOM, once you boot past the early stages of the kernel, OBP is
> considered volatile and thus unusable.

My code does work on an 4.9.0 LDOM (we get to the ok prompt). So you're saying
this was a "lucky accident" and there is no "clean" way of getting back to
the OBP when we are in a LDOM (i.e. all we can do is power off which is
what the current code does)?

> 
> That's why I did a direct LDOM shutdown here.

Unfortunately, the variable "ldom_domaining_enabled" is 1 even when
we are not in an LDOM (I just double checked this by printing
the value out as we shut down) and so the current code will power off in
the bare metal case as well. Is there is a good way of determining that
we are running on bare metal?

Thanks.

Larry

> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] sparc64: shut down to OBP correctly

[David Miller]

From: Larry Bassel <larry.bassel@oracle.com>
Date: Wed, 21 Dec 2016 10:51:33 -0800

> On 21 Dec 16 13:07, David Miller wrote:
>> From: Larry Bassel <larry.bassel@oracle.com>
>> Date: Wed, 21 Dec 2016 09:12:13 -0800
>> 
>> > Orabug: 24789774
>> > 
>> > The command "shutdown -h -H now" should shut down the system to
>> > OBP, however the machine was incorrectly being powered off instead
>> > (on both LDOM and bare metal).
>> > 
>> > The "exit" command to the OBP must be run and then a hard
>> > loop to prevent return to the kernel.
>> > 
>> > Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
>> 
>> On LDOM, once you boot past the early stages of the kernel, OBP is
>> considered volatile and thus unusable.
> 
> My code does work on an 4.9.0 LDOM (we get to the ok prompt). So you're saying
> this was a "lucky accident" and there is no "clean" way of getting back to
> the OBP when we are in a LDOM (i.e. all we can do is power off which is
> what the current code does)?

Yes, for a very long time the firmware/hypervisor engineers told me
exactly this.

> Unfortunately, the variable "ldom_domaining_enabled" is 1 even when
> we are not in an LDOM (I just double checked this by printing
> the value out as we shut down) and so the current code will power off in
> the bare metal case as well. Is there is a good way of determining that
> we are running on bare metal?

Even for bare metal the condition holds.  As long as we use the interfaces
guarded by ldom_domaining_enabled, the OBP is not reliably available after
early boot.

Re: [PATCH] sparc64: shut down to OBP correctly

[Larry Bassel]

On 21 Dec 16 14:06, David Miller wrote:
> From: Larry Bassel <larry.bassel@oracle.com>
> Date: Wed, 21 Dec 2016 10:51:33 -0800
> 
> > On 21 Dec 16 13:07, David Miller wrote:
> >> From: Larry Bassel <larry.bassel@oracle.com>
> >> Date: Wed, 21 Dec 2016 09:12:13 -0800
> >> 
> >> > Orabug: 24789774
> >> > 
> >> > The command "shutdown -h -H now" should shut down the system to
> >> > OBP, however the machine was incorrectly being powered off instead
> >> > (on both LDOM and bare metal).
> >> > 
> >> > The "exit" command to the OBP must be run and then a hard
> >> > loop to prevent return to the kernel.
> >> > 
> >> > Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
> >> 
> >> On LDOM, once you boot past the early stages of the kernel, OBP is
> >> considered volatile and thus unusable.
> > 
> > My code does work on an 4.9.0 LDOM (we get to the ok prompt). So you're saying
> > this was a "lucky accident" and there is no "clean" way of getting back to
> > the OBP when we are in a LDOM (i.e. all we can do is power off which is
> > what the current code does)?
> 
> Yes, for a very long time the firmware/hypervisor engineers told me
> exactly this.
> 
> > Unfortunately, the variable "ldom_domaining_enabled" is 1 even when
> > we are not in an LDOM (I just double checked this by printing
> > the value out as we shut down) and so the current code will power off in
> > the bare metal case as well. Is there is a good way of determining that
> > we are running on bare metal?
> 
> Even for bare metal the condition holds.  As long as we use the interfaces
> guarded by ldom_domaining_enabled, the OBP is not reliably available after
> early boot.

I'm not sure I understand this comment -- are you saying that if we are
bare metal (and somehow can know this), one still cannot call into the OBP?
There are other calls to the OBP (p1275_cmd_direct) in prom/misc_64.c, some
of which are not conditional on ldom_domaining_enabled (which means they
would be called even in the LDOM case).

If ldom_domaining_enabled is always set, what is the purpose of the "exit"
call in prom_halt() at all? It appears that ldc_init() is always called
upon kernel initialization (assuming CONFIG_SUN_LDOMS is set) which (barring
some error in ldc_init()) permanently sets ldom_domaining_enabled. It appears
that CONFIG_SUN_LDOMS is set by defconfig (and I would presume we would want
this set so that the same kernel could work in both an LDOM and a bare metal
environment).

Therefore, I'm looking for some way to know whether I'm really bare metal
or not besides checking ldom_domaining_enabled. Should (during kernel
initialization) ldc_init() *not* be called when we are bare metal
(assuming this can be determined)?

My apologies if I'm missing something obvious here.

Larry

> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] sparc64: shut down to OBP correctly

[David Miller]

From: Larry Bassel <larry.bassel@oracle.com>
Date: Wed, 21 Dec 2016 11:44:07 -0800

> On 21 Dec 16 14:06, David Miller wrote:
>> From: Larry Bassel <larry.bassel@oracle.com>
>> Date: Wed, 21 Dec 2016 10:51:33 -0800
>> 
>> > On 21 Dec 16 13:07, David Miller wrote:
>> >> From: Larry Bassel <larry.bassel@oracle.com>
>> >> Date: Wed, 21 Dec 2016 09:12:13 -0800
>> >> 
>> >> > Orabug: 24789774
>> >> > 
>> >> > The command "shutdown -h -H now" should shut down the system to
>> >> > OBP, however the machine was incorrectly being powered off instead
>> >> > (on both LDOM and bare metal).
>> >> > 
>> >> > The "exit" command to the OBP must be run and then a hard
>> >> > loop to prevent return to the kernel.
>> >> > 
>> >> > Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
>> >> 
>> >> On LDOM, once you boot past the early stages of the kernel, OBP is
>> >> considered volatile and thus unusable.
>> > 
>> > My code does work on an 4.9.0 LDOM (we get to the ok prompt). So you're saying
>> > this was a "lucky accident" and there is no "clean" way of getting back to
>> > the OBP when we are in a LDOM (i.e. all we can do is power off which is
>> > what the current code does)?
>> 
>> Yes, for a very long time the firmware/hypervisor engineers told me
>> exactly this.
>> 
>> > Unfortunately, the variable "ldom_domaining_enabled" is 1 even when
>> > we are not in an LDOM (I just double checked this by printing
>> > the value out as we shut down) and so the current code will power off in
>> > the bare metal case as well. Is there is a good way of determining that
>> > we are running on bare metal?
>> 
>> Even for bare metal the condition holds.  As long as we use the interfaces
>> guarded by ldom_domaining_enabled, the OBP is not reliably available after
>> early boot.
> 
> I'm not sure I understand this comment -- are you saying that if we are
> bare metal (and somehow can know this), one still cannot call into the OBP?
> There are other calls to the OBP (p1275_cmd_direct) in prom/misc_64.c, some
> of which are not conditional on ldom_domaining_enabled (which means they
> would be called even in the LDOM case).

We have to unconditionally allow p1275_cmd_direct() for certain earlier
boot calls.

The problem, I was told, is that when you boot cpus into the machine using
the LDOM cpu start hypercall, you can't use OBP.

This is why we pull the entire device tree from OBP into the kernel on
the boot processer very early in the boot.  That way we don't need to
make OBP calls once we start booting up the non-boot cpus.

Re: [PATCH] sparc64: shut down to OBP correctly

[Alexandre Chartre]

On 12/21/2016 09:15 PM, David Miller wrote:
> From: Larry Bassel <larry.bassel@oracle.com>
> Date: Wed, 21 Dec 2016 11:44:07 -0800
>
>> On 21 Dec 16 14:06, David Miller wrote:
>>> From: Larry Bassel <larry.bassel@oracle.com>
>>> Date: Wed, 21 Dec 2016 10:51:33 -0800
>>>
>>>> On 21 Dec 16 13:07, David Miller wrote:
>>>>> From: Larry Bassel <larry.bassel@oracle.com>
>>>>> Date: Wed, 21 Dec 2016 09:12:13 -0800
>>>>>
>>>>>> Orabug: 24789774
>>>>>>
>>>>>> The command "shutdown -h -H now" should shut down the system to
>>>>>> OBP, however the machine was incorrectly being powered off instead
>>>>>> (on both LDOM and bare metal).
>>>>>>
>>>>>> The "exit" command to the OBP must be run and then a hard
>>>>>> loop to prevent return to the kernel.
>>>>>>
>>>>>> Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
>>>>>
>>>>> On LDOM, once you boot past the early stages of the kernel, OBP is
>>>>> considered volatile and thus unusable.
>>>>
>>>> My code does work on an 4.9.0 LDOM (we get to the ok prompt). So you're saying
>>>> this was a "lucky accident" and there is no "clean" way of getting back to
>>>> the OBP when we are in a LDOM (i.e. all we can do is power off which is
>>>> what the current code does)?
>>>
>>> Yes, for a very long time the firmware/hypervisor engineers told me
>>> exactly this.
>>>
>>>> Unfortunately, the variable "ldom_domaining_enabled" is 1 even when
>>>> we are not in an LDOM (I just double checked this by printing
>>>> the value out as we shut down) and so the current code will power off in
>>>> the bare metal case as well. Is there is a good way of determining that
>>>> we are running on bare metal?
>>>
>>> Even for bare metal the condition holds.  As long as we use the interfaces
>>> guarded by ldom_domaining_enabled, the OBP is not reliably available after
>>> early boot.
>>
>> I'm not sure I understand this comment -- are you saying that if we are
>> bare metal (and somehow can know this), one still cannot call into the OBP?
>> There are other calls to the OBP (p1275_cmd_direct) in prom/misc_64.c, some
>> of which are not conditional on ldom_domaining_enabled (which means they
>> would be called even in the LDOM case).
>
> We have to unconditionally allow p1275_cmd_direct() for certain earlier
> boot calls.
>
> The problem, I was told, is that when you boot cpus into the machine using
> the LDOM cpu start hypercall, you can't use OBP.
>
> This is why we pull the entire device tree from OBP into the kernel on
> the boot processer very early in the boot.  That way we don't need to
> make OBP calls once we start booting up the non-boot cpus.


That's correct. The problem is that the OBP won't be aware of any configuration
change in the domain (like adding/removing cpu, memory or devices) and the OBP
won't update its device tree. So if you return to the OBP, the behavior is
unpredictable because the OBP might have outdated information.

For example, on Solaris, a shutdown/halt command won't directly return to the
OBP. Instead, it will reset the domain and stop at the OBP ok prompt after the
domain has been reset. This is done by setting the OBP reboot-command to "noop"
(to prevent an auto-boot after the reset) and then resetting the domain with
HV_MACH_SIR.


alex.

Re: [PATCH] sparc64: shut down to OBP correctly

[Larry Bassel]

On 22 Dec 16 09:26, Alexandre Chartre wrote:
> 
> On 12/21/2016 09:15 PM, David Miller wrote:
> >From: Larry Bassel <larry.bassel@oracle.com>
> >Date: Wed, 21 Dec 2016 11:44:07 -0800
> >
> >>On 21 Dec 16 14:06, David Miller wrote:
> >>>From: Larry Bassel <larry.bassel@oracle.com>
> >>>Date: Wed, 21 Dec 2016 10:51:33 -0800
> >>>
> >>>>On 21 Dec 16 13:07, David Miller wrote:
> >>>>>From: Larry Bassel <larry.bassel@oracle.com>
> >>>>>Date: Wed, 21 Dec 2016 09:12:13 -0800
> >>>>>
> >>>>>>Orabug: 24789774
> >>>>>>
> >>>>>>The command "shutdown -h -H now" should shut down the system to
> >>>>>>OBP, however the machine was incorrectly being powered off instead
> >>>>>>(on both LDOM and bare metal).
> >>>>>>
> >>>>>>The "exit" command to the OBP must be run and then a hard
> >>>>>>loop to prevent return to the kernel.
> >>>>>>
> >>>>>>Signed-off-by: Larry Bassel <larry.bassel@oracle.com>
> >>>>>
> >>>>>On LDOM, once you boot past the early stages of the kernel, OBP is
> >>>>>considered volatile and thus unusable.
> >>>>
> >>>>My code does work on an 4.9.0 LDOM (we get to the ok prompt). So you're saying
> >>>>this was a "lucky accident" and there is no "clean" way of getting back to
> >>>>the OBP when we are in a LDOM (i.e. all we can do is power off which is
> >>>>what the current code does)?
> >>>
> >>>Yes, for a very long time the firmware/hypervisor engineers told me
> >>>exactly this.
> >>>
> >>>>Unfortunately, the variable "ldom_domaining_enabled" is 1 even when
> >>>>we are not in an LDOM (I just double checked this by printing
> >>>>the value out as we shut down) and so the current code will power off in
> >>>>the bare metal case as well. Is there is a good way of determining that
> >>>>we are running on bare metal?
> >>>
> >>>Even for bare metal the condition holds.  As long as we use the interfaces
> >>>guarded by ldom_domaining_enabled, the OBP is not reliably available after
> >>>early boot.
> >>
> >>I'm not sure I understand this comment -- are you saying that if we are
> >>bare metal (and somehow can know this), one still cannot call into the OBP?
> >>There are other calls to the OBP (p1275_cmd_direct) in prom/misc_64.c, some
> >>of which are not conditional on ldom_domaining_enabled (which means they
> >>would be called even in the LDOM case).
> >
> >We have to unconditionally allow p1275_cmd_direct() for certain earlier
> >boot calls.
> >
> >The problem, I was told, is that when you boot cpus into the machine using
> >the LDOM cpu start hypercall, you can't use OBP.
> >
> >This is why we pull the entire device tree from OBP into the kernel on
> >the boot processer very early in the boot.  That way we don't need to
> >make OBP calls once we start booting up the non-boot cpus.
> 
> 
> That's correct. The problem is that the OBP won't be aware of any configuration
> change in the domain (like adding/removing cpu, memory or devices) and the OBP
> won't update its device tree. So if you return to the OBP, the behavior is
> unpredictable because the OBP might have outdated information.
> 
> For example, on Solaris, a shutdown/halt command won't directly return to the
> OBP. Instead, it will reset the domain and stop at the OBP ok prompt after the
> domain has been reset. This is done by setting the OBP reboot-command to "noop"
> (to prevent an auto-boot after the reset) and then resetting the domain with
> HV_MACH_SIR.

Thanks for mentioning this. I'm aware of what Solaris does and even have an
earlier (on top of 4.6) version of this patch (which I never sent upstream)
which did exactly that (and it worked).

Is there a consensus that this approach is acceptable? If so, I'll clean up
the earlier version, re-test it and then submit that as v3.

> 
> 
> alex.

Larry