[PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Michael Ellerman]

Some Power9 boxes will have this adapter installed, so add it to the
defconfig so we can boot on those machines without an initrd.

Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
 arch/powerpc/configs/powernv_defconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/powerpc/configs/powernv_defconfig b/arch/powerpc/configs/powernv_defconfig
index f89d9b1799b6..dde87c623b74 100644
--- a/arch/powerpc/configs/powernv_defconfig
+++ b/arch/powerpc/configs/powernv_defconfig
@@ -113,6 +113,7 @@ CONFIG_SCSI_CXGB3_ISCSI=m
 CONFIG_SCSI_CXGB4_ISCSI=m
 CONFIG_SCSI_BNX2_ISCSI=m
 CONFIG_BE2ISCSI=m
+CONFIG_SCSI_AACRAID=y
 CONFIG_SCSI_MPT2SAS=m
 CONFIG_SCSI_SYM53C8XX_2=m
 CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0
-- 
2.7.4

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Guilherme G. Piccoli]

On 08/28/2017 02:56 AM, Michael Ellerman wrote:
> Some Power9 boxes will have this adapter installed, so add it to the
> defconfig so we can boot on those machines without an initrd.

Michael, not sure if this affects Petitboot (I know it has its own
default config files), but in the past we had some issues with this
driver and it's nice to have the possibility to blacklist it.

What is the rationale for this change? I mean, how being able to boot
without initrd is interesting here? Sorry if it's silly question!

Thanks,


Guilherme


P.S. If this config was *unset* and you're setting, I'd be glad to see
it as module!

> 
> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
> ---
>  arch/powerpc/configs/powernv_defconfig | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/powerpc/configs/powernv_defconfig b/arch/powerpc/configs/powernv_defconfig
> index f89d9b1799b6..dde87c623b74 100644
> --- a/arch/powerpc/configs/powernv_defconfig
> +++ b/arch/powerpc/configs/powernv_defconfig
> @@ -113,6 +113,7 @@ CONFIG_SCSI_CXGB3_ISCSI=m
>  CONFIG_SCSI_CXGB4_ISCSI=m
>  CONFIG_SCSI_BNX2_ISCSI=m
>  CONFIG_BE2ISCSI=m
> +CONFIG_SCSI_AACRAID=y
>  CONFIG_SCSI_MPT2SAS=m
>  CONFIG_SCSI_SYM53C8XX_2=m
>  CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0
> 

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Michael Ellerman]

"Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes:

> On 08/28/2017 02:56 AM, Michael Ellerman wrote:
>> Some Power9 boxes will have this adapter installed, so add it to the
>> defconfig so we can boot on those machines without an initrd.
>
> Michael, not sure if this affects Petitboot (I know it has its own
> default config files), but in the past we had some issues with this
> driver and it's nice to have the possibility to blacklist it.

Why we were black listing it?

> What is the rationale for this change? I mean, how being able to boot
> without initrd is interesting here? Sorry if it's silly question!

It is a silly question! :)

When you boot as many kernels as me it's preferable to be able to build
a kernel completely remote from the box, throw the kernel on a tftp
server and have the machine boot from that kernel all the way to
userspace.

That requires not using an initrd, and therefore having all the drivers
built-in that are needed to get to the boot disk.

(Yeah it is "possible" to build an initrd remotely but it's a pain).

And the one P9 box I have here uses that driver for its root disk, so I
do really want it to be =y.

cheers

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Guilherme G. Piccoli]

On 08/29/2017 08:22 AM, Michael Ellerman wrote:
> "Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes:
> 
>> On 08/28/2017 02:56 AM, Michael Ellerman wrote:
>>> Some Power9 boxes will have this adapter installed, so add it to the
>>> defconfig so we can boot on those machines without an initrd.
>>
>> Michael, not sure if this affects Petitboot (I know it has its own
>> default config files), but in the past we had some issues with this
>> driver and it's nice to have the possibility to blacklist it.
> 
> Why we were black listing it?

Basically for the same reason you want it built-in - debug purposes heheh

In fact, this driver takes some time in its initialization process, so
if my rootfs is not under it, to speedup my boot-time I could blacklist
it; also, to debug the aacraid driver we could blacklist it in Petitboot
and debug on distro (easier to build kernels there), since the issue
could be in first probe only (or be affected by kexec-out from Petitboot).

Anyway, since make it a module will harm your workflow, I agree built-in
is the best option!

Thanks for your detailed explanation.
Cheers,


Guilherme


> 
>> What is the rationale for this change? I mean, how being able to boot
>> without initrd is interesting here? Sorry if it's silly question!
> 
> It is a silly question! :)
> 
> When you boot as many kernels as me it's preferable to be able to build
> a kernel completely remote from the box, throw the kernel on a tftp
> server and have the machine boot from that kernel all the way to
> userspace.
> 
> That requires not using an initrd, and therefore having all the drivers
> built-in that are needed to get to the boot disk.
> 
> (Yeah it is "possible" to build an initrd remotely but it's a pain).
> 
> And the one P9 box I have here uses that driver for its root disk, so I
> do really want it to be =y.
> 
> cheers
> 

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Michael Ellerman]

"Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes:

> On 08/29/2017 08:22 AM, Michael Ellerman wrote:
>> "Guilherme G. Piccoli" <gpiccoli@linux.vnet.ibm.com> writes:
>> 
>>> On 08/28/2017 02:56 AM, Michael Ellerman wrote:
>>>> Some Power9 boxes will have this adapter installed, so add it to the
>>>> defconfig so we can boot on those machines without an initrd.
>>>
>>> Michael, not sure if this affects Petitboot (I know it has its own
>>> default config files), but in the past we had some issues with this
>>> driver and it's nice to have the possibility to blacklist it.
>> 
>> Why we were black listing it?
>
> Basically for the same reason you want it built-in - debug purposes heheh
>
> In fact, this driver takes some time in its initialization process, so
> if my rootfs is not under it, to speedup my boot-time I could blacklist
> it; also, to debug the aacraid driver we could blacklist it in Petitboot
> and debug on distro (easier to build kernels there), since the issue
> could be in first probe only (or be affected by kexec-out from Petitboot).
>
> Anyway, since make it a module will harm your workflow, I agree built-in
> is the best option!

OK. So maybe when the petitboot kernel moves up to 4.14 they may want to
flip it back to being a module.

I think the majority of machines that have one of these adapters will be
using it for their root disk, so I think the slow initialisation is just
something we have to suffer.

cheers

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Guilherme G. Piccoli]

On 08/30/2017 08:07 AM, Michael Ellerman wrote:
>[...]
> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to
> flip it back to being a module.
> 

OK, I'll be tuned! Thanks


> I think the majority of machines that have one of these adapters will be
> using it for their root disk, so I think the slow initialisation is just
> something we have to suffer.
> 
> cheers
> 

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Stewart Smith]

Michael Ellerman <mpe@ellerman.id.au> writes:
> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to
> flip it back to being a module.

Yeah. We tend to keep all RAID adapters as modules because they almost
all uniquely try to undo all boot time optimization done by anybody,
anywhere, ever.

> I think the majority of machines that have one of these adapters will be
> using it for their root disk, so I think the slow initialisation is just
> something we have to suffer.

It does seem that way currently.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Michael Ellerman]

Stewart Smith <stewart@linux.vnet.ibm.com> writes:

> Michael Ellerman <mpe@ellerman.id.au> writes:
>> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to
>> flip it back to being a module.
>
> Yeah. We tend to keep all RAID adapters as modules because they almost
> all uniquely try to undo all boot time optimization done by anybody,
> anywhere, ever.

Yes I know what you mean. I am thinking of one In ParticulaR.

cheers

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Brian King]

On 08/31/2017 07:37 AM, Michael Ellerman wrote:
> Stewart Smith <stewart@linux.vnet.ibm.com> writes:
> 
>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to
>>> flip it back to being a module.
>>
>> Yeah. We tend to keep all RAID adapters as modules because they almost
>> all uniquely try to undo all boot time optimization done by anybody,
>> anywhere, ever.
> 
> Yes I know what you mean. I am thinking of one In ParticulaR.

Ouch. In all fairness over the life of ipr I've done a couple things to
improve initialization time.

1. On a PowerVM machine, the ipr boot device should not get reset between
   the FCODE driver and the Linux driver, so init is extremely fast. If its
   a non-boot device, then you'll have to wait for the firmware to initialize
   on those adapters.
2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot
   kernel, then we should also avoid resetting the ipr adapter, so ipr
   init on the kernel being kexec booted from skiboot should be extremely fast. 

As to improving ipr init time in the skiboot kernel, that becomes more
difficult to do safely. The firmware stack still needs to boot, do device
discovery, potentially spin up disks, etc, which all takes time.

Additionally, we put in changes in the firmware and driver to address
complaints largely from Anton about it taking too long for ipr to timeout on
bad adapters.

If you've got cases where ipr init is taking a long time, I'd be
interested to know what scenarios are the most annoying to see if there
is any opportunity to improve.

-Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Michael Ellerman]

Brian King <brking@linux.vnet.ibm.com> writes:

> On 08/31/2017 07:37 AM, Michael Ellerman wrote:
>> Stewart Smith <stewart@linux.vnet.ibm.com> writes:
>> 
>>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>>> OK. So maybe when the petitboot kernel moves up to 4.14 they may want to
>>>> flip it back to being a module.
>>>
>>> Yeah. We tend to keep all RAID adapters as modules because they almost
>>> all uniquely try to undo all boot time optimization done by anybody,
>>> anywhere, ever.
>> 
>> Yes I know what you mean. I am thinking of one In ParticulaR.
>
> Ouch. In all fairness over the life of ipr I've done a couple things to
> improve initialization time.

Sorry that wasn't meant as a dig at you, more the hardware. It might
have needed a smiley :)

> 1. On a PowerVM machine, the ipr boot device should not get reset between
>    the FCODE driver and the Linux driver, so init is extremely fast. If its
>    a non-boot device, then you'll have to wait for the firmware to initialize
>    on those adapters.

Yeah I don't see any problem on my LPARs, which just have a single IPR
for the boot device. Init takes ~1 second.

> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot
>    kernel, then we should also avoid resetting the ipr adapter, so ipr
>    init on the kernel being kexec booted from skiboot should be extremely fast. 

OK, I didn't know that was an option, so that might help.

> ...
> If you've got cases where ipr init is taking a long time, I'd be
> interested to know what scenarios are the most annoying to see if there
> is any opportunity to improve.

Yeah booting bare metal is where I see it (not using ipr.fast_reboot).

eg booting a Tuleta:

  [    1.517752] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017)
  ...
  [    1.519884] ipr 0001:08:00.0: Initializing IOA.
  [    1.519941] scsi host0: IBM 0 Storage Adapter
  [   25.695744] ipr 0001:08:00.0: Starting IOA initialization sequence.
  ...
  [   26.329614] sd 0:2:0:0: [sda] Spinning up disk...
  [   26.329752] sd 0:2:1:0: [sdb] Spinning up disk...
  [   26.329990] sd 0:2:2:0: [sdc] Spinning up disk...
  [   26.330065] sd 0:2:3:0: [sdd] Spinning up disk...
  ...
  [   26.349962] sr 0:0:6:0: Attached scsi generic sg15 type 5


Those 24 seconds are a bit of a shame. I assume we're just waiting for
the card firmware to come up?

I don't think there's much we can do in the kernel, PCI has just come
up, so we can't start the IPR init earlier. And the rest of the boot
only takes ~3 seconds, so even if we did everything else in parallel,
we'd still be waiting ~21 seconds for IPR.

cheers

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Stewart Smith]

Michael Ellerman <mpe@ellerman.id.au> writes:
>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot
>>    kernel, then we should also avoid resetting the ipr adapter, so ipr
>>    init on the kernel being kexec booted from skiboot should be extremely fast. 
>
> OK, I didn't know that was an option, so that might help.
>
>> ...
>> If you've got cases where ipr init is taking a long time, I'd be
>> interested to know what scenarios are the most annoying to see if there
>> is any opportunity to improve.
>
> Yeah booting bare metal is where I see it (not using ipr.fast_reboot).

Hrm... We should probably enable that by default for petitboot then.

It'd at least cut some time off booting straight through to OS.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Brian King]

On 09/03/2017 06:19 PM, Stewart Smith wrote:
> Michael Ellerman <mpe@ellerman.id.au> writes:
>>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot
>>>    kernel, then we should also avoid resetting the ipr adapter, so ipr
>>>    init on the kernel being kexec booted from skiboot should be extremely fast. 
>>
>> OK, I didn't know that was an option, so that might help.
>>
>>> ...
>>> If you've got cases where ipr init is taking a long time, I'd be
>>> interested to know what scenarios are the most annoying to see if there
>>> is any opportunity to improve.
>>
>> Yeah booting bare metal is where I see it (not using ipr.fast_reboot).
> 
> Hrm... We should probably enable that by default for petitboot then.
> 
> It'd at least cut some time off booting straight through to OS.

Agreed. I'd be interested to hear if that helps address the issue
Michael is seeing.

You can easily test this by exiting to a petitboot shell:

echo 1 > /sys/module/ipr/parameters/fast_reboot

Then go back to petitboot and boot the OS.

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Michael Ellerman]

Brian King <brking@linux.vnet.ibm.com> writes:
> On 09/03/2017 06:19 PM, Stewart Smith wrote:
>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot
>>>>    kernel, then we should also avoid resetting the ipr adapter, so ipr
>>>>    init on the kernel being kexec booted from skiboot should be extremely fast. 
>>>
>>> OK, I didn't know that was an option, so that might help.
>>>
>>>> ...
>>>> If you've got cases where ipr init is taking a long time, I'd be
>>>> interested to know what scenarios are the most annoying to see if there
>>>> is any opportunity to improve.
>>>
>>> Yeah booting bare metal is where I see it (not using ipr.fast_reboot).
>> 
>> Hrm... We should probably enable that by default for petitboot then.
>> 
>> It'd at least cut some time off booting straight through to OS.
>
> Agreed. I'd be interested to hear if that helps address the issue
> Michael is seeing.
>
> You can easily test this by exiting to a petitboot shell:
>
> echo 1 > /sys/module/ipr/parameters/fast_reboot
>
> Then go back to petitboot and boot the OS.

Just following up on this (!).

This does work, and I've now been running it in my CI for about a month
(~1000 boots) with no problems.

You can also make it persistent by doing:

  $ nvram -p ibm,skiboot --update-config bootargs="ipr.fast_reboot=1"

cheers

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Stewart Smith]

Michael Ellerman <mpe@ellerman.id.au> writes:
> Brian King <brking@linux.vnet.ibm.com> writes:
>> On 09/03/2017 06:19 PM, Stewart Smith wrote:
>>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>>>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot
>>>>>    kernel, then we should also avoid resetting the ipr adapter, so ipr
>>>>>    init on the kernel being kexec booted from skiboot should be extremely fast. 
>>>>
>>>> OK, I didn't know that was an option, so that might help.
>>>>
>>>>> ...
>>>>> If you've got cases where ipr init is taking a long time, I'd be
>>>>> interested to know what scenarios are the most annoying to see if there
>>>>> is any opportunity to improve.
>>>>
>>>> Yeah booting bare metal is where I see it (not using ipr.fast_reboot).
>>> 
>>> Hrm... We should probably enable that by default for petitboot then.
>>> 
>>> It'd at least cut some time off booting straight through to OS.
>>
>> Agreed. I'd be interested to hear if that helps address the issue
>> Michael is seeing.
>>
>> You can easily test this by exiting to a petitboot shell:
>>
>> echo 1 > /sys/module/ipr/parameters/fast_reboot
>>
>> Then go back to petitboot and boot the OS.
>
> Just following up on this (!).
>
> This does work, and I've now been running it in my CI for about a month
> (~1000 boots) with no problems.
>
> You can also make it persistent by doing:
>
>   $ nvram -p ibm,skiboot --update-config bootargs="ipr.fast_reboot=1"

Okay, cool. https://github.com/open-power/op-build/pull/1900 will set it
in firmware - we may as well run with this and fix any bugs we find.

Any reason why it isn't the default behaviour?

-- 
Stewart Smith
OPAL Architect, IBM.

Re: [PATCH] powerpc/powernv: Turn on SCSI_AACRAID in powernv_defconfig

[Brian King]

On 02/22/2018 06:16 PM, Stewart Smith wrote:
> Michael Ellerman <mpe@ellerman.id.au> writes:
>> Brian King <brking@linux.vnet.ibm.com> writes:
>>> On 09/03/2017 06:19 PM, Stewart Smith wrote:
>>>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>>>>> 2. On a bare metal machine, if you set ipr.fast_reboot=1 on the skiboot
>>>>>>    kernel, then we should also avoid resetting the ipr adapter, so ipr
>>>>>>    init on the kernel being kexec booted from skiboot should be extremely fast. 
>>>>>
>>>>> OK, I didn't know that was an option, so that might help.
>>>>>
>>>>>> ...
>>>>>> If you've got cases where ipr init is taking a long time, I'd be
>>>>>> interested to know what scenarios are the most annoying to see if there
>>>>>> is any opportunity to improve.
>>>>>
>>>>> Yeah booting bare metal is where I see it (not using ipr.fast_reboot).
>>>>
>>>> Hrm... We should probably enable that by default for petitboot then.
>>>>
>>>> It'd at least cut some time off booting straight through to OS.
>>>
>>> Agreed. I'd be interested to hear if that helps address the issue
>>> Michael is seeing.
>>>
>>> You can easily test this by exiting to a petitboot shell:
>>>
>>> echo 1 > /sys/module/ipr/parameters/fast_reboot
>>>
>>> Then go back to petitboot and boot the OS.
>>
>> Just following up on this (!).
>>
>> This does work, and I've now been running it in my CI for about a month
>> (~1000 boots) with no problems.
>>
>> You can also make it persistent by doing:
>>
>>   $ nvram -p ibm,skiboot --update-config bootargs="ipr.fast_reboot=1"
> 
> Okay, cool. https://github.com/open-power/op-build/pull/1900 will set it
> in firmware - we may as well run with this and fix any bugs we find.
> 
> Any reason why it isn't the default behaviour?

The primary reason this isn't the default setting is because it would cause
undesired behavior on system reboots. On a kexec reboot, since the PCIe slots
don't get power cycled or even hit with PERST, we can simply quiesce devices
and pick them up at the other side of the kexec. On a real reboot where the
firmware or hardware may end up doing a PERST or power cycle, we need to tell the ipr
adapters that is coming and let them shutdown gracefully. If we don't,
we will likely be OK in the single adapter configuration in most scenarios,
but for a dual ipr adapter configuration, we can then end up with undesired
adapter failovers occurring due to the unexpected power offs / resets, which
can then end up extending the next boot.

-Brian 

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center