Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Jon Hunter]

Hi Greg,

On 23/05/2022 18:03, Greg Kroah-Hartman wrote:
> This is the start of the stable review cycle for the 4.9.316 release.
> There are 25 patches in this series, all will be posted as a response
> to this one.  If anyone has any issues with these being applied, please
> let me know.
> 
> Responses should be made by Wed, 25 May 2022 16:56:55 +0000.
> Anything received after that time might be too late.
> 
> The whole patch series can be found in one patch at:
> 	https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.9.316-rc1.gz
> or in the git tree and branch at:
> 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.9.y
> and the diffstat can be found below.
> 
> thanks,
> 
> greg k-h
> 
> -------------
> Pseudo-Shortlog of commits:

...

> Ard Biesheuvel <ardb@kernel.org>
>      ARM: 9196/1: spectre-bhb: enable for Cortex-A15


I am seeing a boot regression on tegra124-jetson-tk1 and reverting the 
above commit is fixing the problem. This also appears to impact 
linux-4.14.y, 4.19.y and 5.4.y.

Test results for stable-v4.9:
     8 builds:	8 pass, 0 fail
     18 boots:	16 pass, 2 fail
     18 tests:	18 pass, 0 fail

Linux version:	4.9.316-rc1-gbe4ec3e3faa1
Boards tested:	tegra124-jetson-tk1, tegra20-ventana,
                 tegra210-p2371-2180, tegra30-cardhu-a04

Boot failures:	tegra124-jetson-tk1

Thanks
Jon

-- 
nvpublic

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Greg Kroah-Hartman]

On Tue, May 24, 2022 at 09:32:07AM +0100, Jon Hunter wrote:
> Hi Greg,
> 
> On 23/05/2022 18:03, Greg Kroah-Hartman wrote:
> > This is the start of the stable review cycle for the 4.9.316 release.
> > There are 25 patches in this series, all will be posted as a response
> > to this one.  If anyone has any issues with these being applied, please
> > let me know.
> > 
> > Responses should be made by Wed, 25 May 2022 16:56:55 +0000.
> > Anything received after that time might be too late.
> > 
> > The whole patch series can be found in one patch at:
> > 	https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.9.316-rc1.gz
> > or in the git tree and branch at:
> > 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.9.y
> > and the diffstat can be found below.
> > 
> > thanks,
> > 
> > greg k-h
> > 
> > -------------
> > Pseudo-Shortlog of commits:
> 
> ...
> 
> > Ard Biesheuvel <ardb@kernel.org>
> >      ARM: 9196/1: spectre-bhb: enable for Cortex-A15
> 
> 
> I am seeing a boot regression on tegra124-jetson-tk1 and reverting the above
> commit is fixing the problem. This also appears to impact linux-4.14.y,
> 4.19.y and 5.4.y.
> 
> Test results for stable-v4.9:
>     8 builds:	8 pass, 0 fail
>     18 boots:	16 pass, 2 fail
>     18 tests:	18 pass, 0 fail
> 
> Linux version:	4.9.316-rc1-gbe4ec3e3faa1
> Boards tested:	tegra124-jetson-tk1, tegra20-ventana,
>                 tegra210-p2371-2180, tegra30-cardhu-a04
> 
> Boot failures:	tegra124-jetson-tk1

Odd.  This is also in 5.10.y, right?  No issues there?  Are we missing
something?

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Jon Hunter]

On 24/05/2022 13:09, Greg Kroah-Hartman wrote:

...

>> I am seeing a boot regression on tegra124-jetson-tk1 and reverting the above
>> commit is fixing the problem. This also appears to impact linux-4.14.y,
>> 4.19.y and 5.4.y.
>>
>> Test results for stable-v4.9:
>>      8 builds:	8 pass, 0 fail
>>      18 boots:	16 pass, 2 fail
>>      18 tests:	18 pass, 0 fail
>>
>> Linux version:	4.9.316-rc1-gbe4ec3e3faa1
>> Boards tested:	tegra124-jetson-tk1, tegra20-ventana,
>>                  tegra210-p2371-2180, tegra30-cardhu-a04
>>
>> Boot failures:	tegra124-jetson-tk1
> 
> Odd.  This is also in 5.10.y, right?  No issues there?  Are we missing
> something?


Actually, the more I look at this, the more I see various intermittent 
reports with this and it is also impacting the mainline.

The problem is that the commit in question is causing a ton of messages 
to be printed a boot and this sometimes is causing the boot test to fail 
because the boot is taking too long. The console shows ...

[ 1233.327547] CPU0: Spectre BHB: using loop workaround
[ 1233.327795] CPU1: Spectre BHB: using loop workaround
[ 1233.328270] CPU1: Spectre BHB: using loop workaround
[ 1233.328700] CPU1: Spectre BHB: using loop workaround
[ 1233.355477] CPU2: Spectre BHB: using loop workaround
** 7 printk messages dropped **
[ 1233.366271] CPU0: Spectre BHB: using loop workaround
[ 1233.366580] CPU0: Spectre BHB: using loop workaround
[ 1233.366815] CPU1: Spectre BHB: using loop workaround
[ 1233.405475] CPU1: Spectre BHB: using loop workaround
[ 1233.405874] CPU0: Spectre BHB: using loop workaround
[ 1233.406041] CPU1: Spectre BHB: using loop workaround
** 1 printk messages dropped **

There is a similar report of this [0] and I believe that we need a 
similar fix for the above prints as well. I have reported this to Ard 
[1]. So I am not sure that these Spectre BHB patches are quite ready for 
stable.

Cheers
Jon

[0] 
https://lore.kernel.org/linux-arm-kernel/20220519161310.1489625-1-dmitry.osipenko@collabora.com/T/
[1] 
https://lore.kernel.org/linux-arm-kernel/a589f56d-a0e1-328d-e4be-9427342d46b8@nvidia.com/

-- 
nvpublic

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Greg Kroah-Hartman]

On Tue, May 24, 2022 at 03:55:58PM +0100, Jon Hunter wrote:
> 
> On 24/05/2022 13:09, Greg Kroah-Hartman wrote:
> 
> ...
> 
> > > I am seeing a boot regression on tegra124-jetson-tk1 and reverting the above
> > > commit is fixing the problem. This also appears to impact linux-4.14.y,
> > > 4.19.y and 5.4.y.
> > > 
> > > Test results for stable-v4.9:
> > >      8 builds:	8 pass, 0 fail
> > >      18 boots:	16 pass, 2 fail
> > >      18 tests:	18 pass, 0 fail
> > > 
> > > Linux version:	4.9.316-rc1-gbe4ec3e3faa1
> > > Boards tested:	tegra124-jetson-tk1, tegra20-ventana,
> > >                  tegra210-p2371-2180, tegra30-cardhu-a04
> > > 
> > > Boot failures:	tegra124-jetson-tk1
> > 
> > Odd.  This is also in 5.10.y, right?  No issues there?  Are we missing
> > something?
> 
> 
> Actually, the more I look at this, the more I see various intermittent
> reports with this and it is also impacting the mainline.
> 
> The problem is that the commit in question is causing a ton of messages to
> be printed a boot and this sometimes is causing the boot test to fail
> because the boot is taking too long. The console shows ...
> 
> [ 1233.327547] CPU0: Spectre BHB: using loop workaround
> [ 1233.327795] CPU1: Spectre BHB: using loop workaround
> [ 1233.328270] CPU1: Spectre BHB: using loop workaround
> [ 1233.328700] CPU1: Spectre BHB: using loop workaround
> [ 1233.355477] CPU2: Spectre BHB: using loop workaround
> ** 7 printk messages dropped **
> [ 1233.366271] CPU0: Spectre BHB: using loop workaround
> [ 1233.366580] CPU0: Spectre BHB: using loop workaround
> [ 1233.366815] CPU1: Spectre BHB: using loop workaround
> [ 1233.405475] CPU1: Spectre BHB: using loop workaround
> [ 1233.405874] CPU0: Spectre BHB: using loop workaround
> [ 1233.406041] CPU1: Spectre BHB: using loop workaround
> ** 1 printk messages dropped **
> 
> There is a similar report of this [0] and I believe that we need a similar
> fix for the above prints as well. I have reported this to Ard [1]. So I am
> not sure that these Spectre BHB patches are quite ready for stable.

These patches are quite small, and just enable it for this known-broken
cpu type.

If there is an issue enabling it for this cpu type, then we can work on
that upstream, but there shouldn't be a reason to prevent this from
being merged now, especially given that it is supposed to be fixing a
known issue.

thanks,

greg k-h

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Florian Fainelli]

On 5/24/22 08:06, Greg Kroah-Hartman wrote:
> On Tue, May 24, 2022 at 03:55:58PM +0100, Jon Hunter wrote:
>>
>> On 24/05/2022 13:09, Greg Kroah-Hartman wrote:
>>
>> ...
>>
>>>> I am seeing a boot regression on tegra124-jetson-tk1 and reverting the above
>>>> commit is fixing the problem. This also appears to impact linux-4.14.y,
>>>> 4.19.y and 5.4.y.
>>>>
>>>> Test results for stable-v4.9:
>>>>       8 builds:	8 pass, 0 fail
>>>>       18 boots:	16 pass, 2 fail
>>>>       18 tests:	18 pass, 0 fail
>>>>
>>>> Linux version:	4.9.316-rc1-gbe4ec3e3faa1
>>>> Boards tested:	tegra124-jetson-tk1, tegra20-ventana,
>>>>                   tegra210-p2371-2180, tegra30-cardhu-a04
>>>>
>>>> Boot failures:	tegra124-jetson-tk1
>>>
>>> Odd.  This is also in 5.10.y, right?  No issues there?  Are we missing
>>> something?
>>
>>
>> Actually, the more I look at this, the more I see various intermittent
>> reports with this and it is also impacting the mainline.
>>
>> The problem is that the commit in question is causing a ton of messages to
>> be printed a boot and this sometimes is causing the boot test to fail
>> because the boot is taking too long. The console shows ...
>>
>> [ 1233.327547] CPU0: Spectre BHB: using loop workaround
>> [ 1233.327795] CPU1: Spectre BHB: using loop workaround
>> [ 1233.328270] CPU1: Spectre BHB: using loop workaround
>> [ 1233.328700] CPU1: Spectre BHB: using loop workaround
>> [ 1233.355477] CPU2: Spectre BHB: using loop workaround
>> ** 7 printk messages dropped **
>> [ 1233.366271] CPU0: Spectre BHB: using loop workaround
>> [ 1233.366580] CPU0: Spectre BHB: using loop workaround
>> [ 1233.366815] CPU1: Spectre BHB: using loop workaround
>> [ 1233.405475] CPU1: Spectre BHB: using loop workaround
>> [ 1233.405874] CPU0: Spectre BHB: using loop workaround
>> [ 1233.406041] CPU1: Spectre BHB: using loop workaround
>> ** 1 printk messages dropped **
>>
>> There is a similar report of this [0] and I believe that we need a similar
>> fix for the above prints as well. I have reported this to Ard [1]. So I am
>> not sure that these Spectre BHB patches are quite ready for stable.
> 
> These patches are quite small, and just enable it for this known-broken
> cpu type.
> 
> If there is an issue enabling it for this cpu type, then we can work on
> that upstream, but there shouldn't be a reason to prevent this from
> being merged now, especially given that it is supposed to be fixing a
> known issue.

Jonathan any chance this is Tegra specific? Our ARCH_BRCMSTB SoCs which 
use a Brahma-B15 which uses nearly the same ca15 processor functions 
defined in arch/arm/mm/proc-v7.S reports the following *before* changes:

[    0.001641] CPU: Testing write buffer coherency: ok
[    0.001685] CPU0: Spectre v2: using ICIALLU workaround
[    0.001703] ftrace: allocating 30541 entries in 120 pages
[    0.044600] CPU0: update cpu_capacity 1024
[    0.044633] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.044662] Setting up static identity map for 0x200000 - 0x200060
[    0.047410] brcmstb: biuctrl: MCP: Write pairing already disabled
[    0.048974] CPU1: update cpu_capacity 1024
[    0.048978] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.048981] CPU1: Spectre v2: using ICIALLU workaround
[    0.050234] CPU2: update cpu_capacity 1024
[    0.050238] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.050241] CPU2: Spectre v2: using ICIALLU workaround
[    0.051437] CPU3: update cpu_capacity 1024
[    0.051441] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.051444] CPU3: Spectre v2: using ICIALLU workaround
[    0.051532] Brought up 4 CPUs

and this *after* merging 4.9.316-rc1:

[    0.001626] CPU: Testing write buffer coherency: ok
[    0.001670] CPU0: Spectre v2: using ICIALLU workaround
[    0.001689] CPU0: Spectre BHB: using loop workaround
[    0.001705] ftrace: allocating 30542 entries in 120 pages
[    0.043752] CPU0: update cpu_capacity 1024
[    0.043784] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.043813] Setting up static identity map for 0x200000 - 0x200060
[    0.046547] brcmstb: biuctrl: MCP: Write pairing already disabled
[    0.048121] CPU1: update cpu_capacity 1024
[    0.048124] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.048129] CPU1: Spectre v2: using ICIALLU workaround
[    0.048165] CPU1: Spectre BHB: using loop workaround
[    0.049398] CPU2: update cpu_capacity 1024
[    0.049402] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.049405] CPU2: Spectre v2: using ICIALLU workaround
[    0.049440] CPU2: Spectre BHB: using loop workaround
[    0.050613] CPU3: update cpu_capacity 1024
[    0.050617] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.050619] CPU3: Spectre v2: using ICIALLU workaround
[    0.050653] CPU3: Spectre BHB: using loop workaround
[    0.050722] Brought up 4 CPUs
[    0.050738] SMP: Total of 4 processors activated (216.00 BogoMIPS).
[    0.050753] CPU: All CPU(s) started in HYP mode.
-- 
Florian

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Jon Hunter]

On 24/05/2022 17:30, Florian Fainelli wrote:

...

> Jonathan any chance this is Tegra specific? Our ARCH_BRCMSTB SoCs which 
> use a Brahma-B15 which uses nearly the same ca15 processor functions 
> defined in arch/arm/mm/proc-v7.S reports the following *before* changes:
> 
> [    0.001641] CPU: Testing write buffer coherency: ok
> [    0.001685] CPU0: Spectre v2: using ICIALLU workaround
> [    0.001703] ftrace: allocating 30541 entries in 120 pages
> [    0.044600] CPU0: update cpu_capacity 1024
> [    0.044633] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> [    0.044662] Setting up static identity map for 0x200000 - 0x200060
> [    0.047410] brcmstb: biuctrl: MCP: Write pairing already disabled
> [    0.048974] CPU1: update cpu_capacity 1024
> [    0.048978] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> [    0.048981] CPU1: Spectre v2: using ICIALLU workaround
> [    0.050234] CPU2: update cpu_capacity 1024
> [    0.050238] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
> [    0.050241] CPU2: Spectre v2: using ICIALLU workaround
> [    0.051437] CPU3: update cpu_capacity 1024
> [    0.051441] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
> [    0.051444] CPU3: Spectre v2: using ICIALLU workaround
> [    0.051532] Brought up 4 CPUs
> 
> and this *after* merging 4.9.316-rc1:
> 
> [    0.001626] CPU: Testing write buffer coherency: ok
> [    0.001670] CPU0: Spectre v2: using ICIALLU workaround
> [    0.001689] CPU0: Spectre BHB: using loop workaround
> [    0.001705] ftrace: allocating 30542 entries in 120 pages
> [    0.043752] CPU0: update cpu_capacity 1024
> [    0.043784] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> [    0.043813] Setting up static identity map for 0x200000 - 0x200060
> [    0.046547] brcmstb: biuctrl: MCP: Write pairing already disabled
> [    0.048121] CPU1: update cpu_capacity 1024
> [    0.048124] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> [    0.048129] CPU1: Spectre v2: using ICIALLU workaround
> [    0.048165] CPU1: Spectre BHB: using loop workaround
> [    0.049398] CPU2: update cpu_capacity 1024
> [    0.049402] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
> [    0.049405] CPU2: Spectre v2: using ICIALLU workaround
> [    0.049440] CPU2: Spectre BHB: using loop workaround
> [    0.050613] CPU3: update cpu_capacity 1024
> [    0.050617] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
> [    0.050619] CPU3: Spectre v2: using ICIALLU workaround
> [    0.050653] CPU3: Spectre BHB: using loop workaround
> [    0.050722] Brought up 4 CPUs
> [    0.050738] SMP: Total of 4 processors activated (216.00 BogoMIPS).
> [    0.050753] CPU: All CPU(s) started in HYP mode.


Does your platform support CPU idle? I see this being triggered
during CPU idle transitions ...

[    4.415167] CPU0: Spectre BHB: using loop workaround
[    4.417621] [<c01109a0>] (unwind_backtrace) from [<c010b7ac>] (show_stack+0x10/0x14)
[    4.430291] [<c010b7ac>] (show_stack) from [<c09c2b38>] (dump_stack+0xc0/0xd4)
[    4.437512] [<c09c2b38>] (dump_stack) from [<c011a6c8>] (cpu_v7_spectre_bhb_init+0xd8/0x190)
[    4.445943] [<c011a6c8>] (cpu_v7_spectre_bhb_init) from [<c010dee8>] (cpu_suspend+0xac/0xc8)
[    4.454377] [<c010dee8>] (cpu_suspend) from [<c011e7e4>] (tegra114_idle_power_down+0x74/0x78)
[    4.462898] [<c011e7e4>] (tegra114_idle_power_down) from [<c06d3b44>] (cpuidle_enter_state+0x130/0x524)
[    4.472286] [<c06d3b44>] (cpuidle_enter_state) from [<c0164a30>] (do_idle+0x1b0/0x200)
[    4.480199] [<c0164a30>] (do_idle) from [<c0164d28>] (cpu_startup_entry+0x18/0x1c)
[    4.487762] [<c0164d28>] (cpu_startup_entry) from [<801018cc>] (0x801018cc)

Jon

-- 
nvpublic

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Florian Fainelli]

[-- Attachment #1: Type: text/plain, Size: 4336 bytes --]

On 5/24/22 10:50, Jon Hunter wrote:
> 
> On 24/05/2022 17:30, Florian Fainelli wrote:
> 
> ...
> 
>> Jonathan any chance this is Tegra specific? Our ARCH_BRCMSTB SoCs 
>> which use a Brahma-B15 which uses nearly the same ca15 processor 
>> functions defined in arch/arm/mm/proc-v7.S reports the following 
>> *before* changes:
>>
>> [    0.001641] CPU: Testing write buffer coherency: ok
>> [    0.001685] CPU0: Spectre v2: using ICIALLU workaround
>> [    0.001703] ftrace: allocating 30541 entries in 120 pages
>> [    0.044600] CPU0: update cpu_capacity 1024
>> [    0.044633] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
>> [    0.044662] Setting up static identity map for 0x200000 - 0x200060
>> [    0.047410] brcmstb: biuctrl: MCP: Write pairing already disabled
>> [    0.048974] CPU1: update cpu_capacity 1024
>> [    0.048978] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
>> [    0.048981] CPU1: Spectre v2: using ICIALLU workaround
>> [    0.050234] CPU2: update cpu_capacity 1024
>> [    0.050238] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
>> [    0.050241] CPU2: Spectre v2: using ICIALLU workaround
>> [    0.051437] CPU3: update cpu_capacity 1024
>> [    0.051441] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
>> [    0.051444] CPU3: Spectre v2: using ICIALLU workaround
>> [    0.051532] Brought up 4 CPUs
>>
>> and this *after* merging 4.9.316-rc1:
>>
>> [    0.001626] CPU: Testing write buffer coherency: ok
>> [    0.001670] CPU0: Spectre v2: using ICIALLU workaround
>> [    0.001689] CPU0: Spectre BHB: using loop workaround
>> [    0.001705] ftrace: allocating 30542 entries in 120 pages
>> [    0.043752] CPU0: update cpu_capacity 1024
>> [    0.043784] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
>> [    0.043813] Setting up static identity map for 0x200000 - 0x200060
>> [    0.046547] brcmstb: biuctrl: MCP: Write pairing already disabled
>> [    0.048121] CPU1: update cpu_capacity 1024
>> [    0.048124] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
>> [    0.048129] CPU1: Spectre v2: using ICIALLU workaround
>> [    0.048165] CPU1: Spectre BHB: using loop workaround
>> [    0.049398] CPU2: update cpu_capacity 1024
>> [    0.049402] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
>> [    0.049405] CPU2: Spectre v2: using ICIALLU workaround
>> [    0.049440] CPU2: Spectre BHB: using loop workaround
>> [    0.050613] CPU3: update cpu_capacity 1024
>> [    0.050617] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
>> [    0.050619] CPU3: Spectre v2: using ICIALLU workaround
>> [    0.050653] CPU3: Spectre BHB: using loop workaround
>> [    0.050722] Brought up 4 CPUs
>> [    0.050738] SMP: Total of 4 processors activated (216.00 BogoMIPS).
>> [    0.050753] CPU: All CPU(s) started in HYP mode.
> 
> 
> Does your platform support CPU idle? I see this being triggered
> during CPU idle transitions ...

It does, but not with an idle state resulting in powering down secondary 
cores. We do have CPU hotplug with power gating as well as system wide 
suspend states that result in power gating secondaries and they appear 
to be working fine.

I use the attached script to randomly cycle hot plug/unplug through each 
4 cores and it has been running over 10k cycles.

> 
> [    4.415167] CPU0: Spectre BHB: using loop workaround
> [    4.417621] [<c01109a0>] (unwind_backtrace) from [<c010b7ac>] 
> (show_stack+0x10/0x14)
> [    4.430291] [<c010b7ac>] (show_stack) from [<c09c2b38>] 
> (dump_stack+0xc0/0xd4)
> [    4.437512] [<c09c2b38>] (dump_stack) from [<c011a6c8>] 
> (cpu_v7_spectre_bhb_init+0xd8/0x190)
> [    4.445943] [<c011a6c8>] (cpu_v7_spectre_bhb_init) from [<c010dee8>] 
> (cpu_suspend+0xac/0xc8)
> [    4.454377] [<c010dee8>] (cpu_suspend) from [<c011e7e4>] 
> (tegra114_idle_power_down+0x74/0x78)
> [    4.462898] [<c011e7e4>] (tegra114_idle_power_down) from [<c06d3b44>] 
> (cpuidle_enter_state+0x130/0x524)
> [    4.472286] [<c06d3b44>] (cpuidle_enter_state) from [<c0164a30>] 
> (do_idle+0x1b0/0x200)
> [    4.480199] [<c0164a30>] (do_idle) from [<c0164d28>] 
> (cpu_startup_entry+0x18/0x1c)
> [    4.487762] [<c0164d28>] (cpu_startup_entry) from [<801018cc>] 
> (0x801018cc)
> 
> Jon
> 


-- 
Florian

[-- Attachment #2: hotplug.sh --]
[-- Type: application/x-shellscript, Size: 699 bytes --]

Re: [PATCH 4.9 00/25] 4.9.316-rc1 review

[Jon Hunter]

On 24/05/2022 16:06, Greg Kroah-Hartman wrote:
> On Tue, May 24, 2022 at 03:55:58PM +0100, Jon Hunter wrote:
>>
>> On 24/05/2022 13:09, Greg Kroah-Hartman wrote:
>>
>> ...
>>
>>>> I am seeing a boot regression on tegra124-jetson-tk1 and reverting the above
>>>> commit is fixing the problem. This also appears to impact linux-4.14.y,
>>>> 4.19.y and 5.4.y.
>>>>
>>>> Test results for stable-v4.9:
>>>>       8 builds:	8 pass, 0 fail
>>>>       18 boots:	16 pass, 2 fail
>>>>       18 tests:	18 pass, 0 fail
>>>>
>>>> Linux version:	4.9.316-rc1-gbe4ec3e3faa1
>>>> Boards tested:	tegra124-jetson-tk1, tegra20-ventana,
>>>>                   tegra210-p2371-2180, tegra30-cardhu-a04
>>>>
>>>> Boot failures:	tegra124-jetson-tk1
>>>
>>> Odd.  This is also in 5.10.y, right?  No issues there?  Are we missing
>>> something?
>>
>>
>> Actually, the more I look at this, the more I see various intermittent
>> reports with this and it is also impacting the mainline.
>>
>> The problem is that the commit in question is causing a ton of messages to
>> be printed a boot and this sometimes is causing the boot test to fail
>> because the boot is taking too long. The console shows ...
>>
>> [ 1233.327547] CPU0: Spectre BHB: using loop workaround
>> [ 1233.327795] CPU1: Spectre BHB: using loop workaround
>> [ 1233.328270] CPU1: Spectre BHB: using loop workaround
>> [ 1233.328700] CPU1: Spectre BHB: using loop workaround
>> [ 1233.355477] CPU2: Spectre BHB: using loop workaround
>> ** 7 printk messages dropped **
>> [ 1233.366271] CPU0: Spectre BHB: using loop workaround
>> [ 1233.366580] CPU0: Spectre BHB: using loop workaround
>> [ 1233.366815] CPU1: Spectre BHB: using loop workaround
>> [ 1233.405475] CPU1: Spectre BHB: using loop workaround
>> [ 1233.405874] CPU0: Spectre BHB: using loop workaround
>> [ 1233.406041] CPU1: Spectre BHB: using loop workaround
>> ** 1 printk messages dropped **
>>
>> There is a similar report of this [0] and I believe that we need a similar
>> fix for the above prints as well. I have reported this to Ard [1]. So I am
>> not sure that these Spectre BHB patches are quite ready for stable.
> 
> These patches are quite small, and just enable it for this known-broken
> cpu type.
> 
> If there is an issue enabling it for this cpu type, then we can work on
> that upstream, but there shouldn't be a reason to prevent this from
> being merged now, especially given that it is supposed to be fixing a
> known issue.

Yes understand. I have been doing some more testing and with v4.9, this 
is triggering a boot timeout 100% of the time. So with 20 boots, all 20 
timeout. Note the timeout is 2 mins. With v4.14, I saw only 5 out of 20 
timeouts and so it would seem that v4.9 is slower to boot in general. I 
think that the more recent kernels show intermittent timeouts.

We have some verbose logging enabled on this platform, which until now 
has not been a problem, but I will disable this and that should resolve 
this for now.

Cheers
Jon	

-- 
nvpublic