Re: mainline/master bisection: baseline.login on odroid-xu3

Re: mainline/master bisection: baseline.login on odroid-xu3

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 9056 bytes --]

On Sun, Apr 17, 2022 at 04:59:47AM -0700, KernelCI bot wrote:

The KernelCI bisection bot found a boot regression on Odroid-XU3 with
commit 8d9d651ff2270 (ARM: use LOADADDR() to get load address of
section).  It appears to trigger a deadlock somehow at some point after
the secondary CPUs come up:

<6>[    0.133301] smp: Bringing up secondary CPUs ...
<6>[    0.138439] CPU1: thread -1, cpu 1, socket 1, mpidr 80000101
<6>[    0.139400] CPU2: thread -1, cpu 2, socket 1, mpidr 80000102
<6>[    0.140325] CPU3: thread -1, cpu 3, socket 1, mpidr 80000103
<6>[    0.141167] CPU4: thread -1, cpu 0, socket 0, mpidr 80000000
<6>[    0.141185] CPU4: Spectre v2: using ICIALLU workaround
<6>[    0.168676] CPU4: Spectre BHB: using loop workaround
<5>[    0.780471] random: fast init done
<3>[   21.167354] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
<3>[   21.172051] rcu: 	4-....: (0 ticks this GP) idle=001/1/0x40000000 softirq=0/0 fqs=1050 
<4>[   21.180284] 	(detected by 0, t=2102 jiffies, g=-1199, q=1)
<6>[   21.186003] Sending NMI from CPU 0 to CPUs 4:
<5>[   44.300722] random: crng init done
<3>[   94.237962] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
<3>[   94.242664] rcu: 	4-....: (0 ticks this GP) idle=001/1/0x40000000 softirq=0/0 fqs=4202 
<4>[   94.250897] 	(detected by 0, t=8407 jiffies, g=-1199, q=1)
<6>[   94.256616] Sending NMI from CPU 0 to CPUs 4:
<3>[  167.308571] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
<3>[  167.313269] rcu: 	4-....: (0 ticks this GP) idle=001/1/0x40000000 softirq=0/0 fqs=7354 
<4>[  167.321502] 	(detected by 0, t=14712 jiffies, g=-1199, q=1)
<6>[  167.327308] Sending NMI from CPU 0 to CPUs 4:

I've not deleted any context below for reference, this includes links to
full logs of the failure, a log of the bisection and a tag for the
report from the bot:

> * If you do send a fix, please include this trailer:            *
> *   Reported-by: "kernelci.org bot" <bot@kernelci.org>          *
> *                                                               *
> * Hope this helps!                                              *
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

> 
> mainline/master bisection: baseline.login on odroid-xu3
> 
> Summary:
>   Start:      90ea17a9e27b5 Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
>   Plain log:  https://storage.kernelci.org/mainline/master/v5.18-rc2-291-g90ea17a9e27b5/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-collabora/baseline-odroid-xu3.txt
>   HTML log:   https://storage.kernelci.org/mainline/master/v5.18-rc2-291-g90ea17a9e27b5/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-collabora/baseline-odroid-xu3.html
>   Result:     8d9d651ff2270 ARM: use LOADADDR() to get load address of sections
> 
> Checks:
>   revert:     PASS
>   verify:     PASS
> 
> Parameters:
>   Tree:       mainline
>   URL:        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>   Branch:     master
>   Target:     odroid-xu3
>   CPU arch:   arm
>   Lab:        lab-collabora
>   Compiler:   gcc-10
>   Config:     multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y
>   Test case:  baseline.login
> 
> Breaking commit found:
> 
> -------------------------------------------------------------------------------
> commit 8d9d651ff2270a632e9dc497b142db31e8911315
> Author: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
> Date:   Fri Feb 11 19:49:50 2022 +0000
> 
>     ARM: use LOADADDR() to get load address of sections
>     
>     Use the linker's LOADADDR() macro to get the load address of the
>     sections, and provide a macro to set the start and end symbols.
>     
>     Acked-by: Catalin Marinas <catalin.marinas@arm.com>
>     Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
> 
> diff --git a/arch/arm/include/asm/vmlinux.lds.h b/arch/arm/include/asm/vmlinux.lds.h
> index 4a91428c324db..e02710d17cf93 100644
> --- a/arch/arm/include/asm/vmlinux.lds.h
> +++ b/arch/arm/include/asm/vmlinux.lds.h
> @@ -26,6 +26,11 @@
>  #define ARM_MMU_DISCARD(x)	x
>  #endif
>  
> +/* Set start/end symbol names to the LMA for the section */
> +#define ARM_LMA(sym, section)						\
> +	sym##_start = LOADADDR(section);				\
> +	sym##_end = LOADADDR(section) + SIZEOF(section)
> +
>  #define PROC_INFO							\
>  		. = ALIGN(4);						\
>  		__proc_info_begin = .;					\
> @@ -110,19 +115,19 @@
>   * only thing that matters is their relative offsets
>   */
>  #define ARM_VECTORS							\
> -	__vectors_start = .;						\
> +	__vectors_lma = .;						\
>  	.vectors 0xffff0000 : AT(__vectors_start) {			\
>  		*(.vectors)						\
>  	}								\
> -	. = __vectors_start + SIZEOF(.vectors);				\
> -	__vectors_end = .;						\
> +	ARM_LMA(__vectors, .vectors);					\
> +	. = __vectors_lma + SIZEOF(.vectors);				\
>  									\
> -	__stubs_start = .;						\
> -	.stubs ADDR(.vectors) + 0x1000 : AT(__stubs_start) {		\
> +	__stubs_lma = .;						\
> +	.stubs ADDR(.vectors) + 0x1000 : AT(__stubs_lma) {		\
>  		*(.stubs)						\
>  	}								\
> -	. = __stubs_start + SIZEOF(.stubs);				\
> -	__stubs_end = .;						\
> +	ARM_LMA(__stubs, .stubs);					\
> +	. = __stubs_lma + SIZEOF(.stubs);				\
>  									\
>  	PROVIDE(vector_fiq_offset = vector_fiq - ADDR(.vectors));
> -------------------------------------------------------------------------------
> 
> 
> Git bisection log:
> 
> -------------------------------------------------------------------------------
> git bisect start
> # good: [ea4424be16887a37735d6550cfd0611528dbe5d9] Merge tag 'mtd/fixes-for-5.17-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux
> git bisect good ea4424be16887a37735d6550cfd0611528dbe5d9
> # bad: [90ea17a9e27b5778ec517efb1ce0b81d36905654] Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
> git bisect bad 90ea17a9e27b5778ec517efb1ce0b81d36905654
> # bad: [169e77764adc041b1dacba84ea90516a895d43b2] Merge tag 'net-next-5.18' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
> git bisect bad 169e77764adc041b1dacba84ea90516a895d43b2
> # bad: [baaa68a9796ef2cadfe5caaf4c730412eda0f31c] Merge tag 'arm-soc-5.18' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
> git bisect bad baaa68a9796ef2cadfe5caaf4c730412eda0f31c
> # bad: [5191290407668028179f2544a11ae9b57f0bcf07] Merge tag 'for-5.18-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
> git bisect bad 5191290407668028179f2544a11ae9b57f0bcf07
> # bad: [5628b8de1228436d47491c662dc521bc138a3d43] Merge tag 'random-5.18-rc1-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/crng/random
> git bisect bad 5628b8de1228436d47491c662dc521bc138a3d43
> # bad: [eaa54b1458ca84092e513d554dd6d234245e6bef] Merge tag 'x86_cleanups_for_v5.18_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad eaa54b1458ca84092e513d554dd6d234245e6bef
> # bad: [c81801eb7f2476a25d8fb27449e01b0bef46908a] Merge tag 'acpi-5.17-rc9' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
> git bisect bad c81801eb7f2476a25d8fb27449e01b0bef46908a
> # bad: [c30b5b8cfb72cc83c10793588ce94471a4769409] Merge tag 'staging-5.17-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect bad c30b5b8cfb72cc83c10793588ce94471a4769409
> # bad: [3bf7edc84a9eb4007dd9a0cb8878a7e1d5ec6a3b] Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> git bisect bad 3bf7edc84a9eb4007dd9a0cb8878a7e1d5ec6a3b
> # skip: [cd22a8bfcfe07ef35b1b5d96dd468e92dc5e9d8a] Merge tag 'arm64-spectre-bhb-for-v5.17-2' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> git bisect skip cd22a8bfcfe07ef35b1b5d96dd468e92dc5e9d8a
> # good: [bd09128d16fac3c34b80bd6a29088ac632e8ce09] arm64: Add percpu vectors for EL1
> git bisect good bd09128d16fac3c34b80bd6a29088ac632e8ce09
> # bad: [b9baf5c8c5c356757f4f9d8180b5e9d234065bc3] ARM: Spectre-BHB workaround
> git bisect bad b9baf5c8c5c356757f4f9d8180b5e9d234065bc3
> # good: [04e91b7324760a377a725e218b5ee783826d30f5] ARM: early traps initialisation
> git bisect good 04e91b7324760a377a725e218b5ee783826d30f5
> # bad: [8d9d651ff2270a632e9dc497b142db31e8911315] ARM: use LOADADDR() to get load address of sections
> git bisect bad 8d9d651ff2270a632e9dc497b142db31e8911315
> # first bad commit: [8d9d651ff2270a632e9dc497b142db31e8911315] ARM: use LOADADDR() to get load address of sections
> -------------------------------------------------------------------------------
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Groups.io Links: You receive all messages sent to this group.
> View/Reply Online (#25908): https://groups.io/g/kernelci-results/message/25908
> Mute This Topic: https://groups.io/mt/90521049/1131744
> Group Owner: kernelci-results+owner@groups.io
> Unsubscribe: https://groups.io/g/kernelci-results/unsub [broonie@kernel.org]
> -=-=-=-=-=-=-=-=-=-=-=-
> 
> 

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: mainline/master bisection: baseline.login on odroid-xu3

[Russell King (Oracle)]

On Sun, Apr 17, 2022 at 08:35:29PM +0100, Mark Brown wrote:
> On Sun, Apr 17, 2022 at 04:59:47AM -0700, KernelCI bot wrote:
> 
> The KernelCI bisection bot found a boot regression on Odroid-XU3 with
> commit 8d9d651ff2270 (ARM: use LOADADDR() to get load address of
> section).  It appears to trigger a deadlock somehow at some point after
> the secondary CPUs come up:

If the primary CPU is still happy, then the secondary CPUs should be.
Quite simply, if something is broken in the setup of the vectors, the
primary CPU would also be affected by this commit. However...

> <6>[    0.133301] smp: Bringing up secondary CPUs ...
> <6>[    0.138439] CPU1: thread -1, cpu 1, socket 1, mpidr 80000101
> <6>[    0.139400] CPU2: thread -1, cpu 2, socket 1, mpidr 80000102
> <6>[    0.140325] CPU3: thread -1, cpu 3, socket 1, mpidr 80000103
> <6>[    0.141167] CPU4: thread -1, cpu 0, socket 0, mpidr 80000000
> <6>[    0.141185] CPU4: Spectre v2: using ICIALLU workaround
> <6>[    0.168676] CPU4: Spectre BHB: using loop workaround

Given that there was a problem with the loop implementation, please
ensure that when you bisect, you have the following commit in place
every time you test a bisect point:

6c7cb60bff7a ("ARM: fix Thumb2 regression with Spectre BHB")

Otherwise, yes, it is known that Thumb2 will be broken.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: mainline/master bisection: baseline.login on odroid-xu3

[Russell King (Oracle)]

On Sun, Apr 17, 2022 at 10:27:58PM +0100, Russell King (Oracle) wrote:
> On Sun, Apr 17, 2022 at 08:35:29PM +0100, Mark Brown wrote:
> > On Sun, Apr 17, 2022 at 04:59:47AM -0700, KernelCI bot wrote:
> > 
> > The KernelCI bisection bot found a boot regression on Odroid-XU3 with
> > commit 8d9d651ff2270 (ARM: use LOADADDR() to get load address of
> > section).  It appears to trigger a deadlock somehow at some point after
> > the secondary CPUs come up:
> 
> If the primary CPU is still happy, then the secondary CPUs should be.
> Quite simply, if something is broken in the setup of the vectors, the
> primary CPU would also be affected by this commit. However...
> 
> > <6>[    0.133301] smp: Bringing up secondary CPUs ...
> > <6>[    0.138439] CPU1: thread -1, cpu 1, socket 1, mpidr 80000101
> > <6>[    0.139400] CPU2: thread -1, cpu 2, socket 1, mpidr 80000102
> > <6>[    0.140325] CPU3: thread -1, cpu 3, socket 1, mpidr 80000103
> > <6>[    0.141167] CPU4: thread -1, cpu 0, socket 0, mpidr 80000000
> > <6>[    0.141185] CPU4: Spectre v2: using ICIALLU workaround
> > <6>[    0.168676] CPU4: Spectre BHB: using loop workaround
> 
> Given that there was a problem with the loop implementation, please
> ensure that when you bisect, you have the following commit in place
> every time you test a bisect point:
> 
> 6c7cb60bff7a ("ARM: fix Thumb2 regression with Spectre BHB")
> 
> Otherwise, yes, it is known that Thumb2 will be broken.

Also, I should say, if this commit is really the case:

8d9d651ff227 ("ARM: use LOADADDR() to get load address of sections")

then I need to see two vmlinux binaries (the decompressed kernel ELF
image) for analysis of why this commit is causing the failure.
Basically, I need to disassemble the code and see what the toolchain
is doing to screw up the kernel.

If it isn't obvious from that, then I have no idea why this commit
would cause such a strange regression.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: mainline/master bisection: baseline.login on odroid-xu3

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 1643 bytes --]

On Sun, Apr 17, 2022 at 10:27:58PM +0100, Russell King (Oracle) wrote:

> If the primary CPU is still happy, then the secondary CPUs should be.
> Quite simply, if something is broken in the setup of the vectors, the
> primary CPU would also be affected by this commit. However...

Yes, I was quite confused.

> > <6>[    0.141185] CPU4: Spectre v2: using ICIALLU workaround
> > <6>[    0.168676] CPU4: Spectre BHB: using loop workaround

> Given that there was a problem with the loop implementation, please
> ensure that when you bisect, you have the following commit in place
> every time you test a bisect point:

> 6c7cb60bff7a ("ARM: fix Thumb2 regression with Spectre BHB")

> Otherwise, yes, it is known that Thumb2 will be broken.

Ah, that won't have been in there (since the bot is just running off
what it finds in git) and indeed looking at the most recent results:

   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.18-rc2-351-ga2c29ccd94778/plan/baseline/

it does look like it's just Thumb2 that's failing.  However that's got
Linus' tree as of a2c29ccd9477861b16ddc02c411a6c9665250558 which does
feature the above commit so there's something else off (I expect we'll
see newer results soon at https://linux.kernelci.org/job/mainline/ since
it looks like Linus just tagged -rc3).  It does seem to be only this
board, other boards are booting the Thumb2 kernel fine, so I'd not be
surprised if missing the above commit just confused the bisection.

We'd need to rerun a bisection with the above commit in place I guess, I
don't have access to trigger that - Guillaume (in CCs) can hopefully
help next week.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel