[PATCH v2 0/2] bcm4908: Fix secondary CPU initialization

[PATCH v2 0/2] bcm4908: Fix secondary CPU initialization

[Sam Edwards]

Hello list,

This is v2 of my previous patch [1] for resolving a problem preventing
secondary CPU(s) from coming up on bcm4908. After some discussion, I decided to
try dropping the reserved memory region from 64K to only 4K. Looks like it
works!

Changes v1->v2:
- Reduce 64K reserved region to 4K
- Style change to the `reg` property to use hex instead of decimal
- Slight rephrasing to the commit message
- Add a new patch that also moves the `cpu-release-addr` into this reserved
  memory region

Cheers,
Sam

[1]: https://lore.kernel.org/lkml/20241003213007.1339811-1-CFSworks@gmail.com/T/

Sam Edwards (2):
  arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area
  arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr

 .../boot/dts/broadcom/bcmbca/bcm4908.dtsi      | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

-- 
2.44.2

[PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

[Sam Edwards]

The CFE bootloader places a stub program in the first page of physical
memory to hold the secondary CPUs until the boot CPU writes the release
address, but does not splice a /reserved-memory node into the FDT to
protect it. If Linux overwrites this program before execution reaches
smp_prepare_cpus(), the secondary CPUs may become inaccessible.

This is only a problem with CFE, and then only until the secondary CPUs
are brought online. Ideally, there would be some hypothetical mechanism
we could use to indicate that this area of memory is sensitive only
during boot. But as there is none, and since it is such a small amount
of memory, it is easiest to reserve it unconditionally.

Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the
first 4KiB of physical memory.

Signed-off-by: Sam Edwards <CFSworks@gmail.com>
---
 arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
index 8b924812322c..c51b92387fad 100644
--- a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
+++ b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
@@ -68,6 +68,16 @@ l2: l2-cache0 {
 		};
 	};
 
+	reserved-memory {
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges;
+
+		cfe-stub@0 {
+			reg = <0x0 0x0 0x0 0x1000>;
+		};
+	};
+
 	axi@81000000 {
 		compatible = "simple-bus";
 		#address-cells = <1>;
-- 
2.44.2

Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

[Florian Fainelli]

On 10/4/24 22:01, Sam Edwards wrote:
> The CFE bootloader places a stub program in the first page of physical
> memory to hold the secondary CPUs until the boot CPU writes the release
> address, but does not splice a /reserved-memory node into the FDT to
> protect it. If Linux overwrites this program before execution reaches
> smp_prepare_cpus(), the secondary CPUs may become inaccessible.
> 
> This is only a problem with CFE, and then only until the secondary CPUs
> are brought online. Ideally, there would be some hypothetical mechanism
> we could use to indicate that this area of memory is sensitive only
> during boot. But as there is none, and since it is such a small amount
> of memory, it is easiest to reserve it unconditionally.

If we supported CPU hotplug on those platforms (do we?) then it actually 
does matter that this memory remains protected, and it cannot be 
reclaimed. This does not invalidate the commit message and I will take 
it as-is, but it it is not memory that we can necessarily reclaim that 
easily, if we did things properly.

> 
> Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the
> first 4KiB of physical memory.
> 
> Signed-off-by: Sam Edwards <CFSworks@gmail.com>
> ---
>   arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> diff --git a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
> index 8b924812322c..c51b92387fad 100644
> --- a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
> +++ b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
> @@ -68,6 +68,16 @@ l2: l2-cache0 {
>   		};
>   	};
>   
> +	reserved-memory {
> +		#address-cells = <2>;
> +		#size-cells = <2>;
> +		ranges;
> +
> +		cfe-stub@0 {
> +			reg = <0x0 0x0 0x0 0x1000>;
> +		};
> +	};
> +
>   	axi@81000000 {
>   		compatible = "simple-bus";
>   		#address-cells = <1>;


-- 
Florian

Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

[Sam Edwards]

On Thu, Oct 10, 2024 at 3:57 PM Florian Fainelli
<florian.fainelli@broadcom.com> wrote:
>
> On 10/4/24 22:01, Sam Edwards wrote:
> > The CFE bootloader places a stub program in the first page of physical
> > memory to hold the secondary CPUs until the boot CPU writes the release
> > address, but does not splice a /reserved-memory node into the FDT to
> > protect it. If Linux overwrites this program before execution reaches
> > smp_prepare_cpus(), the secondary CPUs may become inaccessible.
> >
> > This is only a problem with CFE, and then only until the secondary CPUs
> > are brought online. Ideally, there would be some hypothetical mechanism
> > we could use to indicate that this area of memory is sensitive only
> > during boot. But as there is none, and since it is such a small amount
> > of memory, it is easiest to reserve it unconditionally.
>
> If we supported CPU hotplug on those platforms (do we?) then it actually
> does matter that this memory remains protected, and it cannot be
> reclaimed. This does not invalidate the commit message and I will take
> it as-is, but it it is not memory that we can necessarily reclaim that
> easily, if we did things properly.

I am looking at only one build of CFE, so don't take what I'm saying
as gospel, but as I understand it:

CFE implements only the spin-table method, which isn't dynamic. Once
the kernel calls for the secondary CPUs to be released, it cannot shut
them down and bring them up again, so they have no need to reenter the
spin stub.

With U-Boot+ATF, it's of course a different story: ATF needs to stay
resident in memory to implement PSCI (which _is_ dynamic), but U-Boot
inserts/overwrites the necessary DT structures to protect ATF and tell
Linux to use PSCI, so we don't need to be thinking about that here.

But, again, that's just my understanding. If there's a version of CFE
out there that loads ATF/similar or implements PSCI itself, then,
well, that'd invalidate this patch('s commit message). I doubt such a
variant of CFE exists, but you're much better equipped to confirm or
refute that possibility, given your familiarity with the platform and
internal details.

Best wishes,
Sam

>
> >
> > Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the
> > first 4KiB of physical memory.
> >
> > Signed-off-by: Sam Edwards <CFSworks@gmail.com>
> > ---
> >   arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++
> >   1 file changed, 10 insertions(+)
> >
> > diff --git a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
> > index 8b924812322c..c51b92387fad 100644
> > --- a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
> > +++ b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
> > @@ -68,6 +68,16 @@ l2: l2-cache0 {
> >               };
> >       };
> >
> > +     reserved-memory {
> > +             #address-cells = <2>;
> > +             #size-cells = <2>;
> > +             ranges;
> > +
> > +             cfe-stub@0 {
> > +                     reg = <0x0 0x0 0x0 0x1000>;
> > +             };
> > +     };
> > +
> >       axi@81000000 {
> >               compatible = "simple-bus";
> >               #address-cells = <1>;
>
>
> --
> Florian

Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

[Florian Fainelli]

From: Florian Fainelli <f.fainelli@gmail.com>

On Fri,  4 Oct 2024 22:01:54 -0700, Sam Edwards <cfsworks@gmail.com> wrote:
> The CFE bootloader places a stub program in the first page of physical
> memory to hold the secondary CPUs until the boot CPU writes the release
> address, but does not splice a /reserved-memory node into the FDT to
> protect it. If Linux overwrites this program before execution reaches
> smp_prepare_cpus(), the secondary CPUs may become inaccessible.
> 
> This is only a problem with CFE, and then only until the secondary CPUs
> are brought online. Ideally, there would be some hypothetical mechanism
> we could use to indicate that this area of memory is sensitive only
> during boot. But as there is none, and since it is such a small amount
> of memory, it is easiest to reserve it unconditionally.
> 
> Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the
> first 4KiB of physical memory.
> 
> Signed-off-by: Sam Edwards <CFSworks@gmail.com>
> ---

Applied to https://github.com/Broadcom/stblinux/commits/devicetree-arm64/next, thanks!
--
Florian

[PATCH v2 2/2] arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr

[Sam Edwards]

The `cpu-release-addr` property is relevant only when the "spin-table"
enable method is used. It is the physical address where the bootloader
expects Linux to write the secondary CPU entry point's physical address.
On this platform, only the CFE bootloader uses this method: U-Boot uses
PSCI instead.

CFE actually walks the FDT to learn this address, so we're free to put
it wherever we want. We only need to make sure that it goes in a
reserved-memory block so that writing to it during early boot does not
risk conflicting with an unrelated memory allocation: this was not done.

Since the previous patch reserved the first page of memory for CFE's
secondary-CPU init stub, which is actually much smaller than a page,
just put this address at the end of that page and it shall be so
protected.

Signed-off-by: Sam Edwards <CFSworks@gmail.com>
---
 arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
index c51b92387fad..613ba7ee43d6 100644
--- a/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
+++ b/arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi
@@ -30,7 +30,7 @@ cpu0: cpu@0 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x0>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
@@ -39,7 +39,7 @@ cpu1: cpu@1 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x1>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
@@ -48,7 +48,7 @@ cpu2: cpu@2 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x2>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
@@ -57,7 +57,7 @@ cpu3: cpu@3 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x3>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
-- 
2.44.2

Re: [PATCH v2 2/2] arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr

[Florian Fainelli]

From: Florian Fainelli <f.fainelli@gmail.com>

On Fri,  4 Oct 2024 22:01:55 -0700, Sam Edwards <cfsworks@gmail.com> wrote:
> The `cpu-release-addr` property is relevant only when the "spin-table"
> enable method is used. It is the physical address where the bootloader
> expects Linux to write the secondary CPU entry point's physical address.
> On this platform, only the CFE bootloader uses this method: U-Boot uses
> PSCI instead.
> 
> CFE actually walks the FDT to learn this address, so we're free to put
> it wherever we want. We only need to make sure that it goes in a
> reserved-memory block so that writing to it during early boot does not
> risk conflicting with an unrelated memory allocation: this was not done.
> 
> Since the previous patch reserved the first page of memory for CFE's
> secondary-CPU init stub, which is actually much smaller than a page,
> just put this address at the end of that page and it shall be so
> protected.
> 
> Signed-off-by: Sam Edwards <CFSworks@gmail.com>
> ---

Applied to https://github.com/Broadcom/stblinux/commits/devicetree-arm64/next, thanks!
--
Florian