[PATCH 0/5] Microwatt updates

[PATCH 0/5] Microwatt updates

[Paul Mackerras]

This patch series updates the kernel support for the Microwatt
soft-core and its implementation on FPGA systems, particularly the
Digilent Arty A7-100 FPGA development board.

Microwatt now supports almost all of the features of the SFFS (Scalar
Fixed-poin and Floating-point Subset) compliancy subset of Power ISA
version 3.1C, including prefixed instructions and the fixed-point hash
(ROP mitigation) instructions.  It is also now SMP-capable, and a
dual-core system will fit on the Arty A7-100 board.

Microwatt does not have broadcast TLB invalidations in SMP systems;
the kernel already has code to deal with this.  One of the patches in
this series provides a config option to allow platforms to select
unconditionally the behaviour where cross-CPU TLB invalidations are
handled using inter-processor interrupts.

Paul.
---
 arch/powerpc/boot/dts/microwatt.dts          | 107 +++++++++++++++++++++++----
 arch/powerpc/mm/book3s64/pgtable.c           |  10 ++-
 arch/powerpc/platforms/Kconfig.cputype       |  12 +++
 arch/powerpc/platforms/microwatt/Kconfig     |   4 +-
 arch/powerpc/platforms/microwatt/Makefile    |   1 +
 arch/powerpc/platforms/microwatt/microwatt.h |   1 +
 arch/powerpc/platforms/microwatt/setup.c     |  19 +++++
 7 files changed, 138 insertions(+), 16 deletions(-)

[PATCH 1/5] powerpc/microwatt: Select COMMON_CLK in order to get the clock framework

[Paul Mackerras]

This is to allow us to select Litex MMC host controller driver, which
drives the litesdcard gateware.

Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
---
 arch/powerpc/platforms/microwatt/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/powerpc/platforms/microwatt/Kconfig b/arch/powerpc/platforms/microwatt/Kconfig
index 6af443a1db99..5e41adadac1f 100644
--- a/arch/powerpc/platforms/microwatt/Kconfig
+++ b/arch/powerpc/platforms/microwatt/Kconfig
@@ -6,6 +6,7 @@ config PPC_MICROWATT
 	select PPC_ICS_NATIVE
 	select PPC_ICP_NATIVE
 	select PPC_UDBG_16550
+	select COMMON_CLK
 	help
           This option enables support for FPGA-based Microwatt implementations.
 
-- 
2.47.1

[PATCH 2/5] powerpc/microwatt: Device-tree updates

[Paul Mackerras]

Microwatt now implements ISA v3.1 (SFFS compliancy subset), including
prefixed instructions, scv/rfscv, and the FSCR, HFSCR, TAR, and CTRL
registers.  The privileged mode of operation is now hypervisor mode
and there is no privileged non-hypervisor mode; the MSR[HV] bit is
forced to 1.

Besides updating the ibm,powerpc-cpu-features property to reflect the
above, this also makes the following changes relating to peripheral
devices:

- Add gpio controller.
- Remove high-speed property from SD controller, for the case where
  the interface is connected through 200 ohm protection resisters.
- Put an alias for the ethernet in /chosen.

Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
---
 arch/powerpc/boot/dts/microwatt.dts | 73 ++++++++++++++++++++++++-----
 1 file changed, 62 insertions(+), 11 deletions(-)

diff --git a/arch/powerpc/boot/dts/microwatt.dts b/arch/powerpc/boot/dts/microwatt.dts
index 269e930b3b0b..6e575e841a7b 100644
--- a/arch/powerpc/boot/dts/microwatt.dts
+++ b/arch/powerpc/boot/dts/microwatt.dts
@@ -1,4 +1,5 @@
 /dts-v1/;
+#include <dt-bindings/gpio/gpio.h>
 
 / {
 	#size-cells = <0x02>;
@@ -8,6 +9,7 @@ / {
 
 	aliases {
 		serial0 = &UART0;
+		ethernet = &enet0;
 	};
 
 	reserved-memory {
@@ -35,40 +37,79 @@ cpus {
 
 		ibm,powerpc-cpu-features {
 			display-name = "Microwatt";
-			isa = <3000>;
+			isa = <3010>;
 			device_type = "cpu-features";
 			compatible = "ibm,powerpc-cpu-features";
 
 			mmu-radix {
 				isa = <3000>;
-				usable-privilege = <2>;
+				usable-privilege = <6>;
+				os-support = <0>;
 			};
 
 			little-endian {
-				isa = <2050>;
-				usable-privilege = <3>;
+				isa = <0>;
+				usable-privilege = <7>;
+				os-support = <0>;
 				hwcap-bit-nr = <1>;
 			};
 
 			cache-inhibited-large-page {
-				isa = <2040>;
-				usable-privilege = <2>;
+				isa = <0>;
+				usable-privilege = <6>;
+				os-support = <0>;
 			};
 
 			fixed-point-v3 {
 				isa = <3000>;
-				usable-privilege = <3>;
+				usable-privilege = <7>;
 			};
 
 			no-execute {
-				isa = <2010>;
+				isa = <0x00>;
 				usable-privilege = <2>;
+				os-support = <0>;
 			};
 
 			floating-point {
+				hfscr-bit-nr = <0>;
 				hwcap-bit-nr = <27>;
 				isa = <0>;
-				usable-privilege = <3>;
+				usable-privilege = <7>;
+				hv-support = <1>;
+				os-support = <0>;
+			};
+
+			prefixed-instructions {
+				hfscr-bit-nr = <13>;
+				fscr-bit-nr = <13>;
+				isa = <3010>;
+				usable-privilege = <7>;
+				os-support = <1>;
+				hv-support = <1>;
+			};
+
+			tar {
+				hfscr-bit-nr = <8>;
+				fscr-bit-nr = <8>;
+				isa = <2070>;
+				usable-privilege = <7>;
+				os-support = <1>;
+				hv-support = <1>;
+				hwcap-bit-nr = <58>;
+			};
+
+			control-register {
+				isa = <0>;
+				usable-privilege = <7>;
+			};
+
+			system-call-vectored {
+				isa = <2070>;
+				usable-privilege = <7>;
+				os-support = <1>;
+				fscr-bit-nr = <12>;
+				hwcap-bit-nr = <52>;
 			};
 		};
 
@@ -138,7 +179,18 @@ UART0: serial@2000 {
 			interrupts = <0x10 0x1>;
 		};
 
-		ethernet@8020000 {
+		gpio: gpio@7000 {
+			device_type = "gpio";
+			compatible = "faraday,ftgpio010";
+			gpio-controller;
+			#gpio-cells = <2>;
+			reg = <0x7000 0x80>;
+			interrupts = <0x14 1>;
+			interrupt-controller;
+			#interrupt-cells = <2>;
+		};
+
+		enet0: ethernet@8020000 {
 			compatible = "litex,liteeth";
 			reg = <0x8021000 0x100
 				0x8020800 0x100
@@ -160,7 +212,6 @@ mmc@8040000 {
 			reg-names = "phy", "core", "reader", "writer", "irq";
 			bus-width = <4>;
 			interrupts = <0x13 1>;
-			cap-sd-highspeed;
 			clocks = <&sys_clk>;
 		};
 	};
-- 
2.47.1

[PATCH 3/5] powerpc/microwatt: Define an idle power-save function

[Paul Mackerras]

This uses the 'wait' instruction to pause instruction execution when
idle until an interrupt occurs.

Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
---
 arch/powerpc/platforms/microwatt/setup.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/powerpc/platforms/microwatt/setup.c b/arch/powerpc/platforms/microwatt/setup.c
index 5e1c0997170d..97828a99780d 100644
--- a/arch/powerpc/platforms/microwatt/setup.c
+++ b/arch/powerpc/platforms/microwatt/setup.c
@@ -34,10 +34,19 @@ static void __init microwatt_setup_arch(void)
 	microwatt_rng_init();
 }
 
+static void microwatt_idle(void)
+{
+	if (!prep_irq_for_idle())
+		return;
+
+	__asm__ __volatile__ ("wait");
+}
+
 define_machine(microwatt) {
 	.name			= "microwatt",
 	.compatible		= "microwatt-soc",
 	.init_IRQ		= microwatt_init_IRQ,
 	.setup_arch		= microwatt_setup_arch,
 	.progress		= udbg_progress,
+	.power_save		= microwatt_idle,
 };
-- 
2.47.1

[PATCH 4/5] powerpc: Define config option for processors without broadcast TLBIE

[Paul Mackerras]

Power ISA v3.1 implementations in the Linux Compliancy Subset and
lower are not required to implement broadcast TLBIE, and in fact
Microwatt doesn't.  To avoid the need to specify "disable_tlbie" on
the kernel command line on SMP Microwatt systems, this defines a
config option that asserts that broadcast TLBIE should never be used
(the kernel will instead use IPIs to trigger local TLBIEs on other
CPUs when required).

Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
---
 arch/powerpc/mm/book3s64/pgtable.c       | 10 ++++++++--
 arch/powerpc/platforms/Kconfig.cputype   | 12 ++++++++++++
 arch/powerpc/platforms/microwatt/Kconfig |  1 +
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index 374542528080..14ee96e2a581 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -588,10 +588,16 @@ int pmd_move_must_withdraw(struct spinlock *new_pmd_ptl,
 }
 #endif
 
+#ifndef CONFIG_PPC_RADIX_NO_BROADCAST_TLBIE
+#define DEFAULT_TLBIE_ENABLE	true
+#else
+#define DEFAULT_TLBIE_ENABLE	false
+#endif
+
 /*
  * Does the CPU support tlbie?
  */
-bool tlbie_capable __read_mostly = true;
+bool tlbie_capable __read_mostly = DEFAULT_TLBIE_ENABLE;
 EXPORT_SYMBOL(tlbie_capable);
 
 /*
@@ -599,7 +605,7 @@ EXPORT_SYMBOL(tlbie_capable);
  * address spaces? tlbie may still be used for nMMU accelerators, and for KVM
  * guest address spaces.
  */
-bool tlbie_enabled __read_mostly = true;
+bool tlbie_enabled __read_mostly = DEFAULT_TLBIE_ENABLE;
 
 static int __init setup_disable_tlbie(char *str)
 {
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index 1453ccc900c4..bd2a4e46ab34 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -449,6 +449,18 @@ config PPC_RADIX_MMU_DEFAULT
 
 	  If you're unsure, say Y.
 
+config PPC_RADIX_NO_BROADCAST_TLBIE
+	depends on PPC_RADIX_MMU
+	help
+	  Power ISA v3.1 implementations in the Linux Compliancy Subset
+	  and lower are not required to implement broadcast TLBIE
+	  instructions, that is, a TLB invalidation instruction
+	  performed on one CPU is not required to operate on the TLBs
+	  in all CPUs in the system.  Instead, the kernel does an IPI
+	  to each relevant CPU to get it to do a local TLBIE instruction.
+	  Select this option to force global invalidations to be done via
+	  IPIs unconditionally.
+
 config PPC_KERNEL_PREFIXED
 	depends on PPC_HAVE_PREFIXED_SUPPORT
 	depends on CC_HAS_PREFIXED
diff --git a/arch/powerpc/platforms/microwatt/Kconfig b/arch/powerpc/platforms/microwatt/Kconfig
index 5e41adadac1f..1d5cc1ae3636 100644
--- a/arch/powerpc/platforms/microwatt/Kconfig
+++ b/arch/powerpc/platforms/microwatt/Kconfig
@@ -7,6 +7,7 @@ config PPC_MICROWATT
 	select PPC_ICP_NATIVE
 	select PPC_UDBG_16550
 	select COMMON_CLK
+	select PPC_RADIX_NO_BROADCAST_TLBIE
 	help
           This option enables support for FPGA-based Microwatt implementations.
 
-- 
2.47.1

[PATCH 5/5] powerpc/microwatt: Add SMP support

[Paul Mackerras]

This adds support for Microwatt systems with more than one core, and
updates the device tree for a 2-core version.  (This does not prevent
the kernel from running on a single-core system.)

Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
---
 arch/powerpc/boot/dts/microwatt.dts          | 34 ++++++++++++++++++--
 arch/powerpc/platforms/microwatt/Kconfig     |  2 +-
 arch/powerpc/platforms/microwatt/Makefile    |  1 +
 arch/powerpc/platforms/microwatt/microwatt.h |  1 +
 arch/powerpc/platforms/microwatt/setup.c     | 10 ++++++
 5 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/boot/dts/microwatt.dts b/arch/powerpc/boot/dts/microwatt.dts
index 6e575e841a7b..89281dc975b3 100644
--- a/arch/powerpc/boot/dts/microwatt.dts
+++ b/arch/powerpc/boot/dts/microwatt.dts
@@ -142,6 +142,36 @@ PowerPC,Microwatt@0 {
 			ibm,mmu-lpid-bits = <12>;
 			ibm,mmu-pid-bits = <20>;
 		};
+
+		PowerPC,Microwatt@1 {
+			i-cache-sets = <2>;
+			ibm,dec-bits = <64>;
+			reservation-granule-size = <64>;
+			clock-frequency = <100000000>;
+			timebase-frequency = <100000000>;
+			i-tlb-sets = <1>;
+			ibm,ppc-interrupt-server#s = <1>;
+			i-cache-block-size = <64>;
+			d-cache-block-size = <64>;
+			d-cache-sets = <2>;
+			i-tlb-size = <64>;
+			cpu-version = <0x990000>;
+			status = "okay";
+			i-cache-size = <0x1000>;
+			ibm,processor-radix-AP-encodings = <0x0c 0xa0000010 0x20000015 0x4000001e>;
+			tlb-size = <0>;
+			tlb-sets = <0>;
+			device_type = "cpu";
+			d-tlb-size = <128>;
+			d-tlb-sets = <2>;
+			reg = <1>;
+			general-purpose;
+			64-bit;
+			d-cache-size = <0x1000>;
+			ibm,chip-id = <0>;
+			ibm,mmu-lpid-bits = <12>;
+			ibm,mmu-pid-bits = <20>;
+		};
 	};
 
 	soc@c0000000 {
@@ -154,8 +184,8 @@ soc@c0000000 {
 
 		interrupt-controller@4000 {
 			compatible = "openpower,xics-presentation", "ibm,ppc-xicp";
-			ibm,interrupt-server-ranges = <0x0 0x1>;
-			reg = <0x4000 0x100>;
+			ibm,interrupt-server-ranges = <0x0 0x2>;
+			reg = <0x4000 0x10 0x4010 0x10>;
 		};
 
 		ICS: interrupt-controller@5000 {
diff --git a/arch/powerpc/platforms/microwatt/Kconfig b/arch/powerpc/platforms/microwatt/Kconfig
index 1d5cc1ae3636..8911ea7cf12e 100644
--- a/arch/powerpc/platforms/microwatt/Kconfig
+++ b/arch/powerpc/platforms/microwatt/Kconfig
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 config PPC_MICROWATT
-	depends on PPC_BOOK3S_64 && !SMP
+	depends on PPC_BOOK3S_64
 	bool "Microwatt SoC platform"
 	select PPC_XICS
 	select PPC_ICS_NATIVE
diff --git a/arch/powerpc/platforms/microwatt/Makefile b/arch/powerpc/platforms/microwatt/Makefile
index 116d6d3ad3f0..d973b2ab4042 100644
--- a/arch/powerpc/platforms/microwatt/Makefile
+++ b/arch/powerpc/platforms/microwatt/Makefile
@@ -1 +1,2 @@
 obj-y	+= setup.o rng.o
+obj-$(CONFIG_SMP) += smp.o
diff --git a/arch/powerpc/platforms/microwatt/microwatt.h b/arch/powerpc/platforms/microwatt/microwatt.h
index 335417e95e66..891aa2800768 100644
--- a/arch/powerpc/platforms/microwatt/microwatt.h
+++ b/arch/powerpc/platforms/microwatt/microwatt.h
@@ -3,5 +3,6 @@
 #define _MICROWATT_H
 
 void microwatt_rng_init(void);
+void microwatt_init_smp(void);
 
 #endif /* _MICROWATT_H */
diff --git a/arch/powerpc/platforms/microwatt/setup.c b/arch/powerpc/platforms/microwatt/setup.c
index 97828a99780d..d966bf1f57f7 100644
--- a/arch/powerpc/platforms/microwatt/setup.c
+++ b/arch/powerpc/platforms/microwatt/setup.c
@@ -29,6 +29,15 @@ static int __init microwatt_populate(void)
 }
 machine_arch_initcall(microwatt, microwatt_populate);
 
+static int __init microwatt_probe(void)
+{
+	/* Main reason for having this is to start the other CPU(s) */
+#ifdef CONFIG_SMP
+	microwatt_init_smp();
+#endif
+	return 1;
+}
+
 static void __init microwatt_setup_arch(void)
 {
 	microwatt_rng_init();
@@ -45,6 +54,7 @@ static void microwatt_idle(void)
 define_machine(microwatt) {
 	.name			= "microwatt",
 	.compatible		= "microwatt-soc",
+	.probe			= microwatt_probe,
 	.init_IRQ		= microwatt_init_IRQ,
 	.setup_arch		= microwatt_setup_arch,
 	.progress		= udbg_progress,
-- 
2.47.1

Re: [PATCH 1/5] powerpc/microwatt: Select COMMON_CLK in order to get the clock framework

[Nicholas Piggin]

On Wed Jan 29, 2025 at 8:51 AM AEST, Paul Mackerras wrote:
> This is to allow us to select Litex MMC host controller driver, which
> drives the litesdcard gateware.
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>

Acked-by: Nicholas Piggin <npiggin@gmail.com>

> ---
>  arch/powerpc/platforms/microwatt/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/arch/powerpc/platforms/microwatt/Kconfig b/arch/powerpc/platforms/microwatt/Kconfig
> index 6af443a1db99..5e41adadac1f 100644
> --- a/arch/powerpc/platforms/microwatt/Kconfig
> +++ b/arch/powerpc/platforms/microwatt/Kconfig
> @@ -6,6 +6,7 @@ config PPC_MICROWATT
>  	select PPC_ICS_NATIVE
>  	select PPC_ICP_NATIVE
>  	select PPC_UDBG_16550
> +	select COMMON_CLK
>  	help
>            This option enables support for FPGA-based Microwatt implementations.
>  

Re: [PATCH 3/5] powerpc/microwatt: Define an idle power-save function

[Nicholas Piggin]

On Wed Jan 29, 2025 at 8:52 AM AEST, Paul Mackerras wrote:
> This uses the 'wait' instruction to pause instruction execution when
> idle until an interrupt occurs.
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
> ---
>  arch/powerpc/platforms/microwatt/setup.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/arch/powerpc/platforms/microwatt/setup.c b/arch/powerpc/platforms/microwatt/setup.c
> index 5e1c0997170d..97828a99780d 100644
> --- a/arch/powerpc/platforms/microwatt/setup.c
> +++ b/arch/powerpc/platforms/microwatt/setup.c
> @@ -34,10 +34,19 @@ static void __init microwatt_setup_arch(void)
>  	microwatt_rng_init();
>  }
>  
> +static void microwatt_idle(void)
> +{
> +	if (!prep_irq_for_idle())
> +		return;
> +
> +	__asm__ __volatile__ ("wait");
> +}

Does wait cause MSR[EE] to be set? If not, do you need to use
prep_irq_for_idle_irqsoff() here maybe?

Thanks,
Nick

> +
>  define_machine(microwatt) {
>  	.name			= "microwatt",
>  	.compatible		= "microwatt-soc",
>  	.init_IRQ		= microwatt_init_IRQ,
>  	.setup_arch		= microwatt_setup_arch,
>  	.progress		= udbg_progress,
> +	.power_save		= microwatt_idle,
>  };

Re: [PATCH 4/5] powerpc: Define config option for processors without broadcast TLBIE

[Nicholas Piggin]

On Wed Jan 29, 2025 at 8:53 AM AEST, Paul Mackerras wrote:
> Power ISA v3.1 implementations in the Linux Compliancy Subset and
> lower are not required to implement broadcast TLBIE, and in fact
> Microwatt doesn't.  To avoid the need to specify "disable_tlbie" on
> the kernel command line on SMP Microwatt systems, this defines a
> config option that asserts that broadcast TLBIE should never be used
> (the kernel will instead use IPIs to trigger local TLBIEs on other
> CPUs when required).
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
> ---
>  arch/powerpc/mm/book3s64/pgtable.c       | 10 ++++++++--
>  arch/powerpc/platforms/Kconfig.cputype   | 12 ++++++++++++
>  arch/powerpc/platforms/microwatt/Kconfig |  1 +
>  3 files changed, 21 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
> index 374542528080..14ee96e2a581 100644
> --- a/arch/powerpc/mm/book3s64/pgtable.c
> +++ b/arch/powerpc/mm/book3s64/pgtable.c
> @@ -588,10 +588,16 @@ int pmd_move_must_withdraw(struct spinlock *new_pmd_ptl,
>  }
>  #endif
>  
> +#ifndef CONFIG_PPC_RADIX_NO_BROADCAST_TLBIE

Hate to bikeshed, but would it be annoying to make this an affirmative
option?

> +#define DEFAULT_TLBIE_ENABLE	true
> +#else
> +#define DEFAULT_TLBIE_ENABLE	false
> +#endif
> +
>  /*
>   * Does the CPU support tlbie?
>   */
> -bool tlbie_capable __read_mostly = true;
> +bool tlbie_capable __read_mostly = DEFAULT_TLBIE_ENABLE;
>  EXPORT_SYMBOL(tlbie_capable);
>  
>  /*
> @@ -599,7 +605,7 @@ EXPORT_SYMBOL(tlbie_capable);
>   * address spaces? tlbie may still be used for nMMU accelerators, and for KVM
>   * guest address spaces.
>   */
> -bool tlbie_enabled __read_mostly = true;
> +bool tlbie_enabled __read_mostly = DEFAULT_TLBIE_ENABLE;
>  
>  static int __init setup_disable_tlbie(char *str)
>  {
> diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
> index 1453ccc900c4..bd2a4e46ab34 100644
> --- a/arch/powerpc/platforms/Kconfig.cputype
> +++ b/arch/powerpc/platforms/Kconfig.cputype
> @@ -449,6 +449,18 @@ config PPC_RADIX_MMU_DEFAULT
>  
>  	  If you're unsure, say Y.
>  
> +config PPC_RADIX_NO_BROADCAST_TLBIE
> +	depends on PPC_RADIX_MMU
> +	help
> +	  Power ISA v3.1 implementations in the Linux Compliancy Subset
> +	  and lower are not required to implement broadcast TLBIE
> +	  instructions, that is, a TLB invalidation instruction
> +	  performed on one CPU is not required to operate on the TLBs
> +	  in all CPUs in the system.  Instead, the kernel does an IPI
> +	  to each relevant CPU to get it to do a local TLBIE instruction.
> +	  Select this option to force global invalidations to be done via
> +	  IPIs unconditionally.

... when using radix MMU. Hash MMU requires broadcast TLBIE.

Nitpicks aside, looks okay to me.

Reviewed-by: Nicholas Piggin <npiggin@gmail.com>

> +
>  config PPC_KERNEL_PREFIXED
>  	depends on PPC_HAVE_PREFIXED_SUPPORT
>  	depends on CC_HAS_PREFIXED
> diff --git a/arch/powerpc/platforms/microwatt/Kconfig b/arch/powerpc/platforms/microwatt/Kconfig
> index 5e41adadac1f..1d5cc1ae3636 100644
> --- a/arch/powerpc/platforms/microwatt/Kconfig
> +++ b/arch/powerpc/platforms/microwatt/Kconfig
> @@ -7,6 +7,7 @@ config PPC_MICROWATT
>  	select PPC_ICP_NATIVE
>  	select PPC_UDBG_16550
>  	select COMMON_CLK
> +	select PPC_RADIX_NO_BROADCAST_TLBIE
>  	help
>            This option enables support for FPGA-based Microwatt implementations.
>  

Re: [PATCH 5/5] powerpc/microwatt: Add SMP support

[Nicholas Piggin]

On Wed Jan 29, 2025 at 8:55 AM AEST, Paul Mackerras wrote:
> This adds support for Microwatt systems with more than one core, and
> updates the device tree for a 2-core version.  (This does not prevent
> the kernel from running on a single-core system.)
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>

Well, I'm impressed you added SMP :)

What happens with a 1 CPU system? Do we time out waiting for secondaries
and continue, or is there something more graceful?

> ---
>  arch/powerpc/boot/dts/microwatt.dts          | 34 ++++++++++++++++++--
>  arch/powerpc/platforms/microwatt/Kconfig     |  2 +-
>  arch/powerpc/platforms/microwatt/Makefile    |  1 +
>  arch/powerpc/platforms/microwatt/microwatt.h |  1 +
>  arch/powerpc/platforms/microwatt/setup.c     | 10 ++++++
>  5 files changed, 45 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/boot/dts/microwatt.dts b/arch/powerpc/boot/dts/microwatt.dts
> index 6e575e841a7b..89281dc975b3 100644
> --- a/arch/powerpc/boot/dts/microwatt.dts
> +++ b/arch/powerpc/boot/dts/microwatt.dts
> @@ -142,6 +142,36 @@ PowerPC,Microwatt@0 {
>  			ibm,mmu-lpid-bits = <12>;
>  			ibm,mmu-pid-bits = <20>;
>  		};
> +
> +		PowerPC,Microwatt@1 {
> +			i-cache-sets = <2>;
> +			ibm,dec-bits = <64>;
> +			reservation-granule-size = <64>;
> +			clock-frequency = <100000000>;
> +			timebase-frequency = <100000000>;
> +			i-tlb-sets = <1>;
> +			ibm,ppc-interrupt-server#s = <1>;
> +			i-cache-block-size = <64>;
> +			d-cache-block-size = <64>;
> +			d-cache-sets = <2>;
> +			i-tlb-size = <64>;
> +			cpu-version = <0x990000>;
> +			status = "okay";
> +			i-cache-size = <0x1000>;
> +			ibm,processor-radix-AP-encodings = <0x0c 0xa0000010 0x20000015 0x4000001e>;
> +			tlb-size = <0>;
> +			tlb-sets = <0>;
> +			device_type = "cpu";
> +			d-tlb-size = <128>;
> +			d-tlb-sets = <2>;
> +			reg = <1>;
> +			general-purpose;
> +			64-bit;
> +			d-cache-size = <0x1000>;
> +			ibm,chip-id = <0>;
> +			ibm,mmu-lpid-bits = <12>;
> +			ibm,mmu-pid-bits = <20>;
> +		};
>  	};
>  
>  	soc@c0000000 {
> @@ -154,8 +184,8 @@ soc@c0000000 {
>  
>  		interrupt-controller@4000 {
>  			compatible = "openpower,xics-presentation", "ibm,ppc-xicp";
> -			ibm,interrupt-server-ranges = <0x0 0x1>;
> -			reg = <0x4000 0x100>;
> +			ibm,interrupt-server-ranges = <0x0 0x2>;
> +			reg = <0x4000 0x10 0x4010 0x10>;
>  		};
>  
>  		ICS: interrupt-controller@5000 {
> diff --git a/arch/powerpc/platforms/microwatt/Kconfig b/arch/powerpc/platforms/microwatt/Kconfig
> index 1d5cc1ae3636..8911ea7cf12e 100644
> --- a/arch/powerpc/platforms/microwatt/Kconfig
> +++ b/arch/powerpc/platforms/microwatt/Kconfig
> @@ -1,6 +1,6 @@
>  # SPDX-License-Identifier: GPL-2.0
>  config PPC_MICROWATT
> -	depends on PPC_BOOK3S_64 && !SMP
> +	depends on PPC_BOOK3S_64
>  	bool "Microwatt SoC platform"
>  	select PPC_XICS
>  	select PPC_ICS_NATIVE
> diff --git a/arch/powerpc/platforms/microwatt/Makefile b/arch/powerpc/platforms/microwatt/Makefile
> index 116d6d3ad3f0..d973b2ab4042 100644
> --- a/arch/powerpc/platforms/microwatt/Makefile
> +++ b/arch/powerpc/platforms/microwatt/Makefile
> @@ -1 +1,2 @@
>  obj-y	+= setup.o rng.o
> +obj-$(CONFIG_SMP) += smp.o

Hmm.... `git add arch/powerpc/platforms/microwatt/smp.c` ?

> diff --git a/arch/powerpc/platforms/microwatt/microwatt.h b/arch/powerpc/platforms/microwatt/microwatt.h
> index 335417e95e66..891aa2800768 100644
> --- a/arch/powerpc/platforms/microwatt/microwatt.h
> +++ b/arch/powerpc/platforms/microwatt/microwatt.h
> @@ -3,5 +3,6 @@
>  #define _MICROWATT_H
>  
>  void microwatt_rng_init(void);
> +void microwatt_init_smp(void);
>  
>  #endif /* _MICROWATT_H */
> diff --git a/arch/powerpc/platforms/microwatt/setup.c b/arch/powerpc/platforms/microwatt/setup.c
> index 97828a99780d..d966bf1f57f7 100644
> --- a/arch/powerpc/platforms/microwatt/setup.c
> +++ b/arch/powerpc/platforms/microwatt/setup.c
> @@ -29,6 +29,15 @@ static int __init microwatt_populate(void)
>  }
>  machine_arch_initcall(microwatt, microwatt_populate);
>  
> +static int __init microwatt_probe(void)
> +{
> +	/* Main reason for having this is to start the other CPU(s) */
> +#ifdef CONFIG_SMP
> +	microwatt_init_smp();
> +#endif

Could use

	if (IS_ENABLED(CONFIG_SMP))
		microwatt_init_smp();

Thanks,
Nick

> +	return 1;
> +}
> +
>  static void __init microwatt_setup_arch(void)
>  {
>  	microwatt_rng_init();
> @@ -45,6 +54,7 @@ static void microwatt_idle(void)
>  define_machine(microwatt) {
>  	.name			= "microwatt",
>  	.compatible		= "microwatt-soc",
> +	.probe			= microwatt_probe,
>  	.init_IRQ		= microwatt_init_IRQ,
>  	.setup_arch		= microwatt_setup_arch,
>  	.progress		= udbg_progress,

Re: [PATCH 2/5] powerpc/microwatt: Device-tree updates

[Nicholas Piggin]

On Wed Jan 29, 2025 at 8:52 AM AEST, Paul Mackerras wrote:
> Microwatt now implements ISA v3.1 (SFFS compliancy subset), including
> prefixed instructions, scv/rfscv, and the FSCR, HFSCR, TAR, and CTRL
> registers.  The privileged mode of operation is now hypervisor mode
> and there is no privileged non-hypervisor mode; the MSR[HV] bit is
> forced to 1.

Cool. Lots of development in microwatt.

Come to think of it we should have put a broadcast-tlbie feature
in there and you wouldn't need the other patch. That can go on
the todo list I guess.

system-call-vectored was available in ISA v3.0. Not that we do much
with it at the moment IIRC, but there were dreams of wiring it in for
compat guests. With that fixed,

Acked-by: Nicholas Piggin <npiggin@gmail.com>

Thanks,
Nick

>
> Besides updating the ibm,powerpc-cpu-features property to reflect the
> above, this also makes the following changes relating to peripheral
> devices:
>
> - Add gpio controller.
> - Remove high-speed property from SD controller, for the case where
>   the interface is connected through 200 ohm protection resisters.
> - Put an alias for the ethernet in /chosen.
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
> ---
>  arch/powerpc/boot/dts/microwatt.dts | 73 ++++++++++++++++++++++++-----
>  1 file changed, 62 insertions(+), 11 deletions(-)
>
> diff --git a/arch/powerpc/boot/dts/microwatt.dts b/arch/powerpc/boot/dts/microwatt.dts
> index 269e930b3b0b..6e575e841a7b 100644
> --- a/arch/powerpc/boot/dts/microwatt.dts
> +++ b/arch/powerpc/boot/dts/microwatt.dts
> @@ -1,4 +1,5 @@
>  /dts-v1/;
> +#include <dt-bindings/gpio/gpio.h>
>  
>  / {
>  	#size-cells = <0x02>;
> @@ -8,6 +9,7 @@ / {
>  
>  	aliases {
>  		serial0 = &UART0;
> +		ethernet = &enet0;
>  	};
>  
>  	reserved-memory {
> @@ -35,40 +37,79 @@ cpus {
>  
>  		ibm,powerpc-cpu-features {
>  			display-name = "Microwatt";
> -			isa = <3000>;
> +			isa = <3010>;
>  			device_type = "cpu-features";
>  			compatible = "ibm,powerpc-cpu-features";
>  
>  			mmu-radix {
>  				isa = <3000>;
> -				usable-privilege = <2>;
> +				usable-privilege = <6>;
> +				os-support = <0>;
>  			};
>  
>  			little-endian {
> -				isa = <2050>;
> -				usable-privilege = <3>;
> +				isa = <0>;
> +				usable-privilege = <7>;
> +				os-support = <0>;
>  				hwcap-bit-nr = <1>;
>  			};
>  
>  			cache-inhibited-large-page {
> -				isa = <2040>;
> -				usable-privilege = <2>;
> +				isa = <0>;
> +				usable-privilege = <6>;
> +				os-support = <0>;
>  			};
>  
>  			fixed-point-v3 {
>  				isa = <3000>;
> -				usable-privilege = <3>;
> +				usable-privilege = <7>;
>  			};
>  
>  			no-execute {
> -				isa = <2010>;
> +				isa = <0x00>;
>  				usable-privilege = <2>;
> +				os-support = <0>;
>  			};
>  
>  			floating-point {
> +				hfscr-bit-nr = <0>;
>  				hwcap-bit-nr = <27>;
>  				isa = <0>;
> -				usable-privilege = <3>;
> +				usable-privilege = <7>;
> +				hv-support = <1>;
> +				os-support = <0>;
> +			};
> +
> +			prefixed-instructions {
> +				hfscr-bit-nr = <13>;
> +				fscr-bit-nr = <13>;
> +				isa = <3010>;
> +				usable-privilege = <7>;
> +				os-support = <1>;
> +				hv-support = <1>;
> +			};
> +
> +			tar {
> +				hfscr-bit-nr = <8>;
> +				fscr-bit-nr = <8>;
> +				isa = <2070>;
> +				usable-privilege = <7>;
> +				os-support = <1>;
> +				hv-support = <1>;
> +				hwcap-bit-nr = <58>;
> +			};
> +
> +			control-register {
> +				isa = <0>;
> +				usable-privilege = <7>;
> +			};
> +
> +			system-call-vectored {
> +				isa = <2070>;
> +				usable-privilege = <7>;
> +				os-support = <1>;
> +				fscr-bit-nr = <12>;
> +				hwcap-bit-nr = <52>;
>  			};
>  		};
>  
> @@ -138,7 +179,18 @@ UART0: serial@2000 {
>  			interrupts = <0x10 0x1>;
>  		};
>  
> -		ethernet@8020000 {
> +		gpio: gpio@7000 {
> +			device_type = "gpio";
> +			compatible = "faraday,ftgpio010";
> +			gpio-controller;
> +			#gpio-cells = <2>;
> +			reg = <0x7000 0x80>;
> +			interrupts = <0x14 1>;
> +			interrupt-controller;
> +			#interrupt-cells = <2>;
> +		};
> +
> +		enet0: ethernet@8020000 {
>  			compatible = "litex,liteeth";
>  			reg = <0x8021000 0x100
>  				0x8020800 0x100
> @@ -160,7 +212,6 @@ mmc@8040000 {
>  			reg-names = "phy", "core", "reader", "writer", "irq";
>  			bus-width = <4>;
>  			interrupts = <0x13 1>;
> -			cap-sd-highspeed;
>  			clocks = <&sys_clk>;
>  		};
>  	};

Re: [PATCH 3/5] powerpc/microwatt: Define an idle power-save function

[Paul Mackerras]

On Wed, Jan 29, 2025 at 04:06:03PM +1000, Nicholas Piggin wrote:
> On Wed Jan 29, 2025 at 8:52 AM AEST, Paul Mackerras wrote:
> > This uses the 'wait' instruction to pause instruction execution when
> > idle until an interrupt occurs.
> >
> > Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
> > ---
> >  arch/powerpc/platforms/microwatt/setup.c | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/arch/powerpc/platforms/microwatt/setup.c b/arch/powerpc/platforms/microwatt/setup.c
> > index 5e1c0997170d..97828a99780d 100644
> > --- a/arch/powerpc/platforms/microwatt/setup.c
> > +++ b/arch/powerpc/platforms/microwatt/setup.c
> > @@ -34,10 +34,19 @@ static void __init microwatt_setup_arch(void)
> >  	microwatt_rng_init();
> >  }
> >  
> > +static void microwatt_idle(void)
> > +{
> > +	if (!prep_irq_for_idle())
> > +		return;
> > +
> > +	__asm__ __volatile__ ("wait");
> > +}
> 
> Does wait cause MSR[EE] to be set? If not, do you need to use
> prep_irq_for_idle_irqsoff() here maybe?

The wait instruction doesn't do anything to MSR[EE], but a pending
external interrupt (or any asynchronous exception condition) will
terminate the wait regardless of MSR[EE].

So yes, it does like I should be using prep_irq_for_idle_irqsoff().

Thanks,
Paul.

Re: [PATCH 5/5] powerpc/microwatt: Add SMP support

[Paul Mackerras]

On Wed, Jan 29, 2025 at 04:21:26PM +1000, Nicholas Piggin wrote:
> On Wed Jan 29, 2025 at 8:55 AM AEST, Paul Mackerras wrote:
> > This adds support for Microwatt systems with more than one core, and
> > updates the device tree for a 2-core version.  (This does not prevent
> > the kernel from running on a single-core system.)
> >
> > Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
> 
> Well, I'm impressed you added SMP :)
> 
> What happens with a 1 CPU system? Do we time out waiting for secondaries
> and continue, or is there something more graceful?

There's a field in the SYSCON register which tells you how many cores
there are.  microwatt_init_smp() looks at that field and only starts
the CPUs that are there.

Oops, sorry, I see that I forgot to do 'git add' on
arch/powerpc/platforms/microwatt/smp.c.  Here it is (I'll include it
properly in v2, of course):

// SPDX-License-Identifier: GPL-2.0-or-later

/*
 * SMP support functions for Microwatt
 * Copyright 2025 Paul Mackerras <paulus@ozlabs.org>
 */

#include <linux/kernel.h>
#include <linux/smp.h>
#include <linux/io.h>
#include <asm/early_ioremap.h>
#include <asm/xics.h>

#include "microwatt.h"

static void __init microwatt_smp_probe(void)
{
	xics_smp_probe();
}

static void microwatt_smp_setup_cpu(int cpu)
{
	if (cpu != 0)
		xics_setup_cpu();
}

static struct smp_ops_t microwatt_smp_ops = {
	.probe		= microwatt_smp_probe,
	.message_pass	= NULL,		/* Use smp_muxed_ipi_message_pass */
	.kick_cpu	= smp_generic_kick_cpu,
	.setup_cpu	= microwatt_smp_setup_cpu,
};

/* XXX get from device tree */
#define SYSCON_BASE	0xc0000000

#define SYSCON_CPU_CTRL	0x58

void __init microwatt_init_smp(void)
{
	volatile unsigned char __iomem *syscon;
	int ncpus;
	int timeout;

	syscon = early_ioremap(SYSCON_BASE, 0x100);
	if (syscon == NULL) {
		pr_err("Failed to map SYSCON\n");
		return;
	}
	ncpus = (readl(syscon + SYSCON_CPU_CTRL) >> 8) & 0xff;
	if (ncpus < 2)
		goto out;

	smp_ops = &microwatt_smp_ops;

	/*
	 * Write two instructions at location 0:
	 * mfspr r3, PIR
	 * b __secondary_hold
	 */
	*(unsigned int *)KERNELBASE = 0x7c7ffaa6;
	*(unsigned int *)(KERNELBASE+4) = 0x4800005c;

	/* enable the other CPUs, they start at location 0 */
	writel((1ul << ncpus) - 1, syscon + SYSCON_CPU_CTRL);

	timeout = 10000;
	while (!__secondary_hold_acknowledge) {
		if (--timeout == 0)
			break;
		barrier();
	}

 out:
	early_iounmap((void *)syscon, 0x100);
}

Paul.

Re: [PATCH 4/5] powerpc: Define config option for processors without broadcast TLBIE

[Paul Mackerras]

On Wed, Jan 29, 2025 at 04:14:25PM +1000, Nicholas Piggin wrote:
> On Wed Jan 29, 2025 at 8:53 AM AEST, Paul Mackerras wrote:
> > Power ISA v3.1 implementations in the Linux Compliancy Subset and
> > lower are not required to implement broadcast TLBIE, and in fact
> > Microwatt doesn't.  To avoid the need to specify "disable_tlbie" on
> > the kernel command line on SMP Microwatt systems, this defines a
> > config option that asserts that broadcast TLBIE should never be used
> > (the kernel will instead use IPIs to trigger local TLBIEs on other
> > CPUs when required).
> >
> > Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
> > ---
> >  arch/powerpc/mm/book3s64/pgtable.c       | 10 ++++++++--
> >  arch/powerpc/platforms/Kconfig.cputype   | 12 ++++++++++++
> >  arch/powerpc/platforms/microwatt/Kconfig |  1 +
> >  3 files changed, 21 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
> > index 374542528080..14ee96e2a581 100644
> > --- a/arch/powerpc/mm/book3s64/pgtable.c
> > +++ b/arch/powerpc/mm/book3s64/pgtable.c
> > @@ -588,10 +588,16 @@ int pmd_move_must_withdraw(struct spinlock *new_pmd_ptl,
> >  }
> >  #endif
> >  
> > +#ifndef CONFIG_PPC_RADIX_NO_BROADCAST_TLBIE
> 
> Hate to bikeshed, but would it be annoying to make this an affirmative
> option?

I guess we'd have to make all the platforms that do have broadcast
tlbie (and a book3s-64 MMU with radix) select that option.  Which
would be powernv and pseries, I would think.  If that's correct then
it's probably not too annoying.  Should I do that in v2?

Paul.

Re: [PATCH 2/5] powerpc/microwatt: Device-tree updates

[Paul Mackerras]

On Wed, Jan 29, 2025 at 04:36:14PM +1000, Nicholas Piggin wrote:
> On Wed Jan 29, 2025 at 8:52 AM AEST, Paul Mackerras wrote:
> > Microwatt now implements ISA v3.1 (SFFS compliancy subset), including
> > prefixed instructions, scv/rfscv, and the FSCR, HFSCR, TAR, and CTRL
> > registers.  The privileged mode of operation is now hypervisor mode
> > and there is no privileged non-hypervisor mode; the MSR[HV] bit is
> > forced to 1.
> 
> Cool. Lots of development in microwatt.
> 
> Come to think of it we should have put a broadcast-tlbie feature
> in there and you wouldn't need the other patch. That can go on
> the todo list I guess.

I thought about doing that, but it would add complexity and I'm not
sure it would actually have any measurable performance benefit.  When
I saw it was optional in the ISA for LCS and below, and that the
kernel has all the machinery for handling the cross-CPU invalidations
via IPI, it became very much the path of least resistance to use the
kernel machinery.

> system-call-vectored was available in ISA v3.0. Not that we do much
> with it at the moment IIRC, but there were dreams of wiring it in for
> compat guests. With that fixed,

Interesting.  I looked in my copy of v2.07 (PowerISA_V2.07_PUBLIC.pdf)
and it mentions rfscv in a couple of places, but has no description of
scv or rfscv.  I'll change it to v3.0.

Thanks,
Paul.

Re: [PATCH 5/5] powerpc/microwatt: Add SMP support

[Nicholas Piggin]

On Wed Jan 29, 2025 at 4:57 PM AEST, Paul Mackerras wrote:
> On Wed, Jan 29, 2025 at 04:21:26PM +1000, Nicholas Piggin wrote:
>> On Wed Jan 29, 2025 at 8:55 AM AEST, Paul Mackerras wrote:
>> > This adds support for Microwatt systems with more than one core, and
>> > updates the device tree for a 2-core version.  (This does not prevent
>> > the kernel from running on a single-core system.)
>> >
>> > Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
>> 
>> Well, I'm impressed you added SMP :)
>> 
>> What happens with a 1 CPU system? Do we time out waiting for secondaries
>> and continue, or is there something more graceful?
>
> There's a field in the SYSCON register which tells you how many cores
> there are.  microwatt_init_smp() looks at that field and only starts
> the CPUs that are there.

Ah, nice.

> Oops, sorry, I see that I forgot to do 'git add' on
> arch/powerpc/platforms/microwatt/smp.c.  Here it is (I'll include it
> properly in v2, of course):
>
> // SPDX-License-Identifier: GPL-2.0-or-later
>
> /*
>  * SMP support functions for Microwatt
>  * Copyright 2025 Paul Mackerras <paulus@ozlabs.org>
>  */
>
> #include <linux/kernel.h>
> #include <linux/smp.h>
> #include <linux/io.h>
> #include <asm/early_ioremap.h>
> #include <asm/xics.h>
>
> #include "microwatt.h"
>
> static void __init microwatt_smp_probe(void)
> {
> 	xics_smp_probe();
> }
>
> static void microwatt_smp_setup_cpu(int cpu)
> {
> 	if (cpu != 0)
> 		xics_setup_cpu();
> }
>
> static struct smp_ops_t microwatt_smp_ops = {
> 	.probe		= microwatt_smp_probe,
> 	.message_pass	= NULL,		/* Use smp_muxed_ipi_message_pass */
> 	.kick_cpu	= smp_generic_kick_cpu,
> 	.setup_cpu	= microwatt_smp_setup_cpu,
> };
>
> /* XXX get from device tree */
> #define SYSCON_BASE	0xc0000000

#define SYSCON_LENGTH 0x100

?

>
> #define SYSCON_CPU_CTRL	0x58
>
> void __init microwatt_init_smp(void)
> {
> 	volatile unsigned char __iomem *syscon;
> 	int ncpus;
> 	int timeout;
>
> 	syscon = early_ioremap(SYSCON_BASE, 0x100);

ioremap is not up by SMP init time? I always have to
trawl through init spaghetti to work it out. I guess it's
early SMP init.

> 	if (syscon == NULL) {
> 		pr_err("Failed to map SYSCON\n");
> 		return;
> 	}
> 	ncpus = (readl(syscon + SYSCON_CPU_CTRL) >> 8) & 0xff;
> 	if (ncpus < 2)
> 		goto out;
>
> 	smp_ops = &microwatt_smp_ops;
>
> 	/*
> 	 * Write two instructions at location 0:
> 	 * mfspr r3, PIR
> 	 * b __secondary_hold
> 	 */
> 	*(unsigned int *)KERNELBASE = 0x7c7ffaa6;
> 	*(unsigned int *)(KERNELBASE+4) = 0x4800005c;

Could move constants to PPC_INST_ ?

>
> 	/* enable the other CPUs, they start at location 0 */
> 	writel((1ul << ncpus) - 1, syscon + SYSCON_CPU_CTRL);
>
> 	timeout = 10000;
> 	while (!__secondary_hold_acknowledge) {
> 		if (--timeout == 0)
> 			break;
> 		barrier();
> 	}
>
>  out:
> 	early_iounmap((void *)syscon, 0x100);
> }

Looks okay otherwise.

Thanks,
Nick

Re: [PATCH 4/5] powerpc: Define config option for processors without broadcast TLBIE

[Nicholas Piggin]

On Wed Jan 29, 2025 at 5:10 PM AEST, Paul Mackerras wrote:
> On Wed, Jan 29, 2025 at 04:14:25PM +1000, Nicholas Piggin wrote:
>> On Wed Jan 29, 2025 at 8:53 AM AEST, Paul Mackerras wrote:
>> > Power ISA v3.1 implementations in the Linux Compliancy Subset and
>> > lower are not required to implement broadcast TLBIE, and in fact
>> > Microwatt doesn't.  To avoid the need to specify "disable_tlbie" on
>> > the kernel command line on SMP Microwatt systems, this defines a
>> > config option that asserts that broadcast TLBIE should never be used
>> > (the kernel will instead use IPIs to trigger local TLBIEs on other
>> > CPUs when required).
>> >
>> > Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
>> > ---
>> >  arch/powerpc/mm/book3s64/pgtable.c       | 10 ++++++++--
>> >  arch/powerpc/platforms/Kconfig.cputype   | 12 ++++++++++++
>> >  arch/powerpc/platforms/microwatt/Kconfig |  1 +
>> >  3 files changed, 21 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
>> > index 374542528080..14ee96e2a581 100644
>> > --- a/arch/powerpc/mm/book3s64/pgtable.c
>> > +++ b/arch/powerpc/mm/book3s64/pgtable.c
>> > @@ -588,10 +588,16 @@ int pmd_move_must_withdraw(struct spinlock *new_pmd_ptl,
>> >  }
>> >  #endif
>> >  
>> > +#ifndef CONFIG_PPC_RADIX_NO_BROADCAST_TLBIE
>> 
>> Hate to bikeshed, but would it be annoying to make this an affirmative
>> option?
>
> I guess we'd have to make all the platforms that do have broadcast
> tlbie (and a book3s-64 MMU with radix) select that option.  Which
> would be powernv and pseries, I would think.  If that's correct then
> it's probably not too annoying.  Should I do that in v2?

I think you're right, powernv and pseries. If you wouldn't mind doing
it please.

Thanks,
Nick

Re: [PATCH 2/5] powerpc/microwatt: Device-tree updates

[Nicholas Piggin]

On Wed Jan 29, 2025 at 5:18 PM AEST, Paul Mackerras wrote:
> On Wed, Jan 29, 2025 at 04:36:14PM +1000, Nicholas Piggin wrote:
>> On Wed Jan 29, 2025 at 8:52 AM AEST, Paul Mackerras wrote:
>> > Microwatt now implements ISA v3.1 (SFFS compliancy subset), including
>> > prefixed instructions, scv/rfscv, and the FSCR, HFSCR, TAR, and CTRL
>> > registers.  The privileged mode of operation is now hypervisor mode
>> > and there is no privileged non-hypervisor mode; the MSR[HV] bit is
>> > forced to 1.
>> 
>> Cool. Lots of development in microwatt.
>> 
>> Come to think of it we should have put a broadcast-tlbie feature
>> in there and you wouldn't need the other patch. That can go on
>> the todo list I guess.
>
> I thought about doing that, but it would add complexity and I'm not
> sure it would actually have any measurable performance benefit.  When
> I saw it was optional in the ISA for LCS and below, and that the
> kernel has all the machinery for handling the cross-CPU invalidations
> via IPI, it became very much the path of least resistance to use the
> kernel machinery.

I was unclear, I meant we (well, I) should have added that feature
to the cpufeatures device tree. I'm sure I did because I also added
the IPI+TLBIEL support but must be mistaken or never submitted it.

Perfectly reasonable to not add broadcast tlbie in microwatt.

>
>> system-call-vectored was available in ISA v3.0. Not that we do much
>> with it at the moment IIRC, but there were dreams of wiring it in for
>> compat guests. With that fixed,
>
> Interesting.  I looked in my copy of v2.07 (PowerISA_V2.07_PUBLIC.pdf)
> and it mentions rfscv in a couple of places, but has no description of
> scv or rfscv.  I'll change it to v3.0.

Yeah that must be a mistake in the 2.07 doc.

Thanks,
Nick

Re: [PATCH 5/5] powerpc/microwatt: Add SMP support

[Michael Ellerman]

Paul Mackerras <paulus@ozlabs.org> writes:
> On Wed, Jan 29, 2025 at 04:21:26PM +1000, Nicholas Piggin wrote:
>> On Wed Jan 29, 2025 at 8:55 AM AEST, Paul Mackerras wrote:
>> > This adds support for Microwatt systems with more than one core, and
>> > updates the device tree for a 2-core version.  (This does not prevent
>> > the kernel from running on a single-core system.)
>> >
>> > Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
>> 
>> Well, I'm impressed you added SMP :)
>> 
>> What happens with a 1 CPU system? Do we time out waiting for secondaries
>> and continue, or is there something more graceful?
>
> There's a field in the SYSCON register which tells you how many cores
> there are.  microwatt_init_smp() looks at that field and only starts
> the CPUs that are there.
>
> Oops, sorry, I see that I forgot to do 'git add' on
> arch/powerpc/platforms/microwatt/smp.c.  Here it is (I'll include it
> properly in v2, of course):
>
> // SPDX-License-Identifier: GPL-2.0-or-later
>
> /*
>  * SMP support functions for Microwatt
>  * Copyright 2025 Paul Mackerras <paulus@ozlabs.org>
>  */
>
> #include <linux/kernel.h>
> #include <linux/smp.h>
> #include <linux/io.h>
> #include <asm/early_ioremap.h>
> #include <asm/xics.h>
>
> #include "microwatt.h"
>
> static void __init microwatt_smp_probe(void)
> {
> 	xics_smp_probe();
> }
>
> static void microwatt_smp_setup_cpu(int cpu)
> {
> 	if (cpu != 0)
> 		xics_setup_cpu();
> }
>
> static struct smp_ops_t microwatt_smp_ops = {
> 	.probe		= microwatt_smp_probe,
> 	.message_pass	= NULL,		/* Use smp_muxed_ipi_message_pass */
> 	.kick_cpu	= smp_generic_kick_cpu,
> 	.setup_cpu	= microwatt_smp_setup_cpu,
> };
>
> /* XXX get from device tree */
> #define SYSCON_BASE	0xc0000000
>
> #define SYSCON_CPU_CTRL	0x58
>
> void __init microwatt_init_smp(void)
> {
> 	volatile unsigned char __iomem *syscon;
> 	int ncpus;
> 	int timeout;
>
> 	syscon = early_ioremap(SYSCON_BASE, 0x100);
> 	if (syscon == NULL) {
> 		pr_err("Failed to map SYSCON\n");
> 		return;
> 	}
> 	ncpus = (readl(syscon + SYSCON_CPU_CTRL) >> 8) & 0xff;
> 	if (ncpus < 2)
> 		goto out;
>
> 	smp_ops = &microwatt_smp_ops;
>
> 	/*
> 	 * Write two instructions at location 0:
> 	 * mfspr r3, PIR
> 	 * b __secondary_hold
> 	 */
> 	*(unsigned int *)KERNELBASE = 0x7c7ffaa6;
> 	*(unsigned int *)(KERNELBASE+4) = 0x4800005c;
 
There's macros that would make these a little nicer, ie. PPC_RAW_MFSPR()
and PPC_RAW_BRANCH().

> 	/* enable the other CPUs, they start at location 0 */
> 	writel((1ul << ncpus) - 1, syscon + SYSCON_CPU_CTRL);
>
> 	timeout = 10000;
> 	while (!__secondary_hold_acknowledge) {
> 		if (--timeout == 0)
> 			break;
> 		barrier();
> 	}

I assume CPU 0 always boots first?

Is the loop actually necessary? It only waits for a single non-zero CPU
to come up after all, not all of them.

cheers

Re: [PATCH 5/5] powerpc/microwatt: Add SMP support

[Paul Mackerras]

On Wed, Jan 29, 2025 at 06:12:55PM +1000, Nicholas Piggin wrote:
> On Wed Jan 29, 2025 at 4:57 PM AEST, Paul Mackerras wrote:

> > void __init microwatt_init_smp(void)
> > {
> > 	volatile unsigned char __iomem *syscon;
> > 	int ncpus;
> > 	int timeout;
> >
> > 	syscon = early_ioremap(SYSCON_BASE, 0x100);
> 
> ioremap is not up by SMP init time? I always have to
> trawl through init spaghetti to work it out. I guess it's
> early SMP init.

I had a little difficulty with this, and ended up with this function
(microwatt_init_smp) being called at platform probe time, which is why
the early_ioremap.  We need the secondary CPUs spinning in
__secondary_hold before smp_release_cpus() gets called, because that
is what sends them to generic_secondary_smp_init().
Now smp_release_cpus() is called in setup_arch() before even the
platform's setup_arch() function.  (And also, smp_setup_cpu_maps()
gets called even earlier.)  Hence the choice of the platform probe
function.

On platforms with OF and RTAS, the secondary CPUs are started and left
spinning in __secondary_hold by prom_init(), which is right at the
beginning, before the kernel proper starts executing.  Microwatt
doesn't have OF or RTAS, and the secondary CPUs are held in reset
until microwatt_init_smp() releases them (and then they start at 0).
Doing that as early as possible therefore more-or-less matches what
OF/RTAS platforms do.

Paul.

Re: [PATCH 5/5] powerpc/microwatt: Add SMP support

[Paul Mackerras]

On Wed, Jan 29, 2025 at 11:50:44PM +1100, Michael Ellerman wrote:
> Paul Mackerras <paulus@ozlabs.org> writes:

> There's macros that would make these a little nicer, ie. PPC_RAW_MFSPR()
> and PPC_RAW_BRANCH().

Thanks, I'll use them.

> > 	/* enable the other CPUs, they start at location 0 */
> > 	writel((1ul << ncpus) - 1, syscon + SYSCON_CPU_CTRL);
> >
> > 	timeout = 10000;
> > 	while (!__secondary_hold_acknowledge) {
> > 		if (--timeout == 0)
> > 			break;
> > 		barrier();
> > 	}
> 
> I assume CPU 0 always boots first?

Yes, on reset, only CPU 0 starts executing, and the rest are held in
the reset state until that writel() to SYSCON_CPU_CTRL enables them.
They all then start executing at location 0.

> Is the loop actually necessary? It only waits for a single non-zero CPU
> to come up after all, not all of them.

Yes, it only waits for the first one to get to the point of setting
__secondary_hold_acknowledge, but that might take a microsecond or so,
meaning we do need a loop (or I suppose a udelay(10) would probably
also be sufficient).  The assumption is that any others will get into
the loop very shortly after the first.  If that proves incorrect, it
would be possible instead to start them one by one, clearing
__secondary_hold_acknowledge each time (but I don't have a way to test
such code at present).

Paul.

Re: [PATCH 0/5] Microwatt updates

[Segher Boessenkool]

Hi!

On Wed, Jan 29, 2025 at 09:49:49AM +1100, Paul Mackerras wrote:
> This patch series updates the kernel support for the Microwatt
> soft-core and its implementation on FPGA systems, particularly the
> Digilent Arty A7-100 FPGA development board.
> 
> Microwatt now supports almost all of the features of the SFFS (Scalar
> Fixed-poin and Floating-point Subset) compliancy subset of Power ISA
> version 3.1C, including prefixed instructions and the fixed-point hash
> (ROP mitigation) instructions.  It is also now SMP-capable, and a
> dual-core system will fit on the Arty A7-100 board.

Congratulations!

> Microwatt does not have broadcast TLB invalidations in SMP systems;

So it isn't *really* SMP.  Compare 603 vs. 604.  With enough software
(OS) trickery you can make some things work, but :-)  (There have been
many 603 multiprocessor systems as well, to draw the analogy further
than wanted :-) )

> the kernel already has code to deal with this.  One of the patches in
> this series provides a config option to allow platforms to select
> unconditionally the behaviour where cross-CPU TLB invalidations are
> handled using inter-processor interrupts.

Are there plans to broadcast the (SMP cache invalidation) messages?
Will uwatt support some real bus protocol, for example?

Again, congrats on this great milestone!  Does this floating point
support do square roots as well (aka "gpopt"; does it do "gfxopt" for
that matter, fsel?)  fsqrt is kinda tricky to get to work fully
correctly :-)


Segher

Re: [PATCH 3/5] powerpc/microwatt: Define an idle power-save function

[Segher Boessenkool]

Hi!

> +static void microwatt_idle(void)
> +{
> +	if (!prep_irq_for_idle())
> +		return;
> +
> +	__asm__ __volatile__ ("wait");
> +}

All asm without outputs is always implicitly volatile (if it wasn't, it
could always be transfirmed whatever way you want, like, optimised away
completely).

It can still be useful for documentation purposes of course, but here it
is the other way around really, it is just cargo cult :-(


Segher

Re: [PATCH 3/5] powerpc/microwatt: Define an idle power-save function

[Segher Boessenkool]

On Wed, Jan 29, 2025 at 04:06:03PM +1000, Nicholas Piggin wrote:
> Does wait cause MSR[EE] to be set? If not, do you need to use
> prep_irq_for_idle_irqsoff() here maybe?

Assuming this does implement the standard ISA 2.03 wait instruction
(and it better), this does not do anything other than to stop fetching
and execution until some later event happens.

What values of the WC field does uwatt implement?


Segher

Re: [PATCH 2/5] powerpc/microwatt: Device-tree updates

[Segher Boessenkool]

On Wed, Jan 29, 2025 at 09:52:09AM +1100, Paul Mackerras wrote:
> -			isa = <3000>;
> +			isa = <3010>;

Does this mean 3.1, or 3.01?  If the former, can this also encode 3.1C?
Should uwatt say to support that?

>  			little-endian {
> -				isa = <2050>;
> -				usable-privilege = <3>;
> +				isa = <0>;
> +				usable-privilege = <7>;
> +				os-support = <0>;
>  				hwcap-bit-nr = <1>;
>  			};

What does "isa" in this node mean?  Why is it changed to 0?  (I don't
know this node at all, I only know a property with the same name).

>  			cache-inhibited-large-page {
> -				isa = <2040>;
> -				usable-privilege = <2>;
> +				isa = <0>;
> +				usable-privilege = <6>;
> +				os-support = <0>;
>  			};

Similar question here.

> -				isa = <2010>;
> +				isa = <0x00>;
>  				usable-privilege = <2>;
> +				os-support = <0>;
>  			};

And here.  That's a quite woolly way to say "0" btw ;-)


Segher

Re: [PATCH 2/5] powerpc/microwatt: Device-tree updates

[Segher Boessenkool]

On Wed, Jan 29, 2025 at 04:36:14PM +1000, Nicholas Piggin wrote:
> On Wed Jan 29, 2025 at 8:52 AM AEST, Paul Mackerras wrote:
> > Microwatt now implements ISA v3.1 (SFFS compliancy subset), including
> > prefixed instructions, scv/rfscv, and the FSCR, HFSCR, TAR, and CTRL
> > registers.  The privileged mode of operation is now hypervisor mode
> > and there is no privileged non-hypervisor mode; the MSR[HV] bit is
> > forced to 1.
> 
> Cool. Lots of development in microwatt.
> 
> Come to think of it we should have put a broadcast-tlbie feature
> in there and you wouldn't need the other patch. That can go on
> the todo list I guess.
> 
> system-call-vectored was available in ISA v3.0. Not that we do much
> with it at the moment IIRC, but there were dreams of wiring it in for
> compat guests. With that fixed,

This patch says 2.07 for it (if that is what 2070 mens(if that is what
2070 means).  Something's not right :-)

> > +			system-call-vectored {
> > +				isa = <2070>;
> > +				usable-privilege = <7>;
> > +				os-support = <1>;
> > +				fscr-bit-nr = <12>;
> > +				hwcap-bit-nr = <52>;
> >  			};


Segher

Re: [PATCH 2/5] powerpc/microwatt: Device-tree updates

[Segher Boessenkool]

On Wed, Jan 29, 2025 at 06:18:54PM +1100, Paul Mackerras wrote:
> Interesting.  I looked in my copy of v2.07 (PowerISA_V2.07_PUBLIC.pdf)
> and it mentions rfscv in a couple of places, but has no description of
> scv or rfscv.  I'll change it to v3.0.

Huh, rfscv is 3.0 and later according to later versions of the ISA.


Segher

Re: [PATCH 2/5] powerpc/microwatt: Device-tree updates

[Segher Boessenkool]

On Wed, Jan 29, 2025 at 06:20:31PM +1000, Nicholas Piggin wrote:
> Perfectly reasonable to not add broadcast tlbie in microwatt.

If you call "the easy way out" reasonable, then sure.  This pretty
trivial hardware addition causes so many software headaches whenn
missing, it isn't funny.  "Friends don't let friends skimp on minimal
system support features", etc. :-)


Segher

Re: [PATCH 4/5] powerpc: Define config option for processors without broadcast TLBIE

[Segher Boessenkool]

On Wed, Jan 29, 2025 at 09:53:44AM +1100, Paul Mackerras wrote:
> Power ISA v3.1 implementations in the Linux Compliancy Subset and
> lower are not required to implement broadcast TLBIE, and in fact
> Microwatt doesn't.

But this pretty much means that such systems cannot be SMP systems at
all.  Implementing the necessary synchronisation using some
cobbled-together rickety jury-rigged contraption is not anyone's goal.

Interesting that you did not see any performance loss, btw!  Well you
didn't try it on anything bigger than a simple dual CPU :-)


Segher

Re: [PATCH 4/5] powerpc: Define config option for processors without broadcast TLBIE

[Segher Boessenkool]

On Wed, Jan 29, 2025 at 06:10:43PM +1100, Paul Mackerras wrote:
> > Hate to bikeshed, but would it be annoying to make this an affirmative
> > option?
> 
> I guess we'd have to make all the platforms that do have broadcast
> tlbie (and a book3s-64 MMU with radix) select that option.  Which
> would be powernv and pseries, I would think.  If that's correct then
> it's probably not too annoying.  Should I do that in v2?

Such an option would mean "this platform implements the Power
architecture correctly".  Interesting :-)


Segher

Re: [PATCH 0/5] Microwatt updates

[Paul Mackerras]

On Fri, Jan 31, 2025 at 10:13:43AM -0600, Segher Boessenkool wrote:
> Hi!
> 
> On Wed, Jan 29, 2025 at 09:49:49AM +1100, Paul Mackerras wrote:
> > This patch series updates the kernel support for the Microwatt
> > soft-core and its implementation on FPGA systems, particularly the
> > Digilent Arty A7-100 FPGA development board.
> > 
> > Microwatt now supports almost all of the features of the SFFS (Scalar
> > Fixed-poin and Floating-point Subset) compliancy subset of Power ISA
> > version 3.1C, including prefixed instructions and the fixed-point hash
> > (ROP mitigation) instructions.  It is also now SMP-capable, and a
> > dual-core system will fit on the Arty A7-100 board.
> 
> Congratulations!

Thanks!

> > Microwatt does not have broadcast TLB invalidations in SMP systems;
> 
> So it isn't *really* SMP.  Compare 603 vs. 604.  With enough software

Actually, the term "SMP" is about latency to memory, indicating that
all CPUs have access to memory with similar latency.  It doesn't say
anything about coherency, either of memory caches or TLBs.  So yes,
Microwatt is SMP.

And for the record, the instruction and data caches are coherent,
which is what matters to user-space.  Stuff to do with the TLB is not
visible to user-space.  And the ISA explicitly says "TLBs are
non-coherent caches of the HTABs and Radix Trees" (Book III section
6.10.1).

> (OS) trickery you can make some things work, but :-)  (There have been
> many 603 multiprocessor systems as well, to draw the analogy further
> than wanted :-) )

603 was a looong time ago, I don't recall the details.

Regarding broadcast TLBIEs, the protocols and mechanisms for doing
that are known to be complex and slow in the IBM Power processors (ask
Derek Williams about that :).  Anton found that in fact doing only
local TLBIEs and using IPIs gave *better* performance on IBM Power
systems than using hardware broadcast TLBIEs in many cases (the reason
being that software knows which other CPUs might have a given TLB
entry, often quite a small set, whereas hardware doesn't, and has to
send the invalidation to every CPU and wait for a response from every
CPU).  Add to that, that most other SMP-capable CPU architectures
don't do broadcast TLB invalidations, Intel x86 for example.

> > the kernel already has code to deal with this.  One of the patches in
> > this series provides a config option to allow platforms to select
> > unconditionally the behaviour where cross-CPU TLB invalidations are
> > handled using inter-processor interrupts.
> 
> Are there plans to broadcast the (SMP cache invalidation) messages?

Cache (i.e. instruction and data cache) - yes, they *are* coherent.
More precisely, the D caches are write-through, and all I and D caches
snoop writes to memory (including DMA writes) and invalidate any cache
lines being written to.

> Will uwatt support some real bus protocol, for example?

"Real" meaning using tri-state bus drivers, like we did in the 90s? :)

> Again, congrats on this great milestone!  Does this floating point
> support do square roots as well (aka "gpopt"; does it do "gfxopt" for
> that matter, fsel?)  fsqrt is kinda tricky to get to work fully
> correctly :-)

Yes, fsqrt and fsel are implemented in hardware, and are accurate to
the last bit.  Also, the FPU handles denormalized values in hardware
(both input and output) and implements all exception handling as per
the ISA, including the trap-enabled overflow cases.  Feel free to run
whatever tests you like and report bugs.  But we're getting a bit
off-topic from the kernel patches. :)

Paul.

Re: [PATCH 3/5] powerpc/microwatt: Define an idle power-save function

[Paul Mackerras]

On Fri, Jan 31, 2025 at 10:32:55AM -0600, Segher Boessenkool wrote:
> On Wed, Jan 29, 2025 at 04:06:03PM +1000, Nicholas Piggin wrote:
> > Does wait cause MSR[EE] to be set? If not, do you need to use
> > prep_irq_for_idle_irqsoff() here maybe?
> 
> Assuming this does implement the standard ISA 2.03 wait instruction

ISA 2.03?  I don't have a copy of 2.03.  I looked in 2.04 and the wait
instruction there has a different extended opcode from the ISA 3.0/3.1
wait instruction.  Why is ISA 2.03 at all relevant to anything here?
In any case, the description of the wait instruction in 2.04 doesn't
actually say that it waits for anything other than all previous
instructions being finished.

> (and it better), this does not do anything other than to stop fetching
> and execution until some later event happens.
> 
> What values of the WC field does uwatt implement?

Just WC=0; for other values it's a no-op.  (Which is still arguably
correct given that execution is allowed to resume when an
implementation-dependent event occurs; P9 for instance just stops for
a few microseconds, if I recall correctly, for any WC value.)

Paul.

Re: [PATCH 0/5] Microwatt updates

[Gabriel Paubert]

[Sorry, I wanted to reply earlier, but it stayed in my drafts folder for a month]

On Sat, Feb 01, 2025 at 12:22:51PM +1100, Paul Mackerras wrote:
[snipped]
> 
> 603 was a looong time ago, I don't recall the details.
> 
> Regarding broadcast TLBIEs, the protocols and mechanisms for doing
> that are known to be complex and slow in the IBM Power processors (ask
> Derek Williams about that :).  Anton found that in fact doing only
> local TLBIEs and using IPIs gave *better* performance on IBM Power
> systems than using hardware broadcast TLBIEs in many cases (the reason
> being that software knows which other CPUs might have a given TLB
> entry, often quite a small set, whereas hardware doesn't, and has to
> send the invalidation to every CPU and wait for a response from every
> CPU).  Add to that, that most other SMP-capable CPU architectures
> don't do broadcast TLB invalidations, Intel x86 for example.

Actually it's coming to x86, at least on the AMD side:

https://lore.kernel.org/all/20250206044346.3810242-1-riel@surriel.com/

with performance numbers which look rather good.

I don't know how it looks like at the level of the hardware protocol,
but implementing it on a single chip/socket is likely relatively simple.

Gabriel

> 
> > > the kernel already has code to deal with this.  One of the patches in
> > > this series provides a config option to allow platforms to select
> > > unconditionally the behaviour where cross-CPU TLB invalidations are
> > > handled using inter-processor interrupts.
> > 
> > Are there plans to broadcast the (SMP cache invalidation) messages?
> 
> Cache (i.e. instruction and data cache) - yes, they *are* coherent.
> More precisely, the D caches are write-through, and all I and D caches
> snoop writes to memory (including DMA writes) and invalidate any cache
> lines being written to.
> 
> > Will uwatt support some real bus protocol, for example?
> 
> "Real" meaning using tri-state bus drivers, like we did in the 90s? :)
> 
> > Again, congrats on this great milestone!  Does this floating point
> > support do square roots as well (aka "gpopt"; does it do "gfxopt" for
> > that matter, fsel?)  fsqrt is kinda tricky to get to work fully
> > correctly :-)
> 
> Yes, fsqrt and fsel are implemented in hardware, and are accurate to
> the last bit.  Also, the FPU handles denormalized values in hardware
> (both input and output) and implements all exception handling as per
> the ISA, including the trap-enabled overflow cases.  Feel free to run
> whatever tests you like and report bugs.  But we're getting a bit
> off-topic from the kernel patches. :)
> 
> Paul.
>