32-bit HIGHMEM and game console downstreams

32-bit HIGHMEM and game console downstreams

[Ash Logan]

Hello!

LWN recently did a piece on 32-bit support in the kernel, and thought as 
a user of 32-bit I would share my 2c. [1]

I maintain a downstream fork of 6.6 to support the Nintendo Wii U 
hardware [2]. I'm also in regular contact with another doing the same 
for the older Wii [3]. Linux on this era of game consoles is doing 
pretty well :)

The Wii and Wii U are both 32-bit PowerPC machines with holes in their 
memory map, which I think makes them interesting for this discussion. 
Let me summarize the hardware and kernels involved:

Wii (2006)
- 1x PowerPC 750CL "Broadway" @ 729MHz
- 24MB "MEM1" + 64MB "MEM2" (non-contiguous - MEM2 starts 256MiB in)
- Kernel 4.19 (+ CIP patchset), dev has been working on forward-porting 
all the drivers one major version at a time (he's currently up to 5.15 
last I checked) + limited upstream support (hardware bringup, UART, not 
many peripherals)

Wii U (2012)
- 3x PowerPC 750CL "Espresso" @ 1.2GHz
- 32MB "MEM1" + 2GB "MEM2" (also starts 256MiB in) + various small SRAMs
- Kernel 6.6 (+ LTS patchset), I also had a run at upstreaming some of 
it in 2022 [4] and would eventually like to go again

Special mention to the GameCube, basically a slower Wii with only 24MB 
direct RAM and 16MB of non-mapped "ARAM". Wii Linux has experimental 
support for this where they use the ARAM as swap.

All of these are flatmem devices, as that's all the 32-bit PowerPC arch 
supports, with the Wii U additionally enabling highmem for its 2GB of 
RAM. Both devices have a small memory area (MEM1) with the "bulk" of RAM 
starting at 256MiB. The Wii U in particular sounds like a candidate 
system for densemem - I would like to read up more about this if I can, 
I was only able to find seemingly unrelated information about CXL when 
searching online.

There is a somewhat active userbase for both devices. I only have stats 
for the Wii U side, but some rough nginx grepping for the last few days 
- Sep 7th-Sep 12th - shows 39 downloads of distribution tarballs and 
bootloader binaries in that period, not including torrents. In the past 
2 weeks - Aug 29th-Sep 12th - 9 people joined the community Discord, 442 
total. Anecdotally, the Wii Linux userbase appears at least twice as big 
(based on their Discord activity).

Distribution-wise, we're supported by ArchPOWER [5], Adélie Linux [6], 
and other distros. The Wii U's Espresso has CPU errata requiring a 
patched compiler, and both distributions ship separate package repos for 
this CPU. ArchPOWER requested I rebase onto 6.6 so they could have 
firmware compression - previously the Wii U was on 4.19 - so there is 
some demand for newer kernel features as well.

I know I'm talking about hobbyist use - and mostly downstream use at 
that - and I do suspect that in the event of a wider 32-bit deprecation 
we'd be fine on the final LTS, whatever that ends up being. Still, I 
think the Wii and Wii U represent a decent number of 32-bit users, so I 
wanted to add to the conversation here.

Thanks,
Ash

[1] https://lwn.net/Articles/1035727/
[2] https://linux-wiiu.org/
[3] https://wii-linux.org/
[4] https://lore.kernel.org/lkml/20221119113041.284419-1-ash@heyquark.com/
[5] https://archlinuxpower.org/
[6] https://www.adelielinux.org/

Re: 32-bit HIGHMEM and game console downstreams

[Arnd Bergmann]

On Sat, Sep 13, 2025, at 12:53, Ash Logan wrote:

Hi Ash,

> All of these are flatmem devices, as that's all the 32-bit PowerPC arch 
> supports, with the Wii U additionally enabling highmem for its 2GB of 
> RAM. Both devices have a small memory area (MEM1) with the "bulk" of RAM 
> starting at 256MiB. The Wii U in particular sounds like a candidate 
> system for densemem - I would like to read up more about this if I can, 
> I was only able to find seemingly unrelated information about CXL when 
> searching online.

I would not expect densemem to make a difference, at least if we go
with 256MB chunks (the size has not been decided, and not much code
exists in the first place).

Like most other machines, this one is probably fine with a combination
of a custom LOWMEM_SIZE setting and using zram-highmem, even if we
end up removing support for highmem page cache.

The smaller devices are probable getting problematic sooner, 96MB
in the Wii is already really tight and this only gets worse over
time.

> I know I'm talking about hobbyist use - and mostly downstream use at 
> that - and I do suspect that in the event of a wider 32-bit deprecation 
> we'd be fine on the final LTS, whatever that ends up being. Still, I 
> think the Wii and Wii U represent a decent number of 32-bit users, so I 
> wanted to add to the conversation here.

Just to be clear: there is no general 32-bit deprecation going on. When
I talked about phasing out 32-bit platforms over time, that is purely
going to be those that have no users left, or the few ones that are
causing more work than they are worth. E.g. The ppc405 ones got
removed recently (after many years of discussion) because they were
making ppc440 maintenance harder and had no known users.

Highmem does get in the way, but unless more -mm folks make a strong
argument in favor of removing it all, it's more likely that we'll
go with Willy's suggestion of keeping highmem on page cache (anon
and file mappings) than just keeling zram, and even that would
still work.

     Arnd

Re: 32-bit HIGHMEM and game console downstreams

[Ash Logan]

On 13/9/25 23:52, Arnd Bergmann wrote:

Hi Arnd! Thanks for your reply.

> Like most other machines, this one is probably fine with a combination
> of a custom LOWMEM_SIZE setting and using zram-highmem, even if we
> end up removing support for highmem page cache.

Good shout - I'm now testing a 2G/2G split which allows for 1536MiB 
lowmem. I know that's a somewhat aggressive setting for userspace, so 
we'll see if anything breaks. I read Rasbian shipped similar kernels and 
had issues with Wine, though that's not a common use case on PowerPC ^^

> The smaller devices are probable getting problematic sooner, 96MB
> in the Wii is already really tight and this only gets worse over
> time.

The maintainer of that downstream claims to be able to boot modern 
text-mode distros on the GameCube' 24MB, which I find really impressive!

> Just to be clear: there is no general 32-bit deprecation going on. When
> I talked about phasing out 32-bit platforms over time, that is purely
> going to be those that have no users left, or the few ones that are
> causing more work than they are worth. E.g. The ppc405 ones got
> removed recently (after many years of discussion) because they were
> making ppc440 maintenance harder and had no known users.
> 
> Highmem does get in the way, but unless more -mm folks make a strong
> argument in favor of removing it all, it's more likely that we'll
> go with Willy's suggestion of keeping highmem on page cache (anon
> and file mappings) than just keeling zram, and even that would
> still work.

That's good to hear. I would like to have the page cache, though I'm 
still going to try and maximise lowmem as well.

Thanks,
Ash

Re: 32-bit HIGHMEM and game console downstreams

[Arnd Bergmann]

On Tue, Sep 16, 2025, at 03:57, Ash Logan wrote:
> On 13/9/25 23:52, Arnd Bergmann wrote:
>
>> Like most other machines, this one is probably fine with a combination
>> of a custom LOWMEM_SIZE setting and using zram-highmem, even if we
>> end up removing support for highmem page cache.
>
> Good shout - I'm now testing a 2G/2G split which allows for 1536MiB 
> lowmem. I know that's a somewhat aggressive setting for userspace, so 
> we'll see if anything breaks. I read Rasbian shipped similar kernels and 
> had issues with Wine, though that's not a common use case on PowerPC ^^

For experiments I would suggest going all the way to 2GB lowmem on MEM2,
which would require running without MEM1. At least on powerpc you have
complete flexibility with the vmsplit, compared to arm and x86 that
only have a few distinct options.

If you run into problems with ~1.8GB user addressing, you can still
see where exactly the problem is and whether it's fixable. E.g. If
there is a single process that tries to actually use most of the
available RAM, it will likely fail when it runs out of address
space in malloc(), and there is not much we can do about that.

>> The smaller devices are probable getting problematic sooner, 96MB
>> in the Wii is already really tight and this only gets worse over
>> time.
>
> The maintainer of that downstream claims to be able to boot modern 
> text-mode distros on the GameCube' 24MB, which I find really impressive!

24MB is impressive indeed. In my latest tests I did not get below 32MB
(+swap) on an ARMv7 kernel with Debian Bookworm, and major features
turned off in both kernel and userland.

On a simpler musl+busybox userland and even more feature reduced
kernel (no network, initramfs-only) I could get to ~10MB, but then it
doesn't really do anything besides showing a shell.

    Arnd

Re: 32-bit HIGHMEM and game console downstreams

[Willy Tarreau]

On Tue, Sep 16, 2025 at 08:20:35AM +0200, Arnd Bergmann wrote:
> > The maintainer of that downstream claims to be able to boot modern 
> > text-mode distros on the GameCube' 24MB, which I find really impressive!
> 
> 24MB is impressive indeed. In my latest tests I did not get below 32MB
> (+swap) on an ARMv7 kernel with Debian Bookworm, and major features
> turned off in both kernel and userland.
> 
> On a simpler musl+busybox userland and even more feature reduced
> kernel (no network, initramfs-only) I could get to ~10MB, but then it
> doesn't really do anything besides showing a shell.

When you build your systems from source and install only the necessary
*files* (not packages) you can get much lower. Here's my reverse-proxy
for example (aarch64):

  $ df /
  Filesystem     1K-blocks  Used Available Use% Mounted on
  /dev/ram0          11520 11520         0 100% /

and my firewall:

  $ df /
  Filesystem     1K-blocks  Used Available Use% Mounted on
  /dev/ram0          16640 16640         0 100% /

these are full-featured images with tcpdump, exim, haproxy, nftables,
pppd, ntpd, wireguard, openvpn, hostapd etc and even glibc. I've been
used to installing them on 25128 SPI NOR flashes (16 MB). But yeah
that requires gcc -Os, getting rid of NLS, deciding on a case by case
basis between shared or static linking, and installing the strict
minimum for each service. I even have an old boot loader recovery
image where kernel+initramfs fit into just 1 MB with a small recovery
shell, mke2fs, dropbear and tftp client (i386).

Cheers,
Willy

Re: 32-bit HIGHMEM and game console downstreams

[Arnd Bergmann]

On Tue, Sep 16, 2025, at 09:00, Willy Tarreau wrote:
> On Tue, Sep 16, 2025 at 08:20:35AM +0200, Arnd Bergmann wrote:
>> > The maintainer of that downstream claims to be able to boot modern 
>> > text-mode distros on the GameCube' 24MB, which I find really impressive!
>> 
>> On a simpler musl+busybox userland and even more feature reduced
>> kernel (no network, initramfs-only) I could get to ~10MB, but then it
>> doesn't really do anything besides showing a shell.
>
> When you build your systems from source and install only the necessary
> *files* (not packages) you can get much lower. Here's my reverse-proxy
> for example (aarch64):
>
>   $ df /
>   Filesystem     1K-blocks  Used Available Use% Mounted on
>   /dev/ram0          11520 11520         0 100% /

Yes, that was what I did in my busybox example, which used a tiny
initramfs in the end. Your ramdisk itself is already bigger than
my total 10MB RAM here, but then it also does one thing instead
of nothing.

    Arnd

Re: 32-bit HIGHMEM and game console downstreams

[Segher Boessenkool]

On Tue, Sep 16, 2025 at 11:57:00AM +1000, Ash Logan wrote:
> On 13/9/25 23:52, Arnd Bergmann wrote:
> > The smaller devices are probable getting problematic sooner, 96MB
> > in the Wii is already really tight and this only gets worse over
> > time.
> 
> The maintainer of that downstream claims to be able to boot modern text-mode
> distros on the GameCube' 24MB, which I find really impressive!

Hey, I still boot up my imac G3 sometimes (if I can find a keyboard for
it!), it has 64MB of RAM and that needs some handholding to boot already
(the zImage needs some manual header editting to fix its load address,
something like that).

But booting it does :-)

> > Just to be clear: there is no general 32-bit deprecation going on. When
> > I talked about phasing out 32-bit platforms over time, that is purely
> > going to be those that have no users left, or the few ones that are
> > causing more work than they are worth. E.g. The ppc405 ones got
> > removed recently (after many years of discussion) because they were
> > making ppc440 maintenance harder and had no known users.

Good good good.  Way too reasonable!

Thanks,


Segher

Re: 32-bit HIGHMEM and game console downstreams

[Segher Boessenkool]

Hi!

On Sat, Sep 13, 2025 at 08:53:08PM +1000, Ash Logan wrote:
> Wii (2006)
> - 1x PowerPC 750CL "Broadway" @ 729MHz
> - 24MB "MEM1" + 64MB "MEM2" (non-contiguous - MEM2 starts 256MiB in)
> - Kernel 4.19 (+ CIP patchset), dev has been working on forward-porting all
> the drivers one major version at a time (he's currently up to 5.15 last I
> checked) + limited upstream support (hardware bringup, UART, not many
> peripherals)

There *aren't* many peripherals, so that is quite okay :-)

> Wii U (2012)
> - 3x PowerPC 750CL "Espresso" @ 1.2GHz

It is not a 750CL.  We never found out what the model # is, if indeed it
has one!  But the CPU cores are compatible to the Broadway, sure, there
even are configuration bits to make it do the bugs that were fixed in
Espresso!  (Just like Broadway can emulate a Gekko, 750CXe, the GCN
thing).

It does have its own PVR value of course, that is something at least :-)

(Espresso is one chip btw, with three mostly symmetrical cores).

> - 32MB "MEM1" + 2GB "MEM2" (also starts 256MiB in) + various small SRAMs

It has 32MB MEM1?  Huh.  Why?

> Special mention to the GameCube, basically a slower Wii with only 24MB
> direct RAM and 16MB of non-mapped "ARAM". Wii Linux has experimental support
> for this where they use the ARAM as swap.

The main memory on the GCN was very fast, low-latency, ram.  "1T-SRAM",
named so because it isn't SRAM, and not really single cycle access
either :-)

There is a block of it for graphics, too.  No idea where that is direct-
mapped, if at all.

> There is a somewhat active userbase for both devices.

Still?  Nice to hear!

> I only have stats for
> the Wii U side, but some rough nginx grepping for the last few days - Sep
> 7th-Sep 12th - shows 39 downloads of distribution tarballs and bootloader
> binaries in that period, not including torrents. In the past 2 weeks - Aug
> 29th-Sep 12th - 9 people joined the community Discord, 442 total.
> Anecdotally, the Wii Linux userbase appears at least twice as big (based on
> their Discord activity).

That makes sense, there are way more Wii devices than WiiU devices, but
the latter is newer and somewhat nicer (bigger) to run Linux on.

> Distribution-wise, we're supported by ArchPOWER [5], Adélie Linux [6], and
> other distros. The Wii U's Espresso has CPU errata requiring a patched
> compiler,

Can you remind me what that is about?  It shouldn't be too hard to
include it in mainline GCC.

> and both distributions ship separate package repos for this CPU.
> ArchPOWER requested I rebase onto 6.6 so they could have firmware
> compression - previously the Wii U was on 4.19 - so there is some demand for
> newer kernel features as well.
> 
> I know I'm talking about hobbyist use - and mostly downstream use at that -
> and I do suspect that in the event of a wider 32-bit deprecation we'd be
> fine on the final LTS, whatever that ends up being. Still, I think the Wii
> and Wii U represent a decent number of 32-bit users, so I wanted to add to
> the conversation here.

Hey, at least you hobbyists are responsive!

I don't see any real reason to no longer support 32-bit systems in
Linux -- all of the ways where that needs different treatment have been
figured out decades ago already, and perhaps not too many people still
use it, but for some architectures it is the only option and that will
never change.  Not all of the world is x86.  Even with Arm there are
many 32-bit only systems in the wild, some pretty new even!


Segher

Re: 32-bit HIGHMEM and game console downstreams

[Ash Logan]

On 15/9/25 00:14, Segher Boessenkool wrote:

Hello!

> On Sat, Sep 13, 2025 at 08:53:08PM +1000, Ash Logan wrote:
>> Wii (2006)
>> - 1x PowerPC 750CL "Broadway" @ 729MHz
>> - 24MB "MEM1" + 64MB "MEM2" (non-contiguous - MEM2 starts 256MiB in)
>> - Kernel 4.19 (+ CIP patchset), dev has been working on forward-porting all
>> the drivers one major version at a time (he's currently up to 5.15 last I
>> checked) + limited upstream support (hardware bringup, UART, not many
>> peripherals)
> 
> There *aren't* many peripherals, so that is quite okay :-)

That's true. The lack of a UART or similar does make USB kinda essential 
for an input device in my opinion, though getting it working is 
Complicated for DMA reasons. (I think this is the main thing holding the 
downstream back in their rebasing efforts)

>> Wii U (2012)
>> - 3x PowerPC 750CL "Espresso" @ 1.2GHz
> 
> It is not a 750CL.  We never found out what the model # is, if indeed it
> has one!  But the CPU cores are compatible to the Broadway, sure, there
> even are configuration bits to make it do the bugs that were fixed in
> Espresso!  (Just like Broadway can emulate a Gekko, 750CXe, the GCN
> thing).
> 
> It does have its own PVR value of course, that is something at least :-)
> 
> (Espresso is one chip btw, with three mostly symmetrical cores).

Yeah, I was just going for the closest "public" chip :) I think the PVR 
is closer to the CXe too, but all the HIDs, missing THRM, missing 
frequency scaling - it's very CL-y...

>> - 32MB "MEM1" + 2GB "MEM2" (also starts 256MiB in) + various small SRAMs
> 
> It has 32MB MEM1?  Huh.  Why?

New generation upgrade? MEM1 does get used for Wii U software too, 
usually to keep framebuffers and other 3D things, so I guess they wanted 
just a little more for all the 1080p buffers the new console juggles.

>> Distribution-wise, we're supported by ArchPOWER [5], Adélie Linux [6], and
>> other distros. The Wii U's Espresso has CPU errata requiring a patched
>> compiler,
> 
> Can you remind me what that is about?  It shouldn't be too hard to
> include it in mainline GCC.

The short version is "every stwcx. should be prefaced with a dcbst" - 
something to do with bus snooping (store-with-flush, store-with-kill) 
I'd guess. I have some GCC patches drafted (activated by -mcpu=espresso) 
here: https://gitlab.com/linux-wiiu/smp-patches

I'm impressed by how often IBM stuffed up atomics during this 
generation. The Xbox 360 has an extremely similar issue despite being an 
entirely different lineage of chip. No surprise the console vendors all 
went to x86 and ARM right after, I suppose.

Thanks,
Ash

Re: 32-bit HIGHMEM and game console downstreams

[Segher Boessenkool]

On Tue, Sep 16, 2025 at 12:10:14PM +1000, Ash Logan wrote:
> On 15/9/25 00:14, Segher Boessenkool wrote:
> > On Sat, Sep 13, 2025 at 08:53:08PM +1000, Ash Logan wrote:
> > > Wii (2006)
> > > - 1x PowerPC 750CL "Broadway" @ 729MHz
> > > - 24MB "MEM1" + 64MB "MEM2" (non-contiguous - MEM2 starts 256MiB in)
> > > - Kernel 4.19 (+ CIP patchset), dev has been working on forward-porting all
> > > the drivers one major version at a time (he's currently up to 5.15 last I
> > > checked) + limited upstream support (hardware bringup, UART, not many
> > > peripherals)
> > 
> > There *aren't* many peripherals, so that is quite okay :-)
> 
> That's true. The lack of a UART or similar does make USB kinda essential for
> an input device in my opinion, though getting it working is Complicated for
> DMA reasons. (I think this is the main thing holding the downstream back in
> their rebasing efforts)

We often threw something on the EXI bus, either something that directly
converts to USB (like the EXIUSB board we did), or anything goes really,
it is a really simple bus so an FPGA or a CPLD or XS1 or even pure
software works.

> > > Wii U (2012)
> > > - 3x PowerPC 750CL "Espresso" @ 1.2GHz
> > 
> > It is not a 750CL.  We never found out what the model # is, if indeed it
> > has one!  But the CPU cores are compatible to the Broadway, sure, there
> > even are configuration bits to make it do the bugs that were fixed in
> > Espresso!  (Just like Broadway can emulate a Gekko, 750CXe, the GCN
> > thing).
> > 
> > It does have its own PVR value of course, that is something at least :-)
> > 
> > (Espresso is one chip btw, with three mostly symmetrical cores).
> 
> Yeah, I was just going for the closest "public" chip :) I think the PVR is
> closer to the CXe too, but all the HIDs, missing THRM, missing frequency
> scaling - it's very CL-y...

The 750CX and 750CXe (and so, Gekko as well) have a PVN of 8, it is
just a 750 after all.  750FX has 0x7000 though, and 750GX has 0x7002.
Espresso is 0x7001, maybe it is somwhat inbetween FX and GX wrt
implementation process or the like.

The numbering is a bit of a mess, yeah :-)

> > > - 32MB "MEM1" + 2GB "MEM2" (also starts 256MiB in) + various small SRAMs
> > 
> > It has 32MB MEM1?  Huh.  Why?
> 
> New generation upgrade? MEM1 does get used for Wii U software too, usually
> to keep framebuffers and other 3D things, so I guess they wanted just a
> little more for all the 1080p buffers the new console juggles.

Ah, it appears the MEM1 is not implemented as 1T-SRAM anymore, but just
is a hunk of off-the-shelf DDR2 (or GDDR3, or DDR3, or whatever they
used; it doesn't matter so much).  Those come in power-of-two sizes
mostly :-)

There only is 3MB of 1T on WiiU, for the graphics memory.

> > > Distribution-wise, we're supported by ArchPOWER [5], Adélie Linux [6], and
> > > other distros. The Wii U's Espresso has CPU errata requiring a patched
> > > compiler,
> > 
> > Can you remind me what that is about?  It shouldn't be too hard to
> > include it in mainline GCC.
> 
> The short version is "every stwcx. should be prefaced with a dcbst" -
> something to do with bus snooping (store-with-flush, store-with-kill) I'd
> guess. I have some GCC patches drafted (activated by -mcpu=espresso) here:
> https://gitlab.com/linux-wiiu/smp-patches

Ah cool.  I'm maintainer for PowerPC GCC these days, I'll be happy to
work with whoever wants to do a good patch (not this one, it is not a
different CPU type, we shouldn't pretend it is, the actual patch can be
just a handful of lines).

> I'm impressed by how often IBM stuffed up atomics during this generation.
> The Xbox 360 has an extremely similar issue despite being an entirely
> different lineage of chip.

It is not very similar at all.  Both have some issues related to
(off-core) coherency stuff, something for which there were no good
testing frameworks in that era.  And that is where the similarity ends.

This needs full-system emulation to test well, probably interpreting
the whole VHDL, for all cores and the caches and the fabric.  Or special
ASICs to implement the whole thing at a hundred of a percent of real
speed.

> No surprise the console vendors all went to x86
> and ARM right after, I suppose.

Both of which had similar (but even worse) bugs, yeah :-)


Segher