NUMA support for dual core Opteron

NUMA support for dual core Opteron

[yhlu]

Someone mentioned that NUMA support for dual core opteron need acpi
support in LinuxBIOS.

there may be some other solution for that.
1. PowerPC already support dual core and it should support NUMA, So
the Open Firmware must have some NUMA entry definition.
Can we make x86-64 kernel support OpenFirmware interface so we can use
OpenBIOS as payload of LinuxBIOS.
2. enable acpi and add the NUMA entries into it, the Linux Kernel will be happy.
3. If we are trying to use ADLO to load Windows/Solaris/FreeBSD, We
need to pass related acpi info to ADLO....

Solution 1 will be ideal one, and can make Solaris for X86-64 use
OpenFirmware interface too.....

which one is better?

YH

Re: [LinuxBIOS] NUMA support for dual core Opteron

[Li-Ta Lo]

On Thu, 2005-07-14 at 10:58 -0700, yhlu wrote:
> Someone mentioned that NUMA support for dual core opteron need acpi
> support in LinuxBIOS.
> 
> there may be some other solution for that.
> 1. PowerPC already support dual core and it should support NUMA, So
> the Open Firmware must have some NUMA entry definition.
> Can we make x86-64 kernel support OpenFirmware interface so we can use
> OpenBIOS as payload of LinuxBIOS.
> 2. enable acpi and add the NUMA entries into it, the Linux Kernel will be happy.
> 3. If we are trying to use ADLO to load Windows/Solaris/FreeBSD, We
> need to pass related acpi info to ADLO....
> 
> Solution 1 will be ideal one, and can make Solaris for X86-64 use
> OpenFirmware interface too.....
> 
> which one is better?
> 


AFIAK, for x86_64 kernel, it will try to read NUMA configuration from
HW directory. We don't have to export any ACPI table.

-- 
Li-Ta Lo <ollie@lanl.gov>
Los Alamos National Lab

Re: [discuss] Re: [LinuxBIOS] NUMA support for dual core Opteron

[Andi Kleen]

> AFIAK, for x86_64 kernel, it will try to read NUMA configuration from
> HW directory. We don't have to export any ACPI table.

It doesn't work for dual core or 8 sockets for some reason. Since the SRAT
code works fine and is in general more future proof we never tracked down 
why. Patches welcome. 

However you'll likely need ACPI for other reasons anyways, e.g. for
better power saving.

-Andi

Re: [discuss] Re: [LinuxBIOS] NUMA support for dual core Opteron

[yhlu]

FYI in Tyan S4881 (8 ways dual core 875 cpu ) with LinuxBIOS I got
also the 1G mem hole is enabled.

So the kernel should be OK with read NUMA directly from HW.

YH

Firmware type: LinuxBIOS
old bootloader convention, maybe loadlin?
Bootdata ok (command line is apic=debug ramdisk_size=65536
root=/dev/ram0 rw console=tty0 console=ttyS0,115200n8 )
Linux version 2.6.12 (root@tst288xsuse9) (gcc version 3.3.3 (SuSE
Linux)) #8 SMP Fri Jun 24 12:41:43 PDT 2005
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 0000000000000e48 (reserved)
 BIOS-e820: 0000000000000e48 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 00000000000f0400 (reserved)
 BIOS-e820: 0000000000100000 - 00000000c0000000 (usable)
 BIOS-e820: 0000000100000000 - 0000000840000000 (usable)
Scanning NUMA topology in Northbridge 24
Number of nodes 8
Node 0 MemBase 0000000000000000 Limit 000000013fffffff
Node 1 MemBase 0000000140000000 Limit 000000023fffffff
Node 2 MemBase 0000000240000000 Limit 000000033fffffff
Node 3 MemBase 0000000340000000 Limit 000000043fffffff
Node 4 MemBase 0000000440000000 Limit 000000053fffffff
Node 5 MemBase 0000000540000000 Limit 000000063fffffff
Node 6 MemBase 0000000640000000 Limit 000000073fffffff
Node 7 MemBase 0000000740000000 Limit 000000083fffffff
node 1 shift 24 addr 140000000 conflict 0
node 1 shift 25 addr 1fe000000 conflict 0
node 3 shift 26 addr 3fc000000 conflict 0
node 7 shift 27 addr 7f8000000 conflict 0
Using node hash shift of 28
Bootmem setup node 0 0000000000000000-000000013fffffff
Bootmem setup node 1 0000000140000000-000000023fffffff
Bootmem setup node 2 0000000240000000-000000033fffffff
Bootmem setup node 3 0000000340000000-000000043fffffff
Bootmem setup node 4 0000000440000000-000000053fffffff
Bootmem setup node 5 0000000540000000-000000063fffffff
Bootmem setup node 6 0000000640000000-000000073fffffff
Bootmem setup node 7 0000000740000000-000000083fffffff

in LB
Setting variable MTRR 0, base:    0MB, range: 32768MB, type WB
Setting variable MTRR 1, base: 32768MB, range: 1024MB, type WB
Setting variable MTRR 2, base: 3072MB, range: 1024MB, type UC


On 7/14/05, Andi Kleen <ak@suse.de> wrote:
> > AFIAK, for x86_64 kernel, it will try to read NUMA configuration from
> > HW directory. We don't have to export any ACPI table.
> 
> It doesn't work for dual core or 8 sockets for some reason. Since the SRAT
> code works fine and is in general more future proof we never tracked down
> why. Patches welcome.
> 
> However you'll likely need ACPI for other reasons anyways, e.g. for
> better power saving.
> 
> -Andi
> 
>

Re: [discuss] Re: [LinuxBIOS] NUMA support for dual core Opteron

[Ronald G. Minnich]

On Thu, 14 Jul 2005, Andi Kleen wrote:

> However you'll likely need ACPI for other reasons anyways, e.g. for
> better power saving.

bummer. What the BIOS vendors are doing (to lock in proprietary BIOS, some
say)  is making ACPI tables copyright the BIOS vendor, not the motherboard 
vendor. So LinuxBIOS will have to reverse engineer their own, somehow. 

Shame we can't free ourselves of ACIP a bit. Oh well.

ron

Re: [discuss] Re: [LinuxBIOS] NUMA support for dual core Opteron

[Andi Kleen]

On Thu, Jul 14, 2005 at 01:04:26PM -0600, Ronald G. Minnich wrote:
> 
> 
> On Thu, 14 Jul 2005, Andi Kleen wrote:
> 
> > However you'll likely need ACPI for other reasons anyways, e.g. for
> > better power saving.
> 
> bummer. What the BIOS vendors are doing (to lock in proprietary BIOS, some
> say)  is making ACPI tables copyright the BIOS vendor, not the motherboard 
> vendor. So LinuxBIOS will have to reverse engineer their own, somehow. 

You don't need full support, many of it is optional and will
fall back to the old methods if not available. e.g. you can
probably leave out most of the PCI support if you don't want to support
PCI hotplug. Longer term it might be needed again for power management
though.

Doing PST objects for power saving shouldn't be that difficult, but you
need knowledge of the CPUs from their data sheet (and some testing 
if the power regulators on the mobo can take all the transitions) 
But it shouldn't be very motherboard specific.

However that said there is a lot of useful information
in the FADT and some other tables and I definitely plan to use more of it 
in the future.

-Andi

Re: [LinuxBIOS] NUMA support for dual core Opteron

[yhlu]

For S2895, with 1Gx8 Installed and E stepping dual core opteron with
1G mem hole emable, got

Bootdata ok (command line is apic=debug ramdisk_size=65536
root=/dev/ram0 rw console=tty0 console=ttyS0,115200n8 )
Linux version 2.6.12-rc5 (root@tst288xsuse9) (gcc version 3.3.3 (SuSE
Linux)) #26 SMP Thu Jun 2 18:15:44 PDT 2005
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 0000000000000dd0 (reserved)
 BIOS-e820: 0000000000000dd0 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 00000000000f0400 (reserved)
 BIOS-e820: 0000000000100000 - 00000000c0000000 (usable)
 BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
ACPI: Unable to locate RSDP
Scanning NUMA topology in Northbridge 24
Number of nodes 2
Node 0 MemBase 0000000000000000 Limit 000000013fffffff
Node 1 MemBase 0000000140000000 Limit 000000023fffffff
node 1 shift 24 addr 140000000 conflict 0
node 1 shift 25 addr 1fe000000 conflict 0
Using node hash shift of 26
Bootmem setup node 0 0000000000000000-000000013fffffff
Bootmem setup node 1 0000000140000000-000000023fffffff


~ # cat /proc/mt~ # cat /proc/mtrr 
reg00: base=0x00000000 (   0MB), size=8192MB: write-back, count=1
reg01: base=0x200000000 (8192MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1

with suse kernel and LinuxBIOS + S2882 I can get

linux:~ # cat /proc/meminfo 
        total:    used:    free:  shared: buffers:  cached:
Mem:  8131170304 145051648 7986118656        0 15220736 66543616
Swap: 2154979328        0 2154979328
MemTotal:      7940596 kB
MemFree:       7798944 kB
MemShared:           0 kB
Buffers:         14864 kB
Cached:          64984 kB
SwapCached:          0 kB
Active:          39056 kB
Inactive:        40844 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      7940596 kB
LowFree:       7798944 kB
SwapTotal:     2104472 kB
SwapFree:      2104472 kB
BigFree:             0 kB

Node 0 MemTotal:      4194300 kB
Node 0 MemFree:      3793788 kB
Node 0 MemUsed:       400512 kB
Node 0 HighTotal:           0 kB
Node 0 HighFree:            0 kB
Node 0 LowTotal:      4194300 kB
Node 0 LowFree:       3793788 kB

Node 1 MemTotal:      4194300 kB
Node 1 MemFree:      4005156 kB
Node 1 MemUsed:       189144 kB
Node 1 HighTotal:           0 kB
Node 1 HighFree:            0 kB
Node 1 LowTotal:      4194300 kB
Node 1 LowFree:       4005156 kB

I wonder if suse kernel have some special code to show that. in
Kernel.org, I can not Node0....Node 1...

YH

On 7/14/05, Li-Ta Lo <ollie@lanl.gov> wrote:
> On Thu, 2005-07-14 at 10:58 -0700, yhlu wrote:
> > Someone mentioned that NUMA support for dual core opteron need acpi
> > support in LinuxBIOS.
> >
> > there may be some other solution for that.
> > 1. PowerPC already support dual core and it should support NUMA, So
> > the Open Firmware must have some NUMA entry definition.
> > Can we make x86-64 kernel support OpenFirmware interface so we can use
> > OpenBIOS as payload of LinuxBIOS.
> > 2. enable acpi and add the NUMA entries into it, the Linux Kernel will be happy.
> > 3. If we are trying to use ADLO to load Windows/Solaris/FreeBSD, We
> > need to pass related acpi info to ADLO....
> >
> > Solution 1 will be ideal one, and can make Solaris for X86-64 use
> > OpenFirmware interface too.....
> >
> > which one is better?
> >
> 
> 
> AFIAK, for x86_64 kernel, it will try to read NUMA configuration from
> HW directory. We don't have to export any ACPI table.
> 
> --
> Li-Ta Lo <ollie@lanl.gov>
> Los Alamos National Lab
> 
>

Re: [discuss] Re: [LinuxBIOS] NUMA support for dual core Opteron

[Andi Kleen]

P.S.: It is very nasty to cc closed mailing lists when posting
to open ones. Please don't do that in the future.

-Andi

Re: NUMA support for dual core Opteron

[Ronald G. Minnich]

if there is any chance of getting along without ACPI entries that is best.  
Linux did do this once already, for SMP K8: K8 can boot and run NUMA
without an SRAT table. What more is needed for dual core, and could Linux
support in this area be extended?

ron

Re: [discuss] Re: NUMA support for dual core Opteron

[Andi Kleen]

[closed mailing list dropped. Sorry I have no plans to argue with
your mailbots]

On Thu, Jul 14, 2005 at 01:00:01PM -0600, Ronald G. Minnich wrote:
> if there is any chance of getting along without ACPI entries that is best.  
> Linux did do this once already, for SMP K8: K8 can boot and run NUMA
> without an SRAT table. What more is needed for dual core, and could Linux
> support in this area be extended?

The dual core NUMA parsing problem could be probably fixed. I personally
have no plans to work on it though, since the ACPI method works fine.

Feel free to submit patches.

However with 90+W CPUs I would strongly recommend having support
for PowerNow! and the old style PST table doesn't support
dual core or SMP, so you need ACPI for that anyways.

-Andi

Re: [discuss] Re: NUMA support for dual core Opteron

[yhlu]

I didn't see any problem about NUMA with LinuxBIOS + 8way dual core system.
of couse the acpi support in Kernel is disabled.

p.s. can you use powernow when acpi is disabled?
p.s.s  Is powerpc64 support ACPI? or ACPI is only can be used by x86?

YH

On 7/14/05, Andi Kleen <ak@suse.de> wrote:
> [closed mailing list dropped. Sorry I have no plans to argue with
> your mailbots]
> 
> On Thu, Jul 14, 2005 at 01:00:01PM -0600, Ronald G. Minnich wrote:
> > if there is any chance of getting along without ACPI entries that is best.
> > Linux did do this once already, for SMP K8: K8 can boot and run NUMA
> > without an SRAT table. What more is needed for dual core, and could Linux
> > support in this area be extended?
> 
> The dual core NUMA parsing problem could be probably fixed. I personally
> have no plans to work on it though, since the ACPI method works fine.
> 
> Feel free to submit patches.
> 
> However with 90+W CPUs I would strongly recommend having support
> for PowerNow! and the old style PST table doesn't support
> dual core or SMP, so you need ACPI for that anyways.
> 
> -Andi
>

Re: [discuss] Re: NUMA support for dual core Opteron

[Andi Kleen]

On Thu, Jul 14, 2005 at 07:46:49PM -0700, yhlu wrote:
> p.s. can you use powernow when acpi is disabled?

Only on uniprocessor machines.

> p.s.s  Is powerpc64 support ACPI? or ACPI is only can be used by x86?

powerpc64 uses openfirmware, not ACPI.

-Andi

Re: [discuss] Re: NUMA support for dual core Opteron

[yhlu]

Andi,

How do yo think about make x86-64 kernel support openfirmware interface?

Can we borrow some code from ppc64 arch?

YH


On 7/14/05, Andi Kleen <ak@suse.de> wrote:
> On Thu, Jul 14, 2005 at 07:46:49PM -0700, yhlu wrote:
> > p.s. can you use powernow when acpi is disabled?
> 
> Only on uniprocessor machines.
> 
> > p.s.s  Is powerpc64 support ACPI? or ACPI is only can be used by x86?
> 
> powerpc64 uses openfirmware, not ACPI.
> 
> -Andi
>

Re: [discuss] Re: NUMA support for dual core Opteron

[Andi Kleen]

On Thu, 14 Jul 2005 20:52:58 -0700
yhlu <yinghailu@gmail.com> wrote:

> Andi,
> 
> How do yo think about make x86-64 kernel support openfirmware interface?

I don't like it. We already have the old x86 BIOS interfaces and ACPI
and at some point EFI. No need for more.

-Andi

Re: [discuss] Re: NUMA support for dual core Opteron

[yhlu]

EFI support in x86-64?

Is EFI only support IA64?

Is acpi in EFI?

YH

On 7/14/05, Andi Kleen <ak@suse.de> wrote:
> On Thu, 14 Jul 2005 20:52:58 -0700
> yhlu <yinghailu@gmail.com> wrote:
> 
> > Andi,
> >
> > How do yo think about make x86-64 kernel support openfirmware interface?
> 
> I don't like it. We already have the old x86 BIOS interfaces and ACPI
> and at some point EFI. No need for more.
> 
> -Andi
>

Re: [discuss] Re: NUMA support for dual core Opteron

[Ronald G. Minnich]

On Fri, 15 Jul 2005, Andi Kleen wrote:

> Only on uniprocessor machines.

Question for the AMD guys: is there a chance of getting
non-proprietary-bios ACPI tables from AMD directly? I.e. ACPI tables as
needed for power-now etc. could be released under GPL, making inclusion
into linuxbios a bit simpler.  Right now, the only ACPI table's I've seen
all bear "IF I COPY THIS PLEASE SUE ME UNDER THE DMCA" notices :-)

If such tables are available, and I'm just out of touch, I'd be very happy
to hear that; please send me a URL. 

It makes no sense at all to me that ACPI would be copyright anyone, since
they merely describe hardware, and even the OS guys might want to copy
them around from node to node in some cases. But that's the problem right
now.

ron

Re: [discuss] Re: NUMA support for dual core Opteron

[Tom Vier]

On Thu, Jul 14, 2005 at 09:09:29PM +0200, Andi Kleen wrote:
> However with 90+W CPUs I would strongly recommend having support
> for PowerNow! and the old style PST table doesn't support
> dual core or SMP, so you need ACPI for that anyways.

Do opterons even support powernow? The proc or sysfs control file never
shows up on mine and the cpu flags don't list it. Then again, neither does
my athlon64. They're all in 32bit mode.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: [discuss] Re: NUMA support for dual core Opteron

[Andi Kleen]

On Fri, Jul 15, 2005 at 02:15:21PM -0400, Tom Vier wrote:
> On Thu, Jul 14, 2005 at 09:09:29PM +0200, Andi Kleen wrote:
> > However with 90+W CPUs I would strongly recommend having support
> > for PowerNow! and the old style PST table doesn't support
> > dual core or SMP, so you need ACPI for that anyways.
> 
> Do opterons even support powernow? The proc or sysfs control file never
> shows up on mine and the cpu flags don't list it. Then again, neither does
> my athlon64. They're all in 32bit mode.

New enough ones when the BIOS supports it too: yes. Older ones didn't.

-Andi