[Linux-ia64] How linux support ccNUMA machine?

[Linux-ia64] How linux support ccNUMA machine?

[Barry Wu]

Hi, all, I am new to ccNUMA machine. I ask Mr. John about linux on
SGI Altix 3000. Because he is not responsible for Altix prom,
he could not give me any information about ccNUMA firmware that
linux should depend on. Besides, he give me a lot of information
about linux kernel. I do not know whether it is linux or firmware
complete ccNUMA initialization. Because in Altix, it can support
64 cpus, but every 4 cpus is a C-brick. I think every C-brick has
a prom. Therefore, it is under firmware complete 64 cpus initialization 
or linux kernel complete. 
Another question is for Itainum2, the system is boot from 4G-16MB address 
space, if this ROM address can be replace with RAM address when system
startup? Otherwise there will be many physical address holes in the system.
Am I right?
If someone knows, please give me a reply. Thanks in advance!

Barry





_________________________________________________________________
享用世界上最大的电子邮件系统― MSN Hotmail。  http://www.hotmail.com  

Re: [Linux-ia64] How linux support ccNUMA machine?

[Gilad Ben-Yossef]

On Fri, 2003-02-14 at 16:25, Barry Wu wrote:
> Hi, all, I am new to ccNUMA machine. I ask Mr. John about linux on
> SGI Altix 3000. Because he is not responsible for Altix prom,
> he could not give me any information about ccNUMA firmware that
> linux should depend on. Besides, he give me a lot of information
> about linux kernel. I do not know whether it is linux or firmware
> complete ccNUMA initialization. Because in Altix, it can support

I parse your question as: is the Linux kernel on the Altix 3000 aware of
ccNUMA or is this something that gets dealt in firmware?

If this was idneed your question ;-) then the answer is that although
you could build hardware/firmaware that will let you run non ccNUMA
aware operating system on ccNUMA operating system your perfomance  will
most likely be very lousy if the scheudler, for example, will not be
aware of the different memory spaces ansd the cost of accessing them
that is associated with a ccNUMA system. Therefore, the Linux kernel is
aware and manages the ccNUMA aspects of the system. Of course, it does
so via the PROM functions in firmware (PAL/SAL in IA64 speak. I think
:-)

> 64 cpus, but every 4 cpus is a C-brick. I think every C-brick has
> a prom. Therefore, it is under firmware complete 64 cpus initialization 
> or linux kernel complete. 

Again, I'm not sure I understood the question. The Linux kernel gets run
and initalized on each of the CPUs. Does this answer the question?

> Another question is for Itainum2, the system is boot from 4G-16MB address 
> space, if this ROM address can be replace with RAM address when system
> startup? Otherwise there will be many physical address holes in the system.
> Am I right?

I am not sure I understand the question. Linux has support for non
contingious memory, if that helps.

Gilad.
-- 
Gilad Ben-Yossef <gilad@benyossef.com>
http://benyossef.com

 "A 24-year-old South Korean man died after playing computer
  games nonstop for 86 hours, police said."

  http://www.smh.com.au/articles/2002/10/10/1034061260831.html

Re: [Linux-ia64] How linux support ccNUMA machine?

[Barry Wu]

Hi, Gilad,

>I parse your question as: is the Linux kernel on the Altix 3000 aware of
>ccNUMA or is this something that gets dealt in firmware?
My question as: linux kernel can support ccNUMA, but it is based on 
firmware.
Then is firmware responsible to finish ccNUMA initialization?
If so, because Itanium2 firmware address is at 4GB-16MB, and 4cpus consist
of mainboard, if to support 64cpus, there will be 16 mainboards. Because
linux is only one copy: one text and one data segment. When linux startup,
if it want to call firmware functions (such as PAL, SAL or even EFI), which
cpu's mainboard firmware to access? Or access its own mainboard, or access
id 0 cpu mainboard? because the memory is global shared.

If you know, please give me some infomation. Thanks again for your reply.

Best Regards,

Barry

_________________________________________________________________
与联机的朋友进行交流，请使用 MSN Messenger:  http://messenger.msn.com/cn  

Re: [Linux-ia64] How linux support ccNUMA machine?

[Rich Altmaier]

Hi Barry, 
Some general comments on our experience with using Intel's SAL and PAL
on our Altix system (up to 64p SSI):
-The PAL spec specifies whether functions must be called by each CPU, or
 has a system wide effect and may be called by any one CPU.  In general
 most all PAL functions operate only on, and retrieve info for, just the
 calling CPU.
-The SAL spec also specifies this.  The SAL spec expects that memory is
 globally accessible.  Most SAL functions operate on a shared global
 data structure, and specify that only one CPU need make a call to set
 state for all CPUs.  
-We have found very little problem with PAL & SAL in our global shared
 memory system.

The right way to figure out your situation is to proceed:
a) get a detailed spec from your hardware designer as to exactly
   what operations are "propagated" between FSBs, and which software
   has to be responsible for.
b) understand very clearly how physical address space is being shared,
   and the global view each CPU will have.   Hardware designs can do
   bizarre things in providing partial sharing.  You should argue these
   things are foolish if they are not readily supportable by software.
c) look over the PAL and SAL specs as noted above.

I hope this is helpful,
Thanks, Rich

Rich Altmaier, SGI


Barry Wu wrote:
> 
> Hi, Gilad,
> 
> >I parse your question as: is the Linux kernel on the Altix 3000 aware of
> >ccNUMA or is this something that gets dealt in firmware?
> My question as: linux kernel can support ccNUMA, but it is based on
> firmware.
> Then is firmware responsible to finish ccNUMA initialization?
> If so, because Itanium2 firmware address is at 4GB-16MB, and 4cpus consist
> of mainboard, if to support 64cpus, there will be 16 mainboards. Because
> linux is only one copy: one text and one data segment. When linux startup,
> if it want to call firmware functions (such as PAL, SAL or even EFI), which
> cpu's mainboard firmware to access? Or access its own mainboard, or access
> id 0 cpu mainboard? because the memory is global shared.
> 
> If you know, please give me some infomation. Thanks again for your reply.
> 
> Best Regards,
> 
> Barry
>