Dual Opteron box, what's the optimal memory placement for the CPUs?

Dual Opteron box, what's the optimal memory placement for the CPUs?

[Jesper Juhl]

Quick question,

I've got an IBM eServer 325 here with 2 Opteron 240 CPUs. The box has 6 
DIMM slots, 4 for the "primary" CPU and 2 for the second one. I've got the 
following memory sticks : 4x512MB and 2x1GB

My plan is to plug the 4 512MB sticks into the slots for the first CPU and 
the 2GB sticks into the two slots for the second CPU, giving them 2GB 
each, but I could also give the first one 2x512MB and 2x1GB and the second 
one 2x512MB giving the first CPU 3GB and the second 1GB. Does it matter at 
all, and if it does, what's the optimal configuration?

--
Jesper Juhl

Re: Dual Opteron box, what's the optimal memory placement for the CPUs?

[Joel Jaeggli]

On Wed, 27 Oct 2004, Jesper Juhl wrote:

>
> Quick question,
>
> I've got an IBM eServer 325 here with 2 Opteron 240 CPUs. The box has 6
> DIMM slots, 4 for the "primary" CPU and 2 for the second one. I've got the
> following memory sticks : 4x512MB and 2x1GB
>
> My plan is to plug the 4 512MB sticks into the slots for the first CPU and
> the 2GB sticks into the two slots for the second CPU, giving them 2GB
> each, but I could also give the first one 2x512MB and 2x1GB and the second
> one 2x512MB giving the first CPU 3GB and the second 1GB. Does it matter at
> all.

doesn't really matter, they just have to be installed in pairs for bank 
interleaving. node interleaving is dependant on having banks on both cpu's 
populated.

> and if it does, what's the optimal configuration?
>
> --
> Jesper Juhl
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
-------------------------------------------------------------------------- 
Joel Jaeggli  	       Unix Consulting 	       joelja@darkwing.uoregon.edu 
GPG Key Fingerprint:     5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2

Re: Dual Opteron box, what's the optimal memory placement for the CPUs?

[Jesper Juhl]

On Wed, 27 Oct 2004, Joel Jaeggli wrote:

> On Wed, 27 Oct 2004, Jesper Juhl wrote:
> 
> > 
> > Quick question,
> > 
> > I've got an IBM eServer 325 here with 2 Opteron 240 CPUs. The box has 6
> > DIMM slots, 4 for the "primary" CPU and 2 for the second one. I've got the
> > following memory sticks : 4x512MB and 2x1GB
> > 
> > My plan is to plug the 4 512MB sticks into the slots for the first CPU and
> > the 2GB sticks into the two slots for the second CPU, giving them 2GB
> > each, but I could also give the first one 2x512MB and 2x1GB and the second
> > one 2x512MB giving the first CPU 3GB and the second 1GB. Does it matter at
> > all.
> 
> doesn't really matter, they just have to be installed in pairs for bank
> interleaving. node interleaving is dependant on having banks on both cpu's
> populated.
> 
Yeah, I know they have to be installed in pairs, but I would have thought 
that it would be best to have an even memory distribution so that an even 
amount of local memory was available to processes executing on either CPU. 
Even if Linux makes sure to execute processes on the CPU where most of 
their memory is local, wouldn't a non-even distribution make more 
processes prefer one CPU and thus not make the best possible use of them?

I don't really know very much about this specific detail (which is why I 
asked), and you tell me it doesn't matter, so I'll assume Linux has some 
intelligent way of dealing with this that I just don't know about - That's 
good enough for me, I'll trust you on that, just wanted to know for now 
and the future. :-)


--
Jesper Juhl

Re: Dual Opteron box, what's the optimal memory placement for the CPUs?

[Petr Vandrovec]

On Thu, Oct 28, 2004 at 12:00:45AM +0200, Jesper Juhl wrote:

> Yeah, I know they have to be installed in pairs, but I would have thought 
> that it would be best to have an even memory distribution so that an even 
> amount of local memory was available to processes executing on either CPU. 
> Even if Linux makes sure to execute processes on the CPU where most of 
> their memory is local, wouldn't a non-even distribution make more 
> processes prefer one CPU and thus not make the best possible use of them?
> 
> I don't really know very much about this specific detail (which is why I 
> asked), and you tell me it doesn't matter, so I'll assume Linux has some 
> intelligent way of dealing with this that I just don't know about - That's 
> good enough for me, I'll trust you on that, just wanted to know for now 
> and the future. :-)

You should also check this with your BIOS.  It will be a bit unfortunate 
if BIOS decides to just throw away 0.5GB of your memory to make space
for PCI devices, and maybe you can help BIOS a bit with right
shuffling (with 3:1 it will hopefully decide to put 1GB above
4GB boundary; with 2:2 who knows; AFAIK Opteron northbridge does not
allow for splitting DRAM on one node into two regions; at least I did not 
found how to do that).

I would prefer 2:2 if both variants give you same amount of accessible
memory.
						Best regards,
							Petr Vandrovec

Re: Dual Opteron box, what's the optimal memory placement for the CPUs?

[Thomas Zehetbauer]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

That is indeed a very good question.

If you use node interleaving you end up with memory accesses to be
distributed matching the amount of memory installed on each node.
Assuming that processes prefer the first node you get better performance
with load < 1 and memory usage < $first_node_memory.

If you use a NUMA configuration you should get better performance from
almost equally distributed memory. Especially when the load goes up.

Tom

PS: I choosed NUMA with equally distributed memory.
-- 
  T h o m a s   Z e h e t b a u e r   ( TZ251 )
  PGP encrypted mail preferred - KeyID 96FFCB89
      finger thomasz@hostmaster.org for key

XT emulator finally available for Pentium-IV: install Windows XP today!




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 481 bytes --]