Linux MIPS Architecture development
 help / color / mirror / Atom feed
From: "Greg Chesson" <greg@xtp.engr.sgi.com>
To: <pjlahaie@atlsci.com>
Cc: Ariel Faigon <ariel@cthulhu.engr.sgi.com>,
	Olivier Galibert <galibert@pobox.com>,
	linux@cthulhu.engr.sgi.com
Subject: Re: help offered
Date: Wed, 25 Nov 1998 13:57:05 -0800	[thread overview]
Message-ID: <9811251357.ZM24157@xtp.engr.sgi.com> (raw)
In-Reply-To: <pjlahaie@atlsci.com> "Re: help offered" (Nov 25,  4:38pm)

On Nov 25,  4:38pm, <pjlahaie@atlsci.com> wrote:
> Subject: Re: help offered
> On Wed, 25 Nov 1998, Greg Chesson wrote:
>
> > But the memory subsystem is ccNUMA.  That means any channel in the system
> > can read/write any memory in the system.  With io buffers that comprise
> > multiple pages, and with the pages of the buffer located on several
different
> > memory controllers, multiple io channels can burst (in parallel) to the
> > "array" of pages that comprise the buffer.
>
>     Except some of this has to go through the CrayLink.  The memory you

there are 8 IO links in the example I gave plus numerous Craylinks  -
I think it's 16 for the example.  The bandwidth definitely does not go
down a single link.

> are "bursting" to is not on the same node.  Therefore, if you have a
> dual-threaded application that runs over the data, at most the max
> bandwidth is 1.6GB/s (seeing as it's advantagous to spread your code to

the application in the example is single-threaded.
Lot's of people just want a bigpipe and a single file descriptor.

> two nodes and split the memory between them).  If you application can make
> use of all processors on that box, then you get the full bandwidth.  The

a single-thread app can easily malloc pages from all the processor slots
on the box.  Can't do that in a cluster or a shared-nothing machine.
You can think of processor slots as just extra memory controllers.
For some applications we ship with "sparse" processor nodes for just
this purpose.

> most any single processor in that Origin can handle is 800MB/s and if it
> needs to get that data, eventually that data is shoveled through the
> CrayLink (and hopefully is gets migrated there).  Is there anything flawed
> with this reasoning?

The single processor limit is set by the memory controller bandwidth.
It can peak at over 600 MB/s, but 500 MB/s is a good number for sustained
random access ops.
>
>     I don't see why it cannot be done.  The page-cache/file system buffer
> cache are supposed to be merged.  If you mmap that data, you should just
> get a pte pointing to that area in the page cache.

ok.

>
>     But that bandwidth isn't single node bandwidth.  No single node can do
> 4GB/s.  All nodes need to use their local memory to achieve max bandwidth.
>

we do make systems where single node bandwidth is many gigabytes/sec.
They're called vector supercomputers.

The max amount of memory on a motherboard is 4 GBytes, I think.
In order to get a bigger memory, more processors must be added.
Do you want to criticize that, too?

The "beauty" of the ccNUMA memory architeture is that by using off-the-shelf
memory circuits, both bandwidth and capacity can be aggregated in a modular
way and still be mapped into a coherent virtual address space.

g



-- 
Greg Chesson

WARNING: multiple messages have this Message-ID (diff)
From: "Greg Chesson" <greg@xtp.engr.sgi.com>
To: pjlahaie@atlsci.com
Cc: Ariel Faigon <ariel@cthulhu.engr.sgi.com>,
	Olivier Galibert <galibert@pobox.com>,
	linux@cthulhu.engr.sgi.com
Subject: Re: help offered
Date: Wed, 25 Nov 1998 13:57:05 -0800	[thread overview]
Message-ID: <9811251357.ZM24157@xtp.engr.sgi.com> (raw)
Message-ID: <19981125215705.n94lasQ0sMbQb1kVOsWGrYY1h-BY00agAoS8ppD0eQ8@z> (raw)
In-Reply-To: <pjlahaie@atlsci.com> "Re: help offered" (Nov 25,  4:38pm)

On Nov 25,  4:38pm, <pjlahaie@atlsci.com> wrote:
> Subject: Re: help offered
> On Wed, 25 Nov 1998, Greg Chesson wrote:
>
> > But the memory subsystem is ccNUMA.  That means any channel in the system
> > can read/write any memory in the system.  With io buffers that comprise
> > multiple pages, and with the pages of the buffer located on several
different
> > memory controllers, multiple io channels can burst (in parallel) to the
> > "array" of pages that comprise the buffer.
>
>     Except some of this has to go through the CrayLink.  The memory you

there are 8 IO links in the example I gave plus numerous Craylinks  -
I think it's 16 for the example.  The bandwidth definitely does not go
down a single link.

> are "bursting" to is not on the same node.  Therefore, if you have a
> dual-threaded application that runs over the data, at most the max
> bandwidth is 1.6GB/s (seeing as it's advantagous to spread your code to

the application in the example is single-threaded.
Lot's of people just want a bigpipe and a single file descriptor.

> two nodes and split the memory between them).  If you application can make
> use of all processors on that box, then you get the full bandwidth.  The

a single-thread app can easily malloc pages from all the processor slots
on the box.  Can't do that in a cluster or a shared-nothing machine.
You can think of processor slots as just extra memory controllers.
For some applications we ship with "sparse" processor nodes for just
this purpose.

> most any single processor in that Origin can handle is 800MB/s and if it
> needs to get that data, eventually that data is shoveled through the
> CrayLink (and hopefully is gets migrated there).  Is there anything flawed
> with this reasoning?

The single processor limit is set by the memory controller bandwidth.
It can peak at over 600 MB/s, but 500 MB/s is a good number for sustained
random access ops.
>
>     I don't see why it cannot be done.  The page-cache/file system buffer
> cache are supposed to be merged.  If you mmap that data, you should just
> get a pte pointing to that area in the page cache.

ok.

>
>     But that bandwidth isn't single node bandwidth.  No single node can do
> 4GB/s.  All nodes need to use their local memory to achieve max bandwidth.
>

we do make systems where single node bandwidth is many gigabytes/sec.
They're called vector supercomputers.

The max amount of memory on a motherboard is 4 GBytes, I think.
In order to get a bigger memory, more processors must be added.
Do you want to criticize that, too?

The "beauty" of the ccNUMA memory architeture is that by using off-the-shelf
memory circuits, both bandwidth and capacity can be aggregated in a modular
way and still be mapped into a coherent virtual address space.

g



-- 
Greg Chesson

  reply	other threads:[~1998-11-25 22:00 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
1998-11-24 12:27 help offered Torbjörn Gannholm
1998-11-24 20:33 ` Ariel Faigon
1998-11-25 19:49   ` Olivier Galibert
1998-11-25 19:57     ` John E. Schimmel
1998-11-25 19:57       ` John E. Schimmel
1998-11-25 20:11     ` Jeffrey Watts
1998-11-25 20:43       ` Greg Chesson
1998-11-25 20:37     ` Ariel Faigon
1998-11-25 20:37       ` Ariel Faigon
1998-11-25 20:51       ` pjlahaie
1998-11-25 21:18         ` William J. Earl
1998-11-25 21:18           ` William J. Earl
1998-11-25 21:24         ` Greg Chesson
1998-11-25 21:24           ` Greg Chesson
1998-11-25 21:38           ` pjlahaie
1998-11-25 21:57             ` Greg Chesson [this message]
1998-11-25 21:57               ` Greg Chesson
1998-11-25 22:09               ` pjlahaie
1998-11-25 22:57                 ` Greg Chesson
1998-11-25 22:57                   ` Greg Chesson
1998-11-25 22:08             ` William J. Earl
1998-11-25 22:08               ` William J. Earl
1998-11-25 22:13         ` Alex Kozlov
1998-11-25 22:15           ` pjlahaie
1998-11-25 22:25           ` William J. Earl
1998-11-25 22:25             ` William J. Earl
1998-11-25 21:46       ` Alan Cox
1998-11-25 21:04         ` Greg Chesson
1998-11-25 21:04           ` Greg Chesson
1998-11-26 12:28       ` ralf
     [not found]         ` <19981126085407.A2201@uni-koblenz.de>
1998-11-27 22:59           ` Olivier Galibert
1998-11-28  2:45             ` ralf
1998-11-26 22:17       ` Miguel de Icaza
1998-11-25 20:56     ` Greg Chesson
1998-11-25 21:12       ` Olivier Galibert

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9811251357.ZM24157@xtp.engr.sgi.com \
    --to=greg@xtp.engr.sgi.com \
    --cc=ariel@cthulhu.engr.sgi.com \
    --cc=galibert@pobox.com \
    --cc=linux@cthulhu.engr.sgi.com \
    --cc=pjlahaie@atlsci.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox