Linux MIPS Architecture development
 help / color / mirror / Atom feed
From: "William J. Earl" <wje@fir.engr.sgi.com>
To: <pjlahaie@atlsci.com>
Cc: Greg Chesson <greg@xtp.engr.sgi.com>,
	Ariel Faigon <ariel@cthulhu.engr.sgi.com>,
	Olivier Galibert <galibert@pobox.com>,
	linux@cthulhu.engr.sgi.com
Subject: Re: help offered
Date: Wed, 25 Nov 1998 14:08:27 -0800	[thread overview]
Message-ID: <199811252208.OAA23040@fir.engr.sgi.com> (raw)
In-Reply-To: <Pine.LNX.4.04.9811251631410.3207-100000@elenuial.atlsci.com>

pjlahaie@atlsci.com writes:
 > On Wed, 25 Nov 1998, Greg Chesson wrote:
 > 
 > > But the memory subsystem is ccNUMA.  That means any channel in the system
 > > can read/write any memory in the system.  With io buffers that comprise
 > > multiple pages, and with the pages of the buffer located on several different
 > > memory controllers, multiple io channels can burst (in parallel) to the
 > > "array" of pages that comprise the buffer.
 > 
 >     Except some of this has to go through the CrayLink.  The memory you
 > are "bursting" to is not on the same node.  Therefore, if you have a
 > dual-threaded application that runs over the data, at most the max
 > bandwidth is 1.6GB/s (seeing as it's advantagous to spread your code to
 > two nodes and split the memory between them).  If you application can make
 > use of all processors on that box, then you get the full bandwidth.  The
 > most any single processor in that Origin can handle is 800MB/s and if it
 > needs to get that data, eventually that data is shoveled through the
 > CrayLink (and hopefully is gets migrated there).  Is there anything flawed
 > with this reasoning?

      There is not a single CrayLink.  Each router port is a CrayLink.
There are many CrayLinks if you have many nodes, so the aggregate bandwidth
scales.  If you want to do some real processing on all of the data, you
will need more processors that would be required to simply read it into
some processor cache anyway, so the node bandwidth is unlikely to be the
limiting factor.  When you add processors, you get more aggregate bandwidth.

...
 > > So, a 16-processor Origin can operate a 2 GB/s file system and use only
 > > 40-50% of its internal bandwidth.  Obviously, many many configurations
 > > of processors, channels, disks and network devices are possible.
 > 
 >     But that bandwidth isn't single node bandwidth.  No single node can do
 > 4GB/s.  All nodes need to use their local memory to achieve max bandwidth.

       Yes.  The point of the operating system is to spread the load of
processing, I/O, and networking over the complete system, not create
bottlenecks at a particular node card. 

WARNING: multiple messages have this Message-ID (diff)
From: "William J. Earl" <wje@fir.engr.sgi.com>
To: pjlahaie@atlsci.com
Cc: Greg Chesson <greg@xtp.engr.sgi.com>,
	Ariel Faigon <ariel@cthulhu.engr.sgi.com>,
	Olivier Galibert <galibert@pobox.com>,
	linux@cthulhu.engr.sgi.com
Subject: Re: help offered
Date: Wed, 25 Nov 1998 14:08:27 -0800	[thread overview]
Message-ID: <199811252208.OAA23040@fir.engr.sgi.com> (raw)
Message-ID: <19981125220827.ZBzjWA--uPOYoJMLRu_aaRDkAZ8FmHKk4nXAC7DYJAE@z> (raw)
In-Reply-To: <Pine.LNX.4.04.9811251631410.3207-100000@elenuial.atlsci.com>

pjlahaie@atlsci.com writes:
 > On Wed, 25 Nov 1998, Greg Chesson wrote:
 > 
 > > But the memory subsystem is ccNUMA.  That means any channel in the system
 > > can read/write any memory in the system.  With io buffers that comprise
 > > multiple pages, and with the pages of the buffer located on several different
 > > memory controllers, multiple io channels can burst (in parallel) to the
 > > "array" of pages that comprise the buffer.
 > 
 >     Except some of this has to go through the CrayLink.  The memory you
 > are "bursting" to is not on the same node.  Therefore, if you have a
 > dual-threaded application that runs over the data, at most the max
 > bandwidth is 1.6GB/s (seeing as it's advantagous to spread your code to
 > two nodes and split the memory between them).  If you application can make
 > use of all processors on that box, then you get the full bandwidth.  The
 > most any single processor in that Origin can handle is 800MB/s and if it
 > needs to get that data, eventually that data is shoveled through the
 > CrayLink (and hopefully is gets migrated there).  Is there anything flawed
 > with this reasoning?

      There is not a single CrayLink.  Each router port is a CrayLink.
There are many CrayLinks if you have many nodes, so the aggregate bandwidth
scales.  If you want to do some real processing on all of the data, you
will need more processors that would be required to simply read it into
some processor cache anyway, so the node bandwidth is unlikely to be the
limiting factor.  When you add processors, you get more aggregate bandwidth.

...
 > > So, a 16-processor Origin can operate a 2 GB/s file system and use only
 > > 40-50% of its internal bandwidth.  Obviously, many many configurations
 > > of processors, channels, disks and network devices are possible.
 > 
 >     But that bandwidth isn't single node bandwidth.  No single node can do
 > 4GB/s.  All nodes need to use their local memory to achieve max bandwidth.

       Yes.  The point of the operating system is to spread the load of
processing, I/O, and networking over the complete system, not create
bottlenecks at a particular node card. 

  parent reply	other threads:[~1998-11-25 22:09 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
1998-11-24 12:27 help offered Torbjörn Gannholm
1998-11-24 20:33 ` Ariel Faigon
1998-11-25 19:49   ` Olivier Galibert
1998-11-25 19:57     ` John E. Schimmel
1998-11-25 19:57       ` John E. Schimmel
1998-11-25 20:11     ` Jeffrey Watts
1998-11-25 20:43       ` Greg Chesson
1998-11-25 20:37     ` Ariel Faigon
1998-11-25 20:37       ` Ariel Faigon
1998-11-25 20:51       ` pjlahaie
1998-11-25 21:18         ` William J. Earl
1998-11-25 21:18           ` William J. Earl
1998-11-25 21:24         ` Greg Chesson
1998-11-25 21:24           ` Greg Chesson
1998-11-25 21:38           ` pjlahaie
1998-11-25 21:57             ` Greg Chesson
1998-11-25 21:57               ` Greg Chesson
1998-11-25 22:09               ` pjlahaie
1998-11-25 22:57                 ` Greg Chesson
1998-11-25 22:57                   ` Greg Chesson
1998-11-25 22:08             ` William J. Earl [this message]
1998-11-25 22:08               ` William J. Earl
1998-11-25 22:13         ` Alex Kozlov
1998-11-25 22:15           ` pjlahaie
1998-11-25 22:25           ` William J. Earl
1998-11-25 22:25             ` William J. Earl
1998-11-25 21:46       ` Alan Cox
1998-11-25 21:04         ` Greg Chesson
1998-11-25 21:04           ` Greg Chesson
1998-11-26 12:28       ` ralf
     [not found]         ` <19981126085407.A2201@uni-koblenz.de>
1998-11-27 22:59           ` Olivier Galibert
1998-11-28  2:45             ` ralf
1998-11-26 22:17       ` Miguel de Icaza
1998-11-25 20:56     ` Greg Chesson
1998-11-25 21:12       ` Olivier Galibert

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=199811252208.OAA23040@fir.engr.sgi.com \
    --to=wje@fir.engr.sgi.com \
    --cc=ariel@cthulhu.engr.sgi.com \
    --cc=galibert@pobox.com \
    --cc=greg@xtp.engr.sgi.com \
    --cc=linux@cthulhu.engr.sgi.com \
    --cc=pjlahaie@atlsci.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox