[patch 0/4] V2 ia64 SPARSEMEM

[patch 0/4] V2 ia64 SPARSEMEM

[Bob Picco]

Tony,

ChangeLog V2:
	Jesse's review input has been applied to patches 3 and 4.
	These were all cosmetic changes.

This is a series of patches which enable SPARSEMEM for ia64.  It's against
rc4-mm2.  The patches have been tested against memory configurations FLATMEM,
DISCONTIG and SPARSEMEM.  This includes NUMA simulated hardware, rx2600
and HPSIM.  An early version of ia64 with SPARSEMEM was tested by Jesse. It
would be optimal to have another test pass on SGI NUMA hardware.

Ultimately I would like to eliminate DISCONTIG and VIRTUAL_MEM_MAP. Before 
this can be accomplished, more performance comparisons between SPARSEMEM 
and DISCONTIG+VIRTUAL_MEM_MAP is required.  I did some preliminary performance
work with 2 CPU rx2600.  The results were comparable and no noticeable 
regression was evidenced.  Further work on multi node NUMA machines is
required.

thanks,

bob

Re: [patch 0/4] V2 ia64 SPARSEMEM

[Jack Steiner]

On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
...
> and HPSIM.  An early version of ia64 with SPARSEMEM was tested by Jesse. It
> would be optimal to have another test pass on SGI NUMA hardware.


I'm away from the office this week, but next week, I'll give the patch
another spin on a big SGI system.

>
>
> VIRTUAL_MEM_MAP was introduced on ia64 because of memory holes on ia64
> platforms. The mem_map of the pglist_data is a pointer to a virtual
> contiguous array of page structures in memory for a node.  To
> eliminate memory waste contiguous memory holes don't have page structures.
> The alternative would be for holes in memory to be represented by reserve
> page structures.  The VIRTUAL_MEM_MAP solutions requires ia64_pfn_valid
> and a hook in the buddy allocator to check for holes in memory.
>
> SPARSEMEM has eliminated mem_map, ia64_pfn_valid and the buddy allocator
> hook.  There is a small cost for SPARSEMEM.  Any section which has both
> memory and holes requires reserved pages for the holes.  I can see

Our systems are notorious for having large holes in memory on nodes. For example,
a node may have memory at node offsets 0, 16GB, 32GB & 48GB. At each offset, you
can have a chunk of 1 to 16GB. For example, a typical system might have memory 
at the following node offsets:
    Memory at
         0 -  4GB
	16 - 20GB
	32 - 36GB
	48 - 52GB

You are guaranteed to have memory at node offset 0. Memory may or
may not exist at the other offsets.

I haven't had a chance to look at the patch, but will page struct entries 
will be allocated for the pages in the holes (4GB-16GB, etc).



-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.

Re: [patch 0/4] V2 ia64 SPARSEMEM

[Bob Picco]

Jack Steiner wrote:	[Wed May 25 2005, 06:44:06PM EDT]
> On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
> ...
> > and HPSIM.  An early version of ia64 with SPARSEMEM was tested by Jesse. It
> > would be optimal to have another test pass on SGI NUMA hardware.
> 
> 
> I'm away from the office this week, but next week, I'll give the patch
> another spin on a big SGI system.
Great!
> 
> >
> >
> > VIRTUAL_MEM_MAP was introduced on ia64 because of memory holes on ia64
> > platforms. The mem_map of the pglist_data is a pointer to a virtual
> > contiguous array of page structures in memory for a node.  To
> > eliminate memory waste contiguous memory holes don't have page structures.
> > The alternative would be for holes in memory to be represented by reserve
> > page structures.  The VIRTUAL_MEM_MAP solutions requires ia64_pfn_valid
> > and a hook in the buddy allocator to check for holes in memory.
> >
> > SPARSEMEM has eliminated mem_map, ia64_pfn_valid and the buddy allocator
> > hook.  There is a small cost for SPARSEMEM.  Any section which has both
> > memory and holes requires reserved pages for the holes.  I can see
> 
> Our systems are notorious for having large holes in memory on nodes. For example,
> a node may have memory at node offsets 0, 16GB, 32GB & 48GB. At each offset, you
> can have a chunk of 1 to 16GB. For example, a typical system might have memory 
> at the following node offsets:
>     Memory at
>          0 -  4GB
> 	16 - 20GB
> 	32 - 36GB
> 	48 - 52GB
> 
> You are guaranteed to have memory at node offset 0. Memory may or
> may not exist at the other offsets.
> 
> I haven't had a chance to look at the patch, but will page struct entries 
> will be allocated for the pages in the holes (4GB-16GB, etc).
No there won't be page structures allocated for regions which are holes.
The default SECTION_BITS value of 28 is for a 256Mb aligned section. So memory
not reported (via efi memory descriptor walk) in a section  won't have page 
structs. The function register_sparse_mem in discontig.c calls SPARSEMEM
to validate a memory section. Only valid sections are subject to page
structure allocation later in initialization.
> 
> 
> 
> -- 
> Thanks
> 
> Jack Steiner (steiner@sgi.com)          651-683-5302
> Principal Engineer                      SGI - Silicon Graphics, Inc.
> 
your welcome and thanks,

bob
> 

Re: [patch 0/4] V2 ia64 SPARSEMEM

[Jack Steiner]

On Wed, May 25, 2005 at 05:44:06PM -0500, Jack Steiner wrote:
> On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
> ...
> > and HPSIM.  An early version of ia64 with SPARSEMEM was tested by Jesse. It
> > would be optimal to have another test pass on SGI NUMA hardware.
> 
> 
> I'm away from the office this week, but next week, I'll give the patch
> another spin on a big SGI system.
> 

Sorry to have taken so long but I had a number of problems getting the latest 
mm kernel to run reliably on our large systems. 

All of the page fault intensive tests that I ran on a 64p showed essentially
the same performance with & w/o SPARSEMEM enabled. 

I also ran AIM7 but was unsucessful drawing any conclusions from the results.
Something in the latest tree is causing large run-to-run variations - almost 2:1
at some load points. I'm trying  to determine the cause of these variations.

I have not had a chance to pull down the latest SPARSEMEM patches. I'll
do that once I understand the cause of the run-to-run variations.

-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.

Re: [patch 0/4] V2 ia64 SPARSEMEM

[Bob Picco]

Jack Steiner wrote:	[Thu Jun 09 2005, 03:47:13PM EDT]
> On Wed, May 25, 2005 at 05:44:06PM -0500, Jack Steiner wrote:
> > On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
> > ...
> > > and HPSIM.  An early version of ia64 with SPARSEMEM was tested by Jesse. It
> > > would be optimal to have another test pass on SGI NUMA hardware.
> > 
> > 
> > I'm away from the office this week, but next week, I'll give the patch
> > another spin on a big SGI system.
> > 
> 
> Sorry to have taken so long but I had a number of problems getting the latest 
> mm kernel to run reliably on our large systems. 
> 
> All of the page fault intensive tests that I ran on a 64p showed essentially
> the same performance with & w/o SPARSEMEM enabled. 
> 
> I also ran AIM7 but was unsucessful drawing any conclusions from the results.
> Something in the latest tree is causing large run-to-run variations - almost 2:1
> at some load points. I'm trying  to determine the cause of these variations.
> 
> I have not had a chance to pull down the latest SPARSEMEM patches. I'll
> do that once I understand the cause of the run-to-run variations.
Jack,

Great. thanks.

Please hold off on fetching latest SPARSEMEM.  I'm about to post V3 of 
ia64 SPARSEMEM.  It depends on an Andy -mm patch acked by Andrew and two 
-mm patches by me not acked by Andrew.

thanks,

bob
> 
> -- 
> Thanks
> 
> Jack Steiner (steiner@sgi.com)          651-683-5302
> Principal Engineer                      SGI - Silicon Graphics, Inc.
> 
>