sparsemem patches in -mm

sparsemem patches in -mm

[Andrew Morton]

Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
please, if you haven't done so...

They're all in http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/

Applying order is:

http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/remove-non-discontig-use-of-pgdat-node_mem_map.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/resubmit-sparsemem-base-early_pfn_to_nid-works-before-sparse-is-initialized.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/resubmit-sparsemem-base-simple-numa-remap-space-allocator.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/resubmit-sparsemem-base-reorganize-page-flags-bit-operations.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/resubmit-sparsemem-base-teach-discontig-about-sparse-ranges.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/create-mm-kconfig-for-arch-independent-memory-options.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/make-each-arch-use-mm-kconfig.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/update-all-defconfigs-for-arch_discontigmem_enable.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/introduce-new-kconfig-option-for-numa-or-discontig.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/sparsemem-fix-minor-defaults-issue-in-mm-kconfig.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/mm-kconfig-kill-unused-arch_flatmem_disable.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/mm-kconfig-hide-memory-model-selection-menu.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/mm-kconfig-give-discontig-more-help-text.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/ppc64-kconfig-memory-models.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/generify-early_pfn_to_nid.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/generify-memory-present.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/sparsemem-memory-model.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/sparsemem-memory-model-for-i386.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/sparsemem-swiss-cheese-numa-layouts.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/sparsemem-hotplug-base.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/ppc64-add-early_pfn_to_nid.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/ppc64-add-memory-present.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/ppc64-sparsemem-memory-model.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/sparsemem-extreme.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/remove-direct-ref-to-contig_page_data-for-x86-64.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/add-x86-64-kconfig-options-for-sparsemem.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/reorganize-x86-64-numa-and-discontigmem-config-options.patch
http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/add-x86-64-specific-support-for-sparsemem.patch

Thanks.

Re: sparsemem patches in -mm

[Andi Kleen]

On Tue, Jun 21, 2005 at 02:13:52AM -0700, Andrew Morton wrote:
> 
> Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> please, if you haven't done so...

The x86-64 part is ok for me from a review standpoint.

However how much testing has been done on these yet? 
It would be good to at least boot it on EM64T and AMD boxes.

-Andi

Re: sparsemem patches in -mm

[Andrew Morton]

Andi Kleen <ak@suse.de> wrote:
>
> On Tue, Jun 21, 2005 at 02:13:52AM -0700, Andrew Morton wrote:
> > 
> > Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> > please, if you haven't done so...
> 
> The x86-64 part is ok for me from a review standpoint.

OK.

> However how much testing has been done on these yet? 
> It would be good to at least boot it on EM64T and AMD boxes.

Works OK on my emt64 box.  Lots of people have Optera, and I haven't heard
anything which sounds like it's sparsemem-related.

Re: sparsemem patches in -mm

[Bob Picco]

Andrew Morton wrote:	[Tue Jun 21 2005, 05:13:52AM EDT]
> 
> Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> please, if you haven't done so...
> 
Well ia64 SPARSEMEM hasn't been accepted yet by ia64.  I did test 
SPARSEMEM with recent -mm patches on ia64 (EXTREME) and x86_64 4 way 
Opteron (!EXTREME) with OSDL aim.

bob

Re: sparsemem patches in -mm

[Matthew Wilcox]

On Tue, Jun 21, 2005 at 02:13:52AM -0700, Andrew Morton wrote:
> 
> Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> please, if you haven't done so...
> 
> They're all in http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/

Pretty confusing.  All kinds of options added, removed, modified, put
back ...  let's merge it, then see how it looks.  Maybe we'll switch
parisc from discontigmem to sparsemem.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: sparsemem patches in -mm

[Andy Whitcroft]

Andi Kleen wrote:
> On Tue, Jun 21, 2005 at 02:13:52AM -0700, Andrew Morton wrote:
> 
>>Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
>>please, if you haven't done so...
> 
> 
> The x86-64 part is ok for me from a review standpoint.
> 
> However how much testing has been done on these yet? 
> It would be good to at least boot it on EM64T and AMD 

I've been testing this on a range of test boxes we have here, mostly x86
and ppc.  Mostly with kernbench and the like.  So far no problems have
reared their heads.

I've not been able to get much or a realistic test on amd64 box, I seem
to be struggling with combinations of scsi problems on the machine from
other changes in -mm.   The good news is that it gets far enough to
break with scsi problems and problems in sparsemem generally lead to
death much earlier than this.

-apw

Re: sparsemem patches in -mm

[Andy Whitcroft]

Matthew Wilcox wrote:
> On Tue, Jun 21, 2005 at 02:13:52AM -0700, Andrew Morton wrote:
> 
>>Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
>>please, if you haven't done so...
>>
>>They're all in http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/
> 
> 
> Pretty confusing.  All kinds of options added, removed, modified, put
> back ...  let's merge it, then see how it looks.  Maybe we'll switch
> parisc from discontigmem to sparsemem.

Would it make sense for me to take the current pile of patches in -mm
and merge the fixes into the appropriate main patches to make a
consolidated patch set?  If we want to do that I am happy to put the
effort in.

-apw

RE: sparsemem patches in -mm

[Luck, Tony]

>Well ia64 SPARSEMEM hasn't been accepted yet by ia64.  I did test 
>SPARSEMEM with recent -mm patches on ia64 (EXTREME) and x86_64 4 way 
>Opteron (!EXTREME) with OSDL aim.

I'm mostly worried about whether SGI are happy.  I saw the patches
to add the "Extreme" option ... but have only seen minimal feedback
from that.

-Tony

Re: sparsemem patches in -mm

[Dave Hansen]

On Tue, 2005-06-21 at 07:00 -0400, Bob Picco wrote:
> Andrew Morton wrote:	[Tue Jun 21 2005, 05:13:52AM EDT]
> > 
> > Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> > please, if you haven't done so...
> > 
> Well ia64 SPARSEMEM hasn't been accepted yet by ia64.  I did test 
> SPARSEMEM with recent -mm patches on ia64 (EXTREME) and x86_64 4 way 
> Opteron (!EXTREME) with OSDL aim.

I think the whole set is just about ready for mainline.  If anybody
wants a consolidated set of patches that's easier to review, that's no
problem.

I'd only suggest that we keep the "extreme" stuff in -mm for a couple
more releases.  It's a pretty clean add-on, and it has only had a very
short time in -mm.  Bob, would you agree?

-- Dave

Re: sparsemem patches in -mm

[Andi Kleen]

> I've not been able to get much or a realistic test on amd64 box, I seem
> to be struggling with combinations of scsi problems on the machine from
> other changes in -mm.   The good news is that it gets far enough to
> break with scsi problems and problems in sparsemem generally lead to
> death much earlier than this.

That was probably the memory map trips on some machines issue
That should be fixed now.

-Andi

Re: sparsemem patches in -mm

[Andy Whitcroft]

Luck, Tony wrote:
>>Well ia64 SPARSEMEM hasn't been accepted yet by ia64.  I did test 
>>SPARSEMEM with recent -mm patches on ia64 (EXTREME) and x86_64 4 way 
>>Opteron (!EXTREME) with OSDL aim.
> 
> 
> I'm mostly worried about whether SGI are happy.  I saw the patches
> to add the "Extreme" option ... but have only seen minimal feedback
> from that.

I'd like to say here that the EXTREME stuff is faily new compared to the
other parts.  I'd be comfortable to push up the basic SPARSEMEM
implementation and leave the EXTREME in -mm for the moment I think there
will be other updates to it before we are done.

That said I'd like to also point out that SPARSEMEM is currently still
an additional memory model.  Any architecture that finds benefit from it
is welcome to use it, any that isn't yet finding it a benefit is not
being forced to use it.  Yes in a perfect world I'd like to minimise the
number of non 'standard' memory models there are for simplicity of code
etc, but it is not the intent that we force everyone into the same
staight jacket here.

-apw

RE: sparsemem patches in -mm

[Luck, Tony]

>I'd only suggest that we keep the "extreme" stuff in -mm for a couple
>more releases.  It's a pretty clean add-on, and it has only had a very
>short time in -mm.  Bob, would you agree?

Without the extreme code, I'll have a hard time putting together a
practical generic config for ia64.  When you say "a couple more releases",
are you referring to "-mm" releases, or Linus releases?

-Tony

RE: sparsemem patches in -mm

[Dave Hansen]

On Tue, 2005-06-21 at 09:09 -0700, Luck, Tony wrote:
> >I'd only suggest that we keep the "extreme" stuff in -mm for a couple
> >more releases.  It's a pretty clean add-on, and it has only had a very
> >short time in -mm.  Bob, would you agree?
> 
> Without the extreme code, I'll have a hard time putting together a
> practical generic config for ia64.  When you say "a couple more releases",
> are you referring to "-mm" releases, or Linus releases?

Just -mm.  I think it's only been in one -mm release so far.  The rest
of sparsemem has been in -mm since early May.

I have memory hotplug working on top of the extreme patches as well now,
which required a few more tweaks.  That kind of tweaking needs to be
completed by the time it's submitted to mainline, and it's just not
quite there yet.  I'm sure it will be soon.

-- Dave

Re: sparsemem patches in -mm

[Bob Picco]

Dave Hansen wrote:	[Tue Jun 21 2005, 11:57:56AM EDT]
> On Tue, 2005-06-21 at 07:00 -0400, Bob Picco wrote:
> > Andrew Morton wrote:	[Tue Jun 21 2005, 05:13:52AM EDT]
> > > 
> > > Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> > > please, if you haven't done so...
> > > 
> > Well ia64 SPARSEMEM hasn't been accepted yet by ia64.  I did test 
> > SPARSEMEM with recent -mm patches on ia64 (EXTREME) and x86_64 4 way 
> > Opteron (!EXTREME) with OSDL aim.
> 
> I think the whole set is just about ready for mainline.  If anybody
> wants a consolidated set of patches that's easier to review, that's no
> problem.
> 
> I'd only suggest that we keep the "extreme" stuff in -mm for a couple
> more releases.  It's a pretty clean add-on, and it has only had a very
> short time in -mm.  Bob, would you agree?
> 
> -- Dave
> 
I agree.

bob

Re: sparsemem patches in -mm

[Andrew Morton]

Andy Whitcroft <apw@shadowen.org> wrote:
>
> Matthew Wilcox wrote:
> > On Tue, Jun 21, 2005 at 02:13:52AM -0700, Andrew Morton wrote:
> > 
> >>Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> >>please, if you haven't done so...
> >>
> >>They're all in http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/
> > 
> > 
> > Pretty confusing.  All kinds of options added, removed, modified, put
> > back ...  let's merge it, then see how it looks.  Maybe we'll switch
> > parisc from discontigmem to sparsemem.
> 
> Would it make sense for me to take the current pile of patches in -mm
> and merge the fixes into the appropriate main patches to make a
> consolidated patch set?  If we want to do that I am happy to put the
> effort in.

I've already done some of that, but I need to do another pass.

<looks>

The patches I have there seem fairly logical, although there's a lot of
Kconfig futzing going on.  Do you spy any patches which should be collapsed
together there?  If so, which?

Re: sparsemem patches in -mm

[Dave Hansen]

On Tue, 2005-06-21 at 13:01 -0700, Andrew Morton wrote: 
> The patches I have there seem fairly logical, although there's a lot of
> Kconfig futzing going on.  Do you spy any patches which should be collapsed
> together there?  If so, which?

I've broken the mm/Kconfig ones up into four parts, each using the title
of the first one in the set that was combined.  There was a bit of
unnecessary context that required rediffing.  

The resultant four patches can be found here:
http://sr71.net/~dave/sparsemem/all-condensed/

I'd be happy to post these to LKML or linux-arch if anyone wants to see
them in normal emailed patch form.

order:
create-mm-kconfig-for-arch-independent-memory-options.patch
make-each-arch-use-mm-kconfig.patch
update-all-defconfigs-for-arch_discontigmem_enable.patch
introduce-new-kconfig-option-for-numa-or-discontig.patch

Applied, they produce the exact same output as the source nine patches
from 2.6.12-mm1:
> create-mm-kconfig-for-arch-independent-memory-options.patch
> make-each-arch-use-mm-kconfig.patch
> make-each-arch-use-mm-kconfig-fix.patch
> update-all-defconfigs-for-arch_discontigmem_enable.patch
> introduce-new-kconfig-option-for-numa-or-discontig.patch
> sparsemem-fix-minor-defaults-issue-in-mm-kconfig.patch
> mm-kconfig-kill-unused-arch_flatmem_disable.patch
> mm-kconfig-hide-memory-model-selection-menu.patch
> mm-kconfig-give-discontig-more-help-text.patch

Into this order, in these groups:

create-mm-kconfig-for-arch-independent-memory-options.patch
sparsemem-fix-minor-defaults-issue-in-mm-kconfig.patch
mm-kconfig-hide-memory-model-selection-menu.patch
mm-kconfig-give-discontig-more-help-text.patch
#
make-each-arch-use-mm-kconfig.patch
mm-kconfig-kill-unused-arch_flatmem_disable.patch
make-each-arch-use-mm-kconfig-fix.patch
# 
update-all-defconfigs-for-arch_discontigmem_enable.patch
#
introduce-new-kconfig-option-for-numa-or-discontig.patch


-- Dave

Re: sparsemem patches in -mm

[David S. Miller]

From: Andrew Morton <akpm@osdl.org>
Date: Tue, 21 Jun 2005 02:13:52 -0700

> http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/

The only thing that sticks out to me in this patch is that
sparc64 uses a field starting at bit 24 in page->flags to
determine the cpu which needs a D-cache flush for a page.

I don't think the page->flags rework makes things any different
than before, but it's something to keep in mind.

Re: sparsemem patches in -mm

[Andrew Morton]

"David S. Miller" <davem@davemloft.net> wrote:
>
> From: Andrew Morton <akpm@osdl.org>
> Date: Tue, 21 Jun 2005 02:13:52 -0700
> 
> > http://www.zip.com.au/~akpm/linux/patches/stuff/sparsemem/
> 
> The only thing that sticks out to me in this patch is that
> sparc64 uses a field starting at bit 24 in page->flags to
> determine the cpu which needs a D-cache flush for a page.

<looks>

That's a bit hacky.  Doesn't it conflict with the way in which we stuff the
page's zone index into the top of page->flags?  I guess with a small number
of zones and a smallish number of CPUs you got lucky.

It'd be safer to use bit 32.

> I don't think the page->flags rework makes things any different
> than before, but it's something to keep in mind.

Think so.

Re: sparsemem patches in -mm

[Roman Zippel]

Hi,

Andrew Morton wrote:

> Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> please, if you haven't done so...

They would need a bit more work to be usable on m68k. I have a basic
patch to use DISCONTIGMEM for m68k. There I have the problem that I have
to deal with almost random memory configurations. So converting a
virtual/physical address or pfn into a section number can't use a
compile time constant, instead an offset/shift is calculated at boot
time and patched into the kernel.

bye, Roman

Re: sparsemem patches in -mm

[Dave Hansen]

On Fri, 2005-06-24 at 00:48 +0200, Roman Zippel wrote:
> Andrew Morton wrote:
> > Could I ask the arch maintainers to review the `sparsemem' patches from -mm,
> > please, if you haven't done so...
> 
> They would need a bit more work to be usable on m68k. I have a basic
> patch to use DISCONTIGMEM for m68k. There I have the problem that I have
> to deal with almost random memory configurations. So converting a
> virtual/physical address or pfn into a section number can't use a
> compile time constant, instead an offset/shift is calculated at boot
> time and patched into the kernel.

As far as the shift, you can simply use the smallest possible size as
the SECTION_SIZE.

Nothing I can think of with sparsemem requires you to have the kernel be
at pfn 0.  Unless you start at some completely huge address or have an
exceedingly low SECTION_SIZE, everything should fit.  Remember,
sparsemem doesn't deal with virt/phys layout, just converting pfns to
and from pages.  

If you want to mail me, or take this to your arch's list, we can
probably discuss more of the specifics for your arch.

-- Dave

Re: sparsemem patches in -mm

[Roman Zippel]

Hi,

On Thu, 23 Jun 2005, Dave Hansen wrote:

> As far as the shift, you can simply use the smallest possible size as
> the SECTION_SIZE.

That would waste too much space. 6 bits should be enough. The problem is I 
don't need the top most bits. Patching the shift later into the kernel has 
no real disadvantage at runtime compared to a compile time constant.

bye, Roman

Re: sparsemem patches in -mm

[David S. Miller]

From: Andrew Morton <akpm@osdl.org>
Date: Tue, 21 Jun 2005 23:00:24 -0700

[ Old email, something I want to bring closure to :-) ]

> "David S. Miller" <davem@davemloft.net> wrote:
> > The only thing that sticks out to me in this patch is that
> > sparc64 uses a field starting at bit 24 in page->flags to
> > determine the cpu which needs a D-cache flush for a page.
> 
> <looks>
> 
> That's a bit hacky.  Doesn't it conflict with the way in which we stuff the
> page's zone index into the top of page->flags?  I guess with a small number
> of zones and a smallish number of CPUs you got lucky.
> 
> It'd be safer to use bit 32.

I disagree.  In fact, using bit 24 is the current optimal place.

When page->flags is 64-bit, FLAGS_RESERVED is 32, which means
that all of this node, zone, section stuff will be stored in
the top 32-bits of the page->flags.  The highest defined flag
for the bottom 32-bits is PG_uncached which is 19.

This leaves bits 24-->31 free to sparc64 to use in this way.

It of course limits the number of cpus I support to 256
but that limitation also exists in the form of thread_info()->cpu
being a "u8" on sparc64, so not a big deal.