[Patch 0/3] Modify how per_cpu allocations are done.

[Patch 0/3] Modify how per_cpu allocations are done.

[holt]

This patchset cleans up how per_cpu reservations are done.

-- 

[Patch 0/3] Modify how per_cpu allocations are done. -V7 repost

[holt]

This patchset cleans up how per_cpu reservations are done.

-- 

[Patch 0/3] Modify how per_cpu allocations are done. -V8

[holt]

This patchset cleans up how per_cpu reservations are done.

-- 

Re: [Patch 0/3] Modify how per_cpu allocations are done. -V8

[Robin Holt]

On Mon, Mar 17, 2008 at 07:12:40AM -0500, holt@sgi.com wrote:
> This patchset cleans up how per_cpu reservations are done.

Argh, this should have been a better description and really only
and RFC instead of a patch.

I updated the patch to have the early_cpu_possible_mask in both
the contig and discontig cases.  This allows the mca stack allocator
to use that for preallocation instead of cpu_possible_map.  In the
case of !CONFIG_ACPI, the early_cpu_possible_mask = cpu_possible_mask.

I did find a tiger box (that is the same as a McKinnley SDV, correct?) to
try this on, but the serial console is messed up on that machine and it
so I will test this out when I get to the office.  If you get a chance
first, please let me know if it works.

Thanks,
Robin

Re: [Patch 0/3] Modify how per_cpu allocations are done. -V8

[Robin Holt]

On Mon, Mar 17, 2008 at 07:22:03AM -0500, Robin Holt wrote:
> On Mon, Mar 17, 2008 at 07:12:40AM -0500, holt@sgi.com wrote:
> > This patchset cleans up how per_cpu reservations are done.
> 
> Argh, this should have been a better description and really only
> and RFC instead of a patch.
> 
> I updated the patch to have the early_cpu_possible_mask in both
> the contig and discontig cases.  This allows the mca stack allocator
> to use that for preallocation instead of cpu_possible_map.  In the
> case of !CONFIG_ACPI, the early_cpu_possible_mask = cpu_possible_mask.
> 
> I did find a tiger box (that is the same as a McKinnley SDV, correct?) to
> try this on, but the serial console is messed up on that machine and it
> so I will test this out when I get to the office.  If you get a chance
> first, please let me know if it works.

The machine in question is in fairly bad disrepair.  I believe we will
finally get it booting again today, but it may not be until tomorrow.

Have you had a chance to test this revised patchset on your tiger box?

Thanks,
Robin

RE: [Patch 0/3] Modify how per_cpu allocations are done. -V8

[Luck, Tony]

> Have you had a chance to test this revised patchset on your tiger box?

Yes.  The new patchset still builds cleanly for all my test configs, and
this time it boots on my Tiger.

I'm still wondering about the direction you've taken though.  One
of the goals of this patchset is to reduce boot-time memory
allocation by making sure that we don't allocate per-cpu resources
for cpus that will never exist.  But not all systems provide
enough information to determine this reliably (and Linux doesn't
get around to looking at that information until after we need to
do these allocations) so you've added the "early_cpu_possible_map"
which in some cases has to err on the safe side in how many cpus
it thinks may ever exist ... and so we may allocate resources for
some non-existant cpus.

The existing MCA part of the code appears to be more conservative
than the code that you are adding.  It allocates for cpu0 using
bootmem_alloc, and then for other cpus on an as-needed basis using
alloc_pages_node().  The current code is very ugly ... needing
an "__init refok" function, and not dealing well with possible
allocation failures.  A clean-up is definitely needed, but can't
we still maintain the alloc-on-demand part (perhaps moving it
from being run by the new cpu itself to some pre-bring-up-code
that will be run by the cpu that is going to initiate bringing
the cpu online ... which would make the error handling path
easier).

-Tony

Re: [Patch 0/3] Modify how per_cpu allocations are done. -V8

[Robin Holt]

On Tue, Mar 18, 2008 at 10:46:59AM -0700, Luck, Tony wrote:
> > Have you had a chance to test this revised patchset on your tiger box?
> 
> Yes.  The new patchset still builds cleanly for all my test configs, and
> this time it boots on my Tiger.
> 
> I'm still wondering about the direction you've taken though.  One
> of the goals of this patchset is to reduce boot-time memory
> allocation by making sure that we don't allocate per-cpu resources
> for cpus that will never exist.  But not all systems provide
> enough information to determine this reliably (and Linux doesn't
> get around to looking at that information until after we need to
> do these allocations) so you've added the "early_cpu_possible_map"
> which in some cases has to err on the safe side in how many cpus
> it thinks may ever exist ... and so we may allocate resources for
> some non-existant cpus.

I am not sure where to go with this.  I have done some more pondering
today and now I feel further from a solution.  The code now does limit
the scope of our overallocation.  In the case where the ACPI tables
have described the cpu's numa placement, we don't overallocate at all.
In all other cases, we overshoot by far fewer.  With a defconfig kernel
on a zx1, the allocation drops from 512 per_cpu areas to 32.  I agree it
is not an ideal solution, but I fail to see a better solution.  Is there a
different table I should be walking to discover the actual number of cpus?

> The existing MCA part of the code appears to be more conservative
> than the code that you are adding.  It allocates for cpu0 using
> bootmem_alloc, and then for other cpus on an as-needed basis using
> alloc_pages_node().  The current code is very ugly ... needing
> an "__init refok" function, and not dealing well with possible
> allocation failures.  A clean-up is definitely needed, but can't
> we still maintain the alloc-on-demand part (perhaps moving it
> from being run by the new cpu itself to some pre-bring-up-code
> that will be run by the cpu that is going to initiate bringing
> the cpu online ... which would make the error handling path
> easier).

I think I am going to give up on that patch entirely.  My swag at this at
least ensured there was no way the allocation could fail, but it does
not make any of the over-allocation issues better.  Unless you feel
there is merit here, I will drop that patch.

Thanks,
Robin

Re: [Patch 0/3] Modify how per_cpu allocations are done. -V8

[Robin Holt]

Tony,

Can you give me some direction on this?  I think the first two patches
are probably in the ball park of what we should take.  The third (MCA
stack allocations) should probably be dropped.  I would like to get one
final (hopefully) set out before the merge window opens.

Thanks,
Robin

On Wed, Mar 19, 2008 at 08:40:42PM -0500, Robin Holt wrote:
> On Tue, Mar 18, 2008 at 10:46:59AM -0700, Luck, Tony wrote:
> > > Have you had a chance to test this revised patchset on your tiger box?
> > 
> > Yes.  The new patchset still builds cleanly for all my test configs, and
> > this time it boots on my Tiger.
> > 
> > I'm still wondering about the direction you've taken though.  One
> > of the goals of this patchset is to reduce boot-time memory
> > allocation by making sure that we don't allocate per-cpu resources
> > for cpus that will never exist.  But not all systems provide
> > enough information to determine this reliably (and Linux doesn't
> > get around to looking at that information until after we need to
> > do these allocations) so you've added the "early_cpu_possible_map"
> > which in some cases has to err on the safe side in how many cpus
> > it thinks may ever exist ... and so we may allocate resources for
> > some non-existant cpus.
> 
> I am not sure where to go with this.  I have done some more pondering
> today and now I feel further from a solution.  The code now does limit
> the scope of our overallocation.  In the case where the ACPI tables
> have described the cpu's numa placement, we don't overallocate at all.
> In all other cases, we overshoot by far fewer.  With a defconfig kernel
> on a zx1, the allocation drops from 512 per_cpu areas to 32.  I agree it
> is not an ideal solution, but I fail to see a better solution.  Is there a
> different table I should be walking to discover the actual number of cpus?
> 
> > The existing MCA part of the code appears to be more conservative
> > than the code that you are adding.  It allocates for cpu0 using
> > bootmem_alloc, and then for other cpus on an as-needed basis using
> > alloc_pages_node().  The current code is very ugly ... needing
> > an "__init refok" function, and not dealing well with possible
> > allocation failures.  A clean-up is definitely needed, but can't
> > we still maintain the alloc-on-demand part (perhaps moving it
> > from being run by the new cpu itself to some pre-bring-up-code
> > that will be run by the cpu that is going to initiate bringing
> > the cpu online ... which would make the error handling path
> > easier).
> 
> I think I am going to give up on that patch entirely.  My swag at this at
> least ensured there was no way the allocation could fail, but it does
> not make any of the over-allocation issues better.  Unless you feel
> there is merit here, I will drop that patch.
> 
> Thanks,
> Robin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [Patch 0/3] Modify how per_cpu allocations are done. -V8

[Luck, Tony]

> Can you give me some direction on this?  I think the first two patches
> are probably in the ball park of what we should take.  The third (MCA
> stack allocations) should probably be dropped.  I would like to get one
> final (hopefully) set out before the merge window opens.

Yes.  The first two patched will be good to have in the next merge.
The MCA allocation one felt like we moved the furniture around a
bit, but it didn't look like a real improvement.

If you repost the first two, then it will save me digging back in the
archives.

-Tony