[PATCH 1/6] xen: Add NUMA support to Xen

[PATCH 1/6] xen: Add NUMA support to Xen

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

The attached tarball contains a script that will generate a patch that
integrates NUMA support from Linux into Xen.

The integration support borrows heavily from linux/arch/i386/srat.c for
ACPI SRAT table parsing rather than using the more heavyweight
ACPI_NUMA.  The SRAT table code has been tested on x86_32 and x86_64 and
works with both 32-bit and 64-bit systems.

The code parses the SRAT table for memory and processor affinity and
creates the proper node to cpu mapping.  We define a new memory
structure, node_memory_chunk_t, which is based on the same structure in
linux, but tweaked for Xen.  This structure is used to keep track of
the discontiguous chunks of memory in the system and the node to which
they belong.

We bring in Linux macros and methods for node function like cpu_to_node,
num_online_nodes, node_to_cpumask, etc.

NUMA support in Xen is a compile-time option via the build system.
Passing numa=y will enable all of the NUMA functionality.  We have
tested the code with numa=y and numa=n.  

To create the first numa patch, unpack the attached tarball, numa.tgz,
and execute numa.sh.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[-- Attachment #2: numa.tgz --]
[-- Type: application/x-gtar, Size: 11681 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH 1/6] xen: Add NUMA support to Xen

[Keir Fraser]

On 1 May 2006, at 22:56, Ryan Harper wrote:

> NUMA support in Xen is a compile-time option via the build system.
> Passing numa=y will enable all of the NUMA functionality.  We have
> tested the code with numa=y and numa=n.

I don't want to add a compile option for this -- I want NUMA enabled 
all the time and to add insignificant overhead on non-NUMA (or 
small-NUMA like AMD64) systems. In fact, there's no reason for it to 
add significant overhead in any situation really.

  -- Keir

Re: [PATCH 1/6] xen: Add NUMA support to Xen

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-05-02 09:16]:
> 
> On 1 May 2006, at 22:56, Ryan Harper wrote:
> 
> >NUMA support in Xen is a compile-time option via the build system.
> >Passing numa=y will enable all of the NUMA functionality.  We have
> >tested the code with numa=y and numa=n.
> 
> I don't want to add a compile option for this -- I want NUMA enabled 
> all the time and to add insignificant overhead on non-NUMA (or 
> small-NUMA like AMD64) systems. In fact, there's no reason for it to 
> add significant overhead in any situation really.

OK, agreed.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-05-02 09:16]:
> 
> On 1 May 2006, at 22:56, Ryan Harper wrote:
> 
> >NUMA support in Xen is a compile-time option via the build system.
> >Passing numa=y will enable all of the NUMA functionality.  We have
> >tested the code with numa=y and numa=n.
> 
> I don't want to add a compile option for this -- I want NUMA enabled 
> all the time and to add insignificant overhead on non-NUMA (or 
> small-NUMA like AMD64) systems. In fact, there's no reason for it to 
> add significant overhead in any situation really.
> 
>  -- Keir

Respun with no build-time option.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com


[-- Attachment #2: numa.tgz --]
[-- Type: application/x-gtar, Size: 11369 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 12 May 2006, at 16:12, Ryan Harper wrote:

>> I don't want to add a compile option for this -- I want NUMA enabled
>> all the time and to add insignificant overhead on non-NUMA (or
>> small-NUMA like AMD64) systems. In fact, there's no reason for it to
>> add significant overhead in any situation really.
>>
>>  -- Keir
>
> Respun with no build-time option.

The obvious file to pick on in this patchset is arch/x86/numa.c, 
derived from Linux's srat.c, but I expect the points can be applied 
more generally:
  1. You re-indented. Normally a good thing but not for copies of Linux 
source files. Please edit them and maintain them in Linux style (inc. 
hard tabs) as it makes it easier to sync with Linux updates.
  2. Pointless EXPORT_SYMBOL() lines added. I can fully understand 
keeping existing ones to make the diff smaller, but *why* would you add 
new ones?
  3. You rename variables: the variable 'cpu_affinity' in 
parse_cpu_affinity_structure() has become 'ca'. Now, I prefer your 
shorter name particularly since you add a bunch of code to that 
function, but it will make forward porting to new Linux source versions 
harder than it needs to be. Please don't do it.

Rather than taking Linux's srat.c and turning it into a more generic 
'numa.c' file for Xen, perhaps it makes sense to define arch/x86/numa/ 
and stick srat.c in that subdir, and then add new file(s) in there for 
non-srat-related stuff?

  -- Keir

Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Andi Kleen]

Keir Fraser <Keir.Fraser@cl.cam.ac.uk> writes:

>   1. You re-indented. Normally a good thing but not for copies of
> Linux source files. Please edit them and maintain them in Linux style
> (inc. hard tabs) as it makes it easier to sync with Linux updates.

What Linux updates? The i386 NUMA support is basically dead and
bitrots slowly (except perhaps on NUMAQ) Every half a year someone
uses it and finds it doesn't work, maybe even fixes it and then it
breaks shortly after again. That's because it isn't used and tested in
any significant way. No distribution I'm aware of uses it either.

If you want maintained NUMA it would be much better to base on the x86-64
NUMA code.

-Andi

Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 15 May 2006, at 17:27, Andi Kleen wrote:

>>   1. You re-indented. Normally a good thing but not for copies of
>> Linux source files. Please edit them and maintain them in Linux style
>> (inc. hard tabs) as it makes it easier to sync with Linux updates.
>
> What Linux updates? The i386 NUMA support is basically dead and
> bitrots slowly (except perhaps on NUMAQ) Every half a year someone
> uses it and finds it doesn't work, maybe even fixes it and then it
> breaks shortly after again. That's because it isn't used and tested in
> any significant way. No distribution I'm aware of uses it either.
>
> If you want maintained NUMA it would be much better to base on the 
> x86-64
> NUMA code.

Sounds like an excellent plan to me: the x86/64 code looks cleaner too. 
Let's go with the x86/64 code then, unless there are any objections 
from the IBM folks?

Of course that will mean that stuff that has 'leaked out' from i386's 
srat.c (node_memory_chunk!) will definitely need to be got rid of, but 
that needed cleaning up anyway.

  -- Keir

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-05-15 11:55]:
> 
> On 15 May 2006, at 17:27, Andi Kleen wrote:
> 
> >>  1. You re-indented. Normally a good thing but not for copies of
> >>Linux source files. Please edit them and maintain them in Linux style
> >>(inc. hard tabs) as it makes it easier to sync with Linux updates.
> >
> >What Linux updates? The i386 NUMA support is basically dead and
> >bitrots slowly (except perhaps on NUMAQ) Every half a year someone
> >uses it and finds it doesn't work, maybe even fixes it and then it
> >breaks shortly after again. That's because it isn't used and tested in
> >any significant way. No distribution I'm aware of uses it either.
> >
> >If you want maintained NUMA it would be much better to base on the 
> >x86-64
> >NUMA code.
> 
> Sounds like an excellent plan to me: the x86/64 code looks cleaner too. 
> Let's go with the x86/64 code then, unless there are any objections 
> from the IBM folks?

When I looked at the x86_64 NUMA code, I noticed that SRAT parsing was
done by ACPI_NUMA (drivers/acpi/numa.c).  That code requires more ACPI
support than Xen currently has available.  Rather than pulling in all of
that ACPI, I chose to start with the i386 SRAT table parser which was
sufficient for both i386 and x86_64.  I would think that using the
modified i386 srat.c for parser gets the job done without bringing in
all of the support needed to compile drivers/acpi/numa.c in Xen.

How about putting the two together: use the i386 SRAT parser and the
x86_64 srat.c for structures and initialization?

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 15 May 2006, at 18:11, Ryan Harper wrote:

> When I looked at the x86_64 NUMA code, I noticed that SRAT parsing was
> done by ACPI_NUMA (drivers/acpi/numa.c).  That code requires more ACPI
> support than Xen currently has available.  Rather than pulling in all 
> of
> that ACPI, I chose to start with the i386 SRAT table parser which was
> sufficient for both i386 and x86_64.  I would think that using the
> modified i386 srat.c for parser gets the job done without bringing in
> all of the support needed to compile drivers/acpi/numa.c in Xen.
>
> How about putting the two together: use the i386 SRAT parser and the
> x86_64 srat.c for structures and initialization?

I guess it depends how cleanly it can be done. Can we use the x86/64 
logic pretty much unmodified, but then pull in bits of the i386 parsing 
code as functions that the x86/64 code calls rather than the full-blown 
acpi routines it would usually call into? Or would it be possible to 
pull in drivers/acpi/numa.c without also having to pull in all 80kloc+ 
of acpi crap? This probably needs some thought and/or experimentation!

  -- Keir

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 15 May 2006, at 18:17, Keir Fraser wrote:

>> When I looked at the x86_64 NUMA code, I noticed that SRAT parsing was
>> done by ACPI_NUMA (drivers/acpi/numa.c).  That code requires more ACPI
>> support than Xen currently has available.  Rather than pulling in all 
>> of
>> that ACPI, I chose to start with the i386 SRAT table parser which was
>> sufficient for both i386 and x86_64.  I would think that using the
>> modified i386 srat.c for parser gets the job done without bringing in
>> all of the support needed to compile drivers/acpi/numa.c in Xen.
>>
>> How about putting the two together: use the i386 SRAT parser and the
>> x86_64 srat.c for structures and initialization?
>
> I guess it depends how cleanly it can be done. Can we use the x86/64 
> logic pretty much unmodified, but then pull in bits of the i386 
> parsing code as functions that the x86/64 code calls rather than the 
> full-blown acpi routines it would usually call into? Or would it be 
> possible to pull in drivers/acpi/numa.c without also having to pull in 
> all 80kloc+ of acpi crap? This probably needs some thought and/or 
> experimentation!

Option (b) looks plausible actually. After all, SRAT and SLIT are both 
static tables so they shouldn't need the full AML machinery. afaics 
acpi/numa.c mostly makes use of the acpi_table_parse*() family of 
functions, which we do (to some extent) already support in Xen.

I'd go for pulling in the acpi/numa.c file and see how far you can run 
with that.

  -- Keir

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Andi Kleen]

On Monday 15 May 2006 19:17, Keir Fraser wrote:
> 
> On 15 May 2006, at 18:11, Ryan Harper wrote:
> 
> > When I looked at the x86_64 NUMA code, I noticed that SRAT parsing was
> > done by ACPI_NUMA (drivers/acpi/numa.c).  That code requires more ACPI
> > support than Xen currently has available.  Rather than pulling in all 
> > of
> > that ACPI, I chose to start with the i386 SRAT table parser which was
> > sufficient for both i386 and x86_64.  I would think that using the
> > modified i386 srat.c for parser gets the job done without bringing in
> > all of the support needed to compile drivers/acpi/numa.c in Xen.
> >
> > How about putting the two together: use the i386 SRAT parser and the
> > x86_64 srat.c for structures and initialization?
> 
> I guess it depends how cleanly it can be done. Can we use the x86/64 
> logic pretty much unmodified, but then pull in bits of the i386 parsing 
> code as functions that the x86/64 code calls rather than the full-blown 
> acpi routines it would usually call into? Or would it be possible to 
> pull in drivers/acpi/numa.c 

Yes it should be. numa parsing doesn't need the interpreter. It only
reads a few tables from memory. You could probably even recode it easily to just
call the callbacks in srat.c after looking for the tables in memory.

Bigger issue might be to make it fully 32bit clean. But probably just a few
long -> long long should be enough. 

One reason the i386 NUMA code is so ugly that limited 32bit address spaces 
and NUMA really don't love each other regarding ZONE_NORMAL handling.
The i386 code has some nasty hacks to work around that, like these
memory chunks. The result was always poor of course because NUMA
essentially didn't apply to any kernel objects.

I guess in Xen that might be a bit easier from simpler zone setup (?) 

Or if all fails limit NUMA again to 
64bit hypervisors where it is much cleaner (AFAIK except for the old x440 
and NUMAQ everything  x86 doing NUMA is 64bit  capable) 

-Andi

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-05-15 12:18]:
> 
> On 15 May 2006, at 18:11, Ryan Harper wrote:
> 
> >When I looked at the x86_64 NUMA code, I noticed that SRAT parsing was
> >done by ACPI_NUMA (drivers/acpi/numa.c).  That code requires more ACPI
> >support than Xen currently has available.  Rather than pulling in all 
> >of
> >that ACPI, I chose to start with the i386 SRAT table parser which was
> >sufficient for both i386 and x86_64.  I would think that using the
> >modified i386 srat.c for parser gets the job done without bringing in
> >all of the support needed to compile drivers/acpi/numa.c in Xen.
> >
> >How about putting the two together: use the i386 SRAT parser and the
> >x86_64 srat.c for structures and initialization?
> 
> I guess it depends how cleanly it can be done. Can we use the x86/64 
> logic pretty much unmodified, but then pull in bits of the i386 parsing 
> code as functions that the x86/64 code calls rather than the full-blown 
> acpi routines it would usually call into? Or would it be possible to 
> pull in drivers/acpi/numa.c without also having to pull in all 80kloc+ 
> of acpi crap? This probably needs some thought and/or experimentation!

Well, I *tried* to use drivers/acpi/numa.c first, and I quit after I had
to pull in more than few headers; it was just too intertwined.  I think
using the i386 parser and calling into the x86_64 logic should be
do-able.  I'll give it a go and post back here with the results.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 15 May 2006, at 18:27, Ryan Harper wrote:

> Well, I *tried* to use drivers/acpi/numa.c first, and I quit after I 
> had
> to pull in more than few headers; it was just too intertwined.  I think
> using the i386 parser and calling into the x86_64 logic should be
> do-able.  I'll give it a go and post back here with the results.

Pulling in a few more header files (presumably from include/acpi?) 
would be okay, as long as you don't end up grabbing loads of .c files. 
But really i don't see the problem here -- drivers/acpi/numa.c hardly 
pulls in very many header files. It grabs acmacros.h and acpi_bus.h, 
both of which Xen has already.

Just take a logical look at that file: it clearly doesn't use much ACPI 
machinery and I really think it should be possible to port it down to 
Xen with not much modification. Unless you can give a concrete example 
why I'm wrong. :-)

I'd _much_ rather do that than have a franken-file of i386 and x86/64 
code.

  -- Keir

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-05-15 12:37]:
> 
> On 15 May 2006, at 18:27, Ryan Harper wrote:
> 
> >Well, I *tried* to use drivers/acpi/numa.c first, and I quit after I 
> >had
> >to pull in more than few headers; it was just too intertwined.  I think
> >using the i386 parser and calling into the x86_64 logic should be
> >do-able.  I'll give it a go and post back here with the results.
> 
> Pulling in a few more header files (presumably from include/acpi?) 
> would be okay, as long as you don't end up grabbing loads of .c files. 
> But really i don't see the problem here -- drivers/acpi/numa.c hardly 
> pulls in very many header files. It grabs acmacros.h and acpi_bus.h, 
> both of which Xen has already.
> 
> Just take a logical look at that file: it clearly doesn't use much ACPI 
> machinery and I really think it should be possible to port it down to 
> Xen with not much modification. Unless you can give a concrete example 
> why I'm wrong. :-)

You weren't.  It has been a while since I gave that a try (last summer
using 2.6.11).  Turns out that it does compile without any significant 
work (couple of defines required, but nothing that wouldn't come with 
NUMA header files).

What I'm fighting now is getting linux/arch/x86_64/mm/numa.c to build.
Some of the structures to track numa info (struct pglist_data) are
rather linux specific and include lots of structures related to Linux mm
zones, check out linux/include/linux/mmzone.h.  I attempted to avoid
bring in that in, but the pfn_to_nid/phys_to_nid macros are based on
memnodemap array.  

My current thought is to trim down struct pglist_data into something
that has the minimum information needed to populate the memnodemap
array.  I'll see how far that goes.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Andi Kleen]

 
> What I'm fighting now is getting linux/arch/x86_64/mm/numa.c to build.
> Some of the structures to track numa info (struct pglist_data) are
> rather linux specific and include lots of structures related to Linux mm
> zones, check out linux/include/linux/mmzone.h.  I attempted to avoid
> bring in that in, but the pfn_to_nid/phys_to_nid macros are based on
> memnodemap array.  


I guess you can just replace setup_node_bootmem / setup_node_zones / 
numa_free_all_bootmem with Xen specific functions. They should be the only 
functions dealing with pglists and they're relatively straight forward.

-Andi

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 15 May 2006, at 21:46, Andi Kleen wrote:

>> What I'm fighting now is getting linux/arch/x86_64/mm/numa.c to build.
>> Some of the structures to track numa info (struct pglist_data) are
>> rather linux specific and include lots of structures related to Linux 
>> mm
>> zones, check out linux/include/linux/mmzone.h.  I attempted to avoid
>> bring in that in, but the pfn_to_nid/phys_to_nid macros are based on
>> memnodemap array.
>
>
> I guess you can just replace setup_node_bootmem / setup_node_zones /
> numa_free_all_bootmem with Xen specific functions. They should be the 
> only
> functions dealing with pglists and they're relatively straight forward.

Yes, my gut feeling looking at x86_64's numa.c is that it's going to 
need some heavier surgery than srat.c. I wouldn't worry so much about 
keeping that one close to the Linux original: if we end up pulling down 
more Linux memory bookkeeping code later then we can always go back and 
sync the file more closely. Keep it as clean as possible though, 
obviously (e.g., replacing whole functions is nicer than functions that 
are a hacky halfway house between Linux and Xen, etc).

  -- Keir

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Andi Kleen]

> Yes, my gut feeling looking at x86_64's numa.c is that it's going to 
> need some heavier surgery than srat.c. I wouldn't worry so much about 
> keeping that one close to the Linux original: if we end up pulling down 
> more Linux memory bookkeeping code later then we can always go back and 
> sync the file more closely. Keep it as clean as possible though, 
> obviously (e.g., replacing whole functions is nicer than functions that 
> are a hacky halfway house between Linux and Xen, etc).

If it helps I can split these functions into smaller ones in the mainline
sources. That could isolate the pglists in only very small functions.

-Andi

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 16 May 2006, at 13:57, Andi Kleen wrote:

>> Yes, my gut feeling looking at x86_64's numa.c is that it's going to
>> need some heavier surgery than srat.c. I wouldn't worry so much about
>> keeping that one close to the Linux original: if we end up pulling 
>> down
>> more Linux memory bookkeeping code later then we can always go back 
>> and
>> sync the file more closely. Keep it as clean as possible though,
>> obviously (e.g., replacing whole functions is nicer than functions 
>> that
>> are a hacky halfway house between Linux and Xen, etc).
>
> If it helps I can split these functions into smaller ones in the 
> mainline
> sources. That could isolate the pglists in only very small functions.

Thanks: I guess we'll see how the new patch turns out. It would 
certainly be very useful if you could glance over the next revision of 
the NUMA patches -- at least the first one of the set which will be the 
one that takes code from x86/64 Linux. I'm certain you'll be able to 
pick up on details that I wouldn't spot.

  -- Keir

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 2518 bytes --]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-05-16 08:03]:
> 
> On 16 May 2006, at 13:57, Andi Kleen wrote:
> 
> >>Yes, my gut feeling looking at x86_64's numa.c is that it's going to
> >>need some heavier surgery than srat.c. I wouldn't worry so much about
> >>keeping that one close to the Linux original: if we end up pulling 
> >>down
> >>more Linux memory bookkeeping code later then we can always go back 
> >>and
> >>sync the file more closely. Keep it as clean as possible though,
> >>obviously (e.g., replacing whole functions is nicer than functions 
> >>that
> >>are a hacky halfway house between Linux and Xen, etc).
> >
> >If it helps I can split these functions into smaller ones in the 
> >mainline
> >sources. That could isolate the pglists in only very small functions.
> 
> Thanks: I guess we'll see how the new patch turns out. It would 

I've got the ACPI numa.c parser function using
linux/arch/x86_64/mm/srat.c almost entirely unmodified.
linux/arch/x86_64/mm/numa.c was gutted retainly only the calls between
itself and srat.c.  I kept the memnodemap which has a nice phys_to_nid()
function.  There is still some cleanup to do, but I'd like some feedback
on what I used and didn't.

I didn't bring in mmzone.h but I did pull a few ideas and defines from
there.  We have a simple node_data structure with start_pfn,
spanned_pages and node_id.

I attempted to follow x86_64 numa.c which stashes the structure on
node-local memory.  This was problematic for 32-bit NUMA support.  On
the opteron  2-way that I had, the starting physical address for the
second node is 80000000, and when turned into a virtual address
(PAGE_OFFSET is FF000000), the resulting va is 17F000000, which
overflows unsigned long in 32-bit.  I started to use u64's, but several
functions (like map_pages_to_xen()) only take unsigned longs.  Rather
than go through getting that function working, I think it is perfectly
fine to just have a static array.  The structure we store out there
isn't accessed in any fast path.  If the structure becomes more
complicated in the future, or someone not having the structure on
node-local memory becomes a performance issue we can revisit.

The patch is function on 32-bit and 64-bit boxes and parse the SRAT
table and fills out the node_data array.  I installed a simple
keyhandler 'u' to dump the info to check that it was function after
booting up.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[-- Attachment #2: acpi_numa.tgz --]
[-- Type: application/x-gtar, Size: 9693 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Keir Fraser]

On 19 May 2006, at 22:08, Ryan Harper wrote:

> I've got the ACPI numa.c parser function using
> linux/arch/x86_64/mm/srat.c almost entirely unmodified.
> linux/arch/x86_64/mm/numa.c was gutted retainly only the calls between
> itself and srat.c.  I kept the memnodemap which has a nice 
> phys_to_nid()
> function.  There is still some cleanup to do, but I'd like some 
> feedback
> on what I used and didn't.
>
> I didn't bring in mmzone.h but I did pull a few ideas and defines from
> there.  We have a simple node_data structure with start_pfn,
> spanned_pages and node_id.

I hope Andi will give his opinion, but one thing I spotted immediately 
is that there are plenty of unnecessary diffs against the original 
files. For example, you don't need to remove __read_mostly: we don't 
implement it (yet) but we do define that attribute to nothing so it 
doesn't hurt to keep it in the source files. And why change printk() to 
DPRINTK(): if the original author thinks it's useful enough to always 
print out that's good enough for us, and it keeps the diff smaller. 
These are just a couple of examples: there might be others. These are 
only minor and easily fixed nits, of course!

  -- Keir

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Andi Kleen]

> The patch is function on 32-bit and 64-bit boxes and parse the SRAT
> table and fills out the node_data array.  I installed a simple
> keyhandler 'u' to dump the info to check that it was function after
> booting up.

It mostly looks reasonable. As Keir noted you seem to have a lot of unnecessary
changes (printk->DPRINTK, white space etc.)

I would also suggest you keep the numa emulation. It's useful for debugging
so that developers without numa machines can reproduce numa issues.

-Andi

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

* Andi Kleen <ak@suse.de> [2006-05-22 06:02]:
> 
> > The patch is function on 32-bit and 64-bit boxes and parse the SRAT
> > table and fills out the node_data array.  I installed a simple
> > keyhandler 'u' to dump the info to check that it was function after
> > booting up.
> 
> It mostly looks reasonable. As Keir noted you seem to have a lot of unnecessary
> changes (printk->DPRINTK, white space etc.)

Cleaned up.

> 
> I would also suggest you keep the numa emulation. It's useful for debugging
> so that developers without numa machines can reproduce numa issues.

Added numa emulation back in.  I've set it up such that if we fail to
find a numa-node we fallback on to emulation of one node.  This allows
the allocator code to not be filled with CONFIG_NUMA ifdefs as Keir
wanted. 

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[-- Attachment #2: acpi_numa.tgz --]
[-- Type: application/x-gtar, Size: 8932 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

* Ryan Harper <ryanh@us.ibm.com> [2006-05-24 14:02]:
> * Andi Kleen <ak@suse.de> [2006-05-22 06:02]:
> > 
> > > The patch is function on 32-bit and 64-bit boxes and parse the SRAT
> > > table and fills out the node_data array.  I installed a simple
> > > keyhandler 'u' to dump the info to check that it was function after
> > > booting up.
> > 
> > It mostly looks reasonable. As Keir noted you seem to have a lot of unnecessary
> > changes (printk->DPRINTK, white space etc.)
> 
> Cleaned up.
> 
> > 
> > I would also suggest you keep the numa emulation. It's useful for debugging
> > so that developers without numa machines can reproduce numa issues.
> 
> Added numa emulation back in.  I've set it up such that if we fail to
> find a numa-node we fallback on to emulation of one node.  This allows
> the allocator code to not be filled with CONFIG_NUMA ifdefs as Keir
> wanted. 

This last patch didn't have a functional cpu_to_node/node_to_cpumask.
This patch fixes that up in addition to the previous cleanups.  Please
take a look.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[-- Attachment #2: acpi_numa.tgz --]
[-- Type: application/x-gtar, Size: 9554 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

* Ryan Harper <ryanh@us.ibm.com> [2006-05-31 13:06]:
> * Ryan Harper <ryanh@us.ibm.com> [2006-05-24 14:02]:
> > * Andi Kleen <ak@suse.de> [2006-05-22 06:02]:
> > > 
> > > > The patch is function on 32-bit and 64-bit boxes and parse the SRAT
> > > > table and fills out the node_data array.  I installed a simple
> > > > keyhandler 'u' to dump the info to check that it was function after
> > > > booting up.
> > > 
> > > It mostly looks reasonable. As Keir noted you seem to have a lot of unnecessary
> > > changes (printk->DPRINTK, white space etc.)
> > 
> > Cleaned up.
> > 
> > > 
> > > I would also suggest you keep the numa emulation. It's useful for debugging
> > > so that developers without numa machines can reproduce numa issues.
> > 
> > Added numa emulation back in.  I've set it up such that if we fail to
> > find a numa-node we fallback on to emulation of one node.  This allows
> > the allocator code to not be filled with CONFIG_NUMA ifdefs as Keir
> > wanted. 
> 
> This last patch didn't have a functional cpu_to_node/node_to_cpumask.
> This patch fixes that up in addition to the previous cleanups.  Please
> take a look.

I wanted to follow up and see if anyone has had a chance to look at the
latest rework (patch 1 and 2).  Wasn't sure if this had fallen off your
radar.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-05-16 02:54]:
> 
> On 15 May 2006, at 21:46, Andi Kleen wrote:
> 
> >>What I'm fighting now is getting linux/arch/x86_64/mm/numa.c to build.
> >>Some of the structures to track numa info (struct pglist_data) are
> >>rather linux specific and include lots of structures related to Linux 
> >>mm
> >>zones, check out linux/include/linux/mmzone.h.  I attempted to avoid
> >>bring in that in, but the pfn_to_nid/phys_to_nid macros are based on
> >>memnodemap array.
> >
> >
> >I guess you can just replace setup_node_bootmem / setup_node_zones /
> >numa_free_all_bootmem with Xen specific functions. They should be the 
> >only
> >functions dealing with pglists and they're relatively straight forward.
> 
> Yes, my gut feeling looking at x86_64's numa.c is that it's going to 
> need some heavier surgery than srat.c. I wouldn't worry so much about 
> keeping that one close to the Linux original: if we end up pulling down 
> more Linux memory bookkeeping code later then we can always go back and 
> sync the file more closely. Keep it as clean as possible though, 
> obviously (e.g., replacing whole functions is nicer than functions that 
> are a hacky halfway house between Linux and Xen, etc).

Is there an equivalent function in Xen for
linux/arch/x86_64/kernel/e820.c:find_e820_area() ?

I'm looking at linux/arch/x86_64/mm/numa.c:setup_node_bootmem() and as I
understand, it finds a free location in the e820 map within a node's
bounds to place the struct pglist_data which will hold node information.

Is this something we should do in Xen, or can we just have a static
array of structures which hold the node information?

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Andi Kleen]

 a hacky halfway house between Linux and Xen, etc).
> 
> Is there an equivalent function in Xen for
> linux/arch/x86_64/kernel/e820.c:find_e820_area() ?

If not it should be easy to port. Most likely you can just use 
the x86-64 one after a few s/long/u64/


> Is this something we should do in Xen, or can we just have a static
> array of structures which hold the node information?

Sooner or late you will need local per node state for the allocator.
You can do the infrastructure for that right now

-Andi

Re: Re: [PATCH 1/6][RESEND] xen: Add NUMA support to Xen

[Ryan Harper]

* Andi Kleen <ak@suse.de> [2006-05-17 14:34]:
>  a hacky halfway house between Linux and Xen, etc).
> > 
> > Is there an equivalent function in Xen for
> > linux/arch/x86_64/kernel/e820.c:find_e820_area() ?
> 
> If not it should be easy to port. Most likely you can just use 
> the x86-64 one after a few s/long/u64/

Yep.

Working bad_addr() and needing some help.  Where should I be looking to
find the bad areas that we need to avoid?  Linux's bad_add() avoids <
8000 but I don't if that holds true for Xen. Also initial_images_start 
to end looks like that should be avoided, as does the frame table.  
Anything else?


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com