[Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

Removing or adding memory via the PowerPC hotplug interface shows
anomalies in the association between memory and nodes.  The code
was updated to initialize more possible nodes to make them available
to subsequent DLPAR hotplug-memory operations, even if they are not
needed at boot time.

Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c |   44 ++++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 12 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 15c2dd5..3d58c1f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -870,7 +870,7 @@ void __init dump_numa_cpu_topology(void)
 }
 
 /* Initialize NODE_DATA for a node on the local memory */
-static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
+static void setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 {
 	u64 spanned_pages = end_pfn - start_pfn;
 	const size_t nd_size = roundup(sizeof(pg_data_t), SMP_CACHE_BYTES);
@@ -878,23 +878,41 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 	void *nd;
 	int tnid;
 
-	nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
-	nd = __va(nd_pa);
+	if (!node_data[nid]) {
+		nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
+		nd = __va(nd_pa);
 
-	/* report and initialize */
-	pr_info("  NODE_DATA [mem %#010Lx-%#010Lx]\n",
-		nd_pa, nd_pa + nd_size - 1);
-	tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
-	if (tnid != nid)
-		pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
+		node_data[nid] = nd;
+		memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
+		NODE_DATA(nid)->node_id = nid;
+
+		/* report and initialize */
+		pr_info("  NODE_DATA [mem %#010Lx-%#010Lx]\n",
+			nd_pa, nd_pa + nd_size - 1);
+		tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
+		if (tnid != nid)
+			pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
+	} else {
+		nd_pa = (u64) node_data[nid];
+		nd = __va(nd_pa);
+	}
 
-	node_data[nid] = nd;
-	memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
-	NODE_DATA(nid)->node_id = nid;
 	NODE_DATA(nid)->node_start_pfn = start_pfn;
 	NODE_DATA(nid)->node_spanned_pages = spanned_pages;
 }
 
+static void setup_nodes(void)
+{
+	int i, l = 32 /* MAX_NUMNODES */;
+
+	for (i = 0; i < l; i++) {
+		if (!node_possible(i)) {
+			setup_node_data(i, 0, 0);
+			node_set(i, node_possible_map);
+		}
+	}
+}
+
 void __init initmem_init(void)
 {
 	int nid, cpu;
@@ -914,6 +932,8 @@ void __init initmem_init(void)
 	 */
 	nodes_and(node_possible_map, node_possible_map, node_online_map);
 
+	setup_nodes();
+
 	for_each_online_node(nid) {
 		unsigned long start_pfn, end_pfn;
 

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Reza Arbab]

On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>+static void setup_nodes(void)
>+{
>+	int i, l = 32 /* MAX_NUMNODES */;
>+
>+	for (i = 0; i < l; i++) {
>+		if (!node_possible(i)) {
>+			setup_node_data(i, 0, 0);
>+			node_set(i, node_possible_map);
>+		}
>+	}
>+}

This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset 
node_possible_map to only node_online_map").

Balbir, you have a patchset which reverts it. Do you think that will be 
getting merged?

http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
(see patch 3/3)

-- 
Reza Arbab

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 05/23/2017 10:52 AM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>> +static void setup_nodes(void)
>> +{
>> +    int i, l = 32 /* MAX_NUMNODES */;
>> +
>> +    for (i = 0; i < l; i++) {
>> +        if (!node_possible(i)) {
>> +            setup_node_data(i, 0, 0);
>> +            node_set(i, node_possible_map);
>> +        }
>> +    }
>> +}
> 
> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").

They may be related, but that commit is not a replacement.  The above patch ensures that
there are enough of the nodes initialized at startup to allow for memory hot-add into a
node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
recording that the node was initialized.

I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.

> 
> Balbir, you have a patchset which reverts it. Do you think that will be getting merged?
> 
> http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
> (see patch 3/3)
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Reza Arbab]

On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>On 05/23/2017 10:52 AM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>> +static void setup_nodes(void)
>>> +{
>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>> +
>>> +    for (i = 0; i < l; i++) {
>>> +        if (!node_possible(i)) {
>>> +            setup_node_data(i, 0, 0);
>>> +            node_set(i, node_possible_map);
>>> +        }
>>> +    }
>>> +}
>>
>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>
>They may be related, but that commit is not a replacement.  The above patch ensures that
>there are enough of the nodes initialized at startup to allow for memory hot-add into a
>node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>recording that the node was initialized.

Is it really necessary to preinitialize these empty nodes using 
setup_node_data()? When you do memory hotadd into a node that was not 
used at boot, the node data already gets set up by

add_memory
  add_memory_resource
    hotadd_new_pgdat
      arch_alloc_nodedata <-- allocs the pg_data_t
      ...
      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.

Removing setup_node_data() from that loop leaves only the call to 
node_set(). If 3af229f2071f (which reduces node_possible_map) was 
reverted, you wouldn't need to do that either.

>I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
>'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.

Are you sure you're looking at 3af229f2071f? It only adds one line of 
code; the reduction of node_possible_map.

-- 
Reza Arbab

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 05/23/2017 04:49 PM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>> +static void setup_nodes(void)
>>>> +{
>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>> +
>>>> +    for (i = 0; i < l; i++) {
>>>> +        if (!node_possible(i)) {
>>>> +            setup_node_data(i, 0, 0);
>>>> +            node_set(i, node_possible_map);
>>>> +        }
>>>> +    }
>>>> +}
>>>
>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>
>> They may be related, but that commit is not a replacement.  The above patch ensures that
>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>> recording that the node was initialized.
> 
> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
> 
> add_memory
>  add_memory_resource
>    hotadd_new_pgdat
>      arch_alloc_nodedata <-- allocs the pg_data_t
>      ...
>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.

I see that code now, but for some reason it did not work when I hot-added
memory.

> 
> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
> 
>> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>> which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
>> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
> 
> Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map.
> 

The 3rd file in the patch set removes,

-	nodes_and(node_possible_map, node_possible_map, node_online_map);

I need to add bits to 'node_possible_map' -- bits which may not be used
for the memory at boot, but which would be used when memory is hot-added
later.  I haven't found anything outside of the boot code that adds bits
to the 'possible' mask.

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 05/23/2017 04:49 PM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>> +static void setup_nodes(void)
>>>> +{
>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>> +
>>>> +    for (i = 0; i < l; i++) {
>>>> +        if (!node_possible(i)) {
>>>> +            setup_node_data(i, 0, 0);
>>>> +            node_set(i, node_possible_map);
>>>> +        }
>>>> +    }
>>>> +}
>>>
>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>
>> They may be related, but that commit is not a replacement.  The above patch ensures that
>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>> recording that the node was initialized.
> 
> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
> 
> add_memory
>  add_memory_resource
>    hotadd_new_pgdat
>      arch_alloc_nodedata <-- allocs the pg_data_t
>      ...
>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
> 
> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.

With or without 3af229f2071f, we would still need to add something, somewhere to add new
bits to the 'node_possible_map'.  That is not being done.

> 
>> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>> which is needed by 'numa.c' and 'workqueue.c'.  The nodemask created and updated by
>> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
> 
> Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map.
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Michael Bringmann <mwb@linux.vnet.ibm.com> writes:

> On 05/23/2017 04:49 PM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>> +static void setup_nodes(void)
>>>>> +{
>>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>>> +
>>>>> +    for (i = 0; i < l; i++) {
>>>>> +        if (!node_possible(i)) {
>>>>> +            setup_node_data(i, 0, 0);
>>>>> +            node_set(i, node_possible_map);
>>>>> +        }
>>>>> +    }
>>>>> +}
>>>>
>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>
>>> They may be related, but that commit is not a replacement.  The above patch ensures that
>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>>> recording that the node was initialized.
>> 
>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>> 
>> add_memory
>>  add_memory_resource
>>    hotadd_new_pgdat
>>      arch_alloc_nodedata <-- allocs the pg_data_t
>>      ...
>>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>> 
>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>
> With or without 3af229f2071f, we would still need to add something, somewhere to add new
> bits to the 'node_possible_map'.  That is not being done.

You mustn't add bits to the possible map after boot.

That's its purpose, to tell you what nodes could ever *possibly* exist.

cheers

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 05/24/2017 06:19 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> 
>> On 05/23/2017 04:49 PM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>>> +static void setup_nodes(void)
>>>>>> +{
>>>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>>>> +
>>>>>> +    for (i = 0; i < l; i++) {
>>>>>> +        if (!node_possible(i)) {
>>>>>> +            setup_node_data(i, 0, 0);
>>>>>> +            node_set(i, node_possible_map);
>>>>>> +        }
>>>>>> +    }
>>>>>> +}
>>>>>
>>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>>
>>>> They may be related, but that commit is not a replacement.  The above patch ensures that
>>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>>>> recording that the node was initialized.
>>>
>>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>>
>>> add_memory
>>>  add_memory_resource
>>>    hotadd_new_pgdat
>>>      arch_alloc_nodedata <-- allocs the pg_data_t
>>>      ...
>>>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>>
>>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>>
>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>> bits to the 'node_possible_map'.  That is not being done.
> 
> You mustn't add bits to the possible map after boot.
> 
> That's its purpose, to tell you what nodes could ever *possibly* exist.

The problem that I have been encountering is that the 'possible map' did *not*
show all of the possible nodes.  Rather, it showed only the nodes that were
assigned memory at boot-up.  If more memory were hot-added to the kernel, it
could be assigned into one of the nodes that were skipped at boot.  However,
nothing was updating the 'node_possible_map' correctly in the kernel memory
code.

Reza pointed out a code change in commit 3af229f2071f that has not made it into
the 4.12 checkout i.e. removing the instruction that reduces the node_possible_map.
This may well be a suitable replacement for the code that I have here, and I
will test it here next.

> 
> cheers
> 
> 
Later.

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Michael Bringmann <mwb@linux.vnet.ibm.com> writes:

> On 05/24/2017 06:19 AM, Michael Ellerman wrote:
>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>>
>>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>>> bits to the 'node_possible_map'.  That is not being done.
>> 
>> You mustn't add bits to the possible map after boot.
>> 
>> That's its purpose, to tell you what nodes could ever *possibly* exist.
>
> The problem that I have been encountering is that the 'possible map' did *not*
> show all of the possible nodes.

OK so how did that happen?

The commit message for 3af229f2071f says:

    In practice, we never see a system with 256 NUMA nodes, and in fact, we
    do not support node hotplug on power in the first place, so the nodes
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    that are online when we come up are the nodes that will be present for
    the lifetime of this kernel.

Is that no longer true?

cheers

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 05/25/2017 01:19 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> 
>> On 05/24/2017 06:19 AM, Michael Ellerman wrote:
>>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>>>
>>>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>>>> bits to the 'node_possible_map'.  That is not being done.
>>>
>>> You mustn't add bits to the possible map after boot.
>>>
>>> That's its purpose, to tell you what nodes could ever *possibly* exist.
>>
>> The problem that I have been encountering is that the 'possible map' did *not*
>> show all of the possible nodes.
> 
> OK so how did that happen?
> 
> The commit message for 3af229f2071f says:
> 
>     In practice, we never see a system with 256 NUMA nodes, and in fact, we
>     do not support node hotplug on power in the first place, so the nodes
>     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>     that are online when we come up are the nodes that will be present for
>     the lifetime of this kernel.
> 
> Is that no longer true?

Take a look at the last part of commit 3af229f2071f for file numa.c.  It undoes
a piece of code that restricts the 'node possible map', created earlier, to the
set of online nodes.  That piece of code has not made it into the mainline, at
least not into 4.12.  I am testing to verify whether it is sufficient for my
configuration now.

> 
> cheers
> 
Regards.

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Reza Arbab]

On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>The commit message for 3af229f2071f says:
>
>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>    do not support node hotplug on power in the first place, so the nodes
>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    that are online when we come up are the nodes that will be present for
>    the lifetime of this kernel.
>
>Is that no longer true?

I don't know what the reasoning behind that statement was at the time, 
but as far as I can tell, the only thing missing for node hotplug now is 
Balbir's patchset [1]. He fixes the resource issue which motivated 
3af229f2071f and reverts it.

With that set, I can instantiate a new numa node just by doing 
add_memory(nid, ...) where nid doesn't currently exist.

[1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com

-- 
Reza Arbab

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 05/25/2017 10:10 AM, Reza Arbab wrote:
> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>> The commit message for 3af229f2071f says:
>>
>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>    do not support node hotplug on power in the first place, so the nodes
>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>    that are online when we come up are the nodes that will be present for
>>    the lifetime of this kernel.
>>
>> Is that no longer true?
> 
> I don't know what the reasoning behind that statement was at the time, but as far as I can tell, the only thing missing for node hotplug now is Balbir's patchset [1]. He fixes the resource issue which motivated 3af229f2071f and reverts it.
> 
> With that set, I can instantiate a new numa node just by doing add_memory(nid, ...) where nid doesn't currently exist.
> 
> [1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
> 

Yes, the change to 'numa.c' looks to be sufficient for my needs as well.

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Balbir Singh]

On Thu, 25 May 2017 10:10:11 -0500
Reza Arbab <arbab@linux.vnet.ibm.com> wrote:

> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
> >The commit message for 3af229f2071f says:
> >
> >    In practice, we never see a system with 256 NUMA nodes, and in fact, we
> >    do not support node hotplug on power in the first place, so the nodes
> >    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >    that are online when we come up are the nodes that will be present for
> >    the lifetime of this kernel.
> >
> >Is that no longer true?  
> 
> I don't know what the reasoning behind that statement was at the time, 
> but as far as I can tell, the only thing missing for node hotplug now is 
> Balbir's patchset [1]. He fixes the resource issue which motivated 
> 3af229f2071f and reverts it.
> 
> With that set, I can instantiate a new numa node just by doing 
> add_memory(nid, ...) where nid doesn't currently exist.
> 
> [1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
> 

I guess I should try and revive that patchset. One of the suggestions of
then was to limit maximum possible nodes in firmware, but I'm double checking
to see if we can do that in a well defined manner.

Balbir Singh

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Reza Arbab <arbab@linux.vnet.ibm.com> writes:

> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>The commit message for 3af229f2071f says:
>>
>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>    do not support node hotplug on power in the first place, so the nodes
>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>    that are online when we come up are the nodes that will be present for
>>    the lifetime of this kernel.
>>
>>Is that no longer true?
>
> I don't know what the reasoning behind that statement was at the time, 
> but as far as I can tell, the only thing missing for node hotplug now is 
> Balbir's patchset [1]. He fixes the resource issue which motivated 
> 3af229f2071f and reverts it.
>
> With that set, I can instantiate a new numa node just by doing 
> add_memory(nid, ...) where nid doesn't currently exist.

But does that actually happen on any real system?

cheers

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

I am running into this problem on PowerPC systems where Balbir's patch set
was targeted.  So, yes, I do need to be able to add/enable a new numa node
during system execution in cases where more resources (memory, virtual
processors) are added to the system dynamically.

On 05/25/2017 10:46 PM, Michael Ellerman wrote:
> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
> 
>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>> The commit message for 3af229f2071f says:
>>>
>>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>    do not support node hotplug on power in the first place, so the nodes
>>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>    that are online when we come up are the nodes that will be present for
>>>    the lifetime of this kernel.
>>>
>>> Is that no longer true?
>>
>> I don't know what the reasoning behind that statement was at the time, 
>> but as far as I can tell, the only thing missing for node hotplug now is 
>> Balbir's patchset [1]. He fixes the resource issue which motivated 
>> 3af229f2071f and reverts it.
>>
>> With that set, I can instantiate a new numa node just by doing 
>> add_memory(nid, ...) where nid doesn't currently exist.
> 
> But does that actually happen on any real system?
> 
> cheers
> 
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Reza Arbab]

On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>
>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>The commit message for 3af229f2071f says:
>>>
>>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>    do not support node hotplug on power in the first place, so the nodes
>>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>    that are online when we come up are the nodes that will be present for
>>>    the lifetime of this kernel.
>>>
>>>Is that no longer true?
>>
>> I don't know what the reasoning behind that statement was at the time,
>> but as far as I can tell, the only thing missing for node hotplug now is
>> Balbir's patchset [1]. He fixes the resource issue which motivated
>> 3af229f2071f and reverts it.
>>
>> With that set, I can instantiate a new numa node just by doing
>> add_memory(nid, ...) where nid doesn't currently exist.
>
>But does that actually happen on any real system?

I don't know if anything currently tries to do this. My interest in 
having this working is so that in the future, our coherent gpu memory 
could be added as a distinct node by the device driver.

-- 
Reza Arbab

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Reza Arbab <arbab@linux.vnet.ibm.com> writes:

> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>
>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>The commit message for 3af229f2071f says:
>>>>
>>>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>>    do not support node hotplug on power in the first place, so the nodes
>>>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>    that are online when we come up are the nodes that will be present for
>>>>    the lifetime of this kernel.
>>>>
>>>>Is that no longer true?
>>>
>>> I don't know what the reasoning behind that statement was at the time,
>>> but as far as I can tell, the only thing missing for node hotplug now is
>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>> 3af229f2071f and reverts it.
>>>
>>> With that set, I can instantiate a new numa node just by doing
>>> add_memory(nid, ...) where nid doesn't currently exist.
>>
>>But does that actually happen on any real system?
>
> I don't know if anything currently tries to do this. My interest in 
> having this working is so that in the future, our coherent gpu memory 
> could be added as a distinct node by the device driver.

Sure. If/when that happens, we would hopefully still have some way to
limit the size of the possible map.

That would ideally be a firmware property that tells us the maximum
number of GPUs that might be hot-added, or we punt and cap it at some
"sane" maximum number.

But until that happens it's silly to say we can have up to 256 nodes
when in practice most of our systems have 8 or less.

So I'm still waiting for an explanation from Michael B on how he's
seeing this bug in practice.

cheers

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 05/29/2017 12:32 AM, Michael Ellerman wrote:
> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
> 
>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>
>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>> The commit message for 3af229f2071f says:
>>>>>
>>>>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>>>    do not support node hotplug on power in the first place, so the nodes
>>>>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>>    that are online when we come up are the nodes that will be present for
>>>>>    the lifetime of this kernel.
>>>>>
>>>>> Is that no longer true?
>>>>
>>>> I don't know what the reasoning behind that statement was at the time,
>>>> but as far as I can tell, the only thing missing for node hotplug now is
>>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>>> 3af229f2071f and reverts it.
>>>>
>>>> With that set, I can instantiate a new numa node just by doing
>>>> add_memory(nid, ...) where nid doesn't currently exist.
>>>
>>> But does that actually happen on any real system?
>>
>> I don't know if anything currently tries to do this. My interest in 
>> having this working is so that in the future, our coherent gpu memory 
>> could be added as a distinct node by the device driver.
> 
> Sure. If/when that happens, we would hopefully still have some way to
> limit the size of the possible map.
> 
> That would ideally be a firmware property that tells us the maximum
> number of GPUs that might be hot-added, or we punt and cap it at some
> "sane" maximum number.
> 
> But until that happens it's silly to say we can have up to 256 nodes
> when in practice most of our systems have 8 or less.
> 
> So I'm still waiting for an explanation from Michael B on how he's
> seeing this bug in practice.

I already answered this in an earlier message.  I will give an example.

* Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP
  and 10G of memory in a shared processor configuration.
* At boot time, 4 nodes are put into the possible map by the PowerPC boot
  code.
* Subsequently, the NUMA code executes and puts the 10G memory into nodes
  4 & 5.  No memory goes into Node 0.  So we now have 2 nodes in the
  node_online_map.
* The VP and its threads get assigned to Node 4.
* Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction,
     node_and(node_possible_map, node_possible_map, node_online_map);
  the content of the node_possible_map is reduced to nodes 4-5.
* Later on we hot-add 90G of memory to the system.  It tries to put the
  memory into nodes 0, 4-5, 8 based on the memory association map.  We
  should see memory put into all 4 nodes.  However, since we have reduced
  the 'node_possible_map' to only nodes 4 & 5, we can now only put memory
  into 2 of the configured nodes.

# We want to be able to put memory into all 4 nodes via hot-add operations,
  not only the nodes that 'survive' boot time initialization.  We could
  make a number of changes to ensure that all of the nodes in the initial
  configuration provided by the pHyp can be used, but this one appears to
  be the simplest, only using resources requested by the pHyp at boot --
  even if those resource are not used immediately.

> 
> cheers
> 

Regards,
Michael

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Michael Bringmann <mwb@linux.vnet.ibm.com> writes:

> On 05/29/2017 12:32 AM, Michael Ellerman wrote:
>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>> 
>>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>>
>>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>>> The commit message for 3af229f2071f says:
>>>>>>
>>>>>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>>>>    do not support node hotplug on power in the first place, so the nodes
>>>>>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>>>    that are online when we come up are the nodes that will be present for
>>>>>>    the lifetime of this kernel.
>>>>>>
>>>>>> Is that no longer true?
>>>>>
>>>>> I don't know what the reasoning behind that statement was at the time,
>>>>> but as far as I can tell, the only thing missing for node hotplug now is
>>>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>>>> 3af229f2071f and reverts it.
>>>>>
>>>>> With that set, I can instantiate a new numa node just by doing
>>>>> add_memory(nid, ...) where nid doesn't currently exist.
>>>>
>>>> But does that actually happen on any real system?
>>>
>>> I don't know if anything currently tries to do this. My interest in 
>>> having this working is so that in the future, our coherent gpu memory 
>>> could be added as a distinct node by the device driver.
>> 
>> Sure. If/when that happens, we would hopefully still have some way to
>> limit the size of the possible map.
>> 
>> That would ideally be a firmware property that tells us the maximum
>> number of GPUs that might be hot-added, or we punt and cap it at some
>> "sane" maximum number.
>> 
>> But until that happens it's silly to say we can have up to 256 nodes
>> when in practice most of our systems have 8 or less.
>> 
>> So I'm still waiting for an explanation from Michael B on how he's
>> seeing this bug in practice.
>
> I already answered this in an earlier message.

Which one? I must have missed it.

> I will give an example.
>
> * Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP
>   and 10G of memory in a shared processor configuration.
> * At boot time, 4 nodes are put into the possible map by the PowerPC boot
>   code.

I'm pretty sure we never add nodes to the possible map, it starts out
with MAX_NUMNODES possible and that's it.

Do you actually see mention of nodes 0 and 8 in the dmesg?

What does it say?

> * Subsequently, the NUMA code executes and puts the 10G memory into nodes
>   4 & 5.  No memory goes into Node 0.  So we now have 2 nodes in the
>   node_online_map.
> * The VP and its threads get assigned to Node 4.
> * Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction,
>      node_and(node_possible_map, node_possible_map, node_online_map);
>   the content of the node_possible_map is reduced to nodes 4-5.
> * Later on we hot-add 90G of memory to the system.  It tries to put the
>   memory into nodes 0, 4-5, 8 based on the memory association map.  We
>   should see memory put into all 4 nodes.  However, since we have reduced
>   the 'node_possible_map' to only nodes 4 & 5, we can now only put memory
>   into 2 of the configured nodes.

Right. So it's not that you're hot adding memory into a previously
unseen node as you implied in earlier mails.

> # We want to be able to put memory into all 4 nodes via hot-add operations,
>   not only the nodes that 'survive' boot time initialization.  We could
>   make a number of changes to ensure that all of the nodes in the initial
>   configuration provided by the pHyp can be used, but this one appears to
>   be the simplest, only using resources requested by the pHyp at boot --
>   even if those resource are not used immediately.

I don't think that's what the patch does. It just marks 32 (!?) nodes as
online. Or if you're talking about reverting 3af229f2071f that leaves
you with 256 possible nodes. Both of which are wasteful.

The right fix is to make sure any nodes which are present at boot remain
in the possible map, even if they don't have memory/CPUs assigned at
boot.

What does your device tree look like? Can you send us the output of:

 $ lsprop /proc/device-tree


cheers

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Reza Arbab]

On Thu, Jun 01, 2017 at 07:36:31PM +1000, Michael Ellerman wrote:
>I don't think that's what the patch does. It just marks 32 (!?) nodes 
>as online. Or if you're talking about reverting 3af229f2071f that 
>leaves you with 256 possible nodes. Both of which are wasteful.

To be clear, with Balbir's set the latter is no longer wasteful.

>The right fix is to make sure any nodes which are present at boot 
>remain in the possible map, even if they don't have memory/CPUs 
>assigned at boot.

I'm still hoping 3af229f2071f could indeed be reverted some day, but 
until then the following would follow your suggestion for our GPU nodes.  
What do you think?

--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -895,6 +895,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 void __init initmem_init(void)
 {
 	int nid, cpu;
+	struct device_node *dn;
 
 	max_low_pfn = memblock_end_of_DRAM() >> PAGE_SHIFT;
 	max_pfn = max_low_pfn;
@@ -911,6 +912,18 @@ void __init initmem_init(void)
 	 */
 	nodes_and(node_possible_map, node_possible_map, node_online_map);
 
+	/*
+	 * Consider an ibm,coherent-device-memory node possible. Even though
+	 * it is not online at boot, it may be hotplugged later.
+	 */
+	for_each_compatible_node(dn, NULL, "ibm,coherent-device-memory") {
+		nid = of_node_to_nid_single(dn);
+		if (nid < 0)
+			continue;
+
+		node_set(nid, node_possible_map);
+	}
+
 	for_each_online_node(nid) {
 		unsigned long start_pfn, end_pfn;
 

-- 
Reza Arbab

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Balbir Singh]

On Thu, 2017-06-01 at 16:33 -0500, Reza Arbab wrote:
> On Thu, Jun 01, 2017 at 07:36:31PM +1000, Michael Ellerman wrote:
> > I don't think that's what the patch does. It just marks 32 (!?) nodes 
> > as online. Or if you're talking about reverting 3af229f2071f that 
> > leaves you with 256 possible nodes. Both of which are wasteful.
> 
> To be clear, with Balbir's set the latter is no longer wasteful.
> 
> > The right fix is to make sure any nodes which are present at boot 
> > remain in the possible map, even if they don't have memory/CPUs 
> > assigned at boot.
> 
> I'm still hoping 3af229f2071f could indeed be reverted some day, but 
> until then the following would follow your suggestion for our GPU nodes.  
> What do you think?
> 
> --- a/arch/powerpc/mm/numa.c
> +++ b/arch/powerpc/mm/numa.c
> @@ -895,6 +895,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>  void __init initmem_init(void)
>  {
>  	int nid, cpu;
> +	struct device_node *dn;
>  
>  	max_low_pfn = memblock_end_of_DRAM() >> PAGE_SHIFT;
>  	max_pfn = max_low_pfn;
> @@ -911,6 +912,18 @@ void __init initmem_init(void)
>  	 */
>  	nodes_and(node_possible_map, node_possible_map, node_online_map);
>  
> +	/*
> +	 * Consider an ibm,coherent-device-memory node possible. Even though
> +	 * it is not online at boot, it may be hotplugged later.
> +	 */
> +	for_each_compatible_node(dn, NULL, "ibm,coherent-device-memory") {
> +		nid = of_node_to_nid_single(dn);
> +		if (nid < 0)
> +			continue;
> +
> +		node_set(nid, node_possible_map);
> +	}
> +

I think it looks reasonable, although I'd really like to set a limit
in firmware on the number of nodes and fix memcg hotplug correctly in
the medium term.

Balbir Singh.

>  	for_each_online_node(nid) {
>  		unsigned long start_pfn, end_pfn;
>  
> 

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Reza Arbab <arbab@linux.vnet.ibm.com> writes:

> On Thu, Jun 01, 2017 at 07:36:31PM +1000, Michael Ellerman wrote:
>>I don't think that's what the patch does. It just marks 32 (!?) nodes 
>>as online. Or if you're talking about reverting 3af229f2071f that 
>>leaves you with 256 possible nodes. Both of which are wasteful.
>
> To be clear, with Balbir's set the latter is no longer wasteful.

AFAIK that series has been rejected.

And even so, it only fixes one known case where having a very large
possible map is wasteful, there could be others we haven't found, or
there could be more introduced in future.

So we will always prefer to keep the possible map as small as it can be.

>>The right fix is to make sure any nodes which are present at boot 
>>remain in the possible map, even if they don't have memory/CPUs 
>>assigned at boot.
>
> I'm still hoping 3af229f2071f could indeed be reverted some day, but 
> until then the following would follow your suggestion for our GPU nodes.  
> What do you think?

Yeah we can do something like that.

We might want to come up with a generic property, rather than looking
specifically for ibm,coherent-device-memory.

And we might also need a way for skiboot to tell us "in addition to
what is in the device tree at boot, up to N more nodes could be hot
plugged".

cheers

> --- a/arch/powerpc/mm/numa.c
> +++ b/arch/powerpc/mm/numa.c
> @@ -895,6 +895,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>  void __init initmem_init(void)
>  {
>  	int nid, cpu;
> +	struct device_node *dn;
>  
>  	max_low_pfn = memblock_end_of_DRAM() >> PAGE_SHIFT;
>  	max_pfn = max_low_pfn;
> @@ -911,6 +912,18 @@ void __init initmem_init(void)
>  	 */
>  	nodes_and(node_possible_map, node_possible_map, node_online_map);
>  
> +	/*
> +	 * Consider an ibm,coherent-device-memory node possible. Even though
> +	 * it is not online at boot, it may be hotplugged later.
> +	 */
> +	for_each_compatible_node(dn, NULL, "ibm,coherent-device-memory") {
> +		nid = of_node_to_nid_single(dn);
> +		if (nid < 0)
> +			continue;
> +
> +		node_set(nid, node_possible_map);
> +	}
> +
>  	for_each_online_node(nid) {
>  		unsigned long start_pfn, end_pfn;
>  
>
> -- 
> Reza Arbab

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

[-- Attachment #1: Type: text/plain, Size: 5747 bytes --]



On 06/01/2017 04:36 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> 
>> On 05/29/2017 12:32 AM, Michael Ellerman wrote:
>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>
>>>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>>>
>>>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>>>> The commit message for 3af229f2071f says:
>>>>>>>
>>>>>>>    In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>>>>>    do not support node hotplug on power in the first place, so the nodes
>>>>>>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>>>>    that are online when we come up are the nodes that will be present for
>>>>>>>    the lifetime of this kernel.
>>>>>>>
>>>>>>> Is that no longer true?
>>>>>>
>>>>>> I don't know what the reasoning behind that statement was at the time,
>>>>>> but as far as I can tell, the only thing missing for node hotplug now is
>>>>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>>>>> 3af229f2071f and reverts it.
>>>>>>
>>>>>> With that set, I can instantiate a new numa node just by doing
>>>>>> add_memory(nid, ...) where nid doesn't currently exist.
>>>>>
>>>>> But does that actually happen on any real system?
>>>>
>>>> I don't know if anything currently tries to do this. My interest in 
>>>> having this working is so that in the future, our coherent gpu memory 
>>>> could be added as a distinct node by the device driver.
>>>
>>> Sure. If/when that happens, we would hopefully still have some way to
>>> limit the size of the possible map.
>>>
>>> That would ideally be a firmware property that tells us the maximum
>>> number of GPUs that might be hot-added, or we punt and cap it at some
>>> "sane" maximum number.
>>>
>>> But until that happens it's silly to say we can have up to 256 nodes
>>> when in practice most of our systems have 8 or less.
>>>
>>> So I'm still waiting for an explanation from Michael B on how he's
>>> seeing this bug in practice.
>>
>> I already answered this in an earlier message.
> 
> Which one? I must have missed it.
> 
>> I will give an example.
>>
>> * Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP
>>   and 10G of memory in a shared processor configuration.
>> * At boot time, 4 nodes are put into the possible map by the PowerPC boot
>>   code.
> 
> I'm pretty sure we never add nodes to the possible map, it starts out
> with MAX_NUMNODES possible and that's it.

Let me reword that.  It enables the nodes in the possible map.

> 
> Do you actually see mention of nodes 0 and 8 in the dmesg?

When the 'numa.c' code is built with debug messages, and the system was
given that configuration by pHyp, yes, I did.

> 
> What does it say?

The debug message for each core thread would be something like,

    removing cpu 64 from node 0
    adding cpu 64 to node 8

repeated for all 8 threads of the CPU, and usually with the messages
for all of the CPUs coming out intermixed on the console/dmesg log.

> 
>> * Subsequently, the NUMA code executes and puts the 10G memory into nodes
>>   4 & 5.  No memory goes into Node 0.  So we now have 2 nodes in the
>>   node_online_map.
>> * The VP and its threads get assigned to Node 4.
>> * Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction,
>>      node_and(node_possible_map, node_possible_map, node_online_map);
>>   the content of the node_possible_map is reduced to nodes 4-5.
>> * Later on we hot-add 90G of memory to the system.  It tries to put the
>>   memory into nodes 0, 4-5, 8 based on the memory association map.  We
>>   should see memory put into all 4 nodes.  However, since we have reduced
>>   the 'node_possible_map' to only nodes 4 & 5, we can now only put memory
>>   into 2 of the configured nodes.
> 
> Right. So it's not that you're hot adding memory into a previously
> unseen node as you implied in earlier mails.

In the sense that the nodes were defined in the device tree, that is correct.

In the sense that those nodes are currently deleted from node_possible_map in
'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map,
node_online_map);', the nodes are no longer available to place memory or CPU.

>> # We want to be able to put memory into all 4 nodes via hot-add operations,
>>   not only the nodes that 'survive' boot time initialization.  We could
>>   make a number of changes to ensure that all of the nodes in the initial
>>   configuration provided by the pHyp can be used, but this one appears to
>>   be the simplest, only using resources requested by the pHyp at boot --
>>   even if those resource are not used immediately.
> 
> I don't think that's what the patch does. It just marks 32 (!?) nodes as
> online. Or if you're talking about reverting 3af229f2071f that leaves
> you with 256 possible nodes. Both of which are wasteful> > The right fix is to make sure any nodes which are present at boot remain
> in the possible map, even if they don't have memory/CPUs assigned at
> boot.

Okay, I can try to insert code that extracts all of the nodes from the
ibm,associativity-lookup-arrays property and merge them with the nodes
put into the online map from the CPUs that were found previously during
boot of the powerpc code.

> What does your device tree look like? Can you send us the output of:
> 
>  $ lsprop /proc/device-tree

See attachment 'device-tree.log'.  Note though that this boot of my test
system only has 2 nodes, 0 and 2.

> 
> cheers
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: device-tree.log --]
[-- Type: text/x-log; name="device-tree.log", Size: 52468 bytes --]

ibm,partition-name
		 "ltcalpine2-lp20"
ibm,aix-diagnostics
		
ibm,drc-names    0000219e 4c4d4233 004c4d42 34004c4d
		 4235004c 4d423600 4c4d4237 004c4d42
		 38004c4d 4239004c 4d423130 004c4d42
		 3131004c 4d423132 004c4d42 3133004c
		 4d423134 004c4d42 3135004c 4d423136
		 004c4d42 3137004c 4d423138 004c4d42
		 3139004c 4d423230 004c4d42 3231004c
		 4d423232 004c4d42 3233004c 4d423234
		 [73812 bytes total]
hmc-managed?     ffffffff ffffffff
ibm,drc-types    0000219e 4d454d00 4d454d00 4d454d00
		 4d454d00 4d454d00 4d454d00 4d454d00
		 4d454d00 4d454d00 4d454d00 4d454d00
		 4d454d00 4d454d00 4d454d00 4d454d00
		 4d454d00 4d454d00 4d454d00 4d454d00
		 4d454d00 4d454d00 4d454d00 4d454d00
		 4d454d00 4d454d00 4d454d00 4d454d00
		 4d454d00 4d454d00 4d454d00 4d454d00
		 [34428 bytes total]
compatible       "IBM,8408-E8E"
ibm,extended-clock-frequency
		 00000000 00000000
ibm,max-boot-devices
		 00000005
ibm,drc-indexes  0000219e 80000002 80000003 80000004
		 80000005 80000006 80000007 80000008
		 80000009 8000000a 8000000b 8000000c
		 8000000d 8000000e 8000000f 80000010
		 80000011 80000012 80000013 80000014
		 80000015 80000016 80000017 80000018
		 80000019 8000001a 8000001b 8000001c
		 8000001d 8000001e 8000001f 80000020
		 [34428 bytes total]
ibm,max-vios-function-level
		 00000000
device_type      "chrp"
ibm,fw-bytes-per-boot-device
		 00000100 (256)
ibm,fw-net-compatibility
		 "1"
ibm,service-indicator-mode
		 00000001
ibm,model-class  "E5"
ibm,partition-performance-parameters-level
		 00000001
model            "IBM,8408-E8E"
ibm,partition-no 00000014 (20)
ibm,migratable-partition
		
#address-cells   00000002
ibm,fru-9006-deactivate
		 00000001
#size-cells      00000002
ibm,drc-power-domains
		 0000219e ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 [34428 bytes total]
ibm,fw-net-version
		 "bPF160504 00:04:36 /gsa/ausgsa-p1/01/pfw/pfw860/PF160504/obj/rios_aix_4/of"
ibm,partition-uuid
		 "205f9615-9df1-4423-afb0-60d21df83633"
ibm,plat-res-int-priorities
		 0000000c 000000e4
clock-frequency  00000000
ibm,converged-loc-codes
		
ibm,phandle      ffffffff (-1)
ibm,fault-behavior
		 00000001
ibm,platform-hardware-notification
		 00000001 504f5745 52385f30 30344230 32303100
ibm,eeh-default  00000001
linux,phandle    00c91548 (13178184)
ibm,ignore-hp-po-fails-for-dlpar
		
ibm,enable-ci64-capable
		
system-id        "IBM,0210A7ACV"
ibm,extended-address
		
name             ""
ibm,pci-full-cfg 00000001
ibm,lpar-capable
ibm,linux-le-capable
		

/proc/device-tree/ibm,serial:
compatible       "hvterm1"
device_type      "serial"
linux,phandle    00c9c140 (13222208)
used-by-rtas    
name             "ibm,serial"

/proc/device-tree/vdevice:
interrupt-ranges 000a0000 000001f3
ibm,drc-names    000001f4 55383430 382e4538 452e3130
		 41374143 562d5632 302d4330 00553834
		 30382e45 38452e31 30413741 43562d56
		 32302d43 31005538 3430382e 4538452e
		 31304137 4143562d 5632302d 43320055
		 38343038 2e453845 2e313041 37414356
		 2d563230 2d433300 55383430 382e4538
		 452e3130 41374143 562d5632 302d4334
		 [13394 bytes total]
ibm,drc-types    000001f4 534c4f54 00534c4f 5400534c
		 4f540053 4c4f5400 534c4f54 00534c4f
		 5400534c 4f540053 4c4f5400 534c4f54
		 00534c4f 5400534c 4f540053 4c4f5400
		 534c4f54 00534c4f 5400534c 4f540053
		 4c4f5400 534c4f54 00534c4f 5400534c
		 4f540053 4c4f5400 534c4f54 00534c4f
		 5400534c 4f540053 4c4f5400 534c4f54
		 [2504 bytes total]
compatible       "IBM,vdevice"
ibm,max-virtual-dma-size
		 00050000 (327680)
ibm,drc-indexes  000001f4 30000000 30000001 30000002
		 30000003 30000004 30000005 30000006
		 30000007 30000008 30000009 3000000a
		 3000000b 3000000c 3000000d 3000000e
		 3000000f 30000010 30000011 30000012
		 30000013 30000014 30000015 30000016
		 30000017 30000018 30000019 3000001a
		 3000001b 3000001c 3000001d 3000001e
		 [2004 bytes total]
device_type      "vdevice"
#interrupt-cells 00000002
#address-cells   00000001
#size-cells      00000000
ibm,drc-power-domains
		 000001f4 ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 [2004 bytes total]
ibm,reserved-virtual-addresses
		 00000fff ffd1fff0 00010000 00000fff
		 ffd2fff0 00010000 00000fff ffd3fff0
		 00010000 00000fff ffd4fff0 00010000
		 00000fff ffd5fff0 00010000 00000fff
		 ffd6fff0 00010000 00000fff ffd7fff0
		 00010000 00000fff ffd8fff0 00010000
		 00000fff ffd9fff0 00010000 00000fff
		 ffdafff0 00010000 00000fff ffdbfff0
		 [540 bytes total]
ibm,phandle      fffffffc (-4)
linux,phandle    00e0f338 (14742328)
name             "vdevice"
interrupt-controller
		

/proc/device-tree/vdevice/rtc@4001:
device_type      "rtc"
reg              00004001 (16385)
linux,phandle    00e3f240 (14938688)
ibm,fw-device-code
		 da190000
used-by-rtas    
name             "rtc"

/proc/device-tree/vdevice/IBM,sp@4000:
device_type      "IBM,sp"
reg              00004000 (16384)
linux,phandle    00e3e738 (14935864)
used-by-rtas    
name             "IBM,sp"

/proc/device-tree/vdevice/gscsi@4004:
device_type      "scsi-2"
#address-cells   00000001
#size-cells      00000000
reg              00004004 (16388)
linux,phandle    00e3fe40 (14941760)
used-by-rtas    
name             "gscsi"

/proc/device-tree/vdevice/gscsi@4004/disk:
device_type      "block"
ibm,write-supported
		
linux,phandle    00e40738 (14944056)
name             "disk"

/proc/device-tree/vdevice/v-scsi@30000078:
ibm,#vscsi-address-bits
		 00000040 (64)
ibm,#dma-size-cells
		 00000002
compatible       "IBM,v-scsi"
ibm,fw-adapter-name
		 "Virtual SCSI"
device_type      "vscsi"
ibm,my-drc-index 30000078 (805306488)
ibm,#dma-address-cells
		 00000002
interrupts       000a0078 00000000
ibm,write-blocks-supported
		
ibm,my-dma-window
		 10000078 00000000 00000000 00000000 20000000
ibm,loc-code     "U8408.E8E.10A7ACV-V20-C120-T1"
reg              30000078 (805306488)
ibm,phandle      30000078 (805306488)
linux,phandle    00e25560 (14832992)
name             "v-scsi"

/proc/device-tree/vdevice/v-scsi@30000078/disk:
device_type      "block"
ibm,write-supported
		
ibm,16byte-cdb-supported
		
linux,phandle    00e2d478 (14865528)
name             "disk"

/proc/device-tree/vdevice/v-scsi@30000078/tape:
device_type      "byte"
linux,phandle    00e2fd90 (14876048)
name             "tape"

/proc/device-tree/vdevice/nvram@4002:
device_type      "nvram"
#bytes           00003c00 (15360)
reg              00004002 (16386)
linux,phandle    00e3fb50 (14941008)
used-by-rtas    
name             "nvram"

/proc/device-tree/vdevice/l-lan@30000002:
ibm,#dma-size-cells
		 00000002
compatible       "IBM,l-lan"
mac-address      ba 68 98 2a 1b 02                              .h.*..
ibm,illan-options
		
local-mac-address
		 ba 68 98 2a 1b 02                              .h.*..
ibm,fw-adapter-name
		 "Interpartition Logical LAN"
device_type      "network"
max-frame-size   000005dc (1500)
ibm,my-drc-index 30000002 (805306370)
supported-network-types
		 "ethernet,auto,none,auto"
ibm,mac-address-filters
		 000000ff (255)
ibm,#dma-address-cells
		 00000002
interrupts       000a0002 00000000
chosen-network-type
		 "ethernet,auto,none,auto"
ibm,vserver     
address-bits     00000030 (48)
ibm,my-dma-window
		 10000002 00000000 00000000 00000000 10000000
ibm,loc-code     "U8408.E8E.10A7ACV-V20-C2-T1"
reg              30000002 (805306370)
ibm,phandle      30000002 (805306370)
linux,phandle    00e1d2b0 (14799536)
name             "l-lan"

/proc/device-tree/vdevice/vty@30000000:
compatible       "hvterm1"
device_type      "serial"
ibm,my-drc-index "0"
		 ""
		 ""
interrupts       000a0000 00000000
ibm,loc-code     "U8408.E8E.10A7ACV-V20-C0"
reg              "0"
		 ""
		 ""
ibm,phandle      "0"
		 ""
		 ""
linux,phandle    00e1b528 (14791976)
name             "vty"

/proc/device-tree/memory@0:
device_type      "memory"
ibm,associativity
		 00000004 00000000 00000001 00000002 00000002
available        00000000 001fffff 00000000 00000001
		 00000000 0021ffff 00000000 00000001
		 00000000 00bfffff 00000000 00000001
		 00000000 0a6bdbf3 00000000 0000240d
		 00000000 0cd02878 00000000 000fd788
		 00000000 0ef84b1a 00000000 0000b4e6
		 00000000 0f090000 00000000 0f9c0000
#address-cells   00000001
#size-cells      00000000
reg              00000000 00000000 00000000 20000000
ibm,phandle      fffffffa (-6)
linux,phandle    00cae9d0 (13298128)
name             "memory"

/proc/device-tree/event-sources:
interrupt-ranges 00090000 00000008
#interrupt-cells 00000002
linux,phandle    00e026e8 (14690024)
name             "event-sources"
ibm,fw-lpevent-xirr
		 00090000 (589824)
interrupt-controller
		

/proc/device-tree/event-sources/epow-events:
interrupts       00090001 00000000
linux,phandle    00e048a0 (14698656)
name             "epow-events"

/proc/device-tree/ibm,dynamic-reconfiguration-memory:
ibm,lmb-size     00000000 10000000
ibm,memory-flags-mask
		 000000ff (255)
ibm,dynamic-memory
		 0000059e 00000000 20000000 80000002
		 00000000 00000001 00000008 00000000
		 30000000 80000003 00000000 00000001
		 00000008 00000000 40000000 80000004
		 00000000 00000001 00000008 00000000
		 50000000 80000005 00000000 00000001
		 00000008 00000000 60000000 80000006
		 00000000 00000001 00000008 00000000
		 [34516 bytes total]
ibm,phandle      fffffff9 (-7)
linux,phandle    00ce70c0 (13529280)
ibm,associativity-lookup-arrays
		 00000002 00000004 00000000 00000000 00000000
		 00000000 00000000 00000001 00000002 00000002
ibm,memory-preservation-time
		 00000000
name             "ibm,dynamic-reconfiguration-memory"

/proc/device-tree/options:
oem-banner?      "false"
screen-#columns  "80"
ibm,shadow-boot-device
		 ",AIX-VDASD-IBM     IPR-0   58C1F90000000220"
ibm,fw-default-mac-address?
		 "false"
ibm,fw-force-ior "false"
real-base        "12582912"
ibm,fw-dc-select "100"
use-nvramrc?     "false"
ibm,fw-n-dbfp    "00000000"
ibm,fw-find-tape-alias
		 "false"
ibm,fw-n-ru      "Y"
ibm,fw-tty-language
		 "1"
boot-command     "boot"
ibm,fw-find-cdrom-alias
		 "false"
menu?            "false"
reboot-command   ""
ibm,last-booted  "AIX-VDASD-IBM     IPR-0   58C1F90000000220"
ibm,fw-nbr-reboots
		 "0"
ibm,fw-src-os-vec5
		 19 00 f3 00 c0 c0 05 00 00 00 00 01 00 00 00 00................
		 00 e0 00 00 00 01 00 00 00 00 00               ...........
input-device     "/vdevice/vty"
real-size        "134217728"
ibm,fw-override-cas
		 "false"
ibm,fw-usedrcinfo?
		 "false"
ibm,fw-new-mem-def
		 "true"
ibm,fw-n-dafp    "00000000"
oem-logo?        "false"
real-mode?       "true"
boot-file        ""
diag-switch?     "false"
nvramrc          ""
boot-device      2f766465 76696365 2f6c2d6c 616e4033
		 30303030 3030323a 73706565 643d6175
		 746f2c64 75706c65 783d6175 746f2c39
		 2e34302e 3139322e 38322c62 6f6f746c
		 6f616465 722d3031 2d62612d 36382d39
		 382d3261 2d31622d 30322c39 2e34302e
		 3139352e 3230332c 392e3430 2e313935
		 2e312c35 2c352c32 35352e32 35352e32
		 [184 bytes total]
ibm,fw-forced-boot
		 ""
virt-base        "4294967295"
output-device    "/vdevice/vty"
oem-banner       ""
ibm,fw-show-true-pcie?
		 "true"
little-endian?   "false"
ibm,fw-oflite-dbg
		 "false"
ibm,fw-n-rc      "A"
ibm,fw-n-bc      "255.255.255.255"
ibm,associativity-form
		 "1"
diag-device      2f766465 76696365 2f6c2d6c 616e4033
		 30303030 3030323a 73706565 643d6175
		 746f2c64 75706c65 783d6175 746f2c39
		 2e34302e 3139322e 38322c2c 392e3430
		 2e313935 2e323033 2c392e34 302e3139
		 352e312c 352c352c 3235352e 3235352e
		 3235352e 302c3531 32202f76 64657669
		 63652f76 2d736373 69403330 30303030
		 [153 bytes total]
fcode-debug?     "true"
ibm,fw-prev-boot-vpd
		
oem-logo        
ibm,fw-prev-neg-vec5
		 14 00 f3 00 00 c0 00 00 00 00 00 01 00 00 00 00................
		 00 e0 00 00 00 00                              ......
ibm,fw-menu-ba68982a1b02
		 "auto,auto,9.40.195.203,9.40.192.82,9.40.195.1,255.255.255.0,Standard,Yes,,5,512,5"
ibm,fw-usedynmemv2?
		 "false"
auto-boot?       "true"
ibm,fw-vlan-menu-ba68982a1b02
		 "0,0"
ibm,phandle      fffffffb (-5)
selftest-#megs   "0"
ibm,dasd-spin-interval
		 "5"
boot-last-label  73657470 6172616d 73202752 65642048
		 61742045 6e746572 70726973 65204c69
		 6e757820 53657276 65722028 342e3132
		 2e302d72 63332e77 69393132 37355f30
		 3534615f 30353031 30312e70 70633634
		 6c652b29 20372e33 20284d61 69706f29
		 270a0a09 6c6f6164 5f766964 656f0a09
		 696e736d 6f642067 7a696f0a 09696e73
		 [832 bytes total]
load-base        "16384"
ibm,fw-src-os-vec4
		 01 00 01                                       ...
ibm,fw-keyboard  "1"
linux,phandle    00c9e0c0 (13230272)
virt-size        "4294967295"
diag-file        "diag"
name             "options"
ibm,os-restart-policy
		 "1"
screen-#rows     "24"

/proc/device-tree/rtas:
power-domains-names
		 "root power domain"
get-sensor-state 0000000c (12)
linux,rtas-entry 1ea51548 (514135368)
ibm,configure-kernel-dump-version
		 00010001 (65537)
rtas-sensors     00000001 00000000 00000005 00000000
		 00000009 00000000 0000232d 00000000
system-reboot    00000014 (20)
stop-self        00000018 (24)
ibm,hypertas-functions
		 6863616c 6c2d7066 74006863 616c6c2d
		 74636500 6863616c 6c2d7370 72673000
		 6863616c 6c2d636f 70790068 63616c6c
		 2d646562 75670068 63616c6c 2d746572
		 6d006863 616c6c2d 696e7465 72727570
		 74006863 616c6c2d 6d696772 61746500
		 6863616c 6c2d7065 72666d6f 6e006863
		 616c6c2d 73706c70 61720053 4c422d62
		 [434 bytes total]
ibm,lpar-perftools
		 00000045 (69)
ibm,create-pe-dma-window
		 00000054 (84)
ibm,nmi-register-2
		 00000030 (48)
ibm,nmi-register 00000030 (48)
ibm,configure-pe 00000052 (82)
ibm,display-line-length
		 00000010 (16)
ibm,lrdr-capacity
		 0000005a 00000000 00000000 10000000 00000020
ibm,dma-delay-time
		 0003e800 (256000)
ibm,nmi-interlock
		 00000031 (49)
ibm,indicator-0002
		 ""
ibm,get-indicator-indices-types
		 0000232e 0000232f
ibm,read-slot-reset-state2
		 00000048 (72)
ibm,slot-error-detail
		 00000036 (54)
ibm,update-nodes 0000004f (79)
platform-power-sources
		 00035b60 00004a38 000038a4 00000032
ibm,errinjct     00000027 (39)
ibm,set-dynamic-indicator
		 00000046 (70)
ibm,update-properties
		 00000050 (80)
ibm,sensor-0009  ""
ibm,request-partition-shutdown
		 00000000
ibm,power-off-ups
		 0000004a (74)
ibm,flash-block-version
		 00000001
ibm,recoverable-epow3
		
ibm,integrated-stop-self
		
set-indicator    0000000b (11)
nvram-fetch      00000001
ibm,change-msi   0000004b (75)
ibm,manage-flash-image
		 0000003c (60)
ibm,configure-connector
		 0000002a (42)
rtas-error-log-max
		 00000800 (2048)
power-domains-tree
		 00000000 00000002 00000000 00000001
		 00000000 000000fa 00000064 00000001
		 00000000 000000fa 00000000
ibm,sensor-9005  ""
ibm,change-msix-capable
		
ibm,get-vpd      00000042 (66)
ibm,configure-kernel-dump-sizes
		 00000001 00000000 00200020 00000002 00000000 00001000
ibm,set-eeh-option
		 0000002b (43)
ibm,form-feed    0000000c (12)
ibm,get-sensor-indices-types
		 0000232e 0000232f
ibm,remove-pe-dma-window
		 00000055 (85)
ibm,sensor-0005  ""
ibm,query-interrupt-source-number
		 0000004c (76)
rtas-display-device
		 00e3e738 (14935864)
ibm,indicator-9005
		 ""
ibm,write-pci-config
		 0000002f (47)
ibm,int-on       00000024 (36)
ibm,reset-pe-dma-window
		 00000056 (86)
ibm,max-associativity-domains
		 00000005 00000001 00000008 00000020 00000020 00000100
ibm,set-system-parameter
		 00000034 (52)
ibm,set-slot-reset
		 0000002c (44)
ibm,rks-hcalls   e0000000
start-cpu        00000019 (25)
ibm,get-config-addr-info
		 0000004d (77)
ibm,reset-capabilities
		 00000001
ibm,set-xive     00000023 (35)
get-power-level  0000000e (14)
ibm,display-truncation-length
		 00000010 00000050
ibm,activate-firmware
		 00000043 (67)
ibm,suspend-me   0000004e (78)
ibm,extended-os-term
		
event-scan       00000006
set-power-level  0000000d (13)
ibm,configure-bridge
		 00000038 (56)
ibm,validate-flash-image
		 0000003d (61)
ibm,configure-kernel-dump
		 00000051 (81)
ibm,get-dynamic-sensor-state
		 00000047 (71)
ibm,sensor-0001  ""
ibm,open-errinjct
		 00000026 (38)
rtas-indicators  00000001 00000000 00000002 00000000 0000232d 00000000
ibm,associativity-reference-points
		 00000004 00000002
ibm,close-errinjct
		 00000028 (40)
ibm,get-config-addr-info2
		 0000004d (77)
ibm,os-term      0000001a (26)
display-character
		 0000000a (10)
rtas-size        015b0000 (22740992)
get-time-of-day  00000003
ibm,update-flash-64-and-reboot
		 00000021 (33)
check-exception  00000007
nvram-store      00000002
ibm,indicator-0001
		 ""
ibm,vpd-size     0004b000 (307200)
ibm,phandle      fffffffd (-3)
power-sources-names
		 "power supply"
set-time-of-day  00000004
power-off        00000011 (17)
set-time-for-power-on
		 00000005
rtas-event-scan-rate
		 00000004
linux,rtas-base  1ea50000 (514129920)
ibm,read-pci-config
		 0000002e (46)
ibm,get-indices  00000037 (55)
linux,phandle    00e068a0 (14706848)
ibm,read-slot-reset-state-functions
		 00000003
power-domains-controllers
		 00e3e738 (14935864)
ibm,int-off      00000025 (37)
ibm,get-system-parameter
		 00000032 (50)
query-cpu-stopped-state
		 0000001e (30)
rtas-version     00000001
ibm,display-number-of-lines
		 00000002
ibm,errinjct-tokens
		 7265636f 76657265 642d7370 65636961
		 6c2d6576 656e7400 00000003 636f7272
		 75707465 642d7061 67650000 00000469
		 6f612d62 75732d65 72726f72 00000000
		 07636f72 72757074 65642d64 63616368
		 652d7374 61727400 00000009 636f7272
		 75707465 642d6463 61636865 2d656e64
		 00000000 0a636f72 72757074 65642d69
		 [236 bytes total]
ibm,get-xive     00000022 (34)
ibm,query-pe-dma-window
		 00000053 (83)
name             "rtas"
rtas-last-error  0000001f (31)
ibm,platform-dump
		 0000003b (59)
power-on-max-latency
		 ffffffff (-1)

/proc/device-tree/interrupt-controller@0:
compatible       "IBM,ppc-xicp"
device_type      "PowerPC-External-Interrupt-Presentation"
model            "IBM, Logical PowerPC-PIC, 00"
ibm,interrupt-buid-size
		 00000014 (20)
built-in        
reg              00000000 00000000 00000000 00000000
ibm,interrupt-server-ranges
		 00000000 00000080
linux,phandle    00e04a10 (14699024)
name             "interrupt-controller"
interrupt-controller
		

/proc/device-tree/aliases:
nvram            "/vdevice/nvram@4002"
net              "/vdevice/l-lan@30000002"
scsi             "/vdevice/v-scsi@30000078"
ibm,sp           "/vdevice/IBM,sp@4000"
network          "/vdevice/l-lan@30000002"
linux,phandle    00cefa00 (13564416)
name             "aliases"
rtc              "/vdevice/rtc@4001"

/proc/device-tree/chosen:
linux,stdout-package
		 00e1b528 (14791976)
ibm,architecture-vec-5
		 14 00 f3 00 00 c0 00 00 00 00 00 01 00 00 00 00................
		 00 e0 00 00 00 00                              ......
nvram            08b05100 (145772800)
bootargs         424f4f54 5f494d41 47453d2f 766d6c69
		 6e757a2d 342e3132 2e302d72 63332e77
		 69393132 37355f30 3534615f 30353031
		 30312e70 70633634 6c652b20 726f6f74
		 3d2f6465 762f6d61 70706572 2f726865
		 6c5f6c74 63616c70 696e6532 2d2d6c70
		 32302d72 6f6f7420 726f2063 72617368
		 6b65726e 656c3d61 75746f20 72642e6c
		 [196 bytes total]
linux,initrd-start
		 00000000 0ce00000
cpu              08bc0480 (146539648)
linux,kernel-end 00000000 02550000
ibm,client-architecture-support-reboot
		 00000001
linux,initrd-end 00000000 0ef84b1a
stdout           08b04f80 (145772416)
memory           08bc0700 (146540288)
ibm,phandle      fffffff7 (-9)
stdin            08b05800 (145774592)
linux,stdout-path
		 "/vdevice/vty@30000000"
linux,phandle    00c9dd50 (13229392)
bootpath         "/vdevice/v-scsi@30000078/disk@8100000000000000"
name             "chosen"
mmu              08bc0480 (146539648)
linux,memory-limit
		 00000000 00000000

/proc/device-tree/ibm,platform-facilities:
interrupt-ranges 00090400 00000400
ibm,interrupt-pool
		 00000000
device_type      "ibm,platform-facilities"
#address-cells   00000001
#size-cells      00000000
ibm,phandle      fffffff6 (-10)
linux,phandle    00d26c88 (13790344)
name             "ibm,platform-facilities"
ibm,max-async-ops-per-processor
		 00000004

/proc/device-tree/ibm,platform-facilities/ibm,compression-v1:
compatible       "ibm,compression"
ibm,max-sg-len   00000ff0 (4080)
ibm,resource-id  00000000
ibm,max-sync-cop 00000001 00010000 000001fe 00000001 00010000 000001fe
status           "okay"
ibm,phandle      fffffff4 (-12)
linux,phandle    00d28550 (13796688)
name             "ibm,compression-v1"
ibm,shared-interrupt-pool
		 00d26c88 (13790344)

/proc/device-tree/ibm,platform-facilities/ibm,random-v1:
compatible       "ibm,random"
ibm,max-sync-cop 00000000
status           "okay"
ibm,phandle      fffffff5 (-11)
linux,phandle    00d28368 (13796200)
name             "ibm,random-v1"
ibm,shared-interrupt-pool
		 00d26c88 (13790344)

/proc/device-tree/ibm,platform-facilities/ibm,sym-encryption-v1:
compatible       "ibm,sym-encryption"
ibm,max-sg-len   00000ff0 (4080)
ibm,resource-id  00000001
ibm,max-sync-cop 00000000 00000000 00000003 00000080
		 00062000 000000c4 000000c0 00062000
		 000000c4 00000100 00062000 000000c4
		 00000000 00000001 00000003 00000080
		 00062000 000000c4 000000c0 00062000
		 000000c4 00000100 00062000 000000c4
		 00000000 00000002 00000003 00000080
		 00062000 000000c4 000000c0 00062000
		 [768 bytes total]
status           "okay"
ibm,phandle      fffffff3 (-13)
linux,phandle    00d287c0 (13797312)
name             "ibm,sym-encryption-v1"
ibm,shared-interrupt-pool
		 00d26c88 (13790344)

/proc/device-tree/openprom:
relative-addressing
		
ibm,fw-vernum_encoded
		 "FW860.00 (TV860_016)"
model            "IBM,FW860.00 (TV860_016)"
ibm,phandle      fffffffe (-2)
linux,phandle    00d25650 (13784656)
ibm,fw-next-bank "T"
name             "openprom"

/proc/device-tree/packages:
linux,phandle    00c9dfc8 (13230024)
name             "packages"

/proc/device-tree/packages/obp-tftp:
network          "bootp"
		 "iscsi"
		 "ipv6"
		 "dhcp"
		 "vtag"
		 "csarch"
iscsi            "iscsi"
		 "ipv6"
		 "dhcp"
linux,phandle    00d5d120 (14012704)
name             "obp-tftp"

/proc/device-tree/packages/lpevents:
linux,phandle    00cf4390 (13583248)
name             "lpevents"

/proc/device-tree/packages/fat-files:
linux,phandle    00dcce70 (14470768)
name             "fat-files"

/proc/device-tree/packages/deblocker:
linux,phandle    00d56a98 (13986456)
name             "deblocker"

/proc/device-tree/packages/gui:
linux,phandle    00e47f28 (14974760)
name             "gui"

/proc/device-tree/packages/gui/iscsi:
linux,phandle    00e668b0 (15100080)
name             "iscsi"

/proc/device-tree/packages/post:
linux,phandle    00e79878 (15177848)
name             "post"

/proc/device-tree/packages/gscsi:
#address-cells   00000001
linux,phandle    00dbd100 (14405888)
name             "gscsi"

/proc/device-tree/packages/gscsi/disk:
device_type      "block"
linux,phandle    00dc2d98 (14429592)
name             "disk"

/proc/device-tree/packages/gscsi/tape:
device_type      "byte"
linux,phandle    00dc4be8 (14437352)
name             "tape"

/proc/device-tree/packages/boot-mgr:
linux,phandle    00dee418 (14607384)
name             "boot-mgr"

/proc/device-tree/packages/elf-loader:
linux,phandle    00de2ee0 (14560992)
name             "elf-loader"

/proc/device-tree/packages/iso-9660-files:
linux,phandle    00de1910 (14555408)
name             "iso-9660-files"

/proc/device-tree/packages/iptest:
linux,phandle    00dc68e0 (14444768)
name             "iptest"

/proc/device-tree/packages/udp:
linux,phandle    00d7f010 (14151696)
name             "udp"

/proc/device-tree/packages/tape-label:
linux,phandle    00d5c3b8 (14009272)
name             "tape-label"

/proc/device-tree/packages/cas:
linux,phandle    00dff6e8 (14677736)
name             "cas"

/proc/device-tree/packages/net:
linux,phandle    00de71a8 (14578088)
name             "net"

/proc/device-tree/packages/ping:
linux,phandle    00d8d678 (14210680)
name             "ping"

/proc/device-tree/packages/bootp:
linux,phandle    00d95620 (14243360)
name             "bootp"

/proc/device-tree/packages/utilities:
linux,phandle    00d28e00 (13798912)
name             "utilities"

/proc/device-tree/packages/utilities/trace:
linux,phandle    00d40900 (13895936)
name             "trace"

/proc/device-tree/packages/disk-label:
linux,phandle    00d57fd8 (13991896)
name             "disk-label"

/proc/device-tree/packages/prep-boot:
linux,phandle    00dcc5e0 (14468576)
name             "prep-boot"

/proc/device-tree/packages/ipv6:
linux,phandle    00d77550 (14120272)
name             "ipv6"

/proc/device-tree/packages/terminal-emulator:
iso6429-1983-colors
		
linux,phandle    00e02070 (14688368)
name             "terminal-emulator"

/proc/device-tree/packages/gen-iso-13346-files:
linux,phandle    00dd8a40 (14518848)
name             "gen-iso-13346-files"

/proc/device-tree/packages/dev-tree:
linux,phandle    00cf2928 (13576488)
name             "dev-tree"

/proc/device-tree/packages/tcp:
linux,phandle    00d86048 (14180424)
name             "tcp"

/proc/device-tree/packages/iso-13346-files:
linux,phandle    00dcf198 (14479768)
name             "iso-13346-files"

/proc/device-tree/packages/ipv4:
linux,phandle    00d70270 (14090864)
name             "ipv4"

/proc/device-tree/packages/tftp:
linux,phandle    00dab1e8 (14332392)
name             "tftp"

/proc/device-tree/packages/iscsi:
linux,phandle    00db5760 (14374752)
name             "iscsi"

/proc/device-tree/packages/nls-support:
linux,phandle    00dfe8f0 (14674160)
name             "nls-support"

/proc/device-tree/packages/dhcp:
linux,phandle    00d9ef28 (14282536)
name             "dhcp"

/proc/device-tree/packages/chrp-loader:
linux,phandle    00de2d38 (14560568)
name             "chrp-loader"

/proc/device-tree/packages/dynamic-reconfig:
linux,phandle    00e02168 (14688616)
name             "dynamic-reconfig"

/proc/device-tree/cpus:
ibm,drc-names    00000020 43505520 31004350 55203900
		 43505520 31370043 50552032 35004350
		 55203333 00435055 20343100 43505520
		 34390043 50552035 37004350 55203635
		 00435055 20373300 43505520 38310043
		 50552038 39004350 55203937 00435055
		 20313035 00435055 20313133 00435055
		 20313231 00435055 20313239 00435055
		 [245 bytes total]
ibm,drc-types    00000020 43505500 43505500 43505500
		 43505500 43505500 43505500 43505500
		 43505500 43505500 43505500 43505500
		 43505500 43505500 43505500 43505500
		 43505500 43505500 43505500 43505500
		 43505500 43505500 43505500 43505500
		 43505500 43505500 43505500 43505500
		 43505500 43505500 43505500 43505500
		 [132 bytes total]
ibm,drc-indexes  00000020 10000000 10000008 10000010
		 10000018 10000020 10000028 10000030
		 10000038 10000040 10000048 10000050
		 10000058 10000060 10000068 10000070
		 10000078 10000080 10000088 10000090
		 10000098 100000a0 100000a8 100000b0
		 100000b8 100000c0 100000c8 100000d0
		 100000d8 100000e0 100000e8 100000f0
		 [132 bytes total]
smp-enabled     
#address-cells   00000001
#size-cells      00000000
ibm,drc-power-domains
		 00000020 ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 ffffffff ffffffff ffffffff ffffffff
		 [132 bytes total]
ibm,phandle      fffffff8 (-8)
linux,phandle    00ca0558 (13239640)
name             "cpus"

/proc/device-tree/cpus/PowerPC,POWER8@8:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000008 (268435464)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000008 00000009 0000000a 0000000b
		 0000000c 0000000d 0000000e 0000000f
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000008 0000ffff 00000009 0000ffff
		 0000000a 0000ffff 0000000b 0000ffff
		 0000000c 0000ffff 0000000d 0000ffff
		 0000000e 0000ffff 0000000f 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000008
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000008 (268435464)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00ca7a50 (13269584)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

/proc/device-tree/cpus/PowerPC,POWER8@30:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000030 (268435504)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000030 00000031 00000032 00000033
		 00000034 00000035 00000036 00000037
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000030 0000ffff 00000031 0000ffff
		 00000032 0000ffff 00000033 0000ffff
		 00000034 0000ffff 00000035 0000ffff
		 00000036 0000ffff 00000037 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000030 (48)
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000030 (268435504)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00cac9b0 (13289904)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

/proc/device-tree/cpus/PowerPC,POWER8@20:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000020 (268435488)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000020 00000021 00000022 00000023
		 00000024 00000025 00000026 00000027
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000020 0000ffff 00000021 0000ffff
		 00000022 0000ffff 00000023 0000ffff
		 00000024 0000ffff 00000025 0000ffff
		 00000026 0000ffff 00000027 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000020 (32)
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000020 (268435488)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00caa9f0 (13281776)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

/proc/device-tree/cpus/PowerPC,POWER8@10:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000010 (268435472)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000010 00000011 00000012 00000013
		 00000014 00000015 00000016 00000017
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000010 0000ffff 00000011 0000ffff
		 00000012 0000ffff 00000013 0000ffff
		 00000014 0000ffff 00000015 0000ffff
		 00000016 0000ffff 00000017 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000010 (16)
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000010 (268435472)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00ca8a30 (13273648)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

/proc/device-tree/cpus/PowerPC,POWER8@0:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000000 (268435456)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000000 00000001 00000002 00000003
		 00000004 00000005 00000006 00000007
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000000 0000ffff 00000001 0000ffff
		 00000002 0000ffff 00000003 0000ffff
		 00000004 0000ffff 00000005 0000ffff
		 00000006 0000ffff 00000007 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000000
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000000 (268435456)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00ca67d0 (13264848)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

/proc/device-tree/cpus/PowerPC,POWER8@38:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000038 (268435512)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000038 00000039 0000003a 0000003b
		 0000003c 0000003d 0000003e 0000003f
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000038 0000ffff 00000039 0000ffff
		 0000003a 0000ffff 0000003b 0000ffff
		 0000003c 0000ffff 0000003d 0000ffff
		 0000003e 0000ffff 0000003f 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000038 (56)
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000038 (268435512)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00cad990 (13293968)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

/proc/device-tree/cpus/PowerPC,POWER8@28:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000028 (268435496)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000028 00000029 0000002a 0000002b
		 0000002c 0000002d 0000002e 0000002f
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000028 0000ffff 00000029 0000ffff
		 0000002a 0000ffff 0000002b 0000ffff
		 0000002c 0000ffff 0000002d 0000ffff
		 0000002e 0000ffff 0000002f 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000028 (40)
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000028 (268435496)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00cab9d0 (13285840)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

/proc/device-tree/cpus/PowerPC,POWER8@18:
bus-frequency    00000000
performance-monitor
		 00000000 00000001
ibm,pft-size     00000000 00000020
ibm,vmx          00000003
ibm,spurr        00000001
d-tlb-sets       00000004
ibm,nominal-tbf  1e848000 (512000000)
64-bit          
timebase-frequency
		 1e848000 (512000000)
ibm,extended-clock-frequency
		 00000000 b45d1880
ibm,estimate-precision
		 66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
		 0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
		 71 72 74 65 73 00 00 00 00 0e                  qrtes.....
tlb-size         00000800 (2048)
ibm,pa-features  18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
		 00 00 80 00 80 00 80 00 80 00                  ..........
graphics        
ibm,slb-size     00000020 (32)
device_type      "cpu"
tlb-sets         00000004
ibm,my-drc-index 10000018 (268435480)
reservation-granule-size
		 00000080 (128)
ibm,processor-storage-keys
		 0000001f 00000000
ibm,processor-vadd-size
		 00000044 (68)
slb-size         00000020 (32)
ibm,dfp          00000002
ibm,ppc-interrupt-server#s
		 00000018 00000019 0000001a 0000001b
		 0000001c 0000001d 0000001e 0000001f
ibm,vrma-page-sizes
		 00000018 00000010 0000000c 00000000
		 00000010 00000011 00000022 00000013
ibm,segment-page-sizes
		 0000000c 00000000 00000003 0000000c
		 00000000 00000010 00000007 00000018
		 00000038 00000010 00000110 00000002
		 00000010 00000001 00000018 00000008
		 00000018 00000100 00000001 00000018
		 00000000 00000022 00000120 00000001
		 00000022 00000003
i-cache-line-size
		 00000080 (128)
i-tlb-size       00000000
ibm,pa-optimizations
		 02001004 (33558532)
status           "okay"
cpu-version      0f000004 (251658244)
ibm,purr         00000001
ibm,ppc-interrupt-gserver#s
		 00000018 0000ffff 00000019 0000ffff
		 0000001a 0000ffff 0000001b 0000ffff
		 0000001c 0000ffff 0000001d 0000ffff
		 0000001e 0000ffff 0000001f 0000ffff
i-tlb-sets       00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features  01 00 a0                                       ...
ibm,negotiated-pa-features
		 01 00 00                                       ...
i-cache-block-size
		 00000080 (128)
32-64-bridge    
ibm,extended-bus-frequency
		 00000000 00000000
reg              00000018 (24)
d-cache-block-size
		 00000080 (128)
clock-frequency  b45d1880
ibm,phandle      10000018 (268435480)
ibm,processor-segment-sizes
		 0000001c 00000028 ffffffff ffffffff
linux,phandle    00ca9a10 (13277712)
d-cache-size     00010000 (65536)
i-cache-size     00008000 (32768)
general-purpose 
ibm,raw-pi-features
		 01 00 00                                       ...
d-tlb-size       00000800 (2048)
name             "PowerPC,POWER8"
ibm,processor-page-sizes
		 0000000c 00000010 00000018 00000022
d-cache-sets     00000008
i-cache-sets     00000004
d-cache-line-size
		 00000080 (128)

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>
> When the 'numa.c' code is built with debug messages, and the system was
> given that configuration by pHyp, yes, I did.
>
>> What does it say?
>
> The debug message for each core thread would be something like,
>
>     removing cpu 64 from node 0
>     adding cpu 64 to node 8
>
> repeated for all 8 threads of the CPU, and usually with the messages
> for all of the CPUs coming out intermixed on the console/dmesg log.

OK. I meant what do you see at boot.

I'm curious how we're discovering node 0 and 8 at all if neither has any
memory or CPUs assigned at boot.

>> Right. So it's not that you're hot adding memory into a previously
>> unseen node as you implied in earlier mails.
>
> In the sense that the nodes were defined in the device tree, that is corr=
ect.

Where are they defined in the device tree? That's what I'm trying to unders=
tand.


> In the sense that those nodes are currently deleted from node_possible_ma=
p in
> 'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map,
> node_online_map);', the nodes are no longer available to place memory or =
CPU.

Yeah I understand that part.

> Okay, I can try to insert code that extracts all of the nodes from the
> ibm,associativity-lookup-arrays property and merge them with the nodes
> put into the online map from the CPUs that were found previously during
> boot of the powerpc code.

Hmm, will that work?

Looking at PAPR it's not clear to me that it will work for nodes that
have no memory assigned at boot.

  This property is used to duplicate the function of the
  =E2=80=9Cibm,associativity=E2=80=9D property in a /memory node. Each =E2=
=80=9Cassigned=E2=80=9D LMB
  represented has an index valued between 0 and M-1 which is used as in
  index into this table to select which associativity list to use for
  the LMB. =E2=80=9Cunassigned=E2=80=9D LMBs are place holders for potentia=
l DLPAR
  additions, for which the associativity list index is meaningless and
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  is given the reserved value of -1. This static property, need only
  contain values relevant for the LMBs presented in the
  =E2=80=9Cibm,dynamicreconfiguration-memory=E2=80=9D node; for a dynamic L=
PAR addition
  of a new LMB, the device tree fragment reported by the
  ibm,configure-connector RTAS function is a /memory node, with the
  inclusion of the =E2=80=9Cibm,associativity=E2=80=9D device tree property=
 defined in
  Section C.6.2.2=E2=80=9A =E2=80=9CProperties of the Children of Root=E2=
=80=9A=E2=80=9D on page 1059.

>> What does your device tree look like? Can you send us the output of:
>>=20
>>  $ lsprop /proc/device-tree

Thanks. I forgot that lsprop will truncate long properties, I actually
wanted to see all of the ibm,dynamic-memory property.

But looking at the code I see the only place we set a nid online is if
there is a CPU assigned to it:

static int __init parse_numa_properties(void)
{
...
	for_each_present_cpu(i) {
        	...
		cpu =3D of_get_cpu_node(i, NULL);
		nid =3D of_node_to_nid_single(cpu);
                ...
		node_set_online(nid);
	}

Or for memory nodes (same function):

	for_each_node_by_type(memory, "memory") {
        	...
		nid =3D of_node_to_nid_single(memory);
                ...
		node_set_online(nid);
                ...
	}

Or for entries in ibm,dynamic-memory that are assigned:

static void __init parse_drconf_memory(struct device_node *memory)
{
	...
	for (; n !=3D 0; --n) {
        	...
		/* skip this block if the reserved bit is set in flags (0x80)
		   or if the block is not assigned to this partition (0x8) */
		if ((drmem.flags & DRCONF_MEM_RESERVED)
		    || !(drmem.flags & DRCONF_MEM_ASSIGNED))
			continue;

		...
		do {
                	...
			nid =3D of_drconf_to_nid_single(&drmem, &aa);
			node_set_online(nid);
                        ...
		} while (--ranges);
	}
}


So I don't see from that how we can even be aware that node 0 and 8
exist at boot based on that. Maybe there's another path I'm missing
though.

cheers

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On 06/06/2017 04:48 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>>
>> When the 'numa.c' code is built with debug messages, and the system was
>> given that configuration by pHyp, yes, I did.
>>
>>> What does it say?
>>
>> The debug message for each core thread would be something like,
>>
>>     removing cpu 64 from node 0
>>     adding cpu 64 to node 8
>>
>> repeated for all 8 threads of the CPU, and usually with the messages
>> for all of the CPUs coming out intermixed on the console/dmesg log.
> 
> OK. I meant what do you see at boot.

Here is an example with nodes 0,2,6,7, node 0 starts out empty:

[    0.000000] Initmem setup node 0
[    0.000000]   NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff]
[    0.000000]     NODE_DATA(0) on node 7
[    0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff]
[    0.000000]   NODE_DATA [mem 0x13ffff6300-0x13ffffffff]
[    0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff]
[    0.000000]   NODE_DATA [mem 0x34afff6300-0x34afffffff]
[    0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff]
[    0.000000]   NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff]

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x0000003bffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   2: [mem 0x0000000000000000-0x00000013ffffffff]
[    0.000000]   node   6: [mem 0x0000001400000000-0x00000034afffffff]
[    0.000000]   node   7: [mem 0x00000034b0000000-0x0000003bffffffff]
[    0.000000] Could not find start_pfn for node 0
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[    0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff]
[    0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff]
[    0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff]
[    0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144
[    0.000000] Built 4 zonelists in Node order, mobility grouping on.  Total pages: 3928320

and,

[root@ltcalpine2-lp20 ~]# numactl --hardware
available: 4 nodes (0,2,6-7)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63
node 2 size: 81792 MB
node 2 free: 81033 MB
node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47
node 6 size: 133743 MB
node 6 free: 133097 MB
node 7 cpus: 48 49 50 51 52 53 54 55
node 7 size: 29877 MB
node 7 free: 29599 MB
node distances:
node   0   2   6   7
  0:  10  40  40  40
  2:  40  10  40  40
  6:  40  40  10  20
  7:  40  40  20  10
[root@ltcalpine2-lp20 ~]#

> 
> I'm curious how we're discovering node 0 and 8 at all if neither has any
> memory or CPUs assigned at boot.

Both are in the memory associativity lookup arrays.  And we are circling
back to the 

> 
>>> Right. So it's not that you're hot adding memory into a previously
>>> unseen node as you implied in earlier mails.
>>
>> In the sense that the nodes were defined in the device tree, that is correct.
> 
> Where are they defined in the device tree? That's what I'm trying to understand.

The nodes for memory are defined one time in "ibm,associativity-lookup-arrays".
I wish that there was an official set of node properties in the device-tree,
instead of having them be the values of other properties.

> 
>> In the sense that those nodes are currently deleted from node_possible_map in
>> 'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map,
>> node_online_map);', the nodes are no longer available to place memory or CPU.
> 
> Yeah I understand that part.
> 
>> Okay, I can try to insert code that extracts all of the nodes from the
>> ibm,associativity-lookup-arrays property and merge them with the nodes
>> put into the online map from the CPUs that were found previously during
>> boot of the powerpc code.
> 
> Hmm, will that work?

The nodes are defined in the associativity lookup array, so they have at least
been reserved for us by the pHyp.  On the other hand, if we are only to use
nodes that have resources at boot, why are there extra node values specified?

What I am not 100% clear on -- and why I preferred letting all possible nodes
originally defined, still be possible for subsequent hot-add operations -- is
whether the nodes to be used for hot-added CPUs would always be a subset of
the nodes used for hot-added memory.

* The hot-added CPUs in Shared CPU configurations may be mapped to nodes by
  the value returned to the kernel by the VPHN hcall.

* So far in my tests, this has not been a problem, but I could not be positive
  from the PAPR.

> 
> Looking at PAPR it's not clear to me that it will work for nodes that
> have no memory assigned at boot.
> 
>   This property is used to duplicate the function of the
>   “ibm,associativity” property in a /memory node. Each “assigned” LMB
>   represented has an index valued between 0 and M-1 which is used as in
>   index into this table to select which associativity list to use for
>   the LMB. “unassigned” LMBs are place holders for potential DLPAR
>   additions, for which the associativity list index is meaningless and
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   is given the reserved value of -1. This static property, need only
>   contain values relevant for the LMBs presented in the
>   “ibm,dynamicreconfiguration-memory” node; for a dynamic LPAR addition
>   of a new LMB, the device tree fragment reported by the
>   ibm,configure-connector RTAS function is a /memory node, with the
>   inclusion of the “ibm,associativity” device tree property defined in
>   Section C.6.2.2‚ “Properties of the Children of Root‚” on page 1059.

I don't see any place that builds new /memory nodes in conjunction with
hot-added memory.  The code for powerpc treats the definitions provided
by 'ibm,dynamic-reconfiguration-memory' as the primary reference wherever
hot-added memory comes into play.  It looks like the '/memory' properties
are backups or used for the 'pseries' according to one comment.

> 
>>> What does your device tree look like? Can you send us the output of:
>>>
>>>  $ lsprop /proc/device-tree
> 
> Thanks. I forgot that lsprop will truncate long properties, I actually
> wanted to see all of the ibm,dynamic-memory property.
> 
> But looking at the code I see the only place we set a nid online is if
> there is a CPU assigned to it:
> 
> static int __init parse_numa_properties(void)
> {
> ...
> 	for_each_present_cpu(i) {
>         	...
> 		cpu = of_get_cpu_node(i, NULL);
> 		nid = of_node_to_nid_single(cpu);
>                 ...
> 		node_set_online(nid);
> 	}
> 
> Or for memory nodes (same function):
> 
> 	for_each_node_by_type(memory, "memory") {
>         	...
> 		nid = of_node_to_nid_single(memory);
>                 ...
> 		node_set_online(nid);
>                 ...
> 	}
> 
> Or for entries in ibm,dynamic-memory that are assigned:
> 
> static void __init parse_drconf_memory(struct device_node *memory)
> {
> 	...
> 	for (; n != 0; --n) {
>         	...
> 		/* skip this block if the reserved bit is set in flags (0x80)
> 		   or if the block is not assigned to this partition (0x8) */
> 		if ((drmem.flags & DRCONF_MEM_RESERVED)
> 		    || !(drmem.flags & DRCONF_MEM_ASSIGNED))
> 			continue;
> 
> 		...
> 		do {
>                 	...
> 			nid = of_drconf_to_nid_single(&drmem, &aa);
> 			node_set_online(nid);
>                         ...
> 		} while (--ranges);
> 	}
> }
> 
> 
> So I don't see from that how we can even be aware that node 0 and 8
> exist at boot based on that. Maybe there's another path I'm missing
> though.

We don't 'fill in' the nodes, but we are aware that they exist per the 'ibm,associativity-lookup-arrays'
or the responsed provided by the pHyp to the VPHN hcall.  We don't associate either of these resources
to them, but does that mean that the nodes do not exist?

The code currently says that only nodes booted with resources "exist" i.e. it can't hot-add new nodes,
but is that a just a problem of the kernel implementation?  I think so.

However, this is the problem for users running systems that hot-add a lot of resources are concerned.

They see the associativity arrays (and 'hypinfo' table internal to the pHyp), and they ask why the
kernel only records new resources into the boot-time nodes, while pHyp appears to distribute across
all of the memory nodes specified to the LPAR of the kernel at boot.

I think that all of those nodes specified by the pHyp should exist to the kernel, and that we are
trying to find the best way to make them visible here.

> 
> cheers
> 
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Michael Bringmann <mwb@linux.vnet.ibm.com> writes:

> On 06/06/2017 04:48 AM, Michael Ellerman wrote:
>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>>>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>>>
>>> When the 'numa.c' code is built with debug messages, and the system was
>>> given that configuration by pHyp, yes, I did.
>>>
>>>> What does it say?
>>>
>>> The debug message for each core thread would be something like,
>>>
>>>     removing cpu 64 from node 0
>>>     adding cpu 64 to node 8
>>>
>>> repeated for all 8 threads of the CPU, and usually with the messages
>>> for all of the CPUs coming out intermixed on the console/dmesg log.
>> 
>> OK. I meant what do you see at boot.
>
> Here is an example with nodes 0,2,6,7, node 0 starts out empty:
>
> [    0.000000] Initmem setup node 0
> [    0.000000]   NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff]
> [    0.000000]     NODE_DATA(0) on node 7
> [    0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff]
> [    0.000000]   NODE_DATA [mem 0x13ffff6300-0x13ffffffff]
> [    0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff]
> [    0.000000]   NODE_DATA [mem 0x34afff6300-0x34afffffff]
> [    0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff]
> [    0.000000]   NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff]
>
> [    0.000000] Zone ranges:
> [    0.000000]   DMA      [mem 0x0000000000000000-0x0000003bffffffff]
> [    0.000000]   DMA32    empty
> [    0.000000]   Normal   empty
> [    0.000000] Movable zone start for each node
> [    0.000000] Early memory node ranges
> [    0.000000]   node   2: [mem 0x0000000000000000-0x00000013ffffffff]
> [    0.000000]   node   6: [mem 0x0000001400000000-0x00000034afffffff]
> [    0.000000]   node   7: [mem 0x00000034b0000000-0x0000003bffffffff]
> [    0.000000] Could not find start_pfn for node 0
> [    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
> [    0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff]
> [    0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff]
> [    0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff]
> [    0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144
> [    0.000000] Built 4 zonelists in Node order, mobility grouping on.  Total pages: 3928320
>
> and,
>
> [root@ltcalpine2-lp20 ~]# numactl --hardware
> available: 4 nodes (0,2,6-7)
> node 0 cpus:
> node 0 size: 0 MB
> node 0 free: 0 MB
> node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63
> node 2 size: 81792 MB
> node 2 free: 81033 MB
> node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47
> node 6 size: 133743 MB
> node 6 free: 133097 MB
> node 7 cpus: 48 49 50 51 52 53 54 55
> node 7 size: 29877 MB
> node 7 free: 29599 MB
> node distances:
> node   0   2   6   7
>   0:  10  40  40  40
>   2:  40  10  40  40
>   6:  40  40  10  20
>   7:  40  40  20  10
> [root@ltcalpine2-lp20 ~]#

What kernel is that running?

And can you show me the full ibm,dynamic-memory and lookup-arrays
properties for that system?

cheers

RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

[-- Attachment #1: Type: text/plain, Size: 3833 bytes --]


Here is the information from 2 different kernels.  I have not been able to retrieve
the information matching yesterday's attachments, yet, as those dumps were
acquired in April.
 
Attached please find 2 dumps of similar material from kernels running with my
current patches (Linux 4.4, Linux 4.12).

On 06/07/2017 07:08 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> 
>> On 06/06/2017 04:48 AM, Michael Ellerman wrote:
>>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>>> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>>>>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>>>>
>>>> When the 'numa.c' code is built with debug messages, and the system was
>>>> given that configuration by pHyp, yes, I did.
>>>>
>>>>> What does it say?
>>>>
>>>> The debug message for each core thread would be something like,
>>>>
>>>>     removing cpu 64 from node 0
>>>>     adding cpu 64 to node 8
>>>>
>>>> repeated for all 8 threads of the CPU, and usually with the messages
>>>> for all of the CPUs coming out intermixed on the console/dmesg log.
>>>
>>> OK. I meant what do you see at boot.
>>
>> Here is an example with nodes 0,2,6,7, node 0 starts out empty:
>>
>> [    0.000000] Initmem setup node 0
>> [    0.000000]   NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff]
>> [    0.000000]     NODE_DATA(0) on node 7
>> [    0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff]
>> [    0.000000]   NODE_DATA [mem 0x13ffff6300-0x13ffffffff]
>> [    0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff]
>> [    0.000000]   NODE_DATA [mem 0x34afff6300-0x34afffffff]
>> [    0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff]
>> [    0.000000]   NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff]
>>
>> [    0.000000] Zone ranges:
>> [    0.000000]   DMA      [mem 0x0000000000000000-0x0000003bffffffff]
>> [    0.000000]   DMA32    empty
>> [    0.000000]   Normal   empty
>> [    0.000000] Movable zone start for each node
>> [    0.000000] Early memory node ranges
>> [    0.000000]   node   2: [mem 0x0000000000000000-0x00000013ffffffff]
>> [    0.000000]   node   6: [mem 0x0000001400000000-0x00000034afffffff]
>> [    0.000000]   node   7: [mem 0x00000034b0000000-0x0000003bffffffff]
>> [    0.000000] Could not find start_pfn for node 0
>> [    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
>> [    0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff]
>> [    0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff]
>> [    0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff]
>> [    0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144
>> [    0.000000] Built 4 zonelists in Node order, mobility grouping on.  Total pages: 3928320
>>
>> and,
>>
>> [root@ltcalpine2-lp20 ~]# numactl --hardware
>> available: 4 nodes (0,2,6-7)
>> node 0 cpus:
>> node 0 size: 0 MB
>> node 0 free: 0 MB
>> node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63
>> node 2 size: 81792 MB
>> node 2 free: 81033 MB
>> node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47
>> node 6 size: 133743 MB
>> node 6 free: 133097 MB
>> node 7 cpus: 48 49 50 51 52 53 54 55
>> node 7 size: 29877 MB
>> node 7 free: 29599 MB
>> node distances:
>> node   0   2   6   7
>>   0:  10  40  40  40
>>   2:  40  10  40  40
>>   6:  40  40  10  20
>>   7:  40  40  20  10
>> [root@ltcalpine2-lp20 ~]#
> 
> What kernel is that running?
> 
> And can you show me the full ibm,dynamic-memory and lookup-arrays
> properties for that system?
> 
> cheers
> 
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

[-- Attachment #2: 1.linux4.12-3nodes.txt --]
[-- Type: text/plain, Size: 4480 bytes --]


Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel 4.12.0-rc3.wi91275_054c_060106.ppc64le+ on an ppc64le

ltcalpine2-lp20 login: root
Password: 
Last login: Wed Jun  7 11:03:12 from oc1554177480.austin.ibm.com
[root@ltcalpine2-lp20 ~]# numactl -H
available: 3 nodes (0,2-3)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 48 49 50 51 52 53 54 55
node 2 size: 188668 MB
node 2 free: 187903 MB
node 3 cpus: 40 41 42 43 44 45 46 47 56 57 58 59 60 61 62 63
node 3 size: 56261 MB
node 3 free: 55324 MB
node distances:
node   0   2   3 
  0:  10  40  40 
  2:  40  10  20 
  3:  40  20  10 
[root@ltcalpine2-lp20 ~]# cd /proc/device-tree/ibm,dynamic-reconfiguration-memory
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,dynamic-memory
ibm,dynamic-memory
		 0000059e 00000000 20000000 80000002
		 00000000 00000001 00000008 00000000
		 30000000 80000003 00000000 00000001
		 00000008 00000000 40000000 80000004
		 00000000 00000001 00000008 00000000
		 50000000 80000005 00000000 00000001
		 00000008 00000000 60000000 80000006
		 00000000 00000001 00000008 00000000
		 [34516 bytes total]
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
		 00000003 00000004 00000000 00000000
		 00000000 00000000 00000000 00000001
		 00000002 00000002 00000000 00000001
		 00000003 00000003
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# cat ~/dmesg.log
[    0.000000] Linux version 4.12.0-rc3.wi91275_054c_060106.ppc64le+ (root@ltcalpine2-lp20.aus.stglabs.ibm.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #28 SMP Tue Jun 6 16:05:01 EDT 2017
[    0.000000] Found initrd at 0xc00000000ce00000:0xc00000000eeb4b12
[    0.000000] Using pSeries machine description
[    0.000000] bootconsole [udbg0] enabled
[    0.000000] Partition configured for 256 cpus.
[    0.000000] CPU maps initialized for 8 threads per core
[    0.000000]  (thread shift is 3)
[    0.000000] Freed 4784128 bytes for unused pacas
[    0.000000] -----------------------------------------------------
[    0.000000] ppc64_pft_size    = 0x20
[    0.000000] phys_mem_size     = 0x3c00000000
[    0.000000] dcache_bsize      = 0x80
[    0.000000] icache_bsize      = 0x80
[    0.000000] cpu_features      = 0x27fc7aec18500249
[    0.000000]   possible        = 0x7fffffff18500649
[    0.000000]   always          = 0x0000000018100040
[    0.000000] cpu_user_features = 0xdc0065c2 0xef000000
[    0.000000] mmu_features      = 0x7c006001
[    0.000000] firmware_features = 0x00000003c45bfc57
[    0.000000] htab_hash_mask    = 0x1ffffff
[    0.000000] -----------------------------------------------------
[    0.000000] numa:   NODE_DATA [mem 0x3bff7d5f00-0x3bff7dffff]
[    0.000000] numa:     NODE_DATA(0) on node 3
[    0.000000] numa:   NODE_DATA [mem 0x2e1fff5f00-0x2e1fffffff]
[    0.000000] numa:   NODE_DATA [mem 0x3bff7cbe00-0x3bff7d5eff]
[    0.000000] Section 15357 and 15359 (node 3) have a circular dependency on usemap and pgdat allocations
[    0.000000] PPC64 nvram contains 15360 bytes
[    0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000
[    0.000000] Memory hole size: 0MB
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x0000003bffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   2: [mem 0x0000000000000000-0x0000002e1fffffff]
[    0.000000]   node   3: [mem 0x0000002e20000000-0x0000003bffffffff]
[    0.000000] Could not find start_pfn for node 0
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[    0.000000] On node 0 totalpages: 0
[    0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x0000002e1fffffff]
[    0.000000] On node 2 totalpages: 3022848
[    0.000000]   DMA zone: 2952 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 3022848 pages, LIFO batch:1
[    0.000000] Initmem setup node 3 [mem 0x0000002e20000000-0x0000003bffffffff]
[    0.000000] On node 3 totalpages: 909312
[    0.000000]   DMA zone: 888 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 909312 pages, LIFO batch:1
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]#


[-- Attachment #3: 2.linux4.4-4nodes.txt --]
[-- Type: text/plain, Size: 88606 bytes --]


Welcome to SUSE Linux Enterprise Server 12 SP3 Beta2 (ppc64le) - Kernel 4.4.63-2.1.wi91275.sp3.055a_27-default (hvc0).


ltcalpine2-lp9 login: root
Password: 
Last login: Mon Jun  5 11:43:45 from 9.53.92.157
ltcalpine2-lp9:ß # numactl -H
available: 4 nodes (0-1,6-7)
node 0 cpus:
node 0 size: 20512 MB
node 0 free: 20371 MB
node 1 cpus:
node 1 size: 2815 MB
node 1 free: 2706 MB
node 6 cpus: 0 1 2 3 4 5 6 7
node 6 size: 134783 MB
node 6 free: 134068 MB
node 7 cpus:
node 7 size: 87210 MB
node 7 free: 87102 MB
node distances:
node   0   1   6   7 
  0:  10  20  40  40 
  1:  20  10  40  40 
  6:  40  40  10  20 
  7:  40  40  20  10 
ltcalpine2-lp9:~ # cd 
ltcalpine2-lp9:~ # cd /proc/device-tree/ibm,dynamic-reconfiguration-memory
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # ls
ibm,associativity-lookup-arrays  ibm,memory-flags-mask         linux,phandle
ibm,dynamic-memory               ibm,memory-preservation-time  name
ibm,lmb-size                     ibm,phandle
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
		 00000004 00000004 00000000 00000000 00000000 00000000
		 00000000 00000000 00000001 00000001 00000000 00000003
		 00000006 00000006 00000000 00000003 00000007 00000007
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,dynamic-memory
ibm,dynamic-memory
		 0000059e 00000000 20000000 80000002
		 00000000 00000000 00000008 00000000
		 30000000 80000003 00000000 00000000
		 00000008 00000000 40000000 80000004
		 00000000 00000000 00000008 00000000
		 50000000 80000005 00000000 00000000
		 00000008 00000000 60000000 80000006
		 00000000 00000000 00000008 00000000
		 70000000 80000007 00000000 00000000
		 00000008 00000000 80000000 80000008
		 00000000 00000000 00000008 00000000
		 90000000 80000009 00000000 00000000
		 00000008 00000000 a0000000 8000000a
		 00000000 00000000 00000008 00000000
		 b0000000 8000000b 00000000 00000000
		 00000008 00000000 c0000000 8000000c
		 00000000 00000000 00000008 00000000
		 d0000000 8000000d 00000000 00000000
		 00000008 00000000 e0000000 8000000e
		 00000000 00000000 00000008 00000000
		 f0000000 8000000f 00000000 00000000
		 00000008 00000001 00000000 80000010
		 00000000 00000000 00000008 00000001
		 10000000 80000011 00000000 00000000
		 00000008 00000001 20000000 80000012
		 00000000 00000000 00000008 00000001
		 30000000 80000013 00000000 00000000
		 00000008 00000001 40000000 80000014
		 00000000 00000000 00000008 00000001
		 50000000 80000015 00000000 00000000
		 00000008 00000001 60000000 80000016
		 00000000 00000000 00000008 00000001
		 70000000 80000017 00000000 00000000
		 00000008 00000001 80000000 80000018
		 00000000 00000000 00000008 00000001
		 90000000 80000019 00000000 00000000
		 00000008 00000001 a0000000 8000001a
		 00000000 00000000 00000008 00000001
		 b0000000 8000001b 00000000 00000000
		 00000008 00000001 c0000000 8000001c
		 00000000 00000000 00000008 00000001
		 d0000000 8000001d 00000000 00000000
		 00000008 00000001 e0000000 8000001e
		 00000000 00000000 00000008 00000001
		 f0000000 8000001f 00000000 00000000
		 00000008 00000002 00000000 80000020
		 00000000 00000000 00000008 00000002
		 10000000 80000021 00000000 00000000
		 00000008 00000002 20000000 80000022
		 00000000 00000000 00000008 00000002
		 30000000 80000023 00000000 00000000
		 00000008 00000002 40000000 80000024
		 00000000 00000000 00000008 00000002
		 50000000 80000025 00000000 00000000
		 00000008 00000002 60000000 80000026
		 00000000 00000000 00000008 00000002
		 70000000 80000027 00000000 00000000
		 00000008 00000002 80000000 80000028
		 00000000 00000000 00000008 00000002
		 90000000 80000029 00000000 00000000
		 00000008 00000002 a0000000 8000002a
		 00000000 00000000 00000008 00000002
		 b0000000 8000002b 00000000 00000000
		 00000008 00000002 c0000000 8000002c
		 00000000 00000000 00000008 00000002
		 d0000000 8000002d 00000000 00000000
		 00000008 00000002 e0000000 8000002e
		 00000000 00000000 00000008 00000002
		 f0000000 8000002f 00000000 00000000
		 00000008 00000003 00000000 80000030
		 00000000 00000000 00000008 00000003
		 10000000 80000031 00000000 00000000
		 00000008 00000003 20000000 80000032
		 00000000 00000000 00000008 00000003
		 30000000 80000033 00000000 00000000
		 00000008 00000003 40000000 80000034
		 00000000 00000000 00000008 00000003
		 50000000 80000035 00000000 00000000
		 00000008 00000003 60000000 80000036
		 00000000 00000000 00000008 00000003
		 70000000 80000037 00000000 00000000
		 00000008 00000003 80000000 80000038
		 00000000 00000000 00000008 00000003
		 90000000 80000039 00000000 00000000
		 00000008 00000003 a0000000 8000003a
		 00000000 00000000 00000008 00000003
		 b0000000 8000003b 00000000 00000000
		 00000008 00000003 c0000000 8000003c
		 00000000 00000000 00000008 00000003
		 d0000000 8000003d 00000000 00000000
		 00000008 00000003 e0000000 8000003e
		 00000000 00000000 00000008 00000003
		 f0000000 8000003f 00000000 00000000
		 00000008 00000004 00000000 80000040
		 00000000 00000000 00000008 00000004
		 10000000 80000041 00000000 00000000
		 00000008 00000004 20000000 80000042
		 00000000 00000000 00000008 00000004
		 30000000 80000043 00000000 00000000
		 00000008 00000004 40000000 80000044
		 00000000 00000000 00000008 00000004
		 50000000 80000045 00000000 00000000
		 00000008 00000004 60000000 80000046
		 00000000 00000000 00000008 00000004
		 70000000 80000047 00000000 00000000
		 00000008 00000004 80000000 80000048
		 00000000 00000000 00000008 00000004
		 90000000 80000049 00000000 00000000
		 00000008 00000004 a0000000 8000004a
		 00000000 00000000 00000008 00000004
		 b0000000 8000004b 00000000 00000000
		 00000008 00000004 c0000000 8000004c
		 00000000 00000000 00000008 00000004
		 d0000000 8000004d 00000000 00000000
		 00000008 00000004 e0000000 8000004e
		 00000000 00000000 00000008 00000004
		 f0000000 8000004f 00000000 00000000
		 00000008 00000005 00000000 80000050
		 00000000 00000000 00000008 00000005
		 10000000 80000051 00000000 00000001
		 00000008 00000005 20000000 80000052
		 00000000 00000001 00000008 00000005
		 30000000 80000053 00000000 00000001
		 00000008 00000005 40000000 80000054
		 00000000 00000001 00000008 00000005
		 50000000 80000055 00000000 00000001
		 00000008 00000005 60000000 80000056
		 00000000 00000001 00000008 00000005
		 70000000 80000057 00000000 00000001
		 00000008 00000005 80000000 80000058
		 00000000 00000001 00000008 00000005
		 90000000 80000059 00000000 00000001
		 00000008 00000005 a0000000 8000005a
		 00000000 00000001 00000008 00000005
		 b0000000 8000005b 00000000 00000001
		 00000008 00000005 c0000000 8000005c
		 00000000 00000002 00000008 00000005
		 d0000000 8000005d 00000000 00000002
		 00000008 00000005 e0000000 8000005e
		 00000000 00000002 00000008 00000005
		 f0000000 8000005f 00000000 00000002
		 00000008 00000006 00000000 80000060
		 00000000 00000002 00000008 00000006
		 10000000 80000061 00000000 00000002
		 00000008 00000006 20000000 80000062
		 00000000 00000002 00000008 00000006
		 30000000 80000063 00000000 00000002
		 00000008 00000006 40000000 80000064
		 00000000 00000002 00000008 00000006
		 50000000 80000065 00000000 00000002
		 00000008 00000006 60000000 80000066
		 00000000 00000002 00000008 00000006
		 70000000 80000067 00000000 00000002
		 00000008 00000006 80000000 80000068
		 00000000 00000002 00000008 00000006
		 90000000 80000069 00000000 00000002
		 00000008 00000006 a0000000 8000006a
		 00000000 00000002 00000008 00000006
		 b0000000 8000006b 00000000 00000002
		 00000008 00000006 c0000000 8000006c
		 00000000 00000002 00000008 00000006
		 d0000000 8000006d 00000000 00000002
		 00000008 00000006 e0000000 8000006e
		 00000000 00000002 00000008 00000006
		 f0000000 8000006f 00000000 00000002
		 00000008 00000007 00000000 80000070
		 00000000 00000002 00000008 00000007
		 10000000 80000071 00000000 00000002
		 00000008 00000007 20000000 80000072
		 00000000 00000002 00000008 00000007
		 30000000 80000073 00000000 00000002
		 00000008 00000007 40000000 80000074
		 00000000 00000002 00000008 00000007
		 50000000 80000075 00000000 00000002
		 00000008 00000007 60000000 80000076
		 00000000 00000002 00000008 00000007
		 70000000 80000077 00000000 00000002
		 00000008 00000007 80000000 80000078
		 00000000 00000002 00000008 00000007
		 90000000 80000079 00000000 00000002
		 00000008 00000007 a0000000 8000007a
		 00000000 00000002 00000008 00000007
		 b0000000 8000007b 00000000 00000002
		 00000008 00000007 c0000000 8000007c
		 00000000 00000002 00000008 00000007
		 d0000000 8000007d 00000000 00000002
		 00000008 00000007 e0000000 8000007e
		 00000000 00000002 00000008 00000007
		 f0000000 8000007f 00000000 00000002
		 00000008 00000008 00000000 80000080
		 00000000 00000002 00000008 00000008
		 10000000 80000081 00000000 00000002
		 00000008 00000008 20000000 80000082
		 00000000 00000002 00000008 00000008
		 30000000 80000083 00000000 00000002
		 00000008 00000008 40000000 80000084
		 00000000 00000002 00000008 00000008
		 50000000 80000085 00000000 00000002
		 00000008 00000008 60000000 80000086
		 00000000 00000002 00000008 00000008
		 70000000 80000087 00000000 00000002
		 00000008 00000008 80000000 80000088
		 00000000 00000002 00000008 00000008
		 90000000 80000089 00000000 00000002
		 00000008 00000008 a0000000 8000008a
		 00000000 00000002 00000008 00000008
		 b0000000 8000008b 00000000 00000002
		 00000008 00000008 c0000000 8000008c
		 00000000 00000002 00000008 00000008
		 d0000000 8000008d 00000000 00000002
		 00000008 00000008 e0000000 8000008e
		 00000000 00000002 00000008 00000008
		 f0000000 8000008f 00000000 00000002
		 00000008 00000009 00000000 80000090
		 00000000 00000002 00000008 00000009
		 10000000 80000091 00000000 00000002
		 00000008 00000009 20000000 80000092
		 00000000 00000002 00000008 00000009
		 30000000 80000093 00000000 00000002
		 00000008 00000009 40000000 80000094
		 00000000 00000002 00000008 00000009
		 50000000 80000095 00000000 00000002
		 00000008 00000009 60000000 80000096
		 00000000 00000002 00000008 00000009
		 70000000 80000097 00000000 00000002
		 00000008 00000009 80000000 80000098
		 00000000 00000002 00000008 00000009
		 90000000 80000099 00000000 00000002
		 00000008 00000009 a0000000 8000009a
		 00000000 00000002 00000008 00000009
		 b0000000 8000009b 00000000 00000002
		 00000008 00000009 c0000000 8000009c
		 00000000 00000002 00000008 00000009
		 d0000000 8000009d 00000000 00000002
		 00000008 00000009 e0000000 8000009e
		 00000000 00000002 00000008 00000009
		 f0000000 8000009f 00000000 00000002
		 00000008 0000000a 00000000 800000a0
		 00000000 00000002 00000008 0000000a
		 10000000 800000a1 00000000 00000002
		 00000008 0000000a 20000000 800000a2
		 00000000 00000002 00000008 0000000a
		 30000000 800000a3 00000000 00000002
		 00000008 0000000a 40000000 800000a4
		 00000000 00000002 00000008 0000000a
		 50000000 800000a5 00000000 00000002
		 00000008 0000000a 60000000 800000a6
		 00000000 00000002 00000008 0000000a
		 70000000 800000a7 00000000 00000002
		 00000008 0000000a 80000000 800000a8
		 00000000 00000002 00000008 0000000a
		 90000000 800000a9 00000000 00000002
		 00000008 0000000a a0000000 800000aa
		 00000000 00000002 00000008 0000000a
		 b0000000 800000ab 00000000 00000002
		 00000008 0000000a c0000000 800000ac
		 00000000 00000002 00000008 0000000a
		 d0000000 800000ad 00000000 00000002
		 00000008 0000000a e0000000 800000ae
		 00000000 00000002 00000008 0000000a
		 f0000000 800000af 00000000 00000002
		 00000008 0000000b 00000000 800000b0
		 00000000 00000002 00000008 0000000b
		 10000000 800000b1 00000000 00000002
		 00000008 0000000b 20000000 800000b2
		 00000000 00000002 00000008 0000000b
		 30000000 800000b3 00000000 00000002
		 00000008 0000000b 40000000 800000b4
		 00000000 00000002 00000008 0000000b
		 50000000 800000b5 00000000 00000002
		 00000008 0000000b 60000000 800000b6
		 00000000 00000002 00000008 0000000b
		 70000000 800000b7 00000000 00000002
		 00000008 0000000b 80000000 800000b8
		 00000000 00000002 00000008 0000000b
		 90000000 800000b9 00000000 00000002
		 00000008 0000000b a0000000 800000ba
		 00000000 00000002 00000008 0000000b
		 b0000000 800000bb 00000000 00000002
		 00000008 0000000b c0000000 800000bc
		 00000000 00000002 00000008 0000000b
		 d0000000 800000bd 00000000 00000002
		 00000008 0000000b e0000000 800000be
		 00000000 00000002 00000008 0000000b
		 f0000000 800000bf 00000000 00000002
		 00000008 0000000c 00000000 800000c0
		 00000000 00000002 00000008 0000000c
		 10000000 800000c1 00000000 00000002
		 00000008 0000000c 20000000 800000c2
		 00000000 00000002 00000008 0000000c
		 30000000 800000c3 00000000 00000002
		 00000008 0000000c 40000000 800000c4
		 00000000 00000002 00000008 0000000c
		 50000000 800000c5 00000000 00000002
		 00000008 0000000c 60000000 800000c6
		 00000000 00000002 00000008 0000000c
		 70000000 800000c7 00000000 00000002
		 00000008 0000000c 80000000 800000c8
		 00000000 00000002 00000008 0000000c
		 90000000 800000c9 00000000 00000002
		 00000008 0000000c a0000000 800000ca
		 00000000 00000002 00000008 0000000c
		 b0000000 800000cb 00000000 00000002
		 00000008 0000000c c0000000 800000cc
		 00000000 00000002 00000008 0000000c
		 d0000000 800000cd 00000000 00000002
		 00000008 0000000c e0000000 800000ce
		 00000000 00000002 00000008 0000000c
		 f0000000 800000cf 00000000 00000002
		 00000008 0000000d 00000000 800000d0
		 00000000 00000002 00000008 0000000d
		 10000000 800000d1 00000000 00000002
		 00000008 0000000d 20000000 800000d2
		 00000000 00000002 00000008 0000000d
		 30000000 800000d3 00000000 00000002
		 00000008 0000000d 40000000 800000d4
		 00000000 00000002 00000008 0000000d
		 50000000 800000d5 00000000 00000002
		 00000008 0000000d 60000000 800000d6
		 00000000 00000002 00000008 0000000d
		 70000000 800000d7 00000000 00000002
		 00000008 0000000d 80000000 800000d8
		 00000000 00000002 00000008 0000000d
		 90000000 800000d9 00000000 00000002
		 00000008 0000000d a0000000 800000da
		 00000000 00000002 00000008 0000000d
		 b0000000 800000db 00000000 00000002
		 00000008 0000000d c0000000 800000dc
		 00000000 00000002 00000008 0000000d
		 d0000000 800000dd 00000000 00000002
		 00000008 0000000d e0000000 800000de
		 00000000 00000002 00000008 0000000d
		 f0000000 800000df 00000000 00000002
		 00000008 0000000e 00000000 800000e0
		 00000000 00000002 00000008 0000000e
		 10000000 800000e1 00000000 00000002
		 00000008 0000000e 20000000 800000e2
		 00000000 00000002 00000008 0000000e
		 30000000 800000e3 00000000 00000002
		 00000008 0000000e 40000000 800000e4
		 00000000 00000002 00000008 0000000e
		 50000000 800000e5 00000000 00000002
		 00000008 0000000e 60000000 800000e6
		 00000000 00000002 00000008 0000000e
		 70000000 800000e7 00000000 00000002
		 00000008 0000000e 80000000 800000e8
		 00000000 00000002 00000008 0000000e
		 90000000 800000e9 00000000 00000002
		 00000008 0000000e a0000000 800000ea
		 00000000 00000002 00000008 0000000e
		 b0000000 800000eb 00000000 00000002
		 00000008 0000000e c0000000 800000ec
		 00000000 00000002 00000008 0000000e
		 d0000000 800000ed 00000000 00000002
		 00000008 0000000e e0000000 800000ee
		 00000000 00000002 00000008 0000000e
		 f0000000 800000ef 00000000 00000002
		 00000008 0000000f 00000000 800000f0
		 00000000 00000002 00000008 0000000f
		 10000000 800000f1 00000000 00000002
		 00000008 0000000f 20000000 800000f2
		 00000000 00000002 00000008 0000000f
		 30000000 800000f3 00000000 00000002
		 00000008 0000000f 40000000 800000f4
		 00000000 00000002 00000008 0000000f
		 50000000 800000f5 00000000 00000002
		 00000008 0000000f 60000000 800000f6
		 00000000 00000002 00000008 0000000f
		 70000000 800000f7 00000000 00000002
		 00000008 0000000f 80000000 800000f8
		 00000000 00000002 00000008 0000000f
		 90000000 800000f9 00000000 00000002
		 00000008 0000000f a0000000 800000fa
		 00000000 00000002 00000008 0000000f
		 b0000000 800000fb 00000000 00000002
		 00000008 0000000f c0000000 800000fc
		 00000000 00000002 00000008 0000000f
		 d0000000 800000fd 00000000 00000002
		 00000008 0000000f e0000000 800000fe
		 00000000 00000002 00000008 0000000f
		 f0000000 800000ff 00000000 00000002
		 00000008 00000010 00000000 80000100
		 00000000 00000002 00000008 00000010
		 10000000 80000101 00000000 00000002
		 00000008 00000010 20000000 80000102
		 00000000 00000002 00000008 00000010
		 30000000 80000103 00000000 00000002
		 00000008 00000010 40000000 80000104
		 00000000 00000002 00000008 00000010
		 50000000 80000105 00000000 00000002
		 00000008 00000010 60000000 80000106
		 00000000 00000002 00000008 00000010
		 70000000 80000107 00000000 00000002
		 00000008 00000010 80000000 80000108
		 00000000 00000002 00000008 00000010
		 90000000 80000109 00000000 00000002
		 00000008 00000010 a0000000 8000010a
		 00000000 00000002 00000008 00000010
		 b0000000 8000010b 00000000 00000002
		 00000008 00000010 c0000000 8000010c
		 00000000 00000002 00000008 00000010
		 d0000000 8000010d 00000000 00000002
		 00000008 00000010 e0000000 8000010e
		 00000000 00000002 00000008 00000010
		 f0000000 8000010f 00000000 00000002
		 00000008 00000011 00000000 80000110
		 00000000 00000002 00000008 00000011
		 10000000 80000111 00000000 00000002
		 00000008 00000011 20000000 80000112
		 00000000 00000002 00000008 00000011
		 30000000 80000113 00000000 00000002
		 00000008 00000011 40000000 80000114
		 00000000 00000002 00000008 00000011
		 50000000 80000115 00000000 00000002
		 00000008 00000011 60000000 80000116
		 00000000 00000002 00000008 00000011
		 70000000 80000117 00000000 00000002
		 00000008 00000011 80000000 80000118
		 00000000 00000002 00000008 00000011
		 90000000 80000119 00000000 00000002
		 00000008 00000011 a0000000 8000011a
		 00000000 00000002 00000008 00000011
		 b0000000 8000011b 00000000 00000002
		 00000008 00000011 c0000000 8000011c
		 00000000 00000002 00000008 00000011
		 d0000000 8000011d 00000000 00000002
		 00000008 00000011 e0000000 8000011e
		 00000000 00000002 00000008 00000011
		 f0000000 8000011f 00000000 00000002
		 00000008 00000012 00000000 80000120
		 00000000 00000002 00000008 00000012
		 10000000 80000121 00000000 00000002
		 00000008 00000012 20000000 80000122
		 00000000 00000002 00000008 00000012
		 30000000 80000123 00000000 00000002
		 00000008 00000012 40000000 80000124
		 00000000 00000002 00000008 00000012
		 50000000 80000125 00000000 00000002
		 00000008 00000012 60000000 80000126
		 00000000 00000002 00000008 00000012
		 70000000 80000127 00000000 00000002
		 00000008 00000012 80000000 80000128
		 00000000 00000002 00000008 00000012
		 90000000 80000129 00000000 00000002
		 00000008 00000012 a0000000 8000012a
		 00000000 00000002 00000008 00000012
		 b0000000 8000012b 00000000 00000002
		 00000008 00000012 c0000000 8000012c
		 00000000 00000002 00000008 00000012
		 d0000000 8000012d 00000000 00000002
		 00000008 00000012 e0000000 8000012e
		 00000000 00000002 00000008 00000012
		 f0000000 8000012f 00000000 00000002
		 00000008 00000013 00000000 80000130
		 00000000 00000002 00000008 00000013
		 10000000 80000131 00000000 00000002
		 00000008 00000013 20000000 80000132
		 00000000 00000002 00000008 00000013
		 30000000 80000133 00000000 00000002
		 00000008 00000013 40000000 80000134
		 00000000 00000002 00000008 00000013
		 50000000 80000135 00000000 00000002
		 00000008 00000013 60000000 80000136
		 00000000 00000002 00000008 00000013
		 70000000 80000137 00000000 00000002
		 00000008 00000013 80000000 80000138
		 00000000 00000002 00000008 00000013
		 90000000 80000139 00000000 00000002
		 00000008 00000013 a0000000 8000013a
		 00000000 00000002 00000008 00000013
		 b0000000 8000013b 00000000 00000002
		 00000008 00000013 c0000000 8000013c
		 00000000 00000002 00000008 00000013
		 d0000000 8000013d 00000000 00000002
		 00000008 00000013 e0000000 8000013e
		 00000000 00000002 00000008 00000013
		 f0000000 8000013f 00000000 00000002
		 00000008 00000014 00000000 80000140
		 00000000 00000002 00000008 00000014
		 10000000 80000141 00000000 00000002
		 00000008 00000014 20000000 80000142
		 00000000 00000002 00000008 00000014
		 30000000 80000143 00000000 00000002
		 00000008 00000014 40000000 80000144
		 00000000 00000002 00000008 00000014
		 50000000 80000145 00000000 00000002
		 00000008 00000014 60000000 80000146
		 00000000 00000002 00000008 00000014
		 70000000 80000147 00000000 00000002
		 00000008 00000014 80000000 80000148
		 00000000 00000002 00000008 00000014
		 90000000 80000149 00000000 00000002
		 00000008 00000014 a0000000 8000014a
		 00000000 00000002 00000008 00000014
		 b0000000 8000014b 00000000 00000002
		 00000008 00000014 c0000000 8000014c
		 00000000 00000002 00000008 00000014
		 d0000000 8000014d 00000000 00000002
		 00000008 00000014 e0000000 8000014e
		 00000000 00000002 00000008 00000014
		 f0000000 8000014f 00000000 00000002
		 00000008 00000015 00000000 80000150
		 00000000 00000002 00000008 00000015
		 10000000 80000151 00000000 00000002
		 00000008 00000015 20000000 80000152
		 00000000 00000002 00000008 00000015
		 30000000 80000153 00000000 00000002
		 00000008 00000015 40000000 80000154
		 00000000 00000002 00000008 00000015
		 50000000 80000155 00000000 00000002
		 00000008 00000015 60000000 80000156
		 00000000 00000002 00000008 00000015
		 70000000 80000157 00000000 00000002
		 00000008 00000015 80000000 80000158
		 00000000 00000002 00000008 00000015
		 90000000 80000159 00000000 00000002
		 00000008 00000015 a0000000 8000015a
		 00000000 00000002 00000008 00000015
		 b0000000 8000015b 00000000 00000002
		 00000008 00000015 c0000000 8000015c
		 00000000 00000002 00000008 00000015
		 d0000000 8000015d 00000000 00000002
		 00000008 00000015 e0000000 8000015e
		 00000000 00000002 00000008 00000015
		 f0000000 8000015f 00000000 00000002
		 00000008 00000016 00000000 80000160
		 00000000 00000002 00000008 00000016
		 10000000 80000161 00000000 00000002
		 00000008 00000016 20000000 80000162
		 00000000 00000002 00000008 00000016
		 30000000 80000163 00000000 00000002
		 00000008 00000016 40000000 80000164
		 00000000 00000002 00000008 00000016
		 50000000 80000165 00000000 00000002
		 00000008 00000016 60000000 80000166
		 00000000 00000002 00000008 00000016
		 70000000 80000167 00000000 00000002
		 00000008 00000016 80000000 80000168
		 00000000 00000002 00000008 00000016
		 90000000 80000169 00000000 00000002
		 00000008 00000016 a0000000 8000016a
		 00000000 00000002 00000008 00000016
		 b0000000 8000016b 00000000 00000002
		 00000008 00000016 c0000000 8000016c
		 00000000 00000002 00000008 00000016
		 d0000000 8000016d 00000000 00000002
		 00000008 00000016 e0000000 8000016e
		 00000000 00000002 00000008 00000016
		 f0000000 8000016f 00000000 00000002
		 00000008 00000017 00000000 80000170
		 00000000 00000002 00000008 00000017
		 10000000 80000171 00000000 00000002
		 00000008 00000017 20000000 80000172
		 00000000 00000002 00000008 00000017
		 30000000 80000173 00000000 00000002
		 00000008 00000017 40000000 80000174
		 00000000 00000002 00000008 00000017
		 50000000 80000175 00000000 00000002
		 00000008 00000017 60000000 80000176
		 00000000 00000002 00000008 00000017
		 70000000 80000177 00000000 00000002
		 00000008 00000017 80000000 80000178
		 00000000 00000002 00000008 00000017
		 90000000 80000179 00000000 00000002
		 00000008 00000017 a0000000 8000017a
		 00000000 00000002 00000008 00000017
		 b0000000 8000017b 00000000 00000002
		 00000008 00000017 c0000000 8000017c
		 00000000 00000002 00000008 00000017
		 d0000000 8000017d 00000000 00000002
		 00000008 00000017 e0000000 8000017e
		 00000000 00000002 00000008 00000017
		 f0000000 8000017f 00000000 00000002
		 00000008 00000018 00000000 80000180
		 00000000 00000002 00000008 00000018
		 10000000 80000181 00000000 00000002
		 00000008 00000018 20000000 80000182
		 00000000 00000002 00000008 00000018
		 30000000 80000183 00000000 00000002
		 00000008 00000018 40000000 80000184
		 00000000 00000002 00000008 00000018
		 50000000 80000185 00000000 00000002
		 00000008 00000018 60000000 80000186
		 00000000 00000002 00000008 00000018
		 70000000 80000187 00000000 00000002
		 00000008 00000018 80000000 80000188
		 00000000 00000002 00000008 00000018
		 90000000 80000189 00000000 00000002
		 00000008 00000018 a0000000 8000018a
		 00000000 00000002 00000008 00000018
		 b0000000 8000018b 00000000 00000002
		 00000008 00000018 c0000000 8000018c
		 00000000 00000002 00000008 00000018
		 d0000000 8000018d 00000000 00000002
		 00000008 00000018 e0000000 8000018e
		 00000000 00000002 00000008 00000018
		 f0000000 8000018f 00000000 00000002
		 00000008 00000019 00000000 80000190
		 00000000 00000002 00000008 00000019
		 10000000 80000191 00000000 00000002
		 00000008 00000019 20000000 80000192
		 00000000 00000002 00000008 00000019
		 30000000 80000193 00000000 00000002
		 00000008 00000019 40000000 80000194
		 00000000 00000002 00000008 00000019
		 50000000 80000195 00000000 00000002
		 00000008 00000019 60000000 80000196
		 00000000 00000002 00000008 00000019
		 70000000 80000197 00000000 00000002
		 00000008 00000019 80000000 80000198
		 00000000 00000002 00000008 00000019
		 90000000 80000199 00000000 00000002
		 00000008 00000019 a0000000 8000019a
		 00000000 00000002 00000008 00000019
		 b0000000 8000019b 00000000 00000002
		 00000008 00000019 c0000000 8000019c
		 00000000 00000002 00000008 00000019
		 d0000000 8000019d 00000000 00000002
		 00000008 00000019 e0000000 8000019e
		 00000000 00000002 00000008 00000019
		 f0000000 8000019f 00000000 00000002
		 00000008 0000001a 00000000 800001a0
		 00000000 00000002 00000008 0000001a
		 10000000 800001a1 00000000 00000002
		 00000008 0000001a 20000000 800001a2
		 00000000 00000002 00000008 0000001a
		 30000000 800001a3 00000000 00000002
		 00000008 0000001a 40000000 800001a4
		 00000000 00000002 00000008 0000001a
		 50000000 800001a5 00000000 00000002
		 00000008 0000001a 60000000 800001a6
		 00000000 00000002 00000008 0000001a
		 70000000 800001a7 00000000 00000002
		 00000008 0000001a 80000000 800001a8
		 00000000 00000002 00000008 0000001a
		 90000000 800001a9 00000000 00000002
		 00000008 0000001a a0000000 800001aa
		 00000000 00000002 00000008 0000001a
		 b0000000 800001ab 00000000 00000002
		 00000008 0000001a c0000000 800001ac
		 00000000 00000002 00000008 0000001a
		 d0000000 800001ad 00000000 00000002
		 00000008 0000001a e0000000 800001ae
		 00000000 00000002 00000008 0000001a
		 f0000000 800001af 00000000 00000002
		 00000008 0000001b 00000000 800001b0
		 00000000 00000002 00000008 0000001b
		 10000000 800001b1 00000000 00000002
		 00000008 0000001b 20000000 800001b2
		 00000000 00000002 00000008 0000001b
		 30000000 800001b3 00000000 00000002
		 00000008 0000001b 40000000 800001b4
		 00000000 00000002 00000008 0000001b
		 50000000 800001b5 00000000 00000002
		 00000008 0000001b 60000000 800001b6
		 00000000 00000002 00000008 0000001b
		 70000000 800001b7 00000000 00000002
		 00000008 0000001b 80000000 800001b8
		 00000000 00000002 00000008 0000001b
		 90000000 800001b9 00000000 00000002
		 00000008 0000001b a0000000 800001ba
		 00000000 00000002 00000008 0000001b
		 b0000000 800001bb 00000000 00000002
		 00000008 0000001b c0000000 800001bc
		 00000000 00000002 00000008 0000001b
		 d0000000 800001bd 00000000 00000002
		 00000008 0000001b e0000000 800001be
		 00000000 00000002 00000008 0000001b
		 f0000000 800001bf 00000000 00000002
		 00000008 0000001c 00000000 800001c0
		 00000000 00000002 00000008 0000001c
		 10000000 800001c1 00000000 00000002
		 00000008 0000001c 20000000 800001c2
		 00000000 00000002 00000008 0000001c
		 30000000 800001c3 00000000 00000002
		 00000008 0000001c 40000000 800001c4
		 00000000 00000002 00000008 0000001c
		 50000000 800001c5 00000000 00000002
		 00000008 0000001c 60000000 800001c6
		 00000000 00000002 00000008 0000001c
		 70000000 800001c7 00000000 00000002
		 00000008 0000001c 80000000 800001c8
		 00000000 00000002 00000008 0000001c
		 90000000 800001c9 00000000 00000002
		 00000008 0000001c a0000000 800001ca
		 00000000 00000002 00000008 0000001c
		 b0000000 800001cb 00000000 00000002
		 00000008 0000001c c0000000 800001cc
		 00000000 00000002 00000008 0000001c
		 d0000000 800001cd 00000000 00000002
		 00000008 0000001c e0000000 800001ce
		 00000000 00000002 00000008 0000001c
		 f0000000 800001cf 00000000 00000002
		 00000008 0000001d 00000000 800001d0
		 00000000 00000002 00000008 0000001d
		 10000000 800001d1 00000000 00000002
		 00000008 0000001d 20000000 800001d2
		 00000000 00000002 00000008 0000001d
		 30000000 800001d3 00000000 00000002
		 00000008 0000001d 40000000 800001d4
		 00000000 00000002 00000008 0000001d
		 50000000 800001d5 00000000 00000002
		 00000008 0000001d 60000000 800001d6
		 00000000 00000002 00000008 0000001d
		 70000000 800001d7 00000000 00000002
		 00000008 0000001d 80000000 800001d8
		 00000000 00000002 00000008 0000001d
		 90000000 800001d9 00000000 00000002
		 00000008 0000001d a0000000 800001da
		 00000000 00000002 00000008 0000001d
		 b0000000 800001db 00000000 00000002
		 00000008 0000001d c0000000 800001dc
		 00000000 00000002 00000008 0000001d
		 d0000000 800001dd 00000000 00000002
		 00000008 0000001d e0000000 800001de
		 00000000 00000002 00000008 0000001d
		 f0000000 800001df 00000000 00000002
		 00000008 0000001e 00000000 800001e0
		 00000000 00000002 00000008 0000001e
		 10000000 800001e1 00000000 00000002
		 00000008 0000001e 20000000 800001e2
		 00000000 00000002 00000008 0000001e
		 30000000 800001e3 00000000 00000002
		 00000008 0000001e 40000000 800001e4
		 00000000 00000002 00000008 0000001e
		 50000000 800001e5 00000000 00000002
		 00000008 0000001e 60000000 800001e6
		 00000000 00000002 00000008 0000001e
		 70000000 800001e7 00000000 00000002
		 00000008 0000001e 80000000 800001e8
		 00000000 00000002 00000008 0000001e
		 90000000 800001e9 00000000 00000002
		 00000008 0000001e a0000000 800001ea
		 00000000 00000002 00000008 0000001e
		 b0000000 800001eb 00000000 00000002
		 00000008 0000001e c0000000 800001ec
		 00000000 00000002 00000008 0000001e
		 d0000000 800001ed 00000000 00000002
		 00000008 0000001e e0000000 800001ee
		 00000000 00000002 00000008 0000001e
		 f0000000 800001ef 00000000 00000002
		 00000008 0000001f 00000000 800001f0
		 00000000 00000002 00000008 0000001f
		 10000000 800001f1 00000000 00000002
		 00000008 0000001f 20000000 800001f2
		 00000000 00000002 00000008 0000001f
		 30000000 800001f3 00000000 00000002
		 00000008 0000001f 40000000 800001f4
		 00000000 00000002 00000008 0000001f
		 50000000 800001f5 00000000 00000002
		 00000008 0000001f 60000000 800001f6
		 00000000 00000002 00000008 0000001f
		 70000000 800001f7 00000000 00000002
		 00000008 0000001f 80000000 800001f8
		 00000000 00000002 00000008 0000001f
		 90000000 800001f9 00000000 00000002
		 00000008 0000001f a0000000 800001fa
		 00000000 00000002 00000008 0000001f
		 b0000000 800001fb 00000000 00000002
		 00000008 0000001f c0000000 800001fc
		 00000000 00000002 00000008 0000001f
		 d0000000 800001fd 00000000 00000002
		 00000008 0000001f e0000000 800001fe
		 00000000 00000002 00000008 0000001f
		 f0000000 800001ff 00000000 00000002
		 00000008 00000020 00000000 80000200
		 00000000 00000002 00000008 00000020
		 10000000 80000201 00000000 00000002
		 00000008 00000020 20000000 80000202
		 00000000 00000002 00000008 00000020
		 30000000 80000203 00000000 00000002
		 00000008 00000020 40000000 80000204
		 00000000 00000002 00000008 00000020
		 50000000 80000205 00000000 00000002
		 00000008 00000020 60000000 80000206
		 00000000 00000002 00000008 00000020
		 70000000 80000207 00000000 00000002
		 00000008 00000020 80000000 80000208
		 00000000 00000002 00000008 00000020
		 90000000 80000209 00000000 00000002
		 00000008 00000020 a0000000 8000020a
		 00000000 00000002 00000008 00000020
		 b0000000 8000020b 00000000 00000002
		 00000008 00000020 c0000000 8000020c
		 00000000 00000002 00000008 00000020
		 d0000000 8000020d 00000000 00000002
		 00000008 00000020 e0000000 8000020e
		 00000000 00000002 00000008 00000020
		 f0000000 8000020f 00000000 00000002
		 00000008 00000021 00000000 80000210
		 00000000 00000002 00000008 00000021
		 10000000 80000211 00000000 00000002
		 00000008 00000021 20000000 80000212
		 00000000 00000002 00000008 00000021
		 30000000 80000213 00000000 00000002
		 00000008 00000021 40000000 80000214
		 00000000 00000002 00000008 00000021
		 50000000 80000215 00000000 00000002
		 00000008 00000021 60000000 80000216
		 00000000 00000002 00000008 00000021
		 70000000 80000217 00000000 00000002
		 00000008 00000021 80000000 80000218
		 00000000 00000002 00000008 00000021
		 90000000 80000219 00000000 00000002
		 00000008 00000021 a0000000 8000021a
		 00000000 00000002 00000008 00000021
		 b0000000 8000021b 00000000 00000002
		 00000008 00000021 c0000000 8000021c
		 00000000 00000002 00000008 00000021
		 d0000000 8000021d 00000000 00000002
		 00000008 00000021 e0000000 8000021e
		 00000000 00000002 00000008 00000021
		 f0000000 8000021f 00000000 00000002
		 00000008 00000022 00000000 80000220
		 00000000 00000002 00000008 00000022
		 10000000 80000221 00000000 00000002
		 00000008 00000022 20000000 80000222
		 00000000 00000002 00000008 00000022
		 30000000 80000223 00000000 00000002
		 00000008 00000022 40000000 80000224
		 00000000 00000002 00000008 00000022
		 50000000 80000225 00000000 00000002
		 00000008 00000022 60000000 80000226
		 00000000 00000002 00000008 00000022
		 70000000 80000227 00000000 00000002
		 00000008 00000022 80000000 80000228
		 00000000 00000002 00000008 00000022
		 90000000 80000229 00000000 00000002
		 00000008 00000022 a0000000 8000022a
		 00000000 00000002 00000008 00000022
		 b0000000 8000022b 00000000 00000002
		 00000008 00000022 c0000000 8000022c
		 00000000 00000002 00000008 00000022
		 d0000000 8000022d 00000000 00000002
		 00000008 00000022 e0000000 8000022e
		 00000000 00000002 00000008 00000022
		 f0000000 8000022f 00000000 00000002
		 00000008 00000023 00000000 80000230
		 00000000 00000002 00000008 00000023
		 10000000 80000231 00000000 00000002
		 00000008 00000023 20000000 80000232
		 00000000 00000002 00000008 00000023
		 30000000 80000233 00000000 00000002
		 00000008 00000023 40000000 80000234
		 00000000 00000002 00000008 00000023
		 50000000 80000235 00000000 00000002
		 00000008 00000023 60000000 80000236
		 00000000 00000002 00000008 00000023
		 70000000 80000237 00000000 00000002
		 00000008 00000023 80000000 80000238
		 00000000 00000002 00000008 00000023
		 90000000 80000239 00000000 00000002
		 00000008 00000023 a0000000 8000023a
		 00000000 00000002 00000008 00000023
		 b0000000 8000023b 00000000 00000002
		 00000008 00000023 c0000000 8000023c
		 00000000 00000002 00000008 00000023
		 d0000000 8000023d 00000000 00000002
		 00000008 00000023 e0000000 8000023e
		 00000000 00000002 00000008 00000023
		 f0000000 8000023f 00000000 00000002
		 00000008 00000024 00000000 80000240
		 00000000 00000002 00000008 00000024
		 10000000 80000241 00000000 00000002
		 00000008 00000024 20000000 80000242
		 00000000 00000002 00000008 00000024
		 30000000 80000243 00000000 00000002
		 00000008 00000024 40000000 80000244
		 00000000 00000002 00000008 00000024
		 50000000 80000245 00000000 00000002
		 00000008 00000024 60000000 80000246
		 00000000 00000002 00000008 00000024
		 70000000 80000247 00000000 00000002
		 00000008 00000024 80000000 80000248
		 00000000 00000002 00000008 00000024
		 90000000 80000249 00000000 00000002
		 00000008 00000024 a0000000 8000024a
		 00000000 00000002 00000008 00000024
		 b0000000 8000024b 00000000 00000002
		 00000008 00000024 c0000000 8000024c
		 00000000 00000002 00000008 00000024
		 d0000000 8000024d 00000000 00000002
		 00000008 00000024 e0000000 8000024e
		 00000000 00000002 00000008 00000024
		 f0000000 8000024f 00000000 00000002
		 00000008 00000025 00000000 80000250
		 00000000 00000002 00000008 00000025
		 10000000 80000251 00000000 00000002
		 00000008 00000025 20000000 80000252
		 00000000 00000002 00000008 00000025
		 30000000 80000253 00000000 00000002
		 00000008 00000025 40000000 80000254
		 00000000 00000002 00000008 00000025
		 50000000 80000255 00000000 00000002
		 00000008 00000025 60000000 80000256
		 00000000 00000002 00000008 00000025
		 70000000 80000257 00000000 00000002
		 00000008 00000025 80000000 80000258
		 00000000 00000002 00000008 00000025
		 90000000 80000259 00000000 00000002
		 00000008 00000025 a0000000 8000025a
		 00000000 00000002 00000008 00000025
		 b0000000 8000025b 00000000 00000002
		 00000008 00000025 c0000000 8000025c
		 00000000 00000002 00000008 00000025
		 d0000000 8000025d 00000000 00000002
		 00000008 00000025 e0000000 8000025e
		 00000000 00000002 00000008 00000025
		 f0000000 8000025f 00000000 00000002
		 00000008 00000026 00000000 80000260
		 00000000 00000002 00000008 00000026
		 10000000 80000261 00000000 00000002
		 00000008 00000026 20000000 80000262
		 00000000 00000002 00000008 00000026
		 30000000 80000263 00000000 00000002
		 00000008 00000026 40000000 80000264
		 00000000 00000002 00000008 00000026
		 50000000 80000265 00000000 00000002
		 00000008 00000026 60000000 80000266
		 00000000 00000002 00000008 00000026
		 70000000 80000267 00000000 00000002
		 00000008 00000026 80000000 80000268
		 00000000 00000002 00000008 00000026
		 90000000 80000269 00000000 00000002
		 00000008 00000026 a0000000 8000026a
		 00000000 00000002 00000008 00000026
		 b0000000 8000026b 00000000 00000003
		 00000008 00000026 c0000000 8000026c
		 00000000 00000003 00000008 00000026
		 d0000000 8000026d 00000000 00000003
		 00000008 00000026 e0000000 8000026e
		 00000000 00000003 00000008 00000026
		 f0000000 8000026f 00000000 00000003
		 00000008 00000027 00000000 80000270
		 00000000 00000003 00000008 00000027
		 10000000 80000271 00000000 00000003
		 00000008 00000027 20000000 80000272
		 00000000 00000003 00000008 00000027
		 30000000 80000273 00000000 00000003
		 00000008 00000027 40000000 80000274
		 00000000 00000003 00000008 00000027
		 50000000 80000275 00000000 00000003
		 00000008 00000027 60000000 80000276
		 00000000 00000003 00000008 00000027
		 70000000 80000277 00000000 00000003
		 00000008 00000027 80000000 80000278
		 00000000 00000003 00000008 00000027
		 90000000 80000279 00000000 00000003
		 00000008 00000027 a0000000 8000027a
		 00000000 00000003 00000008 00000027
		 b0000000 8000027b 00000000 00000003
		 00000008 00000027 c0000000 8000027c
		 00000000 00000003 00000008 00000027
		 d0000000 8000027d 00000000 00000003
		 00000008 00000027 e0000000 8000027e
		 00000000 00000003 00000008 00000027
		 f0000000 8000027f 00000000 00000003
		 00000008 00000028 00000000 80000280
		 00000000 00000003 00000008 00000028
		 10000000 80000281 00000000 00000003
		 00000008 00000028 20000000 80000282
		 00000000 00000003 00000008 00000028
		 30000000 80000283 00000000 00000003
		 00000008 00000028 40000000 80000284
		 00000000 00000003 00000008 00000028
		 50000000 80000285 00000000 00000003
		 00000008 00000028 60000000 80000286
		 00000000 00000003 00000008 00000028
		 70000000 80000287 00000000 00000003
		 00000008 00000028 80000000 80000288
		 00000000 00000003 00000008 00000028
		 90000000 80000289 00000000 00000003
		 00000008 00000028 a0000000 8000028a
		 00000000 00000003 00000008 00000028
		 b0000000 8000028b 00000000 00000003
		 00000008 00000028 c0000000 8000028c
		 00000000 00000003 00000008 00000028
		 d0000000 8000028d 00000000 00000003
		 00000008 00000028 e0000000 8000028e
		 00000000 00000003 00000008 00000028
		 f0000000 8000028f 00000000 00000003
		 00000008 00000029 00000000 80000290
		 00000000 00000003 00000008 00000029
		 10000000 80000291 00000000 00000003
		 00000008 00000029 20000000 80000292
		 00000000 00000003 00000008 00000029
		 30000000 80000293 00000000 00000003
		 00000008 00000029 40000000 80000294
		 00000000 00000003 00000008 00000029
		 50000000 80000295 00000000 00000003
		 00000008 00000029 60000000 80000296
		 00000000 00000003 00000008 00000029
		 70000000 80000297 00000000 00000003
		 00000008 00000029 80000000 80000298
		 00000000 00000003 00000008 00000029
		 90000000 80000299 00000000 00000003
		 00000008 00000029 a0000000 8000029a
		 00000000 00000003 00000008 00000029
		 b0000000 8000029b 00000000 00000003
		 00000008 00000029 c0000000 8000029c
		 00000000 00000003 00000008 00000029
		 d0000000 8000029d 00000000 00000003
		 00000008 00000029 e0000000 8000029e
		 00000000 00000003 00000008 00000029
		 f0000000 8000029f 00000000 00000003
		 00000008 0000002a 00000000 800002a0
		 00000000 00000003 00000008 0000002a
		 10000000 800002a1 00000000 00000003
		 00000008 0000002a 20000000 800002a2
		 00000000 00000003 00000008 0000002a
		 30000000 800002a3 00000000 00000003
		 00000008 0000002a 40000000 800002a4
		 00000000 00000003 00000008 0000002a
		 50000000 800002a5 00000000 00000003
		 00000008 0000002a 60000000 800002a6
		 00000000 00000003 00000008 0000002a
		 70000000 800002a7 00000000 00000003
		 00000008 0000002a 80000000 800002a8
		 00000000 00000003 00000008 0000002a
		 90000000 800002a9 00000000 00000003
		 00000008 0000002a a0000000 800002aa
		 00000000 00000003 00000008 0000002a
		 b0000000 800002ab 00000000 00000003
		 00000008 0000002a c0000000 800002ac
		 00000000 00000003 00000008 0000002a
		 d0000000 800002ad 00000000 00000003
		 00000008 0000002a e0000000 800002ae
		 00000000 00000003 00000008 0000002a
		 f0000000 800002af 00000000 00000003
		 00000008 0000002b 00000000 800002b0
		 00000000 00000003 00000008 0000002b
		 10000000 800002b1 00000000 00000003
		 00000008 0000002b 20000000 800002b2
		 00000000 00000003 00000008 0000002b
		 30000000 800002b3 00000000 00000003
		 00000008 0000002b 40000000 800002b4
		 00000000 00000003 00000008 0000002b
		 50000000 800002b5 00000000 00000003
		 00000008 0000002b 60000000 800002b6
		 00000000 00000003 00000008 0000002b
		 70000000 800002b7 00000000 00000003
		 00000008 0000002b 80000000 800002b8
		 00000000 00000003 00000008 0000002b
		 90000000 800002b9 00000000 00000003
		 00000008 0000002b a0000000 800002ba
		 00000000 00000003 00000008 0000002b
		 b0000000 800002bb 00000000 00000003
		 00000008 0000002b c0000000 800002bc
		 00000000 00000003 00000008 0000002b
		 d0000000 800002bd 00000000 00000003
		 00000008 0000002b e0000000 800002be
		 00000000 00000003 00000008 0000002b
		 f0000000 800002bf 00000000 00000003
		 00000008 0000002c 00000000 800002c0
		 00000000 00000003 00000008 0000002c
		 10000000 800002c1 00000000 00000003
		 00000008 0000002c 20000000 800002c2
		 00000000 00000003 00000008 0000002c
		 30000000 800002c3 00000000 00000003
		 00000008 0000002c 40000000 800002c4
		 00000000 00000003 00000008 0000002c
		 50000000 800002c5 00000000 00000003
		 00000008 0000002c 60000000 800002c6
		 00000000 00000003 00000008 0000002c
		 70000000 800002c7 00000000 00000003
		 00000008 0000002c 80000000 800002c8
		 00000000 00000003 00000008 0000002c
		 90000000 800002c9 00000000 00000003
		 00000008 0000002c a0000000 800002ca
		 00000000 00000003 00000008 0000002c
		 b0000000 800002cb 00000000 00000003
		 00000008 0000002c c0000000 800002cc
		 00000000 00000003 00000008 0000002c
		 d0000000 800002cd 00000000 00000003
		 00000008 0000002c e0000000 800002ce
		 00000000 00000003 00000008 0000002c
		 f0000000 800002cf 00000000 00000003
		 00000008 0000002d 00000000 800002d0
		 00000000 00000003 00000008 0000002d
		 10000000 800002d1 00000000 00000003
		 00000008 0000002d 20000000 800002d2
		 00000000 00000003 00000008 0000002d
		 30000000 800002d3 00000000 00000003
		 00000008 0000002d 40000000 800002d4
		 00000000 00000003 00000008 0000002d
		 50000000 800002d5 00000000 00000003
		 00000008 0000002d 60000000 800002d6
		 00000000 00000003 00000008 0000002d
		 70000000 800002d7 00000000 00000003
		 00000008 0000002d 80000000 800002d8
		 00000000 00000003 00000008 0000002d
		 90000000 800002d9 00000000 00000003
		 00000008 0000002d a0000000 800002da
		 00000000 00000003 00000008 0000002d
		 b0000000 800002db 00000000 00000003
		 00000008 0000002d c0000000 800002dc
		 00000000 00000003 00000008 0000002d
		 d0000000 800002dd 00000000 00000003
		 00000008 0000002d e0000000 800002de
		 00000000 00000003 00000008 0000002d
		 f0000000 800002df 00000000 00000003
		 00000008 0000002e 00000000 800002e0
		 00000000 00000003 00000008 0000002e
		 10000000 800002e1 00000000 00000003
		 00000008 0000002e 20000000 800002e2
		 00000000 00000003 00000008 0000002e
		 30000000 800002e3 00000000 00000003
		 00000008 0000002e 40000000 800002e4
		 00000000 00000003 00000008 0000002e
		 50000000 800002e5 00000000 00000003
		 00000008 0000002e 60000000 800002e6
		 00000000 00000003 00000008 0000002e
		 70000000 800002e7 00000000 00000003
		 00000008 0000002e 80000000 800002e8
		 00000000 00000003 00000008 0000002e
		 90000000 800002e9 00000000 00000003
		 00000008 0000002e a0000000 800002ea
		 00000000 00000003 00000008 0000002e
		 b0000000 800002eb 00000000 00000003
		 00000008 0000002e c0000000 800002ec
		 00000000 00000003 00000008 0000002e
		 d0000000 800002ed 00000000 00000003
		 00000008 0000002e e0000000 800002ee
		 00000000 00000003 00000008 0000002e
		 f0000000 800002ef 00000000 00000003
		 00000008 0000002f 00000000 800002f0
		 00000000 00000003 00000008 0000002f
		 10000000 800002f1 00000000 00000003
		 00000008 0000002f 20000000 800002f2
		 00000000 00000003 00000008 0000002f
		 30000000 800002f3 00000000 00000003
		 00000008 0000002f 40000000 800002f4
		 00000000 00000003 00000008 0000002f
		 50000000 800002f5 00000000 00000003
		 00000008 0000002f 60000000 800002f6
		 00000000 00000003 00000008 0000002f
		 70000000 800002f7 00000000 00000003
		 00000008 0000002f 80000000 800002f8
		 00000000 00000003 00000008 0000002f
		 90000000 800002f9 00000000 00000003
		 00000008 0000002f a0000000 800002fa
		 00000000 00000003 00000008 0000002f
		 b0000000 800002fb 00000000 00000003
		 00000008 0000002f c0000000 800002fc
		 00000000 00000003 00000008 0000002f
		 d0000000 800002fd 00000000 00000003
		 00000008 0000002f e0000000 800002fe
		 00000000 00000003 00000008 0000002f
		 f0000000 800002ff 00000000 00000003
		 00000008 00000030 00000000 80000300
		 00000000 00000003 00000008 00000030
		 10000000 80000301 00000000 00000003
		 00000008 00000030 20000000 80000302
		 00000000 00000003 00000008 00000030
		 30000000 80000303 00000000 00000003
		 00000008 00000030 40000000 80000304
		 00000000 00000003 00000008 00000030
		 50000000 80000305 00000000 00000003
		 00000008 00000030 60000000 80000306
		 00000000 00000003 00000008 00000030
		 70000000 80000307 00000000 00000003
		 00000008 00000030 80000000 80000308
		 00000000 00000003 00000008 00000030
		 90000000 80000309 00000000 00000003
		 00000008 00000030 a0000000 8000030a
		 00000000 00000003 00000008 00000030
		 b0000000 8000030b 00000000 00000003
		 00000008 00000030 c0000000 8000030c
		 00000000 00000003 00000008 00000030
		 d0000000 8000030d 00000000 00000003
		 00000008 00000030 e0000000 8000030e
		 00000000 00000003 00000008 00000030
		 f0000000 8000030f 00000000 00000003
		 00000008 00000031 00000000 80000310
		 00000000 00000003 00000008 00000031
		 10000000 80000311 00000000 00000003
		 00000008 00000031 20000000 80000312
		 00000000 00000003 00000008 00000031
		 30000000 80000313 00000000 00000003
		 00000008 00000031 40000000 80000314
		 00000000 00000003 00000008 00000031
		 50000000 80000315 00000000 00000003
		 00000008 00000031 60000000 80000316
		 00000000 00000003 00000008 00000031
		 70000000 80000317 00000000 00000003
		 00000008 00000031 80000000 80000318
		 00000000 00000003 00000008 00000031
		 90000000 80000319 00000000 00000003
		 00000008 00000031 a0000000 8000031a
		 00000000 00000003 00000008 00000031
		 b0000000 8000031b 00000000 00000003
		 00000008 00000031 c0000000 8000031c
		 00000000 00000003 00000008 00000031
		 d0000000 8000031d 00000000 00000003
		 00000008 00000031 e0000000 8000031e
		 00000000 00000003 00000008 00000031
		 f0000000 8000031f 00000000 00000003
		 00000008 00000032 00000000 80000320
		 00000000 00000003 00000008 00000032
		 10000000 80000321 00000000 00000003
		 00000008 00000032 20000000 80000322
		 00000000 00000003 00000008 00000032
		 30000000 80000323 00000000 00000003
		 00000008 00000032 40000000 80000324
		 00000000 00000003 00000008 00000032
		 50000000 80000325 00000000 00000003
		 00000008 00000032 60000000 80000326
		 00000000 00000003 00000008 00000032
		 70000000 80000327 00000000 00000003
		 00000008 00000032 80000000 80000328
		 00000000 00000003 00000008 00000032
		 90000000 80000329 00000000 00000003
		 00000008 00000032 a0000000 8000032a
		 00000000 00000003 00000008 00000032
		 b0000000 8000032b 00000000 00000003
		 00000008 00000032 c0000000 8000032c
		 00000000 00000003 00000008 00000032
		 d0000000 8000032d 00000000 00000003
		 00000008 00000032 e0000000 8000032e
		 00000000 00000003 00000008 00000032
		 f0000000 8000032f 00000000 00000003
		 00000008 00000033 00000000 80000330
		 00000000 00000003 00000008 00000033
		 10000000 80000331 00000000 00000003
		 00000008 00000033 20000000 80000332
		 00000000 00000003 00000008 00000033
		 30000000 80000333 00000000 00000003
		 00000008 00000033 40000000 80000334
		 00000000 00000003 00000008 00000033
		 50000000 80000335 00000000 00000003
		 00000008 00000033 60000000 80000336
		 00000000 00000003 00000008 00000033
		 70000000 80000337 00000000 00000003
		 00000008 00000033 80000000 80000338
		 00000000 00000003 00000008 00000033
		 90000000 80000339 00000000 00000003
		 00000008 00000033 a0000000 8000033a
		 00000000 00000003 00000008 00000033
		 b0000000 8000033b 00000000 00000003
		 00000008 00000033 c0000000 8000033c
		 00000000 00000003 00000008 00000033
		 d0000000 8000033d 00000000 00000003
		 00000008 00000033 e0000000 8000033e
		 00000000 00000003 00000008 00000033
		 f0000000 8000033f 00000000 00000003
		 00000008 00000034 00000000 80000340
		 00000000 00000003 00000008 00000034
		 10000000 80000341 00000000 00000003
		 00000008 00000034 20000000 80000342
		 00000000 00000003 00000008 00000034
		 30000000 80000343 00000000 00000003
		 00000008 00000034 40000000 80000344
		 00000000 00000003 00000008 00000034
		 50000000 80000345 00000000 00000003
		 00000008 00000034 60000000 80000346
		 00000000 00000003 00000008 00000034
		 70000000 80000347 00000000 00000003
		 00000008 00000034 80000000 80000348
		 00000000 00000003 00000008 00000034
		 90000000 80000349 00000000 00000003
		 00000008 00000034 a0000000 8000034a
		 00000000 00000003 00000008 00000034
		 b0000000 8000034b 00000000 00000003
		 00000008 00000034 c0000000 8000034c
		 00000000 00000003 00000008 00000034
		 d0000000 8000034d 00000000 00000003
		 00000008 00000034 e0000000 8000034e
		 00000000 00000003 00000008 00000034
		 f0000000 8000034f 00000000 00000003
		 00000008 00000035 00000000 80000350
		 00000000 00000003 00000008 00000035
		 10000000 80000351 00000000 00000003
		 00000008 00000035 20000000 80000352
		 00000000 00000003 00000008 00000035
		 30000000 80000353 00000000 00000003
		 00000008 00000035 40000000 80000354
		 00000000 00000003 00000008 00000035
		 50000000 80000355 00000000 00000003
		 00000008 00000035 60000000 80000356
		 00000000 00000003 00000008 00000035
		 70000000 80000357 00000000 00000003
		 00000008 00000035 80000000 80000358
		 00000000 00000003 00000008 00000035
		 90000000 80000359 00000000 00000003
		 00000008 00000035 a0000000 8000035a
		 00000000 00000003 00000008 00000035
		 b0000000 8000035b 00000000 00000003
		 00000008 00000035 c0000000 8000035c
		 00000000 00000003 00000008 00000035
		 d0000000 8000035d 00000000 00000003
		 00000008 00000035 e0000000 8000035e
		 00000000 00000003 00000008 00000035
		 f0000000 8000035f 00000000 00000003
		 00000008 00000036 00000000 80000360
		 00000000 00000003 00000008 00000036
		 10000000 80000361 00000000 00000003
		 00000008 00000036 20000000 80000362
		 00000000 00000003 00000008 00000036
		 30000000 80000363 00000000 00000003
		 00000008 00000036 40000000 80000364
		 00000000 00000003 00000008 00000036
		 50000000 80000365 00000000 00000003
		 00000008 00000036 60000000 80000366
		 00000000 00000003 00000008 00000036
		 70000000 80000367 00000000 00000003
		 00000008 00000036 80000000 80000368
		 00000000 00000003 00000008 00000036
		 90000000 80000369 00000000 00000003
		 00000008 00000036 a0000000 8000036a
		 00000000 00000003 00000008 00000036
		 b0000000 8000036b 00000000 00000003
		 00000008 00000036 c0000000 8000036c
		 00000000 00000003 00000008 00000036
		 d0000000 8000036d 00000000 00000003
		 00000008 00000036 e0000000 8000036e
		 00000000 00000003 00000008 00000036
		 f0000000 8000036f 00000000 00000003
		 00000008 00000037 00000000 80000370
		 00000000 00000003 00000008 00000037
		 10000000 80000371 00000000 00000003
		 00000008 00000037 20000000 80000372
		 00000000 00000003 00000008 00000037
		 30000000 80000373 00000000 00000003
		 00000008 00000037 40000000 80000374
		 00000000 00000003 00000008 00000037
		 50000000 80000375 00000000 00000003
		 00000008 00000037 60000000 80000376
		 00000000 00000003 00000008 00000037
		 70000000 80000377 00000000 00000003
		 00000008 00000037 80000000 80000378
		 00000000 00000003 00000008 00000037
		 90000000 80000379 00000000 00000003
		 00000008 00000037 a0000000 8000037a
		 00000000 00000003 00000008 00000037
		 b0000000 8000037b 00000000 00000003
		 00000008 00000037 c0000000 8000037c
		 00000000 00000003 00000008 00000037
		 d0000000 8000037d 00000000 00000003
		 00000008 00000037 e0000000 8000037e
		 00000000 00000003 00000008 00000037
		 f0000000 8000037f 00000000 00000003
		 00000008 00000038 00000000 80000380
		 00000000 00000003 00000008 00000038
		 10000000 80000381 00000000 00000003
		 00000008 00000038 20000000 80000382
		 00000000 00000003 00000008 00000038
		 30000000 80000383 00000000 00000003
		 00000008 00000038 40000000 80000384
		 00000000 00000003 00000008 00000038
		 50000000 80000385 00000000 00000003
		 00000008 00000038 60000000 80000386
		 00000000 00000003 00000008 00000038
		 70000000 80000387 00000000 00000003
		 00000008 00000038 80000000 80000388
		 00000000 00000003 00000008 00000038
		 90000000 80000389 00000000 00000003
		 00000008 00000038 a0000000 8000038a
		 00000000 00000003 00000008 00000038
		 b0000000 8000038b 00000000 00000003
		 00000008 00000038 c0000000 8000038c
		 00000000 00000003 00000008 00000038
		 d0000000 8000038d 00000000 00000003
		 00000008 00000038 e0000000 8000038e
		 00000000 00000003 00000008 00000038
		 f0000000 8000038f 00000000 00000003
		 00000008 00000039 00000000 80000390
		 00000000 00000003 00000008 00000039
		 10000000 80000391 00000000 00000003
		 00000008 00000039 20000000 80000392
		 00000000 00000003 00000008 00000039
		 30000000 80000393 00000000 00000003
		 00000008 00000039 40000000 80000394
		 00000000 00000003 00000008 00000039
		 50000000 80000395 00000000 00000003
		 00000008 00000039 60000000 80000396
		 00000000 00000003 00000008 00000039
		 70000000 80000397 00000000 00000003
		 00000008 00000039 80000000 80000398
		 00000000 00000003 00000008 00000039
		 90000000 80000399 00000000 00000003
		 00000008 00000039 a0000000 8000039a
		 00000000 00000003 00000008 00000039
		 b0000000 8000039b 00000000 00000003
		 00000008 00000039 c0000000 8000039c
		 00000000 00000003 00000008 00000039
		 d0000000 8000039d 00000000 00000003
		 00000008 00000039 e0000000 8000039e
		 00000000 00000003 00000008 00000039
		 f0000000 8000039f 00000000 00000003
		 00000008 0000003a 00000000 800003a0
		 00000000 00000003 00000008 0000003a
		 10000000 800003a1 00000000 00000003
		 00000008 0000003a 20000000 800003a2
		 00000000 00000003 00000008 0000003a
		 30000000 800003a3 00000000 00000003
		 00000008 0000003a 40000000 800003a4
		 00000000 00000003 00000008 0000003a
		 50000000 800003a5 00000000 00000003
		 00000008 0000003a 60000000 800003a6
		 00000000 00000003 00000008 0000003a
		 70000000 800003a7 00000000 00000003
		 00000008 0000003a 80000000 800003a8
		 00000000 00000003 00000008 0000003a
		 90000000 800003a9 00000000 00000003
		 00000008 0000003a a0000000 800003aa
		 00000000 00000003 00000008 0000003a
		 b0000000 800003ab 00000000 00000003
		 00000008 0000003a c0000000 800003ac
		 00000000 00000003 00000008 0000003a
		 d0000000 800003ad 00000000 00000003
		 00000008 0000003a e0000000 800003ae
		 00000000 00000003 00000008 0000003a
		 f0000000 800003af 00000000 00000003
		 00000008 0000003b 00000000 800003b0
		 00000000 00000003 00000008 0000003b
		 10000000 800003b1 00000000 00000003
		 00000008 0000003b 20000000 800003b2
		 00000000 00000003 00000008 0000003b
		 30000000 800003b3 00000000 00000003
		 00000008 0000003b 40000000 800003b4
		 00000000 00000003 00000008 0000003b
		 50000000 800003b5 00000000 00000003
		 00000008 0000003b 60000000 800003b6
		 00000000 00000003 00000008 0000003b
		 70000000 800003b7 00000000 00000003
		 00000008 0000003b 80000000 800003b8
		 00000000 00000003 00000008 0000003b
		 90000000 800003b9 00000000 00000003
		 00000008 0000003b a0000000 800003ba
		 00000000 00000003 00000008 0000003b
		 b0000000 800003bb 00000000 00000003
		 00000008 0000003b c0000000 800003bc
		 00000000 00000003 00000008 0000003b
		 d0000000 800003bd 00000000 00000003
		 00000008 0000003b e0000000 800003be
		 00000000 00000003 00000008 0000003b
		 f0000000 800003bf 00000000 00000003
		 00000008 0000003c 00000000 800003c0
		 00000000 ffffffff 00000000 0000003c
		 10000000 800003c1 00000000 ffffffff
		 00000000 0000003c 20000000 800003c2
		 00000000 ffffffff 00000000 0000003c
		 30000000 800003c3 00000000 ffffffff
		 00000000 0000003c 40000000 800003c4
		 00000000 ffffffff 00000000 0000003c
		 50000000 800003c5 00000000 ffffffff
		 00000000 0000003c 60000000 800003c6
		 00000000 ffffffff 00000000 0000003c
		 70000000 800003c7 00000000 ffffffff
		 00000000 0000003c 80000000 800003c8
		 00000000 ffffffff 00000000 0000003c
		 90000000 800003c9 00000000 ffffffff
		 00000000 0000003c a0000000 800003ca
		 00000000 ffffffff 00000000 0000003c
		 b0000000 800003cb 00000000 ffffffff
		 00000000 0000003c c0000000 800003cc
		 00000000 ffffffff 00000000 0000003c
		 d0000000 800003cd 00000000 ffffffff
		 00000000 0000003c e0000000 800003ce
		 00000000 ffffffff 00000000 0000003c
		 f0000000 800003cf 00000000 ffffffff
		 00000000 0000003d 00000000 800003d0
		 00000000 ffffffff 00000000 0000003d
		 10000000 800003d1 00000000 ffffffff
		 00000000 0000003d 20000000 800003d2
		 00000000 ffffffff 00000000 0000003d
		 30000000 800003d3 00000000 ffffffff
		 00000000 0000003d 40000000 800003d4
		 00000000 ffffffff 00000000 0000003d
		 50000000 800003d5 00000000 ffffffff
		 00000000 0000003d 60000000 800003d6
		 00000000 ffffffff 00000000 0000003d
		 70000000 800003d7 00000000 ffffffff
		 00000000 0000003d 80000000 800003d8
		 00000000 ffffffff 00000000 0000003d
		 90000000 800003d9 00000000 ffffffff
		 00000000 0000003d a0000000 800003da
		 00000000 ffffffff 00000000 0000003d
		 b0000000 800003db 00000000 ffffffff
		 00000000 0000003d c0000000 800003dc
		 00000000 ffffffff 00000000 0000003d
		 d0000000 800003dd 00000000 ffffffff
		 00000000 0000003d e0000000 800003de
		 00000000 ffffffff 00000000 0000003d
		 f0000000 800003df 00000000 ffffffff
		 00000000 0000003e 00000000 800003e0
		 00000000 ffffffff 00000000 0000003e
		 10000000 800003e1 00000000 ffffffff
		 00000000 0000003e 20000000 800003e2
		 00000000 ffffffff 00000000 0000003e
		 30000000 800003e3 00000000 ffffffff
		 00000000 0000003e 40000000 800003e4
		 00000000 ffffffff 00000000 0000003e
		 50000000 800003e5 00000000 ffffffff
		 00000000 0000003e 60000000 800003e6
		 00000000 ffffffff 00000000 0000003e
		 70000000 800003e7 00000000 ffffffff
		 00000000 0000003e 80000000 800003e8
		 00000000 ffffffff 00000000 0000003e
		 90000000 800003e9 00000000 ffffffff
		 00000000 0000003e a0000000 800003ea
		 00000000 ffffffff 00000000 0000003e
		 b0000000 800003eb 00000000 ffffffff
		 00000000 0000003e c0000000 800003ec
		 00000000 ffffffff 00000000 0000003e
		 d0000000 800003ed 00000000 ffffffff
		 00000000 0000003e e0000000 800003ee
		 00000000 ffffffff 00000000 0000003e
		 f0000000 800003ef 00000000 ffffffff
		 00000000 0000003f 00000000 800003f0
		 00000000 ffffffff 00000000 0000003f
		 10000000 800003f1 00000000 ffffffff
		 00000000 0000003f 20000000 800003f2
		 00000000 ffffffff 00000000 0000003f
		 30000000 800003f3 00000000 ffffffff
		 00000000 0000003f 40000000 800003f4
		 00000000 ffffffff 00000000 0000003f
		 50000000 800003f5 00000000 ffffffff
		 00000000 0000003f 60000000 800003f6
		 00000000 ffffffff 00000000 0000003f
		 70000000 800003f7 00000000 ffffffff
		 00000000 0000003f 80000000 800003f8
		 00000000 ffffffff 00000000 0000003f
		 90000000 800003f9 00000000 ffffffff
		 00000000 0000003f a0000000 800003fa
		 00000000 ffffffff 00000000 0000003f
		 b0000000 800003fb 00000000 ffffffff
		 00000000 0000003f c0000000 800003fc
		 00000000 ffffffff 00000000 0000003f
		 d0000000 800003fd 00000000 ffffffff
		 00000000 0000003f e0000000 800003fe
		 00000000 ffffffff 00000000 0000003f
		 f0000000 800003ff 00000000 ffffffff
		 00000000 00000040 00000000 80000400
		 00000000 ffffffff 00000000 00000040
		 10000000 80000401 00000000 ffffffff
		 00000000 00000040 20000000 80000402
		 00000000 ffffffff 00000000 00000040
		 30000000 80000403 00000000 ffffffff
		 00000000 00000040 40000000 80000404
		 00000000 ffffffff 00000000 00000040
		 50000000 80000405 00000000 ffffffff
		 00000000 00000040 60000000 80000406
		 00000000 ffffffff 00000000 00000040
		 70000000 80000407 00000000 ffffffff
		 00000000 00000040 80000000 80000408
		 00000000 ffffffff 00000000 00000040
		 90000000 80000409 00000000 ffffffff
		 00000000 00000040 a0000000 8000040a
		 00000000 ffffffff 00000000 00000040
		 b0000000 8000040b 00000000 ffffffff
		 00000000 00000040 c0000000 8000040c
		 00000000 ffffffff 00000000 00000040
		 d0000000 8000040d 00000000 ffffffff
		 00000000 00000040 e0000000 8000040e
		 00000000 ffffffff 00000000 00000040
		 f0000000 8000040f 00000000 ffffffff
		 00000000 00000041 00000000 80000410
		 00000000 ffffffff 00000000 00000041
		 10000000 80000411 00000000 ffffffff
		 00000000 00000041 20000000 80000412
		 00000000 ffffffff 00000000 00000041
		 30000000 80000413 00000000 ffffffff
		 00000000 00000041 40000000 80000414
		 00000000 ffffffff 00000000 00000041
		 50000000 80000415 00000000 ffffffff
		 00000000 00000041 60000000 80000416
		 00000000 ffffffff 00000000 00000041
		 70000000 80000417 00000000 ffffffff
		 00000000 00000041 80000000 80000418
		 00000000 ffffffff 00000000 00000041
		 90000000 80000419 00000000 ffffffff
		 00000000 00000041 a0000000 8000041a
		 00000000 ffffffff 00000000 00000041
		 b0000000 8000041b 00000000 ffffffff
		 00000000 00000041 c0000000 8000041c
		 00000000 ffffffff 00000000 00000041
		 d0000000 8000041d 00000000 ffffffff
		 00000000 00000041 e0000000 8000041e
		 00000000 ffffffff 00000000 00000041
		 f0000000 8000041f 00000000 ffffffff
		 00000000 00000042 00000000 80000420
		 00000000 ffffffff 00000000 00000042
		 10000000 80000421 00000000 ffffffff
		 00000000 00000042 20000000 80000422
		 00000000 ffffffff 00000000 00000042
		 30000000 80000423 00000000 ffffffff
		 00000000 00000042 40000000 80000424
		 00000000 ffffffff 00000000 00000042
		 50000000 80000425 00000000 ffffffff
		 00000000 00000042 60000000 80000426
		 00000000 ffffffff 00000000 00000042
		 70000000 80000427 00000000 ffffffff
		 00000000 00000042 80000000 80000428
		 00000000 ffffffff 00000000 00000042
		 90000000 80000429 00000000 ffffffff
		 00000000 00000042 a0000000 8000042a
		 00000000 ffffffff 00000000 00000042
		 b0000000 8000042b 00000000 ffffffff
		 00000000 00000042 c0000000 8000042c
		 00000000 ffffffff 00000000 00000042
		 d0000000 8000042d 00000000 ffffffff
		 00000000 00000042 e0000000 8000042e
		 00000000 ffffffff 00000000 00000042
		 f0000000 8000042f 00000000 ffffffff
		 00000000 00000043 00000000 80000430
		 00000000 ffffffff 00000000 00000043
		 10000000 80000431 00000000 ffffffff
		 00000000 00000043 20000000 80000432
		 00000000 ffffffff 00000000 00000043
		 30000000 80000433 00000000 ffffffff
		 00000000 00000043 40000000 80000434
		 00000000 ffffffff 00000000 00000043
		 50000000 80000435 00000000 ffffffff
		 00000000 00000043 60000000 80000436
		 00000000 ffffffff 00000000 00000043
		 70000000 80000437 00000000 ffffffff
		 00000000 00000043 80000000 80000438
		 00000000 ffffffff 00000000 00000043
		 90000000 80000439 00000000 ffffffff
		 00000000 00000043 a0000000 8000043a
		 00000000 ffffffff 00000000 00000043
		 b0000000 8000043b 00000000 ffffffff
		 00000000 00000043 c0000000 8000043c
		 00000000 ffffffff 00000000 00000043
		 d0000000 8000043d 00000000 ffffffff
		 00000000 00000043 e0000000 8000043e
		 00000000 ffffffff 00000000 00000043
		 f0000000 8000043f 00000000 ffffffff
		 00000000 00000044 00000000 80000440
		 00000000 ffffffff 00000000 00000044
		 10000000 80000441 00000000 ffffffff
		 00000000 00000044 20000000 80000442
		 00000000 ffffffff 00000000 00000044
		 30000000 80000443 00000000 ffffffff
		 00000000 00000044 40000000 80000444
		 00000000 ffffffff 00000000 00000044
		 50000000 80000445 00000000 ffffffff
		 00000000 00000044 60000000 80000446
		 00000000 ffffffff 00000000 00000044
		 70000000 80000447 00000000 ffffffff
		 00000000 00000044 80000000 80000448
		 00000000 ffffffff 00000000 00000044
		 90000000 80000449 00000000 ffffffff
		 00000000 00000044 a0000000 8000044a
		 00000000 ffffffff 00000000 00000044
		 b0000000 8000044b 00000000 ffffffff
		 00000000 00000044 c0000000 8000044c
		 00000000 ffffffff 00000000 00000044
		 d0000000 8000044d 00000000 ffffffff
		 00000000 00000044 e0000000 8000044e
		 00000000 ffffffff 00000000 00000044
		 f0000000 8000044f 00000000 ffffffff
		 00000000 00000045 00000000 80000450
		 00000000 ffffffff 00000000 00000045
		 10000000 80000451 00000000 ffffffff
		 00000000 00000045 20000000 80000452
		 00000000 ffffffff 00000000 00000045
		 30000000 80000453 00000000 ffffffff
		 00000000 00000045 40000000 80000454
		 00000000 ffffffff 00000000 00000045
		 50000000 80000455 00000000 ffffffff
		 00000000 00000045 60000000 80000456
		 00000000 ffffffff 00000000 00000045
		 70000000 80000457 00000000 ffffffff
		 00000000 00000045 80000000 80000458
		 00000000 ffffffff 00000000 00000045
		 90000000 80000459 00000000 ffffffff
		 00000000 00000045 a0000000 8000045a
		 00000000 ffffffff 00000000 00000045
		 b0000000 8000045b 00000000 ffffffff
		 00000000 00000045 c0000000 8000045c
		 00000000 ffffffff 00000000 00000045
		 d0000000 8000045d 00000000 ffffffff
		 00000000 00000045 e0000000 8000045e
		 00000000 ffffffff 00000000 00000045
		 f0000000 8000045f 00000000 ffffffff
		 00000000 00000046 00000000 80000460
		 00000000 ffffffff 00000000 00000046
		 10000000 80000461 00000000 ffffffff
		 00000000 00000046 20000000 80000462
		 00000000 ffffffff 00000000 00000046
		 30000000 80000463 00000000 ffffffff
		 00000000 00000046 40000000 80000464
		 00000000 ffffffff 00000000 00000046
		 50000000 80000465 00000000 ffffffff
		 00000000 00000046 60000000 80000466
		 00000000 ffffffff 00000000 00000046
		 70000000 80000467 00000000 ffffffff
		 00000000 00000046 80000000 80000468
		 00000000 ffffffff 00000000 00000046
		 90000000 80000469 00000000 ffffffff
		 00000000 00000046 a0000000 8000046a
		 00000000 ffffffff 00000000 00000046
		 b0000000 8000046b 00000000 ffffffff
		 00000000 00000046 c0000000 8000046c
		 00000000 ffffffff 00000000 00000046
		 d0000000 8000046d 00000000 ffffffff
		 00000000 00000046 e0000000 8000046e
		 00000000 ffffffff 00000000 00000046
		 f0000000 8000046f 00000000 ffffffff
		 00000000 00000047 00000000 80000470
		 00000000 ffffffff 00000000 00000047
		 10000000 80000471 00000000 ffffffff
		 00000000 00000047 20000000 80000472
		 00000000 ffffffff 00000000 00000047
		 30000000 80000473 00000000 ffffffff
		 00000000 00000047 40000000 80000474
		 00000000 ffffffff 00000000 00000047
		 50000000 80000475 00000000 ffffffff
		 00000000 00000047 60000000 80000476
		 00000000 ffffffff 00000000 00000047
		 70000000 80000477 00000000 ffffffff
		 00000000 00000047 80000000 80000478
		 00000000 ffffffff 00000000 00000047
		 90000000 80000479 00000000 ffffffff
		 00000000 00000047 a0000000 8000047a
		 00000000 ffffffff 00000000 00000047
		 b0000000 8000047b 00000000 ffffffff
		 00000000 00000047 c0000000 8000047c
		 00000000 ffffffff 00000000 00000047
		 d0000000 8000047d 00000000 ffffffff
		 00000000 00000047 e0000000 8000047e
		 00000000 ffffffff 00000000 00000047
		 f0000000 8000047f 00000000 ffffffff
		 00000000 00000048 00000000 80000480
		 00000000 ffffffff 00000000 00000048
		 10000000 80000481 00000000 ffffffff
		 00000000 00000048 20000000 80000482
		 00000000 ffffffff 00000000 00000048
		 30000000 80000483 00000000 ffffffff
		 00000000 00000048 40000000 80000484
		 00000000 ffffffff 00000000 00000048
		 50000000 80000485 00000000 ffffffff
		 00000000 00000048 60000000 80000486
		 00000000 ffffffff 00000000 00000048
		 70000000 80000487 00000000 ffffffff
		 00000000 00000048 80000000 80000488
		 00000000 ffffffff 00000000 00000048
		 90000000 80000489 00000000 ffffffff
		 00000000 00000048 a0000000 8000048a
		 00000000 ffffffff 00000000 00000048
		 b0000000 8000048b 00000000 ffffffff
		 00000000 00000048 c0000000 8000048c
		 00000000 ffffffff 00000000 00000048
		 d0000000 8000048d 00000000 ffffffff
		 00000000 00000048 e0000000 8000048e
		 00000000 ffffffff 00000000 00000048
		 f0000000 8000048f 00000000 ffffffff
		 00000000 00000049 00000000 80000490
		 00000000 ffffffff 00000000 00000049
		 10000000 80000491 00000000 ffffffff
		 00000000 00000049 20000000 80000492
		 00000000 ffffffff 00000000 00000049
		 30000000 80000493 00000000 ffffffff
		 00000000 00000049 40000000 80000494
		 00000000 ffffffff 00000000 00000049
		 50000000 80000495 00000000 ffffffff
		 00000000 00000049 60000000 80000496
		 00000000 ffffffff 00000000 00000049
		 70000000 80000497 00000000 ffffffff
		 00000000 00000049 80000000 80000498
		 00000000 ffffffff 00000000 00000049
		 90000000 80000499 00000000 ffffffff
		 00000000 00000049 a0000000 8000049a
		 00000000 ffffffff 00000000 00000049
		 b0000000 8000049b 00000000 ffffffff
		 00000000 00000049 c0000000 8000049c
		 00000000 ffffffff 00000000 00000049
		 d0000000 8000049d 00000000 ffffffff
		 00000000 00000049 e0000000 8000049e
		 00000000 ffffffff 00000000 00000049
		 f0000000 8000049f 00000000 ffffffff
		 00000000 0000004a 00000000 800004a0
		 00000000 ffffffff 00000000 0000004a
		 10000000 800004a1 00000000 ffffffff
		 00000000 0000004a 20000000 800004a2
		 00000000 ffffffff 00000000 0000004a
		 30000000 800004a3 00000000 ffffffff
		 00000000 0000004a 40000000 800004a4
		 00000000 ffffffff 00000000 0000004a
		 50000000 800004a5 00000000 ffffffff
		 00000000 0000004a 60000000 800004a6
		 00000000 ffffffff 00000000 0000004a
		 70000000 800004a7 00000000 ffffffff
		 00000000 0000004a 80000000 800004a8
		 00000000 ffffffff 00000000 0000004a
		 90000000 800004a9 00000000 ffffffff
		 00000000 0000004a a0000000 800004aa
		 00000000 ffffffff 00000000 0000004a
		 b0000000 800004ab 00000000 ffffffff
		 00000000 0000004a c0000000 800004ac
		 00000000 ffffffff 00000000 0000004a
		 d0000000 800004ad 00000000 ffffffff
		 00000000 0000004a e0000000 800004ae
		 00000000 ffffffff 00000000 0000004a
		 f0000000 800004af 00000000 ffffffff
		 00000000 0000004b 00000000 800004b0
		 00000000 ffffffff 00000000 0000004b
		 10000000 800004b1 00000000 ffffffff
		 00000000 0000004b 20000000 800004b2
		 00000000 ffffffff 00000000 0000004b
		 30000000 800004b3 00000000 ffffffff
		 00000000 0000004b 40000000 800004b4
		 00000000 ffffffff 00000000 0000004b
		 50000000 800004b5 00000000 ffffffff
		 00000000 0000004b 60000000 800004b6
		 00000000 ffffffff 00000000 0000004b
		 70000000 800004b7 00000000 ffffffff
		 00000000 0000004b 80000000 800004b8
		 00000000 ffffffff 00000000 0000004b
		 90000000 800004b9 00000000 ffffffff
		 00000000 0000004b a0000000 800004ba
		 00000000 ffffffff 00000000 0000004b
		 b0000000 800004bb 00000000 ffffffff
		 00000000 0000004b c0000000 800004bc
		 00000000 ffffffff 00000000 0000004b
		 d0000000 800004bd 00000000 ffffffff
		 00000000 0000004b e0000000 800004be
		 00000000 ffffffff 00000000 0000004b
		 f0000000 800004bf 00000000 ffffffff
		 00000000 0000004c 00000000 800004c0
		 00000000 ffffffff 00000000 0000004c
		 10000000 800004c1 00000000 ffffffff
		 00000000 0000004c 20000000 800004c2
		 00000000 ffffffff 00000000 0000004c
		 30000000 800004c3 00000000 ffffffff
		 00000000 0000004c 40000000 800004c4
		 00000000 ffffffff 00000000 0000004c
		 50000000 800004c5 00000000 ffffffff
		 00000000 0000004c 60000000 800004c6
		 00000000 ffffffff 00000000 0000004c
		 70000000 800004c7 00000000 ffffffff
		 00000000 0000004c 80000000 800004c8
		 00000000 ffffffff 00000000 0000004c
		 90000000 800004c9 00000000 ffffffff
		 00000000 0000004c a0000000 800004ca
		 00000000 ffffffff 00000000 0000004c
		 b0000000 800004cb 00000000 ffffffff
		 00000000 0000004c c0000000 800004cc
		 00000000 ffffffff 00000000 0000004c
		 d0000000 800004cd 00000000 ffffffff
		 00000000 0000004c e0000000 800004ce
		 00000000 ffffffff 00000000 0000004c
		 f0000000 800004cf 00000000 ffffffff
		 00000000 0000004d 00000000 800004d0
		 00000000 ffffffff 00000000 0000004d
		 10000000 800004d1 00000000 ffffffff
		 00000000 0000004d 20000000 800004d2
		 00000000 ffffffff 00000000 0000004d
		 30000000 800004d3 00000000 ffffffff
		 00000000 0000004d 40000000 800004d4
		 00000000 ffffffff 00000000 0000004d
		 50000000 800004d5 00000000 ffffffff
		 00000000 0000004d 60000000 800004d6
		 00000000 ffffffff 00000000 0000004d
		 70000000 800004d7 00000000 ffffffff
		 00000000 0000004d 80000000 800004d8
		 00000000 ffffffff 00000000 0000004d
		 90000000 800004d9 00000000 ffffffff
		 00000000 0000004d a0000000 800004da
		 00000000 ffffffff 00000000 0000004d
		 b0000000 800004db 00000000 ffffffff
		 00000000 0000004d c0000000 800004dc
		 00000000 ffffffff 00000000 0000004d
		 d0000000 800004dd 00000000 ffffffff
		 00000000 0000004d e0000000 800004de
		 00000000 ffffffff 00000000 0000004d
		 f0000000 800004df 00000000 ffffffff
		 00000000 0000004e 00000000 800004e0
		 00000000 ffffffff 00000000 0000004e
		 10000000 800004e1 00000000 ffffffff
		 00000000 0000004e 20000000 800004e2
		 00000000 ffffffff 00000000 0000004e
		 30000000 800004e3 00000000 ffffffff
		 00000000 0000004e 40000000 800004e4
		 00000000 ffffffff 00000000 0000004e
		 50000000 800004e5 00000000 ffffffff
		 00000000 0000004e 60000000 800004e6
		 00000000 ffffffff 00000000 0000004e
		 70000000 800004e7 00000000 ffffffff
		 00000000 0000004e 80000000 800004e8
		 00000000 ffffffff 00000000 0000004e
		 90000000 800004e9 00000000 ffffffff
		 00000000 0000004e a0000000 800004ea
		 00000000 ffffffff 00000000 0000004e
		 b0000000 800004eb 00000000 ffffffff
		 00000000 0000004e c0000000 800004ec
		 00000000 ffffffff 00000000 0000004e
		 d0000000 800004ed 00000000 ffffffff
		 00000000 0000004e e0000000 800004ee
		 00000000 ffffffff 00000000 0000004e
		 f0000000 800004ef 00000000 ffffffff
		 00000000 0000004f 00000000 800004f0
		 00000000 ffffffff 00000000 0000004f
		 10000000 800004f1 00000000 ffffffff
		 00000000 0000004f 20000000 800004f2
		 00000000 ffffffff 00000000 0000004f
		 30000000 800004f3 00000000 ffffffff
		 00000000 0000004f 40000000 800004f4
		 00000000 ffffffff 00000000 0000004f
		 50000000 800004f5 00000000 ffffffff
		 00000000 0000004f 60000000 800004f6
		 00000000 ffffffff 00000000 0000004f
		 70000000 800004f7 00000000 ffffffff
		 00000000 0000004f 80000000 800004f8
		 00000000 ffffffff 00000000 0000004f
		 90000000 800004f9 00000000 ffffffff
		 00000000 0000004f a0000000 800004fa
		 00000000 ffffffff 00000000 0000004f
		 b0000000 800004fb 00000000 ffffffff
		 00000000 0000004f c0000000 800004fc
		 00000000 ffffffff 00000000 0000004f
		 d0000000 800004fd 00000000 ffffffff
		 00000000 0000004f e0000000 800004fe
		 00000000 ffffffff 00000000 0000004f
		 f0000000 800004ff 00000000 ffffffff
		 00000000 00000050 00000000 80000500
		 00000000 ffffffff 00000000 00000050
		 10000000 80000501 00000000 ffffffff
		 00000000 00000050 20000000 80000502
		 00000000 ffffffff 00000000 00000050
		 30000000 80000503 00000000 ffffffff
		 00000000 00000050 40000000 80000504
		 00000000 ffffffff 00000000 00000050
		 50000000 80000505 00000000 ffffffff
		 00000000 00000050 60000000 80000506
		 00000000 ffffffff 00000000 00000050
		 70000000 80000507 00000000 ffffffff
		 00000000 00000050 80000000 80000508
		 00000000 ffffffff 00000000 00000050
		 90000000 80000509 00000000 ffffffff
		 00000000 00000050 a0000000 8000050a
		 00000000 ffffffff 00000000 00000050
		 b0000000 8000050b 00000000 ffffffff
		 00000000 00000050 c0000000 8000050c
		 00000000 ffffffff 00000000 00000050
		 d0000000 8000050d 00000000 ffffffff
		 00000000 00000050 e0000000 8000050e
		 00000000 ffffffff 00000000 00000050
		 f0000000 8000050f 00000000 ffffffff
		 00000000 00000051 00000000 80000510
		 00000000 ffffffff 00000000 00000051
		 10000000 80000511 00000000 ffffffff
		 00000000 00000051 20000000 80000512
		 00000000 ffffffff 00000000 00000051
		 30000000 80000513 00000000 ffffffff
		 00000000 00000051 40000000 80000514
		 00000000 ffffffff 00000000 00000051
		 50000000 80000515 00000000 ffffffff
		 00000000 00000051 60000000 80000516
		 00000000 ffffffff 00000000 00000051
		 70000000 80000517 00000000 ffffffff
		 00000000 00000051 80000000 80000518
		 00000000 ffffffff 00000000 00000051
		 90000000 80000519 00000000 ffffffff
		 00000000 00000051 a0000000 8000051a
		 00000000 ffffffff 00000000 00000051
		 b0000000 8000051b 00000000 ffffffff
		 00000000 00000051 c0000000 8000051c
		 00000000 ffffffff 00000000 00000051
		 d0000000 8000051d 00000000 ffffffff
		 00000000 00000051 e0000000 8000051e
		 00000000 ffffffff 00000000 00000051
		 f0000000 8000051f 00000000 ffffffff
		 00000000 00000052 00000000 80000520
		 00000000 ffffffff 00000000 00000052
		 10000000 80000521 00000000 ffffffff
		 00000000 00000052 20000000 80000522
		 00000000 ffffffff 00000000 00000052
		 30000000 80000523 00000000 ffffffff
		 00000000 00000052 40000000 80000524
		 00000000 ffffffff 00000000 00000052
		 50000000 80000525 00000000 ffffffff
		 00000000 00000052 60000000 80000526
		 00000000 ffffffff 00000000 00000052
		 70000000 80000527 00000000 ffffffff
		 00000000 00000052 80000000 80000528
		 00000000 ffffffff 00000000 00000052
		 90000000 80000529 00000000 ffffffff
		 00000000 00000052 a0000000 8000052a
		 00000000 ffffffff 00000000 00000052
		 b0000000 8000052b 00000000 ffffffff
		 00000000 00000052 c0000000 8000052c
		 00000000 ffffffff 00000000 00000052
		 d0000000 8000052d 00000000 ffffffff
		 00000000 00000052 e0000000 8000052e
		 00000000 ffffffff 00000000 00000052
		 f0000000 8000052f 00000000 ffffffff
		 00000000 00000053 00000000 80000530
		 00000000 ffffffff 00000000 00000053
		 10000000 80000531 00000000 ffffffff
		 00000000 00000053 20000000 80000532
		 00000000 ffffffff 00000000 00000053
		 30000000 80000533 00000000 ffffffff
		 00000000 00000053 40000000 80000534
		 00000000 ffffffff 00000000 00000053
		 50000000 80000535 00000000 ffffffff
		 00000000 00000053 60000000 80000536
		 00000000 ffffffff 00000000 00000053
		 70000000 80000537 00000000 ffffffff
		 00000000 00000053 80000000 80000538
		 00000000 ffffffff 00000000 00000053
		 90000000 80000539 00000000 ffffffff
		 00000000 00000053 a0000000 8000053a
		 00000000 ffffffff 00000000 00000053
		 b0000000 8000053b 00000000 ffffffff
		 00000000 00000053 c0000000 8000053c
		 00000000 ffffffff 00000000 00000053
		 d0000000 8000053d 00000000 ffffffff
		 00000000 00000053 e0000000 8000053e
		 00000000 ffffffff 00000000 00000053
		 f0000000 8000053f 00000000 ffffffff
		 00000000 00000054 00000000 80000540
		 00000000 ffffffff 00000000 00000054
		 10000000 80000541 00000000 ffffffff
		 00000000 00000054 20000000 80000542
		 00000000 ffffffff 00000000 00000054
		 30000000 80000543 00000000 ffffffff
		 00000000 00000054 40000000 80000544
		 00000000 ffffffff 00000000 00000054
		 50000000 80000545 00000000 ffffffff
		 00000000 00000054 60000000 80000546
		 00000000 ffffffff 00000000 00000054
		 70000000 80000547 00000000 ffffffff
		 00000000 00000054 80000000 80000548
		 00000000 ffffffff 00000000 00000054
		 90000000 80000549 00000000 ffffffff
		 00000000 00000054 a0000000 8000054a
		 00000000 ffffffff 00000000 00000054
		 b0000000 8000054b 00000000 ffffffff
		 00000000 00000054 c0000000 8000054c
		 00000000 ffffffff 00000000 00000054
		 d0000000 8000054d 00000000 ffffffff
		 00000000 00000054 e0000000 8000054e
		 00000000 ffffffff 00000000 00000054
		 f0000000 8000054f 00000000 ffffffff
		 00000000 00000055 00000000 80000550
		 00000000 ffffffff 00000000 00000055
		 10000000 80000551 00000000 ffffffff
		 00000000 00000055 20000000 80000552
		 00000000 ffffffff 00000000 00000055
		 30000000 80000553 00000000 ffffffff
		 00000000 00000055 40000000 80000554
		 00000000 ffffffff 00000000 00000055
		 50000000 80000555 00000000 ffffffff
		 00000000 00000055 60000000 80000556
		 00000000 ffffffff 00000000 00000055
		 70000000 80000557 00000000 ffffffff
		 00000000 00000055 80000000 80000558
		 00000000 ffffffff 00000000 00000055
		 90000000 80000559 00000000 ffffffff
		 00000000 00000055 a0000000 8000055a
		 00000000 ffffffff 00000000 00000055
		 b0000000 8000055b 00000000 ffffffff
		 00000000 00000055 c0000000 8000055c
		 00000000 ffffffff 00000000 00000055
		 d0000000 8000055d 00000000 ffffffff
		 00000000 00000055 e0000000 8000055e
		 00000000 ffffffff 00000000 00000055
		 f0000000 8000055f 00000000 ffffffff
		 00000000 00000056 00000000 80000560
		 00000000 ffffffff 00000000 00000056
		 10000000 80000561 00000000 ffffffff
		 00000000 00000056 20000000 80000562
		 00000000 ffffffff 00000000 00000056
		 30000000 80000563 00000000 ffffffff
		 00000000 00000056 40000000 80000564
		 00000000 ffffffff 00000000 00000056
		 50000000 80000565 00000000 ffffffff
		 00000000 00000056 60000000 80000566
		 00000000 ffffffff 00000000 00000056
		 70000000 80000567 00000000 ffffffff
		 00000000 00000056 80000000 80000568
		 00000000 ffffffff 00000000 00000056
		 90000000 80000569 00000000 ffffffff
		 00000000 00000056 a0000000 8000056a
		 00000000 ffffffff 00000000 00000056
		 b0000000 8000056b 00000000 ffffffff
		 00000000 00000056 c0000000 8000056c
		 00000000 ffffffff 00000000 00000056
		 d0000000 8000056d 00000000 ffffffff
		 00000000 00000056 e0000000 8000056e
		 00000000 ffffffff 00000000 00000056
		 f0000000 8000056f 00000000 ffffffff
		 00000000 00000057 00000000 80000570
		 00000000 ffffffff 00000000 00000057
		 10000000 80000571 00000000 ffffffff
		 00000000 00000057 20000000 80000572
		 00000000 ffffffff 00000000 00000057
		 30000000 80000573 00000000 ffffffff
		 00000000 00000057 40000000 80000574
		 00000000 ffffffff 00000000 00000057
		 50000000 80000575 00000000 ffffffff
		 00000000 00000057 60000000 80000576
		 00000000 ffffffff 00000000 00000057
		 70000000 80000577 00000000 ffffffff
		 00000000 00000057 80000000 80000578
		 00000000 ffffffff 00000000 00000057
		 90000000 80000579 00000000 ffffffff
		 00000000 00000057 a0000000 8000057a
		 00000000 ffffffff 00000000 00000057
		 b0000000 8000057b 00000000 ffffffff
		 00000000 00000057 c0000000 8000057c
		 00000000 ffffffff 00000000 00000057
		 d0000000 8000057d 00000000 ffffffff
		 00000000 00000057 e0000000 8000057e
		 00000000 ffffffff 00000000 00000057
		 f0000000 8000057f 00000000 ffffffff
		 00000000 00000058 00000000 80000580
		 00000000 ffffffff 00000000 00000058
		 10000000 80000581 00000000 ffffffff
		 00000000 00000058 20000000 80000582
		 00000000 ffffffff 00000000 00000058
		 30000000 80000583 00000000 ffffffff
		 00000000 00000058 40000000 80000584
		 00000000 ffffffff 00000000 00000058
		 50000000 80000585 00000000 ffffffff
		 00000000 00000058 60000000 80000586
		 00000000 ffffffff 00000000 00000058
		 70000000 80000587 00000000 ffffffff
		 00000000 00000058 80000000 80000588
		 00000000 ffffffff 00000000 00000058
		 90000000 80000589 00000000 ffffffff
		 00000000 00000058 a0000000 8000058a
		 00000000 ffffffff 00000000 00000058
		 b0000000 8000058b 00000000 ffffffff
		 00000000 00000058 c0000000 8000058c
		 00000000 ffffffff 00000000 00000058
		 d0000000 8000058d 00000000 ffffffff
		 00000000 00000058 e0000000 8000058e
		 00000000 ffffffff 00000000 00000058
		 f0000000 8000058f 00000000 ffffffff
		 00000000 00000059 00000000 80000590
		 00000000 ffffffff 00000000 00000059
		 10000000 80000591 00000000 ffffffff
		 00000000 00000059 20000000 80000592
		 00000000 ffffffff 00000000 00000059
		 30000000 80000593 00000000 ffffffff
		 00000000 00000059 40000000 80000594
		 00000000 ffffffff 00000000 00000059
		 50000000 80000595 00000000 ffffffff
		 00000000 00000059 60000000 80000596
		 00000000 ffffffff 00000000 00000059
		 70000000 80000597 00000000 ffffffff
		 00000000 00000059 80000000 80000598
		 00000000 ffffffff 00000000 00000059
		 90000000 80000599 00000000 ffffffff
		 00000000 00000059 a0000000 8000059a
		 00000000 ffffffff 00000000 00000059
		 b0000000 8000059b 00000000 ffffffff
		 00000000 00000059 c0000000 8000059c
		 00000000 ffffffff 00000000 00000059
		 d0000000 8000059d 00000000 ffffffff
		 00000000 00000059 e0000000 8000059e
		 00000000 ffffffff 00000000 00000059
		 f0000000 8000059f 00000000 ffffffff
		 00000000
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # 
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # cat ~/dmesg.log
[    0.000000] Linux version 4.4.63-2.1.wi91275.sp3.055a_27-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Tue May 2 10:56:31 EDT 2017 (27e2849)
[    0.000000] Node 0 Memory: 0x0-0x510000000
[    0.000000] Node 1 Memory: 0x510000000-0x5c0000000
[    0.000000] Node 6 Memory: 0x5c0000000-0x26b0000000
[    0.000000] Node 7 Memory: 0x26b0000000-0x3c00000000
[    0.000000] numa: Initmem setup node 0 [mem 0x00000000-0x50fffffff]
[    0.000000] numa:   NODE_DATA [mem 0x50fff6100-0x50fffffff]
[    0.000000] numa: Initmem setup node 1 [mem 0x510000000-0x5bfffffff]
[    0.000000] numa:   NODE_DATA [mem 0x5bfff6100-0x5bfffffff]
[    0.000000] numa: Initmem setup node 6 [mem 0x5c0000000-0x26afffffff]
[    0.000000] numa:   NODE_DATA [mem 0x26afff6100-0x26afffffff]
[    0.000000] numa: Initmem setup node 7 [mem 0x26b0000000-0x3bffffffff]
[    0.000000] numa:   NODE_DATA [mem 0x3bffbe2100-0x3bffbebfff]
[    0.000000] Section 15357 and 15359 (node 7) have a circular dependency on usemap and pgdat allocations
[    0.000000] PPC64 nvram contains 15360 bytes
[    0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000
[    0.000000] Memory hole size: 0MB
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x0000003bffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000050fffffff]
[    0.000000]   node   1: [mem 0x0000000510000000-0x00000005bfffffff]
[    0.000000]   node   6: [mem 0x00000005c0000000-0x00000026afffffff]
[    0.000000]   node   7: [mem 0x00000026b0000000-0x0000003bffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000050fffffff]
[    0.000000] On node 0 totalpages: 331776
[    0.000000]   DMA zone: 324 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 331776 pages, LIFO batch:1
[    0.000000] Initmem setup node 1 [mem 0x0000000510000000-0x00000005bfffffff]
[    0.000000] On node 1 totalpages: 45056
[    0.000000]   DMA zone: 44 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 45056 pages, LIFO batch:1
[    0.000000] Initmem setup node 6 [mem 0x00000005c0000000-0x00000026afffffff]
[    0.000000] On node 6 totalpages: 2158592
[    0.000000]   DMA zone: 2108 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 2158592 pages, LIFO batch:1
[    0.000000] Initmem setup node 7 [mem 0x00000026b0000000-0x0000003bffffffff]
[    0.000000] On node 7 totalpages: 1396736
[    0.000000]   DMA zone: 1364 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 1396736 pages, LIFO batch:1
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory #

Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Ellerman]

Michael Bringmann <mwb@linux.vnet.ibm.com> writes:

> Here is the information from 2 different kernels.  I have not been able to retrieve
> the information matching yesterday's attachments, yet, as those dumps were
> acquired in April.
>  
> Attached please find 2 dumps of similar material from kernels running with my
> current patches (Linux 4.4, Linux 4.12).

OK thanks.

I'd actually like to see the dmesg output from a kernel *without* your
patches.

Looking at the device tree properties:

ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
		 00000004 = 4 arrays
                 00000004 = of 4 entries each
                 00000000 00000000 00000000 00000000
		 00000000 00000000 00000001 00000001
                 00000000 00000003 00000006 00000006
                 00000000 00000003 00000007 00000007


Which does tell us that nodes 0, 1, 6 and 7 exist.

So your idea of looking at that and setting any node found in there
online should work.

My only worry is that behaviour appears to be completely undocumented in
PAPR, ie. PAPR explicitly says that property only needs to contain
values for LMBs present at boot.

But possibly we can talk to the PowerVM/PAPR guys and have that changed
so that it becomes something we can rely on.

cheers

Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

On a related note, we are discussing the addition of 2 new device-tree properties
with Pete Heyrman and his fellows that should simplify the determination of the
set of required nodes.

* One property would provide the total/max number of nodes needed by the kernel
  on the current hardware.
* A second property would provide the total/max number of nodes that the kernel
  could use on any system to which it could be migrated.

These properties aren't available, yet, and it takes time to define new properties
in the PAPR and have them implemented in pHyp and the kernel.  As an intermediary
step, the systems which are doing a lot of dynamic hot-add/hot-remove configuration
could provide equivalent information to the PowerPC kernel with a command line
parameter.  The 'numa.c' code would then read this value and fill in the necessary
entries in the 'node_possible_map'.

Would you foresee any problems with using such a feature?

Thanks.

On 06/13/2017 05:45 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> 
>> Here is the information from 2 different kernels.  I have not been able to retrieve
>> the information matching yesterday's attachments, yet, as those dumps were
>> acquired in April.
>>  
>> Attached please find 2 dumps of similar material from kernels running with my
>> current patches (Linux 4.4, Linux 4.12).
> 
> OK thanks.
> 
> I'd actually like to see the dmesg output from a kernel *without* your
> patches.
> 
> Looking at the device tree properties:
> 
> ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays
> ibm,associativity-lookup-arrays
> 		 00000004 = 4 arrays
>                  00000004 = of 4 entries each
>                  00000000 00000000 00000000 00000000
> 		 00000000 00000000 00000001 00000001
>                  00000000 00000003 00000006 00000006
>                  00000000 00000003 00000007 00000007
> 
> 
> Which does tell us that nodes 0, 1, 6 and 7 exist.
> 
> So your idea of looking at that and setting any node found in there
> online should work.
> 
> My only worry is that behaviour appears to be completely undocumented in
> PAPR, ie. PAPR explicitly says that property only needs to contain
> values for LMBs present at boot.
> 
> But possibly we can talk to the PowerVM/PAPR guys and have that changed
> so that it becomes something we can rely on.
> 
> cheers
> 
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Balbir Singh]

[-- Attachment #1: Type: text/plain, Size: 1233 bytes --]

On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com>
wrote:

> On a related note, we are discussing the addition of 2 new device-tree
> properties
> with Pete Heyrman and his fellows that should simplify the determination
> of the
> set of required nodes.
>
> * One property would provide the total/max number of nodes needed by the
> kernel
>   on the current hardware.
>

Yes, that would be nice to have


> * A second property would provide the total/max number of nodes that the
> kernel
>   could use on any system to which it could be migrated.
>
>
Not sure about this one, are you suggesting more memory can be added
depending on the migration target?



> These properties aren't available, yet, and it takes time to define new
> properties
> in the PAPR and have them implemented in pHyp and the kernel.  As an
> intermediary
> step, the systems which are doing a lot of dynamic hot-add/hot-remove
> configuration
> could provide equivalent information to the PowerPC kernel with a command
> line
> parameter.  The 'numa.c' code would then read this value and fill in the
> necessary
> entries in the 'node_possible_map'.
>
> Would you foresee any problems with using such a feature?
>


Balbir Singh

[-- Attachment #2: Type: text/html, Size: 1949 bytes --]

Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Balbir Singh]

On Wed, Jun 14, 2017 at 3:25 PM, Balbir Singh <bsingharora@gmail.com> wrote:
>
>
> On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com>
> wrote:
>>
>> On a related note, we are discussing the addition of 2 new device-tree
>> properties
>> with Pete Heyrman and his fellows that should simplify the determination
>> of the
>> set of required nodes.
>>
>> * One property would provide the total/max number of nodes needed by the
>> kernel
>>   on the current hardware.
>
>

Yes, that would be nice to have

>
>>
>> * A second property would provide the total/max number of nodes that the
>> kernel
>>   could use on any system to which it could be migrated.
>>
>

Not sure about this one, are you suggesting more memory can be added
depending on the migration target?

>
>
>>
>> These properties aren't available, yet, and it takes time to define new
>> properties
>> in the PAPR and have them implemented in pHyp and the kernel.  As an
>> intermediary
>> step, the systems which are doing a lot of dynamic hot-add/hot-remove
>> configuration
>> could provide equivalent information to the PowerPC kernel with a command
>> line
>> parameter.  The 'numa.c' code would then read this value and fill in the
>> necessary
>> entries in the 'node_possible_map'.
>>
>> Would you foresee any problems with using such a feature?
>
>

Sorry my mailer goofed up, resending

Balbir Singh

Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

Hello:

On 06/14/2017 12:27 AM, Balbir Singh wrote:
> On Wed, Jun 14, 2017 at 3:25 PM, Balbir Singh <bsingharora@gmail.com> wrote:
>>
>>
>> On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com>
>> wrote:
>>>
>>> On a related note, we are discussing the addition of 2 new device-tree
>>> properties
>>> with Pete Heyrman and his fellows that should simplify the determination
>>> of the
>>> set of required nodes.
>>>
>>> * One property would provide the total/max number of nodes needed by the
>>> kernel
>>>   on the current hardware.
>>
>>
> 
> Yes, that would be nice to have
> 
>>
>>>
>>> * A second property would provide the total/max number of nodes that the
>>> kernel
>>>   could use on any system to which it could be migrated.
>>>
>>
> 
> Not sure about this one, are you suggesting more memory can be added
> depending on the migration target?

We would use only one of these numbers to allocate nodes.  I have only been
on the periphery of the discussions, so I can not communicate the full
reasoning as to why both measures would be needed.  We would like to have
the first number for node allocation/initialization, but if only the second
value were provided, we would likely need to use it.

>>
>>
>>>
>>> These properties aren't available, yet, and it takes time to define new
>>> properties
>>> in the PAPR and have them implemented in pHyp and the kernel.  As an
>>> intermediary
>>> step, the systems which are doing a lot of dynamic hot-add/hot-remove
>>> configuration
>>> could provide equivalent information to the PowerPC kernel with a command
>>> line
>>> parameter.  The 'numa.c' code would then read this value and fill in the
>>> necessary
>>> entries in the 'node_possible_map'.
>>>
>>> Would you foresee any problems with using such a feature?
>>
>>
> 
> Sorry my mailer goofed up, resending
> 
> Balbir Singh
> 

Thanks.


-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

[-- Attachment #1: Type: text/html, Size: 7939 bytes --]

[-- Attachment #2: 1.linux4.12-3nodes.txt --]
[-- Type: text/plain, Size: 4576 bytes --]


Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel 4.12.0-rc3.wi91275_054c_060106.ppc64le+ on an ppc64le

ltcalpine2-lp20 login: root
Password: 
Last login: Wed Jun  7 11:03:12 from oc1554177480.austin.ibm.com
[root@ltcalpine2-lp20 ~]# numactl -H
available: 3 nodes (0,2-3)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 48 49 50 51 52 53 54 55
node 2 size: 188668 MB
node 2 free: 187903 MB
node 3 cpus: 40 41 42 43 44 45 46 47 56 57 58 59 60 61 62 63
node 3 size: 56261 MB
node 3 free: 55324 MB
node distances:
node   0   2   3 
  0:  10  40  40 
  2:  40  10  20 
  3:  40  20  10 
[root@ltcalpine2-lp20 ~]# cd /proc/device-tree/ibm,dynamic-reconfiguration-memory
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,dynamic-memory
ibm,dynamic-memory
		 0000059e 00000000 20000000 80000002
		 00000000 00000001 00000008 00000000
		 30000000 80000003 00000000 00000001
		 00000008 00000000 40000000 80000004
		 00000000 00000001 00000008 00000000
		 50000000 80000005 00000000 00000001
		 00000008 00000000 60000000 80000006
		 00000000 00000001 00000008 00000000
		 [34516 bytes total]
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
		 00000003 00000004 00000000 00000000
		 00000000 00000000 00000000 00000001
		 00000002 00000002 00000000 00000001
		 00000003 00000003
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# cat ~/dmesg.log
[    0.000000] Linux version 4.12.0-rc3.wi91275_054c_060106.ppc64le+ (root@ltcalpine2-lp20.aus.stglabs.ibm.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #28 SMP Tue Jun 6 16:05:01 EDT 2017
[    0.000000] Found initrd at 0xc00000000ce00000:0xc00000000eeb4b12
[    0.000000] Using pSeries machine description
[    0.000000] bootconsole [udbg0] enabled
[    0.000000] Partition configured for 256 cpus.
[    0.000000] CPU maps initialized for 8 threads per core
[    0.000000]  (thread shift is 3)
[    0.000000] Freed 4784128 bytes for unused pacas
[    0.000000] -----------------------------------------------------
[    0.000000] ppc64_pft_size    = 0x20
[    0.000000] phys_mem_size     = 0x3c00000000
[    0.000000] dcache_bsize      = 0x80
[    0.000000] icache_bsize      = 0x80
[    0.000000] cpu_features      = 0x27fc7aec18500249
[    0.000000]   possible        = 0x7fffffff18500649
[    0.000000]   always          = 0x0000000018100040
[    0.000000] cpu_user_features = 0xdc0065c2 0xef000000
[    0.000000] mmu_features      = 0x7c006001
[    0.000000] firmware_features = 0x00000003c45bfc57
[    0.000000] htab_hash_mask    = 0x1ffffff
[    0.000000] -----------------------------------------------------
[    0.000000] numa:   NODE_DATA [mem 0x3bff7d5f00-0x3bff7dffff]
[    0.000000] numa:     NODE_DATA(0) on node 3
[    0.000000] numa:   NODE_DATA [mem 0x2e1fff5f00-0x2e1fffffff]
[    0.000000] numa:   NODE_DATA [mem 0x3bff7cbe00-0x3bff7d5eff]
[    0.000000] Section 15357 and 15359 (node 3) have a circular dependency on usemap and pgdat allocations
[    0.000000] PPC64 nvram contains 15360 bytes
[    0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000
[    0.000000] Memory hole size: 0MB
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x0000003bffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   2: [mem 0x0000000000000000-0x0000002e1fffffff]
[    0.000000]   node   3: [mem 0x0000002e20000000-0x0000003bffffffff]
[    0.000000] Could not find start_pfn for node 0
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[    0.000000] On node 0 totalpages: 0
[    0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x0000002e1fffffff]
[    0.000000] On node 2 totalpages: 3022848
[    0.000000]   DMA zone: 2952 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 3022848 pages, LIFO batch:1
[    0.000000] Initmem setup node 3 [mem 0x0000002e20000000-0x0000003bffffffff]
[    0.000000] On node 3 totalpages: 909312
[    0.000000]   DMA zone: 888 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 909312 pages, LIFO batch:1
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]#



[-- Attachment #3: 2.linux4.4-4nodes.txt --]
[-- Type: text/plain, Size: 90857 bytes --]


Welcome to SUSE Linux Enterprise Server 12 SP3 Beta2 (ppc64le) - Kernel 4.4.63-2.1.wi91275.sp3.055a_27-default (hvc0).


ltcalpine2-lp9 login: root
Password: 
Last login: Mon Jun  5 11:43:45 from 9.53.92.157
ltcalpine2-lp9:ß # numactl -H
available: 4 nodes (0-1,6-7)
node 0 cpus:
node 0 size: 20512 MB
node 0 free: 20371 MB
node 1 cpus:
node 1 size: 2815 MB
node 1 free: 2706 MB
node 6 cpus: 0 1 2 3 4 5 6 7
node 6 size: 134783 MB
node 6 free: 134068 MB
node 7 cpus:
node 7 size: 87210 MB
node 7 free: 87102 MB
node distances:
node   0   1   6   7 
  0:  10  20  40  40 
  1:  20  10  40  40 
  6:  40  40  10  20 
  7:  40  40  20  10 
ltcalpine2-lp9:~ # cd 
ltcalpine2-lp9:~ # cd /proc/device-tree/ibm,dynamic-reconfiguration-memory
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # ls
ibm,associativity-lookup-arrays  ibm,memory-flags-mask         linux,phandle
ibm,dynamic-memory               ibm,memory-preservation-time  name
ibm,lmb-size                     ibm,phandle
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
		 00000004 00000004 00000000 00000000 00000000 00000000
		 00000000 00000000 00000001 00000001 00000000 00000003
		 00000006 00000006 00000000 00000003 00000007 00000007
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,dynamic-memory
ibm,dynamic-memory
		 0000059e 00000000 20000000 80000002
		 00000000 00000000 00000008 00000000
		 30000000 80000003 00000000 00000000
		 00000008 00000000 40000000 80000004
		 00000000 00000000 00000008 00000000
		 50000000 80000005 00000000 00000000
		 00000008 00000000 60000000 80000006
		 00000000 00000000 00000008 00000000
		 70000000 80000007 00000000 00000000
		 00000008 00000000 80000000 80000008
		 00000000 00000000 00000008 00000000
		 90000000 80000009 00000000 00000000
		 00000008 00000000 a0000000 8000000a
		 00000000 00000000 00000008 00000000
		 b0000000 8000000b 00000000 00000000
		 00000008 00000000 c0000000 8000000c
		 00000000 00000000 00000008 00000000
		 d0000000 8000000d 00000000 00000000
		 00000008 00000000 e0000000 8000000e
		 00000000 00000000 00000008 00000000
		 f0000000 8000000f 00000000 00000000
		 00000008 00000001 00000000 80000010
		 00000000 00000000 00000008 00000001
		 10000000 80000011 00000000 00000000
		 00000008 00000001 20000000 80000012
		 00000000 00000000 00000008 00000001
		 30000000 80000013 00000000 00000000
		 00000008 00000001 40000000 80000014
		 00000000 00000000 00000008 00000001
		 50000000 80000015 00000000 00000000
		 00000008 00000001 60000000 80000016
		 00000000 00000000 00000008 00000001
		 70000000 80000017 00000000 00000000
		 00000008 00000001 80000000 80000018
		 00000000 00000000 00000008 00000001
		 90000000 80000019 00000000 00000000
		 00000008 00000001 a0000000 8000001a
		 00000000 00000000 00000008 00000001
		 b0000000 8000001b 00000000 00000000
		 00000008 00000001 c0000000 8000001c
		 00000000 00000000 00000008 00000001
		 d0000000 8000001d 00000000 00000000
		 00000008 00000001 e0000000 8000001e
		 00000000 00000000 00000008 00000001
		 f0000000 8000001f 00000000 00000000
		 00000008 00000002 00000000 80000020
		 00000000 00000000 00000008 00000002
		 10000000 80000021 00000000 00000000
		 00000008 00000002 20000000 80000022
		 00000000 00000000 00000008 00000002
		 30000000 80000023 00000000 00000000
		 00000008 00000002 40000000 80000024
		 00000000 00000000 00000008 00000002
		 50000000 80000025 00000000 00000000
		 00000008 00000002 60000000 80000026
		 00000000 00000000 00000008 00000002
		 70000000 80000027 00000000 00000000
		 00000008 00000002 80000000 80000028
		 00000000 00000000 00000008 00000002
		 90000000 80000029 00000000 00000000
		 00000008 00000002 a0000000 8000002a
		 00000000 00000000 00000008 00000002
		 b0000000 8000002b 00000000 00000000
		 00000008 00000002 c0000000 8000002c
		 00000000 00000000 00000008 00000002
		 d0000000 8000002d 00000000 00000000
		 00000008 00000002 e0000000 8000002e
		 00000000 00000000 00000008 00000002
		 f0000000 8000002f 00000000 00000000
		 00000008 00000003 00000000 80000030
		 00000000 00000000 00000008 00000003
		 10000000 80000031 00000000 00000000
		 00000008 00000003 20000000 80000032
		 00000000 00000000 00000008 00000003
		 30000000 80000033 00000000 00000000
		 00000008 00000003 40000000 80000034
		 00000000 00000000 00000008 00000003
		 50000000 80000035 00000000 00000000
		 00000008 00000003 60000000 80000036
		 00000000 00000000 00000008 00000003
		 70000000 80000037 00000000 00000000
		 00000008 00000003 80000000 80000038
		 00000000 00000000 00000008 00000003
		 90000000 80000039 00000000 00000000
		 00000008 00000003 a0000000 8000003a
		 00000000 00000000 00000008 00000003
		 b0000000 8000003b 00000000 00000000
		 00000008 00000003 c0000000 8000003c
		 00000000 00000000 00000008 00000003
		 d0000000 8000003d 00000000 00000000
		 00000008 00000003 e0000000 8000003e
		 00000000 00000000 00000008 00000003
		 f0000000 8000003f 00000000 00000000
		 00000008 00000004 00000000 80000040
		 00000000 00000000 00000008 00000004
		 10000000 80000041 00000000 00000000
		 00000008 00000004 20000000 80000042
		 00000000 00000000 00000008 00000004
		 30000000 80000043 00000000 00000000
		 00000008 00000004 40000000 80000044
		 00000000 00000000 00000008 00000004
		 50000000 80000045 00000000 00000000
		 00000008 00000004 60000000 80000046
		 00000000 00000000 00000008 00000004
		 70000000 80000047 00000000 00000000
		 00000008 00000004 80000000 80000048
		 00000000 00000000 00000008 00000004
		 90000000 80000049 00000000 00000000
		 00000008 00000004 a0000000 8000004a
		 00000000 00000000 00000008 00000004
		 b0000000 8000004b 00000000 00000000
		 00000008 00000004 c0000000 8000004c
		 00000000 00000000 00000008 00000004
		 d0000000 8000004d 00000000 00000000
		 00000008 00000004 e0000000 8000004e
		 00000000 00000000 00000008 00000004
		 f0000000 8000004f 00000000 00000000
		 00000008 00000005 00000000 80000050
		 00000000 00000000 00000008 00000005
		 10000000 80000051 00000000 00000001
		 00000008 00000005 20000000 80000052
		 00000000 00000001 00000008 00000005
		 30000000 80000053 00000000 00000001
		 00000008 00000005 40000000 80000054
		 00000000 00000001 00000008 00000005
		 50000000 80000055 00000000 00000001
		 00000008 00000005 60000000 80000056
		 00000000 00000001 00000008 00000005
		 70000000 80000057 00000000 00000001
		 00000008 00000005 80000000 80000058
		 00000000 00000001 00000008 00000005
		 90000000 80000059 00000000 00000001
		 00000008 00000005 a0000000 8000005a
		 00000000 00000001 00000008 00000005
		 b0000000 8000005b 00000000 00000001
		 00000008 00000005 c0000000 8000005c
		 00000000 00000002 00000008 00000005
		 d0000000 8000005d 00000000 00000002
		 00000008 00000005 e0000000 8000005e
		 00000000 00000002 00000008 00000005
		 f0000000 8000005f 00000000 00000002
		 00000008 00000006 00000000 80000060
		 00000000 00000002 00000008 00000006
		 10000000 80000061 00000000 00000002
		 00000008 00000006 20000000 80000062
		 00000000 00000002 00000008 00000006
		 30000000 80000063 00000000 00000002
		 00000008 00000006 40000000 80000064
		 00000000 00000002 00000008 00000006
		 50000000 80000065 00000000 00000002
		 00000008 00000006 60000000 80000066
		 00000000 00000002 00000008 00000006
		 70000000 80000067 00000000 00000002
		 00000008 00000006 80000000 80000068
		 00000000 00000002 00000008 00000006
		 90000000 80000069 00000000 00000002
		 00000008 00000006 a0000000 8000006a
		 00000000 00000002 00000008 00000006
		 b0000000 8000006b 00000000 00000002
		 00000008 00000006 c0000000 8000006c
		 00000000 00000002 00000008 00000006
		 d0000000 8000006d 00000000 00000002
		 00000008 00000006 e0000000 8000006e
		 00000000 00000002 00000008 00000006
		 f0000000 8000006f 00000000 00000002
		 00000008 00000007 00000000 80000070
		 00000000 00000002 00000008 00000007
		 10000000 80000071 00000000 00000002
		 00000008 00000007 20000000 80000072
		 00000000 00000002 00000008 00000007
		 30000000 80000073 00000000 00000002
		 00000008 00000007 40000000 80000074
		 00000000 00000002 00000008 00000007
		 50000000 80000075 00000000 00000002
		 00000008 00000007 60000000 80000076
		 00000000 00000002 00000008 00000007
		 70000000 80000077 00000000 00000002
		 00000008 00000007 80000000 80000078
		 00000000 00000002 00000008 00000007
		 90000000 80000079 00000000 00000002
		 00000008 00000007 a0000000 8000007a
		 00000000 00000002 00000008 00000007
		 b0000000 8000007b 00000000 00000002
		 00000008 00000007 c0000000 8000007c
		 00000000 00000002 00000008 00000007
		 d0000000 8000007d 00000000 00000002
		 00000008 00000007 e0000000 8000007e
		 00000000 00000002 00000008 00000007
		 f0000000 8000007f 00000000 00000002
		 00000008 00000008 00000000 80000080
		 00000000 00000002 00000008 00000008
		 10000000 80000081 00000000 00000002
		 00000008 00000008 20000000 80000082
		 00000000 00000002 00000008 00000008
		 30000000 80000083 00000000 00000002
		 00000008 00000008 40000000 80000084
		 00000000 00000002 00000008 00000008
		 50000000 80000085 00000000 00000002
		 00000008 00000008 60000000 80000086
		 00000000 00000002 00000008 00000008
		 70000000 80000087 00000000 00000002
		 00000008 00000008 80000000 80000088
		 00000000 00000002 00000008 00000008
		 90000000 80000089 00000000 00000002
		 00000008 00000008 a0000000 8000008a
		 00000000 00000002 00000008 00000008
		 b0000000 8000008b 00000000 00000002
		 00000008 00000008 c0000000 8000008c
		 00000000 00000002 00000008 00000008
		 d0000000 8000008d 00000000 00000002
		 00000008 00000008 e0000000 8000008e
		 00000000 00000002 00000008 00000008
		 f0000000 8000008f 00000000 00000002
		 00000008 00000009 00000000 80000090
		 00000000 00000002 00000008 00000009
		 10000000 80000091 00000000 00000002
		 00000008 00000009 20000000 80000092
		 00000000 00000002 00000008 00000009
		 30000000 80000093 00000000 00000002
		 00000008 00000009 40000000 80000094
		 00000000 00000002 00000008 00000009
		 50000000 80000095 00000000 00000002
		 00000008 00000009 60000000 80000096
		 00000000 00000002 00000008 00000009
		 70000000 80000097 00000000 00000002
		 00000008 00000009 80000000 80000098
		 00000000 00000002 00000008 00000009
		 90000000 80000099 00000000 00000002
		 00000008 00000009 a0000000 8000009a
		 00000000 00000002 00000008 00000009
		 b0000000 8000009b 00000000 00000002
		 00000008 00000009 c0000000 8000009c
		 00000000 00000002 00000008 00000009
		 d0000000 8000009d 00000000 00000002
		 00000008 00000009 e0000000 8000009e
		 00000000 00000002 00000008 00000009
		 f0000000 8000009f 00000000 00000002
		 00000008 0000000a 00000000 800000a0
		 00000000 00000002 00000008 0000000a
		 10000000 800000a1 00000000 00000002
		 00000008 0000000a 20000000 800000a2
		 00000000 00000002 00000008 0000000a
		 30000000 800000a3 00000000 00000002
		 00000008 0000000a 40000000 800000a4
		 00000000 00000002 00000008 0000000a
		 50000000 800000a5 00000000 00000002
		 00000008 0000000a 60000000 800000a6
		 00000000 00000002 00000008 0000000a
		 70000000 800000a7 00000000 00000002
		 00000008 0000000a 80000000 800000a8
		 00000000 00000002 00000008 0000000a
		 90000000 800000a9 00000000 00000002
		 00000008 0000000a a0000000 800000aa
		 00000000 00000002 00000008 0000000a
		 b0000000 800000ab 00000000 00000002
		 00000008 0000000a c0000000 800000ac
		 00000000 00000002 00000008 0000000a
		 d0000000 800000ad 00000000 00000002
		 00000008 0000000a e0000000 800000ae
		 00000000 00000002 00000008 0000000a
		 f0000000 800000af 00000000 00000002
		 00000008 0000000b 00000000 800000b0
		 00000000 00000002 00000008 0000000b
		 10000000 800000b1 00000000 00000002
		 00000008 0000000b 20000000 800000b2
		 00000000 00000002 00000008 0000000b
		 30000000 800000b3 00000000 00000002
		 00000008 0000000b 40000000 800000b4
		 00000000 00000002 00000008 0000000b
		 50000000 800000b5 00000000 00000002
		 00000008 0000000b 60000000 800000b6
		 00000000 00000002 00000008 0000000b
		 70000000 800000b7 00000000 00000002
		 00000008 0000000b 80000000 800000b8
		 00000000 00000002 00000008 0000000b
		 90000000 800000b9 00000000 00000002
		 00000008 0000000b a0000000 800000ba
		 00000000 00000002 00000008 0000000b
		 b0000000 800000bb 00000000 00000002
		 00000008 0000000b c0000000 800000bc
		 00000000 00000002 00000008 0000000b
		 d0000000 800000bd 00000000 00000002
		 00000008 0000000b e0000000 800000be
		 00000000 00000002 00000008 0000000b
		 f0000000 800000bf 00000000 00000002
		 00000008 0000000c 00000000 800000c0
		 00000000 00000002 00000008 0000000c
		 10000000 800000c1 00000000 00000002
		 00000008 0000000c 20000000 800000c2
		 00000000 00000002 00000008 0000000c
		 30000000 800000c3 00000000 00000002
		 00000008 0000000c 40000000 800000c4
		 00000000 00000002 00000008 0000000c
		 50000000 800000c5 00000000 00000002
		 00000008 0000000c 60000000 800000c6
		 00000000 00000002 00000008 0000000c
		 70000000 800000c7 00000000 00000002
		 00000008 0000000c 80000000 800000c8
		 00000000 00000002 00000008 0000000c
		 90000000 800000c9 00000000 00000002
		 00000008 0000000c a0000000 800000ca
		 00000000 00000002 00000008 0000000c
		 b0000000 800000cb 00000000 00000002
		 00000008 0000000c c0000000 800000cc
		 00000000 00000002 00000008 0000000c
		 d0000000 800000cd 00000000 00000002
		 00000008 0000000c e0000000 800000ce
		 00000000 00000002 00000008 0000000c
		 f0000000 800000cf 00000000 00000002
		 00000008 0000000d 00000000 800000d0
		 00000000 00000002 00000008 0000000d
		 10000000 800000d1 00000000 00000002
		 00000008 0000000d 20000000 800000d2
		 00000000 00000002 00000008 0000000d
		 30000000 800000d3 00000000 00000002
		 00000008 0000000d 40000000 800000d4
		 00000000 00000002 00000008 0000000d
		 50000000 800000d5 00000000 00000002
		 00000008 0000000d 60000000 800000d6
		 00000000 00000002 00000008 0000000d
		 70000000 800000d7 00000000 00000002
		 00000008 0000000d 80000000 800000d8
		 00000000 00000002 00000008 0000000d
		 90000000 800000d9 00000000 00000002
		 00000008 0000000d a0000000 800000da
		 00000000 00000002 00000008 0000000d
		 b0000000 800000db 00000000 00000002
		 00000008 0000000d c0000000 800000dc
		 00000000 00000002 00000008 0000000d
		 d0000000 800000dd 00000000 00000002
		 00000008 0000000d e0000000 800000de
		 00000000 00000002 00000008 0000000d
		 f0000000 800000df 00000000 00000002
		 00000008 0000000e 00000000 800000e0
		 00000000 00000002 00000008 0000000e
		 10000000 800000e1 00000000 00000002
		 00000008 0000000e 20000000 800000e2
		 00000000 00000002 00000008 0000000e
		 30000000 800000e3 00000000 00000002
		 00000008 0000000e 40000000 800000e4
		 00000000 00000002 00000008 0000000e
		 50000000 800000e5 00000000 00000002
		 00000008 0000000e 60000000 800000e6
		 00000000 00000002 00000008 0000000e
		 70000000 800000e7 00000000 00000002
		 00000008 0000000e 80000000 800000e8
		 00000000 00000002 00000008 0000000e
		 90000000 800000e9 00000000 00000002
		 00000008 0000000e a0000000 800000ea
		 00000000 00000002 00000008 0000000e
		 b0000000 800000eb 00000000 00000002
		 00000008 0000000e c0000000 800000ec
		 00000000 00000002 00000008 0000000e
		 d0000000 800000ed 00000000 00000002
		 00000008 0000000e e0000000 800000ee
		 00000000 00000002 00000008 0000000e
		 f0000000 800000ef 00000000 00000002
		 00000008 0000000f 00000000 800000f0
		 00000000 00000002 00000008 0000000f
		 10000000 800000f1 00000000 00000002
		 00000008 0000000f 20000000 800000f2
		 00000000 00000002 00000008 0000000f
		 30000000 800000f3 00000000 00000002
		 00000008 0000000f 40000000 800000f4
		 00000000 00000002 00000008 0000000f
		 50000000 800000f5 00000000 00000002
		 00000008 0000000f 60000000 800000f6
		 00000000 00000002 00000008 0000000f
		 70000000 800000f7 00000000 00000002
		 00000008 0000000f 80000000 800000f8
		 00000000 00000002 00000008 0000000f
		 90000000 800000f9 00000000 00000002
		 00000008 0000000f a0000000 800000fa
		 00000000 00000002 00000008 0000000f
		 b0000000 800000fb 00000000 00000002
		 00000008 0000000f c0000000 800000fc
		 00000000 00000002 00000008 0000000f
		 d0000000 800000fd 00000000 00000002
		 00000008 0000000f e0000000 800000fe
		 00000000 00000002 00000008 0000000f
		 f0000000 800000ff 00000000 00000002
		 00000008 00000010 00000000 80000100
		 00000000 00000002 00000008 00000010
		 10000000 80000101 00000000 00000002
		 00000008 00000010 20000000 80000102
		 00000000 00000002 00000008 00000010
		 30000000 80000103 00000000 00000002
		 00000008 00000010 40000000 80000104
		 00000000 00000002 00000008 00000010
		 50000000 80000105 00000000 00000002
		 00000008 00000010 60000000 80000106
		 00000000 00000002 00000008 00000010
		 70000000 80000107 00000000 00000002
		 00000008 00000010 80000000 80000108
		 00000000 00000002 00000008 00000010
		 90000000 80000109 00000000 00000002
		 00000008 00000010 a0000000 8000010a
		 00000000 00000002 00000008 00000010
		 b0000000 8000010b 00000000 00000002
		 00000008 00000010 c0000000 8000010c
		 00000000 00000002 00000008 00000010
		 d0000000 8000010d 00000000 00000002
		 00000008 00000010 e0000000 8000010e
		 00000000 00000002 00000008 00000010
		 f0000000 8000010f 00000000 00000002
		 00000008 00000011 00000000 80000110
		 00000000 00000002 00000008 00000011
		 10000000 80000111 00000000 00000002
		 00000008 00000011 20000000 80000112
		 00000000 00000002 00000008 00000011
		 30000000 80000113 00000000 00000002
		 00000008 00000011 40000000 80000114
		 00000000 00000002 00000008 00000011
		 50000000 80000115 00000000 00000002
		 00000008 00000011 60000000 80000116
		 00000000 00000002 00000008 00000011
		 70000000 80000117 00000000 00000002
		 00000008 00000011 80000000 80000118
		 00000000 00000002 00000008 00000011
		 90000000 80000119 00000000 00000002
		 00000008 00000011 a0000000 8000011a
		 00000000 00000002 00000008 00000011
		 b0000000 8000011b 00000000 00000002
		 00000008 00000011 c0000000 8000011c
		 00000000 00000002 00000008 00000011
		 d0000000 8000011d 00000000 00000002
		 00000008 00000011 e0000000 8000011e
		 00000000 00000002 00000008 00000011
		 f0000000 8000011f 00000000 00000002
		 00000008 00000012 00000000 80000120
		 00000000 00000002 00000008 00000012
		 10000000 80000121 00000000 00000002
		 00000008 00000012 20000000 80000122
		 00000000 00000002 00000008 00000012
		 30000000 80000123 00000000 00000002
		 00000008 00000012 40000000 80000124
		 00000000 00000002 00000008 00000012
		 50000000 80000125 00000000 00000002
		 00000008 00000012 60000000 80000126
		 00000000 00000002 00000008 00000012
		 70000000 80000127 00000000 00000002
		 00000008 00000012 80000000 80000128
		 00000000 00000002 00000008 00000012
		 90000000 80000129 00000000 00000002
		 00000008 00000012 a0000000 8000012a
		 00000000 00000002 00000008 00000012
		 b0000000 8000012b 00000000 00000002
		 00000008 00000012 c0000000 8000012c
		 00000000 00000002 00000008 00000012
		 d0000000 8000012d 00000000 00000002
		 00000008 00000012 e0000000 8000012e
		 00000000 00000002 00000008 00000012
		 f0000000 8000012f 00000000 00000002
		 00000008 00000013 00000000 80000130
		 00000000 00000002 00000008 00000013
		 10000000 80000131 00000000 00000002
		 00000008 00000013 20000000 80000132
		 00000000 00000002 00000008 00000013
		 30000000 80000133 00000000 00000002
		 00000008 00000013 40000000 80000134
		 00000000 00000002 00000008 00000013
		 50000000 80000135 00000000 00000002
		 00000008 00000013 60000000 80000136
		 00000000 00000002 00000008 00000013
		 70000000 80000137 00000000 00000002
		 00000008 00000013 80000000 80000138
		 00000000 00000002 00000008 00000013
		 90000000 80000139 00000000 00000002
		 00000008 00000013 a0000000 8000013a
		 00000000 00000002 00000008 00000013
		 b0000000 8000013b 00000000 00000002
		 00000008 00000013 c0000000 8000013c
		 00000000 00000002 00000008 00000013
		 d0000000 8000013d 00000000 00000002
		 00000008 00000013 e0000000 8000013e
		 00000000 00000002 00000008 00000013
		 f0000000 8000013f 00000000 00000002
		 00000008 00000014 00000000 80000140
		 00000000 00000002 00000008 00000014
		 10000000 80000141 00000000 00000002
		 00000008 00000014 20000000 80000142
		 00000000 00000002 00000008 00000014
		 30000000 80000143 00000000 00000002
		 00000008 00000014 40000000 80000144
		 00000000 00000002 00000008 00000014
		 50000000 80000145 00000000 00000002
		 00000008 00000014 60000000 80000146
		 00000000 00000002 00000008 00000014
		 70000000 80000147 00000000 00000002
		 00000008 00000014 80000000 80000148
		 00000000 00000002 00000008 00000014
		 90000000 80000149 00000000 00000002
		 00000008 00000014 a0000000 8000014a
		 00000000 00000002 00000008 00000014
		 b0000000 8000014b 00000000 00000002
		 00000008 00000014 c0000000 8000014c
		 00000000 00000002 00000008 00000014
		 d0000000 8000014d 00000000 00000002
		 00000008 00000014 e0000000 8000014e
		 00000000 00000002 00000008 00000014
		 f0000000 8000014f 00000000 00000002
		 00000008 00000015 00000000 80000150
		 00000000 00000002 00000008 00000015
		 10000000 80000151 00000000 00000002
		 00000008 00000015 20000000 80000152
		 00000000 00000002 00000008 00000015
		 30000000 80000153 00000000 00000002
		 00000008 00000015 40000000 80000154
		 00000000 00000002 00000008 00000015
		 50000000 80000155 00000000 00000002
		 00000008 00000015 60000000 80000156
		 00000000 00000002 00000008 00000015
		 70000000 80000157 00000000 00000002
		 00000008 00000015 80000000 80000158
		 00000000 00000002 00000008 00000015
		 90000000 80000159 00000000 00000002
		 00000008 00000015 a0000000 8000015a
		 00000000 00000002 00000008 00000015
		 b0000000 8000015b 00000000 00000002
		 00000008 00000015 c0000000 8000015c
		 00000000 00000002 00000008 00000015
		 d0000000 8000015d 00000000 00000002
		 00000008 00000015 e0000000 8000015e
		 00000000 00000002 00000008 00000015
		 f0000000 8000015f 00000000 00000002
		 00000008 00000016 00000000 80000160
		 00000000 00000002 00000008 00000016
		 10000000 80000161 00000000 00000002
		 00000008 00000016 20000000 80000162
		 00000000 00000002 00000008 00000016
		 30000000 80000163 00000000 00000002
		 00000008 00000016 40000000 80000164
		 00000000 00000002 00000008 00000016
		 50000000 80000165 00000000 00000002
		 00000008 00000016 60000000 80000166
		 00000000 00000002 00000008 00000016
		 70000000 80000167 00000000 00000002
		 00000008 00000016 80000000 80000168
		 00000000 00000002 00000008 00000016
		 90000000 80000169 00000000 00000002
		 00000008 00000016 a0000000 8000016a
		 00000000 00000002 00000008 00000016
		 b0000000 8000016b 00000000 00000002
		 00000008 00000016 c0000000 8000016c
		 00000000 00000002 00000008 00000016
		 d0000000 8000016d 00000000 00000002
		 00000008 00000016 e0000000 8000016e
		 00000000 00000002 00000008 00000016
		 f0000000 8000016f 00000000 00000002
		 00000008 00000017 00000000 80000170
		 00000000 00000002 00000008 00000017
		 10000000 80000171 00000000 00000002
		 00000008 00000017 20000000 80000172
		 00000000 00000002 00000008 00000017
		 30000000 80000173 00000000 00000002
		 00000008 00000017 40000000 80000174
		 00000000 00000002 00000008 00000017
		 50000000 80000175 00000000 00000002
		 00000008 00000017 60000000 80000176
		 00000000 00000002 00000008 00000017
		 70000000 80000177 00000000 00000002
		 00000008 00000017 80000000 80000178
		 00000000 00000002 00000008 00000017
		 90000000 80000179 00000000 00000002
		 00000008 00000017 a0000000 8000017a
		 00000000 00000002 00000008 00000017
		 b0000000 8000017b 00000000 00000002
		 00000008 00000017 c0000000 8000017c
		 00000000 00000002 00000008 00000017
		 d0000000 8000017d 00000000 00000002
		 00000008 00000017 e0000000 8000017e
		 00000000 00000002 00000008 00000017
		 f0000000 8000017f 00000000 00000002
		 00000008 00000018 00000000 80000180
		 00000000 00000002 00000008 00000018
		 10000000 80000181 00000000 00000002
		 00000008 00000018 20000000 80000182
		 00000000 00000002 00000008 00000018
		 30000000 80000183 00000000 00000002
		 00000008 00000018 40000000 80000184
		 00000000 00000002 00000008 00000018
		 50000000 80000185 00000000 00000002
		 00000008 00000018 60000000 80000186
		 00000000 00000002 00000008 00000018
		 70000000 80000187 00000000 00000002
		 00000008 00000018 80000000 80000188
		 00000000 00000002 00000008 00000018
		 90000000 80000189 00000000 00000002
		 00000008 00000018 a0000000 8000018a
		 00000000 00000002 00000008 00000018
		 b0000000 8000018b 00000000 00000002
		 00000008 00000018 c0000000 8000018c
		 00000000 00000002 00000008 00000018
		 d0000000 8000018d 00000000 00000002
		 00000008 00000018 e0000000 8000018e
		 00000000 00000002 00000008 00000018
		 f0000000 8000018f 00000000 00000002
		 00000008 00000019 00000000 80000190
		 00000000 00000002 00000008 00000019
		 10000000 80000191 00000000 00000002
		 00000008 00000019 20000000 80000192
		 00000000 00000002 00000008 00000019
		 30000000 80000193 00000000 00000002
		 00000008 00000019 40000000 80000194
		 00000000 00000002 00000008 00000019
		 50000000 80000195 00000000 00000002
		 00000008 00000019 60000000 80000196
		 00000000 00000002 00000008 00000019
		 70000000 80000197 00000000 00000002
		 00000008 00000019 80000000 80000198
		 00000000 00000002 00000008 00000019
		 90000000 80000199 00000000 00000002
		 00000008 00000019 a0000000 8000019a
		 00000000 00000002 00000008 00000019
		 b0000000 8000019b 00000000 00000002
		 00000008 00000019 c0000000 8000019c
		 00000000 00000002 00000008 00000019
		 d0000000 8000019d 00000000 00000002
		 00000008 00000019 e0000000 8000019e
		 00000000 00000002 00000008 00000019
		 f0000000 8000019f 00000000 00000002
		 00000008 0000001a 00000000 800001a0
		 00000000 00000002 00000008 0000001a
		 10000000 800001a1 00000000 00000002
		 00000008 0000001a 20000000 800001a2
		 00000000 00000002 00000008 0000001a
		 30000000 800001a3 00000000 00000002
		 00000008 0000001a 40000000 800001a4
		 00000000 00000002 00000008 0000001a
		 50000000 800001a5 00000000 00000002
		 00000008 0000001a 60000000 800001a6
		 00000000 00000002 00000008 0000001a
		 70000000 800001a7 00000000 00000002
		 00000008 0000001a 80000000 800001a8
		 00000000 00000002 00000008 0000001a
		 90000000 800001a9 00000000 00000002
		 00000008 0000001a a0000000 800001aa
		 00000000 00000002 00000008 0000001a
		 b0000000 800001ab 00000000 00000002
		 00000008 0000001a c0000000 800001ac
		 00000000 00000002 00000008 0000001a
		 d0000000 800001ad 00000000 00000002
		 00000008 0000001a e0000000 800001ae
		 00000000 00000002 00000008 0000001a
		 f0000000 800001af 00000000 00000002
		 00000008 0000001b 00000000 800001b0
		 00000000 00000002 00000008 0000001b
		 10000000 800001b1 00000000 00000002
		 00000008 0000001b 20000000 800001b2
		 00000000 00000002 00000008 0000001b
		 30000000 800001b3 00000000 00000002
		 00000008 0000001b 40000000 800001b4
		 00000000 00000002 00000008 0000001b
		 50000000 800001b5 00000000 00000002
		 00000008 0000001b 60000000 800001b6
		 00000000 00000002 00000008 0000001b
		 70000000 800001b7 00000000 00000002
		 00000008 0000001b 80000000 800001b8
		 00000000 00000002 00000008 0000001b
		 90000000 800001b9 00000000 00000002
		 00000008 0000001b a0000000 800001ba
		 00000000 00000002 00000008 0000001b
		 b0000000 800001bb 00000000 00000002
		 00000008 0000001b c0000000 800001bc
		 00000000 00000002 00000008 0000001b
		 d0000000 800001bd 00000000 00000002
		 00000008 0000001b e0000000 800001be
		 00000000 00000002 00000008 0000001b
		 f0000000 800001bf 00000000 00000002
		 00000008 0000001c 00000000 800001c0
		 00000000 00000002 00000008 0000001c
		 10000000 800001c1 00000000 00000002
		 00000008 0000001c 20000000 800001c2
		 00000000 00000002 00000008 0000001c
		 30000000 800001c3 00000000 00000002
		 00000008 0000001c 40000000 800001c4
		 00000000 00000002 00000008 0000001c
		 50000000 800001c5 00000000 00000002
		 00000008 0000001c 60000000 800001c6
		 00000000 00000002 00000008 0000001c
		 70000000 800001c7 00000000 00000002
		 00000008 0000001c 80000000 800001c8
		 00000000 00000002 00000008 0000001c
		 90000000 800001c9 00000000 00000002
		 00000008 0000001c a0000000 800001ca
		 00000000 00000002 00000008 0000001c
		 b0000000 800001cb 00000000 00000002
		 00000008 0000001c c0000000 800001cc
		 00000000 00000002 00000008 0000001c
		 d0000000 800001cd 00000000 00000002
		 00000008 0000001c e0000000 800001ce
		 00000000 00000002 00000008 0000001c
		 f0000000 800001cf 00000000 00000002
		 00000008 0000001d 00000000 800001d0
		 00000000 00000002 00000008 0000001d
		 10000000 800001d1 00000000 00000002
		 00000008 0000001d 20000000 800001d2
		 00000000 00000002 00000008 0000001d
		 30000000 800001d3 00000000 00000002
		 00000008 0000001d 40000000 800001d4
		 00000000 00000002 00000008 0000001d
		 50000000 800001d5 00000000 00000002
		 00000008 0000001d 60000000 800001d6
		 00000000 00000002 00000008 0000001d
		 70000000 800001d7 00000000 00000002
		 00000008 0000001d 80000000 800001d8
		 00000000 00000002 00000008 0000001d
		 90000000 800001d9 00000000 00000002
		 00000008 0000001d a0000000 800001da
		 00000000 00000002 00000008 0000001d
		 b0000000 800001db 00000000 00000002
		 00000008 0000001d c0000000 800001dc
		 00000000 00000002 00000008 0000001d
		 d0000000 800001dd 00000000 00000002
		 00000008 0000001d e0000000 800001de
		 00000000 00000002 00000008 0000001d
		 f0000000 800001df 00000000 00000002
		 00000008 0000001e 00000000 800001e0
		 00000000 00000002 00000008 0000001e
		 10000000 800001e1 00000000 00000002
		 00000008 0000001e 20000000 800001e2
		 00000000 00000002 00000008 0000001e
		 30000000 800001e3 00000000 00000002
		 00000008 0000001e 40000000 800001e4
		 00000000 00000002 00000008 0000001e
		 50000000 800001e5 00000000 00000002
		 00000008 0000001e 60000000 800001e6
		 00000000 00000002 00000008 0000001e
		 70000000 800001e7 00000000 00000002
		 00000008 0000001e 80000000 800001e8
		 00000000 00000002 00000008 0000001e
		 90000000 800001e9 00000000 00000002
		 00000008 0000001e a0000000 800001ea
		 00000000 00000002 00000008 0000001e
		 b0000000 800001eb 00000000 00000002
		 00000008 0000001e c0000000 800001ec
		 00000000 00000002 00000008 0000001e
		 d0000000 800001ed 00000000 00000002
		 00000008 0000001e e0000000 800001ee
		 00000000 00000002 00000008 0000001e
		 f0000000 800001ef 00000000 00000002
		 00000008 0000001f 00000000 800001f0
		 00000000 00000002 00000008 0000001f
		 10000000 800001f1 00000000 00000002
		 00000008 0000001f 20000000 800001f2
		 00000000 00000002 00000008 0000001f
		 30000000 800001f3 00000000 00000002
		 00000008 0000001f 40000000 800001f4
		 00000000 00000002 00000008 0000001f
		 50000000 800001f5 00000000 00000002
		 00000008 0000001f 60000000 800001f6
		 00000000 00000002 00000008 0000001f
		 70000000 800001f7 00000000 00000002
		 00000008 0000001f 80000000 800001f8
		 00000000 00000002 00000008 0000001f
		 90000000 800001f9 00000000 00000002
		 00000008 0000001f a0000000 800001fa
		 00000000 00000002 00000008 0000001f
		 b0000000 800001fb 00000000 00000002
		 00000008 0000001f c0000000 800001fc
		 00000000 00000002 00000008 0000001f
		 d0000000 800001fd 00000000 00000002
		 00000008 0000001f e0000000 800001fe
		 00000000 00000002 00000008 0000001f
		 f0000000 800001ff 00000000 00000002
		 00000008 00000020 00000000 80000200
		 00000000 00000002 00000008 00000020
		 10000000 80000201 00000000 00000002
		 00000008 00000020 20000000 80000202
		 00000000 00000002 00000008 00000020
		 30000000 80000203 00000000 00000002
		 00000008 00000020 40000000 80000204
		 00000000 00000002 00000008 00000020
		 50000000 80000205 00000000 00000002
		 00000008 00000020 60000000 80000206
		 00000000 00000002 00000008 00000020
		 70000000 80000207 00000000 00000002
		 00000008 00000020 80000000 80000208
		 00000000 00000002 00000008 00000020
		 90000000 80000209 00000000 00000002
		 00000008 00000020 a0000000 8000020a
		 00000000 00000002 00000008 00000020
		 b0000000 8000020b 00000000 00000002
		 00000008 00000020 c0000000 8000020c
		 00000000 00000002 00000008 00000020
		 d0000000 8000020d 00000000 00000002
		 00000008 00000020 e0000000 8000020e
		 00000000 00000002 00000008 00000020
		 f0000000 8000020f 00000000 00000002
		 00000008 00000021 00000000 80000210
		 00000000 00000002 00000008 00000021
		 10000000 80000211 00000000 00000002
		 00000008 00000021 20000000 80000212
		 00000000 00000002 00000008 00000021
		 30000000 80000213 00000000 00000002
		 00000008 00000021 40000000 80000214
		 00000000 00000002 00000008 00000021
		 50000000 80000215 00000000 00000002
		 00000008 00000021 60000000 80000216
		 00000000 00000002 00000008 00000021
		 70000000 80000217 00000000 00000002
		 00000008 00000021 80000000 80000218
		 00000000 00000002 00000008 00000021
		 90000000 80000219 00000000 00000002
		 00000008 00000021 a0000000 8000021a
		 00000000 00000002 00000008 00000021
		 b0000000 8000021b 00000000 00000002
		 00000008 00000021 c0000000 8000021c
		 00000000 00000002 00000008 00000021
		 d0000000 8000021d 00000000 00000002
		 00000008 00000021 e0000000 8000021e
		 00000000 00000002 00000008 00000021
		 f0000000 8000021f 00000000 00000002
		 00000008 00000022 00000000 80000220
		 00000000 00000002 00000008 00000022
		 10000000 80000221 00000000 00000002
		 00000008 00000022 20000000 80000222
		 00000000 00000002 00000008 00000022
		 30000000 80000223 00000000 00000002
		 00000008 00000022 40000000 80000224
		 00000000 00000002 00000008 00000022
		 50000000 80000225 00000000 00000002
		 00000008 00000022 60000000 80000226
		 00000000 00000002 00000008 00000022
		 70000000 80000227 00000000 00000002
		 00000008 00000022 80000000 80000228
		 00000000 00000002 00000008 00000022
		 90000000 80000229 00000000 00000002
		 00000008 00000022 a0000000 8000022a
		 00000000 00000002 00000008 00000022
		 b0000000 8000022b 00000000 00000002
		 00000008 00000022 c0000000 8000022c
		 00000000 00000002 00000008 00000022
		 d0000000 8000022d 00000000 00000002
		 00000008 00000022 e0000000 8000022e
		 00000000 00000002 00000008 00000022
		 f0000000 8000022f 00000000 00000002
		 00000008 00000023 00000000 80000230
		 00000000 00000002 00000008 00000023
		 10000000 80000231 00000000 00000002
		 00000008 00000023 20000000 80000232
		 00000000 00000002 00000008 00000023
		 30000000 80000233 00000000 00000002
		 00000008 00000023 40000000 80000234
		 00000000 00000002 00000008 00000023
		 50000000 80000235 00000000 00000002
		 00000008 00000023 60000000 80000236
		 00000000 00000002 00000008 00000023
		 70000000 80000237 00000000 00000002
		 00000008 00000023 80000000 80000238
		 00000000 00000002 00000008 00000023
		 90000000 80000239 00000000 00000002
		 00000008 00000023 a0000000 8000023a
		 00000000 00000002 00000008 00000023
		 b0000000 8000023b 00000000 00000002
		 00000008 00000023 c0000000 8000023c
		 00000000 00000002 00000008 00000023
		 d0000000 8000023d 00000000 00000002
		 00000008 00000023 e0000000 8000023e
		 00000000 00000002 00000008 00000023
		 f0000000 8000023f 00000000 00000002
		 00000008 00000024 00000000 80000240
		 00000000 00000002 00000008 00000024
		 10000000 80000241 00000000 00000002
		 00000008 00000024 20000000 80000242
		 00000000 00000002 00000008 00000024
		 30000000 80000243 00000000 00000002
		 00000008 00000024 40000000 80000244
		 00000000 00000002 00000008 00000024
		 50000000 80000245 00000000 00000002
		 00000008 00000024 60000000 80000246
		 00000000 00000002 00000008 00000024
		 70000000 80000247 00000000 00000002
		 00000008 00000024 80000000 80000248
		 00000000 00000002 00000008 00000024
		 90000000 80000249 00000000 00000002
		 00000008 00000024 a0000000 8000024a
		 00000000 00000002 00000008 00000024
		 b0000000 8000024b 00000000 00000002
		 00000008 00000024 c0000000 8000024c
		 00000000 00000002 00000008 00000024
		 d0000000 8000024d 00000000 00000002
		 00000008 00000024 e0000000 8000024e
		 00000000 00000002 00000008 00000024
		 f0000000 8000024f 00000000 00000002
		 00000008 00000025 00000000 80000250
		 00000000 00000002 00000008 00000025
		 10000000 80000251 00000000 00000002
		 00000008 00000025 20000000 80000252
		 00000000 00000002 00000008 00000025
		 30000000 80000253 00000000 00000002
		 00000008 00000025 40000000 80000254
		 00000000 00000002 00000008 00000025
		 50000000 80000255 00000000 00000002
		 00000008 00000025 60000000 80000256
		 00000000 00000002 00000008 00000025
		 70000000 80000257 00000000 00000002
		 00000008 00000025 80000000 80000258
		 00000000 00000002 00000008 00000025
		 90000000 80000259 00000000 00000002
		 00000008 00000025 a0000000 8000025a
		 00000000 00000002 00000008 00000025
		 b0000000 8000025b 00000000 00000002
		 00000008 00000025 c0000000 8000025c
		 00000000 00000002 00000008 00000025
		 d0000000 8000025d 00000000 00000002
		 00000008 00000025 e0000000 8000025e
		 00000000 00000002 00000008 00000025
		 f0000000 8000025f 00000000 00000002
		 00000008 00000026 00000000 80000260
		 00000000 00000002 00000008 00000026
		 10000000 80000261 00000000 00000002
		 00000008 00000026 20000000 80000262
		 00000000 00000002 00000008 00000026
		 30000000 80000263 00000000 00000002
		 00000008 00000026 40000000 80000264
		 00000000 00000002 00000008 00000026
		 50000000 80000265 00000000 00000002
		 00000008 00000026 60000000 80000266
		 00000000 00000002 00000008 00000026
		 70000000 80000267 00000000 00000002
		 00000008 00000026 80000000 80000268
		 00000000 00000002 00000008 00000026
		 90000000 80000269 00000000 00000002
		 00000008 00000026 a0000000 8000026a
		 00000000 00000002 00000008 00000026
		 b0000000 8000026b 00000000 00000003
		 00000008 00000026 c0000000 8000026c
		 00000000 00000003 00000008 00000026
		 d0000000 8000026d 00000000 00000003
		 00000008 00000026 e0000000 8000026e
		 00000000 00000003 00000008 00000026
		 f0000000 8000026f 00000000 00000003
		 00000008 00000027 00000000 80000270
		 00000000 00000003 00000008 00000027
		 10000000 80000271 00000000 00000003
		 00000008 00000027 20000000 80000272
		 00000000 00000003 00000008 00000027
		 30000000 80000273 00000000 00000003
		 00000008 00000027 40000000 80000274
		 00000000 00000003 00000008 00000027
		 50000000 80000275 00000000 00000003
		 00000008 00000027 60000000 80000276
		 00000000 00000003 00000008 00000027
		 70000000 80000277 00000000 00000003
		 00000008 00000027 80000000 80000278
		 00000000 00000003 00000008 00000027
		 90000000 80000279 00000000 00000003
		 00000008 00000027 a0000000 8000027a
		 00000000 00000003 00000008 00000027
		 b0000000 8000027b 00000000 00000003
		 00000008 00000027 c0000000 8000027c
		 00000000 00000003 00000008 00000027
		 d0000000 8000027d 00000000 00000003
		 00000008 00000027 e0000000 8000027e
		 00000000 00000003 00000008 00000027
		 f0000000 8000027f 00000000 00000003
		 00000008 00000028 00000000 80000280
		 00000000 00000003 00000008 00000028
		 10000000 80000281 00000000 00000003
		 00000008 00000028 20000000 80000282
		 00000000 00000003 00000008 00000028
		 30000000 80000283 00000000 00000003
		 00000008 00000028 40000000 80000284
		 00000000 00000003 00000008 00000028
		 50000000 80000285 00000000 00000003
		 00000008 00000028 60000000 80000286
		 00000000 00000003 00000008 00000028
		 70000000 80000287 00000000 00000003
		 00000008 00000028 80000000 80000288
		 00000000 00000003 00000008 00000028
		 90000000 80000289 00000000 00000003
		 00000008 00000028 a0000000 8000028a
		 00000000 00000003 00000008 00000028
		 b0000000 8000028b 00000000 00000003
		 00000008 00000028 c0000000 8000028c
		 00000000 00000003 00000008 00000028
		 d0000000 8000028d 00000000 00000003
		 00000008 00000028 e0000000 8000028e
		 00000000 00000003 00000008 00000028
		 f0000000 8000028f 00000000 00000003
		 00000008 00000029 00000000 80000290
		 00000000 00000003 00000008 00000029
		 10000000 80000291 00000000 00000003
		 00000008 00000029 20000000 80000292
		 00000000 00000003 00000008 00000029
		 30000000 80000293 00000000 00000003
		 00000008 00000029 40000000 80000294
		 00000000 00000003 00000008 00000029
		 50000000 80000295 00000000 00000003
		 00000008 00000029 60000000 80000296
		 00000000 00000003 00000008 00000029
		 70000000 80000297 00000000 00000003
		 00000008 00000029 80000000 80000298
		 00000000 00000003 00000008 00000029
		 90000000 80000299 00000000 00000003
		 00000008 00000029 a0000000 8000029a
		 00000000 00000003 00000008 00000029
		 b0000000 8000029b 00000000 00000003
		 00000008 00000029 c0000000 8000029c
		 00000000 00000003 00000008 00000029
		 d0000000 8000029d 00000000 00000003
		 00000008 00000029 e0000000 8000029e
		 00000000 00000003 00000008 00000029
		 f0000000 8000029f 00000000 00000003
		 00000008 0000002a 00000000 800002a0
		 00000000 00000003 00000008 0000002a
		 10000000 800002a1 00000000 00000003
		 00000008 0000002a 20000000 800002a2
		 00000000 00000003 00000008 0000002a
		 30000000 800002a3 00000000 00000003
		 00000008 0000002a 40000000 800002a4
		 00000000 00000003 00000008 0000002a
		 50000000 800002a5 00000000 00000003
		 00000008 0000002a 60000000 800002a6
		 00000000 00000003 00000008 0000002a
		 70000000 800002a7 00000000 00000003
		 00000008 0000002a 80000000 800002a8
		 00000000 00000003 00000008 0000002a
		 90000000 800002a9 00000000 00000003
		 00000008 0000002a a0000000 800002aa
		 00000000 00000003 00000008 0000002a
		 b0000000 800002ab 00000000 00000003
		 00000008 0000002a c0000000 800002ac
		 00000000 00000003 00000008 0000002a
		 d0000000 800002ad 00000000 00000003
		 00000008 0000002a e0000000 800002ae
		 00000000 00000003 00000008 0000002a
		 f0000000 800002af 00000000 00000003
		 00000008 0000002b 00000000 800002b0
		 00000000 00000003 00000008 0000002b
		 10000000 800002b1 00000000 00000003
		 00000008 0000002b 20000000 800002b2
		 00000000 00000003 00000008 0000002b
		 30000000 800002b3 00000000 00000003
		 00000008 0000002b 40000000 800002b4
		 00000000 00000003 00000008 0000002b
		 50000000 800002b5 00000000 00000003
		 00000008 0000002b 60000000 800002b6
		 00000000 00000003 00000008 0000002b
		 70000000 800002b7 00000000 00000003
		 00000008 0000002b 80000000 800002b8
		 00000000 00000003 00000008 0000002b
		 90000000 800002b9 00000000 00000003
		 00000008 0000002b a0000000 800002ba
		 00000000 00000003 00000008 0000002b
		 b0000000 800002bb 00000000 00000003
		 00000008 0000002b c0000000 800002bc
		 00000000 00000003 00000008 0000002b
		 d0000000 800002bd 00000000 00000003
		 00000008 0000002b e0000000 800002be
		 00000000 00000003 00000008 0000002b
		 f0000000 800002bf 00000000 00000003
		 00000008 0000002c 00000000 800002c0
		 00000000 00000003 00000008 0000002c
		 10000000 800002c1 00000000 00000003
		 00000008 0000002c 20000000 800002c2
		 00000000 00000003 00000008 0000002c
		 30000000 800002c3 00000000 00000003
		 00000008 0000002c 40000000 800002c4
		 00000000 00000003 00000008 0000002c
		 50000000 800002c5 00000000 00000003
		 00000008 0000002c 60000000 800002c6
		 00000000 00000003 00000008 0000002c
		 70000000 800002c7 00000000 00000003
		 00000008 0000002c 80000000 800002c8
		 00000000 00000003 00000008 0000002c
		 90000000 800002c9 00000000 00000003
		 00000008 0000002c a0000000 800002ca
		 00000000 00000003 00000008 0000002c
		 b0000000 800002cb 00000000 00000003
		 00000008 0000002c c0000000 800002cc
		 00000000 00000003 00000008 0000002c
		 d0000000 800002cd 00000000 00000003
		 00000008 0000002c e0000000 800002ce
		 00000000 00000003 00000008 0000002c
		 f0000000 800002cf 00000000 00000003
		 00000008 0000002d 00000000 800002d0
		 00000000 00000003 00000008 0000002d
		 10000000 800002d1 00000000 00000003
		 00000008 0000002d 20000000 800002d2
		 00000000 00000003 00000008 0000002d
		 30000000 800002d3 00000000 00000003
		 00000008 0000002d 40000000 800002d4
		 00000000 00000003 00000008 0000002d
		 50000000 800002d5 00000000 00000003
		 00000008 0000002d 60000000 800002d6
		 00000000 00000003 00000008 0000002d
		 70000000 800002d7 00000000 00000003
		 00000008 0000002d 80000000 800002d8
		 00000000 00000003 00000008 0000002d
		 90000000 800002d9 00000000 00000003
		 00000008 0000002d a0000000 800002da
		 00000000 00000003 00000008 0000002d
		 b0000000 800002db 00000000 00000003
		 00000008 0000002d c0000000 800002dc
		 00000000 00000003 00000008 0000002d
		 d0000000 800002dd 00000000 00000003
		 00000008 0000002d e0000000 800002de
		 00000000 00000003 00000008 0000002d
		 f0000000 800002df 00000000 00000003
		 00000008 0000002e 00000000 800002e0
		 00000000 00000003 00000008 0000002e
		 10000000 800002e1 00000000 00000003
		 00000008 0000002e 20000000 800002e2
		 00000000 00000003 00000008 0000002e
		 30000000 800002e3 00000000 00000003
		 00000008 0000002e 40000000 800002e4
		 00000000 00000003 00000008 0000002e
		 50000000 800002e5 00000000 00000003
		 00000008 0000002e 60000000 800002e6
		 00000000 00000003 00000008 0000002e
		 70000000 800002e7 00000000 00000003
		 00000008 0000002e 80000000 800002e8
		 00000000 00000003 00000008 0000002e
		 90000000 800002e9 00000000 00000003
		 00000008 0000002e a0000000 800002ea
		 00000000 00000003 00000008 0000002e
		 b0000000 800002eb 00000000 00000003
		 00000008 0000002e c0000000 800002ec
		 00000000 00000003 00000008 0000002e
		 d0000000 800002ed 00000000 00000003
		 00000008 0000002e e0000000 800002ee
		 00000000 00000003 00000008 0000002e
		 f0000000 800002ef 00000000 00000003
		 00000008 0000002f 00000000 800002f0
		 00000000 00000003 00000008 0000002f
		 10000000 800002f1 00000000 00000003
		 00000008 0000002f 20000000 800002f2
		 00000000 00000003 00000008 0000002f
		 30000000 800002f3 00000000 00000003
		 00000008 0000002f 40000000 800002f4
		 00000000 00000003 00000008 0000002f
		 50000000 800002f5 00000000 00000003
		 00000008 0000002f 60000000 800002f6
		 00000000 00000003 00000008 0000002f
		 70000000 800002f7 00000000 00000003
		 00000008 0000002f 80000000 800002f8
		 00000000 00000003 00000008 0000002f
		 90000000 800002f9 00000000 00000003
		 00000008 0000002f a0000000 800002fa
		 00000000 00000003 00000008 0000002f
		 b0000000 800002fb 00000000 00000003
		 00000008 0000002f c0000000 800002fc
		 00000000 00000003 00000008 0000002f
		 d0000000 800002fd 00000000 00000003
		 00000008 0000002f e0000000 800002fe
		 00000000 00000003 00000008 0000002f
		 f0000000 800002ff 00000000 00000003
		 00000008 00000030 00000000 80000300
		 00000000 00000003 00000008 00000030
		 10000000 80000301 00000000 00000003
		 00000008 00000030 20000000 80000302
		 00000000 00000003 00000008 00000030
		 30000000 80000303 00000000 00000003
		 00000008 00000030 40000000 80000304
		 00000000 00000003 00000008 00000030
		 50000000 80000305 00000000 00000003
		 00000008 00000030 60000000 80000306
		 00000000 00000003 00000008 00000030
		 70000000 80000307 00000000 00000003
		 00000008 00000030 80000000 80000308
		 00000000 00000003 00000008 00000030
		 90000000 80000309 00000000 00000003
		 00000008 00000030 a0000000 8000030a
		 00000000 00000003 00000008 00000030
		 b0000000 8000030b 00000000 00000003
		 00000008 00000030 c0000000 8000030c
		 00000000 00000003 00000008 00000030
		 d0000000 8000030d 00000000 00000003
		 00000008 00000030 e0000000 8000030e
		 00000000 00000003 00000008 00000030
		 f0000000 8000030f 00000000 00000003
		 00000008 00000031 00000000 80000310
		 00000000 00000003 00000008 00000031
		 10000000 80000311 00000000 00000003
		 00000008 00000031 20000000 80000312
		 00000000 00000003 00000008 00000031
		 30000000 80000313 00000000 00000003
		 00000008 00000031 40000000 80000314
		 00000000 00000003 00000008 00000031
		 50000000 80000315 00000000 00000003
		 00000008 00000031 60000000 80000316
		 00000000 00000003 00000008 00000031
		 70000000 80000317 00000000 00000003
		 00000008 00000031 80000000 80000318
		 00000000 00000003 00000008 00000031
		 90000000 80000319 00000000 00000003
		 00000008 00000031 a0000000 8000031a
		 00000000 00000003 00000008 00000031
		 b0000000 8000031b 00000000 00000003
		 00000008 00000031 c0000000 8000031c
		 00000000 00000003 00000008 00000031
		 d0000000 8000031d 00000000 00000003
		 00000008 00000031 e0000000 8000031e
		 00000000 00000003 00000008 00000031
		 f0000000 8000031f 00000000 00000003
		 00000008 00000032 00000000 80000320
		 00000000 00000003 00000008 00000032
		 10000000 80000321 00000000 00000003
		 00000008 00000032 20000000 80000322
		 00000000 00000003 00000008 00000032
		 30000000 80000323 00000000 00000003
		 00000008 00000032 40000000 80000324
		 00000000 00000003 00000008 00000032
		 50000000 80000325 00000000 00000003
		 00000008 00000032 60000000 80000326
		 00000000 00000003 00000008 00000032
		 70000000 80000327 00000000 00000003
		 00000008 00000032 80000000 80000328
		 00000000 00000003 00000008 00000032
		 90000000 80000329 00000000 00000003
		 00000008 00000032 a0000000 8000032a
		 00000000 00000003 00000008 00000032
		 b0000000 8000032b 00000000 00000003
		 00000008 00000032 c0000000 8000032c
		 00000000 00000003 00000008 00000032
		 d0000000 8000032d 00000000 00000003
		 00000008 00000032 e0000000 8000032e
		 00000000 00000003 00000008 00000032
		 f0000000 8000032f 00000000 00000003
		 00000008 00000033 00000000 80000330
		 00000000 00000003 00000008 00000033
		 10000000 80000331 00000000 00000003
		 00000008 00000033 20000000 80000332
		 00000000 00000003 00000008 00000033
		 30000000 80000333 00000000 00000003
		 00000008 00000033 40000000 80000334
		 00000000 00000003 00000008 00000033
		 50000000 80000335 00000000 00000003
		 00000008 00000033 60000000 80000336
		 00000000 00000003 00000008 00000033
		 70000000 80000337 00000000 00000003
		 00000008 00000033 80000000 80000338
		 00000000 00000003 00000008 00000033
		 90000000 80000339 00000000 00000003
		 00000008 00000033 a0000000 8000033a
		 00000000 00000003 00000008 00000033
		 b0000000 8000033b 00000000 00000003
		 00000008 00000033 c0000000 8000033c
		 00000000 00000003 00000008 00000033
		 d0000000 8000033d 00000000 00000003
		 00000008 00000033 e0000000 8000033e
		 00000000 00000003 00000008 00000033
		 f0000000 8000033f 00000000 00000003
		 00000008 00000034 00000000 80000340
		 00000000 00000003 00000008 00000034
		 10000000 80000341 00000000 00000003
		 00000008 00000034 20000000 80000342
		 00000000 00000003 00000008 00000034
		 30000000 80000343 00000000 00000003
		 00000008 00000034 40000000 80000344
		 00000000 00000003 00000008 00000034
		 50000000 80000345 00000000 00000003
		 00000008 00000034 60000000 80000346
		 00000000 00000003 00000008 00000034
		 70000000 80000347 00000000 00000003
		 00000008 00000034 80000000 80000348
		 00000000 00000003 00000008 00000034
		 90000000 80000349 00000000 00000003
		 00000008 00000034 a0000000 8000034a
		 00000000 00000003 00000008 00000034
		 b0000000 8000034b 00000000 00000003
		 00000008 00000034 c0000000 8000034c
		 00000000 00000003 00000008 00000034
		 d0000000 8000034d 00000000 00000003
		 00000008 00000034 e0000000 8000034e
		 00000000 00000003 00000008 00000034
		 f0000000 8000034f 00000000 00000003
		 00000008 00000035 00000000 80000350
		 00000000 00000003 00000008 00000035
		 10000000 80000351 00000000 00000003
		 00000008 00000035 20000000 80000352
		 00000000 00000003 00000008 00000035
		 30000000 80000353 00000000 00000003
		 00000008 00000035 40000000 80000354
		 00000000 00000003 00000008 00000035
		 50000000 80000355 00000000 00000003
		 00000008 00000035 60000000 80000356
		 00000000 00000003 00000008 00000035
		 70000000 80000357 00000000 00000003
		 00000008 00000035 80000000 80000358
		 00000000 00000003 00000008 00000035
		 90000000 80000359 00000000 00000003
		 00000008 00000035 a0000000 8000035a
		 00000000 00000003 00000008 00000035
		 b0000000 8000035b 00000000 00000003
		 00000008 00000035 c0000000 8000035c
		 00000000 00000003 00000008 00000035
		 d0000000 8000035d 00000000 00000003
		 00000008 00000035 e0000000 8000035e
		 00000000 00000003 00000008 00000035
		 f0000000 8000035f 00000000 00000003
		 00000008 00000036 00000000 80000360
		 00000000 00000003 00000008 00000036
		 10000000 80000361 00000000 00000003
		 00000008 00000036 20000000 80000362
		 00000000 00000003 00000008 00000036
		 30000000 80000363 00000000 00000003
		 00000008 00000036 40000000 80000364
		 00000000 00000003 00000008 00000036
		 50000000 80000365 00000000 00000003
		 00000008 00000036 60000000 80000366
		 00000000 00000003 00000008 00000036
		 70000000 80000367 00000000 00000003
		 00000008 00000036 80000000 80000368
		 00000000 00000003 00000008 00000036
		 90000000 80000369 00000000 00000003
		 00000008 00000036 a0000000 8000036a
		 00000000 00000003 00000008 00000036
		 b0000000 8000036b 00000000 00000003
		 00000008 00000036 c0000000 8000036c
		 00000000 00000003 00000008 00000036
		 d0000000 8000036d 00000000 00000003
		 00000008 00000036 e0000000 8000036e
		 00000000 00000003 00000008 00000036
		 f0000000 8000036f 00000000 00000003
		 00000008 00000037 00000000 80000370
		 00000000 00000003 00000008 00000037
		 10000000 80000371 00000000 00000003
		 00000008 00000037 20000000 80000372
		 00000000 00000003 00000008 00000037
		 30000000 80000373 00000000 00000003
		 00000008 00000037 40000000 80000374
		 00000000 00000003 00000008 00000037
		 50000000 80000375 00000000 00000003
		 00000008 00000037 60000000 80000376
		 00000000 00000003 00000008 00000037
		 70000000 80000377 00000000 00000003
		 00000008 00000037 80000000 80000378
		 00000000 00000003 00000008 00000037
		 90000000 80000379 00000000 00000003
		 00000008 00000037 a0000000 8000037a
		 00000000 00000003 00000008 00000037
		 b0000000 8000037b 00000000 00000003
		 00000008 00000037 c0000000 8000037c
		 00000000 00000003 00000008 00000037
		 d0000000 8000037d 00000000 00000003
		 00000008 00000037 e0000000 8000037e
		 00000000 00000003 00000008 00000037
		 f0000000 8000037f 00000000 00000003
		 00000008 00000038 00000000 80000380
		 00000000 00000003 00000008 00000038
		 10000000 80000381 00000000 00000003
		 00000008 00000038 20000000 80000382
		 00000000 00000003 00000008 00000038
		 30000000 80000383 00000000 00000003
		 00000008 00000038 40000000 80000384
		 00000000 00000003 00000008 00000038
		 50000000 80000385 00000000 00000003
		 00000008 00000038 60000000 80000386
		 00000000 00000003 00000008 00000038
		 70000000 80000387 00000000 00000003
		 00000008 00000038 80000000 80000388
		 00000000 00000003 00000008 00000038
		 90000000 80000389 00000000 00000003
		 00000008 00000038 a0000000 8000038a
		 00000000 00000003 00000008 00000038
		 b0000000 8000038b 00000000 00000003
		 00000008 00000038 c0000000 8000038c
		 00000000 00000003 00000008 00000038
		 d0000000 8000038d 00000000 00000003
		 00000008 00000038 e0000000 8000038e
		 00000000 00000003 00000008 00000038
		 f0000000 8000038f 00000000 00000003
		 00000008 00000039 00000000 80000390
		 00000000 00000003 00000008 00000039
		 10000000 80000391 00000000 00000003
		 00000008 00000039 20000000 80000392
		 00000000 00000003 00000008 00000039
		 30000000 80000393 00000000 00000003
		 00000008 00000039 40000000 80000394
		 00000000 00000003 00000008 00000039
		 50000000 80000395 00000000 00000003
		 00000008 00000039 60000000 80000396
		 00000000 00000003 00000008 00000039
		 70000000 80000397 00000000 00000003
		 00000008 00000039 80000000 80000398
		 00000000 00000003 00000008 00000039
		 90000000 80000399 00000000 00000003
		 00000008 00000039 a0000000 8000039a
		 00000000 00000003 00000008 00000039
		 b0000000 8000039b 00000000 00000003
		 00000008 00000039 c0000000 8000039c
		 00000000 00000003 00000008 00000039
		 d0000000 8000039d 00000000 00000003
		 00000008 00000039 e0000000 8000039e
		 00000000 00000003 00000008 00000039
		 f0000000 8000039f 00000000 00000003
		 00000008 0000003a 00000000 800003a0
		 00000000 00000003 00000008 0000003a
		 10000000 800003a1 00000000 00000003
		 00000008 0000003a 20000000 800003a2
		 00000000 00000003 00000008 0000003a
		 30000000 800003a3 00000000 00000003
		 00000008 0000003a 40000000 800003a4
		 00000000 00000003 00000008 0000003a
		 50000000 800003a5 00000000 00000003
		 00000008 0000003a 60000000 800003a6
		 00000000 00000003 00000008 0000003a
		 70000000 800003a7 00000000 00000003
		 00000008 0000003a 80000000 800003a8
		 00000000 00000003 00000008 0000003a
		 90000000 800003a9 00000000 00000003
		 00000008 0000003a a0000000 800003aa
		 00000000 00000003 00000008 0000003a
		 b0000000 800003ab 00000000 00000003
		 00000008 0000003a c0000000 800003ac
		 00000000 00000003 00000008 0000003a
		 d0000000 800003ad 00000000 00000003
		 00000008 0000003a e0000000 800003ae
		 00000000 00000003 00000008 0000003a
		 f0000000 800003af 00000000 00000003
		 00000008 0000003b 00000000 800003b0
		 00000000 00000003 00000008 0000003b
		 10000000 800003b1 00000000 00000003
		 00000008 0000003b 20000000 800003b2
		 00000000 00000003 00000008 0000003b
		 30000000 800003b3 00000000 00000003
		 00000008 0000003b 40000000 800003b4
		 00000000 00000003 00000008 0000003b
		 50000000 800003b5 00000000 00000003
		 00000008 0000003b 60000000 800003b6
		 00000000 00000003 00000008 0000003b
		 70000000 800003b7 00000000 00000003
		 00000008 0000003b 80000000 800003b8
		 00000000 00000003 00000008 0000003b
		 90000000 800003b9 00000000 00000003
		 00000008 0000003b a0000000 800003ba
		 00000000 00000003 00000008 0000003b
		 b0000000 800003bb 00000000 00000003
		 00000008 0000003b c0000000 800003bc
		 00000000 00000003 00000008 0000003b
		 d0000000 800003bd 00000000 00000003
		 00000008 0000003b e0000000 800003be
		 00000000 00000003 00000008 0000003b
		 f0000000 800003bf 00000000 00000003
		 00000008 0000003c 00000000 800003c0
		 00000000 ffffffff 00000000 0000003c
		 10000000 800003c1 00000000 ffffffff
		 00000000 0000003c 20000000 800003c2
		 00000000 ffffffff 00000000 0000003c
		 30000000 800003c3 00000000 ffffffff
		 00000000 0000003c 40000000 800003c4
		 00000000 ffffffff 00000000 0000003c
		 50000000 800003c5 00000000 ffffffff
		 00000000 0000003c 60000000 800003c6
		 00000000 ffffffff 00000000 0000003c
		 70000000 800003c7 00000000 ffffffff
		 00000000 0000003c 80000000 800003c8
		 00000000 ffffffff 00000000 0000003c
		 90000000 800003c9 00000000 ffffffff
		 00000000 0000003c a0000000 800003ca
		 00000000 ffffffff 00000000 0000003c
		 b0000000 800003cb 00000000 ffffffff
		 00000000 0000003c c0000000 800003cc
		 00000000 ffffffff 00000000 0000003c
		 d0000000 800003cd 00000000 ffffffff
		 00000000 0000003c e0000000 800003ce
		 00000000 ffffffff 00000000 0000003c
		 f0000000 800003cf 00000000 ffffffff
		 00000000 0000003d 00000000 800003d0
		 00000000 ffffffff 00000000 0000003d
		 10000000 800003d1 00000000 ffffffff
		 00000000 0000003d 20000000 800003d2
		 00000000 ffffffff 00000000 0000003d
		 30000000 800003d3 00000000 ffffffff
		 00000000 0000003d 40000000 800003d4
		 00000000 ffffffff 00000000 0000003d
		 50000000 800003d5 00000000 ffffffff
		 00000000 0000003d 60000000 800003d6
		 00000000 ffffffff 00000000 0000003d
		 70000000 800003d7 00000000 ffffffff
		 00000000 0000003d 80000000 800003d8
		 00000000 ffffffff 00000000 0000003d
		 90000000 800003d9 00000000 ffffffff
		 00000000 0000003d a0000000 800003da
		 00000000 ffffffff 00000000 0000003d
		 b0000000 800003db 00000000 ffffffff
		 00000000 0000003d c0000000 800003dc
		 00000000 ffffffff 00000000 0000003d
		 d0000000 800003dd 00000000 ffffffff
		 00000000 0000003d e0000000 800003de
		 00000000 ffffffff 00000000 0000003d
		 f0000000 800003df 00000000 ffffffff
		 00000000 0000003e 00000000 800003e0
		 00000000 ffffffff 00000000 0000003e
		 10000000 800003e1 00000000 ffffffff
		 00000000 0000003e 20000000 800003e2
		 00000000 ffffffff 00000000 0000003e
		 30000000 800003e3 00000000 ffffffff
		 00000000 0000003e 40000000 800003e4
		 00000000 ffffffff 00000000 0000003e
		 50000000 800003e5 00000000 ffffffff
		 00000000 0000003e 60000000 800003e6
		 00000000 ffffffff 00000000 0000003e
		 70000000 800003e7 00000000 ffffffff
		 00000000 0000003e 80000000 800003e8
		 00000000 ffffffff 00000000 0000003e
		 90000000 800003e9 00000000 ffffffff
		 00000000 0000003e a0000000 800003ea
		 00000000 ffffffff 00000000 0000003e
		 b0000000 800003eb 00000000 ffffffff
		 00000000 0000003e c0000000 800003ec
		 00000000 ffffffff 00000000 0000003e
		 d0000000 800003ed 00000000 ffffffff
		 00000000 0000003e e0000000 800003ee
		 00000000 ffffffff 00000000 0000003e
		 f0000000 800003ef 00000000 ffffffff
		 00000000 0000003f 00000000 800003f0
		 00000000 ffffffff 00000000 0000003f
		 10000000 800003f1 00000000 ffffffff
		 00000000 0000003f 20000000 800003f2
		 00000000 ffffffff 00000000 0000003f
		 30000000 800003f3 00000000 ffffffff
		 00000000 0000003f 40000000 800003f4
		 00000000 ffffffff 00000000 0000003f
		 50000000 800003f5 00000000 ffffffff
		 00000000 0000003f 60000000 800003f6
		 00000000 ffffffff 00000000 0000003f
		 70000000 800003f7 00000000 ffffffff
		 00000000 0000003f 80000000 800003f8
		 00000000 ffffffff 00000000 0000003f
		 90000000 800003f9 00000000 ffffffff
		 00000000 0000003f a0000000 800003fa
		 00000000 ffffffff 00000000 0000003f
		 b0000000 800003fb 00000000 ffffffff
		 00000000 0000003f c0000000 800003fc
		 00000000 ffffffff 00000000 0000003f
		 d0000000 800003fd 00000000 ffffffff
		 00000000 0000003f e0000000 800003fe
		 00000000 ffffffff 00000000 0000003f
		 f0000000 800003ff 00000000 ffffffff
		 00000000 00000040 00000000 80000400
		 00000000 ffffffff 00000000 00000040
		 10000000 80000401 00000000 ffffffff
		 00000000 00000040 20000000 80000402
		 00000000 ffffffff 00000000 00000040
		 30000000 80000403 00000000 ffffffff
		 00000000 00000040 40000000 80000404
		 00000000 ffffffff 00000000 00000040
		 50000000 80000405 00000000 ffffffff
		 00000000 00000040 60000000 80000406
		 00000000 ffffffff 00000000 00000040
		 70000000 80000407 00000000 ffffffff
		 00000000 00000040 80000000 80000408
		 00000000 ffffffff 00000000 00000040
		 90000000 80000409 00000000 ffffffff
		 00000000 00000040 a0000000 8000040a
		 00000000 ffffffff 00000000 00000040
		 b0000000 8000040b 00000000 ffffffff
		 00000000 00000040 c0000000 8000040c
		 00000000 ffffffff 00000000 00000040
		 d0000000 8000040d 00000000 ffffffff
		 00000000 00000040 e0000000 8000040e
		 00000000 ffffffff 00000000 00000040
		 f0000000 8000040f 00000000 ffffffff
		 00000000 00000041 00000000 80000410
		 00000000 ffffffff 00000000 00000041
		 10000000 80000411 00000000 ffffffff
		 00000000 00000041 20000000 80000412
		 00000000 ffffffff 00000000 00000041
		 30000000 80000413 00000000 ffffffff
		 00000000 00000041 40000000 80000414
		 00000000 ffffffff 00000000 00000041
		 50000000 80000415 00000000 ffffffff
		 00000000 00000041 60000000 80000416
		 00000000 ffffffff 00000000 00000041
		 70000000 80000417 00000000 ffffffff
		 00000000 00000041 80000000 80000418
		 00000000 ffffffff 00000000 00000041
		 90000000 80000419 00000000 ffffffff
		 00000000 00000041 a0000000 8000041a
		 00000000 ffffffff 00000000 00000041
		 b0000000 8000041b 00000000 ffffffff
		 00000000 00000041 c0000000 8000041c
		 00000000 ffffffff 00000000 00000041
		 d0000000 8000041d 00000000 ffffffff
		 00000000 00000041 e0000000 8000041e
		 00000000 ffffffff 00000000 00000041
		 f0000000 8000041f 00000000 ffffffff
		 00000000 00000042 00000000 80000420
		 00000000 ffffffff 00000000 00000042
		 10000000 80000421 00000000 ffffffff
		 00000000 00000042 20000000 80000422
		 00000000 ffffffff 00000000 00000042
		 30000000 80000423 00000000 ffffffff
		 00000000 00000042 40000000 80000424
		 00000000 ffffffff 00000000 00000042
		 50000000 80000425 00000000 ffffffff
		 00000000 00000042 60000000 80000426
		 00000000 ffffffff 00000000 00000042
		 70000000 80000427 00000000 ffffffff
		 00000000 00000042 80000000 80000428
		 00000000 ffffffff 00000000 00000042
		 90000000 80000429 00000000 ffffffff
		 00000000 00000042 a0000000 8000042a
		 00000000 ffffffff 00000000 00000042
		 b0000000 8000042b 00000000 ffffffff
		 00000000 00000042 c0000000 8000042c
		 00000000 ffffffff 00000000 00000042
		 d0000000 8000042d 00000000 ffffffff
		 00000000 00000042 e0000000 8000042e
		 00000000 ffffffff 00000000 00000042
		 f0000000 8000042f 00000000 ffffffff
		 00000000 00000043 00000000 80000430
		 00000000 ffffffff 00000000 00000043
		 10000000 80000431 00000000 ffffffff
		 00000000 00000043 20000000 80000432
		 00000000 ffffffff 00000000 00000043
		 30000000 80000433 00000000 ffffffff
		 00000000 00000043 40000000 80000434
		 00000000 ffffffff 00000000 00000043
		 50000000 80000435 00000000 ffffffff
		 00000000 00000043 60000000 80000436
		 00000000 ffffffff 00000000 00000043
		 70000000 80000437 00000000 ffffffff
		 00000000 00000043 80000000 80000438
		 00000000 ffffffff 00000000 00000043
		 90000000 80000439 00000000 ffffffff
		 00000000 00000043 a0000000 8000043a
		 00000000 ffffffff 00000000 00000043
		 b0000000 8000043b 00000000 ffffffff
		 00000000 00000043 c0000000 8000043c
		 00000000 ffffffff 00000000 00000043
		 d0000000 8000043d 00000000 ffffffff
		 00000000 00000043 e0000000 8000043e
		 00000000 ffffffff 00000000 00000043
		 f0000000 8000043f 00000000 ffffffff
		 00000000 00000044 00000000 80000440
		 00000000 ffffffff 00000000 00000044
		 10000000 80000441 00000000 ffffffff
		 00000000 00000044 20000000 80000442
		 00000000 ffffffff 00000000 00000044
		 30000000 80000443 00000000 ffffffff
		 00000000 00000044 40000000 80000444
		 00000000 ffffffff 00000000 00000044
		 50000000 80000445 00000000 ffffffff
		 00000000 00000044 60000000 80000446
		 00000000 ffffffff 00000000 00000044
		 70000000 80000447 00000000 ffffffff
		 00000000 00000044 80000000 80000448
		 00000000 ffffffff 00000000 00000044
		 90000000 80000449 00000000 ffffffff
		 00000000 00000044 a0000000 8000044a
		 00000000 ffffffff 00000000 00000044
		 b0000000 8000044b 00000000 ffffffff
		 00000000 00000044 c0000000 8000044c
		 00000000 ffffffff 00000000 00000044
		 d0000000 8000044d 00000000 ffffffff
		 00000000 00000044 e0000000 8000044e
		 00000000 ffffffff 00000000 00000044
		 f0000000 8000044f 00000000 ffffffff
		 00000000 00000045 00000000 80000450
		 00000000 ffffffff 00000000 00000045
		 10000000 80000451 00000000 ffffffff
		 00000000 00000045 20000000 80000452
		 00000000 ffffffff 00000000 00000045
		 30000000 80000453 00000000 ffffffff
		 00000000 00000045 40000000 80000454
		 00000000 ffffffff 00000000 00000045
		 50000000 80000455 00000000 ffffffff
		 00000000 00000045 60000000 80000456
		 00000000 ffffffff 00000000 00000045
		 70000000 80000457 00000000 ffffffff
		 00000000 00000045 80000000 80000458
		 00000000 ffffffff 00000000 00000045
		 90000000 80000459 00000000 ffffffff
		 00000000 00000045 a0000000 8000045a
		 00000000 ffffffff 00000000 00000045
		 b0000000 8000045b 00000000 ffffffff
		 00000000 00000045 c0000000 8000045c
		 00000000 ffffffff 00000000 00000045
		 d0000000 8000045d 00000000 ffffffff
		 00000000 00000045 e0000000 8000045e
		 00000000 ffffffff 00000000 00000045
		 f0000000 8000045f 00000000 ffffffff
		 00000000 00000046 00000000 80000460
		 00000000 ffffffff 00000000 00000046
		 10000000 80000461 00000000 ffffffff
		 00000000 00000046 20000000 80000462
		 00000000 ffffffff 00000000 00000046
		 30000000 80000463 00000000 ffffffff
		 00000000 00000046 40000000 80000464
		 00000000 ffffffff 00000000 00000046
		 50000000 80000465 00000000 ffffffff
		 00000000 00000046 60000000 80000466
		 00000000 ffffffff 00000000 00000046
		 70000000 80000467 00000000 ffffffff
		 00000000 00000046 80000000 80000468
		 00000000 ffffffff 00000000 00000046
		 90000000 80000469 00000000 ffffffff
		 00000000 00000046 a0000000 8000046a
		 00000000 ffffffff 00000000 00000046
		 b0000000 8000046b 00000000 ffffffff
		 00000000 00000046 c0000000 8000046c
		 00000000 ffffffff 00000000 00000046
		 d0000000 8000046d 00000000 ffffffff
		 00000000 00000046 e0000000 8000046e
		 00000000 ffffffff 00000000 00000046
		 f0000000 8000046f 00000000 ffffffff
		 00000000 00000047 00000000 80000470
		 00000000 ffffffff 00000000 00000047
		 10000000 80000471 00000000 ffffffff
		 00000000 00000047 20000000 80000472
		 00000000 ffffffff 00000000 00000047
		 30000000 80000473 00000000 ffffffff
		 00000000 00000047 40000000 80000474
		 00000000 ffffffff 00000000 00000047
		 50000000 80000475 00000000 ffffffff
		 00000000 00000047 60000000 80000476
		 00000000 ffffffff 00000000 00000047
		 70000000 80000477 00000000 ffffffff
		 00000000 00000047 80000000 80000478
		 00000000 ffffffff 00000000 00000047
		 90000000 80000479 00000000 ffffffff
		 00000000 00000047 a0000000 8000047a
		 00000000 ffffffff 00000000 00000047
		 b0000000 8000047b 00000000 ffffffff
		 00000000 00000047 c0000000 8000047c
		 00000000 ffffffff 00000000 00000047
		 d0000000 8000047d 00000000 ffffffff
		 00000000 00000047 e0000000 8000047e
		 00000000 ffffffff 00000000 00000047
		 f0000000 8000047f 00000000 ffffffff
		 00000000 00000048 00000000 80000480
		 00000000 ffffffff 00000000 00000048
		 10000000 80000481 00000000 ffffffff
		 00000000 00000048 20000000 80000482
		 00000000 ffffffff 00000000 00000048
		 30000000 80000483 00000000 ffffffff
		 00000000 00000048 40000000 80000484
		 00000000 ffffffff 00000000 00000048
		 50000000 80000485 00000000 ffffffff
		 00000000 00000048 60000000 80000486
		 00000000 ffffffff 00000000 00000048
		 70000000 80000487 00000000 ffffffff
		 00000000 00000048 80000000 80000488
		 00000000 ffffffff 00000000 00000048
		 90000000 80000489 00000000 ffffffff
		 00000000 00000048 a0000000 8000048a
		 00000000 ffffffff 00000000 00000048
		 b0000000 8000048b 00000000 ffffffff
		 00000000 00000048 c0000000 8000048c
		 00000000 ffffffff 00000000 00000048
		 d0000000 8000048d 00000000 ffffffff
		 00000000 00000048 e0000000 8000048e
		 00000000 ffffffff 00000000 00000048
		 f0000000 8000048f 00000000 ffffffff
		 00000000 00000049 00000000 80000490
		 00000000 ffffffff 00000000 00000049
		 10000000 80000491 00000000 ffffffff
		 00000000 00000049 20000000 80000492
		 00000000 ffffffff 00000000 00000049
		 30000000 80000493 00000000 ffffffff
		 00000000 00000049 40000000 80000494
		 00000000 ffffffff 00000000 00000049
		 50000000 80000495 00000000 ffffffff
		 00000000 00000049 60000000 80000496
		 00000000 ffffffff 00000000 00000049
		 70000000 80000497 00000000 ffffffff
		 00000000 00000049 80000000 80000498
		 00000000 ffffffff 00000000 00000049
		 90000000 80000499 00000000 ffffffff
		 00000000 00000049 a0000000 8000049a
		 00000000 ffffffff 00000000 00000049
		 b0000000 8000049b 00000000 ffffffff
		 00000000 00000049 c0000000 8000049c
		 00000000 ffffffff 00000000 00000049
		 d0000000 8000049d 00000000 ffffffff
		 00000000 00000049 e0000000 8000049e
		 00000000 ffffffff 00000000 00000049
		 f0000000 8000049f 00000000 ffffffff
		 00000000 0000004a 00000000 800004a0
		 00000000 ffffffff 00000000 0000004a
		 10000000 800004a1 00000000 ffffffff
		 00000000 0000004a 20000000 800004a2
		 00000000 ffffffff 00000000 0000004a
		 30000000 800004a3 00000000 ffffffff
		 00000000 0000004a 40000000 800004a4
		 00000000 ffffffff 00000000 0000004a
		 50000000 800004a5 00000000 ffffffff
		 00000000 0000004a 60000000 800004a6
		 00000000 ffffffff 00000000 0000004a
		 70000000 800004a7 00000000 ffffffff
		 00000000 0000004a 80000000 800004a8
		 00000000 ffffffff 00000000 0000004a
		 90000000 800004a9 00000000 ffffffff
		 00000000 0000004a a0000000 800004aa
		 00000000 ffffffff 00000000 0000004a
		 b0000000 800004ab 00000000 ffffffff
		 00000000 0000004a c0000000 800004ac
		 00000000 ffffffff 00000000 0000004a
		 d0000000 800004ad 00000000 ffffffff
		 00000000 0000004a e0000000 800004ae
		 00000000 ffffffff 00000000 0000004a
		 f0000000 800004af 00000000 ffffffff
		 00000000 0000004b 00000000 800004b0
		 00000000 ffffffff 00000000 0000004b
		 10000000 800004b1 00000000 ffffffff
		 00000000 0000004b 20000000 800004b2
		 00000000 ffffffff 00000000 0000004b
		 30000000 800004b3 00000000 ffffffff
		 00000000 0000004b 40000000 800004b4
		 00000000 ffffffff 00000000 0000004b
		 50000000 800004b5 00000000 ffffffff
		 00000000 0000004b 60000000 800004b6
		 00000000 ffffffff 00000000 0000004b
		 70000000 800004b7 00000000 ffffffff
		 00000000 0000004b 80000000 800004b8
		 00000000 ffffffff 00000000 0000004b
		 90000000 800004b9 00000000 ffffffff
		 00000000 0000004b a0000000 800004ba
		 00000000 ffffffff 00000000 0000004b
		 b0000000 800004bb 00000000 ffffffff
		 00000000 0000004b c0000000 800004bc
		 00000000 ffffffff 00000000 0000004b
		 d0000000 800004bd 00000000 ffffffff
		 00000000 0000004b e0000000 800004be
		 00000000 ffffffff 00000000 0000004b
		 f0000000 800004bf 00000000 ffffffff
		 00000000 0000004c 00000000 800004c0
		 00000000 ffffffff 00000000 0000004c
		 10000000 800004c1 00000000 ffffffff
		 00000000 0000004c 20000000 800004c2
		 00000000 ffffffff 00000000 0000004c
		 30000000 800004c3 00000000 ffffffff
		 00000000 0000004c 40000000 800004c4
		 00000000 ffffffff 00000000 0000004c
		 50000000 800004c5 00000000 ffffffff
		 00000000 0000004c 60000000 800004c6
		 00000000 ffffffff 00000000 0000004c
		 70000000 800004c7 00000000 ffffffff
		 00000000 0000004c 80000000 800004c8
		 00000000 ffffffff 00000000 0000004c
		 90000000 800004c9 00000000 ffffffff
		 00000000 0000004c a0000000 800004ca
		 00000000 ffffffff 00000000 0000004c
		 b0000000 800004cb 00000000 ffffffff
		 00000000 0000004c c0000000 800004cc
		 00000000 ffffffff 00000000 0000004c
		 d0000000 800004cd 00000000 ffffffff
		 00000000 0000004c e0000000 800004ce
		 00000000 ffffffff 00000000 0000004c
		 f0000000 800004cf 00000000 ffffffff
		 00000000 0000004d 00000000 800004d0
		 00000000 ffffffff 00000000 0000004d
		 10000000 800004d1 00000000 ffffffff
		 00000000 0000004d 20000000 800004d2
		 00000000 ffffffff 00000000 0000004d
		 30000000 800004d3 00000000 ffffffff
		 00000000 0000004d 40000000 800004d4
		 00000000 ffffffff 00000000 0000004d
		 50000000 800004d5 00000000 ffffffff
		 00000000 0000004d 60000000 800004d6
		 00000000 ffffffff 00000000 0000004d
		 70000000 800004d7 00000000 ffffffff
		 00000000 0000004d 80000000 800004d8
		 00000000 ffffffff 00000000 0000004d
		 90000000 800004d9 00000000 ffffffff
		 00000000 0000004d a0000000 800004da
		 00000000 ffffffff 00000000 0000004d
		 b0000000 800004db 00000000 ffffffff
		 00000000 0000004d c0000000 800004dc
		 00000000 ffffffff 00000000 0000004d
		 d0000000 800004dd 00000000 ffffffff
		 00000000 0000004d e0000000 800004de
		 00000000 ffffffff 00000000 0000004d
		 f0000000 800004df 00000000 ffffffff
		 00000000 0000004e 00000000 800004e0
		 00000000 ffffffff 00000000 0000004e
		 10000000 800004e1 00000000 ffffffff
		 00000000 0000004e 20000000 800004e2
		 00000000 ffffffff 00000000 0000004e
		 30000000 800004e3 00000000 ffffffff
		 00000000 0000004e 40000000 800004e4
		 00000000 ffffffff 00000000 0000004e
		 50000000 800004e5 00000000 ffffffff
		 00000000 0000004e 60000000 800004e6
		 00000000 ffffffff 00000000 0000004e
		 70000000 800004e7 00000000 ffffffff
		 00000000 0000004e 80000000 800004e8
		 00000000 ffffffff 00000000 0000004e
		 90000000 800004e9 00000000 ffffffff
		 00000000 0000004e a0000000 800004ea
		 00000000 ffffffff 00000000 0000004e
		 b0000000 800004eb 00000000 ffffffff
		 00000000 0000004e c0000000 800004ec
		 00000000 ffffffff 00000000 0000004e
		 d0000000 800004ed 00000000 ffffffff
		 00000000 0000004e e0000000 800004ee
		 00000000 ffffffff 00000000 0000004e
		 f0000000 800004ef 00000000 ffffffff
		 00000000 0000004f 00000000 800004f0
		 00000000 ffffffff 00000000 0000004f
		 10000000 800004f1 00000000 ffffffff
		 00000000 0000004f 20000000 800004f2
		 00000000 ffffffff 00000000 0000004f
		 30000000 800004f3 00000000 ffffffff
		 00000000 0000004f 40000000 800004f4
		 00000000 ffffffff 00000000 0000004f
		 50000000 800004f5 00000000 ffffffff
		 00000000 0000004f 60000000 800004f6
		 00000000 ffffffff 00000000 0000004f
		 70000000 800004f7 00000000 ffffffff
		 00000000 0000004f 80000000 800004f8
		 00000000 ffffffff 00000000 0000004f
		 90000000 800004f9 00000000 ffffffff
		 00000000 0000004f a0000000 800004fa
		 00000000 ffffffff 00000000 0000004f
		 b0000000 800004fb 00000000 ffffffff
		 00000000 0000004f c0000000 800004fc
		 00000000 ffffffff 00000000 0000004f
		 d0000000 800004fd 00000000 ffffffff
		 00000000 0000004f e0000000 800004fe
		 00000000 ffffffff 00000000 0000004f
		 f0000000 800004ff 00000000 ffffffff
		 00000000 00000050 00000000 80000500
		 00000000 ffffffff 00000000 00000050
		 10000000 80000501 00000000 ffffffff
		 00000000 00000050 20000000 80000502
		 00000000 ffffffff 00000000 00000050
		 30000000 80000503 00000000 ffffffff
		 00000000 00000050 40000000 80000504
		 00000000 ffffffff 00000000 00000050
		 50000000 80000505 00000000 ffffffff
		 00000000 00000050 60000000 80000506
		 00000000 ffffffff 00000000 00000050
		 70000000 80000507 00000000 ffffffff
		 00000000 00000050 80000000 80000508
		 00000000 ffffffff 00000000 00000050
		 90000000 80000509 00000000 ffffffff
		 00000000 00000050 a0000000 8000050a
		 00000000 ffffffff 00000000 00000050
		 b0000000 8000050b 00000000 ffffffff
		 00000000 00000050 c0000000 8000050c
		 00000000 ffffffff 00000000 00000050
		 d0000000 8000050d 00000000 ffffffff
		 00000000 00000050 e0000000 8000050e
		 00000000 ffffffff 00000000 00000050
		 f0000000 8000050f 00000000 ffffffff
		 00000000 00000051 00000000 80000510
		 00000000 ffffffff 00000000 00000051
		 10000000 80000511 00000000 ffffffff
		 00000000 00000051 20000000 80000512
		 00000000 ffffffff 00000000 00000051
		 30000000 80000513 00000000 ffffffff
		 00000000 00000051 40000000 80000514
		 00000000 ffffffff 00000000 00000051
		 50000000 80000515 00000000 ffffffff
		 00000000 00000051 60000000 80000516
		 00000000 ffffffff 00000000 00000051
		 70000000 80000517 00000000 ffffffff
		 00000000 00000051 80000000 80000518
		 00000000 ffffffff 00000000 00000051
		 90000000 80000519 00000000 ffffffff
		 00000000 00000051 a0000000 8000051a
		 00000000 ffffffff 00000000 00000051
		 b0000000 8000051b 00000000 ffffffff
		 00000000 00000051 c0000000 8000051c
		 00000000 ffffffff 00000000 00000051
		 d0000000 8000051d 00000000 ffffffff
		 00000000 00000051 e0000000 8000051e
		 00000000 ffffffff 00000000 00000051
		 f0000000 8000051f 00000000 ffffffff
		 00000000 00000052 00000000 80000520
		 00000000 ffffffff 00000000 00000052
		 10000000 80000521 00000000 ffffffff
		 00000000 00000052 20000000 80000522
		 00000000 ffffffff 00000000 00000052
		 30000000 80000523 00000000 ffffffff
		 00000000 00000052 40000000 80000524
		 00000000 ffffffff 00000000 00000052
		 50000000 80000525 00000000 ffffffff
		 00000000 00000052 60000000 80000526
		 00000000 ffffffff 00000000 00000052
		 70000000 80000527 00000000 ffffffff
		 00000000 00000052 80000000 80000528
		 00000000 ffffffff 00000000 00000052
		 90000000 80000529 00000000 ffffffff
		 00000000 00000052 a0000000 8000052a
		 00000000 ffffffff 00000000 00000052
		 b0000000 8000052b 00000000 ffffffff
		 00000000 00000052 c0000000 8000052c
		 00000000 ffffffff 00000000 00000052
		 d0000000 8000052d 00000000 ffffffff
		 00000000 00000052 e0000000 8000052e
		 00000000 ffffffff 00000000 00000052
		 f0000000 8000052f 00000000 ffffffff
		 00000000 00000053 00000000 80000530
		 00000000 ffffffff 00000000 00000053
		 10000000 80000531 00000000 ffffffff
		 00000000 00000053 20000000 80000532
		 00000000 ffffffff 00000000 00000053
		 30000000 80000533 00000000 ffffffff
		 00000000 00000053 40000000 80000534
		 00000000 ffffffff 00000000 00000053
		 50000000 80000535 00000000 ffffffff
		 00000000 00000053 60000000 80000536
		 00000000 ffffffff 00000000 00000053
		 70000000 80000537 00000000 ffffffff
		 00000000 00000053 80000000 80000538
		 00000000 ffffffff 00000000 00000053
		 90000000 80000539 00000000 ffffffff
		 00000000 00000053 a0000000 8000053a
		 00000000 ffffffff 00000000 00000053
		 b0000000 8000053b 00000000 ffffffff
		 00000000 00000053 c0000000 8000053c
		 00000000 ffffffff 00000000 00000053
		 d0000000 8000053d 00000000 ffffffff
		 00000000 00000053 e0000000 8000053e
		 00000000 ffffffff 00000000 00000053
		 f0000000 8000053f 00000000 ffffffff
		 00000000 00000054 00000000 80000540
		 00000000 ffffffff 00000000 00000054
		 10000000 80000541 00000000 ffffffff
		 00000000 00000054 20000000 80000542
		 00000000 ffffffff 00000000 00000054
		 30000000 80000543 00000000 ffffffff
		 00000000 00000054 40000000 80000544
		 00000000 ffffffff 00000000 00000054
		 50000000 80000545 00000000 ffffffff
		 00000000 00000054 60000000 80000546
		 00000000 ffffffff 00000000 00000054
		 70000000 80000547 00000000 ffffffff
		 00000000 00000054 80000000 80000548
		 00000000 ffffffff 00000000 00000054
		 90000000 80000549 00000000 ffffffff
		 00000000 00000054 a0000000 8000054a
		 00000000 ffffffff 00000000 00000054
		 b0000000 8000054b 00000000 ffffffff
		 00000000 00000054 c0000000 8000054c
		 00000000 ffffffff 00000000 00000054
		 d0000000 8000054d 00000000 ffffffff
		 00000000 00000054 e0000000 8000054e
		 00000000 ffffffff 00000000 00000054
		 f0000000 8000054f 00000000 ffffffff
		 00000000 00000055 00000000 80000550
		 00000000 ffffffff 00000000 00000055
		 10000000 80000551 00000000 ffffffff
		 00000000 00000055 20000000 80000552
		 00000000 ffffffff 00000000 00000055
		 30000000 80000553 00000000 ffffffff
		 00000000 00000055 40000000 80000554
		 00000000 ffffffff 00000000 00000055
		 50000000 80000555 00000000 ffffffff
		 00000000 00000055 60000000 80000556
		 00000000 ffffffff 00000000 00000055
		 70000000 80000557 00000000 ffffffff
		 00000000 00000055 80000000 80000558
		 00000000 ffffffff 00000000 00000055
		 90000000 80000559 00000000 ffffffff
		 00000000 00000055 a0000000 8000055a
		 00000000 ffffffff 00000000 00000055
		 b0000000 8000055b 00000000 ffffffff
		 00000000 00000055 c0000000 8000055c
		 00000000 ffffffff 00000000 00000055
		 d0000000 8000055d 00000000 ffffffff
		 00000000 00000055 e0000000 8000055e
		 00000000 ffffffff 00000000 00000055
		 f0000000 8000055f 00000000 ffffffff
		 00000000 00000056 00000000 80000560
		 00000000 ffffffff 00000000 00000056
		 10000000 80000561 00000000 ffffffff
		 00000000 00000056 20000000 80000562
		 00000000 ffffffff 00000000 00000056
		 30000000 80000563 00000000 ffffffff
		 00000000 00000056 40000000 80000564
		 00000000 ffffffff 00000000 00000056
		 50000000 80000565 00000000 ffffffff
		 00000000 00000056 60000000 80000566
		 00000000 ffffffff 00000000 00000056
		 70000000 80000567 00000000 ffffffff
		 00000000 00000056 80000000 80000568
		 00000000 ffffffff 00000000 00000056
		 90000000 80000569 00000000 ffffffff
		 00000000 00000056 a0000000 8000056a
		 00000000 ffffffff 00000000 00000056
		 b0000000 8000056b 00000000 ffffffff
		 00000000 00000056 c0000000 8000056c
		 00000000 ffffffff 00000000 00000056
		 d0000000 8000056d 00000000 ffffffff
		 00000000 00000056 e0000000 8000056e
		 00000000 ffffffff 00000000 00000056
		 f0000000 8000056f 00000000 ffffffff
		 00000000 00000057 00000000 80000570
		 00000000 ffffffff 00000000 00000057
		 10000000 80000571 00000000 ffffffff
		 00000000 00000057 20000000 80000572
		 00000000 ffffffff 00000000 00000057
		 30000000 80000573 00000000 ffffffff
		 00000000 00000057 40000000 80000574
		 00000000 ffffffff 00000000 00000057
		 50000000 80000575 00000000 ffffffff
		 00000000 00000057 60000000 80000576
		 00000000 ffffffff 00000000 00000057
		 70000000 80000577 00000000 ffffffff
		 00000000 00000057 80000000 80000578
		 00000000 ffffffff 00000000 00000057
		 90000000 80000579 00000000 ffffffff
		 00000000 00000057 a0000000 8000057a
		 00000000 ffffffff 00000000 00000057
		 b0000000 8000057b 00000000 ffffffff
		 00000000 00000057 c0000000 8000057c
		 00000000 ffffffff 00000000 00000057
		 d0000000 8000057d 00000000 ffffffff
		 00000000 00000057 e0000000 8000057e
		 00000000 ffffffff 00000000 00000057
		 f0000000 8000057f 00000000 ffffffff
		 00000000 00000058 00000000 80000580
		 00000000 ffffffff 00000000 00000058
		 10000000 80000581 00000000 ffffffff
		 00000000 00000058 20000000 80000582
		 00000000 ffffffff 00000000 00000058
		 30000000 80000583 00000000 ffffffff
		 00000000 00000058 40000000 80000584
		 00000000 ffffffff 00000000 00000058
		 50000000 80000585 00000000 ffffffff
		 00000000 00000058 60000000 80000586
		 00000000 ffffffff 00000000 00000058
		 70000000 80000587 00000000 ffffffff
		 00000000 00000058 80000000 80000588
		 00000000 ffffffff 00000000 00000058
		 90000000 80000589 00000000 ffffffff
		 00000000 00000058 a0000000 8000058a
		 00000000 ffffffff 00000000 00000058
		 b0000000 8000058b 00000000 ffffffff
		 00000000 00000058 c0000000 8000058c
		 00000000 ffffffff 00000000 00000058
		 d0000000 8000058d 00000000 ffffffff
		 00000000 00000058 e0000000 8000058e
		 00000000 ffffffff 00000000 00000058
		 f0000000 8000058f 00000000 ffffffff
		 00000000 00000059 00000000 80000590
		 00000000 ffffffff 00000000 00000059
		 10000000 80000591 00000000 ffffffff
		 00000000 00000059 20000000 80000592
		 00000000 ffffffff 00000000 00000059
		 30000000 80000593 00000000 ffffffff
		 00000000 00000059 40000000 80000594
		 00000000 ffffffff 00000000 00000059
		 50000000 80000595 00000000 ffffffff
		 00000000 00000059 60000000 80000596
		 00000000 ffffffff 00000000 00000059
		 70000000 80000597 00000000 ffffffff
		 00000000 00000059 80000000 80000598
		 00000000 ffffffff 00000000 00000059
		 90000000 80000599 00000000 ffffffff
		 00000000 00000059 a0000000 8000059a
		 00000000 ffffffff 00000000 00000059
		 b0000000 8000059b 00000000 ffffffff
		 00000000 00000059 c0000000 8000059c
		 00000000 ffffffff 00000000 00000059
		 d0000000 8000059d 00000000 ffffffff
		 00000000 00000059 e0000000 8000059e
		 00000000 ffffffff 00000000 00000059
		 f0000000 8000059f 00000000 ffffffff
		 00000000
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # 
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # cat ~/dmesg.log
[    0.000000] Linux version 4.4.63-2.1.wi91275.sp3.055a_27-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Tue May 2 10:56:31 EDT 2017 (27e2849)
[    0.000000] Node 0 Memory: 0x0-0x510000000
[    0.000000] Node 1 Memory: 0x510000000-0x5c0000000
[    0.000000] Node 6 Memory: 0x5c0000000-0x26b0000000
[    0.000000] Node 7 Memory: 0x26b0000000-0x3c00000000
[    0.000000] numa: Initmem setup node 0 [mem 0x00000000-0x50fffffff]
[    0.000000] numa:   NODE_DATA [mem 0x50fff6100-0x50fffffff]
[    0.000000] numa: Initmem setup node 1 [mem 0x510000000-0x5bfffffff]
[    0.000000] numa:   NODE_DATA [mem 0x5bfff6100-0x5bfffffff]
[    0.000000] numa: Initmem setup node 6 [mem 0x5c0000000-0x26afffffff]
[    0.000000] numa:   NODE_DATA [mem 0x26afff6100-0x26afffffff]
[    0.000000] numa: Initmem setup node 7 [mem 0x26b0000000-0x3bffffffff]
[    0.000000] numa:   NODE_DATA [mem 0x3bffbe2100-0x3bffbebfff]
[    0.000000] Section 15357 and 15359 (node 7) have a circular dependency on usemap and pgdat allocations
[    0.000000] PPC64 nvram contains 15360 bytes
[    0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000
[    0.000000] Memory hole size: 0MB
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x0000003bffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000050fffffff]
[    0.000000]   node   1: [mem 0x0000000510000000-0x00000005bfffffff]
[    0.000000]   node   6: [mem 0x00000005c0000000-0x00000026afffffff]
[    0.000000]   node   7: [mem 0x00000026b0000000-0x0000003bffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000050fffffff]
[    0.000000] On node 0 totalpages: 331776
[    0.000000]   DMA zone: 324 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 331776 pages, LIFO batch:1
[    0.000000] Initmem setup node 1 [mem 0x0000000510000000-0x00000005bfffffff]
[    0.000000] On node 1 totalpages: 45056
[    0.000000]   DMA zone: 44 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 45056 pages, LIFO batch:1
[    0.000000] Initmem setup node 6 [mem 0x00000005c0000000-0x00000026afffffff]
[    0.000000] On node 6 totalpages: 2158592
[    0.000000]   DMA zone: 2108 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 2158592 pages, LIFO batch:1
[    0.000000] Initmem setup node 7 [mem 0x00000026b0000000-0x0000003bffffffff]
[    0.000000] On node 7 totalpages: 1396736
[    0.000000]   DMA zone: 1364 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 1396736 pages, LIFO batch:1
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory #

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Reza Arbab]

On Tue, May 23, 2017 at 05:44:23PM -0500, Michael Bringmann wrote:
>On 05/23/2017 04:49 PM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>> +static void setup_nodes(void)
>>>>> +{
>>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>>> +
>>>>> +    for (i = 0; i < l; i++) {
>>>>> +        if (!node_possible(i)) {
>>>>> +            setup_node_data(i, 0, 0);
>>>>> +            node_set(i, node_possible_map);
>>>>> +        }
>>>>> +    }
>>>>> +}
>>>>
>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>
>>> They may be related, but that commit is not a replacement.  The above patch ensures that
>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>>> recording that the node was initialized.
>>
>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>
>> add_memory
>>  add_memory_resource
>>    hotadd_new_pgdat
>>      arch_alloc_nodedata <-- allocs the pg_data_t
>>      ...
>>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>
>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>
>With or without 3af229f2071f, we would still need to add something, somewhere to add new
>bits to the 'node_possible_map'.  That is not being done.

Without 3af229f2071f, those bits would already BE set in 
node_possible_map. You wouldn't have to do anything.

-- 
Reza Arbab

Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc

[Michael Bringmann]

I will get a log based on the latest 4.12 kernel to show what happens
one way or the other, with this patch removed.

On 05/24/2017 09:36 AM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 05:44:23PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 04:49 PM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>>> +static void setup_nodes(void)
>>>>>> +{
>>>>>> +    int i, l = 32 /* MAX_NUMNODES */;
>>>>>> +
>>>>>> +    for (i = 0; i < l; i++) {
>>>>>> +        if (!node_possible(i)) {
>>>>>> +            setup_node_data(i, 0, 0);
>>>>>> +            node_set(i, node_possible_map);
>>>>>> +        }
>>>>>> +    }
>>>>>> +}
>>>>>
>>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>>
>>>> They may be related, but that commit is not a replacement.  The above patch ensures that
>>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>>> node that was not used at boot.  (See 'setup_node_data' function in 'numa.c'.)  That and
>>>> recording that the node was initialized.
>>>
>>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>>
>>> add_memory
>>>  add_memory_resource
>>>    hotadd_new_pgdat
>>>      arch_alloc_nodedata <-- allocs the pg_data_t
>>>      ...
>>>      free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>>
>>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>>
>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>> bits to the 'node_possible_map'.  That is not being done.
> 
> Without 3af229f2071f, those bits would already BE set in node_possible_map. You wouldn't have to do anything.
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com