[PATCH] Add debugging aid for memory initialisation problems

[PATCH] Add debugging aid for memory initialisation problems

[Mel Gorman]

A number of bug reports have been submitted related to memory initialisation
that would have been easier to debug if the PFN of page addresses were
available. The dmesg output is often insufficient to find that information
so debugging patches need to be sent to the reporting user.

This patch prints out information on the memmap when it is being allocated
and the sizeof(struct page) when loglevel is set high enough. In most
architectures, this output is produced in generic code. x86_64 and ia64 both
setup node_mem_map in an architecture-specific manner requiring arch-specfic
changes. Th memmap information can be used to translate any valid page
address into a PFN. page_to_pfn() cannot be used directly in bad_page()
because there is no guarantee that the address pointer is valid in any way
and the translation can produce garbage.

Information on memmap is not printed out for the SPARSEMEM memory model. This
only applies to FLATMEM and DISCONTIG configurations.

Signed-off-by: Mel Gorman <mel@csn.ul.ie>

diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.19-rc6-mm1-clean/arch/ia64/mm/contig.c linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/arch/ia64/mm/contig.c
--- linux-2.6.19-rc6-mm1-clean/arch/ia64/mm/contig.c	2006-11-27 10:49:10.000000000 +0000
+++ linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/arch/ia64/mm/contig.c	2006-11-27 10:55:12.000000000 +0000
@@ -262,6 +262,9 @@ paging_init (void)
 		 */
 		NODE_DATA(0)->node_mem_map = vmem_map +
 			find_min_pfn_with_active_regions();
+		printk(KERN_DEBUG
+			"Node %d memmap at 0x%p size %lu first pfn 0x%p\n",
+			0, vmem_map, map_size, NODE_DATA(0)->node_mem_map);
 		free_area_init_nodes(max_zone_pfns);
 
 		printk("Virtual mem_map starts at 0x%p\n", mem_map);
diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.19-rc6-mm1-clean/arch/ia64/mm/discontig.c linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/arch/ia64/mm/discontig.c
--- linux-2.6.19-rc6-mm1-clean/arch/ia64/mm/discontig.c	2006-11-27 10:49:10.000000000 +0000
+++ linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/arch/ia64/mm/discontig.c	2006-11-27 10:55:09.000000000 +0000
@@ -708,6 +708,9 @@ void __init paging_init(void)
 
 #ifdef CONFIG_VIRTUAL_MEM_MAP
 		NODE_DATA(node)->node_mem_map = vmem_map + pfn_offset;
+		printk(KERN_DEBUG
+			"Node %d memmap at 0x%p size %lu first pfn 0x%p\n",
+			node, vmem_map, 0, NODE_DATA(node)->node_mem_map);
 #endif
 		if (mem_data[node].max_pfn > max_pfn)
 			max_pfn = mem_data[node].max_pfn;
diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.19-rc6-mm1-clean/arch/x86_64/mm/numa.c linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/arch/x86_64/mm/numa.c
--- linux-2.6.19-rc6-mm1-clean/arch/x86_64/mm/numa.c	2006-11-27 10:49:10.000000000 +0000
+++ linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/arch/x86_64/mm/numa.c	2006-11-27 10:51:36.000000000 +0000
@@ -158,6 +158,9 @@ void __init setup_node_zones(int nodeid)
 				memmapsize, SMP_CACHE_BYTES, 
 				round_down(limit - memmapsize, PAGE_SIZE), 
 				limit);
+	printk(KERN_DEBUG "Node %d memmap at 0x%p size %lu first pfn 0x%p\n",
+			nodeid, NODE_DATA(nodeid)->node_mem_map,
+			memmapsize, NODE_DATA(nodeid)->node_mem_map);
 #endif
 } 
 
diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.19-rc6-mm1-clean/mm/page_alloc.c linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/mm/page_alloc.c
--- linux-2.6.19-rc6-mm1-clean/mm/page_alloc.c	2006-11-27 10:49:13.000000000 +0000
+++ linux-2.6.19-rc6-mm1-debug_bootmem_init_issues/mm/page_alloc.c	2006-11-27 10:51:36.000000000 +0000
@@ -2808,6 +2808,9 @@ static void __init alloc_node_mem_map(st
 		if (!map)
 			map = alloc_bootmem_node(pgdat, size);
 		pgdat->node_mem_map = map + (pgdat->node_start_pfn - start);
+		printk(KERN_DEBUG
+			"Node %d memmap at 0x%p size %lu first pfn 0x%p\n",
+			pgdat->node_id, map, size, pgdat->node_mem_map);
 	}
 #ifdef CONFIG_FLATMEM
 	/*
@@ -3038,6 +3041,9 @@ void __init free_area_init_nodes(unsigne
 	/* Regions in the early_node_map can be in any order */
 	sort_node_map();
 
+	/* Print out the page size for debugging meminit problems */
+	printk(KERN_DEBUG "sizeof(struct page) = %d\n", sizeof(struct page));
+
 	/* Print out the zone ranges */
 	printk("Zone PFN ranges:\n");
 	for (i = 0; i < MAX_NR_ZONES; i++)

Re: [PATCH] Add debugging aid for memory initialisation problems

[Andi Kleen]

On Monday 27 November 2006 15:08, Mel Gorman wrote:
> A number of bug reports have been submitted related to memory initialisation
> that would have been easier to debug if the PFN of page addresses were
> available. The dmesg output is often insufficient to find that information
> so debugging patches need to be sent to the reporting user.

So how many new lines does that add overall? Your memmap patches overall
were already one of the most noisy additions we had for a very long time.

-Andi

Re: [PATCH] Add debugging aid for memory initialisation problems

[Mel Gorman]

On Mon, 27 Nov 2006, Andi Kleen wrote:

> On Monday 27 November 2006 15:08, Mel Gorman wrote:
>> A number of bug reports have been submitted related to memory initialisation
>> that would have been easier to debug if the PFN of page addresses were
>> available. The dmesg output is often insufficient to find that information
>> so debugging patches need to be sent to the reporting user.
>
> So how many new lines does that add overall?

I don't know for sure, but it's probably around the 150 LOC mark from 
about 12 patches, mainly in page_alloc.c. A significant portion of the 
patches were documentation, comments and providing debugging information. 
As it is, any bug fix or improvement in that code will now apply to all 
architectures using the API, not just one.

I expect the net code difference to drop as some of the mem= parsing 
(which currently does not always work AFAIK) gets handled in generic 
rather than arch-specific code. These type of cleanups will take a while. 
I would also expect the net difference to drop if more architectures used 
the API instead of arch-specific code but I couldn't test on all arches.

> Your memmap patches overall
> were already one of the most noisy additions we had for a very long time.
>

The worst of the noise should be over now. The last patch I sent isn't 
necessary. I only sent it on because it would have cut down on the time 
needed to fix Andre's issue.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: [PATCH] Add debugging aid for memory initialisation problems

[Andi Kleen]

> > So how many new lines does that add overall?
> 
> I don't know for sure, but it's probably around the 150 LOC mark from 

Sorry I meant just dmesg foot print. I think before you add anything
new you first need a construct like apic_printk() with a configurable log level
with a low default.

-Andi

Re: [PATCH] Add debugging aid for memory initialisation problems

[Mel Gorman]

On (27/11/06 16:08), Andi Kleen didst pronounce:
> 
> > > So how many new lines does that add overall?
> > 
> > I don't know for sure, but it's probably around the 150 LOC mark from 
> 
> Sorry I meant just dmesg foot print. I think before you add anything
> new you first need a construct like apic_printk() with a configurable log level
> with a low default.
> 

Oh right, my bad.

At normal logging levels, the largest output is from stuff like this

Zone PFN ranges:
  DMA          1024 ->   262144
  Normal     262144 ->   262144
early_node_map[3] active PFN ranges
  0:     1024 ->    30719
  0:    32768 ->    65413
  0:    65440 ->    65505

So, that is one line per zone, one line per map entry (that can get
large). There is also lines outputting pages used for memmap, that's
more lines per zone and the DMA zone reserve.

Ok.... it's adding up.

With loglevel at KERN_DEBUG, there can be a very large number of lines
like

Entering add_active_range(0, 1024, 30719) 0 entries of 256 used

I see your point. I'll look into doing something like apic_printk().

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab