[BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[Lee Schermerhorn]

I'm seeing this on an ia64 platform--HP rx8640--running the numactl
package regression test.  On ia64 a "NaT Consumption" [NaT = "not a
thing"] usually means a bogus pointer.  I verified that it also occurs
on 2.6.30-rc3-mmotm-090424-1814.  The regression test runs to completion
on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.

The bug occurs right after the test suite issues the message:

"testing numactl --interleave=all memhog 15728640"

-------------------------------
Console log:

numactl[7821]: NaT consumption 2216203124768 [2]
Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]

Pid: 7821, CPU 25, comm:              numactl
psr : 0000121008022038 ifs : 8000000000000004 ip  : [<a00000010014ec91>]    Not tainted (2.6.30-rc3-mmotm-090428-1631)
ip is at next_zones_zonelist+0x31/0x120
unat: 0000000000000000 pfs : 000000000000038b rsc : 0000000000000003
rnat: 0000001008022038 bsps: e0000702c094e000 pr  : 0000000000659a65
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c0270033f
csd : 0000000000000000 ssd : 0000000000000000
b0  : a00000010018dd30 b6  : a0000001000414c0 b7  : a000000100474660
f6  : 1003e3d64d824615cef9c f7  : 1003e9e3779b97f4a7c16
f8  : 1003e0a00000010071e2a f9  : 1003e0000000000000001
f10 : 1003e0000000000000003 f11 : 1003e8208208208208209
r1  : a000000100c647c0 r2  : 0000000000001f90 r3  : 0000000000000000
r8  : 0000000000001f88 r9  : 0000000000000000 r10 : 0000000000000000
r11 : 0000000000000000 r12 : e0000803859eaad0 r13 : e0000803859e8000
r14 : 0000000000000000 r15 : ffffffffffffff80 r16 : e000078000113c38
r17 : 0000000000000000 r18 : 0000000000000006 r19 : e000078000113c08
r20 : 0000000000000000 r21 : e000078000113c00 r22 : 0000000000000000
r23 : ffffffffffff04d8 r24 : e0000803859e8cb8 r25 : 000000000000033c
r26 : e0000803859eab60 r27 : e0000803859eab54 r28 : e0000803859eab58
r29 : 000000007fffffff r30 : 000000000000033c r31 : 000000000000033c

Call Trace:
 [<a000000100015660>] show_stack+0x40/0xa0
                                sp=e0000803859ec690 bsp=e0000803859ec670
 [<a000000100015f90>] show_regs+0x870/0x8c0
                                sp=e0000803859ec860 bsp=e0000803859ec618
 [<a0000001000398b0>] die+0x1b0/0x2c0
                                sp=e0000803859ec860 bsp=e0000803859ec5c8
 [<a000000100039a10>] die_if_kernel+0x50/0x80
                                sp=e0000803859ec860 bsp=e0000803859ec598


This is all I see.  Apparently system locked up. 

------------------

Here's the memory init debug info:

Zone PFN ranges:
  DMA      0x00000001 -> 0x00040000
  Normal   0x00040000 -> 0x220ff000
Movable zone start PFN for each node
early_node_map[6] active PFN ranges
    4: 0x00000001 -> 0x00007ffa
    0: 0x1c008000 -> 0x1c0fec00
    1: 0x1e000000 -> 0x1e0ff000
    2: 0x20000000 -> 0x200ff000
    3: 0x22000000 -> 0x220fef67
    3: 0x220fefa0 -> 0x220feff6
mminit::pageflags_layout_widths Section 0 Node 6 Zone 2 Flags 23
mminit::pageflags_layout_shifts Section 20 Node 6 Zone 2
mminit::pageflags_layout_offsets Section 0 Node 58 Zone 56
mminit::pageflags_layout_zoneid Zone ID: 56 -> 64
mminit::pageflags_layout_usage location: 64 -> 56 unused 56 -> 23 flags 23 -> 0
On node 0 totalpages: 1010688
  Normal zone: 3948 pages used for memmap
  Normal zone: 1006740 pages, LIFO batch:7
mminit::memmap_init Initialising map node 0 zone 1 pfns 469794816 -> 470805504
On node 1 totalpages: 1044480
  Normal zone: 4080 pages used for memmap
  Normal zone: 1040400 pages, LIFO batch:7
mminit::memmap_init Initialising map node 1 zone 1 pfns 503316480 -> 504360960
On node 2 totalpages: 1044480
  Normal zone: 4080 pages used for memmap
  Normal zone: 1040400 pages, LIFO batch:7
mminit::memmap_init Initialising map node 2 zone 1 pfns 536870912 -> 537915392
On node 3 totalpages: 1044413
  Normal zone: 4080 pages used for memmap
  Normal zone: 1040333 pages, LIFO batch:7
mminit::memmap_init Initialising map node 3 zone 1 pfns 570425344 -> 571469814
On node 4 totalpages: 32761
  DMA zone: 128 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 32633 pages, LIFO batch:7
mminit::memmap_init Initialising map node 4 zone 0 pfns 1 -> 32762
mminit::zonelist general 0:Normal = 0:Normal 1:Normal 2:Normal 3:Normal 4:DMA
mminit::zonelist thisnode 0:Normal = 0:Normal
mminit::zonelist general 1:Normal = 1:Normal 2:Normal 3:Normal 0:Normal 4:DMA
mminit::zonelist thisnode 1:Normal = 1:Normal
mminit::zonelist general 2:Normal = 2:Normal 3:Normal 0:Normal 1:Normal 4:DMA
mminit::zonelist thisnode 2:Normal = 2:Normal
mminit::zonelist general 3:Normal = 3:Normal 0:Normal 1:Normal 2:Normal 4:DMA
mminit::zonelist thisnode 3:Normal = 3:Normal
mminit::zonelist general 4:DMA = 4:DMA
mminit::zonelist thisnode 4:DMA = 4:DMA
Built 5 zonelists in Zone order, mobility grouping on.  Total pages: 4160506

Note that this platform has a small [~512MB] pseudo-node #4 that
contains DMA only.  Here's the 'numactl --hardware' output:

available: 5 nodes (0-4)
node 0 size: 15792 MB
node 0 free: 14908 MB
node 1 size: 16320 MB
node 1 free: 15985 MB
node 2 size: 16320 MB
node 2 free: 16106 MB
node 3 size: 16318 MB
node 3 free: 16146 MB
node 4 size: 511 MB
node 4 free: 495 MB
node distances:
node   0   1   2   3   4 
  0:  10  17  17  17  14 
  1:  17  10  17  17  14 
  2:  17  17  10  17  14 
  3:  17  17  17  10  14 
  4:  14  14  14  14  10 

If I create a cpuset with "mems" 0-3 -- i.e., eliminate the dma-only
node 4 -- I do not hit the this "Nat Consumption" bug.  The x86_64 test
platform doesn't have this "feature".

I suspect that the page alloc optimizations are making assumptions that
aren't true for this platform.   I know we had to muck around quite a
bit to get this all to work in the "memoryless nodes" and "two zonelist"
patches a while back. 

I'll try to bisect to specific patch--probably tomorrow.

Regards,
Lee

Re: [BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[Mel Gorman]

On Wed, Apr 29, 2009 at 04:34:59PM -0400, Lee Schermerhorn wrote:
> I'm seeing this on an ia64 platform--HP rx8640--running the numactl
> package regression test.  On ia64 a "NaT Consumption" [NaT = "not a
> thing"] usually means a bogus pointer.  I verified that it also occurs
> on 2.6.30-rc3-mmotm-090424-1814.  The regression test runs to completion
> on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.
> 
> The bug occurs right after the test suite issues the message:
> 
> "testing numactl --interleave=all memhog 15728640"
> 
> -------------------------------
> Console log:
> 
> numactl[7821]: NaT consumption 2216203124768 [2]
> Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
> 
> Pid: 7821, CPU 25, comm:              numactl
> psr : 0000121008022038 ifs : 8000000000000004 ip  : [<a00000010014ec91>]    Not tainted (2.6.30-rc3-mmotm-090428-1631)
> ip is at next_zones_zonelist+0x31/0x120

What line is this?

> unat: 0000000000000000 pfs : 000000000000038b rsc : 0000000000000003
> rnat: 0000001008022038 bsps: e0000702c094e000 pr  : 0000000000659a65
> ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c0270033f
> csd : 0000000000000000 ssd : 0000000000000000
> b0  : a00000010018dd30 b6  : a0000001000414c0 b7  : a000000100474660
> f6  : 1003e3d64d824615cef9c f7  : 1003e9e3779b97f4a7c16
> f8  : 1003e0a00000010071e2a f9  : 1003e0000000000000001
> f10 : 1003e0000000000000003 f11 : 1003e8208208208208209
> r1  : a000000100c647c0 r2  : 0000000000001f90 r3  : 0000000000000000
> r8  : 0000000000001f88 r9  : 0000000000000000 r10 : 0000000000000000
> r11 : 0000000000000000 r12 : e0000803859eaad0 r13 : e0000803859e8000
> r14 : 0000000000000000 r15 : ffffffffffffff80 r16 : e000078000113c38
> r17 : 0000000000000000 r18 : 0000000000000006 r19 : e000078000113c08
> r20 : 0000000000000000 r21 : e000078000113c00 r22 : 0000000000000000
> r23 : ffffffffffff04d8 r24 : e0000803859e8cb8 r25 : 000000000000033c
> r26 : e0000803859eab60 r27 : e0000803859eab54 r28 : e0000803859eab58
> r29 : 000000007fffffff r30 : 000000000000033c r31 : 000000000000033c
> 
> Call Trace:
>  [<a000000100015660>] show_stack+0x40/0xa0
>                                 sp=e0000803859ec690 bsp=e0000803859ec670
>  [<a000000100015f90>] show_regs+0x870/0x8c0
>                                 sp=e0000803859ec860 bsp=e0000803859ec618
>  [<a0000001000398b0>] die+0x1b0/0x2c0
>                                 sp=e0000803859ec860 bsp=e0000803859ec5c8
>  [<a000000100039a10>] die_if_kernel+0x50/0x80
>                                 sp=e0000803859ec860 bsp=e0000803859ec598
> 
> 
> This is all I see.  Apparently system locked up. 
> 
> ------------------
> 
> Here's the memory init debug info:
> 
> Zone PFN ranges:
>   DMA      0x00000001 -> 0x00040000
>   Normal   0x00040000 -> 0x220ff000
> Movable zone start PFN for each node
> early_node_map[6] active PFN ranges
>     4: 0x00000001 -> 0x00007ffa
>     0: 0x1c008000 -> 0x1c0fec00
>     1: 0x1e000000 -> 0x1e0ff000
>     2: 0x20000000 -> 0x200ff000
>     3: 0x22000000 -> 0x220fef67
>     3: 0x220fefa0 -> 0x220feff6
> mminit::pageflags_layout_widths Section 0 Node 6 Zone 2 Flags 23
> mminit::pageflags_layout_shifts Section 20 Node 6 Zone 2
> mminit::pageflags_layout_offsets Section 0 Node 58 Zone 56
> mminit::pageflags_layout_zoneid Zone ID: 56 -> 64
> mminit::pageflags_layout_usage location: 64 -> 56 unused 56 -> 23 flags 23 -> 0
> On node 0 totalpages: 1010688
>   Normal zone: 3948 pages used for memmap
>   Normal zone: 1006740 pages, LIFO batch:7
> mminit::memmap_init Initialising map node 0 zone 1 pfns 469794816 -> 470805504
> On node 1 totalpages: 1044480
>   Normal zone: 4080 pages used for memmap
>   Normal zone: 1040400 pages, LIFO batch:7
> mminit::memmap_init Initialising map node 1 zone 1 pfns 503316480 -> 504360960
> On node 2 totalpages: 1044480
>   Normal zone: 4080 pages used for memmap
>   Normal zone: 1040400 pages, LIFO batch:7
> mminit::memmap_init Initialising map node 2 zone 1 pfns 536870912 -> 537915392
> On node 3 totalpages: 1044413
>   Normal zone: 4080 pages used for memmap
>   Normal zone: 1040333 pages, LIFO batch:7
> mminit::memmap_init Initialising map node 3 zone 1 pfns 570425344 -> 571469814
> On node 4 totalpages: 32761
>   DMA zone: 128 pages used for memmap
>   DMA zone: 0 pages reserved
>   DMA zone: 32633 pages, LIFO batch:7
> mminit::memmap_init Initialising map node 4 zone 0 pfns 1 -> 32762
> mminit::zonelist general 0:Normal = 0:Normal 1:Normal 2:Normal 3:Normal 4:DMA
> mminit::zonelist thisnode 0:Normal = 0:Normal
> mminit::zonelist general 1:Normal = 1:Normal 2:Normal 3:Normal 0:Normal 4:DMA
> mminit::zonelist thisnode 1:Normal = 1:Normal
> mminit::zonelist general 2:Normal = 2:Normal 3:Normal 0:Normal 1:Normal 4:DMA
> mminit::zonelist thisnode 2:Normal = 2:Normal
> mminit::zonelist general 3:Normal = 3:Normal 0:Normal 1:Normal 2:Normal 4:DMA
> mminit::zonelist thisnode 3:Normal = 3:Normal
> mminit::zonelist general 4:DMA = 4:DMA
> mminit::zonelist thisnode 4:DMA = 4:DMA
> Built 5 zonelists in Zone order, mobility grouping on.  Total pages: 4160506
> 
> Note that this platform has a small [~512MB] pseudo-node #4 that
> contains DMA only.  Here's the 'numactl --hardware' output:
> 

What is a pseudo-node?

> available: 5 nodes (0-4)
> node 0 size: 15792 MB
> node 0 free: 14908 MB
> node 1 size: 16320 MB
> node 1 free: 15985 MB
> node 2 size: 16320 MB
> node 2 free: 16106 MB
> node 3 size: 16318 MB
> node 3 free: 16146 MB
> node 4 size: 511 MB
> node 4 free: 495 MB
> node distances:
> node   0   1   2   3   4 
>   0:  10  17  17  17  14 
>   1:  17  10  17  17  14 
>   2:  17  17  10  17  14 
>   3:  17  17  17  10  14 
>   4:  14  14  14  14  10 
> 
> If I create a cpuset with "mems" 0-3 -- i.e., eliminate the dma-only
> node 4 -- I do not hit the this "Nat Consumption" bug.  The x86_64 test
> platform doesn't have this "feature".
> 
> I suspect that the page alloc optimizations are making assumptions that
> aren't true for this platform. 

Based on the timing of the bug, the most likely explanation
is that there is a problem in there.  I went through the
zonelist-walker changes but didn't spot anything. Could you try reverting
page-allocator-do-not-check-numa-node-id-when-the-caller-knows-the-node-is-valid
please? It has a few changes with repect to NUMA and ia-64 and the error
might be in there somewhere.

Is it only the interleave policy that is affected or are other NUMA
placement policies with node 4 causing trouble as well? If it's only
interleave, are you aware of any recent changes to the interleave policy
in -mm that might also explain this problem?

> I know we had to muck around quite a
> bit to get this all to work in the "memoryless nodes" and "two zonelist"
> patches a while back. 
> 
> I'll try to bisect to specific patch--probably tomorrow.
> 

Can you also try with this minimal debugging patch applied and the full
console log please? I'll keep thinking on it and hopefully I'll get inspired

diff --git a/mm/mm_init.c b/mm/mm_init.c
index 4e0e265..82e17bb 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -41,8 +41,6 @@ void mminit_verify_zonelist(void)
 			listid = i / MAX_NR_ZONES;
 			zonelist = &pgdat->node_zonelists[listid];
 			zone = &pgdat->node_zones[zoneid];
-			if (!populated_zone(zone))
-				continue;
 
 			/* Print information about the zonelist */
 			printk(KERN_DEBUG "mminit::zonelist %s %d:%s = ",
diff --git a/mm/mmzone.c b/mm/mmzone.c
index 16ce8b9..c8c54d1 100644
--- a/mm/mmzone.c
+++ b/mm/mmzone.c
@@ -57,6 +57,10 @@ struct zoneref *next_zones_zonelist(struct zoneref *z,
 					nodemask_t *nodes,
 					struct zone **zone)
 {
+	/* Should be impossible, check for NULL or near-NULL values for z */
+	BUG_ON(!z);
+	BUG_ON((unsigned long )z < PAGE_SIZE);
+
 	/*
 	 * Find the next suitable zone to use for the allocation.
 	 * Only filter based on nodemask if it's set

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[Lee Schermerhorn]

On Thu, 2009-04-30 at 12:31 +0100, Mel Gorman wrote: 
> On Wed, Apr 29, 2009 at 04:34:59PM -0400, Lee Schermerhorn wrote:
> > I'm seeing this on an ia64 platform--HP rx8640--running the numactl
> > package regression test.  On ia64 a "NaT Consumption" [NaT = "not a
> > thing"] usually means a bogus pointer.  I verified that it also occurs
> > on 2.6.30-rc3-mmotm-090424-1814.  The regression test runs to completion
> > on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.
> > 
> > The bug occurs right after the test suite issues the message:
> > 
> > "testing numactl --interleave=all memhog 15728640"
> > 
> > -------------------------------
> > Console log:
> > 
> > numactl[7821]: NaT consumption 2216203124768 [2]
> > Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
> > 
> > Pid: 7821, CPU 25, comm:              numactl
> > psr : 0000121008022038 ifs : 8000000000000004 ip  : [<a00000010014ec91>]    Not tainted (2.6.30-rc3-mmotm-090428-1631)
> > ip is at next_zones_zonelist+0x31/0x120
> 
> What line is this?

Hi, Mel:

Sorry for the delay.  Swamped.  Was building incrementally patched
kernels and took a while to get back to where I could [sort of] answer
this.  Below I've included part of the disassembly of the mmzone.o for
this kernel.


<snip> 
> > mminit::zonelist general 4:DMA = 4:DMA
> > mminit::zonelist thisnode 4:DMA = 4:DMA
> > Built 5 zonelists in Zone order, mobility grouping on.  Total pages: 4160506
> > 
> > Note that this platform has a small [~512MB] pseudo-node #4 that
> > contains DMA only.  Here's the 'numactl --hardware' output:
> > 
> 
> What is a pseudo-node?

It's an artifact of the firmware and platform architecture.  It's a
memory-only node at physical address zero that contains memory that is
hardware-interleaved across a small slice of the real, physical nodes'
memory.  It shows up in the ACPI SRAT/SLIT tables as a separate
'PXM' [proximity domain] that Linux treats as a "node".  Because it's
<4G [on my test platform], it's all ia64 dma zone.

> 
> > available: 5 nodes (0-4)
> > node 0 size: 15792 MB
> > node 0 free: 14908 MB
> > node 1 size: 16320 MB
> > node 1 free: 15985 MB
> > node 2 size: 16320 MB
> > node 2 free: 16106 MB
> > node 3 size: 16318 MB
> > node 3 free: 16146 MB
> > node 4 size: 511 MB
> > node 4 free: 495 MB
> > node distances:
> > node   0   1   2   3   4 
> >   0:  10  17  17  17  14 
> >   1:  17  10  17  17  14 
> >   2:  17  17  10  17  14 
> >   3:  17  17  17  10  14 
> >   4:  14  14  14  14  10 
> > 
> > If I create a cpuset with "mems" 0-3 -- i.e., eliminate the dma-only
> > node 4 -- I do not hit the this "Nat Consumption" bug.  The x86_64 test
> > platform doesn't have this "feature".
> > 
> > I suspect that the page alloc optimizations are making assumptions that
> > aren't true for this platform. 
> 
> Based on the timing of the bug, the most likely explanation
> is that there is a problem in there.  I went through the
> zonelist-walker changes but didn't spot anything. Could you try reverting
> page-allocator-do-not-check-numa-node-id-when-the-caller-knows-the-node-is-valid
> please? It has a few changes with repect to NUMA and ia-64 and the error
> might be in there somewhere.

Yeah, I've built some kernels to test.  Best to build them all at once,
since it takes a while to reboot this beast.  I'll let you know.

> 
> Is it only the interleave policy that is affected or are other NUMA
> placement policies with node 4 causing trouble as well? If it's only
> interleave, are you aware of any recent changes to the interleave policy
> in -mm that might also explain this problem?

I've tried to find a simple reproducer using memtoy, but haven't found
one yet.  The numactl package regression test hit is every time.  I
tried running a 'membind' test, and it doesn't seem to occur, altho I do
hit oom:

numactl --membind=4 ./memhog $(scale 16G)

But, when I try interleave, it hits the bug:

numactl --interleave=all ./memhog $(scale 16G)

That hits the bug.


> 
> > I know we had to muck around quite a
> > bit to get this all to work in the "memoryless nodes" and "two zonelist"
> > patches a while back. 
> > 
> > I'll try to bisect to specific patch--probably tomorrow.
> > 
> 
> Can you also try with this minimal debugging patch applied and the full
> console log please? I'll keep thinking on it and hopefully I'll get inspired

Will do.  I'll send the results.

Here's a section of the disassembly of mmzone.o.  Looks like the fault
is in the inlined zonelist_zone_idx() -- 0x1a0 + 0x31 ~= 0x1d0/1d6.
Dereferencing the zoneref pointer.


mm/mmzone.o:     file format elf64-ia64-little

Disassembly of section .text:

0000000000000000 <next_online_pgdat>:
	return NODE_DATA(first_online_node);
}

<snip>


00000000000001a0 <next_zones_zonelist>:

static inline int zref_in_nodemask(struct zoneref *zref, nodemask_t *nodes)
{
#ifdef CONFIG_NUMA
	return node_isset(zonelist_node_idx(zref), *nodes);
#else
	return 1;
#endif /* CONFIG_NUMA */
}

/* Returns the next zone at or below highest_zoneidx in a zonelist */
struct zoneref *next_zones_zonelist(struct zoneref *z,
					enum zone_type highest_zoneidx,
					nodemask_t *nodes,
					struct zone **zone)
{
 1a0:	10 40 00 40 00 21 	[MIB]       mov r8=r32
	/*
	 * Find the next suitable zone to use for the allocation.
	 * Only filter based on nodemask if it's set
	 */
	if (likely(nodes == NULL))
 1a6:	60 00 88 0e 72 03 	            cmp.eq p6,p7=0,r34
 1ac:	30 00 00 40       	      (p06) br.cond.sptk.few 1d0 <next_zones_zonelist+0x30>
 1b0:	11 00 00 00 01 00 	[MIB]       nop.m 0x0
 1b6:	00 00 00 02 00 00 	            nop.i 0x0
 1bc:	60 00 00 40       	            br.few 210 <next_zones_zonelist+0x70>;;
		while (zonelist_zone_idx(z) > highest_zoneidx)
			z++;
 1c0:	09 40 40 10 00 21 	[MMI]       adds r8=16,r8
 1c6:	00 00 00 02 00 00 	            nop.m 0x0
 1cc:	00 00 04 00       	            nop.i 0x0;;
}

static inline int zonelist_zone_idx(struct zoneref *zoneref)
{
	return zoneref->zone_idx;
 1d0:	0b 10 20 10 00 21 	[MMI]       adds r2=8,r8;;
 1d6:	e0 00 08 20 20 00 	            ld4 r14=[r2]	<<<< ???
 1dc:	00 00 04 00       	            nop.i 0x0;;
 1e0:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 1e6:	80 08 39 12 69 04 	            cmp4.ltu p8,p9=r33,r14
 1ec:	e0 ff ff 4a       	      (p08) br.cond.dptk.few 1c0 <next_zones_zonelist+0x20>
 1f0:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 1f6:	00 00 00 02 00 00 	            nop.i 0x0
 1fc:	b0 00 00 40       	            br.few 2a0 <next_zones_zonelist+0x100>
	else
		while (zonelist_zone_idx(z) > highest_zoneidx ||
				(z->zone && !zref_in_nodemask(z, nodes)))
			z++;
 200:	09 40 40 10 00 21 	[MMI]       adds r8=16,r8
 206:	00 00 00 02 00 00 	            nop.m 0x0
 20c:	00 00 04 00       	            nop.i 0x0;;
}

static inline int zonelist_zone_idx(struct zoneref *zoneref)
{
	return zoneref->zone_idx;
 210:	0b 48 20 10 00 21 	[MMI]       adds r9=8,r8;;
 216:	30 00 24 20 20 00 	            ld4 r3=[r9]
 21c:	00 00 04 00       	            nop.i 0x0;;
 220:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 226:	a0 08 0d 16 69 05 	            cmp4.ltu p10,p11=r33,r3
 22c:	e0 ff ff 4a       	      (p10) br.cond.dptk.few 200 <next_zones_zonelist+0x60>
 230:	09 00 00 00 01 00 	[MMI]       nop.m 0x0
static inline int zonelist_node_idx(struct zoneref *zoneref)
{
#ifdef CONFIG_NUMA
	/* zone_to_nid not available in this context */
	return zoneref->zone->node;
 236:	a0 00 20 30 20 00 	            ld8 r10=[r8]
 23c:	00 00 04 00       	            nop.i 0x0;;
 240:	11 78 c0 14 00 21 	[MIB]       adds r15=48,r10
 246:	c0 00 28 1a 72 06 	            cmp.eq p12,p13=0,r10
 24c:	60 00 00 43       	      (p12) br.cond.dpnt.few 2a0 <next_zones_zonelist+0x100>;;
static inline int zonelist_node_idx(struct zoneref *zoneref)
{
#ifdef CONFIG_NUMA
	/* zone_to_nid not available in this context */
	return zoneref->zone->node;
 250:	02 a0 00 1e 10 10 	[MII]       ld4 r20=[r15]

static __inline__ int
test_bit (int nr, const volatile void *addr)
{
	return 1 & (((const volatile __u32 *) addr)[nr >> 5] >> (nr & 31));
 256:	00 00 00 02 00 60 	            nop.i 0x0;;
 25c:	b2 a0 68 52       	            extr r19=r20,5,27
 260:	02 00 00 00 01 00 	[MII]       nop.m 0x0

static __inline__ int
test_bit (int nr, const volatile void *addr)
{
	return 1 & (((const volatile __u32 *) addr)[nr >> 5] >> (nr & 31));
 266:	f0 f8 50 58 40 00 	            and r15=31,r20;;
 26c:	00 00 04 00       	            nop.i 0x0
 270:	0b 90 4c 44 11 20 	[MMI]       shladd r18=r19,2,r34;;
 276:	10 01 48 60 21 00 	            ld4.acq r17=[r18]
 27c:	00 00 04 00       	            nop.i 0x0;;
 280:	03 00 00 00 01 00 	[MII]       nop.m 0x0
 286:	00 89 00 10 40 60 	            addp4 r16=r17,r0;;
 28c:	f1 80 00 79       	            shr.u r11=r16,r15;;
 290:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 296:	e0 00 2c 1e 28 07 	            tbit.z p14,p15=r11,0
 29c:	70 ff ff 4a       	      (p14) br.cond.dptk.few 200 <next_zones_zonelist+0x60>
		else

static inline struct zone *zonelist_zone(struct zoneref *zoneref)
{
	return zoneref->zone;
 2a0:	09 00 00 00 01 00 	[MMI]       nop.m 0x0

	*zone = zonelist_zone(z);
 2a6:	50 01 20 30 20 00 	            ld8 r21=[r8]
 2ac:	00 00 04 00       	            nop.i 0x0;;
 2b0:	11 00 54 46 98 11 	[MIB]       st8 [r35]=r21
	return z;
}
 2b6:	00 00 00 02 00 80 	            nop.i 0x0
 2bc:	08 00 84 00       	            br.ret.sptk.many b0;;




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[Lee Schermerhorn]

On Thu, 2009-04-30 at 12:31 +0100, Mel Gorman wrote:
> On Wed, Apr 29, 2009 at 04:34:59PM -0400, Lee Schermerhorn wrote:
> > I'm seeing this on an ia64 platform--HP rx8640--running the numactl
> > package regression test.  On ia64 a "NaT Consumption" [NaT = "not a
> > thing"] usually means a bogus pointer.  I verified that it also occurs
> > on 2.6.30-rc3-mmotm-090424-1814.  The regression test runs to completion
> > on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.
> > 
> > The bug occurs right after the test suite issues the message:
> > 
> > "testing numactl --interleave=all memhog 15728640"
> > 
> > -------------------------------
> > Console log:
> > 
> > numactl[7821]: NaT consumption 2216203124768 [2]
> > Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
> > 
> > Pid: 7821, CPU 25, comm:              numactl
> > psr : 0000121008022038 ifs : 8000000000000004 ip  : [<a00000010014ec91>]    Not tainted (2.6.30-rc3-mmotm-090428-1631)
> > ip is at next_zones_zonelist+0x31/0x120
<snip>
> > 
> > I'll try to bisect to specific patch--probably tomorrow.

Mel:  I think you can rest easy.  I've duplicated the problem with a
kernel that truncates the mmotm 04/28 series just before your patches.
Hope it's not my cpuset-mm fix that occurs just before that!  I'll let
you know.

Did hit one or your BUG_ON's, tho'.  See below.

> > 
> 
> Can you also try with this minimal debugging patch applied and the full
> console log please? I'll keep thinking on it and hopefully I'll get inspired
> 
> diff --git a/mm/mm_init.c b/mm/mm_init.c
> index 4e0e265..82e17bb 100644
> --- a/mm/mm_init.c
> +++ b/mm/mm_init.c
> @@ -41,8 +41,6 @@ void mminit_verify_zonelist(void)
>  			listid = i / MAX_NR_ZONES;
>  			zonelist = &pgdat->node_zonelists[listid];
>  			zone = &pgdat->node_zones[zoneid];
> -			if (!populated_zone(zone))
> -				continue;
>  
>  			/* Print information about the zonelist */
>  			printk(KERN_DEBUG "mminit::zonelist %s %d:%s = ",
> diff --git a/mm/mmzone.c b/mm/mmzone.c
> index 16ce8b9..c8c54d1 100644
> --- a/mm/mmzone.c
> +++ b/mm/mmzone.c
> @@ -57,6 +57,10 @@ struct zoneref *next_zones_zonelist(struct zoneref *z,
>  					nodemask_t *nodes,
>  					struct zone **zone)
>  {
> +	/* Should be impossible, check for NULL or near-NULL values for z */
> +	BUG_ON(!z);
> +	BUG_ON((unsigned long )z < PAGE_SIZE);

The test w/o your patches hit the second BUG_ON().


> +
>  	/*
>  	 * Find the next suitable zone to use for the allocation.
>  	 * Only filter based on nodemask if it's set
> --
> To unsubscribe from this list: send the line "unsubscribe linux-numa" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[Mel Gorman]

On Thu, Apr 30, 2009 at 09:14:49PM -0400, Lee Schermerhorn wrote:
> On Thu, 2009-04-30 at 12:31 +0100, Mel Gorman wrote:
> > On Wed, Apr 29, 2009 at 04:34:59PM -0400, Lee Schermerhorn wrote:
> > > I'm seeing this on an ia64 platform--HP rx8640--running the numactl
> > > package regression test.  On ia64 a "NaT Consumption" [NaT = "not a
> > > thing"] usually means a bogus pointer.  I verified that it also occurs
> > > on 2.6.30-rc3-mmotm-090424-1814.  The regression test runs to completion
> > > on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.
> > > 
> > > The bug occurs right after the test suite issues the message:
> > > 
> > > "testing numactl --interleave=all memhog 15728640"
> > > 
> > > -------------------------------
> > > Console log:
> > > 
> > > numactl[7821]: NaT consumption 2216203124768 [2]
> > > Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
> > > 
> > > Pid: 7821, CPU 25, comm:              numactl
> > > psr : 0000121008022038 ifs : 8000000000000004 ip  : [<a00000010014ec91>]    Not tainted (2.6.30-rc3-mmotm-090428-1631)
> > > ip is at next_zones_zonelist+0x31/0x120
> <snip>
> > > 
> > > I'll try to bisect to specific patch--probably tomorrow.
> 
> Mel:  I think you can rest easy.  I've duplicated the problem with a
> kernel that truncates the mmotm 04/28 series just before your patches.

Ok, I can rest a little easier but I won't that much. I've mucked around
enough in there over the last while that it might still be something I
did.

> Hope it's not my cpuset-mm fix that occurs just before that!  I'll let
> you know.
> 

I don't think so because it was in mmotm before my patchset was and you
didn't spot any problems.

> Did hit one or your BUG_ON's, tho'.  See below.
> 
> > > 
> > 
> > Can you also try with this minimal debugging patch applied and the full
> > console log please? I'll keep thinking on it and hopefully I'll get inspired
> > 
> > diff --git a/mm/mm_init.c b/mm/mm_init.c
> > index 4e0e265..82e17bb 100644
> > --- a/mm/mm_init.c
> > +++ b/mm/mm_init.c
> > @@ -41,8 +41,6 @@ void mminit_verify_zonelist(void)
> >  			listid = i / MAX_NR_ZONES;
> >  			zonelist = &pgdat->node_zonelists[listid];
> >  			zone = &pgdat->node_zones[zoneid];
> > -			if (!populated_zone(zone))
> > -				continue;
> >  
> >  			/* Print information about the zonelist */
> >  			printk(KERN_DEBUG "mminit::zonelist %s %d:%s = ",
> > diff --git a/mm/mmzone.c b/mm/mmzone.c
> > index 16ce8b9..c8c54d1 100644
> > --- a/mm/mmzone.c
> > +++ b/mm/mmzone.c
> > @@ -57,6 +57,10 @@ struct zoneref *next_zones_zonelist(struct zoneref *z,
> >  					nodemask_t *nodes,
> >  					struct zone **zone)
> >  {
> > +	/* Should be impossible, check for NULL or near-NULL values for z */
> > +	BUG_ON(!z);
> > +	BUG_ON((unsigned long )z < PAGE_SIZE);
> 
> The test w/o your patches hit the second BUG_ON().
> 

This implies that z was NULL when it was passed to the iterator

#define for_each_zone_zonelist_nodemask(zone, z, zlist, highidx, nodemask) \
        for (z = first_zones_zonelist(zlist, highidx, nodemask, &zone); \
                zone; \
                z = next_zones_zonelist(++z, highidx, nodemask, &zone)) \

and we ended up with z == ++NULL;

Can you send the full dmesg and what your bisection point was? Maybe I
can spot something. The implication is that a corrupt or badly constructed
zonelist is being passed into the page allocator so I'd like to see where
it is coming from.

Thanks

> 
> > +
> >  	/*
> >  	 * Find the next suitable zone to use for the allocation.
> >  	 * Only filter based on nodemask if it's set
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-numa" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: [BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[Lee Schermerhorn]

[Note:  we're experiencing some e-mail problems at HP today.  I've added
a Reply-to: that should work, but my normal hp address will probably
bounce until this gets resolved.]

On Fri, 2009-05-01 at 10:49 +0100, Mel Gorman wrote:
> On Thu, Apr 30, 2009 at 09:14:49PM -0400, Lee Schermerhorn wrote:
> > On Thu, 2009-04-30 at 12:31 +0100, Mel Gorman wrote:
> > > On Wed, Apr 29, 2009 at 04:34:59PM -0400, Lee Schermerhorn wrote:
> > > > I'm seeing this on an ia64 platform--HP rx8640--running the numactl
> > > > package regression test.  On ia64 a "NaT Consumption" [NaT = "not a
> > > > thing"] usually means a bogus pointer.  I verified that it also occurs
> > > > on 2.6.30-rc3-mmotm-090424-1814.  The regression test runs to completion
> > > > on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.
> > > > 
> > > > The bug occurs right after the test suite issues the message:
> > > > 
> > > > "testing numactl --interleave=all memhog 15728640"
> > > > 
> > > > -------------------------------
> > > > Console log:
> > > > 
> > > > numactl[7821]: NaT consumption 2216203124768 [2]
> > > > Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
> > > > 
> > > > Pid: 7821, CPU 25, comm:              numactl
> > > > psr : 0000121008022038 ifs : 8000000000000004 ip  : [<a00000010014ec91>]    Not tainted (2.6.30-rc3-mmotm-090428-1631)
> > > > ip is at next_zones_zonelist+0x31/0x120
> > <snip>
> > > > 
> > > > I'll try to bisect to specific patch--probably tomorrow.
> > 
> > Mel:  I think you can rest easy.  I've duplicated the problem with a
> > kernel that truncates the mmotm 04/28 series just before your patches.
> 
> Ok, I can rest a little easier but I won't that much. I've mucked around
> enough in there over the last while that it might still be something I
> did.
> 
> > Hope it's not my cpuset-mm fix that occurs just before that!  I'll let
> > you know.
> > 
> 
> I don't think so because it was in mmotm before my patchset was and you
> didn't spot any problems.
> 
> > Did hit one or your BUG_ON's, tho'.  See below.
> > 
> > > > 
> > > 
> > > Can you also try with this minimal debugging patch applied and the full
> > > console log please? I'll keep thinking on it and hopefully I'll get inspired
> > > 
> > > diff --git a/mm/mm_init.c b/mm/mm_init.c
> > > index 4e0e265..82e17bb 100644
> > > --- a/mm/mm_init.c
> > > +++ b/mm/mm_init.c
> > > @@ -41,8 +41,6 @@ void mminit_verify_zonelist(void)
> > >  			listid = i / MAX_NR_ZONES;
> > >  			zonelist = &pgdat->node_zonelists[listid];
> > >  			zone = &pgdat->node_zones[zoneid];
> > > -			if (!populated_zone(zone))
> > > -				continue;
> > >  
> > >  			/* Print information about the zonelist */
> > >  			printk(KERN_DEBUG "mminit::zonelist %s %d:%s = ",
> > > diff --git a/mm/mmzone.c b/mm/mmzone.c
> > > index 16ce8b9..c8c54d1 100644
> > > --- a/mm/mmzone.c
> > > +++ b/mm/mmzone.c
> > > @@ -57,6 +57,10 @@ struct zoneref *next_zones_zonelist(struct zoneref *z,
> > >  					nodemask_t *nodes,
> > >  					struct zone **zone)
> > >  {
> > > +	/* Should be impossible, check for NULL or near-NULL values for z */
> > > +	BUG_ON(!z);
> > > +	BUG_ON((unsigned long )z < PAGE_SIZE);
> > 
> > The test w/o your patches hit the second BUG_ON().
> > 
> 
> This implies that z was NULL when it was passed to the iterator
> 
> #define for_each_zone_zonelist_nodemask(zone, z, zlist, highidx, nodemask) \
>         for (z = first_zones_zonelist(zlist, highidx, nodemask, &zone); \
>                 zone; \
>                 z = next_zones_zonelist(++z, highidx, nodemask, &zone)) \
> 
> and we ended up with z == ++NULL;
> 
> Can you send the full dmesg and what your bisection point was? Maybe I
> can spot something. The implication is that a corrupt or badly constructed
> zonelist is being passed into the page allocator so I'd like to see where
> it is coming from.

Mel:

I've tracked it down to this patch:

	cpusetmm-update-tasks-mems_allowed-in-time.patch

Without that patch, the numademo command:

	numactl --interleave=all ./memhog $(scale 16G)
	
runs to completion w/o error.  With this patch pushed, I capture this
trace today:

numactl[7408]: NaT consumption 17179869216 [1]
Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button tg3 e1000 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]

Pid: 7408, CPU 2, comm:              numactl
psr : 0000101008526030 ifs : 8000000000000897 ip  : [<a0000001001335f0>]    Not tainted (2.6.30-rc3-mmotm-090428-1631+zonelist-debug-7)
ip is at __alloc_pages_internal+0x110/0x720
unat: 0000000000000000 pfs : 0000000000000897 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 00000000001cca31 pr  : 0000000500555a69
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f
csd : 0000000000000000 ssd : 0000000000000000
b0  : a0000001001335d0 b6  : a000000100038fa0 b7  : a00000010000bbc0
f6  : 1003e000000000000003e f7  : 1003e000000003ffffffe
f8  : 100178208141befbf5f00 f9  : 1003effffffffffffffc1
f10 : 1003e00000000001dbeff f11 : 1003e0044b82fa09b5a53
r1  : a000000100c844c0 r2  : e00007038a7a0ce0 r3  : e00007038a7a0cd0
r8  : 0000000000000000 r9  : 0000000000020000 r10 : 0000000000000000
r11 : 0000000000000000 r12 : e00007038a7a7de0 r13 : e00007038a7a0000
r14 : 0000000000000000 r15 : e00007038a7a0cf8 r16 : e000070020103b88
r17 : 0000000000000000 r18 : e000070020103b80 r19 : 0000000000000000
r20 : 0000000000000018 r21 : a000000100a22ec8 r22 : a000000100a22ec0
r23 : a000000100a852f0 r24 : e000070380031808 r25 : e00007038003180c
r26 : 0000000000000000 r27 : 0000000000000000 r28 : 0000000000004000
r29 : 0000000000004000 r30 : 0000000000000000 r31 : e0000783821c9900

Call Trace:
 [<a000000100015720>] show_stack+0x40/0xa0
                                sp=e00007038a7a7830 bsp=e00007038a7a13d0
 [<a000000100016050>] show_regs+0x870/0x8c0
                                sp=e00007038a7a7a00 bsp=e00007038a7a1378
 [<a0000001000399d0>] die+0x1b0/0x2c0
                                sp=e00007038a7a7a00 bsp=e00007038a7a1330
 [<a000000100039b30>] die_if_kernel+0x50/0x80
                                sp=e00007038a7a7a00 bsp=e00007038a7a1300
 [<a000000100733980>] ia64_fault+0x1140/0x1260
                                sp=e00007038a7a7a00 bsp=e00007038a7a12a8
 [<a00000010000c3c0>] ia64_native_leave_kernel+0x0/0x270
                                sp=e00007038a7a7c10 bsp=e00007038a7a12a8
 [<a0000001001335f0>] __alloc_pages_internal+0x110/0x720
                                sp=e00007038a7a7de0 bsp=e00007038a7a11e8
 [<a000000100181b60>] alloc_page_interleave+0xa0/0x160
                                sp=e00007038a7a7df0 bsp=e00007038a7a11a8
 [<a000000100182060>] alloc_page_vma+0x120/0x220
                                sp=e00007038a7a7df0 bsp=e00007038a7a1170
 [<a000000100156a50>] handle_mm_fault+0x330/0xf60
                                sp=e00007038a7a7df0 bsp=e00007038a7a1100
 [<a000000100157b80>] __get_user_pages+0x500/0x820
                                sp=e00007038a7a7e00 bsp=e00007038a7a1090
 [<a000000100157f00>] get_user_pages+0x60/0x80
                                sp=e00007038a7a7e10 bsp=e00007038a7a1038
 [<a0000001001ad3d0>] get_arg_page+0x50/0x160
                                sp=e00007038a7a7e10 bsp=e00007038a7a1008
 [<a0000001001ad940>] copy_strings+0x200/0x3a0
                                sp=e00007038a7a7e20 bsp=e00007038a7a0f78
 [<a0000001001adb20>] copy_strings_kernel+0x40/0x60
                                sp=e00007038a7a7e20 bsp=e00007038a7a0f40
 [<a0000001001b0b80>] do_execve+0x320/0x5e0
                                sp=e00007038a7a7e20 bsp=e00007038a7a0ee0
 [<a000000100014780>] sys_execve+0x60/0xa0
                                sp=e00007038a7a7e30 bsp=e00007038a7a0ea8
 [<a00000010000b910>] ia64_execve+0x30/0x140
                                sp=e00007038a7a7e30 bsp=e00007038a7a0e58
 [<a00000010000c240>] ia64_ret_from_syscall+0x0/0x20
                                sp=e00007038a7a7e30 bsp=e00007038a7a0e58
 [<a000000000010720>] _stext+0xffffffff00010720/0x400
                                sp=e00007038a7a8000 bsp=e00007038a7a0e58
---------------------------------------------------------

So, I'm off trying to figure what's happening with interleaved
allocations with this patch.

Later,
Lee