On Mon, Dec 10, 2012 at 09:59:29AM +0900, HATAYAMA Daisuke wrote:
> From: Cliff Wickman <cpw@sgi.com>
> Subject: Re: [PATCH] makedumpfile: request the kernel do page scans
> Date: Mon, 19 Nov 2012 12:07:10 -0600
> 
> > On Fri, Nov 16, 2012 at 03:39:44PM -0500, Vivek Goyal wrote:
> >> On Thu, Nov 15, 2012 at 04:52:40PM -0600, Cliff Wickman wrote:
> >> > 
> >> > Gentlemen,
> >> > 
> >> > I know this is rather late to the game, given all the recent work to speed
> >> > up makedumpfile and reduce the memory that it consumes.
> >> > But I've been experimenting with asking the kernel to scan the page tables
> >> > instead of reading all those page structures through /proc/vmcore.
> >> > 
> >> > The results are rather dramatic -- if they weren't I would not presume to
> >> > suggest such a radical path.
> >> > On a small, idle UV system: about 4 sec. versus about 40 sec.
> >> > On a 8TB UV the unnecessary page scan alone takes 4 minutes, vs. about 200 min
> >> > through /proc/vmcore.
> >> > 
> >> > I have not compared it to your version 1.5.1, so I don't know if your recent
> >> > work provides similar speedups.
> >> 
> >> I guess try 1.5.1-rc. IIUC, we had the logic of going through page tables
> >> but that required one single bitmap to be present and in constrained
> >> memory environment we will not have that.
> >> 
> >> That's when this idea came up that scan portion of struct page range,
> >> filter it, dump it and then move on to next range.
> >> 
> >> Even after 1.5.1-rc if difference is this dramatic, that means we are
> >> not doing something right in makedumpfile and it needs to be
> >> fixed/optimized.
> >> 
> >> But moving the logic to kernel does not make much sense to me at this
> >> point of time untile and unless there is a good explanation that why
> >> user space can't do a good job of what kernel is doing.
> > 
> > I tested a patch in which makedumpfile does nothing but scan all the
> > page structures using /proc/vmcore.  It is simply reading each consecutive
> > range of page structures in readmem() chunks of 512 structures. And doing
> > nothing more than accumulating a hash total of the 'flags' field in each
> > page (for a sanity check). On my test machine there are 6 blocks of page
> > structures, totaling 12 million structures. This takes 31.1 'units of time'
> > (I won't say seconds, as the speed of the clock seems to be way too fast in
> > the crash kernel). If I increase the buffer size to 5120 structures: 31.0 units.
> > At 51200 structures: 30.9.  So buffer size has virtually no effect.
> > 
> > I also request the kernel to do the same thing.  Each of the 6 requests
> > asks the kernel to scan a range of page structures and accumulate a hash
> > total of the 'flags' field.  (And also copy a 10000-element pfn list back
> > to user space, to test that such copies don't add significant overhead.)
> > And the 12 million pages are scanned in 1.6 'units of time'.
> > 
> > If I compare the time for actual page scanning (unnecessary pages and
> > free pages) through /proc/vmcore vs.  requesting the kernel to do the
> > scanning:  40 units vs. 3.8 units.
> > 
> > My conclusion is that makedumpfile's page scanning procedure is extremely
> > dominated by the overhead of copying page structures through /proc/vmcore.
> > And that is about 20x slower than using the kernel to access pages.
> 
> I have not tested your patch set on the machine with 2TB due to
> reservation problem, but I already tested it on my local machine with
> 32GB and saw big performance improvement.
> 
> I applied your patch set on makedumpfile v1.5.1-rc and added an option
> -N not to dump pages to focus on scanning pages part only.
> 
> By this, while scanning pages in user-space took about 25 seconds,
> scanning pages in kernel-space took about 1 second.
> 
> During the execution I profiled it using perf record/report and its
> results are attached files.
> 
> >From this we can notice that current makedumpfile consumes large part
> of execution time for ioremap and its related processing. copy_to_user
> was less than 2% only relative to a whole processing.
> 
> Looking into the codes around read method of /proc/vmcore, its call
> stack can be broken into as follows:
> 
>   read_vmcore
>     read_from_oldmem
>       copy_oldmem_page
> 
> copy_oldmem_page reads the 1st kernel's memory *per page* using
> ioremap_cache and after completing it, immediately unmaps the remapped
> address using iounmap.
> 
> Because ioremap/iounmap is called *per page*, this number of calling
> ioremap throught scanning a whole pages is unchanged even if
> makedumpfile's cache size is changed. This seems consistent with
> Cliff's explanation that increasing 512 entries of makedumpfile's
> cache was meaningless.
> 
> I think the first step to address this issue is to introduce a kind of
> cache in read_vmcore path to reduce the number of calling
> ioremap/iounmap. Porting scanning logic into kernel-space should be
> considered if it turns out not to work enough.
> 
> Thanks.
> HATAYAMA, Daisuke

(resending this as I seem to have messed up the email header)
Hi Hatayama,

If ioremap/iounmap is the bottleneck then perhaps you could do what
my patch does: it consolidates all the ranges of physical addresses
where the boot kernel's page structures reside (see make_kernel_mmap())
and passes them to the kernel, which then does a handfull of ioremaps's
to
cover all of them.  Then /proc/vmcore could look up the already-mapped
virtual address.
(also note a kludge in get_mm_sparsemem() that verifies that each
section
of the mem_map spans contiguous ranges of page structures.  I had
trouble with some sections when I made that assumption)

I'm attaching 3 patches that might be useful in your testing:
- 121210.proc_vmcore2  my current patch that applies to the released
  makedumpfile 1.5.1
- 121207.vmcore_pagescans.sles applies to a 3.0.13 kernel
- 121207.vmcore_pagescans.rhel applies to a 2.6.32 kernel

-Cliff

> sadump: does not have partition header
> sadump: read dump device as unknown format
> sadump: unknown format
> LOAD (0)
>   phys_start : 1000000
>   phys_end   : 22f1000
>   virt_start : ffffffff81000000
>   virt_end   : ffffffff822f1000
> LOAD (1)
>   phys_start : 10000
>   phys_end   : 9b000
>   virt_start : ffff880000010000
>   virt_end   : ffff88000009b000
> LOAD (2)
>   phys_start : 100000
>   phys_end   : 27000000
>   virt_start : ffff880000100000
>   virt_end   : ffff880027000000
> LOAD (3)
>   phys_start : 37000000
>   phys_end   : 7b00d000
>   virt_start : ffff880037000000
>   virt_end   : ffff88007b00d000
> LOAD (4)
>   phys_start : 100000000
>   phys_end   : 880000000
>   virt_start : ffff880100000000
>   virt_end   : ffff880880000000
> Linux kdump
> page_size    : 4096
> page_size    : 4096
> 
> max_mapnr    : 880000
> 
> Buffer size for the cyclic mode: 2228224
> 
> num of NODEs : 4
> 
> 
> Memory type  : SPARSEMEM_EX
> 
> mem_map (0)
>   mem_map    : ffffea0000000000
>   pfn_start  : 0
>   pfn_end    : 8000
> mem_map (1)
>   mem_map    : ffffea00001c0000
>   pfn_start  : 8000
>   pfn_end    : 10000
> mem_map (2)
>   mem_map    : ffffea0000380000
>   pfn_start  : 10000
>   pfn_end    : 18000
> mem_map (3)
>   mem_map    : ffffea0000540000
>   pfn_start  : 18000
>   pfn_end    : 20000
> mem_map (4)
>   mem_map    : ffffea0000700000
>   pfn_start  : 20000
>   pfn_end    : 28000
> mem_map (5)
>   mem_map    : ffffea00008c0000
>   pfn_start  : 28000
>   pfn_end    : 30000
> mem_map (6)
>   mem_map    : ffffea0000a80000
>   pfn_start  : 30000
>   pfn_end    : 38000
> mem_map (7)
>   mem_map    : ffffea0000c40000
>   pfn_start  : 38000
>   pfn_end    : 40000
> mem_map (8)
>   mem_map    : ffffea0000e00000
>   pfn_start  : 40000
>   pfn_end    : 48000
> mem_map (9)
>   mem_map    : ffffea0000fc0000
>   pfn_start  : 48000
>   pfn_end    : 50000
> mem_map (10)
>   mem_map    : ffffea0001180000
>   pfn_start  : 50000
>   pfn_end    : 58000
> mem_map (11)
>   mem_map    : ffffea0001340000
>   pfn_start  : 58000
>   pfn_end    : 60000
> mem_map (12)
>   mem_map    : ffffea0001500000
>   pfn_start  : 60000
>   pfn_end    : 68000
> mem_map (13)
>   mem_map    : ffffea00016c0000
>   pfn_start  : 68000
>   pfn_end    : 70000
> mem_map (14)
>   mem_map    : ffffea0001880000
>   pfn_start  : 70000
>   pfn_end    : 78000
> mem_map (15)
>   mem_map    : ffffea0001a40000
>   pfn_start  : 78000
>   pfn_end    : 80000
> mem_map (16)
>   mem_map    : 0
>   pfn_start  : 80000
>   pfn_end    : 88000
> mem_map (17)
>   mem_map    : 0
>   pfn_start  : 88000
>   pfn_end    : 90000
> mem_map (18)
>   mem_map    : 0
>   pfn_start  : 90000
>   pfn_end    : 98000
> mem_map (19)
>   mem_map    : 0
>   pfn_start  : 98000
>   pfn_end    : a0000
> mem_map (20)
>   mem_map    : 0
>   pfn_start  : a0000
>   pfn_end    : a8000
> mem_map (21)
>   mem_map    : 0
>   pfn_start  : a8000
>   pfn_end    : b0000
> mem_map (22)
>   mem_map    : 0
>   pfn_start  : b0000
>   pfn_end    : b8000
> mem_map (23)
>   mem_map    : 0
>   pfn_start  : b8000
>   pfn_end    : c0000
> mem_map (24)
>   mem_map    : 0
>   pfn_start  : c0000
>   pfn_end    : c8000
> mem_map (25)
>   mem_map    : 0
>   pfn_start  : c8000
>   pfn_end    : d0000
> mem_map (26)
>   mem_map    : 0
>   pfn_start  : d0000
>   pfn_end    : d8000
> mem_map (27)
>   mem_map    : 0
>   pfn_start  : d8000
>   pfn_end    : e0000
> mem_map (28)
>   mem_map    : 0
>   pfn_start  : e0000
>   pfn_end    : e8000
> mem_map (29)
>   mem_map    : 0
>   pfn_start  : e8000
>   pfn_end    : f0000
> mem_map (30)
>   mem_map    : 0
>   pfn_start  : f0000
>   pfn_end    : f8000
> mem_map (31)
>   mem_map    : 0
>   pfn_start  : f8000
>   pfn_end    : 100000
> mem_map (32)
>   mem_map    : ffffea0003800000
>   pfn_start  : 100000
>   pfn_end    : 108000
> mem_map (33)
>   mem_map    : ffffea00039c0000
>   pfn_start  : 108000
>   pfn_end    : 110000
> mem_map (34)
>   mem_map    : ffffea0003b80000
>   pfn_start  : 110000
>   pfn_end    : 118000
> mem_map (35)
>   mem_map    : ffffea0003d40000
>   pfn_start  : 118000
>   pfn_end    : 120000
> mem_map (36)
>   mem_map    : ffffea0003f00000
>   pfn_start  : 120000
>   pfn_end    : 128000
> mem_map (37)
>   mem_map    : ffffea00040c0000
>   pfn_start  : 128000
>   pfn_end    : 130000
> mem_map (38)
>   mem_map    : ffffea0004280000
>   pfn_start  : 130000
>   pfn_end    : 138000
> mem_map (39)
>   mem_map    : ffffea0004440000
>   pfn_start  : 138000
>   pfn_end    : 140000
> mem_map (40)
>   mem_map    : ffffea0004600000
>   pfn_start  : 140000
>   pfn_end    : 148000
> mem_map (41)
>   mem_map    : ffffea00047c0000
>   pfn_start  : 148000
>   pfn_end    : 150000
> mem_map (42)
>   mem_map    : ffffea0004980000
>   pfn_start  : 150000
>   pfn_end    : 158000
> mem_map (43)
>   mem_map    : ffffea0004b40000
>   pfn_start  : 158000
>   pfn_end    : 160000
> mem_map (44)
>   mem_map    : ffffea0004d00000
>   pfn_start  : 160000
>   pfn_end    : 168000
> mem_map (45)
>   mem_map    : ffffea0004ec0000
>   pfn_start  : 168000
>   pfn_end    : 170000
> mem_map (46)
>   mem_map    : ffffea0005080000
>   pfn_start  : 170000
>   pfn_end    : 178000
> mem_map (47)
>   mem_map    : ffffea0005240000
>   pfn_start  : 178000
>   pfn_end    : 180000
> mem_map (48)
>   mem_map    : ffffea0005400000
>   pfn_start  : 180000
>   pfn_end    : 188000
> mem_map (49)
>   mem_map    : ffffea00055c0000
>   pfn_start  : 188000
>   pfn_end    : 190000
> mem_map (50)
>   mem_map    : ffffea0005780000
>   pfn_start  : 190000
>   pfn_end    : 198000
> mem_map (51)
>   mem_map    : ffffea0005940000
>   pfn_start  : 198000
>   pfn_end    : 1a0000
> mem_map (52)
>   mem_map    : ffffea0005b00000
>   pfn_start  : 1a0000
>   pfn_end    : 1a8000
> mem_map (53)
>   mem_map    : ffffea0005cc0000
>   pfn_start  : 1a8000
>   pfn_end    : 1b0000
> mem_map (54)
>   mem_map    : ffffea0005e80000
>   pfn_start  : 1b0000
>   pfn_end    : 1b8000
> mem_map (55)
>   mem_map    : ffffea0006040000
>   pfn_start  : 1b8000
>   pfn_end    : 1c0000
> mem_map (56)
>   mem_map    : ffffea0006200000
>   pfn_start  : 1c0000
>   pfn_end    : 1c8000
> mem_map (57)
>   mem_map    : ffffea00063c0000
>   pfn_start  : 1c8000
>   pfn_end    : 1d0000
> mem_map (58)
>   mem_map    : ffffea0006580000
>   pfn_start  : 1d0000
>   pfn_end    : 1d8000
> mem_map (59)
>   mem_map    : ffffea0006740000
>   pfn_start  : 1d8000
>   pfn_end    : 1e0000
> mem_map (60)
>   mem_map    : ffffea0006900000
>   pfn_start  : 1e0000
>   pfn_end    : 1e8000
> mem_map (61)
>   mem_map    : ffffea0006ac0000
>   pfn_start  : 1e8000
>   pfn_end    : 1f0000
> mem_map (62)
>   mem_map    : ffffea0006c80000
>   pfn_start  : 1f0000
>   pfn_end    : 1f8000
> mem_map (63)
>   mem_map    : ffffea0006e40000
>   pfn_start  : 1f8000
>   pfn_end    : 200000
> mem_map (64)
>   mem_map    : ffffea0007000000
>   pfn_start  : 200000
>   pfn_end    : 208000
> mem_map (65)
>   mem_map    : ffffea00071c0000
>   pfn_start  : 208000
>   pfn_end    : 210000
> mem_map (66)
>   mem_map    : ffffea0007380000
>   pfn_start  : 210000
>   pfn_end    : 218000
> mem_map (67)
>   mem_map    : ffffea0007540000
>   pfn_start  : 218000
>   pfn_end    : 220000
> mem_map (68)
>   mem_map    : ffffea0007700000
>   pfn_start  : 220000
>   pfn_end    : 228000
> mem_map (69)
>   mem_map    : ffffea00078c0000
>   pfn_start  : 228000
>   pfn_end    : 230000
> mem_map (70)
>   mem_map    : ffffea0007a80000
>   pfn_start  : 230000
>   pfn_end    : 238000
> mem_map (71)
>   mem_map    : ffffea0007c40000
>   pfn_start  : 238000
>   pfn_end    : 240000
> mem_map (72)
>   mem_map    : ffffea0007e00000
>   pfn_start  : 240000
>   pfn_end    : 248000
> mem_map (73)
>   mem_map    : ffffea0007fc0000
>   pfn_start  : 248000
>   pfn_end    : 250000
> mem_map (74)
>   mem_map    : ffffea0008180000
>   pfn_start  : 250000
>   pfn_end    : 258000
> mem_map (75)
>   mem_map    : ffffea0008340000
>   pfn_start  : 258000
>   pfn_end    : 260000
> mem_map (76)
>   mem_map    : ffffea0008500000
>   pfn_start  : 260000
>   pfn_end    : 268000
> mem_map (77)
>   mem_map    : ffffea00086c0000
>   pfn_start  : 268000
>   pfn_end    : 270000
> mem_map (78)
>   mem_map    : ffffea0008880000
>   pfn_start  : 270000
>   pfn_end    : 278000
> mem_map (79)
>   mem_map    : ffffea0008a40000
>   pfn_start  : 278000
>   pfn_end    : 280000
> mem_map (80)
>   mem_map    : ffffea0008c00000
>   pfn_start  : 280000
>   pfn_end    : 288000
> mem_map (81)
>   mem_map    : ffffea0008dc0000
>   pfn_start  : 288000
>   pfn_end    : 290000
> mem_map (82)
>   mem_map    : ffffea0008f80000
>   pfn_start  : 290000
>   pfn_end    : 298000
> mem_map (83)
>   mem_map    : ffffea0009140000
>   pfn_start  : 298000
>   pfn_end    : 2a0000
> mem_map (84)
>   mem_map    : ffffea0009300000
>   pfn_start  : 2a0000
>   pfn_end    : 2a8000
> mem_map (85)
>   mem_map    : ffffea00094c0000
>   pfn_start  : 2a8000
>   pfn_end    : 2b0000
> mem_map (86)
>   mem_map    : ffffea0009680000
>   pfn_start  : 2b0000
>   pfn_end    : 2b8000
> mem_map (87)
>   mem_map    : ffffea0009840000
>   pfn_start  : 2b8000
>   pfn_end    : 2c0000
> mem_map (88)
>   mem_map    : ffffea0009a00000
>   pfn_start  : 2c0000
>   pfn_end    : 2c8000
> mem_map (89)
>   mem_map    : ffffea0009bc0000
>   pfn_start  : 2c8000
>   pfn_end    : 2d0000
> mem_map (90)
>   mem_map    : ffffea0009d80000
>   pfn_start  : 2d0000
>   pfn_end    : 2d8000
> mem_map (91)
>   mem_map    : ffffea0009f40000
>   pfn_start  : 2d8000
>   pfn_end    : 2e0000
> mem_map (92)
>   mem_map    : ffffea000a100000
>   pfn_start  : 2e0000
>   pfn_end    : 2e8000
> mem_map (93)
>   mem_map    : ffffea000a2c0000
>   pfn_start  : 2e8000
>   pfn_end    : 2f0000
> mem_map (94)
>   mem_map    : ffffea000a480000
>   pfn_start  : 2f0000
>   pfn_end    : 2f8000
> mem_map (95)
>   mem_map    : ffffea000a640000
>   pfn_start  : 2f8000
>   pfn_end    : 300000
> mem_map (96)
>   mem_map    : ffffea000a800000
>   pfn_start  : 300000
>   pfn_end    : 308000
> mem_map (97)
>   mem_map    : ffffea000a9c0000
>   pfn_start  : 308000
>   pfn_end    : 310000
> mem_map (98)
>   mem_map    : ffffea000ab80000
>   pfn_start  : 310000
>   pfn_end    : 318000
> mem_map (99)
>   mem_map    : ffffea000ad40000
>   pfn_start  : 318000
>   pfn_end    : 320000
> mem_map (100)
>   mem_map    : ffffea000af00000
>   pfn_start  : 320000
>   pfn_end    : 328000
> mem_map (101)
>   mem_map    : ffffea000b0c0000
>   pfn_start  : 328000
>   pfn_end    : 330000
> mem_map (102)
>   mem_map    : ffffea000b280000
>   pfn_start  : 330000
>   pfn_end    : 338000
> mem_map (103)
>   mem_map    : ffffea000b440000
>   pfn_start  : 338000
>   pfn_end    : 340000
> mem_map (104)
>   mem_map    : ffffea000b600000
>   pfn_start  : 340000
>   pfn_end    : 348000
> mem_map (105)
>   mem_map    : ffffea000b7c0000
>   pfn_start  : 348000
>   pfn_end    : 350000
> mem_map (106)
>   mem_map    : ffffea000b980000
>   pfn_start  : 350000
>   pfn_end    : 358000
> mem_map (107)
>   mem_map    : ffffea000bb40000
>   pfn_start  : 358000
>   pfn_end    : 360000
> mem_map (108)
>   mem_map    : ffffea000bd00000
>   pfn_start  : 360000
>   pfn_end    : 368000
> mem_map (109)
>   mem_map    : ffffea000bec0000
>   pfn_start  : 368000
>   pfn_end    : 370000
> mem_map (110)
>   mem_map    : ffffea000c080000
>   pfn_start  : 370000
>   pfn_end    : 378000
> mem_map (111)
>   mem_map    : ffffea000c240000
>   pfn_start  : 378000
>   pfn_end    : 380000
> mem_map (112)
>   mem_map    : ffffea000c400000
>   pfn_start  : 380000
>   pfn_end    : 388000
> mem_map (113)
>   mem_map    : ffffea000c5c0000
>   pfn_start  : 388000
>   pfn_end    : 390000
> mem_map (114)
>   mem_map    : ffffea000c780000
>   pfn_start  : 390000
>   pfn_end    : 398000
> mem_map (115)
>   mem_map    : ffffea000c940000
>   pfn_start  : 398000
>   pfn_end    : 3a0000
> mem_map (116)
>   mem_map    : ffffea000cb00000
>   pfn_start  : 3a0000
>   pfn_end    : 3a8000
> mem_map (117)
>   mem_map    : ffffea000ccc0000
>   pfn_start  : 3a8000
>   pfn_end    : 3b0000
> mem_map (118)
>   mem_map    : ffffea000ce80000
>   pfn_start  : 3b0000
>   pfn_end    : 3b8000
> mem_map (119)
>   mem_map    : ffffea000d040000
>   pfn_start  : 3b8000
>   pfn_end    : 3c0000
> mem_map (120)
>   mem_map    : ffffea000d200000
>   pfn_start  : 3c0000
>   pfn_end    : 3c8000
> mem_map (121)
>   mem_map    : ffffea000d3c0000
>   pfn_start  : 3c8000
>   pfn_end    : 3d0000
> mem_map (122)
>   mem_map    : ffffea000d580000
>   pfn_start  : 3d0000
>   pfn_end    : 3d8000
> mem_map (123)
>   mem_map    : ffffea000d740000
>   pfn_start  : 3d8000
>   pfn_end    : 3e0000
> mem_map (124)
>   mem_map    : ffffea000d900000
>   pfn_start  : 3e0000
>   pfn_end    : 3e8000
> mem_map (125)
>   mem_map    : ffffea000dac0000
>   pfn_start  : 3e8000
>   pfn_end    : 3f0000
> mem_map (126)
>   mem_map    : ffffea000dc80000
>   pfn_start  : 3f0000
>   pfn_end    : 3f8000
> mem_map (127)
>   mem_map    : ffffea000de40000
>   pfn_start  : 3f8000
>   pfn_end    : 400000
> mem_map (128)
>   mem_map    : ffffea000e000000
>   pfn_start  : 400000
>   pfn_end    : 408000
> mem_map (129)
>   mem_map    : ffffea000e1c0000
>   pfn_start  : 408000
>   pfn_end    : 410000
> mem_map (130)
>   mem_map    : ffffea000e380000
>   pfn_start  : 410000
>   pfn_end    : 418000
> mem_map (131)
>   mem_map    : ffffea000e540000
>   pfn_start  : 418000
>   pfn_end    : 420000
> mem_map (132)
>   mem_map    : ffffea000e700000
>   pfn_start  : 420000
>   pfn_end    : 428000
> mem_map (133)
>   mem_map    : ffffea000e8c0000
>   pfn_start  : 428000
>   pfn_end    : 430000
> mem_map (134)
>   mem_map    : ffffea000ea80000
>   pfn_start  : 430000
>   pfn_end    : 438000
> mem_map (135)
>   mem_map    : ffffea000ec40000
>   pfn_start  : 438000
>   pfn_end    : 440000
> mem_map (136)
>   mem_map    : ffffea000ee00000
>   pfn_start  : 440000
>   pfn_end    : 448000
> mem_map (137)
>   mem_map    : ffffea000efc0000
>   pfn_start  : 448000
>   pfn_end    : 450000
> mem_map (138)
>   mem_map    : ffffea000f180000
>   pfn_start  : 450000
>   pfn_end    : 458000
> mem_map (139)
>   mem_map    : ffffea000f340000
>   pfn_start  : 458000
>   pfn_end    : 460000
> mem_map (140)
>   mem_map    : ffffea000f500000
>   pfn_start  : 460000
>   pfn_end    : 468000
> mem_map (141)
>   mem_map    : ffffea000f6c0000
>   pfn_start  : 468000
>   pfn_end    : 470000
> mem_map (142)
>   mem_map    : ffffea000f880000
>   pfn_start  : 470000
>   pfn_end    : 478000
> mem_map (143)
>   mem_map    : ffffea000fa40000
>   pfn_start  : 478000
>   pfn_end    : 480000
> mem_map (144)
>   mem_map    : ffffea000fc00000
>   pfn_start  : 480000
>   pfn_end    : 488000
> mem_map (145)
>   mem_map    : ffffea000fdc0000
>   pfn_start  : 488000
>   pfn_end    : 490000
> mem_map (146)
>   mem_map    : ffffea000ff80000
>   pfn_start  : 490000
>   pfn_end    : 498000
> mem_map (147)
>   mem_map    : ffffea0010140000
>   pfn_start  : 498000
>   pfn_end    : 4a0000
> mem_map (148)
>   mem_map    : ffffea0010300000
>   pfn_start  : 4a0000
>   pfn_end    : 4a8000
> mem_map (149)
>   mem_map    : ffffea00104c0000
>   pfn_start  : 4a8000
>   pfn_end    : 4b0000
> mem_map (150)
>   mem_map    : ffffea0010680000
>   pfn_start  : 4b0000
>   pfn_end    : 4b8000
> mem_map (151)
>   mem_map    : ffffea0010840000
>   pfn_start  : 4b8000
>   pfn_end    : 4c0000
> mem_map (152)
>   mem_map    : ffffea0010a00000
>   pfn_start  : 4c0000
>   pfn_end    : 4c8000
> mem_map (153)
>   mem_map    : ffffea0010bc0000
>   pfn_start  : 4c8000
>   pfn_end    : 4d0000
> mem_map (154)
>   mem_map    : ffffea0010d80000
>   pfn_start  : 4d0000
>   pfn_end    : 4d8000
> mem_map (155)
>   mem_map    : ffffea0010f40000
>   pfn_start  : 4d8000
>   pfn_end    : 4e0000
> mem_map (156)
>   mem_map    : ffffea0011100000
>   pfn_start  : 4e0000
>   pfn_end    : 4e8000
> mem_map (157)
>   mem_map    : ffffea00112c0000
>   pfn_start  : 4e8000
>   pfn_end    : 4f0000
> mem_map (158)
>   mem_map    : ffffea0011480000
>   pfn_start  : 4f0000
>   pfn_end    : 4f8000
> mem_map (159)
>   mem_map    : ffffea0011640000
>   pfn_start  : 4f8000
>   pfn_end    : 500000
> mem_map (160)
>   mem_map    : ffffea0011800000
>   pfn_start  : 500000
>   pfn_end    : 508000
> mem_map (161)
>   mem_map    : ffffea00119c0000
>   pfn_start  : 508000
>   pfn_end    : 510000
> mem_map (162)
>   mem_map    : ffffea0011b80000
>   pfn_start  : 510000
>   pfn_end    : 518000
> mem_map (163)
>   mem_map    : ffffea0011d40000
>   pfn_start  : 518000
>   pfn_end    : 520000
> mem_map (164)
>   mem_map    : ffffea0011f00000
>   pfn_start  : 520000
>   pfn_end    : 528000
> mem_map (165)
>   mem_map    : ffffea00120c0000
>   pfn_start  : 528000
>   pfn_end    : 530000
> mem_map (166)
>   mem_map    : ffffea0012280000
>   pfn_start  : 530000
>   pfn_end    : 538000
> mem_map (167)
>   mem_map    : ffffea0012440000
>   pfn_start  : 538000
>   pfn_end    : 540000
> mem_map (168)
>   mem_map    : ffffea0012600000
>   pfn_start  : 540000
>   pfn_end    : 548000
> mem_map (169)
>   mem_map    : ffffea00127c0000
>   pfn_start  : 548000
>   pfn_end    : 550000
> mem_map (170)
>   mem_map    : ffffea0012980000
>   pfn_start  : 550000
>   pfn_end    : 558000
> mem_map (171)
>   mem_map    : ffffea0012b40000
>   pfn_start  : 558000
>   pfn_end    : 560000
> mem_map (172)
>   mem_map    : ffffea0012d00000
>   pfn_start  : 560000
>   pfn_end    : 568000
> mem_map (173)
>   mem_map    : ffffea0012ec0000
>   pfn_start  : 568000
>   pfn_end    : 570000
> mem_map (174)
>   mem_map    : ffffea0013080000
>   pfn_start  : 570000
>   pfn_end    : 578000
> mem_map (175)
>   mem_map    : ffffea0013240000
>   pfn_start  : 578000
>   pfn_end    : 580000
> mem_map (176)
>   mem_map    : ffffea0013400000
>   pfn_start  : 580000
>   pfn_end    : 588000
> mem_map (177)
>   mem_map    : ffffea00135c0000
>   pfn_start  : 588000
>   pfn_end    : 590000
> mem_map (178)
>   mem_map    : ffffea0013780000
>   pfn_start  : 590000
>   pfn_end    : 598000
> mem_map (179)
>   mem_map    : ffffea0013940000
>   pfn_start  : 598000
>   pfn_end    : 5a0000
> mem_map (180)
>   mem_map    : ffffea0013b00000
>   pfn_start  : 5a0000
>   pfn_end    : 5a8000
> mem_map (181)
>   mem_map    : ffffea0013cc0000
>   pfn_start  : 5a8000
>   pfn_end    : 5b0000
> mem_map (182)
>   mem_map    : ffffea0013e80000
>   pfn_start  : 5b0000
>   pfn_end    : 5b8000
> mem_map (183)
>   mem_map    : ffffea0014040000
>   pfn_start  : 5b8000
>   pfn_end    : 5c0000
> mem_map (184)
>   mem_map    : ffffea0014200000
>   pfn_start  : 5c0000
>   pfn_end    : 5c8000
> mem_map (185)
>   mem_map    : ffffea00143c0000
>   pfn_start  : 5c8000
>   pfn_end    : 5d0000
> mem_map (186)
>   mem_map    : ffffea0014580000
>   pfn_start  : 5d0000
>   pfn_end    : 5d8000
> mem_map (187)
>   mem_map    : ffffea0014740000
>   pfn_start  : 5d8000
>   pfn_end    : 5e0000
> mem_map (188)
>   mem_map    : ffffea0014900000
>   pfn_start  : 5e0000
>   pfn_end    : 5e8000
> mem_map (189)
>   mem_map    : ffffea0014ac0000
>   pfn_start  : 5e8000
>   pfn_end    : 5f0000
> mem_map (190)
>   mem_map    : ffffea0014c80000
>   pfn_start  : 5f0000
>   pfn_end    : 5f8000
> mem_map (191)
>   mem_map    : ffffea0014e40000
>   pfn_start  : 5f8000
>   pfn_end    : 600000
> mem_map (192)
>   mem_map    : ffffea0015000000
>   pfn_start  : 600000
>   pfn_end    : 608000
> mem_map (193)
>   mem_map    : ffffea00151c0000
>   pfn_start  : 608000
>   pfn_end    : 610000
> mem_map (194)
>   mem_map    : ffffea0015380000
>   pfn_start  : 610000
>   pfn_end    : 618000
> mem_map (195)
>   mem_map    : ffffea0015540000
>   pfn_start  : 618000
>   pfn_end    : 620000
> mem_map (196)
>   mem_map    : ffffea0015700000
>   pfn_start  : 620000
>   pfn_end    : 628000
> mem_map (197)
>   mem_map    : ffffea00158c0000
>   pfn_start  : 628000
>   pfn_end    : 630000
> mem_map (198)
>   mem_map    : ffffea0015a80000
>   pfn_start  : 630000
>   pfn_end    : 638000
> mem_map (199)
>   mem_map    : ffffea0015c40000
>   pfn_start  : 638000
>   pfn_end    : 640000
> mem_map (200)
>   mem_map    : ffffea0015e00000
>   pfn_start  : 640000
>   pfn_end    : 648000
> mem_map (201)
>   mem_map    : ffffea0015fc0000
>   pfn_start  : 648000
>   pfn_end    : 650000
> mem_map (202)
>   mem_map    : ffffea0016180000
>   pfn_start  : 650000
>   pfn_end    : 658000
> mem_map (203)
>   mem_map    : ffffea0016340000
>   pfn_start  : 658000
>   pfn_end    : 660000
> mem_map (204)
>   mem_map    : ffffea0016500000
>   pfn_start  : 660000
>   pfn_end    : 668000
> mem_map (205)
>   mem_map    : ffffea00166c0000
>   pfn_start  : 668000
>   pfn_end    : 670000
> mem_map (206)
>   mem_map    : ffffea0016880000
>   pfn_start  : 670000
>   pfn_end    : 678000
> mem_map (207)
>   mem_map    : ffffea0016a40000
>   pfn_start  : 678000
>   pfn_end    : 680000
> mem_map (208)
>   mem_map    : ffffea0016c00000
>   pfn_start  : 680000
>   pfn_end    : 688000
> mem_map (209)
>   mem_map    : ffffea0016dc0000
>   pfn_start  : 688000
>   pfn_end    : 690000
> mem_map (210)
>   mem_map    : ffffea0016f80000
>   pfn_start  : 690000
>   pfn_end    : 698000
> mem_map (211)
>   mem_map    : ffffea0017140000
>   pfn_start  : 698000
>   pfn_end    : 6a0000
> mem_map (212)
>   mem_map    : ffffea0017300000
>   pfn_start  : 6a0000
>   pfn_end    : 6a8000
> mem_map (213)
>   mem_map    : ffffea00174c0000
>   pfn_start  : 6a8000
>   pfn_end    : 6b0000
> mem_map (214)
>   mem_map    : ffffea0017680000
>   pfn_start  : 6b0000
>   pfn_end    : 6b8000
> mem_map (215)
>   mem_map    : ffffea0017840000
>   pfn_start  : 6b8000
>   pfn_end    : 6c0000
> mem_map (216)
>   mem_map    : ffffea0017a00000
>   pfn_start  : 6c0000
>   pfn_end    : 6c8000
> mem_map (217)
>   mem_map    : ffffea0017bc0000
>   pfn_start  : 6c8000
>   pfn_end    : 6d0000
> mem_map (218)
>   mem_map    : ffffea0017d80000
>   pfn_start  : 6d0000
>   pfn_end    : 6d8000
> mem_map (219)
>   mem_map    : ffffea0017f40000
>   pfn_start  : 6d8000
>   pfn_end    : 6e0000
> mem_map (220)
>   mem_map    : ffffea0018100000
>   pfn_start  : 6e0000
>   pfn_end    : 6e8000
> mem_map (221)
>   mem_map    : ffffea00182c0000
>   pfn_start  : 6e8000
>   pfn_end    : 6f0000
> mem_map (222)
>   mem_map    : ffffea0018480000
>   pfn_start  : 6f0000
>   pfn_end    : 6f8000
> mem_map (223)
>   mem_map    : ffffea0018640000
>   pfn_start  : 6f8000
>   pfn_end    : 700000
> mem_map (224)
>   mem_map    : ffffea0018800000
>   pfn_start  : 700000
>   pfn_end    : 708000
> mem_map (225)
>   mem_map    : ffffea00189c0000
>   pfn_start  : 708000
>   pfn_end    : 710000
> mem_map (226)
>   mem_map    : ffffea0018b80000
>   pfn_start  : 710000
>   pfn_end    : 718000
> mem_map (227)
>   mem_map    : ffffea0018d40000
>   pfn_start  : 718000
>   pfn_end    : 720000
> mem_map (228)
>   mem_map    : ffffea0018f00000
>   pfn_start  : 720000
>   pfn_end    : 728000
> mem_map (229)
>   mem_map    : ffffea00190c0000
>   pfn_start  : 728000
>   pfn_end    : 730000
> mem_map (230)
>   mem_map    : ffffea0019280000
>   pfn_start  : 730000
>   pfn_end    : 738000
> mem_map (231)
>   mem_map    : ffffea0019440000
>   pfn_start  : 738000
>   pfn_end    : 740000
> mem_map (232)
>   mem_map    : ffffea0019600000
>   pfn_start  : 740000
>   pfn_end    : 748000
> mem_map (233)
>   mem_map    : ffffea00197c0000
>   pfn_start  : 748000
>   pfn_end    : 750000
> mem_map (234)
>   mem_map    : ffffea0019980000
>   pfn_start  : 750000
>   pfn_end    : 758000
> mem_map (235)
>   mem_map    : ffffea0019b40000
>   pfn_start  : 758000
>   pfn_end    : 760000
> mem_map (236)
>   mem_map    : ffffea0019d00000
>   pfn_start  : 760000
>   pfn_end    : 768000
> mem_map (237)
>   mem_map    : ffffea0019ec0000
>   pfn_start  : 768000
>   pfn_end    : 770000
> mem_map (238)
>   mem_map    : ffffea001a080000
>   pfn_start  : 770000
>   pfn_end    : 778000
> mem_map (239)
>   mem_map    : ffffea001a240000
>   pfn_start  : 778000
>   pfn_end    : 780000
> mem_map (240)
>   mem_map    : ffffea001a400000
>   pfn_start  : 780000
>   pfn_end    : 788000
> mem_map (241)
>   mem_map    : ffffea001a5c0000
>   pfn_start  : 788000
>   pfn_end    : 790000
> mem_map (242)
>   mem_map    : ffffea001a780000
>   pfn_start  : 790000
>   pfn_end    : 798000
> mem_map (243)
>   mem_map    : ffffea001a940000
>   pfn_start  : 798000
>   pfn_end    : 7a0000
> mem_map (244)
>   mem_map    : ffffea001ab00000
>   pfn_start  : 7a0000
>   pfn_end    : 7a8000
> mem_map (245)
>   mem_map    : ffffea001acc0000
>   pfn_start  : 7a8000
>   pfn_end    : 7b0000
> mem_map (246)
>   mem_map    : ffffea001ae80000
>   pfn_start  : 7b0000
>   pfn_end    : 7b8000
> mem_map (247)
>   mem_map    : ffffea001b040000
>   pfn_start  : 7b8000
>   pfn_end    : 7c0000
> mem_map (248)
>   mem_map    : ffffea001b200000
>   pfn_start  : 7c0000
>   pfn_end    : 7c8000
> mem_map (249)
>   mem_map    : ffffea001b3c0000
>   pfn_start  : 7c8000
>   pfn_end    : 7d0000
> mem_map (250)
>   mem_map    : ffffea001b580000
>   pfn_start  : 7d0000
>   pfn_end    : 7d8000
> mem_map (251)
>   mem_map    : ffffea001b740000
>   pfn_start  : 7d8000
>   pfn_end    : 7e0000
> mem_map (252)
>   mem_map    : ffffea001b900000
>   pfn_start  : 7e0000
>   pfn_end    : 7e8000
> mem_map (253)
>   mem_map    : ffffea001bac0000
>   pfn_start  : 7e8000
>   pfn_end    : 7f0000
> mem_map (254)
>   mem_map    : ffffea001bc80000
>   pfn_start  : 7f0000
>   pfn_end    : 7f8000
> mem_map (255)
>   mem_map    : ffffea001be40000
>   pfn_start  : 7f8000
>   pfn_end    : 800000
> mem_map (256)
>   mem_map    : ffffea001c000000
>   pfn_start  : 800000
>   pfn_end    : 808000
> mem_map (257)
>   mem_map    : ffffea001c1c0000
>   pfn_start  : 808000
>   pfn_end    : 810000
> mem_map (258)
>   mem_map    : ffffea001c380000
>   pfn_start  : 810000
>   pfn_end    : 818000
> mem_map (259)
>   mem_map    : ffffea001c540000
>   pfn_start  : 818000
>   pfn_end    : 820000
> mem_map (260)
>   mem_map    : ffffea001c700000
>   pfn_start  : 820000
>   pfn_end    : 828000
> mem_map (261)
>   mem_map    : ffffea001c8c0000
>   pfn_start  : 828000
>   pfn_end    : 830000
> mem_map (262)
>   mem_map    : ffffea001ca80000
>   pfn_start  : 830000
>   pfn_end    : 838000
> mem_map (263)
>   mem_map    : ffffea001cc40000
>   pfn_start  : 838000
>   pfn_end    : 840000
> mem_map (264)
>   mem_map    : ffffea001ce00000
>   pfn_start  : 840000
>   pfn_end    : 848000
> mem_map (265)
>   mem_map    : ffffea001cfc0000
>   pfn_start  : 848000
>   pfn_end    : 850000
> mem_map (266)
>   mem_map    : ffffea001d180000
>   pfn_start  : 850000
>   pfn_end    : 858000
> mem_map (267)
>   mem_map    : ffffea001d340000
>   pfn_start  : 858000
>   pfn_end    : 860000
> mem_map
Excluding unnecessary pages        : [  0 %] 
Excluding unnecessary pages        : [  3 %] 
Excluding unnecessary pages        : [ 17 %] 
Excluding unnecessary pages        : [ 26 %] 
Excluding unnecessary pages        : [ 34 %] 
Excluding unnecessary pages        : [ 42 %] 
Excluding unnecessary pages        : [ 50 %] 
Excluding unnecessary pages        : [ 59 %] 
Excluding unnecessary pages        : [ 67 %] 
Excluding unnecessary pages        : [ 75 %] 
Excluding unnecessary pages        : [ 84 %] 
Excluding unnecessary pages        : [ 92 %] 
Excluding unnecessary pages        : [100 %] 
Excluding unnecessary pages        : [  0 %] 
Excluding unnecessary pages        : [ 13 %] 
Excluding unnecessary pages        : [ 22 %] 
Excluding unnecessary pages        : [ 30 %] 
Excluding unnecessary pages        : [ 38 %] 
Excluding unnecessary pages        : [ 47 %] 
Excluding unnecessary pages        : [ 55 %] 
Excluding unnecessary pages        : [ 63 %] 
Excluding unnecessary pages        : [ 71 %] 
Excluding unnecessary pages        : [ 80 %] 
Excluding unnecessary pages        : [ 88 %] 
Excluding unnecessary pages        : [ 96 %] 
Excluding unnecessary pages        : [100 %] 
>  (268)
>   mem_map    : ffffea001d500000
>   pfn_start  : 860000
>   pfn_end    : 868000
> mem_map (269)
>   mem_map    : ffffea001d6c0000
>   pfn_start  : 868000
>   pfn_end    : 870000
> mem_map (270)
>   mem_map    : ffffea001d880000
>   pfn_start  : 870000
>   pfn_end    : 878000
> mem_map (271)
>   mem_map    : ffffea001da40000
>   pfn_start  : 878000
>   pfn_end    : 880000
> STEP [Excluding unnecessary pages] : 11.224292 seconds
> STEP [Excluding unnecessary pages] : 11.250834 seconds
> STEP [Copying data               ] : 11.407836 seconds
> Writing erase info...
> offset_eraseinfo: 6c0eb8, size_eraseinfo: 0
> 
> Original pages  : 0x00000000007ec289
>   Excluded pages   : 0x00000000007b9efb
>     Pages filled with zero  : 0x0000000000000000
>     Cache pages             : 0x0000000000012cff
>     Cache pages + private   : 0x0000000000012015
>     User process data pages : 0x0000000000001ece
>     Free pages              : 0x0000000000793319
>   Remaining pages  : 0x000000000003238e
>   (The number of pages is reduced to 2%.)
> Memory Hole     : 0x0000000000093d77
> --------------------------------------------------
> Total pages     : 0x0000000000880000
> 
> 
> The dumpfile is saved to /mnt///127.0.0.1-2012-12-07-16:45:20/vmcore-empty-1.
> 
> makedumpfile Completed.
> [ perf record: Woken up 55 times to write data ]
> [ perf record: Captured and wrote 13.735 MB /mnt///127.0.0.1-2012-12-07-16:45:20/perf.data1 (~600084 samples) ]

> sadump: does not have partition header
> sadump: read dump device as unknown format
> sadump: unknown format
> LOAD (0)
>   phys_start : 1000000
>   phys_end   : 22f1000
>   virt_start : ffffffff81000000
>   virt_end   : ffffffff822f1000
> LOAD (1)
>   phys_start : 10000
>   phys_end   : 9b000
>   virt_start : ffff880000010000
>   virt_end   : ffff88000009b000
> LOAD (2)
>   phys_start : 100000
>   phys_end   : 27000000
>   virt_start : ffff880000100000
>   virt_end   : ffff880027000000
> LOAD (3)
>   phys_start : 37000000
>   phys_end   : 7b00d000
>   virt_start : ffff880037000000
>   virt_end   : ffff88007b00d000
> LOAD (4)
>   phys_start : 100000000
>   phys_end   : 880000000
>   virt_start : ffff880100000000
>   virt_end   : ffff880880000000
> Linux kdump
> page_size    : 4096
> page_size    : 4096
> 
> max_mapnr    : 880000
> 
> Buffer size for the cyclic mode: 2228224
> 
> num of NODEs : 4
> 
> 
> Memory type  : SPARSEMEM_EX
> 
> mem_map (0)
>   mem_map    : ffffea0000000000
>   pfn_start  : 0
>   pfn_end    : 8000
> mem_map (1)
>   mem_map    : ffffea00001c0000
>   pfn_start  : 8000
>   pfn_end    : 10000
> mem_map (2)
>   mem_map    : ffffea0000380000
>   pfn_start  : 10000
>   pfn_end    : 18000
> mem_map (3)
>   mem_map    : ffffea0000540000
>   pfn_start  : 18000
>   pfn_end    : 20000
> mem_map (4)
>   mem_map    : ffffea0000700000
>   pfn_start  : 20000
>   pfn_end    : 28000
> mem_map (5)
>   mem_map    : ffffea00008c0000
>   pfn_start  : 28000
>   pfn_end    : 30000
> mem_map (6)
>   mem_map    : ffffea0000a80000
>   pfn_start  : 30000
>   pfn_end    : 38000
> mem_map (7)
>   mem_map    : ffffea0000c40000
>   pfn_start  : 38000
>   pfn_end    : 40000
> mem_map (8)
>   mem_map    : ffffea0000e00000
>   pfn_start  : 40000
>   pfn_end    : 48000
> mem_map (9)
>   mem_map    : ffffea0000fc0000
>   pfn_start  : 48000
>   pfn_end    : 50000
> mem_map (10)
>   mem_map    : ffffea0001180000
>   pfn_start  : 50000
>   pfn_end    : 58000
> mem_map (11)
>   mem_map    : ffffea0001340000
>   pfn_start  : 58000
>   pfn_end    : 60000
> mem_map (12)
>   mem_map    : ffffea0001500000
>   pfn_start  : 60000
>   pfn_end    : 68000
> mem_map (13)
>   mem_map    : ffffea00016c0000
>   pfn_start  : 68000
>   pfn_end    : 70000
> mem_map (14)
>   mem_map    : ffffea0001880000
>   pfn_start  : 70000
>   pfn_end    : 78000
> mem_map (15)
>   mem_map    : ffffea0001a40000
>   pfn_start  : 78000
>   pfn_end    : 80000
> mem_map (16)
>   mem_map    : 0
>   pfn_start  : 80000
>   pfn_end    : 88000
> mem_map (17)
>   mem_map    : 0
>   pfn_start  : 88000
>   pfn_end    : 90000
> mem_map (18)
>   mem_map    : 0
>   pfn_start  : 90000
>   pfn_end    : 98000
> mem_map (19)
>   mem_map    : 0
>   pfn_start  : 98000
>   pfn_end    : a0000
> mem_map (20)
>   mem_map    : 0
>   pfn_start  : a0000
>   pfn_end    : a8000
> mem_map (21)
>   mem_map    : 0
>   pfn_start  : a8000
>   pfn_end    : b0000
> mem_map (22)
>   mem_map    : 0
>   pfn_start  : b0000
>   pfn_end    : b8000
> mem_map (23)
>   mem_map    : 0
>   pfn_start  : b8000
>   pfn_end    : c0000
> mem_map (24)
>   mem_map    : 0
>   pfn_start  : c0000
>   pfn_end    : c8000
> mem_map (25)
>   mem_map    : 0
>   pfn_start  : c8000
>   pfn_end    : d0000
> mem_map (26)
>   mem_map    : 0
>   pfn_start  : d0000
>   pfn_end    : d8000
> mem_map (27)
>   mem_map    : 0
>   pfn_start  : d8000
>   pfn_end    : e0000
> mem_map (28)
>   mem_map    : 0
>   pfn_start  : e0000
>   pfn_end    : e8000
> mem_map (29)
>   mem_map    : 0
>   pfn_start  : e8000
>   pfn_end    : f0000
> mem_map (30)
>   mem_map    : 0
>   pfn_start  : f0000
>   pfn_end    : f8000
> mem_map (31)
>   mem_map    : 0
>   pfn_start  : f8000
>   pfn_end    : 100000
> mem_map (32)
>   mem_map    : ffffea0003800000
>   pfn_start  : 100000
>   pfn_end    : 108000
> mem_map (33)
>   mem_map    : ffffea00039c0000
>   pfn_start  : 108000
>   pfn_end    : 110000
> mem_map (34)
>   mem_map    : ffffea0003b80000
>   pfn_start  : 110000
>   pfn_end    : 118000
> mem_map (35)
>   mem_map    : ffffea0003d40000
>   pfn_start  : 118000
>   pfn_end    : 120000
> mem_map (36)
>   mem_map    : ffffea0003f00000
>   pfn_start  : 120000
>   pfn_end    : 128000
> mem_map (37)
>   mem_map    : ffffea00040c0000
>   pfn_start  : 128000
>   pfn_end    : 130000
> mem_map (38)
>   mem_map    : ffffea0004280000
>   pfn_start  : 130000
>   pfn_end    : 138000
> mem_map (39)
>   mem_map    : ffffea0004440000
>   pfn_start  : 138000
>   pfn_end    : 140000
> mem_map (40)
>   mem_map    : ffffea0004600000
>   pfn_start  : 140000
>   pfn_end    : 148000
> mem_map (41)
>   mem_map    : ffffea00047c0000
>   pfn_start  : 148000
>   pfn_end    : 150000
> mem_map (42)
>   mem_map    : ffffea0004980000
>   pfn_start  : 150000
>   pfn_end    : 158000
> mem_map (43)
>   mem_map    : ffffea0004b40000
>   pfn_start  : 158000
>   pfn_end    : 160000
> mem_map (44)
>   mem_map    : ffffea0004d00000
>   pfn_start  : 160000
>   pfn_end    : 168000
> mem_map (45)
>   mem_map    : ffffea0004ec0000
>   pfn_start  : 168000
>   pfn_end    : 170000
> mem_map (46)
>   mem_map    : ffffea0005080000
>   pfn_start  : 170000
>   pfn_end    : 178000
> mem_map (47)
>   mem_map    : ffffea0005240000
>   pfn_start  : 178000
>   pfn_end    : 180000
> mem_map (48)
>   mem_map    : ffffea0005400000
>   pfn_start  : 180000
>   pfn_end    : 188000
> mem_map (49)
>   mem_map    : ffffea00055c0000
>   pfn_start  : 188000
>   pfn_end    : 190000
> mem_map (50)
>   mem_map    : ffffea0005780000
>   pfn_start  : 190000
>   pfn_end    : 198000
> mem_map (51)
>   mem_map    : ffffea0005940000
>   pfn_start  : 198000
>   pfn_end    : 1a0000
> mem_map (52)
>   mem_map    : ffffea0005b00000
>   pfn_start  : 1a0000
>   pfn_end    : 1a8000
> mem_map (53)
>   mem_map    : ffffea0005cc0000
>   pfn_start  : 1a8000
>   pfn_end    : 1b0000
> mem_map (54)
>   mem_map    : ffffea0005e80000
>   pfn_start  : 1b0000
>   pfn_end    : 1b8000
> mem_map (55)
>   mem_map    : ffffea0006040000
>   pfn_start  : 1b8000
>   pfn_end    : 1c0000
> mem_map (56)
>   mem_map    : ffffea0006200000
>   pfn_start  : 1c0000
>   pfn_end    : 1c8000
> mem_map (57)
>   mem_map    : ffffea00063c0000
>   pfn_start  : 1c8000
>   pfn_end    : 1d0000
> mem_map (58)
>   mem_map    : ffffea0006580000
>   pfn_start  : 1d0000
>   pfn_end    : 1d8000
> mem_map (59)
>   mem_map    : ffffea0006740000
>   pfn_start  : 1d8000
>   pfn_end    : 1e0000
> mem_map (60)
>   mem_map    : ffffea0006900000
>   pfn_start  : 1e0000
>   pfn_end    : 1e8000
> mem_map (61)
>   mem_map    : ffffea0006ac0000
>   pfn_start  : 1e8000
>   pfn_end    : 1f0000
> mem_map (62)
>   mem_map    : ffffea0006c80000
>   pfn_start  : 1f0000
>   pfn_end    : 1f8000
> mem_map (63)
>   mem_map    : ffffea0006e40000
>   pfn_start  : 1f8000
>   pfn_end    : 200000
> mem_map (64)
>   mem_map    : ffffea0007000000
>   pfn_start  : 200000
>   pfn_end    : 208000
> mem_map (65)
>   mem_map    : ffffea00071c0000
>   pfn_start  : 208000
>   pfn_end    : 210000
> mem_map (66)
>   mem_map    : ffffea0007380000
>   pfn_start  : 210000
>   pfn_end    : 218000
> mem_map (67)
>   mem_map    : ffffea0007540000
>   pfn_start  : 218000
>   pfn_end    : 220000
> mem_map (68)
>   mem_map    : ffffea0007700000
>   pfn_start  : 220000
>   pfn_end    : 228000
> mem_map (69)
>   mem_map    : ffffea00078c0000
>   pfn_start  : 228000
>   pfn_end    : 230000
> mem_map (70)
>   mem_map    : ffffea0007a80000
>   pfn_start  : 230000
>   pfn_end    : 238000
> mem_map (71)
>   mem_map    : ffffea0007c40000
>   pfn_start  : 238000
>   pfn_end    : 240000
> mem_map (72)
>   mem_map    : ffffea0007e00000
>   pfn_start  : 240000
>   pfn_end    : 248000
> mem_map (73)
>   mem_map    : ffffea0007fc0000
>   pfn_start  : 248000
>   pfn_end    : 250000
> mem_map (74)
>   mem_map    : ffffea0008180000
>   pfn_start  : 250000
>   pfn_end    : 258000
> mem_map (75)
>   mem_map    : ffffea0008340000
>   pfn_start  : 258000
>   pfn_end    : 260000
> mem_map (76)
>   mem_map    : ffffea0008500000
>   pfn_start  : 260000
>   pfn_end    : 268000
> mem_map (77)
>   mem_map    : ffffea00086c0000
>   pfn_start  : 268000
>   pfn_end    : 270000
> mem_map (78)
>   mem_map    : ffffea0008880000
>   pfn_start  : 270000
>   pfn_end    : 278000
> mem_map (79)
>   mem_map    : ffffea0008a40000
>   pfn_start  : 278000
>   pfn_end    : 280000
> mem_map (80)
>   mem_map    : ffffea0008c00000
>   pfn_start  : 280000
>   pfn_end    : 288000
> mem_map (81)
>   mem_map    : ffffea0008dc0000
>   pfn_start  : 288000
>   pfn_end    : 290000
> mem_map (82)
>   mem_map    : ffffea0008f80000
>   pfn_start  : 290000
>   pfn_end    : 298000
> mem_map (83)
>   mem_map    : ffffea0009140000
>   pfn_start  : 298000
>   pfn_end    : 2a0000
> mem_map (84)
>   mem_map    : ffffea0009300000
>   pfn_start  : 2a0000
>   pfn_end    : 2a8000
> mem_map (85)
>   mem_map    : ffffea00094c0000
>   pfn_start  : 2a8000
>   pfn_end    : 2b0000
> mem_map (86)
>   mem_map    : ffffea0009680000
>   pfn_start  : 2b0000
>   pfn_end    : 2b8000
> mem_map (87)
>   mem_map    : ffffea0009840000
>   pfn_start  : 2b8000
>   pfn_end    : 2c0000
> mem_map (88)
>   mem_map    : ffffea0009a00000
>   pfn_start  : 2c0000
>   pfn_end    : 2c8000
> mem_map (89)
>   mem_map    : ffffea0009bc0000
>   pfn_start  : 2c8000
>   pfn_end    : 2d0000
> mem_map (90)
>   mem_map    : ffffea0009d80000
>   pfn_start  : 2d0000
>   pfn_end    : 2d8000
> mem_map (91)
>   mem_map    : ffffea0009f40000
>   pfn_start  : 2d8000
>   pfn_end    : 2e0000
> mem_map (92)
>   mem_map    : ffffea000a100000
>   pfn_start  : 2e0000
>   pfn_end    : 2e8000
> mem_map (93)
>   mem_map    : ffffea000a2c0000
>   pfn_start  : 2e8000
>   pfn_end    : 2f0000
> mem_map (94)
>   mem_map    : ffffea000a480000
>   pfn_start  : 2f0000
>   pfn_end    : 2f8000
> mem_map (95)
>   mem_map    : ffffea000a640000
>   pfn_start  : 2f8000
>   pfn_end    : 300000
> mem_map (96)
>   mem_map    : ffffea000a800000
>   pfn_start  : 300000
>   pfn_end    : 308000
> mem_map (97)
>   mem_map    : ffffea000a9c0000
>   pfn_start  : 308000
>   pfn_end    : 310000
> mem_map (98)
>   mem_map    : ffffea000ab80000
>   pfn_start  : 310000
>   pfn_end    : 318000
> mem_map (99)
>   mem_map    : ffffea000ad40000
>   pfn_start  : 318000
>   pfn_end    : 320000
> mem_map (100)
>   mem_map    : ffffea000af00000
>   pfn_start  : 320000
>   pfn_end    : 328000
> mem_map (101)
>   mem_map    : ffffea000b0c0000
>   pfn_start  : 328000
>   pfn_end    : 330000
> mem_map (102)
>   mem_map    : ffffea000b280000
>   pfn_start  : 330000
>   pfn_end    : 338000
> mem_map (103)
>   mem_map    : ffffea000b440000
>   pfn_start  : 338000
>   pfn_end    : 340000
> mem_map (104)
>   mem_map    : ffffea000b600000
>   pfn_start  : 340000
>   pfn_end    : 348000
> mem_map (105)
>   mem_map    : ffffea000b7c0000
>   pfn_start  : 348000
>   pfn_end    : 350000
> mem_map (106)
>   mem_map    : ffffea000b980000
>   pfn_start  : 350000
>   pfn_end    : 358000
> mem_map (107)
>   mem_map    : ffffea000bb40000
>   pfn_start  : 358000
>   pfn_end    : 360000
> mem_map (108)
>   mem_map    : ffffea000bd00000
>   pfn_start  : 360000
>   pfn_end    : 368000
> mem_map (109)
>   mem_map    : ffffea000bec0000
>   pfn_start  : 368000
>   pfn_end    : 370000
> mem_map (110)
>   mem_map    : ffffea000c080000
>   pfn_start  : 370000
>   pfn_end    : 378000
> mem_map (111)
>   mem_map    : ffffea000c240000
>   pfn_start  : 378000
>   pfn_end    : 380000
> mem_map (112)
>   mem_map    : ffffea000c400000
>   pfn_start  : 380000
>   pfn_end    : 388000
> mem_map (113)
>   mem_map    : ffffea000c5c0000
>   pfn_start  : 388000
>   pfn_end    : 390000
> mem_map (114)
>   mem_map    : ffffea000c780000
>   pfn_start  : 390000
>   pfn_end    : 398000
> mem_map (115)
>   mem_map    : ffffea000c940000
>   pfn_start  : 398000
>   pfn_end    : 3a0000
> mem_map (116)
>   mem_map    : ffffea000cb00000
>   pfn_start  : 3a0000
>   pfn_end    : 3a8000
> mem_map (117)
>   mem_map    : ffffea000ccc0000
>   pfn_start  : 3a8000
>   pfn_end    : 3b0000
> mem_map (118)
>   mem_map    : ffffea000ce80000
>   pfn_start  : 3b0000
>   pfn_end    : 3b8000
> mem_map (119)
>   mem_map    : ffffea000d040000
>   pfn_start  : 3b8000
>   pfn_end    : 3c0000
> mem_map (120)
>   mem_map    : ffffea000d200000
>   pfn_start  : 3c0000
>   pfn_end    : 3c8000
> mem_map (121)
>   mem_map    : ffffea000d3c0000
>   pfn_start  : 3c8000
>   pfn_end    : 3d0000
> mem_map (122)
>   mem_map    : ffffea000d580000
>   pfn_start  : 3d0000
>   pfn_end    : 3d8000
> mem_map (123)
>   mem_map    : ffffea000d740000
>   pfn_start  : 3d8000
>   pfn_end    : 3e0000
> mem_map (124)
>   mem_map    : ffffea000d900000
>   pfn_start  : 3e0000
>   pfn_end    : 3e8000
> mem_map (125)
>   mem_map    : ffffea000dac0000
>   pfn_start  : 3e8000
>   pfn_end    : 3f0000
> mem_map (126)
>   mem_map    : ffffea000dc80000
>   pfn_start  : 3f0000
>   pfn_end    : 3f8000
> mem_map (127)
>   mem_map    : ffffea000de40000
>   pfn_start  : 3f8000
>   pfn_end    : 400000
> mem_map (128)
>   mem_map    : ffffea000e000000
>   pfn_start  : 400000
>   pfn_end    : 408000
> mem_map (129)
>   mem_map    : ffffea000e1c0000
>   pfn_start  : 408000
>   pfn_end    : 410000
> mem_map (130)
>   mem_map    : ffffea000e380000
>   pfn_start  : 410000
>   pfn_end    : 418000
> mem_map (131)
>   mem_map    : ffffea000e540000
>   pfn_start  : 418000
>   pfn_end    : 420000
> mem_map (132)
>   mem_map    : ffffea000e700000
>   pfn_start  : 420000
>   pfn_end    : 428000
> mem_map (133)
>   mem_map    : ffffea000e8c0000
>   pfn_start  : 428000
>   pfn_end    : 430000
> mem_map (134)
>   mem_map    : ffffea000ea80000
>   pfn_start  : 430000
>   pfn_end    : 438000
> mem_map (135)
>   mem_map    : ffffea000ec40000
>   pfn_start  : 438000
>   pfn_end    : 440000
> mem_map (136)
>   mem_map    : ffffea000ee00000
>   pfn_start  : 440000
>   pfn_end    : 448000
> mem_map (137)
>   mem_map    : ffffea000efc0000
>   pfn_start  : 448000
>   pfn_end    : 450000
> mem_map (138)
>   mem_map    : ffffea000f180000
>   pfn_start  : 450000
>   pfn_end    : 458000
> mem_map (139)
>   mem_map    : ffffea000f340000
>   pfn_start  : 458000
>   pfn_end    : 460000
> mem_map (140)
>   mem_map    : ffffea000f500000
>   pfn_start  : 460000
>   pfn_end    : 468000
> mem_map (141)
>   mem_map    : ffffea000f6c0000
>   pfn_start  : 468000
>   pfn_end    : 470000
> mem_map (142)
>   mem_map    : ffffea000f880000
>   pfn_start  : 470000
>   pfn_end    : 478000
> mem_map (143)
>   mem_map    : ffffea000fa40000
>   pfn_start  : 478000
>   pfn_end    : 480000
> mem_map (144)
>   mem_map    : ffffea000fc00000
>   pfn_start  : 480000
>   pfn_end    : 488000
> mem_map (145)
>   mem_map    : ffffea000fdc0000
>   pfn_start  : 488000
>   pfn_end    : 490000
> mem_map (146)
>   mem_map    : ffffea000ff80000
>   pfn_start  : 490000
>   pfn_end    : 498000
> mem_map (147)
>   mem_map    : ffffea0010140000
>   pfn_start  : 498000
>   pfn_end    : 4a0000
> mem_map (148)
>   mem_map    : ffffea0010300000
>   pfn_start  : 4a0000
>   pfn_end    : 4a8000
> mem_map (149)
>   mem_map    : ffffea00104c0000
>   pfn_start  : 4a8000
>   pfn_end    : 4b0000
> mem_map (150)
>   mem_map    : ffffea0010680000
>   pfn_start  : 4b0000
>   pfn_end    : 4b8000
> mem_map (151)
>   mem_map    : ffffea0010840000
>   pfn_start  : 4b8000
>   pfn_end    : 4c0000
> mem_map (152)
>   mem_map    : ffffea0010a00000
>   pfn_start  : 4c0000
>   pfn_end    : 4c8000
> mem_map (153)
>   mem_map    : ffffea0010bc0000
>   pfn_start  : 4c8000
>   pfn_end    : 4d0000
> mem_map (154)
>   mem_map    : ffffea0010d80000
>   pfn_start  : 4d0000
>   pfn_end    : 4d8000
> mem_map (155)
>   mem_map    : ffffea0010f40000
>   pfn_start  : 4d8000
>   pfn_end    : 4e0000
> mem_map (156)
>   mem_map    : ffffea0011100000
>   pfn_start  : 4e0000
>   pfn_end    : 4e8000
> mem_map (157)
>   mem_map    : ffffea00112c0000
>   pfn_start  : 4e8000
>   pfn_end    : 4f0000
> mem_map (158)
>   mem_map    : ffffea0011480000
>   pfn_start  : 4f0000
>   pfn_end    : 4f8000
> mem_map (159)
>   mem_map    : ffffea0011640000
>   pfn_start  : 4f8000
>   pfn_end    : 500000
> mem_map (160)
>   mem_map    : ffffea0011800000
>   pfn_start  : 500000
>   pfn_end    : 508000
> mem_map (161)
>   mem_map    : ffffea00119c0000
>   pfn_start  : 508000
>   pfn_end    : 510000
> mem_map (162)
>   mem_map    : ffffea0011b80000
>   pfn_start  : 510000
>   pfn_end    : 518000
> mem_map (163)
>   mem_map    : ffffea0011d40000
>   pfn_start  : 518000
>   pfn_end    : 520000
> mem_map (164)
>   mem_map    : ffffea0011f00000
>   pfn_start  : 520000
>   pfn_end    : 528000
> mem_map (165)
>   mem_map    : ffffea00120c0000
>   pfn_start  : 528000
>   pfn_end    : 530000
> mem_map (166)
>   mem_map    : ffffea0012280000
>   pfn_start  : 530000
>   pfn_end    : 538000
> mem_map (167)
>   mem_map    : ffffea0012440000
>   pfn_start  : 538000
>   pfn_end    : 540000
> mem_map (168)
>   mem_map    : ffffea0012600000
>   pfn_start  : 540000
>   pfn_end    : 548000
> mem_map (169)
>   mem_map    : ffffea00127c0000
>   pfn_start  : 548000
>   pfn_end    : 550000
> mem_map (170)
>   mem_map    : ffffea0012980000
>   pfn_start  : 550000
>   pfn_end    : 558000
> mem_map (171)
>   mem_map    : ffffea0012b40000
>   pfn_start  : 558000
>   pfn_end    : 560000
> mem_map (172)
>   mem_map    : ffffea0012d00000
>   pfn_start  : 560000
>   pfn_end    : 568000
> mem_map (173)
>   mem_map    : ffffea0012ec0000
>   pfn_start  : 568000
>   pfn_end    : 570000
> mem_map (174)
>   mem_map    : ffffea0013080000
>   pfn_start  : 570000
>   pfn_end    : 578000
> mem_map (175)
>   mem_map    : ffffea0013240000
>   pfn_start  : 578000
>   pfn_end    : 580000
> mem_map (176)
>   mem_map    : ffffea0013400000
>   pfn_start  : 580000
>   pfn_end    : 588000
> mem_map (177)
>   mem_map    : ffffea00135c0000
>   pfn_start  : 588000
>   pfn_end    : 590000
> mem_map (178)
>   mem_map    : ffffea0013780000
>   pfn_start  : 590000
>   pfn_end    : 598000
> mem_map (179)
>   mem_map    : ffffea0013940000
>   pfn_start  : 598000
>   pfn_end    : 5a0000
> mem_map (180)
>   mem_map    : ffffea0013b00000
>   pfn_start  : 5a0000
>   pfn_end    : 5a8000
> mem_map (181)
>   mem_map    : ffffea0013cc0000
>   pfn_start  : 5a8000
>   pfn_end    : 5b0000
> mem_map (182)
>   mem_map    : ffffea0013e80000
>   pfn_start  : 5b0000
>   pfn_end    : 5b8000
> mem_map (183)
>   mem_map    : ffffea0014040000
>   pfn_start  : 5b8000
>   pfn_end    : 5c0000
> mem_map (184)
>   mem_map    : ffffea0014200000
>   pfn_start  : 5c0000
>   pfn_end    : 5c8000
> mem_map (185)
>   mem_map    : ffffea00143c0000
>   pfn_start  : 5c8000
>   pfn_end    : 5d0000
> mem_map (186)
>   mem_map    : ffffea0014580000
>   pfn_start  : 5d0000
>   pfn_end    : 5d8000
> mem_map (187)
>   mem_map    : ffffea0014740000
>   pfn_start  : 5d8000
>   pfn_end    : 5e0000
> mem_map (188)
>   mem_map    : ffffea0014900000
>   pfn_start  : 5e0000
>   pfn_end    : 5e8000
> mem_map (189)
>   mem_map    : ffffea0014ac0000
>   pfn_start  : 5e8000
>   pfn_end    : 5f0000
> mem_map (190)
>   mem_map    : ffffea0014c80000
>   pfn_start  : 5f0000
>   pfn_end    : 5f8000
> mem_map (191)
>   mem_map    : ffffea0014e40000
>   pfn_start  : 5f8000
>   pfn_end    : 600000
> mem_map (192)
>   mem_map    : ffffea0015000000
>   pfn_start  : 600000
>   pfn_end    : 608000
> mem_map (193)
>   mem_map    : ffffea00151c0000
>   pfn_start  : 608000
>   pfn_end    : 610000
> mem_map (194)
>   mem_map    : ffffea0015380000
>   pfn_start  : 610000
>   pfn_end    : 618000
> mem_map (195)
>   mem_map    : ffffea0015540000
>   pfn_start  : 618000
>   pfn_end    : 620000
> mem_map (196)
>   mem_map    : ffffea0015700000
>   pfn_start  : 620000
>   pfn_end    : 628000
> mem_map (197)
>   mem_map    : ffffea00158c0000
>   pfn_start  : 628000
>   pfn_end    : 630000
> mem_map (198)
>   mem_map    : ffffea0015a80000
>   pfn_start  : 630000
>   pfn_end    : 638000
> mem_map (199)
>   mem_map    : ffffea0015c40000
>   pfn_start  : 638000
>   pfn_end    : 640000
> mem_map (200)
>   mem_map    : ffffea0015e00000
>   pfn_start  : 640000
>   pfn_end    : 648000
> mem_map (201)
>   mem_map    : ffffea0015fc0000
>   pfn_start  : 648000
>   pfn_end    : 650000
> mem_map (202)
>   mem_map    : ffffea0016180000
>   pfn_start  : 650000
>   pfn_end    : 658000
> mem_map (203)
>   mem_map    : ffffea0016340000
>   pfn_start  : 658000
>   pfn_end    : 660000
> mem_map (204)
>   mem_map    : ffffea0016500000
>   pfn_start  : 660000
>   pfn_end    : 668000
> mem_map (205)
>   mem_map    : ffffea00166c0000
>   pfn_start  : 668000
>   pfn_end    : 670000
> mem_map (206)
>   mem_map    : ffffea0016880000
>   pfn_start  : 670000
>   pfn_end    : 678000
> mem_map (207)
>   mem_map    : ffffea0016a40000
>   pfn_start  : 678000
>   pfn_end    : 680000
> mem_map (208)
>   mem_map    : ffffea0016c00000
>   pfn_start  : 680000
>   pfn_end    : 688000
> mem_map (209)
>   mem_map    : ffffea0016dc0000
>   pfn_start  : 688000
>   pfn_end    : 690000
> mem_map (210)
>   mem_map    : ffffea0016f80000
>   pfn_start  : 690000
>   pfn_end    : 698000
> mem_map (211)
>   mem_map    : ffffea0017140000
>   pfn_start  : 698000
>   pfn_end    : 6a0000
> mem_map (212)
>   mem_map    : ffffea0017300000
>   pfn_start  : 6a0000
>   pfn_end    : 6a8000
> mem_map (213)
>   mem_map    : ffffea00174c0000
>   pfn_start  : 6a8000
>   pfn_end    : 6b0000
> mem_map (214)
>   mem_map    : ffffea0017680000
>   pfn_start  : 6b0000
>   pfn_end    : 6b8000
> mem_map (215)
>   mem_map    : ffffea0017840000
>   pfn_start  : 6b8000
>   pfn_end    : 6c0000
> mem_map (216)
>   mem_map    : ffffea0017a00000
>   pfn_start  : 6c0000
>   pfn_end    : 6c8000
> mem_map (217)
>   mem_map    : ffffea0017bc0000
>   pfn_start  : 6c8000
>   pfn_end    : 6d0000
> mem_map (218)
>   mem_map    : ffffea0017d80000
>   pfn_start  : 6d0000
>   pfn_end    : 6d8000
> mem_map (219)
>   mem_map    : ffffea0017f40000
>   pfn_start  : 6d8000
>   pfn_end    : 6e0000
> mem_map (220)
>   mem_map    : ffffea0018100000
>   pfn_start  : 6e0000
>   pfn_end    : 6e8000
> mem_map (221)
>   mem_map    : ffffea00182c0000
>   pfn_start  : 6e8000
>   pfn_end    : 6f0000
> mem_map (222)
>   mem_map    : ffffea0018480000
>   pfn_start  : 6f0000
>   pfn_end    : 6f8000
> mem_map (223)
>   mem_map    : ffffea0018640000
>   pfn_start  : 6f8000
>   pfn_end    : 700000
> mem_map (224)
>   mem_map    : ffffea0018800000
>   pfn_start  : 700000
>   pfn_end    : 708000
> mem_map (225)
>   mem_map    : ffffea00189c0000
>   pfn_start  : 708000
>   pfn_end    : 710000
> mem_map (226)
>   mem_map    : ffffea0018b80000
>   pfn_start  : 710000
>   pfn_end    : 718000
> mem_map (227)
>   mem_map    : ffffea0018d40000
>   pfn_start  : 718000
>   pfn_end    : 720000
> mem_map (228)
>   mem_map    : ffffea0018f00000
>   pfn_start  : 720000
>   pfn_end    : 728000
> mem_map (229)
>   mem_map    : ffffea00190c0000
>   pfn_start  : 728000
>   pfn_end    : 730000
> mem_map (230)
>   mem_map    : ffffea0019280000
>   pfn_start  : 730000
>   pfn_end    : 738000
> mem_map (231)
>   mem_map    : ffffea0019440000
>   pfn_start  : 738000
>   pfn_end    : 740000
> mem_map (232)
>   mem_map    : ffffea0019600000
>   pfn_start  : 740000
>   pfn_end    : 748000
> mem_map (233)
>   mem_map    : ffffea00197c0000
>   pfn_start  : 748000
>   pfn_end    : 750000
> mem_map (234)
>   mem_map    : ffffea0019980000
>   pfn_start  : 750000
>   pfn_end    : 758000
> mem_map (235)
>   mem_map    : ffffea0019b40000
>   pfn_start  : 758000
>   pfn_end    : 760000
> mem_map (236)
>   mem_map    : ffffea0019d00000
>   pfn_start  : 760000
>   pfn_end    : 768000
> mem_map (237)
>   mem_map    : ffffea0019ec0000
>   pfn_start  : 768000
>   pfn_end    : 770000
> mem_map (238)
>   mem_map    : ffffea001a080000
>   pfn_start  : 770000
>   pfn_end    : 778000
> mem_map (239)
>   mem_map    : ffffea001a240000
>   pfn_start  : 778000
>   pfn_end    : 780000
> mem_map (240)
>   mem_map    : ffffea001a400000
>   pfn_start  : 780000
>   pfn_end    : 788000
> mem_map (241)
>   mem_map    : ffffea001a5c0000
>   pfn_start  : 788000
>   pfn_end    : 790000
> mem_map (242)
>   mem_map    : ffffea001a780000
>   pfn_start  : 790000
>   pfn_end    : 798000
> mem_map (243)
>   mem_map    : ffffea001a940000
>   pfn_start  : 798000
>   pfn_end    : 7a0000
> mem_map (244)
>   mem_map    : ffffea001ab00000
>   pfn_start  : 7a0000
>   pfn_end    : 7a8000
> mem_map (245)
>   mem_map    : ffffea001acc0000
>   pfn_start  : 7a8000
>   pfn_end    : 7b0000
> mem_map (246)
>   mem_map    : ffffea001ae80000
>   pfn_start  : 7b0000
>   pfn_end    : 7b8000
> mem_map (247)
>   mem_map    : ffffea001b040000
>   pfn_start  : 7b8000
>   pfn_end    : 7c0000
> mem_map (248)
>   mem_map    : ffffea001b200000
>   pfn_start  : 7c0000
>   pfn_end    : 7c8000
> mem_map (249)
>   mem_map    : ffffea001b3c0000
>   pfn_start  : 7c8000
>   pfn_end    : 7d0000
> mem_map (250)
>   mem_map    : ffffea001b580000
>   pfn_start  : 7d0000
>   pfn_end    : 7d8000
> mem_map (251)
>   mem_map    : ffffea001b740000
>   pfn_start  : 7d8000
>   pfn_end    : 7e0000
> mem_map (252)
>   mem_map    : ffffea001b900000
>   pfn_start  : 7e0000
>   pfn_end    : 7e8000
> mem_map (253)
>   mem_map    : ffffea001bac0000
>   pfn_start  : 7e8000
>   pfn_end    : 7f0000
> mem_map (254)
>   mem_map    : ffffea001bc80000
>   pfn_start  : 7f0000
>   pfn_end    : 7f8000
> mem_map (255)
>   mem_map    : ffffea001be40000
>   pfn_start  : 7f8000
>   pfn_end    : 800000
> mem_map (256)
>   mem_map    : ffffea001c000000
>   pfn_start  : 800000
>   pfn_end    : 808000
> mem_map (257)
>   mem_map    : ffffea001c1c0000
>   pfn_start  : 808000
>   pfn_end    : 810000
> mem_map (258)
>   mem_map    : ffffea001c380000
>   pfn_start  : 810000
>   pfn_end    : 818000
> mem_map (259)
>   mem_map    : ffffea001c540000
>   pfn_start  : 818000
>   pfn_end    : 820000
> mem_map (260)
>   mem_map    : ffffea001c700000
>   pfn_start  : 820000
>   pfn_end    : 828000
> mem_map (261)
>   mem_map    : ffffea001c8c0000
>   pfn_start  : 828000
>   pfn_end    : 830000
> mem_map (262)
>   mem_map    : ffffea001ca80000
>   pfn_start  : 830000
>   pfn_end    : 838000
> mem_map (263)
>   mem_map    : ffffea001cc40000
>   pfn_start  : 838000
>   pfn_end    : 840000
> mem_map (264)
>   mem_map    : ffffea001ce00000
>   pfn_start  : 840000
>   pfn_end    : 848000
> mem_map (265)
>   mem_map    : ffffea001cfc0000
>   pfn_start  : 848000
>   pfn_end    : 850000
> mem_map (266)
>   mem_map    : ffffea001d180000
>   pfn_start  : 850000
>   pfn_end    : 858000
> mem_map (267)
>   mem_map    : ffffea001d340000
>   pfn_start  : 858000
>   pfn_end    : 860000
> mem_map
Excluding unnecessary pages        : [  0 %] 
Excluding unnecessary pages        : [100 %] 
Excluding unnecessary pages        : [  0 %] 
Excluding unnecessary pages        : [100 %] 
>  (268)
>   mem_map    : ffffea001d500000
>   pfn_start  : 860000
>   pfn_end    : 868000
> mem_map (269)
>   mem_map    : ffffea001d6c0000
>   pfn_start  : 868000
>   pfn_end    : 870000
> mem_map (270)
>   mem_map    : ffffea001d880000
>   pfn_start  : 870000
>   pfn_end    : 878000
> mem_map (271)
>   mem_map    : ffffea001da40000
>   pfn_start  : 878000
>   pfn_end    : 880000
> STEP [Excluding unnecessary pages] : 0.438936 seconds
> STEP [Excluding unnecessary pages] : 0.467304 seconds
> STEP [Copying data               ] : 0.624328 seconds
> Writing erase info...
> offset_eraseinfo: 6c0eb8, size_eraseinfo: 0
> 
> Original pages  : 0x00000000007ec289
>   Excluded pages   : 0x00000000007b9efb
>     Pages filled with zero  : 0x0000000000000000
>     Cache pages             : 0x0000000000012cff
>     Cache pages + private   : 0x0000000000012015
>     User process data pages : 0x0000000000001ece
>     Free pages              : 0x0000000000793319
>   Remaining pages  : 0x000000000003238e
>   (The number of pages is reduced to 2%.)
> Memory Hole     : 0x0000000000093d77
> --------------------------------------------------
> Total pages     : 0x0000000000880000
> 
> 
> The dumpfile is saved to /mnt///127.0.0.1-2012-12-07-16:45:20/vmcore-empty-2.
> 
> makedumpfile Completed.
> [ perf record: Woken up 3 times to write data ]
> [ perf record: Captured and wrote 0.598 MB /mnt///127.0.0.1-2012-12-07-16:45:20/perf.data2 (~26144 samples) ]

> Failed to open [ext4], continuing without symbols
> No kallsyms or vmlinux with build-id a77a2293aab413880b8b361bb5b863a1680c8eab was found
> [qla2xxx] with build id a77a2293aab413880b8b361bb5b863a1680c8eab not found, continuing without symbols
> Failed to open [dm_mirror], continuing without symbols
> No kallsyms or vmlinux with build-id 8e4a472eadb14fb0cde985ef8571b543880472dd was found
> [megaraid_sas] with build id 8e4a472eadb14fb0cde985ef8571b543880472dd not found, continuing without symbols
> No kallsyms or vmlinux with build-id 93346fc362be38e207aeaae310a339fb502d9acb was found
> [jbd2] with build id 93346fc362be38e207aeaae310a339fb502d9acb not found, continuing without symbols
> # ========
> # captured on: Fri Dec  7 16:45:43 2012
> # hostname : (none)
> # os release : 3.7.0-rc8-cliff-bench+
> # perf version : 3.7.rc8.3.ge0fb22
> # arch : x86_64
> # nrcpus online : 1
> # nrcpus avail : 1
> # cpudesc : Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz
> # cpuid : GenuineIntel,6,47,2
> # total memory : 240104 kB
> # cmdline : /var/crash/perf record -g -o /mnt///127.0.0.1-2012-12-07-16:45:20/perf.data1 /var/crash/makedumpfile-cliff --message-level 31 -i /var/crash/vmcoreinfo.txt -N -o -d 31 /proc/vmcore /mnt///127.0.0.1-2012-12-07-16:45:20/vmcore-empty-1 
> # event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, id = { 2 }
> # HEADER_CPU_TOPOLOGY info available, use -I to display
> # HEADER_NUMA_TOPOLOGY info available, use -I to display
> # pmu mappings: cpu = 4, software = 1, tracepoint = 2, uncore_bbox_0 = 15, uncore_bbox_1 = 16, uncore_cbox_0 = 7, uncore_cbox_1 = 8, uncore_cbox_2 = 9, uncore_cbox_3 = 10, uncore_cbox_4 = 11, uncore_cbox_5 = 12, uncore_cbox_6 = 13, uncore_cbox_7 = 14, uncore_mbox_0 = 19, uncore_mbox_1 = 20, uncore_rbox_0 = 21, uncore_rbox_1 = 22, uncore_sbox_0 = 17, uncore_sbox_1 = 18, breakpoint = 5, uncore_ubox = 6, uncore_wbox = 23
> # ========
> #
> # Samples: 91K of event 'cycles'
> # Event count (approx.): 23676246537
> #
> # Overhead          Command         Shared Object
> # ........  ...............  ....................
> #
>     90.38%  makedumpfile-cl  [kernel.kallsyms]   
>             |          
>             |--19.73%-- __purge_vmap_area_lazy
>             |          |          
>             |          |--80.50%-- free_vmap_area_noflush
>             |          |          free_unmap_vmap_area
>             |          |          remove_vm_area
>             |          |          iounmap
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          |          
>             |          |          |--66.59%-- 0x0
>             |          |          |          
>             |          |           --33.41%-- 0x45524f4300000001
>             |          |          
>             |          |--19.43%-- vm_unmap_aliases
>             |          |          change_page_attr_set_clr
>             |          |          _set_memory_wb
>             |          |          ioremap_change_attr
>             |          |          kernel_map_sync_memtype
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |           --0.07%-- [...]
>             |          
>             |--15.58%-- try_preserve_large_page
>             |          |          
>             |          |--99.97%-- __change_page_attr
>             |          |          __change_page_attr_set_clr
>             |          |          change_page_attr_set_clr
>             |          |          _set_memory_wb
>             |          |          ioremap_change_attr
>             |          |          kernel_map_sync_memtype
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |           --0.03%-- [...]
>             |          
>             |--14.51%-- iomem_map_sanity_check
>             |          |          
>             |          |--99.97%-- __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          |          
>             |          |          |--50.01%-- 0x6e6d2f0065726f63
>             |          |          |          
>             |          |           --49.99%-- 0x45524f4300000001
>             |           --0.03%-- [...]
>             |          
>             |--11.06%-- walk_system_ram_range
>             |          |          
>             |          |--64.71%-- pat_pagerange_is_ram
>             |          |          |          
>             |          |          |--50.87%-- free_memtype
>             |          |          |          iounmap
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |           --49.13%-- reserve_memtype
>             |          |                     __ioremap_caller
>             |          |                     ioremap_cache
>             |          |                     copy_oldmem_page
>             |          |                     read_from_oldmem
>             |          |                     read_vmcore
>             |          |                     proc_reg_read
>             |          |                     vfs_read
>             |          |                     sys_read
>             |          |                     system_call_fastpath
>             |          |                     __read_nocancel
>             |          |          
>             |          |--34.75%-- page_is_ram
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |           --0.54%-- [...]
>             |          
>             |--4.84%-- __phys_addr
>             |          |          
>             |          |--52.48%-- __change_page_attr
>             |          |          __change_page_attr_set_clr
>             |          |          change_page_attr_set_clr
>             |          |          _set_memory_wb
>             |          |          ioremap_change_attr
>             |          |          kernel_map_sync_memtype
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |          |--45.72%-- try_preserve_large_page
>             |          |          __change_page_attr
>             |          |          __change_page_attr_set_clr
>             |          |          change_page_attr_set_clr
>             |          |          _set_memory_wb
>             |          |          ioremap_change_attr
>             |          |          kernel_map_sync_memtype
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |           --1.80%-- [...]
>             |          
>             |--4.82%-- __get_vm_area_node
>             |          |          
>             |          |--99.70%-- get_vm_area_caller
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          |          
>             |          |           --100.00%-- 0x0
>             |           --0.30%-- [...]
>             |          
>             |--4.13%-- iounmap
>             |          |          
>             |          |--99.59%-- copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          |          
>             |          |          |--50.39%-- 0x45524f4300000001
>             |          |          |          
>             |          |           --49.61%-- 0x0
>             |           --0.41%-- [...]
>             |          
>             |--3.60%-- read_vmcore
>             |          |          
>             |          |--99.83%-- proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |           --0.17%-- [...]
>             |          
>             |--2.24%-- copy_user_generic_string
>             |          read_from_oldmem
>             |          read_vmcore
>             |          proc_reg_read
>             |          vfs_read
>             |          sys_read
>             |          system_call_fastpath
>             |          __read_nocancel
>             |          
>             |--1.92%-- remove_vm_area
>             |          |          
>             |          |--99.31%-- iounmap
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          |          
>             |          |           --100.00%-- 0x0
>             |          |          
>             |           --0.69%-- copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--1.82%-- do_kernel_range_flush
>             |          |          
>             |          |--99.20%-- on_each_cpu
>             |          |          flush_tlb_kernel_range
>             |          |          __purge_vmap_area_lazy
>             |          |          free_vmap_area_noflush
>             |          |          free_unmap_vmap_area
>             |          |          remove_vm_area
>             |          |          iounmap
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |           --0.80%-- flush_tlb_kernel_range
>             |                     __purge_vmap_area_lazy
>             |                     free_vmap_area_noflush
>             |                     free_unmap_vmap_area
>             |                     remove_vm_area
>             |                     iounmap
>             |                     copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--1.01%-- rbt_memtype_erase
>             |          |          
>             |          |--96.66%-- free_memtype
>             |          |          iounmap
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |           --3.34%-- iounmap
>             |                     copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--0.87%-- kfree
>             |          |          
>             |          |--55.52%-- free_memtype
>             |          |          iounmap
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |          |--29.00%-- iounmap
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |          |--12.42%-- rcu_do_batch
>             |          |          __rcu_process_callbacks
>             |          |          rcu_process_callbacks
>             |          |          __do_softirq
>             |          |          call_softirq
>             |          |          do_softirq
>             |          |          irq_exit
>             |          |          smp_apic_timer_interrupt
>             |          |          apic_timer_interrupt
>             |          |          |          
>             |          |          |--20.23%-- free_vmap_area_noflush
>             |          |          |          free_unmap_vmap_area
>             |          |          |          remove_vm_area
>             |          |          |          iounmap
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--19.11%-- __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--16.82%-- __change_page_attr
>             |          |          |          __change_page_attr_set_clr
>             |          |          |          change_page_attr_set_clr
>             |          |          |          _set_memory_wb
>             |          |          |          ioremap_change_attr
>             |          |          |          kernel_map_sync_memtype
>             |          |          |          __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--6.75%-- pat_pagerange_is_ram
>             |          |          |          |          
>             |          |          |          |--66.67%-- reserve_memtype
>             |          |          |          |          __ioremap_caller
>             |          |          |          |          ioremap_cache
>             |          |          |          |          copy_oldmem_page
>             |          |          |          |          read_from_oldmem
>             |          |          |          |          read_vmcore
>             |          |          |          |          proc_reg_read
>             |          |          |          |          vfs_read
>             |          |          |          |          sys_read
>             |          |          |          |          system_call_fastpath
>             |          |          |          |          __read_nocancel
>             |          |          |          |          
>             |          |          |           --33.33%-- free_memtype
>             |          |          |                     iounmap
>             |          |          |                     copy_oldmem_page
>             |          |          |                     read_from_oldmem
>             |          |          |                     read_vmcore
>             |          |          |                     proc_reg_read
>             |          |          |                     vfs_read
>             |          |          |                     sys_read
>             |          |          |                     system_call_fastpath
>             |          |          |                     __read_nocancel
>             |          |          |          
>             |          |          |--5.62%-- vm_unmap_aliases
>             |          |          |          change_page_attr_set_clr
>             |          |          |          _set_memory_wb
>             |          |          |          ioremap_change_attr
>             |          |          |          kernel_map_sync_memtype
>             |          |          |          __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--4.50%-- proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--4.49%-- copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--3.37%-- get_vm_area_caller
>             |          |          |          __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--3.37%-- readmem
>             |          |          |          
>             |          |          |--2.25%-- page_is_ram
>             |          |          |          __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--2.25%-- __exclude_unnecessary_pages
>             |          |          |          exclude_unnecessary_pages_cyclic
>             |          |          |          writeout_dumpfile
>             |          |          |          0x7fff6c26de6d
>             |          |          |          0x64656b616d2f6873
>             |          |          |          
>             |          |          |--2.25%-- paddr_to_offset
>             |          |          |          exclude_unnecessary_pages_cyclic
>             |          |          |          |          
>             |          |          |           --100.00%-- writeout_dumpfile
>             |          |          |                     0x7fff6c26de6d
>             |          |          |                     0x64656b616d2f6873
>             |          |          |          
>             |          |          |--1.13%-- try_preserve_large_page
>             |          |          |          __change_page_attr
>             |          |          |          __change_page_attr_set_clr
>             |          |          |          change_page_attr_set_clr
>             |          |          |          _set_memory_wb
>             |          |          |          ioremap_change_attr
>             |          |          |          kernel_map_sync_memtype
>             |          |          |          __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--1.12%-- reserve_memtype
>             |          |          |          __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--1.12%-- __lseek_nocancel
>             |          |          |          
>             |          |          |--1.12%-- proc_reg_llseek
>             |          |          |          vfs_llseek
>             |          |          |          sys_lseek
>             |          |          |          system_call_fastpath
>             |          |          |          __lseek_nocancel
>             |          |          |          
>             |          |          |--1.12%-- free_memtype
>             |          |          |          iounmap
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--1.12%-- on_each_cpu
>             |          |          |          flush_tlb_kernel_range
>             |          |          |          __purge_vmap_area_lazy
>             |          |          |          free_vmap_area_noflush
>             |          |          |          free_unmap_vmap_area
>             |          |          |          remove_vm_area
>             |          |          |          iounmap
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |          |--1.12%-- __get_vm_area_node
>             |          |          |          get_vm_area_caller
>             |          |          |          __ioremap_caller
>             |          |          |          ioremap_cache
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          
>             |          |           --1.12%-- vtop4_x86_64
>             |          |          
>             |          |--2.09%-- copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |           --0.98%-- __rcu_process_callbacks
>             |                     rcu_process_callbacks
>             |                     __do_softirq
>             |                     call_softirq
>             |                     do_softirq
>             |                     irq_exit
>             |                     smp_apic_timer_interrupt
>             |                     apic_timer_interrupt
>             |                     |          
>             |                     |--57.11%-- __change_page_attr
>             |                     |          __change_page_attr_set_clr
>             |                     |          change_page_attr_set_clr
>             |                     |          _set_memory_wb
>             |                     |          ioremap_change_attr
>             |                     |          kernel_map_sync_memtype
>             |                     |          __ioremap_caller
>             |                     |          ioremap_cache
>             |                     |          copy_oldmem_page
>             |                     |          read_from_oldmem
>             |                     |          read_vmcore
>             |                     |          proc_reg_read
>             |                     |          vfs_read
>             |                     |          sys_read
>             |                     |          system_call_fastpath
>             |                     |          __read_nocancel
>             |                     |          
>             |                     |--14.30%-- flush_tlb_kernel_range
>             |                     |          __purge_vmap_area_lazy
>             |                     |          free_vmap_area_noflush
>             |                     |          free_unmap_vmap_area
>             |                     |          remove_vm_area
>             |                     |          iounmap
>             |                     |          copy_oldmem_page
>             |                     |          read_from_oldmem
>             |                     |          read_vmcore
>             |                     |          proc_reg_read
>             |                     |          vfs_read
>             |                     |          sys_read
>             |                     |          system_call_fastpath
>             |                     |          __read_nocancel
>             |                     |          
>             |                     |--14.30%-- free_vmap_area_noflush
>             |                     |          free_unmap_vmap_area
>             |                     |          remove_vm_area
>             |                     |          iounmap
>             |                     |          copy_oldmem_page
>             |                     |          read_from_oldmem
>             |                     |          read_vmcore
>             |                     |          proc_reg_read
>             |                     |          vfs_read
>             |                     |          sys_read
>             |                     |          system_call_fastpath
>             |                     |          __read_nocancel
>             |                     |          
>             |                      --14.30%-- copy_oldmem_page
>             |                                read_from_oldmem
>             |                                read_vmcore
>             |                                proc_reg_read
>             |                                vfs_read
>             |                                sys_read
>             |                                system_call_fastpath
>             |                                __read_nocancel
>             |          
>             |--0.58%-- __insert_vmap_area
>             |          |          
>             |          |--98.32%-- alloc_vmap_area
>             |          |          __get_vm_area_node
>             |          |          get_vm_area_caller
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |           --1.68%-- __get_vm_area_node
>             |                     get_vm_area_caller
>             |                     __ioremap_caller
>             |                     ioremap_cache
>             |                     copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--0.57%-- kmem_cache_alloc_node_trace
>             |          |          
>             |          |--55.58%-- __get_vm_area_node
>             |          |          get_vm_area_caller
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |          |--41.85%-- alloc_vmap_area
>             |          |          __get_vm_area_node
>             |          |          get_vm_area_caller
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |           --2.56%-- get_vm_area_caller
>             |                     __ioremap_caller
>             |                     ioremap_cache
>             |                     copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--0.54%-- vfs_read
>             |          |          
>             |          |--98.21%-- sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          |          
>             |          |           --100.00%-- 0x0
>             |          |          
>             |           --1.79%-- system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--0.54%-- memtype_rb_check_conflict
>             |          |          
>             |          |--95.54%-- rbt_memtype_check_insert
>             |          |          reserve_memtype
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |           --4.46%-- reserve_memtype
>             |                     __ioremap_caller
>             |                     ioremap_cache
>             |                     copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--0.51%-- __mtrr_type_lookup
>             |          |          
>             |          |--96.91%-- mtrr_type_lookup
>             |          |          reserve_memtype
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          
>             |           --3.09%-- reserve_memtype
>             |                     __ioremap_caller
>             |                     ioremap_cache
>             |                     copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>              --11.15%-- [...]
> 
>      9.19%  makedumpfile-cl  makedumpfile-cliff  
>             |          
>             |--37.97%-- __exclude_unnecessary_pages
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff6c26de6d
>             |                     0x64656b616d2f6873
>             |          
>             |--18.73%-- set_bitmap_cyclic
>             |          |          
>             |           --100.00%-- exclude_unnecessary_pages_cyclic
>             |                     |          
>             |                      --100.00%-- writeout_dumpfile
>             |                                0x7fff6c26de6d
>             |                                0x64656b616d2f6873
>             |          
>             |--13.79%-- paddr_to_offset
>             |          |          
>             |           --100.00%-- exclude_unnecessary_pages_cyclic
>             |                     |          
>             |                      --100.00%-- writeout_dumpfile
>             |                                0x7fff6c26de6d
>             |                                0x64656b616d2f6873
>             |          
>             |--4.71%-- readmem
>             |          |          
>             |           --100.00%-- exclude_unnecessary_pages_cyclic
>             |                     writeout_dumpfile
>             |                     0x7fff6c26de6d
>             |                     0x64656b616d2f6873
>             |          
>             |--4.12%-- get_num_dumpable_cyclic
>             |          
>             |--2.90%-- is_in_same_page
>             |          
>             |--2.66%-- page_is_buddy_v3
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff6c26de6d
>             |                     0x64656b616d2f6873
>             |          
>             |--2.37%-- clear_bit_on_2nd_bitmap_for_kernel
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff6c26de6d
>             |                     0x64656b616d2f6873
>             |          
>             |--2.35%-- create_1st_bitmap_cyclic
>             |          
>             |--2.04%-- is_xen_memory
>             |          |          
>             |           --100.00%-- exclude_unnecessary_pages_cyclic
>             |                     |          
>             |                      --100.00%-- writeout_dumpfile
>             |                                0x7fff6c26de6d
>             |                                0x64656b616d2f6873
>             |          
>             |--2.02%-- vtop4_x86_64
>             |          
>             |--1.67%-- set_bit_on_1st_bitmap
>             |          
>             |--1.53%-- clear_bit_on_2nd_bitmap
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff6c26de6d
>             |                     0x64656b616d2f6873
>             |          
>             |--1.25%-- write_kdump_pages_and_bitmap_cyclic
>             |          writeout_dumpfile
>             |          0x7fff6c26de6d
>             |          0x64656b616d2f6873
>             |          
>             |--1.15%-- update_cyclic_region
>             |          
>             |--0.66%-- vaddr_to_paddr_x86_64
>              --0.07%-- [...]
> 
>      0.42%  makedumpfile-cl  libc.so.6           
>             |          
>             |--40.25%-- __lseek_nocancel
>             |          
>             |--36.91%-- __read_nocancel
>             |          
>             |--8.64%-- __GI___libc_read
>             |          
>             |--5.01%-- __memset_sse2
>             |          |          
>             |           --100.00%-- 0x7fff6c26de6d
>             |                     0x64656b616d2f6873
>             |          
>             |--3.90%-- __GI___libc_lseek64
>             |          
>             |--3.72%-- memcpy
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff6c26de6d
>             |                     0x64656b616d2f6873
>             |          
>             |--0.53%-- memchr
>              --1.04%-- [...]
> 
>      0.01%  makedumpfile-cl  ld-linux-x86-64.so.2
>             |          
>             |--49.69%-- strcmp
>             |          |          
>             |          |--67.41%-- 0x8922974
>             |          |          0x42494c4700342e33
>             |          |          
>             |           --32.59%-- 0x9691a75
>             |                     0x42494c4700352e32
>             |          
>             |--17.46%-- _dl_name_match_p
>             |          _dl_check_all_versions
>             |          
>             |--16.51%-- do_lookup_x
>             |          
>              --16.34%-- _dl_lookup_symbol_x
>                        _dl_relocate_object
>                        dl_main
>                        _dl_sysdep_start
>                        0x4156415741e58948
> 
>      0.00%  makedumpfile-cl  libstdc++.so.6      
>             |
>             --- 0x37282bb470
>                 0x3728253b43
>                 0x7fff6c26de6d
>                 0x64656b616d2f6873
> 
>      0.00%  makedumpfile-cl  [megaraid_sas]      
>             |
>             --- megasas_isr
>                 handle_irq_event_percpu
>                 handle_irq_event
>                 handle_edge_irq
>                 handle_irq
>                 do_IRQ
>                 ret_from_intr
>                 __exclude_unnecessary_pages
>                 exclude_unnecessary_pages_cyclic
>                 writeout_dumpfile
>                 0x7fff6c26de6d
>                 0x64656b616d2f6873
> 
>      0.00%  makedumpfile-cl  [qla2xxx]           
>             |
>             --- qla24xx_mbx_completion
>                 qla24xx_msix_default
>                 handle_irq_event_percpu
>                 handle_irq_event
>                 handle_edge_irq
>                 handle_irq
>                 do_IRQ
>                 ret_from_intr
>                 __change_page_attr
>                 __change_page_attr_set_clr
>                 change_page_attr_set_clr
>                 _set_memory_wb
>                 ioremap_change_attr
>                 kernel_map_sync_memtype
>                 __ioremap_caller
>                 ioremap_cache
>                 copy_oldmem_page
>                 read_from_oldmem
>                 read_vmcore
>                 proc_reg_read
>                 vfs_read
>                 sys_read
>                 system_call_fastpath
>                 __read_nocancel
> 
>      0.00%  makedumpfile-cl  [jbd2]              
>             |
>             --- jbd2_journal_start
>                 ext4_dirty_inode
>                 __mark_inode_dirty
>                 update_time
>                 file_update_time
>                 __generic_file_aio_write
>                 generic_file_aio_write
>                 ext4_file_write
>                 do_sync_write
>                 vfs_write
>                 sys_write
>                 system_call_fastpath
>                 __write_nocancel
>                 0xffffff0000003725
> 
> 
> 

> Failed to open [dm_mirror], continuing without symbols
> Failed to open [ext4], continuing without symbols
> # ========
> # captured on: Fri Dec  7 16:45:46 2012
> # hostname : (none)
> # os release : 3.7.0-rc8-cliff-bench+
> # perf version : 3.7.rc8.3.ge0fb22
> # arch : x86_64
> # nrcpus online : 1
> # nrcpus avail : 1
> # cpudesc : Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz
> # cpuid : GenuineIntel,6,47,2
> # total memory : 240104 kB
> # cmdline : /var/crash/perf record -g -o /mnt///127.0.0.1-2012-12-07-16:45:20/perf.data2 /var/crash/makedumpfile-cliff --message-level 31 -i /var/crash/vmcoreinfo.txt -N -d 31 /proc/vmcore /mnt///127.0.0.1-2012-12-07-16:45:20/vmcore-empty-2 
> # event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, id = { 3 }
> # HEADER_CPU_TOPOLOGY info available, use -I to display
> # HEADER_NUMA_TOPOLOGY info available, use -I to display
> # pmu mappings: cpu = 4, software = 1, tracepoint = 2, uncore_bbox_0 = 15, uncore_bbox_1 = 16, uncore_cbox_0 = 7, uncore_cbox_1 = 8, uncore_cbox_2 = 9, uncore_cbox_3 = 10, uncore_cbox_4 = 11, uncore_cbox_5 = 12, uncore_cbox_6 = 13, uncore_cbox_7 = 14, uncore_mbox_0 = 19, uncore_mbox_1 = 20, uncore_rbox_0 = 21, uncore_rbox_1 = 22, uncore_sbox_0 = 17, uncore_sbox_1 = 18, breakpoint = 5, uncore_ubox = 6, uncore_wbox = 23
> # ========
> #
> # Samples: 5K of event 'cycles'
> # Event count (approx.): 1493942335
> #
> # Overhead          Command         Shared Object
> # ........  ...............  ....................
> #
>     51.45%  makedumpfile-cl  makedumpfile-cliff  
>             |          
>             |--51.45%-- set_bitmap_cyclic
>             |          |          
>             |           --100.00%-- exclude_unnecessary_pages_cyclic
>             |                     |          
>             |                      --100.00%-- writeout_dumpfile
>             |                                0x7fff0b66fe70
>             |                                0x64656b616d2f6873
>             |          
>             |--11.66%-- get_num_dumpable_cyclic
>             |          
>             |--7.96%-- clear_bit_on_2nd_bitmap_for_kernel
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff0b66fe70
>             |                     0x64656b616d2f6873
>             |          
>             |--6.93%-- create_1st_bitmap_cyclic
>             |          
>             |--4.52%-- __exclude_unnecessary_pages_kernel
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff0b66fe70
>             |                     0x64656b616d2f6873
>             |          
>             |--4.35%-- clear_bit_on_2nd_bitmap
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff0b66fe70
>             |                     0x64656b616d2f6873
>             |          
>             |--4.05%-- set_bit_on_1st_bitmap
>             |          
>             |--3.63%-- update_cyclic_region
>             |          
>             |--3.52%-- write_kdump_pages_and_bitmap_cyclic
>             |          writeout_dumpfile
>             |          0x7fff0b66fe70
>             |          0x64656b616d2f6873
>             |          
>             |--1.87%-- is_xen_memory
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff0b66fe70
>             |                     0x64656b616d2f6873
>              --0.07%-- [...]
> 
>     47.68%  makedumpfile-cl  [kernel.kallsyms]   
>             |          
>             |--77.89%-- write_vmcore_get_excludes
>             |          write_vmcore_pfn_lists
>             |          proc_reg_write
>             |          vfs_write
>             |          sys_write
>             |          system_call_fastpath
>             |          __write_nocancel
>             |          exclude_unnecessary_pages_cyclic
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff0b66fe70
>             |                     0x64656b616d2f6873
>             |          
>             |--15.34%-- walk_system_ram_range
>             |          |          
>             |          |--96.92%-- page_is_ram
>             |          |          __ioremap_caller
>             |          |          ioremap_cache
>             |          |          |          
>             |          |          |--99.51%-- write_vmcore_get_memmap
>             |          |          |          write_vmcore_pfn_lists
>             |          |          |          proc_reg_write
>             |          |          |          vfs_write
>             |          |          |          sys_write
>             |          |          |          system_call_fastpath
>             |          |          |          __write_nocancel
>             |          |          |          0x7fff0b66fe70
>             |          |          |          0x64656b616d2f6873
>             |          |           --0.49%-- [...]
>             |          |          
>             |          |--2.84%-- pat_pagerange_is_ram
>             |          |          |          
>             |          |          |--58.30%-- free_memtype
>             |          |          |          iounmap
>             |          |          |          copy_oldmem_page
>             |          |          |          read_from_oldmem
>             |          |          |          read_vmcore
>             |          |          |          proc_reg_read
>             |          |          |          vfs_read
>             |          |          |          sys_read
>             |          |          |          system_call_fastpath
>             |          |          |          __read_nocancel
>             |          |          |          |          
>             |          |          |           --100.00%-- 0x45524f4300000001
>             |          |          |          
>             |          |           --41.70%-- reserve_memtype
>             |          |                     __ioremap_caller
>             |          |                     ioremap_cache
>             |          |                     copy_oldmem_page
>             |          |                     read_from_oldmem
>             |          |                     read_vmcore
>             |          |                     proc_reg_read
>             |          |                     vfs_read
>             |          |                     sys_read
>             |          |                     system_call_fastpath
>             |          |                     __read_nocancel
>             |           --0.24%-- [...]
>             |          
>             |--0.76%-- __purge_vmap_area_lazy
>             |          |          
>             |          |--76.20%-- free_vmap_area_noflush
>             |          |          free_unmap_vmap_area
>             |          |          remove_vm_area
>             |          |          iounmap
>             |          |          copy_oldmem_page
>             |          |          read_from_oldmem
>             |          |          read_vmcore
>             |          |          proc_reg_read
>             |          |          vfs_read
>             |          |          sys_read
>             |          |          system_call_fastpath
>             |          |          __read_nocancel
>             |          |          |          
>             |          |          |--50.02%-- 0x45524f4300000001
>             |          |          |          
>             |          |           --49.98%-- 0x0
>             |          |          
>             |           --23.80%-- vm_unmap_aliases
>             |                     change_page_attr_set_clr
>             |                     _set_memory_wb
>             |                     ioremap_change_attr
>             |                     kernel_map_sync_memtype
>             |                     __ioremap_caller
>             |                     ioremap_cache
>             |                     copy_oldmem_page
>             |                     read_from_oldmem
>             |                     read_vmcore
>             |                     proc_reg_read
>             |                     vfs_read
>             |                     sys_read
>             |                     system_call_fastpath
>             |                     __read_nocancel
>             |          
>             |--0.66%-- iomem_map_sanity_check
>             |          __ioremap_caller
>             |          ioremap_cache
>             |          copy_oldmem_page
>             |          read_from_oldmem
>             |          read_vmcore
>             |          proc_reg_read
>             |          vfs_read
>             |          sys_read
>             |          system_call_fastpath
>             |          __read_nocancel
>             |          |          
>             |          |--50.44%-- 0x45524f4300000001
>             |          |          
>             |           --49.56%-- 0x0
>             |          
>             |--0.51%-- try_preserve_large_page
>             |          __change_page_attr
>             |          __change_page_attr_set_clr
>             |          change_page_attr_set_clr
>             |          _set_memory_wb
>             |          ioremap_change_attr
>             |          kernel_map_sync_memtype
>             |          __ioremap_caller
>             |          ioremap_cache
>             |          copy_oldmem_page
>             |          read_from_oldmem
>             |          read_vmcore
>             |          proc_reg_read
>             |          vfs_read
>             |          sys_read
>             |          system_call_fastpath
>             |          __read_nocancel
>              --4.84%-- [...]
> 
>      0.72%  makedumpfile-cl  libc.so.6           
>             |          
>             |--36.59%-- __memset_sse2
>             |          |          
>             |           --100.00%-- 0x7fff0b66fe70
>             |                     0x64656b616d2f6873
>             |          
>             |--31.88%-- memcpy
>             |          |          
>             |           --100.00%-- writeout_dumpfile
>             |                     0x7fff0b66fe70
>             |                     0x64656b616d2f6873
>             |          
>             |--4.88%-- time
>             |          
>             |--4.80%-- _IO_file_xsputn@@GLIBC_2.2.5
>             |          0x7f7e97029000
>             |          
>             |--2.46%-- _IO_file_init@@GLIBC_2.2.5
>             |          
>             |--2.45%-- _int_free
>             |          
>             |--2.45%-- 0x372527ffa0
>             |          
>             |--2.44%-- _IO_default_xsputn
>             |          0x71e4ef
>             |          
>             |--2.42%-- __lseek_nocancel
>             |          
>             |--2.41%-- _IO_getline_info
>             |          
>             |--2.41%-- __strlen_sse42
>             |          0x61705f6769746e6f
>             |          
>             |--2.41%-- _IO_fgets
>             |          0x3638343533313d45
>             |          
>              --2.40%-- vfprintf
>                        fprintf
> 
>      0.13%  makedumpfile-cl  ld-linux-x86-64.so.2
>             |          
>             |--28.61%-- _dl_relocate_object
>             |          dl_main
>             |          _dl_sysdep_start
>             |          0x4156415741e58948
>             |          
>             |--28.61%-- do_lookup_x
>             |          
>             |--14.96%-- strcmp
>             |          0x6e696c2d646c0036
>             |          
>             |--14.34%-- _dl_lookup_symbol_x
>             |          _dl_relocate_object
>             |          dl_main
>             |          _dl_sysdep_start
>             |          0x4156415741e58948
>             |          
>              --13.48%-- _dl_sort_fini
>                        _dl_fini
>                        exit
> 
>      0.02%  makedumpfile-cl  libstdc++.so.6      
>             |
>             --- std::basic_ios<char, std::char_traits<char> >::init(std::basic_streambuf<char, std::char_traits<char> >*)
>                 0x37284ebbe0
>                 std::basic_ostream<char, std::char_traits<char> >::~basic_ostream()
> 
> 
> 


-- 
Cliff Wickman
SGI
cpw@sgi.com
(651) 683-3824