From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: 2.6.39 domU - page allocation failure Date: Tue, 24 May 2011 12:13:29 -0400 Message-ID: <20110524161329.GA6716@dumpdata.com> References: <6C609111-074F-4604-B6EC-44C589621E17@theshore.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <6C609111-074F-4604-B6EC-44C589621E17@theshore.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Christopher S. Aker" Cc: xen devel List-Id: xen-devel@lists.xenproject.org On Mon, May 23, 2011 at 10:13:21PM -0400, Christopher S. Aker wrote: > Moments after deploying 2.6.39 32 bit this evening, got a report of the following. Recent pv_ops kernels seem to suffer from a too-low vm/min_free_kbytes value. Raising it from the default of ~2700 to 16k (on a 512M instance) seems to relieve some of these symptoms, but it's not a scientific analysis. > > Noteworthy is the "HighMem free:0kB min:128kB low:136kB high:148kB" line from the spew. HighMem ratio calculations seem to be insufficient ... maybe I'll try just setting it to 0. > > Any thoughts? Thanks! > > index.scgi: page allocation failure. order:5, mode:0x20 > Pid: 1997, comm: index.scgi Not tainted 2.6.39-linode33 #1 > Call Trace: > [] ? __alloc_pages_nodemask+0x516/0x6d0 > [] ? T.819+0x30/0xc0 > [] ? T.816+0x95/0x250 > [] ? cache_alloc_refill+0x246/0x290 > [] ? __kmalloc+0xb4/0xe0 > [] ? pskb_expand_head+0x12e/0x200 > [] ? __pskb_pull_tail+0x4d/0x2b0 > [] ? ipv4_confirm+0xd3/0x180 > [] ? dev_hard_start_xmit+0x1dd/0x3e0 > [] ? ip_finish_output2+0x250/0x250 > [] ? ip_finish_output2+0x250/0x250 > [] ? sch_direct_xmit+0xb2/0x170 > [] ? dev_queue_xmit+0xf9/0x320 > [] ? ip_finish_output+0x137/0x300 > [] ? ip_output+0xaa/0xe0 > [] ? ip_finish_output2+0x250/0x250 > [] ? ip_local_out+0x18/0x20 > [] ? ip_queue_xmit+0x117/0x3d0 > [] ? get_page_from_freelist+0x119/0x340 > [] ? __pollwait+0xf0/0xf0 > [] ? xen_restore_fl_direct_reloc+0x4/0x4 > [] ? tcp_transmit_skb+0x37f/0x660 > [] ? tcp_write_xmit+0x18b/0x4a0 > [] ? check_events+0x8/0xc > [] ? tcp_push_one+0x2d/0x40 > [] ? tcp_sendmsg+0x9d4/0xab0 > [] ? inet_sendmsg+0x47/0xb0 > [] ? sock_sendmsg+0xf1/0x110 > [] ? pvclock_clocksource_read+0x116/0x180 > [] ? pvclock_clocksource_read+0x116/0x180 > [] ? xen_vcpuop_set_next_event+0x48/0x80 > [] ? clockevents_program_event+0xa0/0x160 > [] ? sys_sendto+0x11e/0x170 > [] ? do_stolen_accounting+0x21a/0x250 > [] ? pvclock_clocksource_read+0x116/0x180 > [] ? sys_send+0x37/0x40 > [] ? sys_socketcall+0x1f4/0x2c0 > [] ? syscall_call+0x7/0xb > [] ? migrate_timers+0x80/0xbe > Mem-Info: > DMA per-cpu: > CPU 0: hi: 0, btch: 1 usd: 0 > CPU 1: hi: 0, btch: 1 usd: 0 > CPU 2: hi: 0, btch: 1 usd: 0 > CPU 3: hi: 0, btch: 1 usd: 0 > Normal per-cpu: > CPU 0: hi: 186, btch: 31 usd: 130 > CPU 1: hi: 186, btch: 31 usd: 168 > CPU 2: hi: 186, btch: 31 usd: 159 > CPU 3: hi: 186, btch: 31 usd: 193 > HighMem per-cpu: > CPU 0: hi: 0, btch: 1 usd: 0 > CPU 1: hi: 0, btch: 1 usd: 0 > CPU 2: hi: 0, btch: 1 usd: 0 > CPU 3: hi: 0, btch: 1 usd: 0 > active_anon:32680 inactive_anon:32950 isolated_anon:43 > active_file:22048 inactive_file:23990 isolated_file:19 > unevictable:0 dirty:122 writeback:8 unstable:0 > free:2486 slab_reclaimable:1069 slab_unreclaimable:1605 > mapped:5588 shmem:33178 pagetables:255 bounce:0 > DMA free:2092kB min:84kB low:104kB high:124kB active_anon:588kB inactive_anon:632kB active_file:32kB inactive_file:2996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:1220kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > lowmem_reserve[]: 0 490 498 498 > Normal free:7852kB min:2788kB low:3484kB high:4180kB active_anon:130132kB inactive_anon:131168kB active_file:88160kB inactive_file:92964kB unevictable:0kB isolated(anon):172kB isolated(file):76kB present:502240kB mlocked:0kB dirty:488kB writeback:32kB mapped:22332kB shmem:131492kB slab_reclaimable:4276kB slab_unreclaimable:6416kB kernel_stack:568kB pagetables:1020kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:62 all_unreclaimable? no > lowmem_reserve[]: 0 0 64 64 > HighMem free:0kB min:128kB low:136kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > lowmem_reserve[]: 0 0 0 0 > DMA: 3*4kB 4*8kB 4*16kB 2*32kB 4*64kB 3*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2092kB > Normal: 507*4kB 197*8kB 110*16kB 31*32kB 23*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7828kB > HighMem: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB > 79628 total pagecache pages > 460 pages in swap cache > Swap cache stats: add 1791, delete 1330, find 1257/1321 > Free swap = 259640kB > Total swap = 262140kB > 1050608 pages RAM Ok, so 4G > 865282 pages HighMem > 930420 pages reserved That is a large amount of reserved.. Did the previous guest (2.6.38) have less of a reserve amount? Can you for fun try to merge #stable/e820.bugfixes ? > 18088 pages shared > 104397 pages non-shared > > -Chris > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel