Re: virtio net regression

[Antoine Martin <antoine@devloop.org.uk>]

Avi Kivity wrote:
> Antoine Martin wrote:
>> Hi,
>>
>> Here is another one, any ideas?
>> These oopses do look quite deep. Is it normal to end up in tcp_send_ack
>> from pdflush??
>>
>>   
>
> I think it can happen anywhere, part of the net softirq.
Hah, gotcha.
>
>> Cheers
>> Antoine
>>
>> [929492.154634] pdflush: page allocation failure. order:0, mode:0x20
>>   
>
> You're out of memory.
That's quite odd, the guest wasn't even hitting the swap at the tine.
>   How much memory did you allocate to the guest?  did you balloon it?
512MB, no ballooning.

>
>> [929492.154637] Pid: 291, comm: pdflush Not tainted 2.6.29.2 #5
>> [929492.154639] Call Trace:
>> [929492.154641]  <IRQ>  [<ffffffff8027e8bc>]
>> __alloc_pages_internal+0x3e1/0x401
>> [929492.154649]  [<ffffffff8055b5ea>] try_fill_recv+0xa1/0x182
>> [929492.154652]  [<ffffffff8055c1fc>] virtnet_poll+0x533/0x5ab
>> [929492.154655]  [<ffffffff80632bba>] net_rx_action+0x70/0x143
>> [929492.154658]  [<ffffffff8023f18c>] __do_softirq+0x83/0x123
>> [929492.154661]  [<ffffffff8020d35c>] call_softirq+0x1c/0x28
>> [929492.154664]  [<ffffffff8020e2c0>] do_softirq+0x3c/0x85
>> [929492.154666]  [<ffffffff8023eea3>] irq_exit+0x3f/0x7a
>> [929492.154668]  [<ffffffff8020e59c>] do_IRQ+0x12b/0x14f
>> [929492.154670]  [<ffffffff8020cad3>] ret_from_intr+0x0/0x29
>> [929492.154672]  <EOI>  [<ffffffff802c22b1>]
>> __set_page_dirty_buffers+0x0/0x8f
>> [929492.154677]  [<ffffffff8031702b>] bget_one+0x0/0xb
>> [929492.154680]  [<ffffffff80316fa2>] walk_page_buffers+0x2/0x8b
>> [929492.154682]  [<ffffffff803185bc>] ext3_ordered_writepage+0xae/0x134
>> [929492.154685]  [<ffffffff8027ea46>] __writepage+0xa/0x25
>> [929492.154687]  [<ffffffff8027f19f>] write_cache_pages+0x206/0x322
>> [929492.154689]  [<ffffffff8027ea3c>] __writepage+0x0/0x25
>> [929492.154691]  [<ffffffff8027f2fe>] do_writepages+0x27/0x2d
>> [929492.154694]  [<ffffffff802bd3f6>]
>> __writeback_single_inode+0x1a7/0x3b5
>> [929492.154696]  [<ffffffff8020a68c>] __switch_to+0xb4/0x38c
>> [929492.154698]  [<ffffffff802bda76>] generic_sync_sb_inodes+0x2a7/0x458
>> [929492.154701]  [<ffffffff802bde00>] writeback_inodes+0x8d/0xe6
>> [929492.154704]  [<ffffffff807296e2>] _spin_lock+0x5/0x7
>> [929492.155056]  [<ffffffff8027f432>] wb_kupdate+0x9f/0x116
>> [929492.155058]  [<ffffffff80280095>] pdflush+0x14b/0x202
>> [929492.155061]  [<ffffffff8027f393>] wb_kupdate+0x0/0x116
>> [929492.155063]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
>> [929492.155065]  [<ffffffff8027ff4a>] pdflush+0x0/0x202
>> [929492.155068]  [<ffffffff8024c127>] kthread+0x47/0x73
>> [929492.155070]  [<ffffffff8020d25a>] child_rip+0xa/0x20
>> [929492.155072]  [<ffffffff8024c0e0>] kthread+0x0/0x73
>> [929492.183142]  [<ffffffff8020d250>] child_rip+0x0/0x20
>> [929492.183145] Mem-Info:
>> [929492.183147] DMA per-cpu:
>> [929492.183149] CPU    0: hi:    0, btch:   1 usd:   0
>> [929492.183151] DMA32 per-cpu:
>> [929492.183154] CPU    0: hi:  186, btch:  31 usd: 184
>> [929492.183158] Active_anon:2755 active_file:39849 inactive_anon:2972
>> [929492.183159]  inactive_file:70353 unevictable:0 dirty:4172
>> writeback:1580 unstable:0
>> [929492.183161]  free:734 slab:5619 mapped:15047 pagetables:927 bounce:0
>> [929492.183166] DMA free:1968kB min:28kB low:32kB high:40kB
>> active_anon:0kB inactive_anon:40kB active_file:2116kB
>> inactive_file:1880kB unevictable:0kB present:5448kB pages_scanned:0
>> all_unreclaimable? no
>> [929492.183169] lowmem_reserve[]: 0 489 489 489
>> [929492.183176] DMA32 free:968kB min:2812kB low:3512kB high:4216kB
>> active_anon:11020kB inactive_anon:11848kB active_file:157280kB
>> inactive_file:279532kB unevictable:0kB present:500896kB pages_scanned:0
>> all_unreclaimable? no
>> [929492.183180] lowmem_reserve[]: 0 0 0 0
>> [929492.183183] DMA: 6*4kB 2*8kB 3*16kB 1*32kB 1*64kB 2*128kB 0*256kB
>> 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1976kB
>> [929492.183235] DMA32: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 3*128kB 2*256kB
>> 0*512kB 0*1024kB 0*2048kB 0*4096kB = 968kB
>> [929492.183244] 110992 total pagecache pages
>> [929492.183246] 739 pages in swap cache
>> [929492.183248] Swap cache stats: add 8996, delete 8257, find
>> 92604/93191
>> [929492.183250] Free swap  = 1040016kB
>> [929492.183252] Total swap = 1048568kB
>> [929492.186003] 131056 pages RAM
>> [929492.186006] 4799 pages reserved
>> [929492.186007] 44697 pages shared
>> [929492.186008] 90516 pages non-shared
>> [930274.380075] eth0: no IPv6 routers present
>>
>>   
> Strange, seems to be a bit of free memory here.
There should be lots, all this host is doing is apache+sftp...

Assuming I can make it re-occur (stress testing it?), how would I dig
further to find the cause of this memory exhaustion? /proc/meminfo and
friends?

Cheers
Antoine