Re: ppoll() stuck on POLLIN while TCP peer is sending

Eric Wong
2013-01-09T03:55:14Z

Eric Dumazet <erdnetdev@gmail.com> wrote:
> On Tue, 2013-01-08 at 18:32 -0800, Eric Dumazet wrote:
> > Hmm, it seems sk_filter() can return -ENOMEM because skb has the
> > pfmemalloc() set.
>
> >
> > One TCP socket keeps retransmitting an SKB via loopback, and TCP stack
> > drops the packet again and again.
>
> sock_init_data() sets sk->sk_allocation to GFP_KERNEL
>
> Shouldnt it use (GFP_KERNEL | __GFP_NOMEMALLOC) instead ?

Thanks, things are running good after ~35 minutes so far.
Will report back if things break (hopefully I don't run out
of laptop battery power :x).

I'm now getting allocation failure warnings (which I don't believe
happened before, and should be expected, I think...)

kworker/1:1: page allocation failure: order:0, mode:0x20
Pid: 236, comm: kworker/1:1 Not tainted 3.8.0-rc2w5+ #76
Call Trace:
<IRQ> [<ffffffff810a2411>] warn_alloc_failed+0xe1/0x130
[<ffffffff810a5779>] __alloc_pages_nodemask+0x5e9/0x840
[<ffffffff8133df8d>] ? ip_rcv+0x24d/0x340
[<ffffffff811f35b3>] ? sg_init_table+0x23/0x50
[<ffffffffa002162a>] get_a_page.isra.25+0x3a/0x40 [virtio_net]
[<ffffffffa0022258>] try_fill_recv+0x318/0x4a0 [virtio_net]
[<ffffffffa00227bd>] virtnet_poll+0x3dd/0x610 [virtio_net]
[<ffffffff8131767d>] net_rx_action+0x9d/0x1a0
[<ffffffff8104284a>] __do_softirq+0xba/0x170
[<ffffffff813b199c>] call_softirq+0x1c/0x30
<EOI> [<ffffffff8100c61d>] do_softirq+0x6d/0xa0
[<ffffffff81042424>] local_bh_enable+0x94/0xa0
[<ffffffff813aed45>] __cond_resched_softirq+0x35/0x50
[<ffffffff81305e7c>] release_sock+0x9c/0x150
[<ffffffff8134b90e>] tcp_sendmsg+0x11e/0xd80
[<ffffffff81370cee>] inet_sendmsg+0x5e/0xa0
[<ffffffff81300a77>] sock_sendmsg+0x87/0xa0
[<ffffffff810a48c9>] ? __free_memcg_kmem_pages+0x9/0x10
[<ffffffff81067819>] ? select_task_rq_fair+0x699/0x6b0
[<ffffffff81300acb>] kernel_sendmsg+0x3b/0x50
[<ffffffffa0052dc9>] xs_send_kvec+0x89/0xa0 [sunrpc]
[<ffffffffa00534bf>] xs_sendpages+0x5f/0x1e0 [sunrpc]
[<ffffffff81047d63>] ? lock_timer_base.isra.32+0x33/0x60
[<ffffffffa00548e7>] xs_tcp_send_request+0x57/0x110 [sunrpc]
[<ffffffffa0051c0d>] xprt_transmit+0x6d/0x260 [sunrpc]
[<ffffffffa004f108>] call_transmit+0x1a8/0x240 [sunrpc]
[<ffffffffa0056316>] __rpc_execute+0x56/0x250 [sunrpc]
[<ffffffffa0056535>] rpc_async_schedule+0x25/0x40 [sunrpc]
[<ffffffff810515cc>] process_one_work+0x12c/0x480
[<ffffffffa0056510>] ? __rpc_execute+0x250/0x250 [sunrpc]
[<ffffffff810538ad>] worker_thread+0x15d/0x460
[<ffffffff81053750>] ? flush_delayed_work+0x60/0x60
[<ffffffff8105865b>] kthread+0xbb/0xc0
[<ffffffff810585a0>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813b063c>] ret_from_fork+0x7c/0xb0
[<ffffffff810585a0>] ? kthread_create_on_node+0x120/0x120
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 0
active_anon:3620 inactive_anon:3624 isolated_anon:0
active_file:4290 inactive_file:101218 isolated_file:0
unevictable:0 dirty:2306 writeback:0 unstable:0
free:1711 slab_reclaimable:1529 slab_unreclaimable:5796
mapped:2325 shmem:66 pagetables:759 bounce:0
free_cma:0
DMA free:2012kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:13624kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB managed:15900kB mlocked:0kB dirty:244kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:80kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 488 488 488
DMA32 free:4832kB min:2784kB low:3480kB high:4176kB active_anon:14480kB inactive_anon:14496kB active_file:17156kB inactive_file:391248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:499960kB managed:491256kB mlocked:0kB dirty:8980kB writeback:0kB mapped:9300kB shmem:264kB slab_reclaimable:6100kB slab_unreclaimable:23104kB kernel_stack:1336kB pagetables:3036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 2*4kB (U) 1*8kB (R) 1*16kB (U) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 2016kB
DMA32: 207*4kB (UEM) 116*8kB (UEM) 32*16kB (UM) 58*32kB (UM) 13*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4956kB
108890 total pagecache pages
3302 pages in swap cache
Swap cache stats: add 4086, delete 784, find 494/535
Free swap = 378980kB
Total swap = 392188kB
131054 pages RAM
3820 pages reserved
541743 pages shared
117221 pages non-shared
cat: page allocation failure: order:0, mode:0x20
Pid: 23684, comm: cat Not tainted 3.8.0-rc2w5+ #76
Call Trace:
<IRQ> [<ffffffff810a2411>] warn_alloc_failed+0xe1/0x130
[<ffffffff810a5779>] __alloc_pages_nodemask+0x5e9/0x840
[<ffffffff8133df8d>] ? ip_rcv+0x24d/0x340
[<ffffffff811f35b3>] ? sg_init_table+0x23/0x50
[<ffffffffa002162a>] get_a_page.isra.25+0x3a/0x40 [virtio_net]
[<ffffffffa0022258>] try_fill_recv+0x318/0x4a0 [virtio_net]
[<ffffffffa00227bd>] virtnet_poll+0x3dd/0x610 [virtio_net]
[<ffffffff8131767d>] net_rx_action+0x9d/0x1a0
[<ffffffff8104284a>] __do_softirq+0xba/0x170
[<ffffffff813b199c>] call_softirq+0x1c/0x30
[<ffffffff8100c61d>] do_softirq+0x6d/0xa0
[<ffffffff81042a75>] irq_exit+0xa5/0xb0
[<ffffffff8100c25e>] do_IRQ+0x5e/0xd0
[<ffffffff813afe2d>] common_interrupt+0x6d/0x6d
<EOI> [<ffffffff813af82c>] ? _raw_spin_unlock_irqrestore+0xc/0x20
[<ffffffff810a91b6>] pagevec_lru_move_fn+0xb6/0xe0
[<ffffffff810a8780>] ? compound_unlock_irqrestore+0x20/0x20
[<ffffffffa00acd30>] ? nfs_read_completion+0x190/0x190 [nfs]
[<ffffffff810a91f7>] __pagevec_lru_add+0x17/0x20
[<ffffffff810a95c8>] __lru_cache_add+0x68/0x90
[<ffffffff8109e869>] add_to_page_cache_lru+0x29/0x40
[<ffffffff810a80cc>] read_cache_pages+0x6c/0x100
[<ffffffffa00ad4dc>] nfs_readpages+0xcc/0x160 [nfs]
[<ffffffff810a7f57>] __do_page_cache_readahead+0x1c7/0x280
[<ffffffff810a827c>] ra_submit+0x1c/0x20
[<ffffffff810a83ad>] ondemand_readahead+0x12d/0x250
[<ffffffff8109f37d>] ? __generic_file_aio_write+0x1bd/0x3c0
[<ffffffff810a8550>] page_cache_async_readahead+0x80/0xa0
[<ffffffff8109e0b8>] ? find_get_page+0x28/0xd0
[<ffffffff8109fb73>] generic_file_aio_read+0x503/0x6c0
[<ffffffffa00a4231>] nfs_file_read+0x91/0xb0 [nfs]
[<ffffffff810e4477>] do_sync_read+0xa7/0xe0
[<ffffffff810e4b50>] vfs_read+0xa0/0x160
[<ffffffff810e4c5d>] sys_read+0x4d/0x90
[<ffffffff813b06e9>] system_call_fastpath+0x16/0x1b
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 86
CPU 1: hi: 186, btch: 31 usd: 167
active_anon:2376 inactive_anon:2431 isolated_anon:0
active_file:3712 inactive_file:103686 isolated_file:17
unevictable:0 dirty:646 writeback:0 unstable:0
free:807 slab_reclaimable:1485 slab_unreclaimable:5873
mapped:2343 shmem:66 pagetables:791 bounce:0
free_cma:0
DMA free:2032kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:12916kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:124kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 488 488 488
DMA32 free:1196kB min:2784kB low:3480kB high:4176kB active_anon:9504kB inactive_anon:9724kB active_file:14824kB inactive_file:401828kB unevictable:0kB isolated(anon):0kB isolated(file):68kB present:499960kB managed:491256kB mlocked:0kB dirty:2588kB writeback:0kB mapped:9360kB shmem:264kB slab_reclaimable:5924kB slab_unreclaimable:23368kB kernel_stack:1336kB pagetables:3164kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 11*4kB (MR) 1*8kB (R) 0*16kB 2*32kB (R) 4*64kB (R) 3*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 2036kB
DMA32: 40*4kB (UEM) 28*8kB (UM) 22*16kB (UEM) 5*32kB (UM) 5*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1216kB
108199 total pagecache pages
714 pages in swap cache
Swap cache stats: add 4829, delete 4115, find 583/626
Free swap = 376280kB
Total swap = 392188kB
131054 pages RAM
3820 pages reserved
542491 pages shared
117305 pages non-shared
cat: page allocation failure: order:0, mode:0x20
Pid: 24171, comm: cat Not tainted 3.8.0-rc2w5+ #76
Call Trace:
<IRQ> [<ffffffff810a2411>] warn_alloc_failed+0xe1/0x130
[<ffffffff8137871e>] ? fib_table_lookup+0x26e/0x2d0
[<ffffffff810a5779>] __alloc_pages_nodemask+0x5e9/0x840
[<ffffffff8130a6c0>] __netdev_alloc_frag+0xa0/0x150
[<ffffffff8130d9d2>] __netdev_alloc_skb+0x82/0xe0
[<ffffffffa00225b7>] virtnet_poll+0x1d7/0x610 [virtio_net]
[<ffffffff8131767d>] net_rx_action+0x9d/0x1a0
[<ffffffff8104284a>] __do_softirq+0xba/0x170
[<ffffffff813b199c>] call_softirq+0x1c/0x30
[<ffffffff8100c61d>] do_softirq+0x6d/0xa0
[<ffffffff81042a75>] irq_exit+0xa5/0xb0
[<ffffffff8100c25e>] do_IRQ+0x5e/0xd0
[<ffffffff813afe2d>] common_interrupt+0x6d/0x6d
<EOI> [<ffffffff813aeeb5>] ? io_schedule+0xa5/0xd0
[<ffffffff811ef000>] ? copy_user_generic_string+0x30/0x40
[<ffffffff8109db32>] ? __lock_page_killable+0x62/0x70
[<ffffffff8109da85>] ? file_read_actor+0x135/0x180
[<ffffffff8109f950>] generic_file_aio_read+0x2e0/0x6c0
[<ffffffffa00a4231>] nfs_file_read+0x91/0xb0 [nfs]
[<ffffffff810e4477>] do_sync_read+0xa7/0xe0
[<ffffffff810e4b50>] vfs_read+0xa0/0x160
[<ffffffff810e4c5d>] sys_read+0x4d/0x90
[<ffffffff813b06e9>] system_call_fastpath+0x16/0x1b
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 50
CPU 1: hi: 186, btch: 31 usd: 30
active_anon:2367 inactive_anon:2431 isolated_anon:0
active_file:3719 inactive_file:103732 isolated_file:0
unevictable:0 dirty:1302 writeback:0 unstable:0
free:754 slab_reclaimable:1589 slab_unreclaimable:5896
mapped:2343 shmem:66 pagetables:781 bounce:0
free_cma:0
DMA free:1980kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:9704kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB managed:15900kB mlocked:0kB dirty:84kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:180kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 488 488 488
DMA32 free:1036kB min:2784kB low:3480kB high:4176kB active_anon:9468kB inactive_anon:9724kB active_file:14852kB inactive_file:405224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:499960kB managed:491256kB mlocked:0kB dirty:5124kB writeback:0kB mapped:9360kB shmem:264kB slab_reclaimable:6340kB slab_unreclaimable:23404kB kernel_stack:1368kB pagetables:3124kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 34*4kB (UR) 23*8kB (UMR) 40*16kB (UMR) 2*32kB (UM) 1*64kB (R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 1984kB
DMA32: 1*4kB (U) 33*8kB (UEM) 8*16kB (UM) 4*32kB (UM) 8*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1036kB
108257 total pagecache pages
712 pages in swap cache
Swap cache stats: add 4829, delete 4117, find 585/628
Free swap = 376288kB
Total swap = 392188kB
131054 pages RAM
3820 pages reserved
280574 pages shared
116800 pages non-shared
--
Eric Wong