Re: 2.6.39.1: Intel I340-T4: irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20

Re: 2.6.39.1: Intel I340-T4: irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20

[Mark Lord]

On 11-06-17 09:16 PM, Justin Piszcz wrote:
> Hi,
>
> Kernel 2.6.39.1, x86_64.
> Has anyone seen a page allocation failure on a NIC before?
..
> [60295.925691] irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20
> [60295.945328] Pid: 2299, comm: irq/64-eth3-TxR Not tainted 2.6.39.1 #1
> [60295.945329] Call Trace:
> [60295.945330]  <IRQ>  [<ffffffff810882f6>] ? __alloc_pages_nodemask+0x606/0x890
> [60295.945341]  [<ffffffff810b1435>] ? cache_alloc_refill+0x2c5/0x530
> [60295.945343]  [<ffffffff810b180b>] ? kmem_cache_alloc+0x7b/0xa0
> [60295.945347]  [<ffffffff815031ac>] ? sk_prot_alloc.clone.35+0x3c/0x120
> [60295.945349]  [<ffffffff81503320>] ? sk_clone+0x10/0x2b0
> [60295.945352]  [<ffffffff815580bb>] ? inet_csk_clone+0xb/0x90
> [60295.945355]  [<ffffffff8156fa31>] ? tcp_create_openreq_child+0x21/0x4e0
> [60295.945357]  [<ffffffff8156cbd3>] ? tcp_v4_syn_recv_sock+0x53/0x250
> [60295.945359]  [<ffffffff8156f790>] ? tcp_check_req+0x200/0x480
> [60295.945362]  [<ffffffff8156cab1>] ? tcp_v4_do_rcv+0x1c1/0x290
> [60295.945365]  [<ffffffff8154dd30>] ? ip_rcv_finish+0x340/0x340
> [60295.945367]  [<ffffffff8156f047>] ? tcp_v4_rcv+0x5f7/0x8b0
> [60295.945369]  [<ffffffff8154ddf4>] ? ip_local_deliver_finish+0xc4/0x200
> [60295.945373]  [<ffffffff8151158b>] ? __netif_receive_skb+0x4eb/0x610
> [60295.945375]  [<ffffffff81511898>] ? netif_receive_skb+0x78/0x80
> [60295.945377]  [<ffffffff81511f03>] ? napi_gro_receive+0xa3/0xc0
> [60295.945379]  [<ffffffff815119b8>] ? napi_skb_finish+0x38/0x50
> [60295.945383]  [<ffffffff813e6208>] ? igb_poll+0x8b8/0xd00
> [60295.945386]  [<ffffffff8102e5f1>] ? enqueue_task_rt+0x121/0x320
> [60295.945388]  [<ffffffff815120c9>] ? net_rx_action+0xf9/0x180
> [60295.945391]  [<ffffffff8103df38>] ? __do_softirq+0x98/0x120
> [60295.945395]  [<ffffffff81070010>] ? irq_thread_fn+0x40/0x40
> [60295.945397]  [<ffffffff81619a4c>] ? call_softirq+0x1c/0x30
> [60295.945398]  <EOI>  [<ffffffff81003d8d>] ? do_softirq+0x4d/0x80
> [60295.945402]  [<ffffffff8103de94>] ? local_bh_enable+0x94/0xa0
> [60295.945405]  [<ffffffff8106ff70>] ? irq_thread+0x150/0x1b0
> [60295.945407]  [<ffffffff8106fe20>] ? irq_finalize_oneshot+0x130/0x130
> [60295.945409]  [<ffffffff8106fe20>] ? irq_finalize_oneshot+0x130/0x130
> [60295.945412]  [<ffffffff81052746>] ? kthread+0x96/0xa0
> [60295.945414]  [<ffffffff81619954>] ? kernel_thread_helper+0x4/0x10
> [60295.945417]  [<ffffffff810526b0>] ? kthread_worker_fn+0x120/0x120
> [60295.945418]  [<ffffffff81619950>] ? gs_change+0xb/0xb
..

Not on a NIC, but also with 2.6.39:

[35850.612899] sd 4:0:0:0: [sdc] Attached SCSI disk
[35943.085264] mount: page allocation failure. order:5, mode:0xc0d0
[35943.085277] Pid: 14228, comm: mount Not tainted 2.6.39 #10
[35943.085284] Call Trace:
[35943.085306]  [<ffffffff8106fa96>] ? __alloc_pages_nodemask+0x710/0x74d
[35943.085322]  [<ffffffff8106fb5b>] ? __get_free_pages+0x12/0x50
[35943.085335]  [<ffffffff810f9120>] ? ext4_fill_super+0xe4f/0x20ff
[35943.085347]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e
[35943.085359]  [<ffffffff81148ef0>] ? snprintf+0x36/0x3b
[35943.085371]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e
[35943.085384]  [<ffffffff8109e05e>] ? mount_bdev+0x136/0x17d
[35943.085397]  [<ffffffff8109537d>] ? __kmalloc_track_caller+0xa9/0x116
[35943.085410]  [<ffffffff8109cfa6>] ? mount_fs+0xc/0xa6
[35943.085423]  [<ffffffff810b225d>] ? vfs_kern_mount+0x61/0x97
[35943.085434]  [<ffffffff810b22f2>] ? do_kern_mount+0x49/0xd6
[35943.085445]  [<ffffffff810b2a70>] ? do_mount+0x6f1/0x758
[35943.085457]  [<ffffffff81078f01>] ? memdup_user+0x3f/0x5b
[35943.085468]  [<ffffffff810b2b5f>] ? sys_mount+0x88/0xcd
[35943.085482]  [<ffffffff812cc47b>] ? system_call_fastpath+0x16/0x1b
[35943.085490] Mem-Info:
[35943.085496] DMA per-cpu:
[35943.085503] CPU    0: hi:    0, btch:   1 usd:   0
[35943.085511] CPU    1: hi:    0, btch:   1 usd:   0
[35943.085517] DMA32 per-cpu:
[35943.085524] CPU    0: hi:  186, btch:  31 usd:   0
[35943.085532] CPU    1: hi:  186, btch:  31 usd: 114
[35943.085549] active_anon:64179 inactive_anon:31764 isolated_anon:0
[35943.085554]  active_file:90242 inactive_file:223697 isolated_file:0
[35943.085558]  unevictable:2 dirty:24616 writeback:0 unstable:0
[35943.085562]  free:19204 slab_reclaimable:64266 slab_unreclaimable:6283
[35943.085566]  mapped:7463 shmem:31597 pagetables:5475 bounce:0
[35943.085592] DMA free:8308kB min:340kB low:424kB high:508kB active_anon:0kB
inactive_anon:1056kB active_file:1736kB inactive_file:4712kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:92kB slab_unreclaimable:8kB
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
[35943.085612] lowmem_reserve[]: 0 1993 1993 1993
[35943.085643] DMA32 free:68508kB min:44712kB low:55888kB high:67068kB
active_anon:256716kB inactive_anon:126000kB active_file:359232kB
inactive_file:890076kB unevictable:8kB isolated(anon):0kB isolated(file):0kB
present:2041776kB mlocked:0kB dirty:98464kB writeback:0kB mapped:29852kB
shmem:126388kB slab_reclaimable:256972kB slab_unreclaimable:25124kB
kernel_stack:2160kB pagetables:21900kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:115 all_unreclaimable? no
[35943.085663] lowmem_reserve[]: 0 0 0 0
[35943.085673] DMA: 3*4kB 3*8kB 3*16kB 15*32kB 17*64kB 6*128kB 3*256kB 4*512kB
1*1024kB 1*2048kB 0*4096kB = 8308kB
[35943.085700] DMA32: 5597*4kB 2221*8kB 948*16kB 330*32kB 33*64kB 4*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68508kB
[35943.085727] 345589 total pagecache pages
[35943.085733] 50 pages in swap cache
[35943.085739] Swap cache stats: add 58, delete 8, find 0/0
[35943.085744] Free swap  = 1975060kB
[35943.085749] Total swap = 1975292kB
[35943.113312] 521600 pages RAM
[35943.113318] 9355 pages reserved
[35943.113322] 290443 pages shared
[35943.113326] 248448 pages non-shared
[35943.181471] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts:
(null)

------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

Re: 2.6.39.1: Intel I340-T4: irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20

[Mark Lord]

On 11-06-18 12:19 PM, Mark Lord wrote:
> On 11-06-17 09:16 PM, Justin Piszcz wrote:
>> > Hi,
>> >
>> > Kernel 2.6.39.1, x86_64.
>> > Has anyone seen a page allocation failure on a NIC before?
> ..
>> > [60295.925691] irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20
>> > [60295.945328] Pid: 2299, comm: irq/64-eth3-TxR Not tainted 2.6.39.1 #1
>> > [60295.945329] Call Trace:
>> > [60295.945330]  <IRQ>  [<ffffffff810882f6>] ? __alloc_pages_nodemask+0x606/0x890
>> > [60295.945341]  [<ffffffff810b1435>] ? cache_alloc_refill+0x2c5/0x530
>> > [60295.945343]  [<ffffffff810b180b>] ? kmem_cache_alloc+0x7b/0xa0
>> > [60295.945347]  [<ffffffff815031ac>] ? sk_prot_alloc.clone.35+0x3c/0x120
>> > [60295.945349]  [<ffffffff81503320>] ? sk_clone+0x10/0x2b0
>> > [60295.945352]  [<ffffffff815580bb>] ? inet_csk_clone+0xb/0x90
>> > [60295.945355]  [<ffffffff8156fa31>] ? tcp_create_openreq_child+0x21/0x4e0
>> > [60295.945357]  [<ffffffff8156cbd3>] ? tcp_v4_syn_recv_sock+0x53/0x250
>> > [60295.945359]  [<ffffffff8156f790>] ? tcp_check_req+0x200/0x480
>> > [60295.945362]  [<ffffffff8156cab1>] ? tcp_v4_do_rcv+0x1c1/0x290
>> > [60295.945365]  [<ffffffff8154dd30>] ? ip_rcv_finish+0x340/0x340
>> > [60295.945367]  [<ffffffff8156f047>] ? tcp_v4_rcv+0x5f7/0x8b0
>> > [60295.945369]  [<ffffffff8154ddf4>] ? ip_local_deliver_finish+0xc4/0x200
>> > [60295.945373]  [<ffffffff8151158b>] ? __netif_receive_skb+0x4eb/0x610
>> > [60295.945375]  [<ffffffff81511898>] ? netif_receive_skb+0x78/0x80
>> > [60295.945377]  [<ffffffff81511f03>] ? napi_gro_receive+0xa3/0xc0
>> > [60295.945379]  [<ffffffff815119b8>] ? napi_skb_finish+0x38/0x50
>> > [60295.945383]  [<ffffffff813e6208>] ? igb_poll+0x8b8/0xd00
>> > [60295.945386]  [<ffffffff8102e5f1>] ? enqueue_task_rt+0x121/0x320
>> > [60295.945388]  [<ffffffff815120c9>] ? net_rx_action+0xf9/0x180
>> > [60295.945391]  [<ffffffff8103df38>] ? __do_softirq+0x98/0x120
>> > [60295.945395]  [<ffffffff81070010>] ? irq_thread_fn+0x40/0x40
>> > [60295.945397]  [<ffffffff81619a4c>] ? call_softirq+0x1c/0x30
>> > [60295.945398]  <EOI>  [<ffffffff81003d8d>] ? do_softirq+0x4d/0x80
>> > [60295.945402]  [<ffffffff8103de94>] ? local_bh_enable+0x94/0xa0
>> > [60295.945405]  [<ffffffff8106ff70>] ? irq_thread+0x150/0x1b0
>> > [60295.945407]  [<ffffffff8106fe20>] ? irq_finalize_oneshot+0x130/0x130
>> > [60295.945409]  [<ffffffff8106fe20>] ? irq_finalize_oneshot+0x130/0x130
>> > [60295.945412]  [<ffffffff81052746>] ? kthread+0x96/0xa0
>> > [60295.945414]  [<ffffffff81619954>] ? kernel_thread_helper+0x4/0x10
>> > [60295.945417]  [<ffffffff810526b0>] ? kthread_worker_fn+0x120/0x120
>> > [60295.945418]  [<ffffffff81619950>] ? gs_change+0xb/0xb
> ..
>
> Not on a NIC, but also with 2.6.39:
>
> [35850.612899] sd 4:0:0:0: [sdc] Attached SCSI disk
> [35943.085264] mount: page allocation failure. order:5, mode:0xc0d0
> [35943.085277] Pid: 14228, comm: mount Not tainted 2.6.39 #10
> [35943.085284] Call Trace:
> [35943.085306]  [<ffffffff8106fa96>] ? __alloc_pages_nodemask+0x710/0x74d
> [35943.085322]  [<ffffffff8106fb5b>] ? __get_free_pages+0x12/0x50
> [35943.085335]  [<ffffffff810f9120>] ? ext4_fill_super+0xe4f/0x20ff
> [35943.085347]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e
> [35943.085359]  [<ffffffff81148ef0>] ? snprintf+0x36/0x3b
> [35943.085371]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e
> [35943.085384]  [<ffffffff8109e05e>] ? mount_bdev+0x136/0x17d
> [35943.085397]  [<ffffffff8109537d>] ? __kmalloc_track_caller+0xa9/0x116
> [35943.085410]  [<ffffffff8109cfa6>] ? mount_fs+0xc/0xa6
> [35943.085423]  [<ffffffff810b225d>] ? vfs_kern_mount+0x61/0x97
> [35943.085434]  [<ffffffff810b22f2>] ? do_kern_mount+0x49/0xd6
> [35943.085445]  [<ffffffff810b2a70>] ? do_mount+0x6f1/0x758
> [35943.085457]  [<ffffffff81078f01>] ? memdup_user+0x3f/0x5b
> [35943.085468]  [<ffffffff810b2b5f>] ? sys_mount+0x88/0xcd
> [35943.085482]  [<ffffffff812cc47b>] ? system_call_fastpath+0x16/0x1b
> [35943.085490] Mem-Info:
> [35943.085496] DMA per-cpu:
> [35943.085503] CPU    0: hi:    0, btch:   1 usd:   0
> [35943.085511] CPU    1: hi:    0, btch:   1 usd:   0
> [35943.085517] DMA32 per-cpu:
> [35943.085524] CPU    0: hi:  186, btch:  31 usd:   0
> [35943.085532] CPU    1: hi:  186, btch:  31 usd: 114
> [35943.085549] active_anon:64179 inactive_anon:31764 isolated_anon:0
> [35943.085554]  active_file:90242 inactive_file:223697 isolated_file:0
> [35943.085558]  unevictable:2 dirty:24616 writeback:0 unstable:0
> [35943.085562]  free:19204 slab_reclaimable:64266 slab_unreclaimable:6283
> [35943.085566]  mapped:7463 shmem:31597 pagetables:5475 bounce:0
> [35943.085592] DMA free:8308kB min:340kB low:424kB high:508kB active_anon:0kB
> inactive_anon:1056kB active_file:1736kB inactive_file:4712kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB
> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:92kB slab_unreclaimable:8kB
> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? no
> [35943.085612] lowmem_reserve[]: 0 1993 1993 1993
> [35943.085643] DMA32 free:68508kB min:44712kB low:55888kB high:67068kB
> active_anon:256716kB inactive_anon:126000kB active_file:359232kB
> inactive_file:890076kB unevictable:8kB isolated(anon):0kB isolated(file):0kB
> present:2041776kB mlocked:0kB dirty:98464kB writeback:0kB mapped:29852kB
> shmem:126388kB slab_reclaimable:256972kB slab_unreclaimable:25124kB
> kernel_stack:2160kB pagetables:21900kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:115 all_unreclaimable? no
> [35943.085663] lowmem_reserve[]: 0 0 0 0
> [35943.085673] DMA: 3*4kB 3*8kB 3*16kB 15*32kB 17*64kB 6*128kB 3*256kB 4*512kB
> 1*1024kB 1*2048kB 0*4096kB = 8308kB
> [35943.085700] DMA32: 5597*4kB 2221*8kB 948*16kB 330*32kB 33*64kB 4*128kB
> 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68508kB
> [35943.085727] 345589 total pagecache pages
> [35943.085733] 50 pages in swap cache
> [35943.085739] Swap cache stats: add 58, delete 8, find 0/0
> [35943.085744] Free swap  = 1975060kB
> [35943.085749] Total swap = 1975292kB
> [35943.113312] 521600 pages RAM
> [35943.113318] 9355 pages reserved
> [35943.113322] 290443 pages shared
> [35943.113326] 248448 pages non-shared
> [35943.181471] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts:
> (null)

Oh, and another one, not quite identical:

[297246.722573] mount: page allocation failure. order:5, mode:0xc0d0
[297246.722584] Pid: 25863, comm: mount Not tainted 2.6.39 #10
[297246.722590] Call Trace:
[297246.722610]  [<ffffffff8106fa96>] ? __alloc_pages_nodemask+0x710/0x74d
[297246.722622]  [<ffffffff810bbd2c>] ? unmap_underlying_metadata+0x4b/0x4b
[297246.722633]  [<ffffffff8106fb5b>] ? __get_free_pages+0x12/0x50
[297246.722643]  [<ffffffff810f9120>] ? ext4_fill_super+0xe4f/0x20ff
[297246.722652]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e
[297246.722662]  [<ffffffff81148ef0>] ? snprintf+0x36/0x3b
[297246.722670]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e
[297246.722680]  [<ffffffff8109e05e>] ? mount_bdev+0x136/0x17d
[297246.722690]  [<ffffffff8109537d>] ? __kmalloc_track_caller+0xa9/0x116
[297246.722700]  [<ffffffff8109cfa6>] ? mount_fs+0xc/0xa6
[297246.722710]  [<ffffffff810b225d>] ? vfs_kern_mount+0x61/0x97
[297246.722720]  [<ffffffff810b22f2>] ? do_kern_mount+0x49/0xd6
[297246.722729]  [<ffffffff810b2a70>] ? do_mount+0x6f1/0x758
[297246.722740]  [<ffffffff81078f01>] ? memdup_user+0x3f/0x5b
[297246.722749]  [<ffffffff810b2b5f>] ? sys_mount+0x88/0xcd
[297246.722761]  [<ffffffff812cc47b>] ? system_call_fastpath+0x16/0x1b
[297246.722767] Mem-Info:
[297246.722772] DMA per-cpu:
[297246.722778] CPU    0: hi:    0, btch:   1 usd:   0
[297246.722784] CPU    1: hi:    0, btch:   1 usd:   0
[297246.722788] DMA32 per-cpu:
[297246.722794] CPU    0: hi:  186, btch:  31 usd:  14
[297246.722800] CPU    1: hi:  186, btch:  31 usd:   0
[297246.722813] active_anon:73864 inactive_anon:32029 isolated_anon:0
[297246.722817]  active_file:76404 inactive_file:195583 isolated_file:0
[297246.722821]  unevictable:2 dirty:19997 writeback:0 unstable:0
[297246.722824]  free:20421 slab_reclaimable:96332 slab_unreclaimable:4325
[297246.722827]  mapped:7904 shmem:29851 pagetables:5963 bounce:0
[297246.722848] DMA free:8396kB min:340kB low:424kB high:508kB active_anon:0kB
inactive_anon:2048kB active_file:3476kB inactive_file:400kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB
writeback:0kB mapped:52kB shmem:0kB slab_reclaimable:1588kB
slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[297246.722864] lowmem_reserve[]: 0 1993 1993 1993
[297246.722888] DMA32 free:73288kB min:44712kB low:55888kB high:67068kB
active_anon:295456kB inactive_anon:126068kB active_file:302140kB
inactive_file:781932kB unevictable:8kB isolated(anon):0kB isolated(file):0kB
present:2041776kB mlocked:0kB dirty:79988kB writeback:0kB mapped:31564kB
shmem:119404kB slab_reclaimable:383740kB slab_unreclaimable:17296kB
kernel_stack:2248kB pagetables:23852kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
[297246.722904] lowmem_reserve[]: 0 0 0 0
[297246.722912] DMA: 57*4kB 29*8kB 10*16kB 1*32kB 3*64kB 3*128kB 4*256kB 2*512kB
3*1024kB 1*2048kB 0*4096kB = 8396kB
[297246.722936] DMA32: 3186*4kB 2220*8kB 936*16kB 455*32kB 191*64kB 8*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 73288kB
[297246.722962] 310158 total pagecache pages
[297246.722968] 8321 pages in swap cache
[297246.722974] Swap cache stats: add 33604, delete 25283, find 46658/48112
[297246.722980] Free swap  = 1899068kB
[297246.722984] Total swap = 1975292kB
[297246.747988] 521600 pages RAM
[297246.747995] 9355 pages reserved
[297246.748000] 258772 pages shared
[297246.748004] 291766 pages non-shared
[297246.815211] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts:
(null)

Re: 2.6.39.1: Intel I340-T4: irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20

[Andreas Dilger]

On 2011-06-18, at 10:19 AM, Mark Lord <kernel@teksavvy.com> wrote:
> On 11-06-17 09:16 PM, Justin Piszcz wrote:
>> 
>> Kernel 2.6.39.1, x86_64.
>> Has anyone seen a page allocation failure on a NIC before?
> ..
>> [60295.925691] irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20
>> [60295.945328] Pid: 2299, comm: irq/64-eth3-TxR Not tainted 2.6.39.1 #1
>> [60295.945329] Call Trace:
>> [60295.945330]  <IRQ>  [<ffffffff810882f6>] ? __alloc_pages_nodemask+0x606/0x890
>> [60295.945341]  [<ffffffff810b1435>] ? cache_alloc_refill+0x2c5/0x530
>> [60295.945343]  [<ffffffff810b180b>] ? kmem_cache_alloc+0x7b/0xa0
>> [60295.945347]  [<ffffffff815031ac>] ? sk_prot_alloc.clone.35+0x3c/0x120
>> [60295.945349]  [<ffffffff81503320>] ? sk_clone+0x10/0x2b0
>> [60295.945352]  [<ffffffff815580bb>] 
> 
> Not on a NIC, but also with 2.6.39:
> 
> [35850.612899] sd 4:0:0:0: [sdc] Attached SCSI disk
> [35943.085264] mount: page allocation failure. order:5, mode:0xc0d0
> [35943.085277] Pid: 14228, comm: mount Not tainted 2.6.39 #10
> [35943.085284] Call Trace:
> [35943.085306]  [<ffffffff8106fa96>] ? __alloc_pages_nodemask+0x710/0x74d
> [35943.085322]  [<ffffffff8106fb5b>] ? __get_free_pages+0x12/0x50
> [35943.085335]  [<ffffffff810f9120>] ? ext4_fill_super+0xe4f/0x20ff
> [35943.085347]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e

There are a few places in the ext4 mount that are doing large allocations. In some places they fall back to vmalloc, so they should really be done with GFP_NOWARN.

 A few places don't yet fall back to vmalloc(), which is a problem with fragmented memory or very large filesystems. We were trying to test a 192TB ext4 filesystem, but were unable to mount it without patching the kernel.

Cheers, Andreas

Re: 2.6.39.1: Intel I340-T4: irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20

[Stephan Boettcher]

Andreas Dilger <aedilger@gmail.com> writes:

> On 2011-06-18, at 10:19 AM, Mark Lord <kernel@teksavvy.com> wrote:
>> On 11-06-17 09:16 PM, Justin Piszcz wrote:
>>> 
>>> Kernel 2.6.39.1, x86_64.
>>> Has anyone seen a page allocation failure on a NIC before?
>> ..
>>> [60295.925691] irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20
>>> [60295.945328] Pid: 2299, comm: irq/64-eth3-TxR Not tainted 2.6.39.1 #1
>>> [60295.945329] Call Trace:
>>> [60295.945330]  <IRQ>  [<ffffffff810882f6>] ? __alloc_pages_nodemask+0x606/0x890
>>> [60295.945341]  [<ffffffff810b1435>] ? cache_alloc_refill+0x2c5/0x530
>>> [60295.945343]  [<ffffffff810b180b>] ? kmem_cache_alloc+0x7b/0xa0
>>> [60295.945347]  [<ffffffff815031ac>] ? sk_prot_alloc.clone.35+0x3c/0x120
>>> [60295.945349]  [<ffffffff81503320>] ? sk_clone+0x10/0x2b0
>>> [60295.945352]  [<ffffffff815580bb>] 
>> 
>> Not on a NIC, but also with 2.6.39:
>> 
>> [35850.612899] sd 4:0:0:0: [sdc] Attached SCSI disk
>> [35943.085264] mount: page allocation failure. order:5, mode:0xc0d0
>> [35943.085277] Pid: 14228, comm: mount Not tainted 2.6.39 #10
>> [35943.085284] Call Trace:
>> [35943.085306]  [<ffffffff8106fa96>] ? __alloc_pages_nodemask+0x710/0x74d
>> [35943.085322]  [<ffffffff8106fb5b>] ? __get_free_pages+0x12/0x50
>> [35943.085335]  [<ffffffff810f9120>] ? ext4_fill_super+0xe4f/0x20ff
>> [35943.085347]  [<ffffffff810f82d1>] ? ext4_remount+0x40e/0x40e
>
> There are a few places in the ext4 mount that are doing large
> allocations. In some places they fall back to vmalloc, so they should
> really be done with GFP_NOWARN.
>
>  A few places don't yet fall back to vmalloc(), which is a problem
>  with fragmented memory or very large filesystems. We were trying to
>  test a 192TB ext4 filesystem, but were unable to mount it without
>  patching the kernel.

:-O ...  my puny 20TB ext4 filesystem did not do something like
this, yet.

> Cheers, Andreas--

-- 
Stephan 

Re: 2.6.39.1: Intel I340-T4: irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20

[Andreas Dilger]

On 2011-06-18, at 11:39 AM, Stephan Boettcher wrote:
> Andreas Dilger <adilger@dilger.ca> writes:
>> There are a few places in the ext4 mount that are doing large
>> allocations. In some places they fall back to vmalloc, so they should
>> really be done with GFP_NOWARN.
>> 
>> A few places don't yet fall back to vmalloc(), which is a problem
>> with fragmented memory or very large filesystems. We were trying to
>> test a 192TB ext4 filesystem, but were unable to mount it without
>> patching the kernel.
> 
> :-O ...  my puny 20TB ext4 filesystem did not do something like
> this, yet.

What sort of experience do you have with using a filesystem > 20TB?
I don't think there are many users out there yet that are doing this
today, so it would be great if you could share some data with us.

So far, we've only been doing testing and benchmarking (mke2fs, e2fsck
times, IO and metadata load tests, etc) and I don't know that all of
the "real world" corner cases have been tested yet.

Cheers, Andreas

Re: 2.6.39.1: Intel I340-T4: irq/64-eth3-TxR: page allocation failure. order:1, mode:0x20

[Stephan Boettcher]

Andreas Dilger <adilger@dilger.ca> writes:

> On 2011-06-18, at 11:39 AM, Stephan Boettcher wrote:
>> Andreas Dilger <adilger@dilger.ca> writes:
>>> There are a few places in the ext4 mount that are doing large
>>> allocations. In some places they fall back to vmalloc, so they should
>>> really be done with GFP_NOWARN.
>>> 
>>> A few places don't yet fall back to vmalloc(), which is a problem
>>> with fragmented memory or very large filesystems. We were trying to
>>> test a 192TB ext4 filesystem, but were unable to mount it without
>>> patching the kernel.
>> 
>> :-O ...  my puny 20TB ext4 filesystem did not do something like
>> this, yet.
>
> What sort of experience do you have with using a filesystem > 20TB?
> I don't think there are many users out there yet that are doing this
> today, so it would be great if you could share some data with us.

I will, as soon as something interesting shows up.  Currently it is
offline, I need to buy some hardware for the frontend.

The setup is nfs-md-nbd-md-sata, RAID5², 3*(6*2TB), mostly for backups.  The
aim is to keep some old solid 32-bit servers usefull for a little longer.

Three 32-bit servers each provide a 10TB nbd to the frontend, which
must be 64-bit.  The frontend, that run the outer md-RAID5 on three nbd
was an Atom525, which I had to return to it's original duties last week.

So far I filled it about 25% with backups via rsync.  

I did not observe any problems with the filesystem.  I did run several
fsck, which were surprisingly fast.  The problem I had were of the kind
that I could not login to any of the servers while they were busy
rebuilding the RAID.  This will get solved with a little more networking
gear.

As soon as I get new frontend hardware, I can run some tests, if
somebody tells me what and how to do it.  The data that is currently on
there is expendable.  The tests shall not target performance of any kind,
for obvious reasons.

> So far, we've only been doing testing and benchmarking (mke2fs, e2fsck
> times, IO and metadata load tests, etc) and I don't know that all of
> the "real world" corner cases have been tested yet.

Well, all the real world corner cases will be well out of my reach with
this setup.

-- 
Stephan
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html