2.6.10-rc1-bk page allocation failure. order:2, mode:0x20

2.6.10-rc1-bk page allocation failure. order:2, mode:0x20

[Brad Campbell]

G'day all,

I'm still getting quite a lot of these come up in the logs when the system is under mild load.
I suspect it might have something to do with running an MTU of 9000 on the main ethernet port which 
is directly feeding a workstation with an NFS root (and thus gets quite a high load at times)

It's not so much an issue but it does cause the workstation to stall for up to a second while it 
waits for data every time it occurs.

The loaded ethernet port is this one on an PCI card

0000:00:0d.0 Ethernet controller: Marvell Technology Group Ltd. Yukon Gigabit Ethernet 
10/100/1000Base-T Adapter (rev 12)

This started rearing its ugly head when I moved from 2.6.5 to 2.6.9-preX and persists with BK as of 
about 2 days ago.

Regards,
Brad

srv:/home/brad# uname -a
Linux srv 2.6.10-rc1 #2 Sun Oct 31 10:56:30 GST 2004 i686 GNU/Linux

srv:/home/brad# free
              total       used       free     shared    buffers     cached
Mem:        514676     251088     263588          0      13280     156328
-/+ buffers/cache:      81480     433196
Swap:       987988        284     987704

srv:/home/brad# lspci
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT8377 [KT400/KT600 AGP] Host Bridge (rev 80)
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI Bridge
0000:00:09.0 Ethernet controller: 3Com Corporation 3c940 10/100/1000Base-T [Marvell] (rev 12)
0000:00:0b.0 Unknown mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02)
0000:00:0c.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46)
0000:00:0d.0 Ethernet controller: Marvell Technology Group Ltd. Yukon Gigabit Ethernet 
10/100/1000Base-T Adapter (rev 12)
0000:00:0e.0 Unknown mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02)
0000:00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
0000:00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus 
Master IDE (rev 06)
0000:00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
0000:00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [K8T800 South]
0000:00:13.0 Unknown mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02)
0000:01:00.0 VGA compatible controller: Silicon Integrated Systems [SiS] 315PRO PCI/AGP VGA Display 
Adapter

srv:/home/brad# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0E:A6:41:45:94
           inet addr:192.168.3.82  Bcast:192.168.3.255  Mask:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
           Interrupt:18 Memory:de800000-0

eth1      Link encap:Ethernet  HWaddr 00:04:E2:8E:1E:AD
           inet addr:192.168.2.82  Bcast:192.168.2.255  Mask:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
           RX packets:5207091 errors:430 dropped:430 overruns:0 frame:0
           TX packets:4805338 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:322428903 (307.4 MiB)  TX bytes:2838601552 (2.6 GiB)
           Interrupt:16 Memory:dc800000-0

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:138 errors:0 dropped:0 overruns:0 frame:0
           TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:9378 (9.1 KiB)  TX bytes:9378 (9.1 KiB)


srv:/home/brad# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1              19G  9.4G  8.1G  54% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/hda3             165G   38G  119G  24% /raid0
/dev/md0              2.1T  2.1T   12G 100% /raid
/dev/md2              459G  315G  145G  69% /raid2

swapper: page allocation failure. order:2, mode:0x20
  [<c0138a98>] __alloc_pages+0x1c8/0x390
  [<c0138c7f>] __get_free_pages+0x1f/0x40
  [<c013c18d>] kmem_getpages+0x1d/0xb0
  [<c013ce26>] cache_grow+0xb6/0x170
  [<c013d03e>] cache_alloc_refill+0x15e/0x220
  [<c013d400>] __kmalloc+0x80/0xa0
  [<c02e61e7>] alloc_skb+0x47/0xe0
  [<e08bc871>] FillRxDescriptor+0x31/0xc0 [sk98lin]
  [<e08bc824>] FillRxRing+0x54/0x70 [sk98lin]
  [<e08bb89a>] SkGeIsrOnePort+0x17a/0x190 [sk98lin]
  [<c0132b24>] handle_IRQ_event+0x34/0x70
  [<c0132c42>] __do_IRQ+0xe2/0x160
  [<c0106666>] do_IRQ+0x26/0x40
  [<c0104aa8>] common_interrupt+0x18/0x20
  [<c0101fb0>] default_idle+0x0/0x30
  [<c0101fd3>] default_idle+0x23/0x30
  [<c010204a>] cpu_idle+0x3a/0x60
  [<c047e7c2>] start_kernel+0x142/0x160
  [<c047e3b0>] unknown_bootoption+0x0/0x1b0
syslogd: page allocation failure. order:2, mode:0x20
  [<c0138a98>] __alloc_pages+0x1c8/0x390
  [<c0138c7f>] __get_free_pages+0x1f/0x40
  [<c013c18d>] kmem_getpages+0x1d/0xb0
  [<c013ce26>] cache_grow+0xb6/0x170
  [<c013d03e>] cache_alloc_refill+0x15e/0x220
  [<c013d400>] __kmalloc+0x80/0xa0
  [<c02e61e7>] alloc_skb+0x47/0xe0
  [<e08bc871>] FillRxDescriptor+0x31/0xc0 [sk98lin]
  [<e08bc824>] FillRxRing+0x54/0x70 [sk98lin]
  [<e08bb89a>] SkGeIsrOnePort+0x17a/0x190 [sk98lin]
  [<c0132b24>] handle_IRQ_event+0x34/0x70
  [<c0132c42>] __do_IRQ+0xe2/0x160
  [<c0106666>] do_IRQ+0x26/0x40
  [<c0104aa8>] common_interrupt+0x18/0x20
syslogd: page allocation failure. order:2, mode:0x20
  [<c0138a98>] __alloc_pages+0x1c8/0x390
  [<c0104e15>] show_trace+0x35/0x90
  [<c0138c7f>] __get_free_pages+0x1f/0x40
  [<c013c18d>] kmem_getpages+0x1d/0xb0
  [<c013ce26>] cache_grow+0xb6/0x170
  [<c013d03e>] cache_alloc_refill+0x15e/0x220
  [<c0138c7f>] __get_free_pages+0x1f/0x40
  [<c013d400>] __kmalloc+0x80/0xa0
  [<c02e61e7>] alloc_skb+0x47/0xe0
  [<e08bc871>] FillRxDescriptor+0x31/0xc0 [sk98lin]
  [<e08bc824>] FillRxRing+0x54/0x70 [sk98lin]
  [<e08c0584>] SkDrvEvent+0xa44/0xaa0 [sk98lin]
  [<e08cd953>] SkGeSirqIsr+0x1d3/0x890 [sk98lin]
  [<e08d06e4>] SkEventDispatcher+0xc4/0x160 [sk98lin]
  [<e08bb7cc>] SkGeIsrOnePort+0xac/0x190 [sk98lin]
  [<c0132b24>] handle_IRQ_event+0x34/0x70
  [<c0132c42>] __do_IRQ+0xe2/0x160
  [<c0106666>] do_IRQ+0x26/0x40
  [<c0104aa8>] common_interrupt+0x18/0x20

Re: 2.6.10-rc1-bk page allocation failure. order:2, mode:0x20

[Nick Piggin]

Brad Campbell wrote:
> G'day all,
> 
> I'm still getting quite a lot of these come up in the logs when the 
> system is under mild load.
> I suspect it might have something to do with running an MTU of 9000 on 
> the main ethernet port which is directly feeding a workstation with an 
> NFS root (and thus gets quite a high load at times)
> 
> It's not so much an issue but it does cause the workstation to stall for 
> up to a second while it waits for data every time it occurs.
> 
> The loaded ethernet port is this one on an PCI card
> 
> 0000:00:0d.0 Ethernet controller: Marvell Technology Group Ltd. Yukon 
> Gigabit Ethernet 10/100/1000Base-T Adapter (rev 12)
> 
> This started rearing its ugly head when I moved from 2.6.5 to 2.6.9-preX 
> and persists with BK as of about 2 days ago.
> 

There are patches in the newest -mm kernels that should help the
problem. If you're willing to test them, the feedback would be
welcome.

Thanks
Nick

Re: 2.6.10-rc1-bk page allocation failure. order:2, mode:0x20

[Brad Campbell]

Nick Piggin wrote:
> Brad Campbell wrote:
> 
>> G'day all,
>>
>> I'm still getting quite a lot of these come up in the logs when the 
>> system is under mild load.
>> I suspect it might have something to do with running an MTU of 9000 on 
>> the main ethernet port which is directly feeding a workstation with an 
>> NFS root (and thus gets quite a high load at times)
>>
>> It's not so much an issue but it does cause the workstation to stall 
>> for up to a second while it waits for data every time it occurs.
>>
>> The loaded ethernet port is this one on an PCI card
>>
>> 0000:00:0d.0 Ethernet controller: Marvell Technology Group Ltd. Yukon 
>> Gigabit Ethernet 10/100/1000Base-T Adapter (rev 12)
>>
>> This started rearing its ugly head when I moved from 2.6.5 to 
>> 2.6.9-preX and persists with BK as of about 2 days ago.
>>
> 
> There are patches in the newest -mm kernels that should help the
> problem. If you're willing to test them, the feedback would be
> welcome.

Always willing to test specific patches. Can I just grab the broken out patches, or pull some 
specific csets from a bk tree? I'm not particularly keen on running an -mm kernel on this box if I 
can avoid it (It's a server in 24hr use with 2.5TB of data where the backup media is 7,000km away).

Regards,
Brad

Re: 2.6.10-rc1-bk page allocation failure. order:2, mode:0x20

[Brad Campbell]

Nick Piggin wrote:

>> Always willing to test specific patches. Can I just grab the broken 
>> out patches, or pull some specific csets from a bk tree? I'm not 
>> particularly keen on running an -mm kernel on this box if I can avoid 
>> it (It's a server in 24hr use with 2.5TB of data where the backup 
>> media is 7,000km away).
>>
> 
> OK fair enough.
> 
> Here is a rollup of the 3 patches that are supposed to help with
> the problem. It is diffed against 2.6.10-rc1-bk8, which you probably
> wouldn't want to run either.
> 
> Not sure how cleanly it will apply onto 2.6.9... shouldn't be too bad
> I think.
>

I'm actually running a reasonably recent BK pull of 2.6.10-rc1 as of a couple of days ago, but I did 
some pretty severe testing and evaluation with my raid disks removed and replaced with spares before 
I let it loose on the real media. I have applied those patches and I'll beat on it for a few hours 
and see how it goes. I have some pretty defined cron jobs that make it easy to reproduce.

Regards,
Brad