Re: [Bug #14141] order 2 page allocation failures in iwlagn

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Friday 02 October 2009, Frans Pop wrote:
> On Thursday 01 October 2009, Rafael J. Wysocki wrote:
> > Bug-Entry     : http://bugzilla.kernel.org/show_bug.cgi?id=14141
> > Subject       : order 2 page allocation failures in iwlagn
> > Submitter     : Frans Pop <elendil@planet.nl>
> > Date          : 2009-09-06 7:40 (26 days old)
> > References    : http://marc.info/?l=linux-kernel&m=125222287419691&w=4
> > Handled-By    : Pekka Enberg <penberg@cs.helsinki.fi>
>
> I'm not sure about this.
>
> The error messages from failed allocations should now be a lot less as a
> result of this commit:
> commit f82a924cc88a5541df1d4b9d38a0968cd077a051
> Author: Reinette Chatre <reinette.chatre@intel.com>
> Date:   Thu Sep 17 10:43:56 2009 -0700
>     iwlwifi: reduce noise when skb allocation fails
>
> That commit is in mainline, and I'm not sure if it is important enough
> for a stable update (AFAICT it's not listed for 2.6.31.2).
>
> That commit is mostly cosmetic, but possibly the real regression is not
> in iwlagn but in the way memory is freed/defragmented. That aspect was
> also reported by Bartlomiej (#14016) and was extensively discussed
> (without a clear conclusion) here: http://lkml.org/lkml/2009/8/26/140.

I may be getting somewhere with this. I just got the allocation failures
included below on .32-rc3. Note that these are not the "fixable" failures
that got suppressed with the commit referenced above, but the "this could
affect networking" failures that are still reported.

What I was doing when I got them is also interesting:
- a kernel build
- a gitk for the kernel tree (with full history this uses ~40% of memory)
- by mistake I then started a _second_ gitk

The second gitk (which shows as 'wish8.5' in top) caused a massive swap
out which brought the system to a standstill for a while (with huge
latencies as well, including a completely stuck mouse cursor, which
happens rarely).
The system has 2GB RAM + 2GB swap, so IIUC there is no danger of getting
into an OOM as the first gitk can be swapped out completely.

I'll dig into this a bit more as it looks like this should be reproducible,
probably even without the kernel build. Next step is to see how .30 behaves
in the same situation.

Even if it is reproducible with .30, I wonder if the kernel shouldn't be
more robust in this situation. Currently it seems to allow one single
process to claim so much memory before swapping out that "normal" operation
of other processes is affected. I can understand that such a situation may
be hard to avoid on a very busy system where multiple processes start
claiming (a lot of) memory at roughly the same time, but I'd say it should
be avoidable if a single process is the culprit.

BTW, the system recovered completely, although that took some time (the
first gitk remained visible in top long after I closed its window; I think
because the system was busy swapping it back in before terminating it).

Cheers,
FJP

kcryptd: page allocation failure. order:2, mode:0x4020
Pid: 1483, comm: kcryptd Not tainted 2.6.32-rc3 #22
Call Trace:
 <IRQ>  [<ffffffff8107c3d5>] __alloc_pages_nodemask+0x5a2/0x5ec
 [<ffffffff81264892>] ? _spin_unlock+0x9/0xb
 [<ffffffff811e73cd>] ? __alloc_skb+0x3c/0x15b
 [<ffffffffa03202cb>] ? iwl_rx_allocate+0x8f/0x305 [iwlcore]
 [<ffffffff8107c431>] __get_free_pages+0x12/0x41
 [<ffffffff8109cb1a>] __kmalloc_track_caller+0x3b/0xed
 [<ffffffff811e73f7>] __alloc_skb+0x66/0x15b
 [<ffffffffa03202cb>] iwl_rx_allocate+0x8f/0x305 [iwlcore]
 [<ffffffffa0320557>] iwl_rx_replenish_now+0x16/0x23 [iwlcore]
 [<ffffffffa035c0c8>] iwl_rx_handle+0x3a8/0x3c1 [iwlagn]
 [<ffffffff81051add>] ? sched_clock_local+0x1c/0x80
 [<ffffffffa035c60d>] iwl_irq_tasklet_legacy+0x52c/0x7a4 [iwlagn]
 [<ffffffffa0317aaf>] ? __iwl_read32+0xa5/0xb4 [iwlcore]
 [<ffffffff8103efb8>] tasklet_action+0x71/0xbc
 [<ffffffff8103f837>] __do_softirq+0x96/0x11b
 [<ffffffff8100cabc>] call_softirq+0x1c/0x28
 [<ffffffff8100e5ef>] do_softirq+0x33/0x6b
 [<ffffffff8103f5c5>] irq_exit+0x36/0x75
 [<ffffffff8100dcf1>] do_IRQ+0xa3/0xba
 [<ffffffff8100c353>] ret_from_intr+0x0/0xa
 <EOI>  [<ffffffff811199dd>] ? scatterwalk_start+0x11/0x19
 [<ffffffff8111bbca>] ? blkcipher_walk_first+0x173/0x196
 [<ffffffff8111b67b>] ? blkcipher_walk_done+0xe6/0x1b8
 [<ffffffff8111bc35>] ? blkcipher_walk_virt+0x1a/0x1d
 [<ffffffffa02001cf>] ? crypto_cbc_encrypt+0x43/0x18e [cbc]
 [<ffffffff81127efd>] ? blk_recount_segments+0x1b/0x2c
 [<ffffffffa021371e>] ? aes_encrypt+0x0/0xf [aes_x86_64]
 [<ffffffff8111af64>] ? async_encrypt+0x38/0x3a
 [<ffffffffa01f7b54>] ? crypt_convert+0x1f9/0x28b [dm_crypt]
 [<ffffffffa01f8009>] ? kcryptd_crypt+0x423/0x449 [dm_crypt]
 [<ffffffffa01f7be6>] ? kcryptd_crypt+0x0/0x449 [dm_crypt]
 [<ffffffff81049bfd>] ? worker_thread+0x146/0x1d8
 [<ffffffff8104d706>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81049ab7>] ? worker_thread+0x0/0x1d8
 [<ffffffff8104d3f4>] ? kthread+0x7d/0x85
 [<ffffffff8100c9ba>] ? child_rip+0xa/0x20
 [<ffffffff8104d377>] ? kthread+0x0/0x85
 [<ffffffff8100c9b0>] ? child_rip+0x0/0x20
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 171
CPU    1: hi:  186, btch:  31 usd: 177
active_anon:298532 inactive_anon:100163 isolated_anon:52
 active_file:3993 inactive_file:4001 isolated_file:12
 unevictable:399 dirty:0 writeback:76102 unstable:0 buffer:125
 free:14107 slab_reclaimable:4510 slab_unreclaimable:20421
 mapped:7949 shmem:0 pagetables:4437 bounce:0
DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3340kB inactive_anon:3608kB active_file:384kB
   inactive_file:472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB
   dirty:0kB writeback:80kB mapped:256kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:104kB kernel_stack:0kB
   pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1976 1976 1976
DMA32 free:48500kB min:5664kB low:7080kB high:8496kB active_anon:1190788kB inactive_anon:397044kB active_file:15588kB
   inactive_file:15532kB unevictable:1596kB isolated(anon):208kB isolated(file):48kB present:2023748kB mlocked:1596kB
   dirty:0kB writeback:304328kB mapped:31540kB shmem:0kB slab_reclaimable:18028kB slab_unreclaimable:81496kB kernel_stack:1672kB
   pagetables:17732kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 19*4kB 13*8kB 3*16kB 7*32kB 11*64kB 11*128kB 5*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 7940kB
DMA32: 9299*4kB 1341*8kB 4*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 48500kB
98572 total pagecache pages
90213 pages in swap cache
Swap cache stats: add 175874, delete 85661, find 7850/8731
Free swap  = 1425944kB
Total swap = 2097144kB
518064 pages RAM
10350 pages reserved
82388 pages shared
437481 pages non-shared
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
swapper: page allocation failure. order:2, mode:0x4020
Pid: 0, comm: swapper Not tainted 2.6.32-rc3 #22
Call Trace:
 <IRQ>  [<ffffffff8107c3d5>] __alloc_pages_nodemask+0x5a2/0x5ec
 [<ffffffff81264892>] ? _spin_unlock+0x9/0xb
 [<ffffffff811e73cd>] ? __alloc_skb+0x3c/0x15b
 [<ffffffffa03202cb>] ? iwl_rx_allocate+0x8f/0x305 [iwlcore]
 [<ffffffff8107c431>] __get_free_pages+0x12/0x41
 [<ffffffff8109cb1a>] __kmalloc_track_caller+0x3b/0xed
 [<ffffffff811e73f7>] __alloc_skb+0x66/0x15b
 [<ffffffffa03202cb>] iwl_rx_allocate+0x8f/0x305 [iwlcore]
 [<ffffffffa0320557>] iwl_rx_replenish_now+0x16/0x23 [iwlcore]
 [<ffffffffa035c0c8>] iwl_rx_handle+0x3a8/0x3c1 [iwlagn]
 [<ffffffffa035c60d>] iwl_irq_tasklet_legacy+0x52c/0x7a4 [iwlagn]
 [<ffffffffa0317aaf>] ? __iwl_read32+0xa5/0xb4 [iwlcore]
 [<ffffffff8103efb8>] tasklet_action+0x71/0xbc
 [<ffffffff8103f837>] __do_softirq+0x96/0x11b
 [<ffffffff8100cabc>] call_softirq+0x1c/0x28
 [<ffffffff8100e5ef>] do_softirq+0x33/0x6b
 [<ffffffff8103f5c5>] irq_exit+0x36/0x75
 [<ffffffff8100dcf1>] do_IRQ+0xa3/0xba
 [<ffffffff8100c353>] ret_from_intr+0x0/0xa
 <EOI>  [<ffffffffa0278ec7>] ? acpi_idle_enter_simple+0xf9/0x127 [processor]
 [<ffffffffa0278ebd>] ? acpi_idle_enter_simple+0xef/0x127 [processor]
 [<ffffffff811da545>] ? cpuidle_idle_call+0x8c/0xc7
 [<ffffffff8100ae2e>] ? cpu_idle+0x55/0x8d
 [<ffffffff8125432d>] ? rest_init+0x61/0x63
 [<ffffffff81436c3e>] ? start_kernel+0x348/0x353
 [<ffffffff8143629a>] ? x86_64_start_reservations+0xaa/0xae
 [<ffffffff8143637f>] ? x86_64_start_kernel+0xe1/0xe8
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 171
CPU    1: hi:  186, btch:  31 usd: 155
active_anon:297901 inactive_anon:99948 isolated_anon:52
 active_file:3920 inactive_file:3948 isolated_file:12
 unevictable:399 dirty:0 writeback:34634 unstable:0 buffer:125
 free:23390 slab_reclaimable:4510 slab_unreclaimable:11714
 mapped:7819 shmem:0 pagetables:4437 bounce:0
DMA free:7908kB min:40kB low:48kB high:60kB active_anon:3340kB inactive_anon:3608kB active_file:384kB
   inactive_file:472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB
   dirty:0kB writeback:36kB mapped:256kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:12kB kernel_stack:0kB
   pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1976 1976 1976
DMA32 free:85652kB min:5664kB low:7080kB high:8496kB active_anon:1188264kB inactive_anon:396184kB active_file:15296kB
   inactive_file:15320kB unevictable:1596kB isolated(anon):208kB isolated(file):48kB present:2023748kB mlocked:1596kB
   dirty:0kB writeback:138500kB mapped:31020kB shmem:0kB slab_reclaimable:18028kB slab_unreclaimable:46844kB
   kernel_stack:1672kB pagetables:17732kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 17*4kB 12*8kB 4*16kB 6*32kB 11*64kB 11*128kB 5*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 7908kB
DMA32: 12419*4kB 4439*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85652kB
97616 total pagecache pages
89394 pages in swap cache
Swap cache stats: add 175906, delete 86512, find 7850/8733
Free swap  = 1425864kB
Total swap = 2097144kB
518064 pages RAM
10350 pages reserved
82282 pages shared
428383 pages non-shared
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 05 October 2009, Frans Pop wrote:
> I'll dig into this a bit more as it looks like this should be
> reproducible, probably even without the kernel build. Next step is to
> see how .30 behaves in the same situation.

This looks conclusive. I tested .30 and .32-rc3 from clean reboots and
only starting gitk. I only started music playing in the background
(amarok) from an NFS share to ensure network activity.

With .32-rc3 I got 4 SKB allocation errors while starting the *second* gitk
instance. And the system was completely frozen with music stopped until gitk
finished loading.

With .30 I was able to start *three* gitk's (which meant 2 of them got
(partially) swapped out) without any allocation errors. And with the system
remaining relatively responsive. There was a short break in the music while
I started the 2nd instance, but it just continued playing afterwards. There
was also some mild latency in the mouse cursor, but nothing like the full
desktop freeze I get with .32-rc3.

With .30 I looked at /proc/buddyinfo while the 3rd gitk was being started,
and that looked fairly healthy all the time:
Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
Node 0, zone    DMA32    579     67     25      8      5      1      1      0      1      1      0
Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
Node 0, zone    DMA32    276     54     13     15      8     10      3      1      1      1      0
Node 0, zone      DMA      4      9     22     20     21     11      0      0      0      0      1
Node 0, zone    DMA32    119     45     24     18     12      4      5      2      1      1      0
Node 0, zone      DMA      4      9     22     20     21     11      0      0      0      0      1
Node 0, zone    DMA32    527     13      9      5      5      3      2      1      1      1      0
Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
Node 0, zone    DMA32   1375     24      7      7      8      5      1      1      0      1      0
Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
Node 0, zone    DMA32    329     21      3      3     17      8      5      1      0      1      0

With .32 it was obviously impossible to get that info due to the total
freeze of the desktop. Not sure if the scheduler changes in .32 contribute
to this. Guess I could find out by doing the same test with .31.

One thing I should mention: my swap is an LVM volume that's in a VG that's
on a LUKS encrypted partition.

Does this give you enough info to go on, or should I try a bisection?

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 05 October 2009, Frans Pop wrote:
> With .32 it was obviously impossible to get that info due to the total
> freeze of the desktop. Not sure if the scheduler changes in .32
> contribute to this. Guess I could find out by doing the same test with
> .31.

I've tried with .31.1 too now and there does seem to be a scheduler
component too. With .31.1 I also get the SKB allocation errors, but the
desktop freeze seems to be less severe than with .32-rc3. I would suggest
looking into that _after_ the allocation issue has been traced/solved.

I did manage to really (partially) hang up the desktop with .31.1: music
did not come back and the task manager of the KDE desktop remained frozen,
but I could still use konsole [1].
I suspect this is because I also got an OOPS in between the SKB failures:

IP: [<ffffffffa0444ea2>] rpcauth_checkverf+0x4e/0x5a[sunrpc]
PGD 77b83067 PUD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/class/power_supply/C23D/charge_full
CPU 0
Modules linked in: i915 drm i2c_algo_bit i2c_core ppdev parport_pc lp parport cpufreq_conservative
   cpufreq_userspace cpufreq_stats cpufreq_powersave ipv6 nfsd exportfs nfs lockd nfs_acl auth_rpcgss
   sunrpc ext2 coretemp hp_wmi acpi_cpufreq loop snd_hda_codec_analog snd_hda_intel snd_hda_codec
   arc4 snd_pcm_oss snd_mixer_oss ecb snd_pcm snd_seq_dummy snd_seq_oss iwlagn iwlcore snd_seq_midi
   pcmcia mac80211 snd_rawmidi usblp snd_seq_midi_event snd_seq pcspkr cfg80211 yenta_socket
   rsrc_nonstatic pcmcia_core psmouse snd_timer snd_seq_device rfkill serio_raw snd soundcore
   snd_page_alloc hp_accel lis3lv02d video container output wmi intel_agp input_polldev battery ac
   processor button joydev evdev ext3 jbd mbcache sha256_generic aes_x86_64 aes_generic cbc usbhid hid
   dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sg sr_mod sd_mod cdrom ide_pci_generic piix
   ide_core pata_acpi uhci_hcd ata_piix ohci1394 sdhci_pci sdhci mmc_core led_class ieee1394 ricoh_mmc
   ata_generic ehci_hcd libta e1000e scsi_mod thermal fan thermal_sys [last unloaded: scsi_wait_scan]
Pid: 3226, comm: rpciod/0 Not tainted 2.6.31.1 #20 HP Compaq 2510p Notebook PC
RIP: 0010:[<ffffffffa0444ea2>]  [<ffffffffa0444ea2>]rpcauth_checkverf+0x4e/0x5a [sunrpc]
RSP: 0018:ffff88007aafbda0  EFLAGS: 00010246
RAX: 0000000400001000 RBX: ffff88003a718e40 RCX: 0000000000000001
RDX: ffff880038b821bc RSI: ffff880038b821c8 RDI: ffff8800618358c8
RBP: ffff88007aafbdc0 R08: 0000000000000000 R09: 0000000000000000
R10: ffff880001514d80 R11: ffff8800536401f0 R12: ffff8800618358c8
R13: ffff880038b821c8 R14: ffff880037bb4bd0 R15: ffffffffa04bf52b
FS:  0000000000000000(0000) GS:ffff880001504000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000400001038 CR3: 0000000067ee5000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rpciod/0 (pid: 3226, threadinfo ffff88007aafa000, task ffff88007c431670)
Stack:
 ffff88007aafbde0 ffff880037bb4bd0 ffff8800618358c8 ffff880061835958
<0> ffff88007aafbe00 ffffffffa043e24a ffff88007c4319e0 ffff8800618358c8
<0> ffff880061835970 ffff880061835958 0000000000000000 0000000000000001
Call Trace:
 [<ffffffffa043e24a>] call_decode+0x374/0x68e [sunrpc]
 [<ffffffffa044430e>] __rpc_execute+0x86/0x244 [sunrpc]
 [<ffffffffa04444f8>] ? rpc_async_schedule+0x0/0x12 [sunrpc]
 [<ffffffffa0444508>] rpc_async_schedule+0x10/0x12 [sunrpc]
 [<ffffffff81048bd5>] worker_thread+0x132/0x1ca
 [<ffffffff8104c657>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81048aa3>] ? worker_thread+0x0/0x1ca
 [<ffffffff8104c335>] kthread+0x8f/0x97
 [<ffffffff8100ca7a>] child_rip+0xa/0x20
 [<ffffffff8104c2a6>] ? kthread+0x0/0x97
 [<ffffffff8100ca70>] ? child_rip+0x0/0x20
Code: 30 0f b7 b7 06 01 00 00 48 89 d9 48 c7 c7 30 42
 45 a0 48 8b 40 10 48 8b 50 10 31 c0 e8 73 f8 e0 e0 48 8b 43 38 4c 89 ee 4c 89 e7 <ff> 50 38 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 41 55 49 89 f5
RIP  [<ffffffffa0444ea2>] rpcauth_checkverf+0x4e/0x5a [sunrpc]
 RSP <ffff88007aafbda0>
CR2: 0000000400001038

Not sure whether it's worth following up on that as a separate issue.

Cheers,
FJP

[1] KDE's task manager freezing for short periods is normal for me while
amarok is blocked by NFS. This normally only happens when I start amarok
for the first time, but it does explain how the NFS oops can have the
same effect.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Mon, Oct 05, 2009 at 08:50:58AM +0200, Frans Pop wrote:
> On Monday 05 October 2009, Frans Pop wrote:
> > I'll dig into this a bit more as it looks like this should be
> > reproducible, probably even without the kernel build. Next step is to
> > see how .30 behaves in the same situation.
> 
> This looks conclusive. I tested .30 and .32-rc3 from clean reboots and
> only starting gitk. I only started music playing in the background
> (amarok) from an NFS share to ensure network activity.
> 
> With .32-rc3 I got 4 SKB allocation errors while starting the *second* gitk
> instance. And the system was completely frozen with music stopped until gitk
> finished loading.
> 
> With .30 I was able to start *three* gitk's (which meant 2 of them got
> (partially) swapped out) without any allocation errors. And with the system
> remaining relatively responsive. There was a short break in the music while
> I started the 2nd instance, but it just continued playing afterwards. There
> was also some mild latency in the mouse cursor, but nothing like the full
> desktop freeze I get with .32-rc3.
> 
> With .30 I looked at /proc/buddyinfo while the 3rd gitk was being started,
> and that looked fairly healthy all the time:
> Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
> Node 0, zone    DMA32    579     67     25      8      5      1      1      0      1      1      0
> Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
> Node 0, zone    DMA32    276     54     13     15      8     10      3      1      1      1      0
> Node 0, zone      DMA      4      9     22     20     21     11      0      0      0      0      1
> Node 0, zone    DMA32    119     45     24     18     12      4      5      2      1      1      0
> Node 0, zone      DMA      4      9     22     20     21     11      0      0      0      0      1
> Node 0, zone    DMA32    527     13      9      5      5      3      2      1      1      1      0
> Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
> Node 0, zone    DMA32   1375     24      7      7      8      5      1      1      0      1      0
> Node 0, zone      DMA      5      9     22     20     21     11      0      0      0      0      1
> Node 0, zone    DMA32    329     21      3      3     17      8      5      1      0      1      0
> 
> With .32 it was obviously impossible to get that info due to the total
> freeze of the desktop. Not sure if the scheduler changes in .32 contribute
> to this. Guess I could find out by doing the same test with .31.
> 
> One thing I should mention: my swap is an LVM volume that's in a VG that's
> on a LUKS encrypted partition.
> 
> Does this give you enough info to go on, or should I try a bisection?
> 

I'll be trying to reproduce it, but it's unlikely I'll manage to
reproduce it reliably as there may be a specific combination of hardware
necessary as well. What I'm going to try is writing a module that
allocates order-5 every second GFP_ATOMIC and see can I reproduce using
scenarios similar to yours but it'll take some time with no guarantee of
success. If you could bisect it, it would be fantastic.

Thanks

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 05 October 2009, Mel Gorman wrote:
> On Mon, Oct 05, 2009 at 08:50:58AM +0200, Frans Pop wrote:
> > On Monday 05 October 2009, Frans Pop wrote:
> > > I'll dig into this a bit more as it looks like this should be
> > > reproducible, probably even without the kernel build. Next step is
> > > to see how .30 behaves in the same situation.
> >
> > This looks conclusive. I tested .30 and .32-rc3 from clean reboots and
> > only starting gitk. I only started music playing in the background
> > (amarok) from an NFS share to ensure network activity.
> >
> > With .32-rc3 I got 4 SKB allocation errors while starting the *second*
> > gitk instance. And the system was completely frozen with music stopped
> > until gitk finished loading.
> >
> > With .30 I was able to start *three* gitk's (which meant 2 of them got
> > (partially) swapped out) without any allocation errors. And with the
> > system remaining relatively responsive. There was a short break in the
> > music while I started the 2nd instance, but it just continued playing
> > afterwards. There was also some mild latency in the mouse cursor, but
> > nothing like the full desktop freeze I get with .32-rc3.
> >
> > One thing I should mention: my swap is an LVM volume that's in a VG
> > that's on a LUKS encrypted partition.
> >
> > Does this give you enough info to go on, or should I try a bisection?
>
> I'll be trying to reproduce it, but it's unlikely I'll manage to
> reproduce it reliably as there may be a specific combination of hardware
> necessary as well. What I'm going to try is writing a module that
> allocates order-5 every second GFP_ATOMIC and see can I reproduce using
> scenarios similar to yours but it'll take some time with no guarantee of
> success. If you could bisect it, it would be fantastic.

And the winner is:
2ff05b2b4eac2e63d345fc731ea151a060247f53 is first bad commit
commit 2ff05b2b4eac2e63d345fc731ea151a060247f53
Author: David Rientjes <rientjes@google.com>
Date:   Tue Jun 16 15:32:56 2009 -0700

    oom: move oom_adj value from task_struct to mm_struct

I'm confident that the bisection is good. The test case was very reliable 
while zooming in on the merge from akpm.

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[David Rientjes]

On Mon, 5 Oct 2009, Frans Pop wrote:

> And the winner is:
> 2ff05b2b4eac2e63d345fc731ea151a060247f53 is first bad commit
> commit 2ff05b2b4eac2e63d345fc731ea151a060247f53
> Author: David Rientjes <rientjes@google.com>
> Date:   Tue Jun 16 15:32:56 2009 -0700
> 
>     oom: move oom_adj value from task_struct to mm_struct
> 
> I'm confident that the bisection is good. The test case was very reliable 
> while zooming in on the merge from akpm.
> 

I doubt it for two reasons: (i) this commit was reverted in 0753ba0 since 
2.6.31-rc7 and is no longer in the kernel, and (ii) these are GFP_ATOMIC 
allocations which would be unaffected by oom killer scores.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[KOSAKI Motohiro]

> On Mon, 5 Oct 2009, Frans Pop wrote:
> 
> > And the winner is:
> > 2ff05b2b4eac2e63d345fc731ea151a060247f53 is first bad commit
> > commit 2ff05b2b4eac2e63d345fc731ea151a060247f53
> > Author: David Rientjes <rientjes@google.com>
> > Date:   Tue Jun 16 15:32:56 2009 -0700
> > 
> >     oom: move oom_adj value from task_struct to mm_struct
> > 
> > I'm confident that the bisection is good. The test case was very reliable 
> > while zooming in on the merge from akpm.
> > 
> 
> I doubt it for two reasons: (i) this commit was reverted in 0753ba0 since 
> 2.6.31-rc7 and is no longer in the kernel, and (ii) these are GFP_ATOMIC 
> allocations which would be unaffected by oom killer scores.

I agree. this patch is pretty obvious correct. it was reverted by
one unfortunately regression.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Mon, Oct 05, 2009 at 05:04:55PM -0700, David Rientjes wrote:
> On Mon, 5 Oct 2009, Frans Pop wrote:
> 
> > And the winner is:
> > 2ff05b2b4eac2e63d345fc731ea151a060247f53 is first bad commit
> > commit 2ff05b2b4eac2e63d345fc731ea151a060247f53
> > Author: David Rientjes <rientjes@google.com>
> > Date:   Tue Jun 16 15:32:56 2009 -0700
> > 
> >     oom: move oom_adj value from task_struct to mm_struct
> > 
> > I'm confident that the bisection is good. The test case was very reliable 
> > while zooming in on the merge from akpm.
> > 
> 
> I doubt it for two reasons: (i) this commit was reverted in 0753ba0 since 
> 2.6.31-rc7 and is no longer in the kernel, and (ii) these are GFP_ATOMIC 
> allocations which would be unaffected by oom killer scores.
> 

However, the problem was reported to start showing up in 2.6.31-rc1 so
while it might not be *the* patch, it might be making the type of change
that caused more fragmentation. This patch adjusted the size of
mm_struct and maybe it was enough to change the "order" required for the
slab. Maybe there are other slabs that have changed size as well in that
timeframe.

Frans, what is the size of mm_struct before and after this patch was
applied? Find it with either

grep mm_struct /proc/slabinfo

and if the information is not available there, try

cat /sys/kernel/slab/mm_struct/slab_size and
/sys/kernel/slab/mm_struct/order

Thanks

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[David Rientjes]

On Tue, 6 Oct 2009, Mel Gorman wrote:

> > > And the winner is:
> > > 2ff05b2b4eac2e63d345fc731ea151a060247f53 is first bad commit
> > > commit 2ff05b2b4eac2e63d345fc731ea151a060247f53
> > > Author: David Rientjes <rientjes@google.com>
> > > Date:   Tue Jun 16 15:32:56 2009 -0700
> > > 
> > >     oom: move oom_adj value from task_struct to mm_struct
> > > 
> > > I'm confident that the bisection is good. The test case was very reliable 
> > > while zooming in on the merge from akpm.
> > > 
> > 
> > I doubt it for two reasons: (i) this commit was reverted in 0753ba0 since 
> > 2.6.31-rc7 and is no longer in the kernel, and (ii) these are GFP_ATOMIC 
> > allocations which would be unaffected by oom killer scores.
> > 
> 
> However, the problem was reported to start showing up in 2.6.31-rc1 so
> while it might not be *the* patch, it might be making the type of change
> that caused more fragmentation. This patch adjusted the size of
> mm_struct and maybe it was enough to change the "order" required for the
> slab. Maybe there are other slabs that have changed size as well in that
> timeframe.
> 
> Frans, what is the size of mm_struct before and after this patch was
> applied? Find it with either
> 
> grep mm_struct /proc/slabinfo
> 
> and if the information is not available there, try
> 
> cat /sys/kernel/slab/mm_struct/slab_size and
> /sys/kernel/slab/mm_struct/order
> 

If that's the case and the problem still persists in 2.6.31-rc7 as 
reported, then you'd need to compare the current slab order for both 
mm_struct and signal_struct to the previously known working kernel 
since the latter is where oom_adj was moved.  (You'd still have to check 
the former to see if there were any mm_struct additions between rc1 and 
rc7 between the commit and revert, though.)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Tue, Oct 06, 2009 at 02:14:26AM -0700, David Rientjes wrote:
> On Tue, 6 Oct 2009, Mel Gorman wrote:
> 
> > > > And the winner is:
> > > > 2ff05b2b4eac2e63d345fc731ea151a060247f53 is first bad commit
> > > > commit 2ff05b2b4eac2e63d345fc731ea151a060247f53
> > > > Author: David Rientjes <rientjes@google.com>
> > > > Date:   Tue Jun 16 15:32:56 2009 -0700
> > > > 
> > > >     oom: move oom_adj value from task_struct to mm_struct
> > > > 
> > > > I'm confident that the bisection is good. The test case was very reliable 
> > > > while zooming in on the merge from akpm.
> > > > 
> > > 
> > > I doubt it for two reasons: (i) this commit was reverted in 0753ba0 since 
> > > 2.6.31-rc7 and is no longer in the kernel, and (ii) these are GFP_ATOMIC 
> > > allocations which would be unaffected by oom killer scores.
> > > 
> > 
> > However, the problem was reported to start showing up in 2.6.31-rc1 so
> > while it might not be *the* patch, it might be making the type of change
> > that caused more fragmentation. This patch adjusted the size of
> > mm_struct and maybe it was enough to change the "order" required for the
> > slab. Maybe there are other slabs that have changed size as well in that
> > timeframe.
> > 
> > Frans, what is the size of mm_struct before and after this patch was
> > applied? Find it with either
> > 
> > grep mm_struct /proc/slabinfo
> > 
> > and if the information is not available there, try
> > 
> > cat /sys/kernel/slab/mm_struct/slab_size and
> > /sys/kernel/slab/mm_struct/order
> > 
> 
> If that's the case and the problem still persists in 2.6.31-rc7 as 
> reported, then you'd need to compare the current slab order for both 
> mm_struct and signal_struct to the previously known working kernel 
> since the latter is where oom_adj was moved.  (You'd still have to check 
> the former to see if there were any mm_struct additions between rc1 and 
> rc7 between the commit and revert, though.)
> 

Best to just grab all of slabinfo for a poke around. I know task_struct
has increases in size since 2.6.29 but not enough on the machines I've
changed to make a difference to the order of pages requested. It might
be different on the problem machines.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Tuesday 06 October 2009, David Rientjes wrote:
> On Mon, 5 Oct 2009, Frans Pop wrote:
> > And the winner is:
> > 2ff05b2b4eac2e63d345fc731ea151a060247f53 is first bad commit
> > commit 2ff05b2b4eac2e63d345fc731ea151a060247f53
> > Author: David Rientjes <rientjes@google.com>
> > Date:   Tue Jun 16 15:32:56 2009 -0700
> >
> >     oom: move oom_adj value from task_struct to mm_struct
> >
> > I'm confident that the bisection is good. The test case was very
> > reliable while zooming in on the merge from akpm.
>
> I doubt it for two reasons: (i) this commit was reverted in 0753ba0
> since 2.6.31-rc7 and is no longer in the kernel, and (ii) these are
> GFP_ATOMIC allocations which would be unaffected by oom killer scores.

OK. Looks like I have been getting some false "good" results. I've been 
redoing part of the bisect and am getting close to a new candidate. Will 
explain further when I have that.

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

Sorry for going quiet on this issue for a few days, but I have been 
spending *a lot* of time on it. I've done what amounts to 5 bisection 
rounds at ~20 minutes per iteration and in total over 80 boots.

The problem with my first bisection was that there are *at least two* 
changes at the root of this issue, both committed between .30 and .30-rc1. 
Because of this a normal bisection will not lead to a reliable result and 
even with my last effort I can only narrow it down to two different areas, 
and not 100% to specific commits.

The two identified areas are:
1) a wireless merge which causes the SKB errors to appear in the first
   place, but not always;
2) an mm merge which makes the SKB errors occur *much* quicker; IMHO this
   is the change that also causes the regressions reported by Pekka and
   Karol.

So below my results. The issue is both complex and subtle. Now it's up to 
you, domain experts for both mm *and* wireless/networking, to make sense of 
it all and come up with suggestions on how to proceed.

I've improved my test and it's now a lot more reliable, but there are still 
timing influences. Also, because this is all merge-window stuff, I'm 
hitting quite a few minor and major regressions between commits that can 
affect tests.

Please study the information below carefully. I know it's long, but I think 
this issue justifies that.

On Monday 05 October 2009, Frans Pop wrote:
> This looks conclusive. I tested .30 and .32-rc3 from clean reboots and
> only starting gitk. I only started music playing in the background
> (amarok) from an NFS share to ensure network activity.
>
> With .32-rc3 I got 4 SKB allocation errors while starting the *second*
> gitk instance. And the system was completely frozen with music stopped
> until gitk finished loading.

With .32-rc3, .31.1 and vanilla .31 I will get multiple SKB allocation 
errors the *first time* I run the test, *every* time.

> With .30 I was able to start *three* gitk's (which meant 2 of them got
> (partially) swapped out) without any allocation errors. And with the
> system remaining relatively responsive. There was a short break in the
> music while I started the 2nd instance, but it just continued playing
> afterwards. There was also some mild latency in the mouse cursor, but
> nothing like the full desktop freeze I get with .32-rc3.

With both .30.2 and vanilla .30 I have *never* been able to get any SKB 
allocation errors. No matter how often I repeat the test.

So, the start and end position are 100% reproducible. Problem is that this 
changes during the bisection. At some point the test will fail (no SKB 
errors) the first time I run it, but it will fail on the second or third 
attempt.
Apparently at some point memory must already be fragmented (or higher 
orders already used up) to some extend for the errors to trigger.

TEST METHOD
-----------
As a normal bisection (I tried 3 times...) did not lead anywhere, I had to 
think of an alternative approach. I decided to start by manually selecting 
merges by Linus into mainline. The advantage is that that makes the 
bisection linear and makes it a lot easier to see patterns.
After narrowing down to a specific merge, I bisected (again semi-manually) 
inside that merge.

Because I suspected there were multiple changes involved, I deliberately 
tried to find two points:
- where do I first start seeing SKB errors at all, even if it is only at
  the second or third try;
- where do I start getting SKB errors reliably on the first try.

I worked from "good" to "bad", i.e. I started at .30. The merges were not 
chosen completely randomly. From the first 3 bisections I strongly 
suspected the first 'net-next' merge and the first 'akpm' merge, but I did 
make sure to confirm that suspicion.

TEST DESCRIPTION
----------------
The test I've ended up using is:
1) clean boot
2) start music in amarok from NFS share; use very long song to avoid file
   changes and thus ensure a fluent stream of network data during the test
3) start 'gitk v2.6.29..master &' - to use up some memory
4) start first 'gitk master &' - after this all normal memory is as good as
   used up, with minor swap; this never resulted in SKB errors
5) start second 'gitk master &' - this causes heavy swapping (>700 MB) and
   is the real test
6) if there were no SKB errors after 5), kill the gitk processes and repeat
   steps 3) to 5). I've done this up to 4 times in some cases
7) if the results are not clear or when there is doubt later, repeat from
   step 1) with same kernel

Memory after initial 'gitk v2.6.29..master &':
             total       used       free     shared    buffers     cached
Mem:       2030776    1153008     877768          0      41572     333968
-/+ buffers/cache:     777468    1253308
Swap:      2097144          0    2097144

Memory after first 'gitk master &':
             total       used       free     shared    buffers     cached
Mem:       2030776    1979040      51736          0      35684     238420
-/+ buffers/cache:    1704936     325840
Swap:      2097144      21876    2075268

Memory after second 'gitk master &' (with .30.2):
             total       used       free     shared    buffers     cached
Mem:       2030776    2011608      19168          0      21836      92336
-/+ buffers/cache:    1897436     133340
Swap:      2097144     776160    1320984

OVERVIEW OF RESULTS
-------------------
Below I list the most relevant merges and commits. Note that they are 
listed in commit order; my kernel version shows the order of testing.

For the commits I tested the test results are listed on the next line.
The first number on that line consists of the test series + the iteration 
(and also identifies the kernel I used).
A "+" means I got no SKB errors, a "-" that I did get them. A "|" means I 
rebooted for a second series of tests.

v2.6.30-2330-gdb8e7f1	'x86-fixes-for-linus' of linux-2.6-tip
	1.1	+++	iwlagn sw-error during first test
v2.6.30-4127-g0fa2133	'merge' of powerpc (last merge before net-next-2.6)
	1.2	+++
v2.6.30-5398-g2ed0e21	net-next-2.6 (mega-merge!)
	1.4	+-	system reboot fails after testing
v2.6.30-5517-g609106b	'merge' of powerpc
	1.3	+-	system reboot fails after testing
v2.6.30-5927-gf83b1e6	'for-linus' of linux1394-2.6 (last merge before akpm)
	2.2	++-
v2.6.30-6111-g517d086	'akpm'
	2.1	-|-

BISECTION OF net-next-2.6 MERGE
-------------------------------
Note that this merge was based not on .30 vanilla, but partly on 
v2.6.30-rc1 and partly on v2.6.30-rc6.
I think this had an influence on the latencies I saw (i.e. because some 
post-rc6 bug fixes were not present it changes the general behavior of the 
system during the swapping). For example: with v2.6.30-4127-g0fa2133 the 
system remained more responsive (smaller music skips) than with 
v2.6.30-rc1-1219-g82d0481.

I started again by testing merges, this time those by David.

v2.6.30-rc1-1219-g82d0481	'master' of wireless-next-2.6
	1.5	++++	bad latencies
v2.6.30-rc6-660-gbb803cf	'master' of net-2.6
v2.6.30-rc6-808-g45ea4ea	'master' of wireless-next-2.6
v2.6.30-rc6-850-gc649c0e	'master' of net-2.6
v2.6.30-rc6-922-g3f1f39c	'linux-2.6.31.y' of wimax
v2.6.30-rc6-999-gb2f8f75	'master' of net-2.6
v2.6.30-rc6-1028-ga8c617e	'net-next' of lksctp-dev
	1.7	++++|++++|++++
	I went back to this one twice because the bisection inside the
	next merge (see below) did not give a clear result.
v2.6.30-rc6-1103-gb1bc81a	'master' of wireless-next-2.6
	1.8	+-
v2.6.30-rc6-1224-g84503dd	'master' of wireless-next-2.6
	1.6	+-

So the problem started in the v2.6.30-rc6-1103-gb1bc81a merge.
I was unable to narrow it down to an exact commit; AFAICT the remaining 
ones (between v2.6.30-rc6-1028-g8fc0fee and v2.6.30-rc6-1032-g7ba10a8) are 
uninteresting. But it *must* be in this area!

For a good overview of the area, use 'gitk 3f1f39c4..b1bc81a0'.

v2.6.30-rc6-1028-g8fc0fee	cfg80211: use key size constants
	1.11	++++
v2.6.30-rc6-1031-g1bb5633	iwmc3200wifi: fix printk format
	1.14	+++-	not quite conclusive...
v2.6.30-rc6-1032-g7ba10a8	mac80211: fix transposed min/max CW values
	1.13	-
	This is a bugfix for aa837ee1d from an earlier merge! Could this maybe
	influence the test results in between? There are various SKB related
	changes there, for example: dfbf97f3..e5b9215e.
v2.6.30-rc6-1037-g2c5b9e5	wireless: libertas: fix unaligned accesses
	1.12	+-
v2.6.30-rc6-1044-g729e9c7	cfg80211: fix for duplicate userspace replies
	1.10	+-
v2.6.30-rc6-1075-gc587de0	iwlwifi: unify station management
	1.9	++-|+-
v2.6.30-rc6-1076-gd14d444	iwl3945: port allow skb allocation in tasklet
	I thought this was a prime candidate, but as you can see several commits
	before failed too. Still worth looking at I think!

BISECTION of akpm (mm) MERGE
----------------------------
So here I went looking for "where does the test start failing on the first 
try". Again, I was unable to narrow it down to a single commit.

For a good overview of the area, use 'gitk f83b1e61..517d0869'.

v2.6.30-5466-ga1dd268	mm: use alloc_pages_exact in alloc_large_system_hash
	2.3	+-
v2.6.30-5478-ge9bb35d	mm: setup_per_zone_inactive_ratio - fix comment and..
	2.5	+-
v2.6.30-5486-g35282a2	migration: only migrate_prep() once per move_pages()
	2.6	-|+|-	not quite conclusive...
v2.6.30-5492-gbce7394	page-allocator: reset wmark_min and inactive ratio..
	2.4	-|-

WHERE NEXT?
===========
I think the results confirm there is definitely an issue here and that my 
test is reliable and consistent enough to show it. And as it currently is 
the only test we have...

I hope that the info above is enough for the mm and wireless domain 
experts to identify likely candidates in the areas I've identified.

The next step could be trying specific reverts or debug patches, either on 
top of current git, or 2.6.31, or inside the identified areas.
I'll run anything you care to throw at me and will try to provide any 
additional info you need, but at this point it's up to you.

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 12 October 2009, Frans Pop wrote:
> BISECTION of akpm (mm) MERGE
> ----------------------------
> So here I went looking for "where does the test start failing on the
> first try". Again, I was unable to narrow it down to a single commit.

Note that this merge is based on mainline at v2.6.30-5415-g03347e2, so a
number of merges "drop out" once I  started bisecting into this merge. But
that point is still *after* the net-next-2.6 merge, which is all that's
really relevant for this issue.

> For a good overview of the area, use 'gitk f83b1e61..517d0869'.
>
> v2.6.30-5466-ga1dd268   mm: use alloc_pages_exact in alloc_large_system_hash
>	2.3     +- 
> v2.6.30-5478-ge9bb35d   mm: setup_per_zone_inactive_ratio - fix comment and..
>	2.5     +- 
> v2.6.30-5486-g35282a2   migration: only migrate_prep() once per move_pages()
>	2.6     -|+|-   not quite conclusive... 
> v2.6.30-5492-gbce7394   page-allocator: reset wmark_min and inactive ratio..
>	2.4     -|- 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Mon, Oct 12, 2009 at 01:10:25AM +0200, Frans Pop wrote:
> Sorry for going quiet on this issue for a few days, but I have been 
> spending *a lot* of time on it. I've done what amounts to 5 bisection 
> rounds at ~20 minutes per iteration and in total over 80 boots.
> 
> The problem with my first bisection was that there are *at least two* 
> changes at the root of this issue, both committed between .30 and .30-rc1. 
> Because of this a normal bisection will not lead to a reliable result and 
> even with my last effort I can only narrow it down to two different areas, 
> and not 100% to specific commits.
> 

Thanks very much for your detailed work on this.

> The two identified areas are:
> 1) a wireless merge which causes the SKB errors to appear in the first
>    place, but not always;
> 2) an mm merge which makes the SKB errors occur *much* quicker; IMHO this
>    is the change that also causes the regressions reported by Pekka and
>    Karol.
> 
> So below my results. The issue is both complex and subtle. Now it's up to 
> you, domain experts for both mm *and* wireless/networking, to make sense of 
> it all and come up with suggestions on how to proceed.
> 
> I've improved my test and it's now a lot more reliable, but there are still 
> timing influences.

The timing influences is probably because kswapd is working from the
time memory gets full. High-order allocation failures would cause it to
start reclaiming at that order so it's a race always to see can it do
its work before an atomic allocation fails or not.

> Also, because this is all merge-window stuff, I'm 
> hitting quite a few minor and major regressions between commits that can 
> affect tests.
> 
> Please study the information below carefully. I know it's long, but I think 
> this issue justifies that.
> 

Agreed. I'll be looking at commits, both wireless and mm but obviously
anything I saw about wireless needs to be taken with a generous dose of
salt.

> On Monday 05 October 2009, Frans Pop wrote:
> > This looks conclusive. I tested .30 and .32-rc3 from clean reboots and
> > only starting gitk. I only started music playing in the background
> > (amarok) from an NFS share to ensure network activity.
> >
> > With .32-rc3 I got 4 SKB allocation errors while starting the *second*
> > gitk instance. And the system was completely frozen with music stopped
> > until gitk finished loading.
> 
> With .32-rc3, .31.1 and vanilla .31 I will get multiple SKB allocation 
> errors the *first time* I run the test, *every* time.
> 

So, this remains a current problem that wasn't solved by accident.

> > With .30 I was able to start *three* gitk's (which meant 2 of them got
> > (partially) swapped out) without any allocation errors. And with the
> > system remaining relatively responsive. There was a short break in the
> > music while I started the 2nd instance, but it just continued playing
> > afterwards. There was also some mild latency in the mouse cursor, but
> > nothing like the full desktop freeze I get with .32-rc3.
> 
> With both .30.2 and vanilla .30 I have *never* been able to get any SKB 
> allocation errors. No matter how often I repeat the test.
> 
> So, the start and end position are 100% reproducible. Problem is that this 
> changes during the bisection. At some point the test will fail (no SKB 
> errors) the first time I run it, but it will fail on the second or third 
> attempt.
> Apparently at some point memory must already be fragmented (or higher 
> orders already used up) to some extend for the errors to trigger.
> 

That is a reasonable assessment. It could be because

1. Something in the intevening commits greatly increases the number of
   GFP_ATOMIC allocations that are occuring. It's a pity that the allocator
   tracepoints are not available in those kernels. It would have made
   investigating this theory easier.

2. kswapd is no longer reclaiming high-order pages as well as it used
   to be it due to changes in kswapd itself or lumpy reclaim

3. Fragmentation avoidance has been broken in some subtle manner

I think 3 is particularly unlikely and am expecting it to be 1 or 2.

> TEST METHOD
> -----------
> As a normal bisection (I tried 3 times...) did not lead anywhere, I had to 
> think of an alternative approach. I decided to start by manually selecting 
> merges by Linus into mainline. The advantage is that that makes the 
> bisection linear and makes it a lot easier to see patterns.
> After narrowing down to a specific merge, I bisected (again semi-manually) 
> inside that merge.
> 
> Because I suspected there were multiple changes involved, I deliberately 
> tried to find two points:
> - where do I first start seeing SKB errors at all, even if it is only at
>   the second or third try;
> - where do I start getting SKB errors reliably on the first try.
> 
> I worked from "good" to "bad", i.e. I started at .30. The merges were not 
> chosen completely randomly. From the first 3 bisections I strongly 
> suspected the first 'net-next' merge and the first 'akpm' merge, but I did 
> make sure to confirm that suspicion.
> 

A very good approach.

> TEST DESCRIPTION
> ----------------
> The test I've ended up using is:
> 1) clean boot
> 2) start music in amarok from NFS share; use very long song to avoid file
>    changes and thus ensure a fluent stream of network data during the test
> 3) start 'gitk v2.6.29..master &' - to use up some memory
> 4) start first 'gitk master &' - after this all normal memory is as good as
>    used up, with minor swap; this never resulted in SKB errors
> 5) start second 'gitk master &' - this causes heavy swapping (>700 MB) and
>    is the real test
> 6) if there were no SKB errors after 5), kill the gitk processes and repeat
>    steps 3) to 5). I've done this up to 4 times in some cases
> 7) if the results are not clear or when there is doubt later, repeat from
>    step 1) with same kernel
> 
> Memory after initial 'gitk v2.6.29..master &':
>              total       used       free     shared    buffers     cached
> Mem:       2030776    1153008     877768          0      41572     333968
> -/+ buffers/cache:     777468    1253308
> Swap:      2097144          0    2097144
> 
> Memory after first 'gitk master &':
>              total       used       free     shared    buffers     cached
> Mem:       2030776    1979040      51736          0      35684     238420
> -/+ buffers/cache:    1704936     325840
> Swap:      2097144      21876    2075268
> 
> Memory after second 'gitk master &' (with .30.2):
>              total       used       free     shared    buffers     cached
> Mem:       2030776    2011608      19168          0      21836      92336
> -/+ buffers/cache:    1897436     133340
> Swap:      2097144     776160    1320984
> 
> OVERVIEW OF RESULTS
> -------------------
> Below I list the most relevant merges and commits. Note that they are 
> listed in commit order; my kernel version shows the order of testing.
> 
> For the commits I tested the test results are listed on the next line.
> The first number on that line consists of the test series + the iteration 
> (and also identifies the kernel I used).
> A "+" means I got no SKB errors, a "-" that I did get them. A "|" means I 
> rebooted for a second series of tests.
> 
> v2.6.30-2330-gdb8e7f1	'x86-fixes-for-linus' of linux-2.6-tip
> 	1.1	+++	iwlagn sw-error during first test
> v2.6.30-4127-g0fa2133	'merge' of powerpc (last merge before net-next-2.6)
> 	1.2	+++
> v2.6.30-5398-g2ed0e21	net-next-2.6 (mega-merge!)
> 	1.4	+-	system reboot fails after testing
> v2.6.30-5517-g609106b	'merge' of powerpc
> 	1.3	+-	system reboot fails after testing
> v2.6.30-5927-gf83b1e6	'for-linus' of linux1394-2.6 (last merge before akpm)
> 	2.2	++-
> v2.6.30-6111-g517d086	'akpm'
> 	2.1	-|-
> 
> BISECTION OF net-next-2.6 MERGE
> -------------------------------
> Note that this merge was based not on .30 vanilla, but partly on 
> v2.6.30-rc1 and partly on v2.6.30-rc6.
> I think this had an influence on the latencies I saw (i.e. because some 
> post-rc6 bug fixes were not present it changes the general behavior of the 
> system during the swapping). For example: with v2.6.30-4127-g0fa2133 the 
> system remained more responsive (smaller music skips) than with 
> v2.6.30-rc1-1219-g82d0481.
> 
> I started again by testing merges, this time those by David.
> 
> v2.6.30-rc1-1219-g82d0481	'master' of wireless-next-2.6
> 	1.5	++++	bad latencies

The bad latencies might imply that there are a lot more allocations
going on than there used to be. Maybe it was just because of a wireless
bug though that was later fixed.

> v2.6.30-rc6-660-gbb803cf	'master' of net-2.6
> v2.6.30-rc6-808-g45ea4ea	'master' of wireless-next-2.6
> v2.6.30-rc6-850-gc649c0e	'master' of net-2.6
> v2.6.30-rc6-922-g3f1f39c	'linux-2.6.31.y' of wimax
> v2.6.30-rc6-999-gb2f8f75	'master' of net-2.6
> v2.6.30-rc6-1028-ga8c617e	'net-next' of lksctp-dev
> 	1.7	++++|++++|++++
> 	I went back to this one twice because the bisection inside the
> 	next merge (see below) did not give a clear result.
> v2.6.30-rc6-1103-gb1bc81a	'master' of wireless-next-2.6
> 	1.8	+-
> v2.6.30-rc6-1224-g84503dd	'master' of wireless-next-2.6
> 	1.6	+-
> 
> So the problem started in the v2.6.30-rc6-1103-gb1bc81a merge.
> I was unable to narrow it down to an exact commit; AFAICT the remaining 
> ones (between v2.6.30-rc6-1028-g8fc0fee and v2.6.30-rc6-1032-g7ba10a8) are 
> uninteresting. But it *must* be in this area!
> 
> For a good overview of the area, use 'gitk 3f1f39c4..b1bc81a0'.
> 
> v2.6.30-rc6-1028-g8fc0fee	cfg80211: use key size constants
> 	1.11	++++
> v2.6.30-rc6-1031-g1bb5633	iwmc3200wifi: fix printk format
> 	1.14	+++-	not quite conclusive...
> v2.6.30-rc6-1032-g7ba10a8	mac80211: fix transposed min/max CW values
> 	1.13	-
> 	This is a bugfix for aa837ee1d from an earlier merge! Could this maybe
> 	influence the test results in between? There are various SKB related
> 	changes there, for example: dfbf97f3..e5b9215e.

Maybe. Your commit id's are different to what I see. Maybe it's because your
tree has been shuffled around a bit but after some digging around in this
general area, I saw this patch

4752c93c30 iwlcore: Allow skb allocation from tasklet

This patch increases the number of GFP_ATOMIC allocations that can occur by
allocating GFP_ATOMIC in some cases and GFP_KERNEL in others. Previously,
only GFP_KERNEL was used and I didn't realise this allocation method was
so recent. Problems of this sort have cropped up before and while there
are later changes that suppress some of these warnings, I believe this is
a strong candidate for where the allocation failures started appearing.

> v2.6.30-rc6-1037-g2c5b9e5	wireless: libertas: fix unaligned accesses
> 	1.12	+-
> v2.6.30-rc6-1044-g729e9c7	cfg80211: fix for duplicate userspace replies
> 	1.10	+-
> v2.6.30-rc6-1075-gc587de0	iwlwifi: unify station management
> 	1.9	++-|+-
> v2.6.30-rc6-1076-gd14d444	iwl3945: port allow skb allocation in tasklet
> 	I thought this was a prime candidate, but as you can see several commits
> 	before failed too. Still worth looking at I think!
> 

Your commit IDs are different to what I see but it's the commit merge at
b1bc81a0ef86b86fa410dd303d84c8c7bd09a64d. I agree that the last commit
(d14d44407b9f06e3cf967fcef28ccb780caf0583) could make the problem worse
because it expands the use of GFP_ATOMIC for another driver.

> BISECTION of akpm (mm) MERGE
> ----------------------------
> So here I went looking for "where does the test start failing on the first 
> try". Again, I was unable to narrow it down to a single commit.
> 
> For a good overview of the area, use 'gitk f83b1e61..517d0869'.
> 
> v2.6.30-5466-ga1dd268	mm: use alloc_pages_exact in alloc_large_system_hash
> 	2.3	+-
> v2.6.30-5478-ge9bb35d	mm: setup_per_zone_inactive_ratio - fix comment and..
> 	2.5	+-
> v2.6.30-5486-g35282a2	migration: only migrate_prep() once per move_pages()
> 	2.6	-|+|-	not quite conclusive...
> v2.6.30-5492-gbce7394	page-allocator: reset wmark_min and inactive ratio..
> 	2.4	-|-
> 

While I didn't spot anything too out of the ordinary here, they did occur
shortly after a number of other page allocator related patches.  One small
thing I noticed there is that kswapd is getting woken up less now than it did
previously. Generally, I wouldn't have expected it to make a difference but
it's possible that kswapd is not being woken up to reclaim at a higher order
than it was previously. I have a patch for this below. It'd be nice if you
could apply it and see do fewer allocation failures occur on current mainline.

> WHERE NEXT?
> ===========
> I think the results confirm there is definitely an issue here and that my 
> test is reliable and consistent enough to show it. And as it currently is 
> the only test we have...
> 
> I hope that the info above is enough for the mm and wireless domain 
> experts to identify likely candidates in the areas I've identified.
> 
> The next step could be trying specific reverts or debug patches, either on 
> top of current git, or 2.6.31, or inside the identified areas.
> I'll run anything you care to throw at me and will try to provide any 
> additional info you need, but at this point it's up to you.
> 

For the wireless people in mainline - iwl_rx_replenish_now() is doing
a GFP_ATOMIC allocation that does not use __GFP_NOWARN. As part of
investigating allocation failures, iwl_rx_allocate() was taught to
distinguish between a benign and serious allocation failure - serious
being there are very few RX buffers left and packet loss could occur soon
(see commit f82a924cc88a5541df1d4b9d38a0968cd077a051). I think this GFP mask
should be made GFP_ATOMIC|__GFP_NOWARN so that warnings only appear when the
failure is serious, dump stack after the warning if you need it. I have a
feeling that almost all these warnings have been benign and are related to
the introduction of GFP_ATOMIC being used so heavily to move more expensive
allocations to the tasklet (presumably to reduce user-visible latency).

Frans, could you try the following kswapd-related patch please? I'd be
interested in seeing if the number of allocation failure warnings are
reduced with it. After that, could you edit
drivers/net/wireless/iwlwifi/iwl-rx.c and make the GFP_ATOMIC in
iwl_rx_replenish_now() GFP_ATOMIC|__GFP_NOWARN and see do any of the
"serious" allocation failure messages appear.

Thanks again for your persistence.

==== CUT HERE ====

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 12 October 2009, Mel Gorman wrote:
> Maybe. Your commit id's are different to what I see. Maybe it's because
> your tree has been shuffled around a bit

No, the commit IDs should be identical. My tree is just plain mainline.

Just to make sure... You did remove the "g" from the IDs, right?
So v2.6.30-rc6-1103-gb1bc81a becomes 'b1bc81a' and if you do
'git describe b1bc81a' you really should end up with the same IDs I have.

> but after some digging around in this general area, I saw this patch
>
> 4752c93c30 iwlcore: Allow skb allocation from tasklet

That is v2.6.30-rc6-773-g4752c93, which is part of the first wireless
merge I tested and where I saw no issues. But see below.

> This patch increases the number of GFP_ATOMIC allocations that can occur
> by allocating GFP_ATOMIC in some cases and GFP_KERNEL in others.
> Previously, only GFP_KERNEL was used and I didn't realise this
> allocation method was so recent. Problems of this sort have cropped up
> before and while there are later changes that suppress some of these
> warnings, I believe this is a strong candidate for where the allocation
> failures started appearing.
>
> > v2.6.30-rc6-1032-g7ba10a8       mac80211: fix transposed min/max CW values
> >     1.13    -
> >     This is a bugfix for aa837ee1d from an earlier merge! Could this maybe

There's a typo here. That ID should be: aa837e1d.

> >     influence the test results in between? There are various SKB related
> >     changes there, for example: dfbf97f3..e5b9215e.
> > v2.6.30-rc6-1037-g2c5b9e5	wireless: libertas: fix unaligned accesses
> > 	1.12    +-
> > v2.6.30-rc6-1044-g729e9c7	cfg80211: fix for duplicate userspace replies
> >     1.10    +- 
> > v2.6.30-rc6-1075-gc587de0	iwlwifi: unify station management
> >     1.9     ++-|+-
> > v2.6.30-rc6-1076-gd14d444	iwl3945: port allow skb allocation in tasklet
> >     I thought this was a prime candidate, but as you can see 
> >     several commits before failed too. Still worth looking at I think!
>
> Your commit IDs are different to what I see but it's the commit merge at
> b1bc81a0ef86b86fa410dd303d84c8c7bd09a64d. I agree that the last commit
> (d14d44407b9f06e3cf967fcef28ccb780caf0583) could make the problem worse
> because it expands the use of GFP_ATOMIC for another driver.

No, that was a mistake of mine. d14d444 is in a driver I don't even compile.
The one you identified (which is the same change for iwlagn) is much more
interesting.

I really do think that v2.6.30-rc6-1032-g7ba10a8 could play a role here.
That's a fix for v2.6.30-rc1-1131-gaa837e1. So that bug was introduced
_before_ the merge 82d0481 and may thus well explain both the latencies I
saw _and_ why that merge tested without problems. And that would also go a
long way to explain my test results.
So I'm going to retest 82d0481 with 7ba10a8 cherry-picked on top.

> > BISECTION of akpm (mm) MERGE
> > ----------------------------
[...]
> While I didn't spot anything too out of the ordinary here, they did
> occur shortly after a number of other page allocator related patches. 
> One small thing I noticed there is that kswapd is getting woken up less
> now than it did previously. Generally, I wouldn't have expected it to
> make a difference but it's possible that kswapd is not being woken up to
> reclaim at a higher order than it was previously. I have a patch for
> this below. It'd be nice if you could apply it and see do fewer
> allocation failures occur on current mainline.

I'll give that patch a try and report back.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Mon, Oct 12, 2009 at 07:32:11PM +0200, Frans Pop wrote:
> On Monday 12 October 2009, Mel Gorman wrote:
> > Maybe. Your commit id's are different to what I see. Maybe it's because
> > your tree has been shuffled around a bit
> 
> No, the commit IDs should be identical. My tree is just plain mainline.
> 
> Just to make sure... You did remove the "g" from the IDs, right?
> So v2.6.30-rc6-1103-gb1bc81a becomes 'b1bc81a' and if you do
> 'git describe b1bc81a' you really should end up with the same IDs I have.
> 

Bah, that's what I was doing all right. No excuse, that was just plain
stupid of me.

> > but after some digging around in this general area, I saw this patch
> >
> > 4752c93c30 iwlcore: Allow skb allocation from tasklet
> 
> That is v2.6.30-rc6-773-g4752c93, which is part of the first wireless
> merge I tested and where I saw no issues. But see below.
> 

While there were no issues at that point, I think it might have been the
beginning of a few patches that made things progressively worse. It is
possible there is more than one patch causing trouble here and bisecting
each of them is unlikely to be an option. More on this later though.

> > This patch increases the number of GFP_ATOMIC allocations that can occur
> > by allocating GFP_ATOMIC in some cases and GFP_KERNEL in others.
> > Previously, only GFP_KERNEL was used and I didn't realise this
> > allocation method was so recent. Problems of this sort have cropped up
> > before and while there are later changes that suppress some of these
> > warnings, I believe this is a strong candidate for where the allocation
> > failures started appearing.
> >
> > > v2.6.30-rc6-1032-g7ba10a8       mac80211: fix transposed min/max CW values
> > >     1.13    -
> > >     This is a bugfix for aa837ee1d from an earlier merge! Could this maybe
> 
> There's a typo here. That ID should be: aa837e1d.
> 
> > >     influence the test results in between? There are various SKB related
> > >     changes there, for example: dfbf97f3..e5b9215e.
> > > v2.6.30-rc6-1037-g2c5b9e5	wireless: libertas: fix unaligned accesses
> > > 	1.12    +-
> > > v2.6.30-rc6-1044-g729e9c7	cfg80211: fix for duplicate userspace replies
> > >     1.10    +- 
> > > v2.6.30-rc6-1075-gc587de0	iwlwifi: unify station management
> > >     1.9     ++-|+-
> > > v2.6.30-rc6-1076-gd14d444	iwl3945: port allow skb allocation in tasklet
> > >     I thought this was a prime candidate, but as you can see 
> > >     several commits before failed too. Still worth looking at I think!
> >
> > Your commit IDs are different to what I see but it's the commit merge at
> > b1bc81a0ef86b86fa410dd303d84c8c7bd09a64d. I agree that the last commit
> > (d14d44407b9f06e3cf967fcef28ccb780caf0583) could make the problem worse
> > because it expands the use of GFP_ATOMIC for another driver.
> 
> No, that was a mistake of mine. d14d444 is in a driver I don't even compile.
> The one you identified (which is the same change for iwlagn) is much more
> interesting.
> 

I had forgotten what model your card was and assumed it must have been based
on this driver for the problem to get worse for you that point.

> I really do think that v2.6.30-rc6-1032-g7ba10a8 could play a role here.
> That's a fix for v2.6.30-rc1-1131-gaa837e1. So that bug was introduced
> _before_ the merge 82d0481 and may thus well explain both the latencies I
> saw _and_ why that merge tested without problems. And that would also go a
> long way to explain my test results.

Very good point.

> So I'm going to retest 82d0481 with 7ba10a8 cherry-picked on top.
> 

Great.

> > > BISECTION of akpm (mm) MERGE
> > > ----------------------------
> [...]
> > While I didn't spot anything too out of the ordinary here, they did
> > occur shortly after a number of other page allocator related patches. 
> > One small thing I noticed there is that kswapd is getting woken up less
> > now than it did previously. Generally, I wouldn't have expected it to
> > make a difference but it's possible that kswapd is not being woken up to
> > reclaim at a higher order than it was previously. I have a patch for
> > this below. It'd be nice if you could apply it and see do fewer
> > allocation failures occur on current mainline.
> 
> I'll give that patch a try and report back.
> 

Thanks a lot.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 12 October 2009, Frans Pop wrote:
> On Monday 12 October 2009, Mel Gorman wrote:
> > but after some digging around in this general area, I saw this patch
> >
> > 4752c93c30 iwlcore: Allow skb allocation from tasklet
>
> That is v2.6.30-rc6-773-g4752c93, which is part of the first wireless
> merge I tested and where I saw no issues. But see below.
>
> > This patch increases the number of GFP_ATOMIC allocations that can
> > occur by allocating GFP_ATOMIC in some cases and GFP_KERNEL in others.
> > Previously, only GFP_KERNEL was used and I didn't realise this
> > allocation method was so recent. Problems of this sort have cropped up
> > before and while there are later changes that suppress some of these
> > warnings, I believe this is a strong candidate for where the
> > allocation failures started appearing.

I have tried reverting this patch and that does make a significant 
difference, but the results are still not really conclusive.
I tested the revert on top of:
- the first net-next-2.6 merge (2ed0e21), i.e. before the mm merge
- 2.6.31.1

In both cases I no longer get SKB errors, but instead (?) I get firmware 
errors:
iwlagn 0000:10:00.0: Microcode SW error detected.  Restarting 0x2000000.

So on the wireless side it does look as if there is more than one change 
involved. Remember that with .30 I don't get any errors, only relatively 
mild latencies and skips in the music.

> I really do think that v2.6.30-rc6-1032-g7ba10a8 could play a role here.
> That's a fix for v2.6.30-rc1-1131-gaa837e1. So that bug was introduced
> _before_ the merge 82d0481 and may thus well explain both the latencies
> I saw _and_ why that merge tested without problems. And that would also
> go a long way to explain my test results.
> So I'm going to retest 82d0481 with 7ba10a8 cherry-picked on top.
                         ^^^^^^^-- should be 45ea4ea

I've tried this but still don't get any SKB errors, so that bug does not 
seem to make a difference.

> > > BISECTION of akpm (mm) MERGE
> > > ----------------------------
> > While I didn't spot anything too out of the ordinary here, they did
> > occur shortly after a number of other page allocator related patches.
> > One small thing I noticed there is that kswapd is getting woken up
> > less now than it did previously. Generally, I wouldn't have expected
> > it to make a difference but it's possible that kswapd is not being
> > woken up to reclaim at a higher order than it was previously. I have a
> > patch for this below. It'd be nice if you could apply it and see do
> > fewer allocation failures occur on current mainline.
>
> I'll give that patch a try and report back.

With your patch on .32-rc4 I still get the SKB errors, so it does not seem 
to help. The only change there may have been is that the desktop was 
frozen longer than without the patch, but that is an impression, not a 
hard fact.


Although identifying the problem on the wireless side is important, I still 
feel that tracing the mm change should have priority as it influences much 
more than just iwlagn, as the other reports prove.

> > After that, could you edit drivers/net/wireless/iwlwifi/iwl-rx.c and
> > make the GFP_ATOMIC in iwl_rx_replenish_now() GFP_ATOMIC|__GFP_NOWARN
> > and see do any of the "serious" allocation failure messages appear.

For the above reason I've not yet tried this. It seems to me that this 
change will not really solve anything, but just suppress errors.

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Tue, Oct 13, 2009 at 10:38:37PM +0200, Frans Pop wrote:
> On Monday 12 October 2009, Frans Pop wrote:
> > On Monday 12 October 2009, Mel Gorman wrote:
> > > but after some digging around in this general area, I saw this patch
> > >
> > > 4752c93c30 iwlcore: Allow skb allocation from tasklet
> >
> > That is v2.6.30-rc6-773-g4752c93, which is part of the first wireless
> > merge I tested and where I saw no issues. But see below.
> >
> > > This patch increases the number of GFP_ATOMIC allocations that can
> > > occur by allocating GFP_ATOMIC in some cases and GFP_KERNEL in others.
> > > Previously, only GFP_KERNEL was used and I didn't realise this
> > > allocation method was so recent. Problems of this sort have cropped up
> > > before and while there are later changes that suppress some of these
> > > warnings, I believe this is a strong candidate for where the
> > > allocation failures started appearing.
> 
> I have tried reverting this patch and that does make a significant 
> difference, but the results are still not really conclusive.
> I tested the revert on top of:
> - the first net-next-2.6 merge (2ed0e21), i.e. before the mm merge
> - 2.6.31.1
> 

I think this is very significant. Either that change needs to be backed
out or more likely, __GFP_NOWARN needs to be specified and warnings
*only* printed when the RX buffers are really low. My expectation would
be that some GFP_ATOMIC allocations fail during refill but the fact they
fail wakes kswapd to reclaim order-2 pages while the RX buffers in the
pool are consumed.

> In both cases I no longer get SKB errors, but instead (?) I get firmware 
> errors:
> iwlagn 0000:10:00.0: Microcode SW error detected.  Restarting 0x2000000.
> 

I am no wireless expert, but that looks like an separate problem to me.
I don't see how an allocation failure could trigger errors in the
microcode.

I really really hate to say it, but this might need a separate bisection
with 4752c93c30 either reverted or patched as I do below.

> So on the wireless side it does look as if there is more than one change 
> involved. Remember that with .30 I don't get any errors, only relatively 
> mild latencies and skips in the music.
> 

2.6.31 does not appear to have done wireless any favours.

> > I really do think that v2.6.30-rc6-1032-g7ba10a8 could play a role here.
> > That's a fix for v2.6.30-rc1-1131-gaa837e1. So that bug was introduced
> > _before_ the merge 82d0481 and may thus well explain both the latencies
> > I saw _and_ why that merge tested without problems. And that would also
> > go a long way to explain my test results.
> > So I'm going to retest 82d0481 with 7ba10a8 cherry-picked on top.
>                          ^^^^^^^-- should be 45ea4ea
> 
> I've tried this but still don't get any SKB errors, so that bug does not 
> seem to make a difference.
> 
> > > > BISECTION of akpm (mm) MERGE
> > > > ----------------------------
> > > While I didn't spot anything too out of the ordinary here, they did
> > > occur shortly after a number of other page allocator related patches.
> > > One small thing I noticed there is that kswapd is getting woken up
> > > less now than it did previously. Generally, I wouldn't have expected
> > > it to make a difference but it's possible that kswapd is not being
> > > woken up to reclaim at a higher order than it was previously. I have a
> > > patch for this below. It'd be nice if you could apply it and see do
> > > fewer allocation failures occur on current mainline.
> >
> > I'll give that patch a try and report back.
> 
> With your patch on .32-rc4 I still get the SKB errors, so it does not seem 
> to help. The only change there may have been is that the desktop was 
> frozen longer than without the patch, but that is an impression, not a 
> hard fact.
> 

Actually, that's fairly interesting and I think justifies pushing the
patch. Direct reclaim can stall processes in a user-visible manner which kswapd
is meant to avoid in the majority of cases but is tricky to quantify without
instrumenting the kernel to measure direct reclaim frequency and latency (I
have WIP tracepoints for this but it's still a WIP). If you notice shorter
stalls with the patch applied, it means that kswapd really did need to be
informed of the problems.

> Although identifying the problem on the wireless side is important, I still 
> feel that tracing the mm change should have priority as it influences much 
> more than just iwlagn, as the other reports prove.
> 

There still has not been a mm-change identified that makes fragmentation
significantly worse. The majority of the wireless reports have been in this
driver and I think we have the problem commit there. The only other is a
firmware loading problem in e100 after resume that fails to make an atomic
order-5 fail. It's possible that something has changed in resume in the
2.6.31 window there - maybe something like drivers now reload during resume
where they didn't previously or less memory being pushed to swap during resume.

> > > After that, could you edit drivers/net/wireless/iwlwifi/iwl-rx.c and
> > > make the GFP_ATOMIC in iwl_rx_replenish_now() GFP_ATOMIC|__GFP_NOWARN
> > > and see do any of the "serious" allocation failure messages appear.
> 
> For the above reason I've not yet tried this. It seems to me that this 
> change will not really solve anything, but just suppress errors.
> 

I disagree. Harmless allocation errors get suppressed but it still warns when
things get really bad. See the following patch that suppresses the
warnings from GFP_ATOMIC but warns for GFP_KERNEL failures and dumps a
stack on serious allocation failure.

We either need a patch like this or the
GFP_ATOMIC-direct-with-refills-from-tasklet patch needs to be reverted.

=== CUT HERE ===

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

[-- Attachment #1: Type: text/plain, Size: 4524 bytes --]

On Wednesday 14 October 2009, Mel Gorman wrote:
> I think this is very significant. Either that change needs to be backed
> out or more likely, __GFP_NOWARN needs to be specified and warnings
> *only* printed when the RX buffers are really low. My expectation would
> be that some GFP_ATOMIC allocations fail during refill but the fact they
> fail wakes kswapd to reclaim order-2 pages while the RX buffers in the
> pool are consumed.

Sorry I did not actually mention this, but the SKB failures I get with .32 
have loads of the "Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 
free buffers remaining." errors. That's why I don't think your patch will 
help anything.

zgrep "Only 0 free buffers remaining" /var/log/kern.log* | wc -l
84

OK, they are all GPF_ATOMIC and not GPF_KERNEL, but they also almost all 
have "0 free buffers"! Next to the 84 warnings for 0 remaining I only have 
one with "3 free buffers" and one with "1 free buffers".

And that does not even count the rate limitting:
Oct 12 20:15:07 aragorn kernel: __ratelimit: 45 callbacks suppressed
Oct 12 20:25:19 aragorn kernel: __ratelimit: 27 callbacks suppressed
Oct 12 20:25:20 aragorn kernel: __ratelimit: 2 callbacks suppressed

Attached the kernel log for one test I did with .32.

> > In both cases I no longer get SKB errors, but instead (?) I get
> > firmware errors:
> > iwlagn 0000:10:00.0: Microcode SW error detected.  Restarting
> > 0x2000000.
>
> I am no wireless expert, but that looks like an separate problem to me.
> I don't see how an allocation failure could trigger errors in the
> microcode.

Yes, it is a separate problem, but it is still significant that reverting 
that patch triggers them in the extreme swap situation.

> > With your patch on .32-rc4 I still get the SKB errors, so it does not
> > seem to help. The only change there may have been is that the desktop
> > was frozen longer than without the patch, but that is an impression,
> > not a hard fact.
>
> Actually, that's fairly interesting and I think justifies pushing the
> patch. Direct reclaim can stall processes in a user-visible manner which
> kswapd is meant to avoid in the majority of cases but is tricky to
> quantify without instrumenting the kernel to measure direct reclaim
> frequency and latency (I have WIP tracepoints for this but it's still a
> WIP). If you notice shorter stalls with the patch applied, it means that
> kswapd really did need to be informed of the problems.

No, I thought I saw _longer_ stalls with your patch applied...

> There still has not been a mm-change identified that makes fragmentation
> significantly worse.

My bisection shows a very clear point, even if not an individual commit, in 
the 'akpm' merge where SKB errors suddenly become *much* more frequent and 
easy to trigger.
I'm sorry to say this, but the fact that nothing has been identified yet is 
IMO the result of a lack of effort, not because there is no such change.

> The majority of the wireless reports have been in 
> this driver and I think we have the problem commit there. The only other
> is a firmware loading problem in e100 after resume that fails to make an
> atomic order-5 fail.

Not exactly true. Bartlomiej's report was about ipw2200, so there are at 
least 3 different drivers involved, two wireless and one wired. Besides 
that one report is related to heavy swap, one to resume and one to driver 
reload.
So it's much more likely that there is some common regression (in mm) that 
affected all three than that there are three unrelated regressions.
And although both of the others did extremely high allocations, they both 
started appearing in the same timeframe. And Bart's very first report 
linked it to mm changes.

> It's possible that something has changed in resume 
> in the 2.6.31 window there - maybe something like drivers now reload
> during resume where they didn't previously or less memory being pushed
> to swap during resume.

IMO you're sticking your head in the sand here. 
I'm not saying that mm is the only issue here, but I'm convinced that there 
_is_ an mm change that has contributed in a major way to these issues, 
even if we've not yet been able to identify it.

> -			    net_ratelimit())
> +			    net_ratelimit()) {
>  				IWL_CRIT(priv, "Failed to allocate SKB buffer with %s. Only %u free
> buffers remaining.\n", priority == GFP_ATOMIC ?  "GFP_ATOMIC" :
> "GFP_KERNEL",

Haven't you broken the test 'priority == GFP_ATOMIC' here by setting 
priority to GFP_ATOMIC|__GFP_NOWARN?

Cheers,
FJP


[-- Attachment #2: kern.log --]
[-- Type: text/x-log, Size: 114900 bytes --]

Oct 12 17:13:07 aragorn kernel: Linux version 2.6.32-rc4 (root@aragorn) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #29 SMP Mon Oct 12 13:12:50 CEST 2009
Oct 12 17:13:07 aragorn kernel: Command line: root=/dev/mapper/main-root ro vga=791 quiet
Oct 12 17:13:07 aragorn kernel: KERNEL supported cpus:
Oct 12 17:13:07 aragorn kernel:   Intel GenuineIntel
Oct 12 17:13:07 aragorn kernel:   AMD AuthenticAMD
Oct 12 17:13:07 aragorn kernel:   Centaur CentaurHauls
Oct 12 17:13:07 aragorn kernel: BIOS-provided physical RAM map:
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 0000000000100000 - 000000007e7b0000 (usable)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 000000007e7b0000 - 000000007e7c5400 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 000000007e7c5400 - 000000007e7e7fb8 (ACPI NVS)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 000000007e7e7fb8 - 000000007f000000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 00000000fed20000 - 00000000fed9a000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 00000000feda0000 - 00000000fedc0000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 00000000ffb00000 - 00000000ffc00000 (reserved)
Oct 12 17:13:07 aragorn kernel:  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
Oct 12 17:13:07 aragorn kernel: DMI 2.4 present.
Oct 12 17:13:07 aragorn kernel: last_pfn = 0x7e7b0 max_arch_pfn = 0x400000000
Oct 12 17:13:07 aragorn kernel: MTRR default type: uncachable
Oct 12 17:13:07 aragorn kernel: MTRR fixed ranges enabled:
Oct 12 17:13:07 aragorn kernel:   00000-9FFFF write-back
Oct 12 17:13:07 aragorn kernel:   A0000-BFFFF uncachable
Oct 12 17:13:07 aragorn kernel:   C0000-D3FFF write-protect
Oct 12 17:13:07 aragorn kernel:   D4000-EFFFF uncachable
Oct 12 17:13:07 aragorn kernel:   F0000-FFFFF write-protect
Oct 12 17:13:07 aragorn kernel: MTRR variable ranges enabled:
Oct 12 17:13:07 aragorn kernel:   0 base 000000000 mask F80000000 write-back
Oct 12 17:13:07 aragorn kernel:   1 base 07F000000 mask FFF000000 uncachable
Oct 12 17:13:07 aragorn kernel:   2 base 07E800000 mask FFF800000 uncachable
Oct 12 17:13:07 aragorn kernel:   3 base 0FEDA0000 mask FFFFE0000 uncachable
Oct 12 17:13:07 aragorn kernel:   4 disabled
Oct 12 17:13:07 aragorn kernel:   5 disabled
Oct 12 17:13:07 aragorn kernel:   6 disabled
Oct 12 17:13:07 aragorn kernel:   7 disabled
Oct 12 17:13:07 aragorn kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Oct 12 17:13:07 aragorn kernel: initial memory mapped : 0 - 20000000
Oct 12 17:13:07 aragorn kernel: init_memory_mapping: 0000000000000000-000000007e7b0000
Oct 12 17:13:07 aragorn kernel:  0000000000 - 007e600000 page 2M
Oct 12 17:13:07 aragorn kernel:  007e600000 - 007e7b0000 page 4k
Oct 12 17:13:07 aragorn kernel: kernel direct mapping tables up to 7e7b0000 @ 8000-c000
Oct 12 17:13:07 aragorn kernel: RAMDISK: 37a79000 - 37fef2f8
Oct 12 17:13:07 aragorn kernel: ACPI: RSDP 00000000000f7960 00024 (v02 HP    )
Oct 12 17:13:07 aragorn kernel: ACPI: XSDT 000000007e7c81c8 0007C (v01 HPQOEM SLIC-MPC 00000001 HP   00000001)
Oct 12 17:13:07 aragorn kernel: ACPI: FACP 000000007e7c8084 000F4 (v04 HP     30C9     00000003 HP   00000001)
Oct 12 17:13:07 aragorn kernel: ACPI: DSDT 000000007e7c8538 13484 (v01 HP       nc2500 00010000 MSFT 03000001)
Oct 12 17:13:07 aragorn kernel: ACPI: FACS 000000007e7e7d80 00040
Oct 12 17:13:07 aragorn kernel: ACPI: SLIC 000000007e7c8244 00176 (v01 HPQOEM SLIC-MPC 00000001 HP   00000001)
Oct 12 17:13:07 aragorn kernel: ACPI: HPET 000000007e7c83bc 00038 (v01 HP     30C9     00000001 HP   00000001)
Oct 12 17:13:07 aragorn kernel: ACPI: APIC 000000007e7c83f4 00068 (v01 HP     30C9     00000001 HP   00000001)
Oct 12 17:13:07 aragorn kernel: ACPI: MCFG 000000007e7c845c 0003C (v01 HP     30C9     00000001 HP   00000001)
Oct 12 17:13:07 aragorn kernel: ACPI: TCPA 000000007e7c8498 00032 (v02 HP     30C9     00000001 HP   00000001)
Oct 12 17:13:07 aragorn kernel: ACPI: ASF! 000000007e7c84cc 00069 (v16 HP     CHIMAYU  00000001 HP   00000000)
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7db9bc 002BE (v01 HP       HPQPAT 00000001 MSFT 03000001)
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7dc640 0025F (v01 HP      Cpu0Tst 00003000 INTL 20060317)
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7dc89f 000A6 (v01 HP      Cpu1Tst 00003000 INTL 20060317)
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7dc945 004D7 (v01 HP        CpuPm 00003000 INTL 20060317)
Oct 12 17:13:07 aragorn kernel: ACPI: Local APIC address 0xfee00000
Oct 12 17:13:07 aragorn kernel: (7 early reservations) ==> bootmem [0000000000 - 007e7b0000]
Oct 12 17:13:07 aragorn kernel:   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Oct 12 17:13:07 aragorn kernel:   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Oct 12 17:13:07 aragorn kernel:   #2 [0001000000 - 00015bdb30]    TEXT DATA BSS ==> [0001000000 - 00015bdb30]
Oct 12 17:13:07 aragorn kernel:   #3 [0037a79000 - 0037fef2f8]          RAMDISK ==> [0037a79000 - 0037fef2f8]
Oct 12 17:13:07 aragorn kernel:   #4 [000009fc00 - 0000100000]    BIOS reserved ==> [000009fc00 - 0000100000]
Oct 12 17:13:07 aragorn kernel:   #5 [00015be000 - 00015be18c]              BRK ==> [00015be000 - 00015be18c]
Oct 12 17:13:07 aragorn kernel:   #6 [0000008000 - 000000a000]          PGTABLE ==> [0000008000 - 000000a000]
Oct 12 17:13:07 aragorn kernel:  [ffffea0000000000-ffffea0001ffffff] PMD -> [ffff880001a00000-ffff8800039fffff] on node 0
Oct 12 17:13:07 aragorn kernel: Zone PFN ranges:
Oct 12 17:13:07 aragorn kernel:   DMA      0x00000000 -> 0x00001000
Oct 12 17:13:07 aragorn kernel:   DMA32    0x00001000 -> 0x00100000
Oct 12 17:13:07 aragorn kernel:   Normal   0x00100000 -> 0x00100000
Oct 12 17:13:07 aragorn kernel: Movable zone start PFN for each node
Oct 12 17:13:07 aragorn kernel: early_node_map[2] active PFN ranges
Oct 12 17:13:07 aragorn kernel:     0: 0x00000000 -> 0x0000009f
Oct 12 17:13:07 aragorn kernel:     0: 0x00000100 -> 0x0007e7b0
Oct 12 17:13:07 aragorn kernel: On node 0 totalpages: 517967
Oct 12 17:13:07 aragorn kernel:   DMA zone: 64 pages used for memmap
Oct 12 17:13:07 aragorn kernel:   DMA zone: 101 pages reserved
Oct 12 17:13:07 aragorn kernel:   DMA zone: 3834 pages, LIFO batch:0
Oct 12 17:13:07 aragorn kernel:   DMA32 zone: 8031 pages used for memmap
Oct 12 17:13:07 aragorn kernel:   DMA32 zone: 505937 pages, LIFO batch:31
Oct 12 17:13:07 aragorn kernel: ACPI: PM-Timer IO Port: 0x1008
Oct 12 17:13:07 aragorn kernel: ACPI: Local APIC address 0xfee00000
Oct 12 17:13:07 aragorn kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Oct 12 17:13:07 aragorn kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Oct 12 17:13:07 aragorn kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
Oct 12 17:13:07 aragorn kernel: ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
Oct 12 17:13:07 aragorn kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
Oct 12 17:13:07 aragorn kernel: IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
Oct 12 17:13:07 aragorn kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Oct 12 17:13:07 aragorn kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Oct 12 17:13:07 aragorn kernel: ACPI: IRQ0 used by override.
Oct 12 17:13:07 aragorn kernel: ACPI: IRQ2 used by override.
Oct 12 17:13:07 aragorn kernel: ACPI: IRQ9 used by override.
Oct 12 17:13:07 aragorn kernel: Using ACPI (MADT) for SMP configuration information
Oct 12 17:13:07 aragorn kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000
Oct 12 17:13:07 aragorn kernel: SMP: Allowing 2 CPUs, 0 hotplug CPUs
Oct 12 17:13:07 aragorn kernel: nr_irqs_gsi: 24
Oct 12 17:13:07 aragorn kernel: PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
Oct 12 17:13:07 aragorn kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Oct 12 17:13:07 aragorn kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Oct 12 17:13:07 aragorn kernel: Allocating PCI resources starting at 7f000000 (gap: 7f000000:7fc00000)
Oct 12 17:13:07 aragorn kernel: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:2 nr_node_ids:1
Oct 12 17:13:07 aragorn kernel: PERCPU: Embedded 27 pages/cpu @ffff880001600000 s81752 r8192 d20648 u1048576
Oct 12 17:13:07 aragorn kernel: pcpu-alloc: s81752 r8192 d20648 u1048576 alloc=1*2097152
Oct 12 17:13:07 aragorn kernel: pcpu-alloc: [0] 0 1 
Oct 12 17:13:07 aragorn kernel: Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 509771
Oct 12 17:13:07 aragorn kernel: Kernel command line: root=/dev/mapper/main-root ro vga=791 quiet
Oct 12 17:13:07 aragorn kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Oct 12 17:13:07 aragorn kernel: Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Oct 12 17:13:07 aragorn kernel: Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Oct 12 17:13:07 aragorn kernel: Initializing CPU#0
Oct 12 17:13:07 aragorn kernel: Checking aperture...
Oct 12 17:13:07 aragorn kernel: No AGP bridge found
Oct 12 17:13:07 aragorn kernel: Memory: 2023496k/2072256k available (2758k kernel code, 388k absent, 47724k reserved, 1927k data, 508k init)
Oct 12 17:13:07 aragorn kernel: SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Oct 12 17:13:07 aragorn kernel: Hierarchical RCU implementation.
Oct 12 17:13:07 aragorn kernel: NR_IRQS:512
Oct 12 17:13:07 aragorn kernel: Extended CMOS year: 2000
Oct 12 17:13:07 aragorn kernel: Console: colour dummy device 80x25
Oct 12 17:13:07 aragorn kernel: console [tty0] enabled
Oct 12 17:13:07 aragorn kernel: hpet clockevent registered
Oct 12 17:13:07 aragorn kernel: HPET: 3 timers in total, 0 timers will be used for per-cpu timer
Oct 12 17:13:07 aragorn kernel: Fast TSC calibration using PIT
Oct 12 17:13:07 aragorn kernel: Detected 1330.067 MHz processor.
Oct 12 17:13:07 aragorn kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 2660.13 BogoMIPS (lpj=5320268)
Oct 12 17:13:07 aragorn kernel: Security Framework initialized
Oct 12 17:13:07 aragorn kernel: SELinux:  Disabled at boot.
Oct 12 17:13:07 aragorn kernel: Mount-cache hash table entries: 256
Oct 12 17:13:07 aragorn kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 12 17:13:07 aragorn kernel: CPU: L2 cache: 2048K
Oct 12 17:13:07 aragorn kernel: CPU: Physical Processor ID: 0
Oct 12 17:13:07 aragorn kernel: CPU: Processor Core ID: 0
Oct 12 17:13:07 aragorn kernel: mce: CPU supports 6 MCE banks
Oct 12 17:13:07 aragorn kernel: CPU0: Thermal monitoring handled by SMI
Oct 12 17:13:07 aragorn kernel: using mwait in idle threads.
Oct 12 17:13:07 aragorn kernel: Performance Events: Core2 events, Intel PMU driver.
Oct 12 17:13:07 aragorn kernel: ... version:                2
Oct 12 17:13:07 aragorn kernel: ... bit width:              40
Oct 12 17:13:07 aragorn kernel: ... generic registers:      2
Oct 12 17:13:07 aragorn kernel: ... value mask:             000000ffffffffff
Oct 12 17:13:07 aragorn kernel: ... max period:             000000007fffffff
Oct 12 17:13:07 aragorn kernel: ... fixed-purpose events:   3
Oct 12 17:13:07 aragorn kernel: ... event mask:             0000000700000003
Oct 12 17:13:07 aragorn kernel: ACPI: Core revision 20090903
Oct 12 17:13:07 aragorn kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Oct 12 17:13:07 aragorn kernel: ftrace: allocating 13965 entries in 55 pages
Oct 12 17:13:07 aragorn kernel: Setting APIC routing to flat
Oct 12 17:13:07 aragorn kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Oct 12 17:13:07 aragorn kernel: CPU0: Intel(R) Core(TM)2 Duo CPU     U7700  @ 1.33GHz stepping 0d
Oct 12 17:13:07 aragorn kernel: Booting processor 1 APIC 0x1 ip 0x6000
Oct 12 17:13:07 aragorn kernel: Initializing CPU#1
Oct 12 17:13:07 aragorn kernel: Calibrating delay using timer specific routine.. 2659.98 BogoMIPS (lpj=5319962)
Oct 12 17:13:07 aragorn kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 12 17:13:07 aragorn kernel: CPU: L2 cache: 2048K
Oct 12 17:13:07 aragorn kernel: CPU: Physical Processor ID: 0
Oct 12 17:13:07 aragorn kernel: CPU: Processor Core ID: 1
Oct 12 17:13:07 aragorn kernel: mce: CPU supports 6 MCE banks
Oct 12 17:13:07 aragorn kernel: CPU1: Thermal monitoring enabled (TM2)
Oct 12 17:13:07 aragorn kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     U7700  @ 1.33GHz stepping 0d
Oct 12 17:13:07 aragorn kernel: checking TSC synchronization [CPU#0 -> CPU#1]: passed.
Oct 12 17:13:07 aragorn kernel: Brought up 2 CPUs
Oct 12 17:13:07 aragorn kernel: Total of 2 processors activated (5320.11 BogoMIPS).
Oct 12 17:13:07 aragorn kernel: CPU0 attaching sched-domain:
Oct 12 17:13:07 aragorn kernel:  domain 0: span 0-1 level MC
Oct 12 17:13:07 aragorn kernel:   groups: 0 1
Oct 12 17:13:07 aragorn kernel: CPU1 attaching sched-domain:
Oct 12 17:13:07 aragorn kernel:  domain 0: span 0-1 level MC
Oct 12 17:13:07 aragorn kernel:   groups: 1 0
Oct 12 17:13:07 aragorn kernel: NET: Registered protocol family 16
Oct 12 17:13:07 aragorn kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Oct 12 17:13:07 aragorn kernel: ACPI: bus type pci registered
Oct 12 17:13:07 aragorn kernel: PCI: MCFG configuration 0: base f8000000 segment 0 buses 0 - 63
Oct 12 17:13:07 aragorn kernel: PCI: Not using MMCONFIG.
Oct 12 17:13:07 aragorn kernel: PCI: Using configuration type 1 for base access
Oct 12 17:13:07 aragorn kernel: bio: create slab <bio-0> at 0
Oct 12 17:13:07 aragorn kernel: ACPI: EC: Look up EC in DSDT
Oct 12 17:13:07 aragorn kernel: ACPI: Interpreter enabled
Oct 12 17:13:07 aragorn kernel: ACPI: (supports S0 S3 S4 S5)
Oct 12 17:13:07 aragorn kernel: ACPI: Using IOAPIC for interrupt routing
Oct 12 17:13:07 aragorn kernel: PCI: MCFG configuration 0: base f8000000 segment 0 buses 0 - 63
Oct 12 17:13:07 aragorn kernel: PCI: MCFG area at f8000000 reserved in ACPI motherboard resources
Oct 12 17:13:07 aragorn kernel: PCI: Using MMCONFIG at f8000000 - fbffffff
Oct 12 17:13:07 aragorn kernel: ACPI: EC: GPE = 0x16, I/O: command/status = 0x66, data = 0x62
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C29F] (on)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C1C7] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C3AD] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C3B0] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C3C3] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C3C4] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C3C5] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C3C6] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: Power Resource [C3C7] (off)
Oct 12 17:13:07 aragorn kernel: ACPI: No dock devices found.
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Root Bridge [C003] (0000:00)
Oct 12 17:13:07 aragorn kernel: pci 0000:00:02.0: reg 10 64bit mmio: [0xe0400000-0xe04fffff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:02.0: reg 18 64bit mmio pref: [0xd0000000-0xdfffffff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:02.0: reg 20 io port: [0x2000-0x2007]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:02.1: reg 10 64bit mmio: [0xe0500000-0xe05fffff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.0: reg 10 64bit mmio: [0xe0600000-0xe060000f]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.0: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.2: reg 10 io port: [0x2008-0x200f]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.2: reg 14 io port: [0x2010-0x2013]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.2: reg 18 io port: [0x2018-0x201f]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.2: reg 1c io port: [0x2020-0x2023]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.2: reg 20 io port: [0x2030-0x203f]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.3: reg 10 io port: [0x2040-0x2047]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:03.3: reg 14 32bit mmio: [0xe0601000-0xe0601fff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:19.0: reg 10 32bit mmio: [0xe0620000-0xe063ffff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:19.0: reg 14 32bit mmio: [0xe0640000-0xe0640fff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:19.0: reg 18 io port: [0x2060-0x207f]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:19.0: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:00:19.0: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1a.0: reg 20 io port: [0x2080-0x209f]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1a.1: reg 20 io port: [0x20a0-0x20bf]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1a.7: reg 10 32bit mmio: [0xe0641000-0xe06413ff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1a.7: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1b.0: reg 10 64bit mmio: [0xe0644000-0xe0647fff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1b.0: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1d.0: reg 20 io port: [0x20c0-0x20df]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1d.1: reg 20 io port: [0x20e0-0x20ff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1d.2: reg 20 io port: [0x2100-0x211f]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1d.7: reg 10 32bit mmio: [0xe0648000-0xe06483ff]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1d.7: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.0: quirk: region 1000-107f claimed by ICH6 ACPI/GPIO/TCO
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.0: quirk: region 1100-113f claimed by ICH6 GPIO
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.0: ICH7 LPC Generic IO decode 1 PIO at 0500 (mask 007f)
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.0: ICH7 LPC Generic IO decode 4 PIO at 02e8 (mask 0007)
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.1: reg 10 io port: [0x00-0x07]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.1: reg 14 io port: [0x00-0x03]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.1: reg 18 io port: [0x00-0x07]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.1: reg 1c io port: [0x00-0x03]
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1f.1: reg 20 io port: [0x2120-0x212f]
Oct 12 17:13:07 aragorn kernel: pci 0000:10:00.0: reg 10 64bit mmio: [0xe0000000-0xe0001fff]
Oct 12 17:13:07 aragorn kernel: pci 0000:10:00.0: PME# supported from D0 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:10:00.0: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1: bridge 32bit mmio: [0xe0000000-0xe00fffff]
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0: reg 10 32bit mmio: [0xe0100000-0xe0100fff]
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0: supports D1 D2
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0: PME# supported from D0 D1 D2 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.1: reg 10 32bit mmio: [0xe0101000-0xe01017ff]
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.1: supports D1 D2
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.1: PME# supported from D0 D1 D2 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.1: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.2: reg 10 32bit mmio: [0xe0102000-0xe01020ff]
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.2: supports D1 D2
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.2: PME# supported from D0 D1 D2 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.2: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.3: reg 10 32bit mmio: [0xe0103000-0xe01030ff]
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.3: supports D1 D2
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.3: PME# supported from D0 D1 D2 D3hot D3cold
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.3: PME# disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1e.0: transparent bridge
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1e.0: bridge 32bit mmio: [0xe0100000-0xe03fffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:00: on NUMA node 0
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Routing Table [\_SB_.C003._PRT]
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Routing Table [\_SB_.C003.C0B2._PRT]
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Routing Table [\_SB_.C003.C11F._PRT]
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Routing Table [\_SB_.C003.C133._PRT]
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Link [C12F] (IRQs *10 11)
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Link [C130] (IRQs *10 11)
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Link [C131] (IRQs 10 *11)
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Link [C132] (IRQs 10 11) *5
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Link [C142] (IRQs *10 11)
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Link [C143] (IRQs 10 11) *0, disabled.
Oct 12 17:13:07 aragorn kernel: ACPI: PCI Interrupt Link [C144] (IRQs 10 *11)
Oct 12 17:13:07 aragorn kernel: ACPI Exception: AE_NOT_FOUND, Evaluating _PRS (20090903/pci_link-184)
Oct 12 17:13:07 aragorn kernel: vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
Oct 12 17:13:07 aragorn kernel: vgaarb: loaded
Oct 12 17:13:07 aragorn kernel: usbcore: registered new interface driver usbfs
Oct 12 17:13:07 aragorn kernel: usbcore: registered new interface driver hub
Oct 12 17:13:07 aragorn kernel: usbcore: registered new device driver usb
Oct 12 17:13:07 aragorn kernel: PCI: Using ACPI for IRQ routing
Oct 12 17:13:07 aragorn kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
Oct 12 17:13:07 aragorn kernel: hpet0: 3 comparators, 64-bit 14.318180 MHz counter
Oct 12 17:13:07 aragorn kernel: Switching to clocksource tsc
Oct 12 17:13:07 aragorn kernel: pnp: PnP ACPI init
Oct 12 17:13:07 aragorn kernel: ACPI: bus type pnp registered
Oct 12 17:13:07 aragorn kernel: pnp: PnP ACPI: found 14 devices
Oct 12 17:13:07 aragorn kernel: ACPI: ACPI bus type pnp unregistered
Oct 12 17:13:07 aragorn kernel: system 00:00: iomem range 0x0-0x9ffff could not be reserved
Oct 12 17:13:07 aragorn kernel: (The fact that a range could not be reserved is generally harmless.)
Oct 12 17:13:07 aragorn kernel: system 00:00: iomem range 0xe0000-0xfffff could not be reserved
Oct 12 17:13:07 aragorn kernel: system 00:00: iomem range 0x100000-0x7e7fffff could not be reserved
Oct 12 17:13:07 aragorn kernel: system 00:0a: ioport range 0x500-0x55f has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0a: ioport range 0x800-0x80f has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0a: iomem range 0xffb00000-0xffbfffff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0a: iomem range 0xfff00000-0xffffffff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: ioport range 0x4d0-0x4d1 has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: ioport range 0x2f8-0x2ff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: ioport range 0x3f8-0x3ff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: ioport range 0x1000-0x107f has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: ioport range 0x1100-0x113f has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: ioport range 0x1200-0x121f has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: iomem range 0xf8000000-0xfbffffff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: iomem range 0xfec00000-0xfec000ff could not be reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: iomem range 0xfed20000-0xfed3ffff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: iomem range 0xfed45000-0xfed8ffff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0c: iomem range 0xfed90000-0xfed99fff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0d: iomem range 0xcee00-0xcffff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0d: iomem range 0xd2000-0xd3fff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0d: iomem range 0xfeda0000-0xfedbffff has been reserved
Oct 12 17:13:07 aragorn kernel: system 00:0d: iomem range 0xfee00000-0xfee00fff has been reserved
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0: PCI bridge, secondary bus 0000:08
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0:   IO window: disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0:   MEM window: disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0:   PREFETCH window: disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1: PCI bridge, secondary bus 0000:10
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1:   IO window: disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1:   MEM window: 0xe0000000-0xe00fffff
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1:   PREFETCH window: disabled
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0: CardBus bridge, secondary bus 0000:03
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0:   IO window: 0x003000-0x0030ff
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0:   IO window: 0x003400-0x0034ff
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0:   PREFETCH window: 0x80000000-0x83ffffff
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0:   MEM window: 0x84000000-0x87ffffff
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1e.0: PCI bridge, secondary bus 0000:02
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1e.0:   IO window: 0x3000-0x3fff
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1e.0:   MEM window: 0xe0100000-0xe03fffff
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1e.0:   PREFETCH window: 0x80000000-0x83ffffff
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1c.1: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: pci 0000:00:1e.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: pci 0000:02:06.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:00: resource 0 io:  [0x00-0xffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffffffffffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:10: resource 1 mem: [0xe0000000-0xe00fffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:02: resource 0 io:  [0x3000-0x3fff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:02: resource 1 mem: [0xe0100000-0xe03fffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:02: resource 2 pref mem [0x80000000-0x83ffffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:02: resource 3 io:  [0x00-0xffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:02: resource 4 mem: [0x000000-0xffffffffffffffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:03: resource 0 io:  [0x3000-0x30ff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:03: resource 1 io:  [0x3400-0x34ff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:03: resource 2 pref mem [0x80000000-0x83ffffff]
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:03: resource 3 mem: [0x84000000-0x87ffffff]
Oct 12 17:13:07 aragorn kernel: NET: Registered protocol family 2
Oct 12 17:13:07 aragorn kernel: IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
Oct 12 17:13:07 aragorn kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
Oct 12 17:13:07 aragorn kernel: TCP bind hash table entries: 65536 (order: 9, 2097152 bytes)
Oct 12 17:13:07 aragorn kernel: TCP: Hash tables configured (established 262144 bind 65536)
Oct 12 17:13:07 aragorn kernel: TCP reno registered
Oct 12 17:13:07 aragorn kernel: NET: Registered protocol family 1
Oct 12 17:13:07 aragorn kernel: Trying to unpack rootfs image as initramfs...
Oct 12 17:13:07 aragorn kernel: Freeing initrd memory: 5592k freed
Oct 12 17:13:07 aragorn kernel: audit: initializing netlink socket (disabled)
Oct 12 17:13:07 aragorn kernel: type=2000 audit(1255360345.659:1): initialized
Oct 12 17:13:07 aragorn kernel: VFS: Disk quotas dquot_6.5.2
Oct 12 17:13:07 aragorn kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Oct 12 17:13:07 aragorn kernel: msgmni has been set to 3964
Oct 12 17:13:07 aragorn kernel: alg: No test for stdrng (krng)
Oct 12 17:13:07 aragorn kernel: io scheduler noop registered
Oct 12 17:13:07 aragorn kernel: io scheduler anticipatory registered
Oct 12 17:13:07 aragorn kernel: io scheduler deadline registered
Oct 12 17:13:07 aragorn kernel: io scheduler cfq registered (default)
Oct 12 17:13:07 aragorn kernel: pci 0000:00:02.0: Boot video device
Oct 12 17:13:07 aragorn kernel: pcieport-driver 0000:00:1c.0: irq 24 for MSI/MSI-X
Oct 12 17:13:07 aragorn kernel: pcieport-driver 0000:00:1c.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: pcieport-driver 0000:00:1c.1: irq 25 for MSI/MSI-X
Oct 12 17:13:07 aragorn kernel: pcieport-driver 0000:00:1c.1: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: vesafb: framebuffer at 0xd0000000, mapped to 0xffffc90004100000, using 3072k, total 7616k
Oct 12 17:13:07 aragorn kernel: vesafb: mode is 1024x768x16, linelength=2048, pages=3
Oct 12 17:13:07 aragorn kernel: vesafb: scrolling: redraw
Oct 12 17:13:07 aragorn kernel: vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
Oct 12 17:13:07 aragorn kernel: Console: switching to colour frame buffer device 128x48
Oct 12 17:13:07 aragorn kernel: fb0: VESA VGA frame buffer device
Oct 12 17:13:07 aragorn kernel: Linux agpgart interface v0.103
Oct 12 17:13:07 aragorn kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Oct 12 17:13:07 aragorn kernel: serial 0000:00:03.3: PCI INT B -> GSI 17 (level, low) -> IRQ 17
Oct 12 17:13:07 aragorn kernel: 0000:00:03.3: ttyS0 at I/O 0x2040 (irq = 17) is a 16550A
Oct 12 17:13:07 aragorn kernel: brd: module loaded
Oct 12 17:13:07 aragorn kernel: PNP: PS/2 Controller [PNP0303:C29C,PNP0f13:C29D] at 0x60,0x64 irq 1,12
Oct 12 17:13:07 aragorn kernel: i8042.c: Detected active multiplexing controller, rev 1.1.
Oct 12 17:13:07 aragorn kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
Oct 12 17:13:07 aragorn kernel: serio: i8042 AUX0 port at 0x60,0x64 irq 12
Oct 12 17:13:07 aragorn kernel: serio: i8042 AUX1 port at 0x60,0x64 irq 12
Oct 12 17:13:07 aragorn kernel: serio: i8042 AUX2 port at 0x60,0x64 irq 12
Oct 12 17:13:07 aragorn kernel: serio: i8042 AUX3 port at 0x60,0x64 irq 12
Oct 12 17:13:07 aragorn kernel: mice: PS/2 mouse device common for all mice
Oct 12 17:13:07 aragorn kernel: Driver 'rtc_cmos' needs updating - please use bus_type methods
Oct 12 17:13:07 aragorn kernel: rtc_cmos 00:06: RTC can wake from S4
Oct 12 17:13:07 aragorn kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Oct 12 17:13:07 aragorn kernel: rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
Oct 12 17:13:07 aragorn kernel: cpuidle: using governor ladder
Oct 12 17:13:07 aragorn kernel: cpuidle: using governor menu
Oct 12 17:13:07 aragorn kernel: TCP cubic registered
Oct 12 17:13:07 aragorn kernel: NET: Registered protocol family 17
Oct 12 17:13:07 aragorn kernel: registered taskstats version 1
Oct 12 17:13:07 aragorn kernel: rtc_cmos 00:06: setting system clock to 2009-10-12 15:12:26 UTC (1255360346)
Oct 12 17:13:07 aragorn kernel: Freeing unused kernel memory: 508k freed
Oct 12 17:13:07 aragorn kernel: input: AT Translated Set 2 keyboard as /class/input/input0
Oct 12 17:13:07 aragorn kernel: fan PNP0C0B:00: registered as cooling_device0
Oct 12 17:13:07 aragorn kernel: ACPI: Fan [C3B1] (off)
Oct 12 17:13:07 aragorn kernel: fan PNP0C0B:01: registered as cooling_device1
Oct 12 17:13:07 aragorn kernel: ACPI: Fan [C3B2] (off)
Oct 12 17:13:07 aragorn kernel: fan PNP0C0B:02: registered as cooling_device2
Oct 12 17:13:07 aragorn kernel: ACPI: Fan [C3C8] (off)
Oct 12 17:13:07 aragorn kernel: fan PNP0C0B:03: registered as cooling_device3
Oct 12 17:13:07 aragorn kernel: ACPI: Fan [C3C9] (off)
Oct 12 17:13:07 aragorn kernel: fan PNP0C0B:04: registered as cooling_device4
Oct 12 17:13:07 aragorn kernel: ACPI: Fan [C3CA] (off)
Oct 12 17:13:07 aragorn kernel: fan PNP0C0B:05: registered as cooling_device5
Oct 12 17:13:07 aragorn kernel: ACPI: Fan [C3CB] (off)
Oct 12 17:13:07 aragorn kernel: fan PNP0C0B:06: registered as cooling_device6
Oct 12 17:13:07 aragorn kernel: ACPI: Fan [C3CC] (off)
Oct 12 17:13:07 aragorn kernel: thermal LNXTHERM:01: registered as thermal_zone0
Oct 12 17:13:07 aragorn kernel: ACPI: Thermal Zone [TZ6] (25 C)
Oct 12 17:13:07 aragorn kernel: thermal LNXTHERM:02: registered as thermal_zone1
Oct 12 17:13:07 aragorn kernel: ACPI: Thermal Zone [TZ0] (55 C)
Oct 12 17:13:07 aragorn kernel: thermal LNXTHERM:03: registered as thermal_zone2
Oct 12 17:13:07 aragorn kernel: ACPI: Thermal Zone [TZ1] (57 C)
Oct 12 17:13:07 aragorn kernel: thermal LNXTHERM:04: registered as thermal_zone3
Oct 12 17:13:07 aragorn kernel: ACPI: Thermal Zone [TZ3] (45 C)
Oct 12 17:13:07 aragorn kernel: thermal LNXTHERM:05: registered as thermal_zone4
Oct 12 17:13:07 aragorn kernel: ACPI: Thermal Zone [TZ4] (33 C)
Oct 12 17:13:07 aragorn kernel: thermal LNXTHERM:06: registered as thermal_zone5
Oct 12 17:13:07 aragorn kernel: ACPI: Thermal Zone [TZ5] (49 C)
Oct 12 17:13:07 aragorn kernel: e1000e: Intel(R) PRO/1000 Network Driver - 1.0.2-k2
Oct 12 17:13:07 aragorn kernel: e1000e: Copyright (c) 1999-2008 Intel Corporation.
Oct 12 17:13:07 aragorn kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
Oct 12 17:13:07 aragorn kernel: e1000e 0000:00:19.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: e1000e 0000:00:19.0: irq 26 for MSI/MSI-X
Oct 12 17:13:07 aragorn kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Oct 12 17:13:07 aragorn kernel: SCSI subsystem initialized
Oct 12 17:13:07 aragorn kernel: libata version 3.00 loaded.
Oct 12 17:13:07 aragorn kernel: ricoh-mmc: Ricoh MMC Controller disabling driver
Oct 12 17:13:07 aragorn kernel: ricoh-mmc: Copyright(c) Philip Langdale
Oct 12 17:13:07 aragorn kernel: ricoh-mmc: Ricoh MMC controller found at 0000:02:06.3 [1180:0843] (rev 11)
Oct 12 17:13:07 aragorn kernel: ricoh-mmc: Controller is now disabled.
Oct 12 17:13:07 aragorn kernel: ohci1394 0000:02:06.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19
Oct 12 17:13:07 aragorn kernel: sdhci: Secure Digital Host Controller Interface driver
Oct 12 17:13:07 aragorn kernel: sdhci: Copyright(c) Pierre Ossman
Oct 12 17:13:07 aragorn kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[19]  MMIO=[e0101000-e01017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
Oct 12 17:13:07 aragorn kernel: sdhci-pci 0000:02:06.2: SDHCI controller found [1180:0822] (rev 21)
Oct 12 17:13:07 aragorn kernel: sdhci-pci 0000:02:06.2: PCI INT C -> GSI 20 (level, low) -> IRQ 20
Oct 12 17:13:07 aragorn kernel: Registered led device: mmc0::
Oct 12 17:13:07 aragorn kernel: mmc0: SDHCI controller on PCI [0000:02:06.2] using PIO
Oct 12 17:13:07 aragorn kernel: 0000:00:19.0: eth0: (PCI Express:2.5GB/s:Width x1) 00:1e:68:5e:3b:04
Oct 12 17:13:07 aragorn kernel: 0000:00:19.0: eth0: Intel(R) PRO/1000 Network Connection
Oct 12 17:13:07 aragorn kernel: 0000:00:19.0: eth0: MAC: 6, PHY: 6, PBA No: ffffff-0ff
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 18 (level, low) -> IRQ 18
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: EHCI Host Controller
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: debug port 1
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: cache line size of 32 is not supported
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: irq 18, io mem 0xe0641000
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00
Oct 12 17:13:07 aragorn kernel: usb usb1: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 1-0:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 1-0:1.0: 4 ports detected
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: debug port 1
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: cache line size of 32 is not supported
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: irq 20, io mem 0xe0648000
Oct 12 17:13:07 aragorn kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
Oct 12 17:13:07 aragorn kernel: usb usb2: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 2-0:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 2-0:1.0: 6 ports detected
Oct 12 17:13:07 aragorn kernel: pata_acpi 0000:00:03.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
Oct 12 17:13:07 aragorn kernel: pata_acpi 0000:00:03.2: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: pata_acpi 0000:00:03.2: PCI INT C disabled
Oct 12 17:13:07 aragorn kernel: pata_acpi 0000:00:1f.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 12 17:13:07 aragorn kernel: pata_acpi 0000:00:1f.1: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: pata_acpi 0000:00:1f.1: PCI INT A disabled
Oct 12 17:13:07 aragorn kernel: ata_piix 0000:00:1f.1: version 2.13
Oct 12 17:13:07 aragorn kernel: ata_piix 0000:00:1f.1: quirky BIOS, skipping spindown on poweroff and hibernation
Oct 12 17:13:07 aragorn kernel: ata_piix 0000:00:1f.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 12 17:13:07 aragorn kernel: ata_piix 0000:00:1f.1: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: uhci_hcd: USB Universal Host Controller Interface driver
Oct 12 17:13:07 aragorn kernel: Uniform Multi-Platform E-IDE driver
Oct 12 17:13:07 aragorn kernel: scsi0 : ata_piix
Oct 12 17:13:07 aragorn kernel: scsi1 : ata_piix
Oct 12 17:13:07 aragorn kernel: ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x2120 irq 14
Oct 12 17:13:07 aragorn kernel: ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x2128 irq 15
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.0: UHCI Host Controller
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.0: irq 16, io base 0x00002080
Oct 12 17:13:07 aragorn kernel: usb usb3: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 3-0:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 3-0:1.0: 2 ports detected
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.1: UHCI Host Controller
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1a.1: irq 17, io base 0x000020a0
Oct 12 17:13:07 aragorn kernel: usb usb4: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 4-0:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 4-0:1.0: 2 ports detected
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 5
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.0: irq 20, io base 0x000020c0
Oct 12 17:13:07 aragorn kernel: usb usb5: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 5-0:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 5-0:1.0: 2 ports detected
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 6
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.1: irq 22, io base 0x000020e0
Oct 12 17:13:07 aragorn kernel: usb usb6: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 6-0:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 6-0:1.0: 2 ports detected
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 7
Oct 12 17:13:07 aragorn kernel: uhci_hcd 0000:00:1d.2: irq 18, io base 0x00002100
Oct 12 17:13:07 aragorn kernel: usb usb7: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 7-0:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 7-0:1.0: 2 ports detected
Oct 12 17:13:07 aragorn kernel: ata2: port disabled. ignoring.
Oct 12 17:13:07 aragorn kernel: ata1.00: ATA-7: SAMSUNG HS122JC, GQ100-04, max UDMA/100
Oct 12 17:13:07 aragorn kernel: ata1.00: 234441648 sectors, multi 16: LBA 
Oct 12 17:13:07 aragorn kernel: ata1.01: ATAPI: MATSHITADVD-RAM UJ-852S, 1.02, max MWDMA2
Oct 12 17:13:07 aragorn kernel: ata1.00: configured for UDMA/100
Oct 12 17:13:07 aragorn kernel: ata1.01: configured for MWDMA2
Oct 12 17:13:07 aragorn kernel: scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HS122JC  GQ10 PQ: 0 ANSI: 5
Oct 12 17:13:07 aragorn kernel: scsi 0:0:1:0: CD-ROM            MATSHITA DVD-RAM UJ-852S  1.02 PQ: 0 ANSI: 5
Oct 12 17:13:07 aragorn kernel: sd 0:0:0:0: [sda] 234441648 512-byte logical blocks: (120 GB/111 GiB)
Oct 12 17:13:07 aragorn kernel: sd 0:0:0:0: [sda] Write Protect is off
Oct 12 17:13:07 aragorn kernel: sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Oct 12 17:13:07 aragorn kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 12 17:13:07 aragorn kernel:  sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 >
Oct 12 17:13:07 aragorn kernel: sd 0:0:0:0: [sda] Attached SCSI disk
Oct 12 17:13:07 aragorn kernel: sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Oct 12 17:13:07 aragorn kernel: Uniform CD-ROM driver Revision: 3.20
Oct 12 17:13:07 aragorn kernel: sr 0:0:1:0: Attached scsi CD-ROM sr0
Oct 12 17:13:07 aragorn kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Oct 12 17:13:07 aragorn kernel: sr 0:0:1:0: Attached scsi generic sg1 type 5
Oct 12 17:13:07 aragorn kernel: usb 1-2: new high speed USB device using ehci_hcd and address 3
Oct 12 17:13:07 aragorn kernel: usb 1-2: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: hub 1-2:1.0: USB hub found
Oct 12 17:13:07 aragorn kernel: hub 1-2:1.0: 4 ports detected
Oct 12 17:13:07 aragorn kernel: usb 3-1: new full speed USB device using uhci_hcd and address 2
Oct 12 17:13:07 aragorn kernel: device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
Oct 12 17:13:07 aragorn kernel: usb 3-1: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: ieee1394: Host added: ID:BUS[0-00:1023]  GUID[001b249929192210]
Oct 12 17:13:07 aragorn kernel: usb 5-2: new full speed USB device using uhci_hcd and address 2
Oct 12 17:13:07 aragorn kernel: usb 5-2: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: usb 1-2.2: new full speed USB device using ehci_hcd and address 4
Oct 12 17:13:07 aragorn kernel: usb 1-2.2: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: usb 1-2.3: new low speed USB device using ehci_hcd and address 5
Oct 12 17:13:07 aragorn kernel: usb 1-2.3: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: usbcore: registered new interface driver hiddev
Oct 12 17:13:07 aragorn kernel: input: Logitech USB Receiver as /class/input/input1
Oct 12 17:13:07 aragorn kernel: generic-usb 0003:046D:C50D.0001: input: USB HID v1.10 Mouse [Logitech USB Receiver] on usb-0000:00:1a.7-2.3/input0
Oct 12 17:13:07 aragorn kernel: usbcore: registered new interface driver usbhid
Oct 12 17:13:07 aragorn kernel: usbhid: v2.6:USB HID core driver
Oct 12 17:13:07 aragorn kernel: usb 1-2.4: new low speed USB device using ehci_hcd and address 6
Oct 12 17:13:07 aragorn kernel: usb 1-2.4: configuration #1 chosen from 1 choice
Oct 12 17:13:07 aragorn kernel: input: USB Compliant Keyboard as /class/input/input2
Oct 12 17:13:07 aragorn kernel: generic-usb 0003:05A4:9841.0002: input: USB HID v1.10 Keyboard [USB Compliant Keyboard] on usb-0000:00:1a.7-2.4/input0
Oct 12 17:13:07 aragorn kernel: input: USB Compliant Keyboard as /class/input/input3
Oct 12 17:13:07 aragorn kernel: generic-usb 0003:05A4:9841.0003: input: USB HID v1.10 Device [USB Compliant Keyboard] on usb-0000:00:1a.7-2.4/input1
Oct 12 17:13:07 aragorn kernel: PM: Starting manual resume from disk
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: udevd version 125 started
Oct 12 17:13:07 aragorn kernel: input: Sleep Button as /class/input/input4
Oct 12 17:13:07 aragorn kernel: ACPI: Sleep Button [C2BF]
Oct 12 17:13:07 aragorn kernel: input: Lid Switch as /class/input/input5
Oct 12 17:13:07 aragorn kernel: ACPI: Lid Switch [C155]
Oct 12 17:13:07 aragorn kernel: input: Power Button as /class/input/input6
Oct 12 17:13:07 aragorn kernel: ACPI: Power Button [PWRF]
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7dbd42 0027F (v01 HP      Cpu0Ist 00003000 INTL 20060317)
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7dc046 005FA (v01 HP      Cpu0Cst 00003001 INTL 20060317)
Oct 12 17:13:07 aragorn kernel: ACPI: AC Adapter [C23B] (on-line)
Oct 12 17:13:07 aragorn kernel: ACPI: WMI: Mapper loaded
Oct 12 17:13:07 aragorn kernel: Monitor-Mwait will be used to enter C-1 state
Oct 12 17:13:07 aragorn kernel: Monitor-Mwait will be used to enter C-2 state
Oct 12 17:13:07 aragorn kernel: Marking TSC unstable due to TSC halts in idle
Oct 12 17:13:07 aragorn kernel: processor LNXCPU:00: registered as cooling_device7
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7dbc7a 000C8 (v01 HP      Cpu1Ist 00003000 INTL 20060317)
Oct 12 17:13:07 aragorn kernel: ACPI: SSDT 000000007e7dbfc1 00085 (v01 HP      Cpu1Cst 00003000 INTL 20060317)
Oct 12 17:13:07 aragorn kernel: Switching to clocksource hpet
Oct 12 17:13:07 aragorn kernel: agpgart-intel 0000:00:00.0: Intel 965GM Chipset
Oct 12 17:13:07 aragorn kernel: agpgart-intel 0000:00:00.0: detected 7676K stolen memory
Oct 12 17:13:07 aragorn kernel: agpgart-intel 0000:00:00.0: AGP aperture is 256M @ 0xd0000000
Oct 12 17:13:07 aragorn kernel: processor LNXCPU:01: registered as cooling_device8
Oct 12 17:13:07 aragorn kernel: ACPI: Battery Slot [C23D] (battery present)
Oct 12 17:13:07 aragorn kernel: lis3lv02d: hardware type NC2510 found
Oct 12 17:13:07 aragorn kernel: lis3lv02d: 2-byte sensor found
Oct 12 17:13:07 aragorn kernel: input: ST LIS3LV02DL Accelerometer as /class/input/input7
Oct 12 17:13:07 aragorn kernel: Registered led device: hp::hddprotect
Oct 12 17:13:07 aragorn kernel: input: PC Speaker as /class/input/input8
Oct 12 17:13:07 aragorn kernel: cfg80211: Using static regulatory domain info
Oct 12 17:13:07 aragorn kernel: cfg80211: Regulatory domain: EU
Oct 12 17:13:07 aragorn kernel: ^I(start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
Oct 12 17:13:07 aragorn kernel: ^I(2402000 KHz - 2482000 KHz @ 40000 KHz), (600 mBi, 2000 mBm)
Oct 12 17:13:07 aragorn kernel: ^I(5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
Oct 12 17:13:07 aragorn kernel: ^I(5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
Oct 12 17:13:07 aragorn kernel: ^I(5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
Oct 12 17:13:07 aragorn kernel: ^I(5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2000 mBm)
Oct 12 17:13:07 aragorn kernel: ^I(5490000 KHz - 5710000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
Oct 12 17:13:07 aragorn kernel: cfg80211: Calling CRDA for country: EU
Oct 12 17:13:07 aragorn kernel: cfg80211: Calling CRDA for country: EU
Oct 12 17:13:07 aragorn kernel: input: PS/2 Generic Mouse as /class/input/input9
Oct 12 17:13:07 aragorn kernel: usblp0: USB Bidirectional printer dev 4 if 0 alt 1 proto 2 vid 0x03F0 pid 0x3102
Oct 12 17:13:07 aragorn kernel: usbcore: registered new interface driver usblp
Oct 12 17:13:07 aragorn kernel: yenta_cardbus 0000:02:06.0: CardBus bridge found [103c:30c9]
Oct 12 17:13:07 aragorn kernel: yenta_cardbus 0000:02:06.0: ISA IRQ mask 0x0cb8, PCI irq 18
Oct 12 17:13:07 aragorn kernel: yenta_cardbus 0000:02:06.0: Socket status: 30000006
Oct 12 17:13:07 aragorn kernel: pci_bus 0000:02: Raising subordinate bus# of parent bus (#02) from #03 to #06
Oct 12 17:13:07 aragorn kernel: yenta_cardbus 0000:02:06.0: pcmcia: parent PCI bridge I/O window: 0x3000 - 0x3fff
Oct 12 17:13:07 aragorn kernel: yenta_cardbus 0000:02:06.0: pcmcia: parent PCI bridge Memory window: 0xe0100000 - 0xe03fffff
Oct 12 17:13:07 aragorn kernel: yenta_cardbus 0000:02:06.0: pcmcia: parent PCI bridge Memory window: 0x80000000 - 0x83ffffff
Oct 12 17:13:07 aragorn kernel: Synaptics Touchpad, model: 1, fw: 6.3, id: 0x1a0b1, caps: 0xa04711/0xa00000
Oct 12 17:13:07 aragorn kernel: input: SynPS/2 Synaptics TouchPad as /class/input/input10
Oct 12 17:13:07 aragorn kernel: iwlagn: Intel(R) Wireless WiFi Link AGN driver for Linux, 1.3.27kd
Oct 12 17:13:07 aragorn kernel: iwlagn: Copyright(c) 2003-2009 Intel Corporation
Oct 12 17:13:07 aragorn kernel: iwlagn 0000:10:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Oct 12 17:13:07 aragorn kernel: iwlagn 0000:10:00.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: iwlagn 0000:10:00.0: Detected Intel Wireless WiFi Link 4965AGN REV=0x4
Oct 12 17:13:07 aragorn kernel: iwlagn 0000:10:00.0: Tunable channels: 11 802.11bg, 13 802.11a channels
Oct 12 17:13:07 aragorn kernel: iwlagn 0000:10:00.0: irq 27 for MSI/MSI-X
Oct 12 17:13:07 aragorn kernel: phy0: Selected rate control algorithm 'iwl-agn-rs'
Oct 12 17:13:07 aragorn kernel: HDA Intel 0000:00:1b.0: power state changed by ACPI to D0
Oct 12 17:13:07 aragorn kernel: HDA Intel 0000:00:1b.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Oct 12 17:13:07 aragorn kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
Oct 12 17:13:07 aragorn kernel: input: HDA Digital PCBeep as /class/input/input11
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-1, internal journal
Oct 12 17:13:07 aragorn kernel: loop: module loaded
Oct 12 17:13:07 aragorn kernel: input: HP WMI hotkeys as /class/input/input12
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-5, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-2, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-3, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-4, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-6, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-9, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-12, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-8, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 17:13:07 aragorn kernel: EXT3 FS on dm-10, internal journal
Oct 12 17:13:07 aragorn kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 12 17:13:07 aragorn kernel: Adding 2097144k swap on /dev/mapper/main-swap.  Priority:-1 extents:1 across:2097144k 
Oct 12 17:13:07 aragorn kernel: iwlagn 0000:10:00.0: firmware: requesting iwlwifi-4965-2.ucode
Oct 12 17:13:07 aragorn kernel: iwlagn 0000:10:00.0: loaded firmware version 228.57.2.23
Oct 12 17:13:07 aragorn kernel: Registered led device: iwl-phy0::radio
Oct 12 17:13:07 aragorn kernel: Registered led device: iwl-phy0::assoc
Oct 12 17:13:07 aragorn kernel: Registered led device: iwl-phy0::RX
Oct 12 17:13:07 aragorn kernel: Registered led device: iwl-phy0::TX
Oct 12 17:13:07 aragorn kernel: wlan0: deauthenticating from 00:14:c1:38:e5:15 by local choice (reason=3)
Oct 12 17:13:07 aragorn kernel: wlan0: direct probe to AP 00:14:c1:38:e5:15 (try 1)
Oct 12 17:13:07 aragorn kernel: wlan0: direct probe responded
Oct 12 17:13:07 aragorn kernel: wlan0: authenticate with AP 00:14:c1:38:e5:15 (try 1)
Oct 12 17:13:07 aragorn kernel: wlan0: authenticated
Oct 12 17:13:07 aragorn kernel: wlan0: associate with AP 00:14:c1:38:e5:15 (try 1)
Oct 12 17:13:07 aragorn kernel: wlan0: RX AssocResp from 00:14:c1:38:e5:15 (capab=0x411 status=0 aid=1)
Oct 12 17:13:07 aragorn kernel: wlan0: associated
Oct 12 17:13:07 aragorn kernel: RPC: Registered udp transport module.
Oct 12 17:13:07 aragorn kernel: RPC: Registered tcp transport module.
Oct 12 17:13:07 aragorn kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Oct 12 17:13:07 aragorn kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Oct 12 17:13:07 aragorn kernel: NET: Registered protocol family 10
Oct 12 17:13:07 aragorn kernel: lo: Disabled Privacy Extensions
Oct 12 17:13:09 aragorn kernel: lp: driver loaded but no devices found
Oct 12 17:13:09 aragorn kernel: ppdev: user-space parallel port driver
Oct 12 17:13:15 aragorn kernel: wlan0: no IPv6 routers present
Oct 12 17:13:20 aragorn kernel: [drm] Initialized drm 1.1.0 20060810
Oct 12 17:13:20 aragorn kernel: pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 12 17:13:20 aragorn kernel: pci 0000:00:02.0: setting latency timer to 64
Oct 12 17:13:20 aragorn kernel: pci 0000:00:02.0: irq 28 for MSI/MSI-X
Oct 12 17:13:20 aragorn kernel: acpi device:00: registered as cooling_device9
Oct 12 17:13:20 aragorn kernel: input: Video Bus as /class/input/input13
Oct 12 17:13:20 aragorn kernel: ACPI: Video Device [C09A] (multi-head: yes  rom: no  post: no)
Oct 12 17:13:20 aragorn kernel: [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
Oct 12 17:16:11 aragorn kernel: sr0: CDROM not ready.  Make sure there is a disc in the drive.
Oct 12 17:16:11 aragorn kernel: sr0: CDROM not ready.  Make sure there is a disc in the drive.
Oct 12 17:55:15 aragorn kernel: sr0: CDROM not ready.  Make sure there is a disc in the drive.
Oct 12 17:55:15 aragorn kernel: sr0: CDROM not ready.  Make sure there is a disc in the drive.
Oct 12 18:10:35 aragorn kernel: e1000e 0000:00:19.0: irq 26 for MSI/MSI-X
Oct 12 18:10:35 aragorn kernel: e1000e 0000:00:19.0: irq 26 for MSI/MSI-X
Oct 12 18:10:35 aragorn kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Oct 12 18:10:37 aragorn kernel: e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
Oct 12 18:10:37 aragorn kernel: 0000:00:19.0: eth0: 10/100 speed: disabling TSO
Oct 12 18:10:37 aragorn kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 12 18:10:39 aragorn kernel: wlan0: deauthenticating from 00:14:c1:38:e5:15 by local choice (reason=3)
Oct 12 18:10:48 aragorn kernel: eth0: no IPv6 routers present
Oct 12 20:13:29 aragorn kernel: Registered led device: iwl-phy0::radio
Oct 12 20:13:29 aragorn kernel: Registered led device: iwl-phy0::assoc
Oct 12 20:13:29 aragorn kernel: Registered led device: iwl-phy0::RX
Oct 12 20:13:29 aragorn kernel: Registered led device: iwl-phy0::TX
Oct 12 20:13:29 aragorn kernel: ADDRCONF(NETDEV_UP): wlan0: link is not ready
Oct 12 20:13:29 aragorn kernel: wlan0: direct probe to AP 00:14:c1:38:e5:15 (try 1)
Oct 12 20:13:29 aragorn kernel: wlan0: direct probe responded
Oct 12 20:13:29 aragorn kernel: wlan0: authenticate with AP 00:14:c1:38:e5:15 (try 1)
Oct 12 20:13:29 aragorn kernel: wlan0: authenticated
Oct 12 20:13:29 aragorn kernel: wlan0: associate with AP 00:14:c1:38:e5:15 (try 1)
Oct 12 20:13:29 aragorn kernel: wlan0: RX ReassocResp from 00:14:c1:38:e5:15 (capab=0x411 status=0 aid=0)
Oct 12 20:13:29 aragorn kernel: wlan0: invalid aid value 0; bits 15:14 not set
Oct 12 20:13:29 aragorn kernel: wlan0: associated
Oct 12 20:13:29 aragorn kernel: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Oct 12 20:13:30 aragorn kernel: wlan0: deauthenticating from 00:14:c1:38:e5:15 by local choice (reason=3)
Oct 12 20:13:31 aragorn kernel: wlan0: deauthenticating from 00:14:c1:38:e5:15 by local choice (reason=3)
Oct 12 20:13:31 aragorn kernel: wlan0: direct probe to AP 00:14:c1:38:e5:15 (try 1)
Oct 12 20:13:31 aragorn kernel: wlan0: direct probe responded
Oct 12 20:13:31 aragorn kernel: wlan0: authenticate with AP 00:14:c1:38:e5:15 (try 1)
Oct 12 20:13:31 aragorn kernel: wlan0: authenticated
Oct 12 20:13:31 aragorn kernel: wlan0: associate with AP 00:14:c1:38:e5:15 (try 1)
Oct 12 20:13:31 aragorn kernel: wlan0: RX ReassocResp from 00:14:c1:38:e5:15 (capab=0x411 status=0 aid=1)
Oct 12 20:13:31 aragorn kernel: wlan0: associated
Oct 12 20:13:40 aragorn kernel: wlan0: no IPv6 routers present
Oct 12 20:15:06 aragorn kernel: swapper: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 0, comm: swapper Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7243>] iwl_rx_handle+0x3ad/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8125732b>] ? ip_rcv+0x2b8/0x2ef
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0362b95>] ? __iwl_read32+0xaa/0xb9 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02b1fc9>] ? acpi_idle_enter_simple+0xfe/0x12c [processor]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa02b1fbf>] ? acpi_idle_enter_simple+0xf4/0x12c [processor]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81220131>] ? cpuidle_idle_call+0x98/0xf3
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100aed0>] ? cpu_idle+0x5a/0x92
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812aa175>] ? start_secondary+0x17a/0x17f
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 187
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 159
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102886 isolated_anon:32
Oct 12 20:15:07 aragorn kernel:  active_file:10730 inactive_file:10749 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:35291 unstable:0 buffer:50
Oct 12 20:15:07 aragorn kernel:  free:3047 slab_reclaimable:3967 slab_unreclaimable:11608
Oct 12 20:15:07 aragorn kernel:  mapped:21571 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4260kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407564kB active_file:42904kB inactive_file:42868kB unevictable:1600kB isolated(anon):128kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:141164kB mapped:86256kB shmem:76kB slab_reclaimable:15864kB slab_unreclaimable:46424kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 549*4kB 196*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4340kB
Oct 12 20:15:07 aragorn kernel: 59407 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 37518 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 205147, delete 167629, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1361488kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93830 pages shared
Oct 12 20:15:07 aragorn kernel: 433823 pages non-shared
Oct 12 20:15:07 aragorn kernel: iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
Oct 12 20:15:07 aragorn kernel: swapper: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 0, comm: swapper Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7243>] iwl_rx_handle+0x3ad/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0362b95>] ? __iwl_read32+0xaa/0xb9 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02b1fc9>] ? acpi_idle_enter_simple+0xfe/0x12c [processor]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa02b1fbf>] ? acpi_idle_enter_simple+0xf4/0x12c [processor]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81220131>] ? cpuidle_idle_call+0x98/0xf3
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100aed0>] ? cpu_idle+0x5a/0x92
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8129e042>] ? rest_init+0x66/0x68
Oct 12 20:15:07 aragorn kernel:  [<ffffffff814a9c48>] ? start_kernel+0x34d/0x358
Oct 12 20:15:07 aragorn kernel:  [<ffffffff814a929a>] ? x86_64_start_reservations+0xaa/0xae
Oct 12 20:15:07 aragorn kernel:  [<ffffffff814a937f>] ? x86_64_start_kernel+0xe1/0xe8
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 187
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 167
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102888 isolated_anon:32
Oct 12 20:15:07 aragorn kernel:  active_file:10737 inactive_file:10749 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:35575 unstable:0 buffer:50
Oct 12 20:15:07 aragorn kernel:  free:2981 slab_reclaimable:3935 slab_unreclaimable:11684
Oct 12 20:15:07 aragorn kernel:  mapped:21563 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:3996kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407572kB active_file:42932kB inactive_file:42868kB unevictable:1600kB isolated(anon):128kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:142300kB mapped:86224kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:46728kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 609*4kB 135*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4028kB
Oct 12 20:15:07 aragorn kernel: 59639 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 37774 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 205403, delete 167629, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1360464kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93766 pages shared
Oct 12 20:15:07 aragorn kernel: 433957 pages non-shared
Oct 12 20:15:07 aragorn kernel: iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 3 free buffers remaining.
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7218>] iwl_rx_handle+0x382/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8125732b>] ? ip_rcv+0x2b8/0x2ef
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0362b95>] ? __iwl_read32+0xaa/0xb9 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 200
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102838 isolated_anon:65
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36939 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11957
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407376kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):256kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147756kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47820kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:131 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60755 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39168 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206820, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354820kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7243>] iwl_rx_handle+0x3ad/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8125732b>] ? ip_rcv+0x2b8/0x2ef
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0362b95>] ? __iwl_read32+0xaa/0xb9 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 190
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102848 isolated_anon:66
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36970 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11978
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407412kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):264kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147880kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47904kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:165 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60786 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39195 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206846, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354716kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7218>] iwl_rx_handle+0x382/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122c0e6>] ? skb_dequeue+0x60/0x6c
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 190
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102848 isolated_anon:66
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36970 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11978
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407412kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):264kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147880kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47904kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60786 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39195 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206846, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354716kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7218>] iwl_rx_handle+0x382/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122c0e6>] ? skb_dequeue+0x60/0x6c
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 190
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102848 isolated_anon:66
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36970 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11978
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407412kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):264kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147880kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47904kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60786 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39195 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206846, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354716kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7218>] iwl_rx_handle+0x382/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122c0e6>] ? skb_dequeue+0x60/0x6c
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 190
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102848 isolated_anon:66
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36970 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11978
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407412kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):264kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147880kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47904kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60786 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39195 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206846, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354716kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7218>] iwl_rx_handle+0x382/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122c0e6>] ? skb_dequeue+0x60/0x6c
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 190
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102848 isolated_anon:66
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36970 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11978
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407412kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):264kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147880kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47904kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60786 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39195 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206846, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354716kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7218>] iwl_rx_handle+0x382/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122c0e6>] ? skb_dequeue+0x60/0x6c
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 190
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102848 isolated_anon:66
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36970 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11978
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407412kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):264kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147880kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47904kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60786 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39195 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206846, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354716kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: kcryptd: page allocation failure. order:2, mode:0x4020
Oct 12 20:15:07 aragorn kernel: Pid: 1523, comm: kcryptd Not tainted 2.6.32-rc4 #29
Oct 12 20:15:07 aragorn kernel: Call Trace:
Oct 12 20:15:07 aragorn kernel:  <IRQ>  [<ffffffff810a9d87>] __alloc_pages_nodemask+0x5b9/0x632
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810a9e17>] __get_free_pages+0x17/0x46
Oct 12 20:15:07 aragorn kernel:  [<ffffffff810cecff>] __kmalloc_track_caller+0x4e/0x146
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] ? iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122d96d>] __alloc_skb+0x6b/0x161
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b583>] iwl_rx_allocate+0x94/0x30a [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa036b814>] iwl_rx_replenish_now+0x1b/0x28 [iwlcore]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b7218>] iwl_rx_handle+0x382/0x3c6 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa03b778d>] iwl_irq_tasklet_legacy+0x531/0x7a9 [iwlagn]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8122c0e6>] ? skb_dequeue+0x60/0x6c
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81048abf>] tasklet_action+0x76/0xc1
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a31a>] __do_softirq+0xdd/0x197
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100ccdc>] call_softirq+0x1c/0x28
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100e81c>] do_softirq+0x38/0x70
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8104a164>] irq_exit+0x3b/0x7a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff812af5d5>] do_IRQ+0xad/0xc4
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100c553>] ret_from_intr+0x0/0xa
Oct 12 20:15:07 aragorn kernel:  <EOI>  [<ffffffffa02419f4>] ? enc128+0x67f/0x80b [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa024273a>] ? aes_encrypt+0x12/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa022f2cc>] ? crypto_cbc_encrypt+0x131/0x193 [cbc]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0242728>] ? aes_encrypt+0x0/0x14 [aes_x86_64]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81152601>] ? async_encrypt+0x3d/0x3f
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201bbd>] ? crypt_convert+0x1fe/0x290 [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0202077>] ? kcryptd_crypt+0x428/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058e77>] ? worker_thread+0x195/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffffa0201c4f>] ? kcryptd_crypt+0x0/0x44e [dm_crypt]
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105cb19>] ? autoremove_wake_function+0x0/0x3d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff81058ce2>] ? worker_thread+0x0/0x22d
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c79b>] ? kthread+0x82/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbda>] ? child_rip+0xa/0x20
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8105c719>] ? kthread+0x0/0x8a
Oct 12 20:15:07 aragorn kernel:  [<ffffffff8100cbd0>] ? child_rip+0x0/0x20
Oct 12 20:15:07 aragorn kernel: Mem-Info:
Oct 12 20:15:07 aragorn kernel: DMA per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 12 20:15:07 aragorn kernel: DMA32 per-cpu:
Oct 12 20:15:07 aragorn kernel: CPU    0: hi:  186, btch:  31 usd: 161
Oct 12 20:15:07 aragorn kernel: CPU    1: hi:  186, btch:  31 usd: 190
Oct 12 20:15:07 aragorn kernel: active_anon:306195 inactive_anon:102848 isolated_anon:66
Oct 12 20:15:07 aragorn kernel:  active_file:10600 inactive_file:10613 isolated_file:0
Oct 12 20:15:07 aragorn kernel:  unevictable:400 dirty:0 writeback:36970 unstable:0 buffer:51
Oct 12 20:15:07 aragorn kernel:  free:3171 slab_reclaimable:3935 slab_unreclaimable:11978
Oct 12 20:15:07 aragorn kernel:  mapped:21315 shmem:19 pagetables:4243 bounce:0
Oct 12 20:15:07 aragorn kernel: DMA free:7928kB min:40kB low:48kB high:60kB active_anon:3880kB inactive_anon:3980kB active_file:16kB inactive_file:128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 1976 1976 1976
Oct 12 20:15:07 aragorn kernel: DMA32 free:4756kB min:5664kB low:7080kB high:8496kB active_anon:1220900kB inactive_anon:407412kB active_file:42384kB inactive_file:42324kB unevictable:1600kB isolated(anon):264kB isolated(file):0kB present:2023748kB mlocked:1600kB dirty:0kB writeback:147880kB mapped:85232kB shmem:76kB slab_reclaimable:15736kB slab_unreclaimable:47904kB kernel_stack:1440kB pagetables:16956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 12 20:15:07 aragorn kernel: lowmem_reserve[]: 0 0 0 0
Oct 12 20:15:07 aragorn kernel: DMA: 6*4kB 6*8kB 5*16kB 5*32kB 5*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 1*4096kB = 7928kB
Oct 12 20:15:07 aragorn kernel: DMA32: 923*4kB 17*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4756kB
Oct 12 20:15:07 aragorn kernel: 60786 total pagecache pages
Oct 12 20:15:07 aragorn kernel: 39195 pages in swap cache
Oct 12 20:15:07 aragorn kernel: Swap cache stats: add 206846, delete 167651, find 11222/12777
Oct 12 20:15:07 aragorn kernel: Free swap  = 1354716kB
Oct 12 20:15:07 aragorn kernel: Total swap = 2097144kB
Oct 12 20:15:07 aragorn kernel: 518064 pages RAM
Oct 12 20:15:07 aragorn kernel: 10503 pages reserved
Oct 12 20:15:07 aragorn kernel: 93541 pages shared
Oct 12 20:15:07 aragorn kernel: 433976 pages non-shared
Oct 12 20:15:07 aragorn kernel: __ratelimit: 45 callbacks suppressed

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Wed, Oct 14, 2009 at 03:10:08PM +0200, Frans Pop wrote:
> On Wednesday 14 October 2009, Mel Gorman wrote:
> > I think this is very significant. Either that change needs to be backed
> > out or more likely, __GFP_NOWARN needs to be specified and warnings
> > *only* printed when the RX buffers are really low. My expectation would
> > be that some GFP_ATOMIC allocations fail during refill but the fact they
> > fail wakes kswapd to reclaim order-2 pages while the RX buffers in the
> > pool are consumed.
> 
> Sorry I did not actually mention this, but the SKB failures I get with .32 
> have loads of the "Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 
> free buffers remaining." errors. That's why I don't think your patch will 
> help anything.
> 
> zgrep "Only 0 free buffers remaining" /var/log/kern.log* | wc -l
> 84
> 
> OK, they are all GPF_ATOMIC and not GPF_KERNEL, but they also almost all 
> have "0 free buffers"! Next to the 84 warnings for 0 remaining I only have 
> one with "3 free buffers" and one with "1 free buffers".
> 

This is fairly important. It shows that the refills are not keeping up
with the GFP_ATOMIC usage. I'm not sure what to do with this. As the
driver introduced GFP_ATOMIC usage at all, I'm tempted to say revert the
changes in the driver that makes use of GFP_ATOMIC but I'm not the
maintainer. They could also consider having a GFP_ATOMIC-optimistic,
GFP_KERNEL-if-no-buffers-free-and-directly-allocating with GFP_KERNEL
refills always happening in the tasklet.

However, it might be just avoiding the MM problem on my part. It's possible
that if I figure out what went wrong in mm and drivers use of GFP_ATOMIC
will be swept under the carpet.

> And that does not even count the rate limitting:
> Oct 12 20:15:07 aragorn kernel: __ratelimit: 45 callbacks suppressed
> Oct 12 20:25:19 aragorn kernel: __ratelimit: 27 callbacks suppressed
> Oct 12 20:25:20 aragorn kernel: __ratelimit: 2 callbacks suppressed
> 
> Attached the kernel log for one test I did with .32.
> 
> > > In both cases I no longer get SKB errors, but instead (?) I get
> > > firmware errors:
> > > iwlagn 0000:10:00.0: Microcode SW error detected.  Restarting
> > > 0x2000000.
> >
> > I am no wireless expert, but that looks like an separate problem to me.
> > I don't see how an allocation failure could trigger errors in the
> > microcode.
> 
> Yes, it is a separate problem, but it is still significant that reverting 
> that patch triggers them in the extreme swap situation.
> 

True.

> > > With your patch on .32-rc4 I still get the SKB errors, so it does not
> > > seem to help. The only change there may have been is that the desktop
> > > was frozen longer than without the patch, but that is an impression,
> > > not a hard fact.
> >
> > Actually, that's fairly interesting and I think justifies pushing the
> > patch. Direct reclaim can stall processes in a user-visible manner which
> > kswapd is meant to avoid in the majority of cases but is tricky to
> > quantify without instrumenting the kernel to measure direct reclaim
> > frequency and latency (I have WIP tracepoints for this but it's still a
> > WIP). If you notice shorter stalls with the patch applied, it means that
> > kswapd really did need to be informed of the problems.
> 
> No, I thought I saw _longer_ stalls with your patch applied...
> 

Sorry, I misinterpreted. If the stalls are longer, it likely means that
kswapd is doing more work and causing more IO when applied as it tries to
get order-2 pages free. You said you still got SKB errors. Were there any
significant change to the number of failures or can that be told?

> > There still has not been a mm-change identified that makes fragmentation
> > significantly worse.
> 
> My bisection shows a very clear point, even if not an individual commit, in 
> the 'akpm' merge where SKB errors suddenly become *much* more frequent and 
> easy to trigger.
> I'm sorry to say this, but the fact that nothing has been identified yet is 
> IMO the result of a lack of effort, not because there is no such change.
> 

I apologise if I've given that impression. I've been starting at the commits
but could not find an obvious candidate within the page allocator itself which
is why I've been looking at other areas. I put together a hack that allocated
order-2 atomics at a constant rate and order-5 atomics at a lower rate to
try replicate the problem without drivers. I ran some workloads but I wasn't
able to get reliable figures that would have allowed me to investigate further.

> > The majority of the wireless reports have been in 
> > this driver and I think we have the problem commit there. The only other
> > is a firmware loading problem in e100 after resume that fails to make an
> > atomic order-5 fail.
> 
> Not exactly true. Bartlomiej's report was about ipw2200, so there are at 
> least 3 different drivers involved, two wireless and one wired. Besides 
> that one report is related to heavy swap, one to resume and one to driver 
> reload.
> So it's much more likely that there is some common regression (in mm) that 
> affected all three than that there are three unrelated regressions.

Very very likely, I'm not denying this.

> And although both of the others did extremely high allocations, they both 
> started appearing in the same timeframe. And Bart's very first report 
> linked it to mm changes.
> 
> > It's possible that something has changed in resume 
> > in the 2.6.31 window there - maybe something like drivers now reload
> > during resume where they didn't previously or less memory being pushed
> > to swap during resume.
> 
> IMO you're sticking your head in the sand here. 

No. If I was sticking my head in the sand, I would have dismissed this
entirely as "GFP_ATOMIC allocations can fail boo hoo hoo deal with it".

What I'm trying to identify what changed that would affect fragmentation
but that is not within the page allocator itself - largely because with
the exception of the patch I gave you, I couldn't find obvious breakage.

You highlighted the first akpm merge so lets look closer at that as I don't
think there is anything more I can do with the wireless driver other than the
suggestions made already. I looked at this already but I felt fixing GFP_ATOMIC
in wireless was the more likely fix. 

Here is what you said about the merge.

====
For a good overview of the area, use 'gitk f83b1e61..517d0869'.

v2.6.30-5466-ga1dd268   mm: use alloc_pages_exact in alloc_large_system_hash
        2.3     +-
v2.6.30-5478-ge9bb35d   mm: setup_per_zone_inactive_ratio - fix comment and..
        2.5     +-
v2.6.30-5486-g35282a2   migration: only migrate_prep() once per move_pages()
        2.6     -|+|-   not quite conclusive...
v2.6.30-5492-gbce7394   page-allocator: reset wmark_min and inactive ratio..
        2.4     -|-
====

This is what I found. The following were the possible commits that might
be causing the problem.

d239171..72807a7 -- page allocator
	These are the bulk of the page-allocator changes that happened int
	the 2.6.30..2.6.31 cycle. It's also the location of the change to
	kswapd that I sent you a patch for. If there was a marked increase
	in the number of failures before and after this patchset, it means
	that I was wrong about the problem not being in the page allocator
	and I have to go back and keep looking. However, you report that

	commit e9bb35d   mm: setup_per_zone_inactive_ratio - fix comment

	had relatively good results - relatively being that it didn't fail
	on the first try. In my head, these patches have been struck off the
	list of possibilities and is why I've been looking in other subsystems.

56e49d2..f166777 -- reclaim
	I would have considered this strong candidates except again, the last
	good commit happened after this point. If other obvious candidates
	don't crop up, it might be worth double checking within this range, particularly
	commit 56e49d2 vmscan: evict use-once pages first
	as it is targeted at streaming-IO workloads which would include
	your music workload. This commit also will cleanly revert on
	mainline so is relatively easy to test

5c87ead..e9bb35d -- inactive ratio changes
	These patches should be harmless but just in case, please
	compare the output of
	# grep inactive_ratio /proc/zoneinfo
	on 2.6.30 and 2.6.31 and make sure the ratios are the same.

e9bb35d..bce7394 -- various changes
	According to your analysis, this is the most likely location of
	the problem commit.

	Commit b70d94e altered how zonelists were selected during
	allocation. This was tested fairly heavily but if the testing
	missed something, it would mean that some allocations are not
	using the zones they should be.  However, my expectation would
	be that mistakes here would have severe consequences affecting a
	large number of people. This does not revert cleanly but there is
	an untested patch below that should do the job. While it's hard to
	imagine this patch being the problem, it's the most likely commit
	with the range of commits your analysis identified.

	Commit bc75d33 is totally harmless but it mentions min_free_kbytes. I
	checked on my machine to make sure min_free_kbytes was the same on both
	2.6.30 and 2.6.31. Can you check that this is true for your machine? If
	min_free_kbytes decreased, it could explain GFP_ATOMIC failures.

	An extremely unlikely candidate is 75927af8. For this to be a problem,
	much of your userspace would have to be calling madvise() with
	stupid parameters and depending on it silently ignore the
	parameters

	A vague potential candidate for swapless systems is 69c85481 but
	your machine has swap so it can't be this.

	Commit bce7394 affects min_free_kbytes but only on hotplug so it
	can't be this either for your machine

After this point, your analysis indicates that things are already broken
but lets look at some of the candidates anyway.  Out of curiousity,
was CONFIG_UNEVICTABLE_LRU unset in your .config for 2.6.30? I could
only find your 2.6.31 .config. If it was, it might be worth reverting
6837765963f1723e80ca97b1fae660f3a60d77df and unsetting it in 2.6.31 and
seeing what happens.

Commit 8cab4754d24a0f2e05920170c845bd84472814c6 keeps pages on the active
lists for longer than 2.6.30 did. It's possible the fewer reclaim decisions
is delaying lumpy reclaim.

CONFIG_NUMA is not set in your config, so the zone_reclaim() changes
around 24cf72518c79cdcda486ed26074ff8151291cf65 can be discounted.

Commit ee993b135ec75a93bd5c45e636bb210d2975159b altered how lumpy
reclaim works but it should have been harmless. It does not cleanly
revert but it's easy to manually revert.

I didn't spot any other patches that might be potential problems in the
commits.

> I'm not saying that mm is the only issue here, but I'm convinced that there 
> _is_ an mm change that has contributed in a major way to these issues, 
> even if we've not yet been able to identify it.
> 
> > -			    net_ratelimit())
> > +			    net_ratelimit()) {
> >  				IWL_CRIT(priv, "Failed to allocate SKB buffer with %s. Only %u free
> > buffers remaining.\n", priority == GFP_ATOMIC ?  "GFP_ATOMIC" :
> > "GFP_KERNEL",
> 
> Haven't you broken the test 'priority == GFP_ATOMIC' here by setting 
> priority to GFP_ATOMIC|__GFP_NOWARN?
> 

Yes, I did, but as you say that this error message is showing up and buffers
are all depleted, it's not even close to being the right fix. It'd only
be relevant if that error message was showing up with buffers remaining in
the queue.


Revert commit b70d94ee

---

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 557bdad..3a94e4b 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -21,8 +21,7 @@ struct vm_area_struct;
 #define __GFP_DMA	((__force gfp_t)0x01u)
 #define __GFP_HIGHMEM	((__force gfp_t)0x02u)
 #define __GFP_DMA32	((__force gfp_t)0x04u)
-#define __GFP_MOVABLE	((__force gfp_t)0x08u)  /* Page is movable */
-#define GFP_ZONEMASK	(__GFP_DMA|__GFP_HIGHMEM|__GFP_DMA32|__GFP_MOVABLE)
+
 /*
  * Action modifiers - doesn't change the zoning
  *
@@ -52,6 +51,7 @@ struct vm_area_struct;
 #define __GFP_HARDWALL   ((__force gfp_t)0x20000u) /* Enforce hardwall cpuset memory allocs */
 #define __GFP_THISNODE	((__force gfp_t)0x40000u)/* No fallback, no policies */
 #define __GFP_RECLAIMABLE ((__force gfp_t)0x80000u) /* Page is reclaimable */
+#define __GFP_MOVABLE	((__force gfp_t)0x100000u)  /* Page is movable */
 
 #ifdef CONFIG_KMEMCHECK
 #define __GFP_NOTRACK	((__force gfp_t)0x200000u)  /* Don't track with kmemcheck */
@@ -128,105 +128,24 @@ static inline int allocflags_to_migratetype(gfp_t gfp_flags)
 		((gfp_flags & __GFP_RECLAIMABLE) != 0);
 }
 
-#ifdef CONFIG_HIGHMEM
-#define OPT_ZONE_HIGHMEM ZONE_HIGHMEM
-#else
-#define OPT_ZONE_HIGHMEM ZONE_NORMAL
-#endif
-
+static inline enum zone_type gfp_zone(gfp_t flags)
+{
 #ifdef CONFIG_ZONE_DMA
-#define OPT_ZONE_DMA ZONE_DMA
-#else
-#define OPT_ZONE_DMA ZONE_NORMAL
+	if (flags & __GFP_DMA)
+		return ZONE_DMA;
 #endif
-
 #ifdef CONFIG_ZONE_DMA32
-#define OPT_ZONE_DMA32 ZONE_DMA32
-#else
-#define OPT_ZONE_DMA32 ZONE_NORMAL
-#endif
-
-/*
- * GFP_ZONE_TABLE is a word size bitstring that is used for looking up the
- * zone to use given the lowest 4 bits of gfp_t. Entries are ZONE_SHIFT long
- * and there are 16 of them to cover all possible combinations of
- * __GFP_DMA, __GFP_DMA32, __GFP_MOVABLE and __GFP_HIGHMEM
- *
- * The zone fallback order is MOVABLE=>HIGHMEM=>NORMAL=>DMA32=>DMA.
- * But GFP_MOVABLE is not only a zone specifier but also an allocation
- * policy. Therefore __GFP_MOVABLE plus another zone selector is valid.
- * Only 1bit of the lowest 3 bit (DMA,DMA32,HIGHMEM) can be set to "1".
- *
- *       bit       result
- *       =================
- *       0x0    => NORMAL
- *       0x1    => DMA or NORMAL
- *       0x2    => HIGHMEM or NORMAL
- *       0x3    => BAD (DMA+HIGHMEM)
- *       0x4    => DMA32 or DMA or NORMAL
- *       0x5    => BAD (DMA+DMA32)
- *       0x6    => BAD (HIGHMEM+DMA32)
- *       0x7    => BAD (HIGHMEM+DMA32+DMA)
- *       0x8    => NORMAL (MOVABLE+0)
- *       0x9    => DMA or NORMAL (MOVABLE+DMA)
- *       0xa    => MOVABLE (Movable is valid only if HIGHMEM is set too)
- *       0xb    => BAD (MOVABLE+HIGHMEM+DMA)
- *       0xc    => DMA32 (MOVABLE+HIGHMEM+DMA32)
- *       0xd    => BAD (MOVABLE+DMA32+DMA)
- *       0xe    => BAD (MOVABLE+DMA32+HIGHMEM)
- *       0xf    => BAD (MOVABLE+DMA32+HIGHMEM+DMA)
- *
- * ZONES_SHIFT must be <= 2 on 32 bit platforms.
- */
-
-#if 16 * ZONES_SHIFT > BITS_PER_LONG
-#error ZONES_SHIFT too large to create GFP_ZONE_TABLE integer
+	if (flags & __GFP_DMA32)
+		return ZONE_DMA32;
 #endif
-
-#define GFP_ZONE_TABLE ( \
-	(ZONE_NORMAL << 0 * ZONES_SHIFT)				\
-	| (OPT_ZONE_DMA << __GFP_DMA * ZONES_SHIFT) 			\
-	| (OPT_ZONE_HIGHMEM << __GFP_HIGHMEM * ZONES_SHIFT)		\
-	| (OPT_ZONE_DMA32 << __GFP_DMA32 * ZONES_SHIFT)			\
-	| (ZONE_NORMAL << __GFP_MOVABLE * ZONES_SHIFT)			\
-	| (OPT_ZONE_DMA << (__GFP_MOVABLE | __GFP_DMA) * ZONES_SHIFT)	\
-	| (ZONE_MOVABLE << (__GFP_MOVABLE | __GFP_HIGHMEM) * ZONES_SHIFT)\
-	| (OPT_ZONE_DMA32 << (__GFP_MOVABLE | __GFP_DMA32) * ZONES_SHIFT)\
-)
-
-/*
- * GFP_ZONE_BAD is a bitmap for all combination of __GFP_DMA, __GFP_DMA32
- * __GFP_HIGHMEM and __GFP_MOVABLE that are not permitted. One flag per
- * entry starting with bit 0. Bit is set if the combination is not
- * allowed.
- */
-#define GFP_ZONE_BAD ( \
-	1 << (__GFP_DMA | __GFP_HIGHMEM)				\
-	| 1 << (__GFP_DMA | __GFP_DMA32)				\
-	| 1 << (__GFP_DMA32 | __GFP_HIGHMEM)				\
-	| 1 << (__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM)		\
-	| 1 << (__GFP_MOVABLE | __GFP_HIGHMEM | __GFP_DMA)		\
-	| 1 << (__GFP_MOVABLE | __GFP_DMA32 | __GFP_DMA)		\
-	| 1 << (__GFP_MOVABLE | __GFP_DMA32 | __GFP_HIGHMEM)		\
-	| 1 << (__GFP_MOVABLE | __GFP_DMA32 | __GFP_DMA | __GFP_HIGHMEM)\
-)
-
-static inline enum zone_type gfp_zone(gfp_t flags)
-{
-	enum zone_type z;
-	int bit = flags & GFP_ZONEMASK;
-
-	z = (GFP_ZONE_TABLE >> (bit * ZONES_SHIFT)) &
-					 ((1 << ZONES_SHIFT) - 1);
-
-	if (__builtin_constant_p(bit))
-		MAYBE_BUILD_BUG_ON((GFP_ZONE_BAD >> bit) & 1);
-	else {
-#ifdef CONFIG_DEBUG_VM
-		BUG_ON((GFP_ZONE_BAD >> bit) & 1);
+	if ((flags & (__GFP_HIGHMEM | __GFP_MOVABLE)) ==
+			(__GFP_HIGHMEM | __GFP_MOVABLE))
+		return ZONE_MOVABLE;
+#ifdef CONFIG_HIGHMEM
+	if (flags & __GFP_HIGHMEM)
+		return ZONE_HIGHMEM;
 #endif
-	}
-	return z;
+	return ZONE_NORMAL;
 }
 
 /*
-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Wednesday 14 October 2009, Mel Gorman wrote:
> You highlighted the first akpm merge so lets look closer at that as I
> don't think there is anything more I can do with the wireless driver
> other than the suggestions made already.

Thanks a lot for that analysis Mel. I'll see if I can come up with 
additional data based of the info you provide here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

Some initial results; all negative I'm afraid.

On Wednesday 14 October 2009, Mel Gorman wrote:
> This is what I found. The following were the possible commits that might
> be causing the problem.

> 56e49d2..f166777 -- reclaim
>         I would have considered this strong candidates except again, the
>         last good commit happened after this point. If other obvious
>         candidates don't crop up, it might be worth double checking
>         within this range, particularly commit 56e49d2 vmscan: evict
>         use-once pages first as it is targeted at streaming-IO workloads
>         which would include your music workload.

Reverted 56e49d2 on top of .31.1; no change.

> 5c87ead..e9bb35d -- inactive ratio changes
>         These patches should be harmless but just in case, please
>         compare the output of
>         # grep inactive_ratio /proc/zoneinfo
>         on 2.6.30 and 2.6.31 and make sure the ratios are the same.

The same for both (and for .32). DMA: 1; DMA32: 3

>         Commit b70d94e altered how zonelists were selected during
>         allocation. This was tested fairly heavily but if the testing
>         missed something, it would mean that some allocations are not
>         using the zones they should be.

Reverted on top of .31.1; no change.

>         Commit bc75d33 is totally harmless but it mentions
>         min_free_kbytes. I checked on my machine to make sure
>         min_free_kbytes was the same on both 2.6.30 and 2.6.31. Can you
>         check that this is true for your machine? If min_free_kbytes
>         decreased, it could explain GFP_ATOMIC failures.

Virtually identical. .30: 5704; .31/.32: 5711

> After this point, your analysis indicates that things are already broken
> but lets look at some of the candidates anyway.  Out of curiousity,
> was CONFIG_UNEVICTABLE_LRU unset in your .config for 2.6.30? I could
> only find your 2.6.31 .config. If it was, it might be worth reverting
> 6837765963f1723e80ca97b1fae660f3a60d77df and unsetting it in 2.6.31 and
> seeing what happens.

CONFIG_UNEVICTABLE_LRU was set and during bisections I've always accepted 
the default, which was "y".

> Commit ee993b135ec75a93bd5c45e636bb210d2975159b altered how lumpy
> reclaim works but it should have been harmless. It does not cleanly
> revert but it's easy to manually revert.

Reverted on top of .31.1; no change.

I'll do some more digging in the 'akpm' merge.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Wed, Oct 14, 2009 at 08:34:56PM +0200, Frans Pop wrote:
> Some initial results; all negative I'm afraid.
> 

These are highly unlikely candidates. I say highly unlikely because they
are before the page allocator patches when your analysis indicated
things were ok.

Commit 70ac23c readahead: sequential mmap readahead
	This affects readahead for mmap() and could have an impact on the
	number of allocations made by the streaming IO. This might be
	generating more bursty network traffic in 2.6.31 than 2.6.30 and
	affecting the allocation apttern enough to cause problems

Commit 2fad6f5 readahead: enforce full readahead size on async mmap readahead
	Another readahead change that may affect the rate of network
	traffic being generated when streaming IO over the network

Commit 10be0b3 readahead: introduce context readahead algorithm
	By using readahead in more situations, it again may be affecting
	the burst rate of network traffic and the rate of GFP_ATOMIC arrivals

Commit 78dc583 vmscan: low order lumpy reclaim also should use PAGEOUT_IO_SYNC
	Very low probability that this is a problem, but it affects
	lumpy reclaim and so has to be considered. It's an awkward
	revert but I think the most important part is just to revert the
	condition that checks if congestion_wait() should be called or not

I relooked at the page allocator patches themselves just in case. Of the
patches in there, I came up with

Commit 11e33f6 page allocator: break up the allocator entry point into fast and slow paths
	This is possibly the most disruptive patch in the set. It should
	not have affected behaviour but the complexity of the patch is
	quite high. I did spot an oddity whereby a process exiting making
	a __GFP_NOFAIL allocation can ignore watermarks. It's unlikely
	this is the problem but as the journal layer uses __GFP_NOFAIL,
	you never know - it might be pushing things down low enough for
	other watermark checks to fail. Patch is below. This is also the
	patch that cause kswapd to wake up less. I sent a patch for that
	problem but I still don't know if it reduced the number of
	failures for you or not.

Commit f2260e6 page allocator: update NR_FREE_PAGES only as necessary
	This patch affects the timing of when NR_FREE_PAGES is updated.
	The reclaim algorithm makes decisions based on this NR_FREE_PAGES
	value.	Crucially, the value can determine if the anon list is force
	scanned or not. The window during which this can make a difference
	should be extremely small but maybe it's enough to make a difference.

Outside the range of commits suspected of causing problems was the
following. It's extremely low probability

Commit 8aa7e84 Fix congestion_wait() sync/async vs read/write confusion
	This patch alters the call to congestion_wait() in the page
	allocator. Frankly, I don't get the change but it might worth
	checking if replacing BLK_RW_ASYNC with WRITE on top of 2.6.31
	makes any difference

After a lot more eyeballing, the best next candidate within mm is the
following patch. Should be tested on it's own and in combination with
the wakeup-kswapd patch sent before.

====

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Thursday 15 October 2009, Mel Gorman wrote:
> After a lot more eyeballing, the best next candidate within mm is the
> following patch. Should be tested on it's own and in combination with
> the wakeup-kswapd patch sent before.
>
> From 4e8b5217f51a00caee527e4e8d8e46fe9f82b482 Mon Sep 17 00:00:00 2001
> From: Mel Gorman <mel@csn.ul.ie>
> Date: Thu, 15 Oct 2009 00:17:05 +0100
> Subject: [PATCH] page allocator: Direct reclaim should always obey
> watermarks
>
> ALLOC_NO_WATERMARKS should be cleared when trying to allocate from the
> free-lists after a direct reclaim.

I've tested the two patches together and this seems like a definite
improvement. I still get SKB errors on the first test, but the desktop
freezes are a lot shorter and the total time needed to load the 3rd gitk
goes down from ~2:15 to ~1:15. The counter in gitk of the number of
loaded commits goes up visibly faster and with fewer halts.

This is on top of current mainline with the RX_LOW_WATERMARK in iwlagn
at it's current value (8).

Here are the allocation failures for 2 consecutive tests. Note that the
first test still shows quite a lot of failures, but the second test hardly
had any at all (I still had music skips though).

[  232.845116] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
[  232.845116] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
[  232.873009] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
[  232.884545] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
[  240.121577] __ratelimit: 26 callbacks suppressed
[  240.121634] __ratelimit: 6 callbacks suppressed
[  240.124006] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 6 free buffers remaining.
[  304.335767] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
[  304.335767] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
[  304.374729] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
[  309.446481] __ratelimit: 5 callbacks suppressed
[  309.450197] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.

[  525.912934] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 5 free buffers remaining.
[  525.953939] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 7 free buffers remaining.
[  536.058171] __ratelimit: 1 callbacks suppressed

Thanks,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Thu, Oct 15, 2009 at 10:15:09PM +0200, Frans Pop wrote:
> On Thursday 15 October 2009, Mel Gorman wrote:
> > After a lot more eyeballing, the best next candidate within mm is the
> > following patch. Should be tested on it's own and in combination with
> > the wakeup-kswapd patch sent before.
> >
> > From 4e8b5217f51a00caee527e4e8d8e46fe9f82b482 Mon Sep 17 00:00:00 2001
> > From: Mel Gorman <mel@csn.ul.ie>
> > Date: Thu, 15 Oct 2009 00:17:05 +0100
> > Subject: [PATCH] page allocator: Direct reclaim should always obey
> > watermarks
> >
> > ALLOC_NO_WATERMARKS should be cleared when trying to allocate from the
> > free-lists after a direct reclaim.
> 
> I've tested the two patches together and this seems like a definite
> improvement.

You probably don't need the mental image, but this made me do a happy
dance. Certainly helps my cold!

> I still get SKB errors on the first test, but the desktop
> freezes are a lot shorter and the total time needed to load the 3rd gitk
> goes down from ~2:15 to ~1:15. The counter in gitk of the number of
> loaded commits goes up visibly faster and with fewer halts.
> 

This brings us close to the state of affairs before the akpm merge.
There might still be something missing in either the mm area or the wireless
driver but any improvement is better than none.

> This is on top of current mainline with the RX_LOW_WATERMARK in iwlagn
> at it's current value (8).
> 
> Here are the allocation failures for 2 consecutive tests. Note that the
> first test still shows quite a lot of failures, but the second test hardly
> had any at all (I still had music skips though).
> 

So, we are still dealing with three problems.

1. GFP_ATOMICS were introduced to the wireless driver in the
   2.6.30..2.6.31 timeframe. It has been more or less identified as being
   the tasklet off-loading and the pools being depleted too easily. This
   still needs to be fixed. 

2. There is also some firmware reloading problem of an unknown source

3. There was an mm regression that made GFP_ATOMIC failures much worse.
   This is at least partially due to tasks exiting being able to go below the
   watermarks and kswapd not being woken up when it should be.  This could
   be the source of the allocation failures on resume that have nothing to
   do with the iwlagn wireless driver.

I am going to put together the pair of patches against mainline with a
recommendation they be also applied for 2.6.31.5. I'll keep looking to
see can I spot another possible candidate for GFP_ATOMIC being worse
than it was.

> [  232.845116] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> [  232.845116] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> [  232.873009] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> [  232.884545] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> [  240.121577] __ratelimit: 26 callbacks suppressed
> [  240.121634] __ratelimit: 6 callbacks suppressed
> [  240.124006] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 6 free buffers remaining.
> [  304.335767] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> [  304.335767] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> [  304.374729] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> [  309.446481] __ratelimit: 5 callbacks suppressed
> [  309.450197] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> 
> [  525.912934] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 5 free buffers remaining.
> [  525.953939] iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 7 free buffers remaining.
> [  536.058171] __ratelimit: 1 callbacks suppressed
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

On Wed, 2009-10-14 at 06:10 -0700, Frans Pop wrote:
> On Wednesday 14 October 2009, Mel Gorman wrote:
> > The majority of the wireless reports have been in 
> > this driver and I think we have the problem commit there. The only other
> > is a firmware loading problem in e100 after resume that fails to make an
> > atomic order-5 fail.
> 
> Not exactly true. Bartlomiej's report was about ipw2200, so there are at 
> least 3 different drivers involved, two wireless and one wired. Besides 
> that one report is related to heavy swap, one to resume and one to driver 
> reload.

Another report arrived today. Please see
http://thread.gmane.org/gmane.linux.kernel.wireless.general/40858 - it
is an order-5 allocation failure during driver reload.

Reinette

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

Another long mail, sorry.

On Wednesday 14 October 2009, Frans Pop wrote:
> > There still has not been a mm-change identified that makes
> > fragmentation significantly worse.
>
> My bisection shows a very clear point, even if not an individual commit,
> in the 'akpm' merge where SKB errors suddenly become *much* more
> frequent and easy to trigger.
> I'm sorry to say this, but the fact that nothing has been identified yet
> is IMO the result of a lack of effort, not because there is no such
> change.

I was wrong. It turns out that I was creating the variations in the test 
results around the akpm merge myself by tiny changes in the way I ran the 
tests. It took another round of about 30 compilations and tests purely in 
this range to show that, but those same tests also made me aware of other 
patterns I should look at.

Until a few days ago I was concentrating on "do I see SKB allocation errors 
or not". Since then I've also been looking more consciously at when they 
happen, at disk access patterns and at desktop freeze patterns.

I think I did mention before that this whole issue is rather subtle :-/
So, my apologies for finguering the wrong area for so long, but it looked 
solid given the info available at the time.

On Thursday 15 October 2009, Mel Gorman wrote:
> Outside the range of commits suspected of causing problems was the
> following. It's extremely low probability
>
> Commit 8aa7e84 Fix congestion_wait() sync/async vs read/write confusion
>         This patch alters the call to congestion_wait() in the page
>         allocator. Frankly, I don't get the change but it might worth
>         checking if replacing BLK_RW_ASYNC with WRITE on top of 2.6.31
>         makes any difference

This is the real culprit. Mel: thanks very much for looking beyond the 
area I identified. Your overview of mm changes was exactly what I needed 
and really helped a lot during my later tests.

This commit definitely causes most of the problems; confirmed by reverting 
it on top of 2.6.31 (also requires reverting 373c0a7e, which is a later 
build fix).

The rest of this mail gives details on my tests and how I reached the above 
conclusion.

TEST BASELINE (2.6.30)
======================
I mentioned in an earlier mail that I run three instances of gitk for my 
tests. Loading gitk seems to consist of 3 phases:
1) general initial scan of the repository (branches?)
2) reading commits: commit counter increases
3) reading references (including bisection good/bad points) and
   uncommitted changes

Below times and comments per stage when the test is run with 2.6.30. As my 
test starts after a clean boot, buffers are mostly empty.

1st instance: 'gitk v2.6.29..master' (preparation)
1) ~20 seconds; user interface is mostly blank
2) ~5 seconds to read 35.000 commits; user interface is updated and counter
   increases steadily as they are read
3) ~10 seconds; "branch"/"follows"/"precedes" info and tags are filled
   in; fairly heavy disk activity

2st instance: 'gitk master' (preparation)
1) 0 seconds (because data is already buffered)
2) ~25 seconds to read 167500 commits; counter increases steadily
3) 1-2 seconds (because data is already buffered)

3st instance: 'gitk master' (the actual test)
1) 0 seconds because data is already buffered
2) ~55 seconds due to swapping overhead; minor music skip around commit
   110.000; counter slower after 90.000, some short halts, but generally
   increases steadily; moderate disk activity
3) ~55-60 seconds; because buffers have been emptied data must by read
   again, with swapping; very heavy disk activity; fairly long music
   skip (15-20 seconds), but no SKB allocation errors

So, the loading of the 3rd instance takes 1.5 minutes longer than the 
second because of the swapping. And phase 3) is most affected by it.

AFTER WIRELESS CHANGE
=====================
After commit 4752c93c30 ("iwlcore: Allow skb allocation from tasklet") I 
start getting the SKB errors. They can be triggered reliably if the whole 
test is repeated 1 or 2 times, but generally not the first time the test 
is run.

Or so I thought for a long time.
It turns out that I will get SKB errors during the first run if I'm
"sloppy" in the test execution. For example if I wait too long before 
switching from the last gitk instance to konsole where I have 
a 'tail -f /var/log/kern.log' running.
Another factor is the state of the repository: do I have master checked 
out, or an older branch, or am I in the middle of a bisection. This 
influences how data is read from the disk and thus the test results.
A last factor may be the size of the kernel I'm using: my test/bisect 
kernel is significantly smaller than my regular kernel.

If the test is run completely cleanly, I will not get SKB errors during the 
first run. Also, this change does not affect the timings of the test at 
all: the total load time of the 3rd instance is still ~1:55 and music 
skips happen in roughly the same places. The pattern of disk activity also 
remains unchanged.

If I do *not* run the test cleanly, any SKB errors during the first test 
run will always be during phase 3), never during phase 2). This is what I 
saw during tests in the 'akpm' range, and explains the inconsistent 
results there.

After discovering this I've made a copy of the git repo so that I always 
test using the exact same state and tightened my test procedure.

AFTER congestion_wait CHANGE
============================
If I test commit 9f2d8be, which is just before the congestion_wait() 
change, I still get the same pattern as described above. But when I test 
with 8aa7e84 ("Fix congestion_wait() sync/async vs read/write confusion"), 
things change dramatically when the 3rd gitk instance is started.

During the 2nd phase I see the first SKB allocation errors with a music 
skip between reading commits 95.000 and 110.000.
About commit 115.000 there is a very long pause during which the counter 
does not increase, music stops and the desktop freezes completely. The 
first 30 seconds of that freeze there is only very low disk activity (which 
seems strange); the next 25 seconds there suddenly is very high disk  
activity during which things gradually unfreeze and more SKB errors are 
displayed. After that the commit counter runs up fairly steadily again.

Phase 2) ends at ~1:45. Phase 3) (with more SKB errors) ends at ~2:05.

So this change almost doubles the time needed for phase 2) and causes SKB 
allocation errors to occur during that phase. Also, before this commit the 
desktop freezes are much shorter and less severe. With this change the 
desktop is completely unusable for almost a minute during phase 2), with 
even the mouse pointer frozen solid.
Note that phase 3) becomes shorter, but that the total time needed to load 
the 3rd instance increases by about 10-15 seconds.

Note: -rc2 and -rc3 had broken NFS, so I had to cherry-pick 3 NFS commits 
from -rc4 on top of the commits I wanted to test.

WITH congestion_wait CHANGE REVERTED
====================================
I've done quite a few tests of 2.6.31 with 373c0a7e and 8aa7e847 reverted 
to confirm that's really the culprit. I've done this for .31-rc3, .31-rc4,
.31-rc5, .31 and .31.1.

In all cases the huge freeze in phase 2) is gone and the general behavior 
and timings are again as it was after the wireless change. During most 
tests I did not get any SKB allocation errors during phase 2) or phase 3).

However with .31-rc5, .31 and .31.1 I have had some tests where I would see 
a few SKB allocation errors during phase 3) (which is somewhat likely), 
but also during phase 2). At this point I'm unsure whether this is just 
noise, or maybe a minor influence from some change merged after .31-rc4.
Looking through the commits there are several mm/page allocation changes.

For now I suggest ignoring this though as the impact (if any) is very minor 
and it is not reproducible reliably enough.


Next I'll retest Mel's patches and also test Reinette's patches.

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Pekka Enberg]

(Adding Jens to CC.)

On Wednesday 14 October 2009, Frans Pop wrote:
> > > There still has not been a mm-change identified that makes
> > > fragmentation significantly worse.

On Mon, 2009-10-19 at 01:33 +0200, Frans Pop wrote:
> > My bisection shows a very clear point, even if not an individual commit,
> > in the 'akpm' merge where SKB errors suddenly become *much* more
> > frequent and easy to trigger.
> > I'm sorry to say this, but the fact that nothing has been identified yet
> > is IMO the result of a lack of effort, not because there is no such
> > change.
> 
> I was wrong. It turns out that I was creating the variations in the test 
> results around the akpm merge myself by tiny changes in the way I ran the 
> tests. It took another round of about 30 compilations and tests purely in 
> this range to show that, but those same tests also made me aware of other 
> patterns I should look at.
> 
> Until a few days ago I was concentrating on "do I see SKB allocation errors 
> or not". Since then I've also been looking more consciously at when they 
> happen, at disk access patterns and at desktop freeze patterns.
> 
> I think I did mention before that this whole issue is rather subtle :-/
> So, my apologies for finguering the wrong area for so long, but it looked 
> solid given the info available at the time.
> 
> On Thursday 15 October 2009, Mel Gorman wrote:
> > Outside the range of commits suspected of causing problems was the
> > following. It's extremely low probability
> >
> > Commit 8aa7e84 Fix congestion_wait() sync/async vs read/write confusion
> >         This patch alters the call to congestion_wait() in the page
> >         allocator. Frankly, I don't get the change but it might worth
> >         checking if replacing BLK_RW_ASYNC with WRITE on top of 2.6.31
> >         makes any difference
> 
> This is the real culprit. Mel: thanks very much for looking beyond the 
> area I identified. Your overview of mm changes was exactly what I needed 
> and really helped a lot during my later tests.
> 
> This commit definitely causes most of the problems; confirmed by reverting 
> it on top of 2.6.31 (also requires reverting 373c0a7e, which is a later 
> build fix).

Mel/Jens, any ideas why commit 8aa7e84 makes us run out of high order
pages? Should we be using BLK_RW_SYNC in mm/page_alloc.c instead of
BLK_RW_ASYNC?

			Pekka

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index bf72055..fa8380a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1727,7 +1727,7 @@ __alloc_pages_high_priority(gfp_t gfp_mask, unsigned int order,
 			preferred_zone, migratetype);
 
 		if (!page && gfp_mask & __GFP_NOFAIL)
-			congestion_wait(BLK_RW_ASYNC, HZ/50);
+			congestion_wait(BLK_RW_SYNC, HZ/50);
 	} while (!page && (gfp_mask & __GFP_NOFAIL));
 
 	return page;
@@ -1898,7 +1898,7 @@ rebalance:
 	pages_reclaimed += did_some_progress;
 	if (should_alloc_retry(gfp_mask, order, pages_reclaimed)) {
 		/* Wait for some write requests to complete then retry */
-		congestion_wait(BLK_RW_ASYNC, HZ/50);
+		congestion_wait(BLK_RW_SYNC, HZ/50);
 		goto rebalance;
 	}
 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 19 October 2009, Pekka Enberg wrote:
> On Wednesday 14 October 2009, Frans Pop wrote:
> > On Thursday 15 October 2009, Mel Gorman wrote:
> > > Outside the range of commits suspected of causing problems was the
> > > following. It's extremely low probability
> > >
> > > Commit 8aa7e84 Fix congestion_wait() sync/async vs read/write
> > > confusion This patch alters the call to congestion_wait() in the
> > > page allocator. Frankly, I don't get the change but it might worth
> > > checking if replacing BLK_RW_ASYNC with WRITE on top of 2.6.31 makes
> > > any difference
> >
> > This is the real culprit. Mel: thanks very much for looking beyond the
> > area I identified. Your overview of mm changes was exactly what I
> > needed and really helped a lot during my later tests.
> >
> > This commit definitely causes most of the problems; confirmed by
> > reverting it on top of 2.6.31 (also requires reverting 373c0a7e, which
> > is a later build fix).
>
> Mel/Jens, any ideas why commit 8aa7e84 makes us run out of high order
> pages? Should we be using BLK_RW_SYNC in mm/page_alloc.c instead of
> BLK_RW_ASYNC?

I'm starting to think that this commit may not be directly related to high 
order allocation failures. The fact that I'm seeing SKB allocation 
failures earlier because of this commit could be just a side effect.
It could be that instead the main impact of this commit is on encrypted 
file system and/or encrypted swap (kcryptd).

Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm 
only reading from NFS that's unlikely).

Reason for thinking this is that reverting it makes no difference for Karol 
[1]. It will be interesting to see if it does make a difference for Sven 
Geggus [2].

/me wonders if we'll ever get to the bottom of this...

[1] http://lkml.org/lkml/2009/10/18/138
[2] http://lkml.org/lkml/2009/10/17/113

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobi Oetiker]

Today Frans Pop wrote:

>
> I'm starting to think that this commit may not be directly related to high
> order allocation failures. The fact that I'm seeing SKB allocation
> failures earlier because of this commit could be just a side effect.
> It could be that instead the main impact of this commit is on encrypted
> file system and/or encrypted swap (kcryptd).
>
> Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm
> only reading from NFS that's unlikely).

I have updated a fileserver to 2.6.31 today and I see page
allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
So I guess the problem must be quite generic:


Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]


Oct 19 08:59:16 johan kernel: [30120.685647] __ratelimit: 13 callbacks suppressed [kern.warning]
Oct 19 08:59:16 johan kernel: [30120.685654] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
Oct 19 08:59:16 johan kernel: [30120.685660] Pid: 6071, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
Oct 19 08:59:16 johan kernel: [30120.685663] Call Trace: [kern.warning]
Oct 19 08:59:16 johan kernel: [30120.685666]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]

Oct 19 09:36:31 johan kernel: [32355.708345] __ratelimit: 16 callbacks suppressed [kern.warning]
Oct 19 09:36:31 johan kernel: [32355.708352] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
Oct 19 09:36:31 johan kernel: [32355.708358] Pid: 6087, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
Oct 19 09:36:31 johan kernel: [32355.708361] Call Trace: [kern.warning]
Oct 19 09:36:31 johan kernel: [32355.708364]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]

Oct 19 10:52:01 johan kernel: [36885.358312] __ratelimit: 31 callbacks suppressed [kern.warning]
Oct 19 10:52:01 johan kernel: [36885.358319] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
Oct 19 10:52:01 johan kernel: [36885.358325] Pid: 6057, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
Oct 19 10:52:01 johan kernel: [36885.358327] Call Trace: [kern.warning]
Oct 19 10:52:01 johan kernel: [36885.358331]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]

Oct 19 11:12:01 johan kernel: [38085.163831] events/3: page allocation failure. order:5, mode:0x4020 [kern.warning]
Oct 19 11:12:01 johan kernel: [38085.163840] Pid: 18, comm: events/3 Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
Oct 19 11:12:01 johan kernel: [38085.163843] Call Trace: [kern.warning]
Oct 19 11:12:01 johan kernel: [38085.163846]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]



-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Pekka Enberg]

On Mon, 2009-10-19 at 11:49 +0200, Tobi Oetiker wrote:
> Today Frans Pop wrote:
> 
> >
> > I'm starting to think that this commit may not be directly related to high
> > order allocation failures. The fact that I'm seeing SKB allocation
> > failures earlier because of this commit could be just a side effect.
> > It could be that instead the main impact of this commit is on encrypted
> > file system and/or encrypted swap (kcryptd).
> >
> > Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm
> > only reading from NFS that's unlikely).
> 
> I have updated a fileserver to 2.6.31 today and I see page
> allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> So I guess the problem must be quite generic:

Yup, it almost certainly is. Does this patch help?

http://lkml.org/lkml/2009/10/16/89

Frans, did you ever get around retesting with just the above patch
applied?

			Pekka

> Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> 
> Oct 19 08:59:16 johan kernel: [30120.685647] __ratelimit: 13 callbacks suppressed [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685654] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685660] Pid: 6071, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685663] Call Trace: [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685666]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> Oct 19 09:36:31 johan kernel: [32355.708345] __ratelimit: 16 callbacks suppressed [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708352] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708358] Pid: 6087, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708361] Call Trace: [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708364]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> Oct 19 10:52:01 johan kernel: [36885.358312] __ratelimit: 31 callbacks suppressed [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358319] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358325] Pid: 6057, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358327] Call Trace: [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358331]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> Oct 19 11:12:01 johan kernel: [38085.163831] events/3: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 11:12:01 johan kernel: [38085.163840] Pid: 18, comm: events/3 Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 11:12:01 johan kernel: [38085.163843] Call Trace: [kern.warning]
> Oct 19 11:12:01 johan kernel: [38085.163846]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Karol Lewandowski]

On Mon, Oct 19, 2009 at 12:54:11PM +0300, Pekka Enberg wrote:
> On Mon, 2009-10-19 at 11:49 +0200, Tobi Oetiker wrote:
> > I have updated a fileserver to 2.6.31 today and I see page
> > allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> > So I guess the problem must be quite generic:
> 
> Yup, it almost certainly is. Does this patch help?
> 
> http://lkml.org/lkml/2009/10/16/89

This patch seems to help in some cases.  Before applying this patch I
was able to trigger alloc failures on different machine by booting
kernel with "mem=256MB" and doing:

  $ gitk on-full-tree &
  # rmmod e100
  ... wait for few MBs in swap
  # modprobe e100; ifup --force ethX

So here this patch helped -- with it, I was unable to trigger page
allocation failures (testing was short, tough).  However, as I said
here[1], I applied both of Mel's patches (including this one) and that
didn't help my orginal issue (failures after suspend).

[1] http://lkml.org/lkml/2009/10/17/109

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Mon, Oct 19, 2009 at 04:01:45PM +0200, Karol Lewandowski wrote:
> On Mon, Oct 19, 2009 at 12:54:11PM +0300, Pekka Enberg wrote:
> > On Mon, 2009-10-19 at 11:49 +0200, Tobi Oetiker wrote:
> > > I have updated a fileserver to 2.6.31 today and I see page
> > > allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> > > So I guess the problem must be quite generic:
> > 
> > Yup, it almost certainly is. Does this patch help?
> > 
> > http://lkml.org/lkml/2009/10/16/89
> 
> This patch seems to help in some cases.  Before applying this patch I
> was able to trigger alloc failures on different machine by booting
> kernel with "mem=256MB" and doing:
> 
>   $ gitk on-full-tree &
>   # rmmod e100
>   ... wait for few MBs in swap
>   # modprobe e100; ifup --force ethX
> 
> So here this patch helped -- with it, I was unable to trigger page
> allocation failures (testing was short, tough).  However, as I said
> here[1], I applied both of Mel's patches (including this one) and that
> didn't help my orginal issue (failures after suspend).
> 
> [1] http://lkml.org/lkml/2009/10/17/109
> 

Can you test with my kswapd patch applied and commits 373c0a7e,8aa7e847
reverted please?

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Karol Lewandowski]

On Mon, Oct 19, 2009 at 03:06:19PM +0100, Mel Gorman wrote:
> Can you test with my kswapd patch applied and commits 373c0a7e,8aa7e847
> reverted please?

It seems that your patch and Frans' reverts together *do* make
difference.

With these patches I haven't been able to trigger failures so far
(in about 6 attempts). I'll continue testing and let you know if
anything changes.

If nothing changes this looks like fix for my problem.

Thanks.  Thanks a lot!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Karol Lewandowski]

On Mon, Oct 19, 2009 at 07:09:47PM +0200, Karol Lewandowski wrote:
> On Mon, Oct 19, 2009 at 03:06:19PM +0100, Mel Gorman wrote:
> > Can you test with my kswapd patch applied and commits 373c0a7e,8aa7e847
> > reverted please?
> 
> It seems that your patch and Frans' reverts together *do* make
> difference.
> 
> With these patches I haven't been able to trigger failures so far
> (in about 6 attempts). I'll continue testing and let you know if
> anything changes.

Damn it.

I'm sorry to inform you that yes, I still get failures (less often,
but still).

Thanks.


e100: Intel(R) PRO/100 Network Driver, 3.5.24-k2-NAPI
e100: Copyright(c) 1999-2006 Intel Corporation
e100 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 9 (level, low) -> IRQ 9
e100 0000:00:03.0: PME# disabled
e100: eth0: e100_probe: addr 0xe8120000, irq 9, MAC addr 00:10:a4:89:e8:84
ifconfig: page allocation failure. order:5, mode:0x8020
Pid: 5151, comm: ifconfig Not tainted 2.6.31+frans2+mel-00002-g90702f9-dirty #2
Call Trace:
 [<c015c4e1>] ? __alloc_pages_nodemask+0x423/0x468
 [<c0104de7>] ? dma_generic_alloc_coherent+0x4a/0xab
 [<c0104d9d>] ? dma_generic_alloc_coherent+0x0/0xab
 [<d1614b6f>] ? e100_alloc_cbs+0xc7/0x174 [e100]
 [<d1615bfe>] ? e100_up+0x1b/0xf5 [e100]
 [<d1615cef>] ? e100_open+0x17/0x41 [e100]
 [<c02f871f>] ? dev_open+0x8f/0xc5
 [<c02f7ed9>] ? dev_change_flags+0xa2/0x155
 [<c032daa6>] ? devinet_ioctl+0x22a/0x51c
 [<c02ebabe>] ? sock_ioctl+0x0/0x1e4
 [<c02ebc7e>] ? sock_ioctl+0x1c0/0x1e4
 [<c02ebabe>] ? sock_ioctl+0x0/0x1e4
 [<c017f23a>] ? vfs_ioctl+0x16/0x4a
 [<c017fb01>] ? do_vfs_ioctl+0x48a/0x4c1
 [<c0168137>] ? handle_mm_fault+0x1e0/0x42c
 [<c0348c6b>] ? do_page_fault+0x2ce/0x2e4
 [<c017fb64>] ? sys_ioctl+0x2c/0x42
 [<c0102748>] ? sysenter_do_call+0x12/0x26
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:   90, btch:  15 usd:  35
Active_anon:14778 active_file:10836 inactive_anon:22033
 inactive_file:11854 unevictable:0 dirty:6 writeback:0 unstable:0
 free:1031 slab:2083 mapped:6193 pagetables:417 bounce:0
DMA free:1096kB min:124kB low:152kB high:184kB active_anon:528kB inactive_anon:3440kB active_file:1076kB inactive_file:5580kB unevictable:0kB present:15868kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 238 238
Normal free:3028kB min:1908kB low:2384kB high:2860kB active_anon:58584kB inactive_anon:84692kB active_file:42268kB inactive_file:41836kB unevictable:0kB present:243776kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 46*4kB 0*8kB 5*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1096kB
Normal: 135*4kB 213*8kB 21*16kB 4*32kB 5*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3028kB
25927 total pagecache pages
3010 pages in swap cache
Swap cache stats: add 205613, delete 202603, find 63665/79800
Free swap  = 485236kB
Total swap = 514040kB
65520 pages RAM
1663 pages reserved
14633 pages shared
52919 pages non-shared
ifconfig: page allocation failure. order:5, mode:0x8020
Pid: 5151, comm: ifconfig Not tainted 2.6.31+frans2+mel-00002-g90702f9-dirty #2
Call Trace:
 [<c015c4e1>] ? __alloc_pages_nodemask+0x423/0x468
 [<c0104de7>] ? dma_generic_alloc_coherent+0x4a/0xab
 [<c0104d9d>] ? dma_generic_alloc_coherent+0x0/0xab
 [<d1614b6f>] ? e100_alloc_cbs+0xc7/0x174 [e100]
 [<d1615bfe>] ? e100_up+0x1b/0xf5 [e100]
 [<d1615cef>] ? e100_open+0x17/0x41 [e100]
 [<c02f871f>] ? dev_open+0x8f/0xc5
 [<c02f7ed9>] ? dev_change_flags+0xa2/0x155
 [<c032daa6>] ? devinet_ioctl+0x22a/0x51c
 [<c02ebabe>] ? sock_ioctl+0x0/0x1e4
 [<c02ebc7e>] ? sock_ioctl+0x1c0/0x1e4
 [<c02ebabe>] ? sock_ioctl+0x0/0x1e4
 [<c017f23a>] ? vfs_ioctl+0x16/0x4a
 [<c017fb01>] ? do_vfs_ioctl+0x48a/0x4c1
 [<c0175fd1>] ? vfs_write+0xf4/0x105
 [<c017fb64>] ? sys_ioctl+0x2c/0x42
 [<c0102748>] ? sysenter_do_call+0x12/0x26
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:   90, btch:  15 usd:  67
Active_anon:14760 active_file:10798 inactive_anon:22052
 inactive_file:11862 unevictable:0 dirty:6 writeback:30 unstable:0
 free:1031 slab:2083 mapped:6187 pagetables:417 bounce:0
DMA free:1096kB min:124kB low:152kB high:184kB active_anon:528kB inactive_anon:3440kB active_file:1076kB inactive_file:5580kB unevictable:0kB present:15868kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 238 238
Normal free:3028kB min:1908kB low:2384kB high:2860kB active_anon:58512kB inactive_anon:84768kB active_file:42116kB inactive_file:41868kB unevictable:0kB present:243776kB pages_scanned:100 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 46*4kB 0*8kB 5*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1096kB
Normal: 135*4kB 213*8kB 21*16kB 4*32kB 5*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3028kB
25924 total pagecache pages
3037 pages in swap cache
Swap cache stats: add 205644, delete 202607, find 63666/79802
Free swap  = 485116kB
Total swap = 514040kB
65520 pages RAM
1663 pages reserved
14638 pages shared
52896 pages non-shared
e100 0000:00:03.0: firmware: requesting e100/d101s_ucode.bin
ADDRCONF(NETDEV_UP): eth0: link is not ready
e100: eth0 NIC Link is Up 100 Mbps Full Duplex
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
eth0: no IPv6 routers present

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Mon, Oct 19, 2009 at 11:49:08AM +0200, Tobi Oetiker wrote:
> Today Frans Pop wrote:
> 
> >
> > I'm starting to think that this commit may not be directly related to high
> > order allocation failures. The fact that I'm seeing SKB allocation
> > failures earlier because of this commit could be just a side effect.
> > It could be that instead the main impact of this commit is on encrypted
> > file system and/or encrypted swap (kcryptd).
> >
> > Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm
> > only reading from NFS that's unlikely).
> 
> I have updated a fileserver to 2.6.31 today and I see page
> allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> So I guess the problem must be quite generic:
> 
> 
> Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 

What's the rest of the stack trace? I'm wondering where a large number
of order-5 GFP_ATOMIC allocations are coming from. It seems different to
the e100 problem where there is one GFP_ATOMIC allocation while the
firmware is being loaded.

Thanks

> 
> Oct 19 08:59:16 johan kernel: [30120.685647] __ratelimit: 13 callbacks suppressed [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685654] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685660] Pid: 6071, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685663] Call Trace: [kern.warning]
> Oct 19 08:59:16 johan kernel: [30120.685666]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> Oct 19 09:36:31 johan kernel: [32355.708345] __ratelimit: 16 callbacks suppressed [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708352] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708358] Pid: 6087, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708361] Call Trace: [kern.warning]
> Oct 19 09:36:31 johan kernel: [32355.708364]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> Oct 19 10:52:01 johan kernel: [36885.358312] __ratelimit: 31 callbacks suppressed [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358319] nfsd: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358325] Pid: 6057, comm: nfsd Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358327] Call Trace: [kern.warning]
> Oct 19 10:52:01 johan kernel: [36885.358331]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> Oct 19 11:12:01 johan kernel: [38085.163831] events/3: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 11:12:01 johan kernel: [38085.163840] Pid: 18, comm: events/3 Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 11:12:01 johan kernel: [38085.163843] Call Trace: [kern.warning]
> Oct 19 11:12:01 johan kernel: [38085.163846]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> 
> 
> 
> -- 
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Mel Gorman wrote:

> On Mon, Oct 19, 2009 at 11:49:08AM +0200, Tobi Oetiker wrote:
> > Today Frans Pop wrote:
> >
> > >
> > > I'm starting to think that this commit may not be directly related to high
> > > order allocation failures. The fact that I'm seeing SKB allocation
> > > failures earlier because of this commit could be just a side effect.
> > > It could be that instead the main impact of this commit is on encrypted
> > > file system and/or encrypted swap (kcryptd).
> > >
> > > Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm
> > > only reading from NFS that's unlikely).
> >
> > I have updated a fileserver to 2.6.31 today and I see page
> > allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> > So I guess the problem must be quite generic:
> >
> >
> > Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> >
>
> What's the rest of the stack trace? I'm wondering where a large number
> of order-5 GFP_ATOMIC allocations are coming from. It seems different to
> the e100 problem where there is one GFP_ATOMIC allocation while the
> firmware is being loaded.

Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684157]  [<ffffffff810da7e5>] __alloc_pages_nodemask+0x135/0x140 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684164]  [<ffffffff815065b4>] ? _spin_unlock_bh+0x14/0x20 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684170]  [<ffffffff8110b368>] kmalloc_large_node+0x68/0xc0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684175]  [<ffffffff8110f15a>] __kmalloc_node_track_caller+0x11a/0x180 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684181]  [<ffffffff8140ffd2>] ? skb_copy+0x32/0xa0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684185]  [<ffffffff8140d8b6>] __alloc_skb+0x76/0x180 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684205]  [<ffffffff8140ffd2>] skb_copy+0x32/0xa0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684221]  [<ffffffffa050f33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684227]  [<ffffffff81416a6d>] dev_queue_xmit_nit+0x10d/0x170 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684231]  [<ffffffff81416f79>] dev_hard_start_xmit+0x189/0x1c0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684236]  [<ffffffff8142f071>] __qdisc_run+0x1a1/0x230 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684240]  [<ffffffff81418a88>] dev_queue_xmit+0x238/0x310 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684246]  [<ffffffff8144864b>] ip_finish_output+0x11b/0x2f0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684250]  [<ffffffff814488a9>] ip_output+0x89/0xd0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684254]  [<ffffffff814478c0>] ip_local_out+0x20/0x30 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684258]  [<ffffffff814481ab>] ip_queue_xmit+0x22b/0x3f0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684264]  [<ffffffff8145d5e5>] tcp_transmit_skb+0x345/0x4e0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684269]  [<ffffffff8145eaf6>] tcp_write_xmit+0xb6/0x2e0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684273]  [<ffffffff8145ed8b>] __tcp_push_pending_frames+0x2b/0xa0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684277]  [<ffffffff8145b249>] tcp_rcv_established+0x459/0x6d0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684282]  [<ffffffff814630bd>] tcp_v4_do_rcv+0x12d/0x140 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684285]  [<ffffffff8146365e>] tcp_v4_rcv+0x58e/0x7c0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684289]  [<ffffffff8144276d>] ip_local_deliver_finish+0x11d/0x2b0 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684293]  [<ffffffff8144293b>] ip_local_deliver+0x3b/0x90 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684297]  [<ffffffff81442ad6>] ip_rcv_finish+0x146/0x420 [kern.warning]
Oct 19 07:10:02 johan kernel: [23565.684301]  [<ffffffff8144304b>] ip_rcv+0x29b/0x370 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684304]  [<ffffffff81418f9a>] netif_receive_skb+0x38a/0x4d0 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684308]  [<ffffffff81419268>] napi_skb_finish+0x48/0x60 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684311]  [<ffffffff81419724>] napi_gro_receive+0x34/0x40 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684330]  [<ffffffffa006b623>] tg3_rx+0x373/0x4b0 [tg3] [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684339]  [<ffffffffa006cbf0>] tg3_poll_work+0x70/0xf0 [tg3] [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684347]  [<ffffffffa006ccae>] tg3_poll+0x3e/0xe0 [tg3] [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684350]  [<ffffffff814198d2>] net_rx_action+0x102/0x210 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684357]  [<ffffffff81061d24>] __do_softirq+0xc4/0x1f0 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684362]  [<ffffffff8101314c>] call_softirq+0x1c/0x30 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684365]  [<ffffffff81014945>] do_softirq+0x55/0x90 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684369]  [<ffffffff8106116b>] irq_exit+0x7b/0x90 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684372]  [<ffffffff81013e93>] do_IRQ+0x73/0xe0 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684378]  [<ffffffff81012993>] ret_from_intr+0x0/0x11 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684381]  <EOI>  [<ffffffff810318b6>] ? native_safe_halt+0x6/0x10 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684391]  [<ffffffff81019cd8>] ? default_idle+0x48/0xe0 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684396]  [<ffffffff8150929d>] ? __atomic_notifier_call_chain+0xd/0x10 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684400]  [<ffffffff815092b1>] ? atomic_notifier_call_chain+0x11/0x20 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684404]  [<ffffffff810107c8>] ? cpu_idle+0x98/0xe0 [kern.warning]
Oct 19 07:10:04 johan kernel: [23565.684410]  [<ffffffff81500d95>] ? start_secondary+0x95/0xc0 [kern.warning]

if you need more, I can send you a whole bunch of them ...

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Mon, Oct 19, 2009 at 03:40:05PM +0200, Tobias Oetiker wrote:
> Hi Mel,
> 
> Today Mel Gorman wrote:
> 
> > On Mon, Oct 19, 2009 at 11:49:08AM +0200, Tobi Oetiker wrote:
> > > Today Frans Pop wrote:
> > >
> > > >
> > > > I'm starting to think that this commit may not be directly related to high
> > > > order allocation failures. The fact that I'm seeing SKB allocation
> > > > failures earlier because of this commit could be just a side effect.
> > > > It could be that instead the main impact of this commit is on encrypted
> > > > file system and/or encrypted swap (kcryptd).
> > > >
> > > > Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm
> > > > only reading from NFS that's unlikely).
> > >
> > > I have updated a fileserver to 2.6.31 today and I see page
> > > allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> > > So I guess the problem must be quite generic:
> > >
> > >
> > > Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> > >
> >
> > What's the rest of the stack trace? I'm wondering where a large number
> > of order-5 GFP_ATOMIC allocations are coming from. It seems different to
> > the e100 problem where there is one GFP_ATOMIC allocation while the
> > firmware is being loaded.
> 
> Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684157]  [<ffffffff810da7e5>] __alloc_pages_nodemask+0x135/0x140 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684164]  [<ffffffff815065b4>] ? _spin_unlock_bh+0x14/0x20 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684170]  [<ffffffff8110b368>] kmalloc_large_node+0x68/0xc0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684175]  [<ffffffff8110f15a>] __kmalloc_node_track_caller+0x11a/0x180 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684181]  [<ffffffff8140ffd2>] ? skb_copy+0x32/0xa0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684185]  [<ffffffff8140d8b6>] __alloc_skb+0x76/0x180 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684205]  [<ffffffff8140ffd2>] skb_copy+0x32/0xa0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684221]  [<ffffffffa050f33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]

Is the MTU set very high between the host and virtualised machine?

Can you test please with the patch at http://lkml.org/lkml/2009/10/16/89
applied and with commits 373c0a7e and 8aa7e847 reverted please?

> Oct 19 07:10:02 johan kernel: [23565.684231]  [<ffffffff81416f79>] dev_hard_start_xmit+0x189/0x1c0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684236]  [<ffffffff8142f071>] __qdisc_run+0x1a1/0x230 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684240]  [<ffffffff81418a88>] dev_queue_xmit+0x238/0x310 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684246]  [<ffffffff8144864b>] ip_finish_output+0x11b/0x2f0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684250]  [<ffffffff814488a9>] ip_output+0x89/0xd0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684254]  [<ffffffff814478c0>] ip_local_out+0x20/0x30 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684258]  [<ffffffff814481ab>] ip_queue_xmit+0x22b/0x3f0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684264]  [<ffffffff8145d5e5>] tcp_transmit_skb+0x345/0x4e0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684269]  [<ffffffff8145eaf6>] tcp_write_xmit+0xb6/0x2e0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684273]  [<ffffffff8145ed8b>] __tcp_push_pending_frames+0x2b/0xa0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684277]  [<ffffffff8145b249>] tcp_rcv_established+0x459/0x6d0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684282]  [<ffffffff814630bd>] tcp_v4_do_rcv+0x12d/0x140 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684285]  [<ffffffff8146365e>] tcp_v4_rcv+0x58e/0x7c0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684289]  [<ffffffff8144276d>] ip_local_deliver_finish+0x11d/0x2b0 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684293]  [<ffffffff8144293b>] ip_local_deliver+0x3b/0x90 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684297]  [<ffffffff81442ad6>] ip_rcv_finish+0x146/0x420 [kern.warning]
> Oct 19 07:10:02 johan kernel: [23565.684301]  [<ffffffff8144304b>] ip_rcv+0x29b/0x370 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684304]  [<ffffffff81418f9a>] netif_receive_skb+0x38a/0x4d0 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684308]  [<ffffffff81419268>] napi_skb_finish+0x48/0x60 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684311]  [<ffffffff81419724>] napi_gro_receive+0x34/0x40 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684330]  [<ffffffffa006b623>] tg3_rx+0x373/0x4b0 [tg3] [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684339]  [<ffffffffa006cbf0>] tg3_poll_work+0x70/0xf0 [tg3] [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684347]  [<ffffffffa006ccae>] tg3_poll+0x3e/0xe0 [tg3] [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684350]  [<ffffffff814198d2>] net_rx_action+0x102/0x210 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684357]  [<ffffffff81061d24>] __do_softirq+0xc4/0x1f0 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684362]  [<ffffffff8101314c>] call_softirq+0x1c/0x30 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684365]  [<ffffffff81014945>] do_softirq+0x55/0x90 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684369]  [<ffffffff8106116b>] irq_exit+0x7b/0x90 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684372]  [<ffffffff81013e93>] do_IRQ+0x73/0xe0 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684378]  [<ffffffff81012993>] ret_from_intr+0x0/0x11 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684381]  <EOI>  [<ffffffff810318b6>] ? native_safe_halt+0x6/0x10 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684391]  [<ffffffff81019cd8>] ? default_idle+0x48/0xe0 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684396]  [<ffffffff8150929d>] ? __atomic_notifier_call_chain+0xd/0x10 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684400]  [<ffffffff815092b1>] ? atomic_notifier_call_chain+0x11/0x20 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684404]  [<ffffffff810107c8>] ? cpu_idle+0x98/0xe0 [kern.warning]
> Oct 19 07:10:04 johan kernel: [23565.684410]  [<ffffffff81500d95>] ? start_secondary+0x95/0xc0 [kern.warning]
> 
> if you need more, I can send you a whole bunch of them ...
> 

I'm assuming they are all more or less the same.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Mel Gorman wrote:

> On Mon, Oct 19, 2009 at 03:40:05PM +0200, Tobias Oetiker wrote:
> > Hi Mel,
> >
> > Today Mel Gorman wrote:
> >
> > > On Mon, Oct 19, 2009 at 11:49:08AM +0200, Tobi Oetiker wrote:
> > > > Today Frans Pop wrote:
> > > >
> > > > >
> > > > > I'm starting to think that this commit may not be directly related to high
> > > > > order allocation failures. The fact that I'm seeing SKB allocation
> > > > > failures earlier because of this commit could be just a side effect.
> > > > > It could be that instead the main impact of this commit is on encrypted
> > > > > file system and/or encrypted swap (kcryptd).
> > > > >
> > > > > Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm
> > > > > only reading from NFS that's unlikely).
> > > >
> > > > I have updated a fileserver to 2.6.31 today and I see page
> > > > allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> > > > So I guess the problem must be quite generic:
> > > >
> > > >
> > > > Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> > > > Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> > > > Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> > > > Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> > > >
> > >
> > > What's the rest of the stack trace? I'm wondering where a large number
> > > of order-5 GFP_ATOMIC allocations are coming from. It seems different to
> > > the e100 problem where there is one GFP_ATOMIC allocation while the
> > > firmware is being loaded.
> >
> > Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684157]  [<ffffffff810da7e5>] __alloc_pages_nodemask+0x135/0x140 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684164]  [<ffffffff815065b4>] ? _spin_unlock_bh+0x14/0x20 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684170]  [<ffffffff8110b368>] kmalloc_large_node+0x68/0xc0 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684175]  [<ffffffff8110f15a>] __kmalloc_node_track_caller+0x11a/0x180 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684181]  [<ffffffff8140ffd2>] ? skb_copy+0x32/0xa0 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684185]  [<ffffffff8140d8b6>] __alloc_skb+0x76/0x180 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684205]  [<ffffffff8140ffd2>] skb_copy+0x32/0xa0 [kern.warning]
> > Oct 19 07:10:02 johan kernel: [23565.684221]  [<ffffffffa050f33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
>
> Is the MTU set very high between the host and virtualised machine?
>
> Can you test please with the patch at http://lkml.org/lkml/2009/10/16/89
> applied and with commits 373c0a7e and 8aa7e847 reverted please?

if you can send me a consolidated patch which does apply to
2.6.31.4 I will be glad to try ...

your patch in http://lkml.org/lkml/2009/10/16/89 seems not to be
for 2.6.31 ... I assume it would be but then again I I don't realy
understand the code so this is just pattern matching ...


--- a/mm/page_alloc.c   2009-10-05 19:12:06.000000000 +0200
+++ b/mm/page_alloc.c   2009-10-19 14:52:15.000000000 +0200
@@ -1763,6 +1763,7 @@
        if (NUMA_BUILD && (gfp_mask & GFP_THISNODE) == GFP_THISNODE)
                goto nopage;

+restart:
        wake_all_kswapd(order, zonelist, high_zoneidx);

        /*
@@ -1772,7 +1773,6 @@
         */
        alloc_flags = gfp_to_alloc_flags(gfp_mask);

-restart:
        /* This is the last chance, in general, before the goto nopage. */
        page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
                        high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,



cheers
tobi
-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Mon, Oct 19, 2009 at 04:16:36PM +0200, Tobias Oetiker wrote:
> Hi Mel,
> 
> Today Mel Gorman wrote:
> 
> > On Mon, Oct 19, 2009 at 03:40:05PM +0200, Tobias Oetiker wrote:
> > > Hi Mel,
> > >
> > > Today Mel Gorman wrote:
> > >
> > > > On Mon, Oct 19, 2009 at 11:49:08AM +0200, Tobi Oetiker wrote:
> > > > > Today Frans Pop wrote:
> > > > >
> > > > > >
> > > > > > I'm starting to think that this commit may not be directly related to high
> > > > > > order allocation failures. The fact that I'm seeing SKB allocation
> > > > > > failures earlier because of this commit could be just a side effect.
> > > > > > It could be that instead the main impact of this commit is on encrypted
> > > > > > file system and/or encrypted swap (kcryptd).
> > > > > >
> > > > > > Besides mm the commit also touches dm-crypt (and nfs/write.c, but as I'm
> > > > > > only reading from NFS that's unlikely).
> > > > >
> > > > > I have updated a fileserver to 2.6.31 today and I see page
> > > > > allocation failures from several parts of the system ... mostly nfs though ... (it is a nfs server).
> > > > > So I guess the problem must be quite generic:
> > > > >
> > > > >
> > > > > Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> > > > > Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> > > > > Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> > > > > Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> > > > >
> > > >
> > > > What's the rest of the stack trace? I'm wondering where a large number
> > > > of order-5 GFP_ATOMIC allocations are coming from. It seems different to
> > > > the e100 problem where there is one GFP_ATOMIC allocation while the
> > > > firmware is being loaded.
> > >
> > > Oct 19 07:10:02 johan kernel: [23565.684110] swapper: page allocation failure. order:5, mode:0x4020 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684118] Pid: 0, comm: swapper Not tainted 2.6.31-02063104-generic #02063104 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684121] Call Trace: [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684124]  <IRQ>  [<ffffffff810da5a2>] __alloc_pages_slowpath+0x3b2/0x4c0 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684157]  [<ffffffff810da7e5>] __alloc_pages_nodemask+0x135/0x140 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684164]  [<ffffffff815065b4>] ? _spin_unlock_bh+0x14/0x20 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684170]  [<ffffffff8110b368>] kmalloc_large_node+0x68/0xc0 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684175]  [<ffffffff8110f15a>] __kmalloc_node_track_caller+0x11a/0x180 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684181]  [<ffffffff8140ffd2>] ? skb_copy+0x32/0xa0 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684185]  [<ffffffff8140d8b6>] __alloc_skb+0x76/0x180 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684205]  [<ffffffff8140ffd2>] skb_copy+0x32/0xa0 [kern.warning]
> > > Oct 19 07:10:02 johan kernel: [23565.684221]  [<ffffffffa050f33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
> >
> > Is the MTU set very high between the host and virtualised machine?
> >
> > Can you test please with the patch at http://lkml.org/lkml/2009/10/16/89
> > applied and with commits 373c0a7e and 8aa7e847 reverted please?
> 
> if you can send me a consolidated patch which does apply to
> 2.6.31.4 I will be glad to try ...
> 

Sure

==== CUT HERE ====

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Mel Gorman wrote:

> >
> > if you can send me a consolidated patch which does apply to
> > 2.6.31.4 I will be glad to try ...
> >
>
> Sure
>
> ==== CUT HERE ====
>
> From 6c0215af3b7c39ef7b8083ea38ca3ad93cd3f51f Mon Sep 17 00:00:00 2001
> From: Mel Gorman <mel@csn.ul.ie>
> Date: Mon, 19 Oct 2009 15:40:43 +0100
> Subject: [PATCH] Kick off kswapd after direct reclaim and revert congestion changes
>
> The following patch is http://lkml.org/lkml/2009/10/16/89 on top of
> 2.6.31.4 as well as patches 373c0a7e and 8aa7e847 reverted.

it seems to help ... the server has been running for 3 hours now
without incident, but then again it is not as active as during the
day, ... will report tomorrow.

cheers
tobi


-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Tobias Oetiker wrote:

> Hi Mel,
>
> Today Mel Gorman wrote:
>
> > >
> > > if you can send me a consolidated patch which does apply to
> > > 2.6.31.4 I will be glad to try ...
> > >
> >
> > Sure
> >
> > ==== CUT HERE ====
> >
> > From 6c0215af3b7c39ef7b8083ea38ca3ad93cd3f51f Mon Sep 17 00:00:00 2001
> > From: Mel Gorman <mel@csn.ul.ie>
> > Date: Mon, 19 Oct 2009 15:40:43 +0100
> > Subject: [PATCH] Kick off kswapd after direct reclaim and revert congestion changes
> >
> > The following patch is http://lkml.org/lkml/2009/10/16/89 on top of
> > 2.6.31.4 as well as patches 373c0a7e and 8aa7e847 reverted.
>
> it seems to help ... the server has been running for 3 hours now
> without incident, but then again it is not as active as during the
> day, ... will report tomorrow.

while I was writing, the system found that the patch does not realy
help:

Oct 19 22:09:52 johan kernel: [11157.121506] smtpd: page allocation failure. order:5, mode:0x4020 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121514] Pid: 19324, comm: smtpd Tainted: G      D    2.6.31.4-oep #1 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121518] Call Trace: [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121521]  <IRQ>  [<ffffffff810cb599>] __alloc_pages_nodemask+0x549/0x650 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121563]  [<ffffffffa02bde3b>] ? __nf_ct_refresh_acct+0xab/0x110 [nf_conntrack] [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121572]  [<ffffffffa02a8337>] ? ipt_do_table+0x2f7/0x610 [ip_tables] [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121580]  [<ffffffff810fac18>] kmalloc_large_node+0x68/0xc0 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121585]  [<ffffffff810fe90a>] __kmalloc_node_track_caller+0x11a/0x180 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121592]  [<ffffffff813ebd42>] ? skb_copy+0x32/0xa0 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121596]  [<ffffffff813e9606>] __alloc_skb+0x76/0x180 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121600]  [<ffffffff813ebd42>] skb_copy+0x32/0xa0 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121615]  [<ffffffffa07dd33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121620]  [<ffffffff813f2512>] dev_hard_start_xmit+0x142/0x320 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121632]  [<ffffffff8140a2c1>] __qdisc_run+0x1a1/0x230 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121637]  [<ffffffff813f41e0>] dev_queue_xmit+0x2b0/0x3a0 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121642]  [<ffffffff8142349b>] ip_finish_output+0x11b/0x2f0 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121646]  [<ffffffff814236f9>] ip_output+0x89/0xd0 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121650]  [<ffffffff81422710>] ip_local_out+0x20/0x30 [kern.warning]
Oct 19 22:09:52 johan kernel: [11157.121654]  [<ffffffff81422ffb>] ip_queue_xmit+0x22b/0x3f0 [kern.warning]

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Mon, Oct 19, 2009 at 10:17:06PM +0200, Tobias Oetiker wrote:
> Hi Mel,
> 
> Today Tobias Oetiker wrote:
> 
> > Hi Mel,
> >
> > Today Mel Gorman wrote:
> >
> > > >
> > > > if you can send me a consolidated patch which does apply to
> > > > 2.6.31.4 I will be glad to try ...
> > > >
> > >
> > > Sure
> > >
> > > ==== CUT HERE ====
> > >
> > > From 6c0215af3b7c39ef7b8083ea38ca3ad93cd3f51f Mon Sep 17 00:00:00 2001
> > > From: Mel Gorman <mel@csn.ul.ie>
> > > Date: Mon, 19 Oct 2009 15:40:43 +0100
> > > Subject: [PATCH] Kick off kswapd after direct reclaim and revert congestion changes
> > >
> > > The following patch is http://lkml.org/lkml/2009/10/16/89 on top of
> > > 2.6.31.4 as well as patches 373c0a7e and 8aa7e847 reverted.
> >
> > it seems to help ... the server has been running for 3 hours now
> > without incident, but then again it is not as active as during the
> > day, ... will report tomorrow.
> 
> while I was writing, the system found that the patch does not realy
> help:
> 
> Oct 19 22:09:52 johan kernel: [11157.121506] smtpd: page allocation failure. order:5, mode:0x4020 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121514] Pid: 19324, comm: smtpd Tainted: G      D    2.6.31.4-oep #1 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121518] Call Trace: [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121521]  <IRQ>  [<ffffffff810cb599>] __alloc_pages_nodemask+0x549/0x650 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121563]  [<ffffffffa02bde3b>] ? __nf_ct_refresh_acct+0xab/0x110 [nf_conntrack] [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121572]  [<ffffffffa02a8337>] ? ipt_do_table+0x2f7/0x610 [ip_tables] [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121580]  [<ffffffff810fac18>] kmalloc_large_node+0x68/0xc0 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121585]  [<ffffffff810fe90a>] __kmalloc_node_track_caller+0x11a/0x180 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121592]  [<ffffffff813ebd42>] ? skb_copy+0x32/0xa0 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121596]  [<ffffffff813e9606>] __alloc_skb+0x76/0x180 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121600]  [<ffffffff813ebd42>] skb_copy+0x32/0xa0 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121615]  [<ffffffffa07dd33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121620]  [<ffffffff813f2512>] dev_hard_start_xmit+0x142/0x320 [kern.warning]

Are the number of failures at least reduced or are they occuring at the
same rate? Also, what was the last kernel that worked for you with this
configuration?

Thanks

> Oct 19 22:09:52 johan kernel: [11157.121632]  [<ffffffff8140a2c1>] __qdisc_run+0x1a1/0x230 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121637]  [<ffffffff813f41e0>] dev_queue_xmit+0x2b0/0x3a0 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121642]  [<ffffffff8142349b>] ip_finish_output+0x11b/0x2f0 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121646]  [<ffffffff814236f9>] ip_output+0x89/0xd0 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121650]  [<ffffffff81422710>] ip_local_out+0x20/0x30 [kern.warning]
> Oct 19 22:09:52 johan kernel: [11157.121654]  [<ffffffff81422ffb>] ip_queue_xmit+0x22b/0x3f0 [kern.warning]
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Mel Gorman wrote:

> On Mon, Oct 19, 2009 at 10:17:06PM +0200, Tobias Oetiker wrote:

> > Oct 19 22:09:52 johan kernel: [11157.121600]  [<ffffffff813ebd42>] skb_copy+0x32/0xa0 [kern.warning]
> > Oct 19 22:09:52 johan kernel: [11157.121615]  [<ffffffffa07dd33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
> > Oct 19 22:09:52 johan kernel: [11157.121620]  [<ffffffff813f2512>] dev_hard_start_xmit+0x142/0x320 [kern.warning]
>
> Are the number of failures at least reduced or are they occuring at the
> same rate?

not that it would have any statistical significance, but I had 5
failure (clusters) yesterday morning and 5 this morning ...

the failures often show up in groups I saved one on
http://tobi.oetiker.ch/cluster-2009-10-20-08-31.txt

> Also, what was the last kernel that worked for you with this
> configuration?

that would be 2.6.24 ... I have not upgraded in quite some time.
But since the io performance of 2.6.31 is about double in my tests
I thought it would be a good thing todo ...

cheers
tobi

> Thanks
>
> > Oct 19 22:09:52 johan kernel: [11157.121632]  [<ffffffff8140a2c1>] __qdisc_run+0x1a1/0x230 [kern.warning]
> > Oct 19 22:09:52 johan kernel: [11157.121637]  [<ffffffff813f41e0>] dev_queue_xmit+0x2b0/0x3a0 [kern.warning]
> > Oct 19 22:09:52 johan kernel: [11157.121642]  [<ffffffff8142349b>] ip_finish_output+0x11b/0x2f0 [kern.warning]
> > Oct 19 22:09:52 johan kernel: [11157.121646]  [<ffffffff814236f9>] ip_output+0x89/0xd0 [kern.warning]
> > Oct 19 22:09:52 johan kernel: [11157.121650]  [<ffffffff81422710>] ip_local_out+0x20/0x30 [kern.warning]
> > Oct 19 22:09:52 johan kernel: [11157.121654]  [<ffffffff81422ffb>] ip_queue_xmit+0x22b/0x3f0 [kern.warning]
> >
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Tue, Oct 20, 2009 at 01:44:50PM +0200, Tobias Oetiker wrote:
> Hi Mel,
> 
> Today Mel Gorman wrote:
> 
> > On Mon, Oct 19, 2009 at 10:17:06PM +0200, Tobias Oetiker wrote:
> 
> > > Oct 19 22:09:52 johan kernel: [11157.121600]  [<ffffffff813ebd42>] skb_copy+0x32/0xa0 [kern.warning]
> > > Oct 19 22:09:52 johan kernel: [11157.121615]  [<ffffffffa07dd33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
> > > Oct 19 22:09:52 johan kernel: [11157.121620]  [<ffffffff813f2512>] dev_hard_start_xmit+0x142/0x320 [kern.warning]
> >
> > Are the number of failures at least reduced or are they occuring at the
> > same rate?
> 
> not that it would have any statistical significance, but I had 5
> failure (clusters) yesterday morning and 5 this morning ...
> 

Before the patches were applied, how many failures were you seeing in
the morning?

> the failures often show up in groups I saved one on
> http://tobi.oetiker.ch/cluster-2009-10-20-08-31.txt
> 
> > Also, what was the last kernel that worked for you with this
> > configuration?
> 
> that would be 2.6.24 ... I have not upgraded in quite some time.
> But since the io performance of 2.6.31 is about double in my tests
> I thought it would be a good thing todo ...
> 

That significant a different in performance may explain differences in timing
as well. i.e. the allocator is being put under more pressure now than it
was previously as more processes make forward progress.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Mel Gorman wrote:

> On Tue, Oct 20, 2009 at 01:44:50PM +0200, Tobias Oetiker wrote:
> > Hi Mel,
> >
> > Today Mel Gorman wrote:
> >
> > > On Mon, Oct 19, 2009 at 10:17:06PM +0200, Tobias Oetiker wrote:
> >
> > > > Oct 19 22:09:52 johan kernel: [11157.121600]  [<ffffffff813ebd42>] skb_copy+0x32/0xa0 [kern.warning]
> > > > Oct 19 22:09:52 johan kernel: [11157.121615]  [<ffffffffa07dd33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
> > > > Oct 19 22:09:52 johan kernel: [11157.121620]  [<ffffffff813f2512>] dev_hard_start_xmit+0x142/0x320 [kern.warning]
> > >
> > > Are the number of failures at least reduced or are they occuring at the
> > > same rate?
> >
> > not that it would have any statistical significance, but I had 5
> > failure (clusters) yesterday morning and 5 this morning ...
> >
>
> Before the patches were applied, how many failures were you seeing in
> the morning?

5 as well ... before an after ...

> > the failures often show up in groups I saved one on
> > http://tobi.oetiker.ch/cluster-2009-10-20-08-31.txt
> >
> > > Also, what was the last kernel that worked for you with this
> > > configuration?
> >
> > that would be 2.6.24 ... I have not upgraded in quite some time.
> > But since the io performance of 2.6.31 is about double in my tests
> > I thought it would be a good thing todo ...
> >
>
> That significant a different in performance may explain differences in timing
> as well. i.e. the allocator is being put under more pressure now than it
> was previously as more processes make forward progress.

you are saing that the problem might be even older ?

we do have 8GB ram and 16 GB swap, so it should not fail to allocate all that
often

top - 14:58:34 up 19:54,  6 users,  load average: 2.09, 1.94, 1.97
Tasks: 451 total,   1 running, 449 sleeping,   0 stopped,   1 zombie
Cpu(s):  3.5%us, 15.5%sy,  2.0%ni, 72.2%id,  6.5%wa,  0.1%hi,  0.3%si,  0.0%st
Mem:   8198504k total,  7599132k used,   599372k free,  1212636k buffers
Swap: 16777208k total,    83568k used, 16693640k free,   610136k cached


cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Tue, Oct 20, 2009 at 02:58:53PM +0200, Tobias Oetiker wrote:
> Hi Mel,
> 
> Today Mel Gorman wrote:
> 
> > On Tue, Oct 20, 2009 at 01:44:50PM +0200, Tobias Oetiker wrote:
> > > Hi Mel,
> > >
> > > Today Mel Gorman wrote:
> > >
> > > > On Mon, Oct 19, 2009 at 10:17:06PM +0200, Tobias Oetiker wrote:
> > >
> > > > > Oct 19 22:09:52 johan kernel: [11157.121600]  [<ffffffff813ebd42>] skb_copy+0x32/0xa0 [kern.warning]
> > > > > Oct 19 22:09:52 johan kernel: [11157.121615]  [<ffffffffa07dd33c>] vboxNetFltLinuxPacketHandler+0x5c/0xd0 [vboxnetflt] [kern.warning]
> > > > > Oct 19 22:09:52 johan kernel: [11157.121620]  [<ffffffff813f2512>] dev_hard_start_xmit+0x142/0x320 [kern.warning]
> > > >
> > > > Are the number of failures at least reduced or are they occuring at the
> > > > same rate?
> > >
> > > not that it would have any statistical significance, but I had 5
> > > failure (clusters) yesterday morning and 5 this morning ...
> > >
> >
> > Before the patches were applied, how many failures were you seeing in
> > the morning?
> 
> 5 as well ... before an after ...
> 
> > > the failures often show up in groups I saved one on
> > > http://tobi.oetiker.ch/cluster-2009-10-20-08-31.txt
> > >
> > > > Also, what was the last kernel that worked for you with this
> > > > configuration?
> > >
> > > that would be 2.6.24 ... I have not upgraded in quite some time.
> > > But since the io performance of 2.6.31 is about double in my tests
> > > I thought it would be a good thing todo ...
> > >
> >
> > That significant a different in performance may explain differences in timing
> > as well. i.e. the allocator is being put under more pressure now than it
> > was previously as more processes make forward progress.
> 
> you are saing that the problem might be even older ?
> 
> we do have 8GB ram and 16 GB swap, so it should not fail to allocate all that
> often
> 
> top - 14:58:34 up 19:54,  6 users,  load average: 2.09, 1.94, 1.97
> Tasks: 451 total,   1 running, 449 sleeping,   0 stopped,   1 zombie
> Cpu(s):  3.5%us, 15.5%sy,  2.0%ni, 72.2%id,  6.5%wa,  0.1%hi,  0.3%si,  0.0%st
> Mem:   8198504k total,  7599132k used,   599372k free,  1212636k buffers
> Swap: 16777208k total,    83568k used, 16693640k free,   610136k cached
> 

High-order atomic allocations of the type you are trying at that frequency
were always a very long shot. The most likely outcome is that something
has changed that means a burst of allocations trigger an allocation failure
where as before processes would delay long enough for the system not to notice.

1. Have MTU settings changed?
2. As order-5 allocations are required to succeed, I'm surprised in a
   sense that there are only 5 failures because it implies the machine is
   actually recovering and continueing on as normal. Can you think of what
   happens in the morning that causes a burst of allocations to occur?
3. Other than the failures, have you noticed any other problems with the
   machine or does it continue along happily?
4. Does the following patch help by any chance?

Thanks

==== CUT HERE ====
vmscan: Force kswapd to take notice faster when high-order watermarks are being hit

When a high-order allocation fails, kswapd is kicked so that it reclaims
at a higher-order to avoid direct reclaimers stall and to help GFP_ATOMIC
allocations. Something has changed in recent kernels that affect the timing
where high-order GFP_ATOMIC allocations are now failing with more frequency,
particularly under pressure. This patch forces kswapd to notice sooner that
high-order allocations are occuring by checking when watermarks are hit early
and by having kswapd restart quickly when the reclaim order is increased.

Not-signed-off-by-because-this-is-a-hatchet-job: Mel Gorman <mel@csn.ul.ie>
--- 
 mm/page_alloc.c |   14 ++++++++++++--
 mm/vmscan.c     |    9 +++++++++
 2 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 2fd7b20..fdbf8c9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1907,6 +1906,17 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
 				zonelist, high_zoneidx, nodemask,
 				preferred_zone, migratetype);
 
+	/*
+	 * If after a high-order allocation we are now below watermarks,
+	 * pre-emptively kick kswapd rather than having the next allocation
+	 * fail and have to wake up kswapd, potentially failing GFP_ATOMIC
+	 * allocations or entering direct reclaim
+	 */
+	if (unlikely(order) && page && !zone_watermark_ok(preferred_zone, order,
+				preferred_zone->watermark[ALLOC_WMARK_LOW],
+				zone_idx(preferred_zone), ALLOC_WMARK_LOW))
+		wake_all_kswapd(order, zonelist, high_zoneidx);
+
 	return page;
 }
 EXPORT_SYMBOL(__alloc_pages_nodemask);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9219beb..0e66a6b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1925,6 +1925,15 @@ loop_again:
 					priority != DEF_PRIORITY)
 				continue;
 
+			/*
+			 * Exit quickly to restart if it has been indicated
+			 * that higher orders are required
+			 */
+			if (pgdat->kswapd_max_order > order) {
+				all_zones_ok = 1;
+				goto out;
+			}
+
 			if (!zone_watermark_ok(zone, order,
 					high_wmark_pages(zone), end_zone, 0))
 				all_zones_ok = 0;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Mel Gorman wrote:

> On Tue, Oct 20, 2009 at 02:58:53PM +0200, Tobias Oetiker wrote:
> > you are saing that the problem might be even older ?
> >
> > we do have 8GB ram and 16 GB swap, so it should not fail to allocate all that
> > often
> >
> > top - 14:58:34 up 19:54,  6 users,  load average: 2.09, 1.94, 1.97
> > Tasks: 451 total,   1 running, 449 sleeping,   0 stopped,   1 zombie
> > Cpu(s):  3.5%us, 15.5%sy,  2.0%ni, 72.2%id,  6.5%wa,  0.1%hi,  0.3%si,  0.0%st
> > Mem:   8198504k total,  7599132k used,   599372k free,  1212636k buffers
> > Swap: 16777208k total,    83568k used, 16693640k free,   610136k cached
> >
>
> High-order atomic allocations of the type you are trying at that frequency
> were always a very long shot. The most likely outcome is that something
> has changed that means a burst of allocations trigger an allocation failure
> where as before processes would delay long enough for the system not to notice.
>
> 1. Have MTU settings changed?

no not to my knowledge

> 2. As order-5 allocations are required to succeed, I'm surprised in a
>    sense that there are only 5 failures because it implies the machine is
>    actually recovering and continueing on as normal. Can you think of what
>    happens in the morning that causes a burst of allocations to occur?

the burts occur all day while the machine is in use ... its just
that I was  writing this at noon so only the morning had passed. So
I compared things to the day before ...

> 3. Other than the failures, have you noticed any other problems with the
>    machine or does it continue along happily?

The machine seems to be fine.

> 4. Does the following patch help by any chance?

should I try this on vanilla 2.6.31.4 or ontop of your previous
patch?

we are running virtualbox 3.0.8 on this machine, virtualbox is using
the physical network interface in bridge mode access the network.
Could this have something todo with the problem ?

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Mel Gorman]

On Tue, Oct 20, 2009 at 03:50:12PM +0200, Tobias Oetiker wrote:
> Hi Mel,
> 
> Today Mel Gorman wrote:
> 
> > On Tue, Oct 20, 2009 at 02:58:53PM +0200, Tobias Oetiker wrote:
> > > you are saing that the problem might be even older ?
> > >
> > > we do have 8GB ram and 16 GB swap, so it should not fail to allocate all that
> > > often
> > >
> > > top - 14:58:34 up 19:54,  6 users,  load average: 2.09, 1.94, 1.97
> > > Tasks: 451 total,   1 running, 449 sleeping,   0 stopped,   1 zombie
> > > Cpu(s):  3.5%us, 15.5%sy,  2.0%ni, 72.2%id,  6.5%wa,  0.1%hi,  0.3%si,  0.0%st
> > > Mem:   8198504k total,  7599132k used,   599372k free,  1212636k buffers
> > > Swap: 16777208k total,    83568k used, 16693640k free,   610136k cached
> > >
> >
> > High-order atomic allocations of the type you are trying at that frequency
> > were always a very long shot. The most likely outcome is that something
> > has changed that means a burst of allocations trigger an allocation failure
> > where as before processes would delay long enough for the system not to notice.
> >
> > 1. Have MTU settings changed?
> 
> no not to my knowledge
> 
> > 2. As order-5 allocations are required to succeed, I'm surprised in a
> >    sense that there are only 5 failures because it implies the machine is
> >    actually recovering and continueing on as normal. Can you think of what
> >    happens in the morning that causes a burst of allocations to occur?
> 
> the burts occur all day while the machine is in use ... its just
> that I was  writing this at noon so only the morning had passed. So
> I compared things to the day before ...
> 

Over the course of a day, how many would you see? By and large, it seems
that the problem yourself and Frans are similar except his is a lot more
severe.

> > 3. Other than the failures, have you noticed any other problems with the
> >    machine or does it continue along happily?
> 
> The machine seems to be fine.
> 
> > 4. Does the following patch help by any chance?
> 
> should I try this on vanilla 2.6.31.4 or ontop of your previous
> patch?
> 

Try on top of vanilla 2.6.31.4 first plase and if failures still occur,
then on top of the previous patch.

> we are running virtualbox 3.0.8 on this machine, virtualbox is using
> the physical network interface in bridge mode access the network.
> Could this have something todo with the problem ?
> 

I do not know for sure. I'm assuming the configuration is the same on
both kernels so it's unlikely to be the issue.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Today Mel Gorman wrote:

>
> Over the course of a day, how many would you see? By and large, it seems
> that the problem yourself and Frans are similar except his is a lot more
> severe.

yesterday it was 19 for 24 hours, today it is 9 for 16 hours (day
is not done yet).

> Try on top of vanilla 2.6.31.4 first plase and if failures still occur,
> then on top of the previous patch.

ok

> > we are running virtualbox 3.0.8 on this machine, virtualbox is using
> > the physical network interface in bridge mode access the network.
> > Could this have something todo with the problem ?
> >
>
> I do not know for sure. I'm assuming the configuration is the same on
> both kernels so it's unlikely to be the issue.

just to be on the sure side I created a tickt with the virtualbox
people ... http://www.virtualbox.org/ticket/5260

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures (generic)

[Tobias Oetiker]

Hi Mel,

Tuesday Mel Gorman wrote:
> 4. Does the following patch help by any chance?
>
> Thanks
>
> ==== CUT HERE ====
> vmscan: Force kswapd to take notice faster when high-order watermarks are being hit
>
> When a high-order allocation fails, kswapd is kicked so that it reclaims
> at a higher-order to avoid direct reclaimers stall and to help GFP_ATOMIC
> allocations. Something has changed in recent kernels that affect the timing
> where high-order GFP_ATOMIC allocations are now failing with more frequency,
> particularly under pressure. This patch forces kswapd to notice sooner that
> high-order allocations are occuring by checking when watermarks are hit early
> and by having kswapd restart quickly when the reclaim order is increased.
>
> Not-signed-off-by-because-this-is-a-hatchet-job: Mel Gorman <mel@csn.ul.ie>
> ---

it does seem to help ... I have been running it from 6am to 12am on
our server now and have not yet seen any issues ...

will shout if I do ...

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Jens Axboe]

On Mon, Oct 19 2009, Pekka Enberg wrote:
> (Adding Jens to CC.)
> 
> On Wednesday 14 October 2009, Frans Pop wrote:
> > > > There still has not been a mm-change identified that makes
> > > > fragmentation significantly worse.
> 
> On Mon, 2009-10-19 at 01:33 +0200, Frans Pop wrote:
> > > My bisection shows a very clear point, even if not an individual commit,
> > > in the 'akpm' merge where SKB errors suddenly become *much* more
> > > frequent and easy to trigger.
> > > I'm sorry to say this, but the fact that nothing has been identified yet
> > > is IMO the result of a lack of effort, not because there is no such
> > > change.
> > 
> > I was wrong. It turns out that I was creating the variations in the test 
> > results around the akpm merge myself by tiny changes in the way I ran the 
> > tests. It took another round of about 30 compilations and tests purely in 
> > this range to show that, but those same tests also made me aware of other 
> > patterns I should look at.
> > 
> > Until a few days ago I was concentrating on "do I see SKB allocation errors 
> > or not". Since then I've also been looking more consciously at when they 
> > happen, at disk access patterns and at desktop freeze patterns.
> > 
> > I think I did mention before that this whole issue is rather subtle :-/
> > So, my apologies for finguering the wrong area for so long, but it looked 
> > solid given the info available at the time.
> > 
> > On Thursday 15 October 2009, Mel Gorman wrote:
> > > Outside the range of commits suspected of causing problems was the
> > > following. It's extremely low probability
> > >
> > > Commit 8aa7e84 Fix congestion_wait() sync/async vs read/write confusion
> > >         This patch alters the call to congestion_wait() in the page
> > >         allocator. Frankly, I don't get the change but it might worth
> > >         checking if replacing BLK_RW_ASYNC with WRITE on top of 2.6.31
> > >         makes any difference
> > 
> > This is the real culprit. Mel: thanks very much for looking beyond the 
> > area I identified. Your overview of mm changes was exactly what I needed 
> > and really helped a lot during my later tests.
> > 
> > This commit definitely causes most of the problems; confirmed by reverting 
> > it on top of 2.6.31 (also requires reverting 373c0a7e, which is a later 
> > build fix).
> 
> Mel/Jens, any ideas why commit 8aa7e84 makes us run out of high order
> pages? Should we be using BLK_RW_SYNC in mm/page_alloc.c instead of
> BLK_RW_ASYNC?

No, I think that is definitely broken since the page freeing should be
using async writes. If the commit in question is making the difference
and the below does indeed fix it, I think that's primarliy due to timing
issues and the general brokeness of the congestion bits. With the below
change, you essentially guarenteed to be waiting 20ms every time and
it's quite likely that that is enough to change the picture.

So I'd like elsewhere for the real problem, it's not likely to be caused
by the sync vs async bits themselves.

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Mon, Oct 19, 2009 at 01:33:29AM +0200, Frans Pop wrote:
> Another long mail, sorry.
> 
> On Wednesday 14 October 2009, Frans Pop wrote:
> > > There still has not been a mm-change identified that makes
> > > fragmentation significantly worse.
> >
> > My bisection shows a very clear point, even if not an individual commit,
> > in the 'akpm' merge where SKB errors suddenly become *much* more
> > frequent and easy to trigger.
> > I'm sorry to say this, but the fact that nothing has been identified yet
> > is IMO the result of a lack of effort, not because there is no such
> > change.
> 
> I was wrong. It turns out that I was creating the variations in the test 
> results around the akpm merge myself by tiny changes in the way I ran the 
> tests. It took another round of about 30 compilations and tests purely in 
> this range to show that, but those same tests also made me aware of other 
> patterns I should look at.
> 

Once again, thanks for persisting with this for so long. That many tests
and searching is a miserable undertaking.

> Until a few days ago I was concentrating on "do I see SKB allocation errors 
> or not". Since then I've also been looking more consciously at when they 
> happen, at disk access patterns and at desktop freeze patterns.
> 
> I think I did mention before that this whole issue is rather subtle :-/

Indeed

> So, my apologies for finguering the wrong area for so long, but it looked 
> solid given the info available at the time.
> 
> On Thursday 15 October 2009, Mel Gorman wrote:
> > Outside the range of commits suspected of causing problems was the
> > following. It's extremely low probability
> >
> > Commit 8aa7e84 Fix congestion_wait() sync/async vs read/write confusion
> >         This patch alters the call to congestion_wait() in the page
> >         allocator. Frankly, I don't get the change but it might worth
> >         checking if replacing BLK_RW_ASYNC with WRITE on top of 2.6.31
> >         makes any difference
> 
> This is the real culprit. Mel: thanks very much for looking beyond the 
> area I identified. Your overview of mm changes was exactly what I needed 
> and really helped a lot during my later tests.
> 

I'm surprised this made such a big difference which is why I described
it as "extremely low probability". It implies that the real problem isn't
fragmentation per-se but the timing of when pages get consumed.

Maybe what has really changed is how long direct reclaimers wait before trying
to allocate again. After the commit, if direct reclaimers are waiting longer
between direct reclaim attempts, it might mean that the GFP_KERNEL reclaimers
of high-order pages are doing less work before and hurting parallel GFP_ATOMIC
users. Jens, does this sound plausible?

> This commit definitely causes most of the problems; confirmed by reverting 
> it on top of 2.6.31 (also requires reverting 373c0a7e, which is a later 
> build fix).
> 
> The rest of this mail gives details on my tests and how I reached the above 
> conclusion.
> 
> TEST BASELINE (2.6.30)
> ======================
> I mentioned in an earlier mail that I run three instances of gitk for my 
> tests. Loading gitk seems to consist of 3 phases:
> 1) general initial scan of the repository (branches?)
> 2) reading commits: commit counter increases
> 3) reading references (including bisection good/bad points) and
>    uncommitted changes
> 
> Below times and comments per stage when the test is run with 2.6.30. As my 
> test starts after a clean boot, buffers are mostly empty.
> 
> 1st instance: 'gitk v2.6.29..master' (preparation)
> 1) ~20 seconds; user interface is mostly blank
> 2) ~5 seconds to read 35.000 commits; user interface is updated and counter
>    increases steadily as they are read
> 3) ~10 seconds; "branch"/"follows"/"precedes" info and tags are filled
>    in; fairly heavy disk activity
> 
> 2st instance: 'gitk master' (preparation)
> 1) 0 seconds (because data is already buffered)
> 2) ~25 seconds to read 167500 commits; counter increases steadily
> 3) 1-2 seconds (because data is already buffered)
> 
> 3st instance: 'gitk master' (the actual test)
> 1) 0 seconds because data is already buffered
> 2) ~55 seconds due to swapping overhead; minor music skip around commit
>    110.000; counter slower after 90.000, some short halts, but generally
>    increases steadily; moderate disk activity
> 3) ~55-60 seconds; because buffers have been emptied data must by read
>    again, with swapping; very heavy disk activity; fairly long music
>    skip (15-20 seconds), but no SKB allocation errors
> 
> So, the loading of the 3rd instance takes 1.5 minutes longer than the 
> second because of the swapping. And phase 3) is most affected by it.
> 
> AFTER WIRELESS CHANGE
> =====================
> After commit 4752c93c30 ("iwlcore: Allow skb allocation from tasklet") I 
> start getting the SKB errors. They can be triggered reliably if the whole 
> test is repeated 1 or 2 times, but generally not the first time the test 
> is run.

It's up to the wireless driver maintainer what to do here, but it seems
like that patch needs to be reverted and thought about some more before
trying again.

> 
> Or so I thought for a long time.
> It turns out that I will get SKB errors during the first run if I'm
> "sloppy" in the test execution. For example if I wait too long before 
> switching from the last gitk instance to konsole where I have 
> a 'tail -f /var/log/kern.log' running.

So the timing is critical of when the high-order atomic allocations
start kicking in.

> Another factor is the state of the repository: do I have master checked 
> out, or an older branch, or am I in the middle of a bisection. This 
> influences how data is read from the disk and thus the test results.
> A last factor may be the size of the kernel I'm using: my test/bisect 
> kernel is significantly smaller than my regular kernel.
> 
> If the test is run completely cleanly, I will not get SKB errors during the 
> first run. Also, this change does not affect the timings of the test at 
> all: the total load time of the 3rd instance is still ~1:55 and music 
> skips happen in roughly the same places. The pattern of disk activity also 
> remains unchanged.
> 
> If I do *not* run the test cleanly, any SKB errors during the first test 
> run will always be during phase 3), never during phase 2). This is what I 
> saw during tests in the 'akpm' range, and explains the inconsistent 
> results there.
> 
> After discovering this I've made a copy of the git repo so that I always 
> test using the exact same state and tightened my test procedure.
> 
> AFTER congestion_wait CHANGE
> ============================
> If I test commit 9f2d8be, which is just before the congestion_wait() 
> change, I still get the same pattern as described above. But when I test 
> with 8aa7e84 ("Fix congestion_wait() sync/async vs read/write confusion"), 
> things change dramatically when the 3rd gitk instance is started.
> 

So, assuming this is a timing problem, this commit affects the timing of
when pages are consumed by processes doing direct reclaim.

> During the 2nd phase I see the first SKB allocation errors with a music 
> skip between reading commits 95.000 and 110.000.
> About commit 115.000 there is a very long pause during which the counter 
> does not increase, music stops and the desktop freezes completely. The 
> first 30 seconds of that freeze there is only very low disk activity (which 
> seems strange);

I'm just going to have to depend on Jens here. Jens, the congestion_wait() is
on BLK_RW_ASYNC after the commit. Reclaim usually writes pages asynchronously
but lumpy reclaim actually waits of pages to write out synchronously so
it's not always async.

Either way, reclaim is usually worried about writing pages but it would appear
after this change that a lot of read activity can also stall a process in
direct reclaim. What might be happening in Frans's particular case is that the
tasklet that allocates high-order pages for the RX buffers is getting stalled
by congestion caused by other processes doing reads from the filesystem.
While it makes sense from a congestion point of view to halt the IO, the
reclaim operations from direct reclaimers is getting delayed for long enough
to cause problems for GFP_ATOMIC.

Does this sound plausible to you? If so, what's the best way of
addressing this? Changing congestion_wait back to WRITE (assuming that
works for Frans)? Changing it to SYNC (again, assuming it actually
works) or a revert?

> the next 25 seconds there suddenly is very high disk  
> activity during which things gradually unfreeze and more SKB errors are 
> displayed. After that the commit counter runs up fairly steadily again.
> 
> Phase 2) ends at ~1:45. Phase 3) (with more SKB errors) ends at ~2:05.
> 
> So this change almost doubles the time needed for phase 2) and causes SKB 
> allocation errors to occur during that phase. Also, before this commit the 
> desktop freezes are much shorter and less severe. With this change the 
> desktop is completely unusable for almost a minute during phase 2), with 
> even the mouse pointer frozen solid.
> Note that phase 3) becomes shorter, but that the total time needed to load 
> the 3rd instance increases by about 10-15 seconds.
> 
> Note: -rc2 and -rc3 had broken NFS, so I had to cherry-pick 3 NFS commits 
> from -rc4 on top of the commits I wanted to test.
> 
> WITH congestion_wait CHANGE REVERTED
> ====================================
> I've done quite a few tests of 2.6.31 with 373c0a7e and 8aa7e847 reverted 
> to confirm that's really the culprit. I've done this for .31-rc3, .31-rc4,
> .31-rc5, .31 and .31.1.
> 
> In all cases the huge freeze in phase 2) is gone and the general behavior 
> and timings are again as it was after the wireless change. During most 
> tests I did not get any SKB allocation errors during phase 2) or phase 3).
> 
> However with .31-rc5, .31 and .31.1 I have had some tests where I would see 
> a few SKB allocation errors during phase 3) (which is somewhat likely), 
> but also during phase 2). At this point I'm unsure whether this is just 
> noise, or maybe a minor influence from some change merged after .31-rc4.
> Looking through the commits there are several mm/page allocation changes.
> 

It could still be kswapd not being woken up often enough after direct
reclaimers. I took a look through the commits but none of the mm or
allocator changes struck me as likely candidates for making
fragmentation worse or altering the timing.

> For now I suggest ignoring this though as the impact (if any) is very minor 
> and it is not reproducible reliably enough.
> 
> Next I'll retest Mel's patches and also test Reinette's patches.
> 

Of the two patches, only the kswapd one should have any significance. As
David pointed out, the second patch is essentially a no-op as it should
not have been possible to enter direct reclaim with ALLOC_NO_WATERMARKS
set.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Chris Mason]

On Mon, Oct 19, 2009 at 03:01:52PM +0100, Mel Gorman wrote:
> 
> > During the 2nd phase I see the first SKB allocation errors with a music 
> > skip between reading commits 95.000 and 110.000.
> > About commit 115.000 there is a very long pause during which the counter 
> > does not increase, music stops and the desktop freezes completely. The 
> > first 30 seconds of that freeze there is only very low disk activity (which 
> > seems strange);
> 
> I'm just going to have to depend on Jens here. Jens, the congestion_wait() is
> on BLK_RW_ASYNC after the commit. Reclaim usually writes pages asynchronously
> but lumpy reclaim actually waits of pages to write out synchronously so
> it's not always async.

Waiting doesn't make it synchronous from the elevator point of view ;)
If you're using WB_SYNC_NONE, it's a async write.  WB_SYNC_ALL makes it
a sync write.  I only see WB_SYNC_NONE in vmscan.c, so we should be
using the async congestion wait.  (the exception is xfs which always
does async writes).

But I'm honestly not 100% sure.  Looking back through the emails, the
test case is doing IO on top of a whole lot of things on top of
dm-crypt?  I just tried to figure out if dm-crypt is turning the async
IO into sync IOs, but didn't quite make sense of it.

Could you also please include which filesystems were being abused during
the test and how?  Reading through the emails, I think you've got:

gitk being run 3 times on some FS (NFS?)
streaming reads on NFS
swap on dm-crypt

If other filesystems are being used, please correct me.  Also please
include if they are on crypto or straight block device.

> 
> Either way, reclaim is usually worried about writing pages but it would appear
> after this change that a lot of read activity can also stall a process in
> direct reclaim. What might be happening in Frans's particular case is that the
> tasklet that allocates high-order pages for the RX buffers is getting stalled
> by congestion caused by other processes doing reads from the filesystem.
> While it makes sense from a congestion point of view to halt the IO, the
> reclaim operations from direct reclaimers is getting delayed for long enough
> to cause problems for GFP_ATOMIC.

The congestion_wait code either waits for congestion to clear or for
a given timeout.  The part that isn't clear is if before the patch
we waited a very short time (congestion cleared quickly) or a very long
time (we hit the timeout or congestion cleared slowly).

The easiest way to tell is to just replace the congestion_wait() calls
in direct reclaim with schedule_timeout_interruptible(10), test, then
schedule_timeout_interruptible(HZ/20), then test again.

> 
> Does this sound plausible to you? If so, what's the best way of
> addressing this? Changing congestion_wait back to WRITE (assuming that
> works for Frans)? Changing it to SYNC (again, assuming it actually
> works) or a revert?

I don't think changing it to SYNC is a good plan unless we're actually
doing sync io.  It would be better to just wait on one of the pages that
you've sent down (or its hashed waitqueue since the page can go away).

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Christoph Hellwig]

On Tue, Oct 20, 2009 at 01:18:15AM +0900, Chris Mason wrote:
> Waiting doesn't make it synchronous from the elevator point of view ;)
> If you're using WB_SYNC_NONE, it's a async write.  WB_SYNC_ALL makes it
> a sync write.  I only see WB_SYNC_NONE in vmscan.c, so we should be
> using the async congestion wait.  (the exception is xfs which always
> does async writes).

That's only because those people who did the global sweep did not bother
to convert it or even tell the list about it.  I have a patch in my
QA queue to change it..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Christoph Hellwig]

On Tue, Oct 20, 2009 at 01:18:15AM +0900, Chris Mason wrote:
> Waiting doesn't make it synchronous from the elevator point of view ;)
> If you're using WB_SYNC_NONE, it's a async write.  WB_SYNC_ALL makes it
> a sync write.  I only see WB_SYNC_NONE in vmscan.c, so we should be
> using the async congestion wait.  (the exception is xfs which always
> does async writes).

That's only because those people who did the global sweep did not bother
to convert it or even tell the list about it.  I have a patch in my
QA queue to change it..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Chris Mason]

On Mon, Oct 19, 2009 at 01:01:15PM -0400, Christoph Hellwig wrote:
> On Tue, Oct 20, 2009 at 01:18:15AM +0900, Chris Mason wrote:
> > Waiting doesn't make it synchronous from the elevator point of view ;)
> > If you're using WB_SYNC_NONE, it's a async write.  WB_SYNC_ALL makes it
> > a sync write.  I only see WB_SYNC_NONE in vmscan.c, so we should be
> > using the async congestion wait.  (the exception is xfs which always
> > does async writes).
> 
> That's only because those people who did the global sweep did not bother
> to convert it or even tell the list about it.  I have a patch in my
> QA queue to change it..

Yes, we just didn't realize XFS was missed.  Sorry.  I wasn't trying to
blame xfs for being behind, just mentioning that we've got about 10
different variables here and I'm having a hard time figuring out which
ones to push on.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Tue, Oct 20, 2009 at 01:18:15AM +0900, Chris Mason wrote:
> On Mon, Oct 19, 2009 at 03:01:52PM +0100, Mel Gorman wrote:
> > 
> > > During the 2nd phase I see the first SKB allocation errors with a music 
> > > skip between reading commits 95.000 and 110.000.
> > > About commit 115.000 there is a very long pause during which the counter 
> > > does not increase, music stops and the desktop freezes completely. The 
> > > first 30 seconds of that freeze there is only very low disk activity (which 
> > > seems strange);
> > 
> > I'm just going to have to depend on Jens here. Jens, the congestion_wait() is
> > on BLK_RW_ASYNC after the commit. Reclaim usually writes pages asynchronously
> > but lumpy reclaim actually waits of pages to write out synchronously so
> > it's not always async.
> 
> Waiting doesn't make it synchronous from the elevator point of view ;)
> If you're using WB_SYNC_NONE, it's a async write.  WB_SYNC_ALL makes it
> a sync write.  I only see WB_SYNC_NONE in vmscan.c, so we should be
> using the async congestion wait.  (the exception is xfs which always
> does async writes).
> 

Right, reclaim always queues the pages for async IO but for lumpy reclaim,
it calls wait_on_page_writeback() but as you say, from an elevator point of
view, it's still async.

> But I'm honestly not 100% sure.  Looking back through the emails, the
> test case is doing IO on top of a whole lot of things on top of
> dm-crypt?  I just tried to figure out if dm-crypt is turning the async
> IO into sync IOs, but didn't quite make sense of it.
> 

I'm not overly sure either.

> Could you also please include which filesystems were being abused during
> the test and how?  Reading through the emails, I think you've got:
> 
> gitk being run 3 times on some FS (NFS?)
> streaming reads on NFS
> swap on dm-crypt
> 
> If other filesystems are being used, please correct me.  Also please
> include if they are on crypto or straight block device.
> 

I've attached a patch below that should allow us to cheat. When it's applied,
it outputs who called congestion_wait(), how long the timeout was and how
long it waited for. By comparing before and after sleep times, we should
be able to see which of the callers has significantly changed and if
it's something easily addressable.

> > Either way, reclaim is usually worried about writing pages but it would appear
> > after this change that a lot of read activity can also stall a process in
> > direct reclaim. What might be happening in Frans's particular case is that the
> > tasklet that allocates high-order pages for the RX buffers is getting stalled
> > by congestion caused by other processes doing reads from the filesystem.
> > While it makes sense from a congestion point of view to halt the IO, the
> > reclaim operations from direct reclaimers is getting delayed for long enough
> > to cause problems for GFP_ATOMIC.
> 
> The congestion_wait code either waits for congestion to clear or for
> a given timeout.  The part that isn't clear is if before the patch
> we waited a very short time (congestion cleared quickly) or a very long
> time (we hit the timeout or congestion cleared slowly).
> 

Using the instrumentation patch, I found with a very basic test that we
are waiting for short periods of time more often with the patch applied

      1 congestion_wait   rw=1 delay 6 timeout 25 :: before commit
      7 kswapd            congestion_wait   rw=1 delay 0 timeout 25 :: before commit
     32 kswapd            congestion_wait sync=0 delay 0 timeout 25 :: after commit
     61 kswapd            congestion_wait   rw=1 delay 1 timeout 25 :: before commit
    133 kswapd            congestion_wait sync=0 delay 1 timeout 25 :: after commit
     16 kswapd            congestion_wait   rw=1 delay 2 timeout 25 :: before commit
     70 kswapd            congestion_wait sync=0 delay 2 timeout 25 :: after commit
      1 try_to_free_pages congestion_wait sync=0 delay 2 timeout 25 :: after commit
     17 kswapd            congestion_wait   rw=1 delay 3 timeout 25 :: before commit
     28 kswapd            congestion_wait sync=0 delay 3 timeout 25 :: after commit
      1 try_to_free_pages congestion_wait sync=0 delay 3 timeout 25 :: after commit
     23 kswapd            congestion_wait   rw=1 delay 4 timeout 25 :: before commit
     16 kswapd            congestion_wait sync=0 delay 4 timeout 25 :: after commit
      5 try_to_free_pages congestion_wait sync=0 delay 4 timeout 25 :: after commit
     20 kswapd            congestion_wait   rw=1 delay 5 timeout 25 :: before commit
     18 kswapd            congestion_wait sync=0 delay 5 timeout 25 :: after commit
      3 try_to_free_pages congestion_wait sync=0 delay 5 timeout 25 :: after commit
     21 kswapd            congestion_wait   rw=1 delay 6 timeout 25 :: before commit
      8 kswapd            congestion_wait sync=0 delay 6 timeout 25 :: after commit
      2 try_to_free_pages congestion_wait sync=0 delay 6 timeout 25 :: after commit
     13 kswapd            congestion_wait   rw=1 delay 7 timeout 25 :: before commit
     12 kswapd            congestion_wait sync=0 delay 7 timeout 25 :: after commit
      2 try_to_free_pages congestion_wait sync=0 delay 7 timeout 25 :: after commit
      8 kswapd            congestion_wait   rw=1 delay 8 timeout 25 :: before commit
      7 kswapd            congestion_wait sync=0 delay 8 timeout 25 :: after commit
      9 kswapd            congestion_wait   rw=1 delay 9 timeout 25 :: before commit
      5 kswapd            congestion_wait sync=0 delay 9 timeout 25 :: after commit
      2 try_to_free_pages congestion_wait sync=0 delay 9 timeout 25 :: after commit
      4 kswapd            congestion_wait   rw=1 delay 10 timeout 25 :: before commit
      5 kswapd            congestion_wait sync=0 delay 10 timeout 25 :: after commit
      1 try_to_free_pages congestion_wait sync=0 delay 10 timeout 25 :: after commit
[... remaining output snipped ...]

The before and after commit are really 2.6.31 and 2.6.31-patch-reverted.
The first column is how many times we delayed for that length of time.
To generate the output, I just took the console log from both kernels with
a basic test, put the congestion_wait lines into two separate files and

cat congestion-*-sorted | sort -n -k5 | uniq -c

to give a count of how many times we delayed for a particular caller.

> The easiest way to tell is to just replace the congestion_wait() calls
> in direct reclaim with schedule_timeout_interruptible(10), test, then
> schedule_timeout_interruptible(HZ/20), then test again.
> 

Reclaim can also call congestion_wait() and maybe the problem isn't
within the page allocator at all but that it's indirectly affected by
timing.

> > 
> > Does this sound plausible to you? If so, what's the best way of
> > addressing this? Changing congestion_wait back to WRITE (assuming that
> > works for Frans)? Changing it to SYNC (again, assuming it actually
> > works) or a revert?
> 
> I don't think changing it to SYNC is a good plan unless we're actually
> doing sync io.  It would be better to just wait on one of the pages that
> you've sent down (or its hashed waitqueue since the page can go away).
> 

Frans, is there any chance you could apply the following patch and get
the console logs for a vanilla kernel and with the congestion patches
reverted? I'm hoping it'll be able to tell us which of the callers has
significantly changed in timing. If there is one caller that has
significantly changed, it might be enough to address just that caller.

=====

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

[-- Attachment #1: Type: text/plain, Size: 2324 bytes --]

On Tuesday 20 October 2009, Mel Gorman wrote:
> I've attached a patch below that should allow us to cheat. When it's
> applied, it outputs who called congestion_wait(), how long the timeout
> was and how long it waited for. By comparing before and after sleep
> times, we should be able to see which of the callers has significantly
> changed and if it's something easily addressable.

The results from this look fairly interesting (although I may be a bad 
judge as I don't really know what I'm looking at ;-).

I've tested with two kernels:
1) 2.6.31.1: 1 test run
2) 2.6.31.1 + congestion_wait() reverts: 2 test runs

The 1st kernel had the expected "freeze" while reading commits in gitk; 
reading commits with the 2nd kernel was more fluent.
I did 2 runs with the 2nd kernel as the first run had a fairly long music 
skip and more SKB errors than expected. The second run was fairly normal 
with no music skips at all even though it had a few SKB errors.

Data for the tests:
				1st kernel	2nd kernel 1	2nd kernel 2
end reading commits		1:15		1:00		0:55
  "freeze"			yes		no		no
branch data shown		1:55		1:15		1:10
system quiet			2:25		1:50		1:45
# SKB allocation errors		10		53		5

Note that the test is substantially faster with the 2nd kernel and that the 
SKB errors don't really affect the duration of the test.


Attached a tarball with the kernel logs, both the full logs and a stripped 
version with only the lines generated during the actual test.
Something like this will extract the debug data from the logs:
$ grep "delay " <file> | sed "s/^.*\] //"

Also attached a ODF spreadsheet with a summary of the data for all 3 tests.
I've dropped the congestion_wait and sync/rw= columns as they were always 
the same (rw=1 for 1st kernel and sync=0 for 2nd kernel).
I've added a column "weighed delay" and totals for that column and the 
count column.

My layman's observations are:
- without the revert 'background_writeout' is called a lot less frequently,
  but when it's called it gets long delays
- without the revert you have 'wb_kupdate', which is relatively expensive
- with the revert 'shrink_list' is relatively expensive, although not
  really in absolute terms

You people may want to look at exactly what happens directly around the SKB 
allocation errors. I've only looked at totals.

Cheers,
FJP


[-- Attachment #2: logs.tgz --]
[-- Type: application/x-tgz, Size: 151463 bytes --]

[-- Attachment #3: results.ods --]
[-- Type: application/vnd.oasis.opendocument.spreadsheet, Size: 20051 bytes --]

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Mon, Oct 26, 2009 at 10:06:09PM +0100, Frans Pop wrote:
> On Tuesday 20 October 2009, Mel Gorman wrote:
> > I've attached a patch below that should allow us to cheat. When it's
> > applied, it outputs who called congestion_wait(), how long the timeout
> > was and how long it waited for. By comparing before and after sleep
> > times, we should be able to see which of the callers has significantly
> > changed and if it's something easily addressable.
> 
> The results from this look fairly interesting (although I may be a bad 
> judge as I don't really know what I'm looking at ;-).
> 
> I've tested with two kernels:
> 1) 2.6.31.1: 1 test run
> 2) 2.6.31.1 + congestion_wait() reverts: 2 test runs
> 
> The 1st kernel had the expected "freeze" while reading commits in gitk; 
> reading commits with the 2nd kernel was more fluent.
> I did 2 runs with the 2nd kernel as the first run had a fairly long music 
> skip and more SKB errors than expected. The second run was fairly normal 
> with no music skips at all even though it had a few SKB errors.
> 
> Data for the tests:
> 				1st kernel	2nd kernel 1	2nd kernel 2
> end reading commits		1:15		1:00		0:55
>   "freeze"			yes		no		no
> branch data shown		1:55		1:15		1:10
> system quiet			2:25		1:50		1:45
> # SKB allocation errors		10		53		5
> 
> Note that the test is substantially faster with the 2nd kernel and that the 
> SKB errors don't really affect the duration of the test.
> 

Ok. I think that despite expectations, the writeback changes have
changed the timing significantly enough to be worth examining closer.

> 
> - without the revert 'background_writeout' is called a lot less frequently,
>   but when it's called it gets long delays
> - without the revert you have 'wb_kupdate', which is relatively expensive
> - with the revert 'shrink_list' is relatively expensive, although not
>   really in absolute terms
> 

Lets look at the callers that waited in congestion_wait() for at least
25 jiffies.

2.6.31.1-async-sync-congestion-wait i.e. vanilla kernel
generated with: cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
     24  background_writeout  congestion_wait sync=0 delay 25 timeout 25
    203  kswapd               congestion_wait sync=0 delay 25 timeout 25
      5  shrink_list          congestion_wait sync=0 delay 25 timeout 25
    155  try_to_free_pages    congestion_wait sync=0 delay 25 timeout 25
    145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
      2  kswapd               congestion_wait sync=0 delay 26 timeout 25
      8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25
      1  try_to_free_pages    congestion_wait sync=0 delay 54 timeout 25

2.6.31.1-write-congestion-wait i.e. kernel with patch reverted
generated with: cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
      2  background_writeout  congestion_wait rw=1 delay 25 timeout 25
    188  kswapd               congestion_wait rw=1 delay 25 timeout 25
     14  shrink_list          congestion_wait rw=1 delay 25 timeout 25
    181  try_to_free_pages    congestion_wait rw=1 delay 25 timeout 25
      5  kswapd               congestion_wait rw=1 delay 26 timeout 25
     10  try_to_free_pages    congestion_wait rw=1 delay 26 timeout 25
      3  try_to_free_pages    congestion_wait rw=1 delay 27 timeout 25
      1  kswapd               congestion_wait rw=1 delay 29 timeout 25
      1  __alloc_pages_nodemask congestion_wait rw=1 delay 30 timeout 5
      1  try_to_free_pages    congestion_wait rw=1 delay 31 timeout 25
      1  try_to_free_pages    congestion_wait rw=1 delay 35 timeout 25
      1  kswapd               congestion_wait rw=1 delay 51 timeout 25
      1  try_to_free_pages    congestion_wait rw=1 delay 56 timeout 25

So, wb_kupdate and background_writeout are the big movers in terms of waiting,
not the direct reclaimers which is what we were expecting. Of those big
movers, wb_kupdate is the most interested because compare the following

$ cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
[ no output ]
$ $ cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
      1  wb_kupdate           congestion_wait sync=0 delay 15 timeout 25
      1  wb_kupdate           congestion_wait sync=0 delay 23 timeout 25
    145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
      8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25

The vanilla kernel is not waiting in wb_kupdate at all.

Jens, before the congestion_wait() changes, wb_kupdate was waiting on
congestion and afterwards it's not. Furthermore, look at the number of pages
that are queued for writeback in the two page allocation failure reports.

without-revert:	writeback:65653
with-revert:	writeback:21713

So, after the move to async/sync, a lot more pages are getting queued
for writeback - more than three times the number of pages are queued for
writeback with the vanilla kernel. This amount of congestion might be why
direct reclaimers and kswapd's timings have changed so much.

Chris Mason hinted at this but I didn't quite "get it" at the time but is it
possible that writeback_inodes() is converting what is expected to be async
IO into sync IO? One way of checking this is if Frans could test the patch
below that makes wb_kupdate wait on sync instead of async.

If this makes a difference, I think the three main areas of trouble we
are now seeing are

	1. page allocator regressions - mostly fixed hopefully
	2. page writeback change in timing - theory yet to be confirmed
	3. drivers using more atomics - iwlagn specific, being dealt with

Of course, the big problem is if the changes are due to major timing
differences in page writeback, then mainline is a totally different
shape of problem as pdflush has been replaced there.

==== 
Have wb_kupdate wait on sync IO congestion instead of async

wb_kupdate is expected to only have queued up pages for async IO.
However, something screwy is happening because it never appears to go to
sleep. Frans, can you test with this patch instead of the revert please?
Preferably, keep the verbose-congestion_wait patch applied so we can
still get an idea who is going to sleep and for how long when calling
congestion_wait. thanks

Not-signed-off-hacket-job: Mel Gorman <mel@csn.ul.ie>
--- 

diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 81627eb..cb646dd 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -787,7 +787,7 @@ static void wb_kupdate(unsigned long arg)
 		writeback_inodes(&wbc);
 		if (wbc.nr_to_write > 0) {
 			if (wbc.encountered_congestion || wbc.more_io)
-				congestion_wait(BLK_RW_ASYNC, HZ/10);
+				congestion_wait(BLK_RW_SYNC, HZ/10);
 			else
 				break;	/* All the old data is written */
 		}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[KOSAKI Motohiro]

2009/10/27 Mel Gorman <mel@csn.ul.ie>:
> On Mon, Oct 26, 2009 at 10:06:09PM +0100, Frans Pop wrote:
>> On Tuesday 20 October 2009, Mel Gorman wrote:
>> > I've attached a patch below that should allow us to cheat. When it's
>> > applied, it outputs who called congestion_wait(), how long the timeout
>> > was and how long it waited for. By comparing before and after sleep
>> > times, we should be able to see which of the callers has significantly
>> > changed and if it's something easily addressable.
>>
>> The results from this look fairly interesting (although I may be a bad
>> judge as I don't really know what I'm looking at ;-).
>>
>> I've tested with two kernels:
>> 1) 2.6.31.1: 1 test run
>> 2) 2.6.31.1 + congestion_wait() reverts: 2 test runs
>>
>> The 1st kernel had the expected "freeze" while reading commits in gitk;
>> reading commits with the 2nd kernel was more fluent.
>> I did 2 runs with the 2nd kernel as the first run had a fairly long music
>> skip and more SKB errors than expected. The second run was fairly normal
>> with no music skips at all even though it had a few SKB errors.
>>
>> Data for the tests:
>>                               1st kernel      2nd kernel 1    2nd kernel 2
>> end reading commits           1:15            1:00            0:55
>>   "freeze"                    yes             no              no
>> branch data shown             1:55            1:15            1:10
>> system quiet                  2:25            1:50            1:45
>> # SKB allocation errors               10              53              5
>>
>> Note that the test is substantially faster with the 2nd kernel and that the
>> SKB errors don't really affect the duration of the test.
>>
>
> Ok. I think that despite expectations, the writeback changes have
> changed the timing significantly enough to be worth examining closer.
>
>>
>> - without the revert 'background_writeout' is called a lot less frequently,
>>   but when it's called it gets long delays
>> - without the revert you have 'wb_kupdate', which is relatively expensive
>> - with the revert 'shrink_list' is relatively expensive, although not
>>   really in absolute terms
>>
>
> Lets look at the callers that waited in congestion_wait() for at least
> 25 jiffies.
>
> 2.6.31.1-async-sync-congestion-wait i.e. vanilla kernel
> generated with: cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
>     24  background_writeout  congestion_wait sync=0 delay 25 timeout 25
>    203  kswapd               congestion_wait sync=0 delay 25 timeout 25
>      5  shrink_list          congestion_wait sync=0 delay 25 timeout 25
>    155  try_to_free_pages    congestion_wait sync=0 delay 25 timeout 25
>    145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
>      2  kswapd               congestion_wait sync=0 delay 26 timeout 25
>      8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25
>      1  try_to_free_pages    congestion_wait sync=0 delay 54 timeout 25
>
> 2.6.31.1-write-congestion-wait i.e. kernel with patch reverted
> generated with: cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
>      2  background_writeout  congestion_wait rw=1 delay 25 timeout 25
>    188  kswapd               congestion_wait rw=1 delay 25 timeout 25
>     14  shrink_list          congestion_wait rw=1 delay 25 timeout 25
>    181  try_to_free_pages    congestion_wait rw=1 delay 25 timeout 25
>      5  kswapd               congestion_wait rw=1 delay 26 timeout 25
>     10  try_to_free_pages    congestion_wait rw=1 delay 26 timeout 25
>      3  try_to_free_pages    congestion_wait rw=1 delay 27 timeout 25
>      1  kswapd               congestion_wait rw=1 delay 29 timeout 25
>      1  __alloc_pages_nodemask congestion_wait rw=1 delay 30 timeout 5
>      1  try_to_free_pages    congestion_wait rw=1 delay 31 timeout 25
>      1  try_to_free_pages    congestion_wait rw=1 delay 35 timeout 25
>      1  kswapd               congestion_wait rw=1 delay 51 timeout 25
>      1  try_to_free_pages    congestion_wait rw=1 delay 56 timeout 25
>
> So, wb_kupdate and background_writeout are the big movers in terms of waiting,
> not the direct reclaimers which is what we were expecting. Of those big
> movers, wb_kupdate is the most interested because compare the following
>
> $ cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
> [ no output ]
> $ $ cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
>      1  wb_kupdate           congestion_wait sync=0 delay 15 timeout 25
>      1  wb_kupdate           congestion_wait sync=0 delay 23 timeout 25
>    145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
>      8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25
>
> The vanilla kernel is not waiting in wb_kupdate at all.
>
> Jens, before the congestion_wait() changes, wb_kupdate was waiting on
> congestion and afterwards it's not. Furthermore, look at the number of pages
> that are queued for writeback in the two page allocation failure reports.
>
> without-revert: writeback:65653
> with-revert:    writeback:21713
>
> So, after the move to async/sync, a lot more pages are getting queued
> for writeback - more than three times the number of pages are queued for
> writeback with the vanilla kernel. This amount of congestion might be why
> direct reclaimers and kswapd's timings have changed so much.
>
> Chris Mason hinted at this but I didn't quite "get it" at the time but is it
> possible that writeback_inodes() is converting what is expected to be async
> IO into sync IO? One way of checking this is if Frans could test the patch
> below that makes wb_kupdate wait on sync instead of async.
>
> If this makes a difference, I think the three main areas of trouble we
> are now seeing are
>
>        1. page allocator regressions - mostly fixed hopefully
>        2. page writeback change in timing - theory yet to be confirmed
>        3. drivers using more atomics - iwlagn specific, being dealt with
>
> Of course, the big problem is if the changes are due to major timing
> differences in page writeback, then mainline is a totally different
> shape of problem as pdflush has been replaced there.
>
> ====
> Have wb_kupdate wait on sync IO congestion instead of async
>
> wb_kupdate is expected to only have queued up pages for async IO.
> However, something screwy is happening because it never appears to go to
> sleep. Frans, can you test with this patch instead of the revert please?
> Preferably, keep the verbose-congestion_wait patch applied so we can
> still get an idea who is going to sleep and for how long when calling
> congestion_wait. thanks
>
> Not-signed-off-hacket-job: Mel Gorman <mel@csn.ul.ie>
> ---
>
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index 81627eb..cb646dd 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -787,7 +787,7 @@ static void wb_kupdate(unsigned long arg)
>                writeback_inodes(&wbc);
>                if (wbc.nr_to_write > 0) {
>                        if (wbc.encountered_congestion || wbc.more_io)
> -                               congestion_wait(BLK_RW_ASYNC, HZ/10);
> +                               congestion_wait(BLK_RW_SYNC, HZ/10);
>                        else
>                                break;  /* All the old data is written */
>                }

Hmm, This doesn't looks correct to me.

BLK_RW_ASYNC mean async write.
BLK_RW_SYNC  mean read and sync-write.

wb_kupdate use WB_SYNC_NONE. it's async write.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Wed, Oct 28, 2009 at 12:16:30AM +0900, KOSAKI Motohiro wrote:
> 2009/10/27 Mel Gorman <mel@csn.ul.ie>:
> > On Mon, Oct 26, 2009 at 10:06:09PM +0100, Frans Pop wrote:
> >> On Tuesday 20 October 2009, Mel Gorman wrote:
> >> > I've attached a patch below that should allow us to cheat. When it's
> >> > applied, it outputs who called congestion_wait(), how long the timeout
> >> > was and how long it waited for. By comparing before and after sleep
> >> > times, we should be able to see which of the callers has significantly
> >> > changed and if it's something easily addressable.
> >>
> >> The results from this look fairly interesting (although I may be a bad
> >> judge as I don't really know what I'm looking at ;-).
> >>
> >> I've tested with two kernels:
> >> 1) 2.6.31.1: 1 test run
> >> 2) 2.6.31.1 + congestion_wait() reverts: 2 test runs
> >>
> >> The 1st kernel had the expected "freeze" while reading commits in gitk;
> >> reading commits with the 2nd kernel was more fluent.
> >> I did 2 runs with the 2nd kernel as the first run had a fairly long music
> >> skip and more SKB errors than expected. The second run was fairly normal
> >> with no music skips at all even though it had a few SKB errors.
> >>
> >> Data for the tests:
> >>                               1st kernel      2nd kernel 1    2nd kernel 2
> >> end reading commits           1:15            1:00            0:55
> >>   "freeze"                    yes             no              no
> >> branch data shown             1:55            1:15            1:10
> >> system quiet                  2:25            1:50            1:45
> >> # SKB allocation errors               10              53              5
> >>
> >> Note that the test is substantially faster with the 2nd kernel and that the
> >> SKB errors don't really affect the duration of the test.
> >>
> >
> > Ok. I think that despite expectations, the writeback changes have
> > changed the timing significantly enough to be worth examining closer.
> >
> >>
> >> - without the revert 'background_writeout' is called a lot less frequently,
> >>   but when it's called it gets long delays
> >> - without the revert you have 'wb_kupdate', which is relatively expensive
> >> - with the revert 'shrink_list' is relatively expensive, although not
> >>   really in absolute terms
> >>
> >
> > Lets look at the callers that waited in congestion_wait() for at least
> > 25 jiffies.
> >
> > 2.6.31.1-async-sync-congestion-wait i.e. vanilla kernel
> > generated with: cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
> >     24  background_writeout  congestion_wait sync=0 delay 25 timeout 25
> >    203  kswapd               congestion_wait sync=0 delay 25 timeout 25
> >      5  shrink_list          congestion_wait sync=0 delay 25 timeout 25
> >    155  try_to_free_pages    congestion_wait sync=0 delay 25 timeout 25
> >    145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
> >      2  kswapd               congestion_wait sync=0 delay 26 timeout 25
> >      8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25
> >      1  try_to_free_pages    congestion_wait sync=0 delay 54 timeout 25
> >
> > 2.6.31.1-write-congestion-wait i.e. kernel with patch reverted
> > generated with: cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
> >      2  background_writeout  congestion_wait rw=1 delay 25 timeout 25
> >    188  kswapd               congestion_wait rw=1 delay 25 timeout 25
> >     14  shrink_list          congestion_wait rw=1 delay 25 timeout 25
> >    181  try_to_free_pages    congestion_wait rw=1 delay 25 timeout 25
> >      5  kswapd               congestion_wait rw=1 delay 26 timeout 25
> >     10  try_to_free_pages    congestion_wait rw=1 delay 26 timeout 25
> >      3  try_to_free_pages    congestion_wait rw=1 delay 27 timeout 25
> >      1  kswapd               congestion_wait rw=1 delay 29 timeout 25
> >      1  __alloc_pages_nodemask congestion_wait rw=1 delay 30 timeout 5
> >      1  try_to_free_pages    congestion_wait rw=1 delay 31 timeout 25
> >      1  try_to_free_pages    congestion_wait rw=1 delay 35 timeout 25
> >      1  kswapd               congestion_wait rw=1 delay 51 timeout 25
> >      1  try_to_free_pages    congestion_wait rw=1 delay 56 timeout 25
> >
> > So, wb_kupdate and background_writeout are the big movers in terms of waiting,
> > not the direct reclaimers which is what we were expecting. Of those big
> > movers, wb_kupdate is the most interested because compare the following
> >
> > $ cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
> > [ no output ]
> > $ $ cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
> >      1  wb_kupdate           congestion_wait sync=0 delay 15 timeout 25
> >      1  wb_kupdate           congestion_wait sync=0 delay 23 timeout 25
> >    145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
> >      8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25
> >
> > The vanilla kernel is not waiting in wb_kupdate at all.
> >
> > Jens, before the congestion_wait() changes, wb_kupdate was waiting on
> > congestion and afterwards it's not. Furthermore, look at the number of pages
> > that are queued for writeback in the two page allocation failure reports.
> >
> > without-revert: writeback:65653
> > with-revert:    writeback:21713
> >
> > So, after the move to async/sync, a lot more pages are getting queued
> > for writeback - more than three times the number of pages are queued for
> > writeback with the vanilla kernel. This amount of congestion might be why
> > direct reclaimers and kswapd's timings have changed so much.
> >
> > Chris Mason hinted at this but I didn't quite "get it" at the time but is it
> > possible that writeback_inodes() is converting what is expected to be async
> > IO into sync IO? One way of checking this is if Frans could test the patch
> > below that makes wb_kupdate wait on sync instead of async.
> >
> > If this makes a difference, I think the three main areas of trouble we
> > are now seeing are
> >
> >        1. page allocator regressions - mostly fixed hopefully
> >        2. page writeback change in timing - theory yet to be confirmed
> >        3. drivers using more atomics - iwlagn specific, being dealt with
> >
> > Of course, the big problem is if the changes are due to major timing
> > differences in page writeback, then mainline is a totally different
> > shape of problem as pdflush has been replaced there.
> >
> > ====
> > Have wb_kupdate wait on sync IO congestion instead of async
> >
> > wb_kupdate is expected to only have queued up pages for async IO.
> > However, something screwy is happening because it never appears to go to
> > sleep. Frans, can you test with this patch instead of the revert please?
> > Preferably, keep the verbose-congestion_wait patch applied so we can
> > still get an idea who is going to sleep and for how long when calling
> > congestion_wait. thanks
> >
> > Not-signed-off-hacket-job: Mel Gorman <mel@csn.ul.ie>
> > ---
> >
> > diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> > index 81627eb..cb646dd 100644
> > --- a/mm/page-writeback.c
> > +++ b/mm/page-writeback.c
> > @@ -787,7 +787,7 @@ static void wb_kupdate(unsigned long arg)
> >                writeback_inodes(&wbc);
> >                if (wbc.nr_to_write > 0) {
> >                        if (wbc.encountered_congestion || wbc.more_io)
> > -                               congestion_wait(BLK_RW_ASYNC, HZ/10);
> > +                               congestion_wait(BLK_RW_SYNC, HZ/10);
> >                        else
> >                                break;  /* All the old data is written */
> >                }
> 
> Hmm, This doesn't looks correct to me.
> 
> BLK_RW_ASYNC mean async write.
> BLK_RW_SYNC  mean read and sync-write.
> 
> wb_kupdate use WB_SYNC_NONE. it's async write.
> 

I don't think it's correct either which is why I described it as
"something screwy is happening because it never appears to go to sleep".
This is despite there being a whole lot of pages queued for writeback
according to the page allocation failure reports. 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Tue, Oct 27, 2009 at 02:54:35PM +0000, Mel Gorman wrote:
> On Mon, Oct 26, 2009 at 10:06:09PM +0100, Frans Pop wrote:
> > On Tuesday 20 October 2009, Mel Gorman wrote:
> > > I've attached a patch below that should allow us to cheat. When it's
> > > applied, it outputs who called congestion_wait(), how long the timeout
> > > was and how long it waited for. By comparing before and after sleep
> > > times, we should be able to see which of the callers has significantly
> > > changed and if it's something easily addressable.
> > 
> > The results from this look fairly interesting (although I may be a bad 
> > judge as I don't really know what I'm looking at ;-).
> > 
> > I've tested with two kernels:
> > 1) 2.6.31.1: 1 test run
> > 2) 2.6.31.1 + congestion_wait() reverts: 2 test runs
> > 
> > The 1st kernel had the expected "freeze" while reading commits in gitk; 
> > reading commits with the 2nd kernel was more fluent.
> > I did 2 runs with the 2nd kernel as the first run had a fairly long music 
> > skip and more SKB errors than expected. The second run was fairly normal 
> > with no music skips at all even though it had a few SKB errors.
> > 
> > Data for the tests:
> > 				1st kernel	2nd kernel 1	2nd kernel 2
> > end reading commits		1:15		1:00		0:55
> >   "freeze"			yes		no		no
> > branch data shown		1:55		1:15		1:10
> > system quiet			2:25		1:50		1:45
> > # SKB allocation errors		10		53		5
> > 
> > Note that the test is substantially faster with the 2nd kernel and that the 
> > SKB errors don't really affect the duration of the test.
> > 
> 
> Ok. I think that despite expectations, the writeback changes have
> changed the timing significantly enough to be worth examining closer.
> 
> > 
> > - without the revert 'background_writeout' is called a lot less frequently,
> >   but when it's called it gets long delays
> > - without the revert you have 'wb_kupdate', which is relatively expensive
> > - with the revert 'shrink_list' is relatively expensive, although not
> >   really in absolute terms
> > 
> 
> Lets look at the callers that waited in congestion_wait() for at least
> 25 jiffies.
> 
> 2.6.31.1-async-sync-congestion-wait i.e. vanilla kernel
> generated with: cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
>      24  background_writeout  congestion_wait sync=0 delay 25 timeout 25
>     203  kswapd               congestion_wait sync=0 delay 25 timeout 25
>       5  shrink_list          congestion_wait sync=0 delay 25 timeout 25
>     155  try_to_free_pages    congestion_wait sync=0 delay 25 timeout 25
>     145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
>       2  kswapd               congestion_wait sync=0 delay 26 timeout 25
>       8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25
>       1  try_to_free_pages    congestion_wait sync=0 delay 54 timeout 25
> 
> 2.6.31.1-write-congestion-wait i.e. kernel with patch reverted
> generated with: cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c
>       2  background_writeout  congestion_wait rw=1 delay 25 timeout 25
>     188  kswapd               congestion_wait rw=1 delay 25 timeout 25
>      14  shrink_list          congestion_wait rw=1 delay 25 timeout 25
>     181  try_to_free_pages    congestion_wait rw=1 delay 25 timeout 25
>       5  kswapd               congestion_wait rw=1 delay 26 timeout 25
>      10  try_to_free_pages    congestion_wait rw=1 delay 26 timeout 25
>       3  try_to_free_pages    congestion_wait rw=1 delay 27 timeout 25
>       1  kswapd               congestion_wait rw=1 delay 29 timeout 25
>       1  __alloc_pages_nodemask congestion_wait rw=1 delay 30 timeout 5
>       1  try_to_free_pages    congestion_wait rw=1 delay 31 timeout 25
>       1  try_to_free_pages    congestion_wait rw=1 delay 35 timeout 25
>       1  kswapd               congestion_wait rw=1 delay 51 timeout 25
>       1  try_to_free_pages    congestion_wait rw=1 delay 56 timeout 25
> 
> So, wb_kupdate and background_writeout are the big movers in terms of waiting,
> not the direct reclaimers which is what we were expecting. Of those big
> movers, wb_kupdate is the most interested because compare the following
> 

Bah, this part is right, but I got the next section the wrong way
around. I should have renamed the damn things instead of remember what
was 1 and what was 2.

1 == vanilla
2 == with-revert

> $ cat kern.log_2.1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
> [ no output ]
> $ $ cat kern.log_1_test | awk -F ] '{print $2}' | sort -k 5 -n | uniq -c | grep wb_kup
>       1  wb_kupdate           congestion_wait sync=0 delay 15 timeout 25
>       1  wb_kupdate           congestion_wait sync=0 delay 23 timeout 25
>     145  wb_kupdate           congestion_wait sync=0 delay 25 timeout 25
>       8  wb_kupdate           congestion_wait sync=0 delay 26 timeout 25
> 
> The vanilla kernel is not waiting in wb_kupdate at all.
> 

The vanilla kernel *is* waiting. The reverted kernel is not. If my patch
makes any difference, it's not for the right reasons.

> Jens, before the congestion_wait() changes, wb_kupdate was waiting on
> congestion and afterwards it's not. Furthermore, look at the number of pages
> that are queued for writeback in the two page allocation failure reports.
> 
> without-revert:	writeback:65653
> with-revert:	writeback:21713
> 

and got it back right again.

kernel 1 == vanilla kernel == without-revert	writeback:65653
kernel 2 == revert  kernel == with-revert	writeback:21713

> So, after the move to async/sync, a lot more pages are getting queued
> for writeback - more than three times the number of pages are queued for
> writeback with the vanilla kernel. This amount of congestion might be why
> direct reclaimers and kswapd's timings have changed so much.
> 

Or more accurately, the vanilla kernel has queued up a lot more pages for
IO than when the patch is reverted. I'm not seeing yet why this is.

> Chris Mason hinted at this but I didn't quite "get it" at the time but is it
> possible that writeback_inodes() is converting what is expected to be async
> IO into sync IO? One way of checking this is if Frans could test the patch
> below that makes wb_kupdate wait on sync instead of async.
> 

This reasoning is rubbish. If the patch makes any difference, it's because
it changes timing. It's probably more important to figure out if a) if the
different number of pages for writeback is relevant and if so b) why has
it changed.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Chris Mason]

On Tue, Oct 27, 2009 at 03:52:24PM +0000, Mel Gorman wrote:
> 
> > So, after the move to async/sync, a lot more pages are getting queued
> > for writeback - more than three times the number of pages are queued for
> > writeback with the vanilla kernel. This amount of congestion might be why
> > direct reclaimers and kswapd's timings have changed so much.
> > 
> 
> Or more accurately, the vanilla kernel has queued up a lot more pages for
> IO than when the patch is reverted. I'm not seeing yet why this is.

[ sympathies over confusion about congestion...lots of variables here ]

If wb_kupdate has been able to queue more writes it is because the
congestion logic isn't stopping it.  We have congestion_wait(), but
before calling that in the writeback paths it says: are you congested?
and then backs off if the answer is yes.

Ideally, direct reclaim will never do writeback.  We want it to be able
to find clean pages that kupdate and friends have already processed.

Waiting for congestion is a funny thing, it only tells us the device has
managed to finish some IO or that a timeout has passed.  Neither event has
any relation to figuring out if the IO for reclaimable pages has
finished.

One option is to have the VM remember the hashed waitqueue for one of
the pages it direct reclaims and then wait on it.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Tuesday 27 October 2009, Chris Mason wrote:
> On Tue, Oct 27, 2009 at 03:52:24PM +0000, Mel Gorman wrote:
> > > So, after the move to async/sync, a lot more pages are getting
> > > queued for writeback - more than three times the number of pages are
> > > queued for writeback with the vanilla kernel. This amount of
> > > congestion might be why direct reclaimers and kswapd's timings have
> > > changed so much.
> >
> > Or more accurately, the vanilla kernel has queued up a lot more pages
> > for IO than when the patch is reverted. I'm not seeing yet why this
> > is.
>
> [ sympathies over confusion about congestion...lots of variables here ]
>
> If wb_kupdate has been able to queue more writes it is because the
> congestion logic isn't stopping it.  We have congestion_wait(), but
> before calling that in the writeback paths it says: are you congested?
> and then backs off if the answer is yes.
>
> Ideally, direct reclaim will never do writeback.  We want it to be able
> to find clean pages that kupdate and friends have already processed.
>
> Waiting for congestion is a funny thing, it only tells us the device has
> managed to finish some IO or that a timeout has passed.  Neither event
> has any relation to figuring out if the IO for reclaimable pages has
> finished.
>
> One option is to have the VM remember the hashed waitqueue for one of
> the pages it direct reclaims and then wait on it.

What people should be aware of is the behavior of the system I see at this 
point. I've already mentioned this in other mails, but it's probably good 
to repeat it here.

While gitk is reading commits with vanilla .31 and .32 kernels there is at 
some point a fairly long period (10-20 seconds) where I see:
- a completely frozen desktop, including frozen mouse cursor
- really very little disk activity (HD led flashes very briefly less than
  once per second)
- reading commits stops completely during this period
- no music.
After that there is a period (another 5-15 seconds) with a huge amount of 
disk activity during which the system gradually becomes responsive again 
and in gitk the count of commits that have been read starts increasing 
again (without a jump in the counter which confirms that no commits were 
read during the freeze).

I cannot really tell what the system is doing during those freezes. Because 
of the frozen desktop I cannot for example see CPU usage. I suspect that, 
as there is hardly any disk activity, the system must be reorganizing RAM 
or something. But it seems quite bad that that gets "bunched up" instead 
of happening more gradually.

With the congestion_wait() change reverted I never see these freezes, only 
much more normal minor latencies (< 2 seconds; mostly < 0.5 seconds), 
which is probably unavoidable during heavy swapping.

Hth,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Tuesday 27 October 2009, Chris Mason wrote:
> On Tue, Oct 27, 2009 at 03:52:24PM +0000, Mel Gorman wrote:
> > > So, after the move to async/sync, a lot more pages are getting
> > > queued for writeback - more than three times the number of pages are
> > > queued for writeback with the vanilla kernel. This amount of
> > > congestion might be why direct reclaimers and kswapd's timings have
> > > changed so much.
> >
> > Or more accurately, the vanilla kernel has queued up a lot more pages
> > for IO than when the patch is reverted. I'm not seeing yet why this
> > is.
>
> [ sympathies over confusion about congestion...lots of variables here ]
>
> If wb_kupdate has been able to queue more writes it is because the
> congestion logic isn't stopping it.  We have congestion_wait(), but
> before calling that in the writeback paths it says: are you congested?
> and then backs off if the answer is yes.
>
> Ideally, direct reclaim will never do writeback.  We want it to be able
> to find clean pages that kupdate and friends have already processed.
>
> Waiting for congestion is a funny thing, it only tells us the device has
> managed to finish some IO or that a timeout has passed.  Neither event
> has any relation to figuring out if the IO for reclaimable pages has
> finished.
>
> One option is to have the VM remember the hashed waitqueue for one of
> the pages it direct reclaims and then wait on it.

What people should be aware of is the behavior of the system I see at this 
point. I've already mentioned this in other mails, but it's probably good 
to repeat it here.

While gitk is reading commits with vanilla .31 and .32 kernels there is at 
some point a fairly long period (10-20 seconds) where I see:
- a completely frozen desktop, including frozen mouse cursor
- really very little disk activity (HD led flashes very briefly less than
  once per second)
- reading commits stops completely during this period
- no music.
After that there is a period (another 5-15 seconds) with a huge amount of 
disk activity during which the system gradually becomes responsive again 
and in gitk the count of commits that have been read starts increasing 
again (without a jump in the counter which confirms that no commits were 
read during the freeze).

I cannot really tell what the system is doing during those freezes. Because 
of the frozen desktop I cannot for example see CPU usage. I suspect that, 
as there is hardly any disk activity, the system must be reorganizing RAM 
or something. But it seems quite bad that that gets "bunched up" instead 
of happening more gradually.

With the congestion_wait() change reverted I never see these freezes, only 
much more normal minor latencies (< 2 seconds; mostly < 0.5 seconds), 
which is probably unavoidable during heavy swapping.

Hth,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Monday 26 October 2009, Frans Pop wrote:
> On Tuesday 20 October 2009, Mel Gorman wrote:
> > I've attached a patch below that should allow us to cheat. When it's
> > applied, it outputs who called congestion_wait(), how long the timeout
> > was and how long it waited for. By comparing before and after sleep
> > times, we should be able to see which of the callers has significantly
> > changed and if it's something easily addressable.
>
> The results from this look fairly interesting (although I may be a bad
> judge as I don't really know what I'm looking at ;-).
>
> I've tested with two kernels:
> 1) 2.6.31.1: 1 test run
> 2) 2.6.31.1 + congestion_wait() reverts: 2 test runs

I've taken another look at the data from this debug patch, resulting in 
these graphs: http://people.debian.org/~fjp/tmp/kernel/congestion.pdf

I think the graph may show the reason for the congestion_wait() regression. 
Horizontal axis shows time, vertical axis shows number of logged 
congestion_wait calls per type.

The top chart is without the revert, the bottom one after the revert.

Note how before the revert the graph shows distinct steps: first you get 
almost exclusively kwapd, followed by almost exclusively alloc_pages and 
try_to_free. I suspect the periods where kswapd is almost horizontal 
correspond to the freezes.

With the revert the lines for the different functions are almost straight 
and everything happens much better interspersed.

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Thursday 05 November 2009, Frans Pop wrote:
> On Monday 26 October 2009, Frans Pop wrote:
> > On Tuesday 20 October 2009, Mel Gorman wrote:
> > > I've attached a patch below that should allow us to cheat. When it's
> > > applied, it outputs who called congestion_wait(), how long the
> > > timeout was and how long it waited for. By comparing before and
> > > after sleep times, we should be able to see which of the callers has
> > > significantly changed and if it's something easily addressable.
> >
> > The results from this look fairly interesting (although I may be a bad
> > judge as I don't really know what I'm looking at ;-).
> >
> > I've tested with two kernels:
> > 1) 2.6.31.1: 1 test run
> > 2) 2.6.31.1 + congestion_wait() reverts: 2 test runs
>
> I've taken another look at the data from this debug patch, resulting in
> these graphs: http://people.debian.org/~fjp/tmp/kernel/congestion.pdf
>
> I think the graph may show the reason for the congestion_wait()
> regression. Horizontal axis shows time, vertical axis shows number of
> logged congestion_wait calls per type.

I'm sorry. My initial version had a skewed time axis (showed occurrences 
instead of actual time). I've now uploaded a corrected version:
http://people.debian.org/~fjp/tmp/kernel/congestion.pdf

I've also uploaded a second version that shows cumulative delay per type, 
which probably gives a better insight:
http://people.debian.org/~fjp/tmp/kernel/congestion2.pdf

For both the top chart is without the revert, the bottom one after the 
revert.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Fri, Nov 06, 2009 at 10:51:37AM +0100, Frans Pop wrote:
> On Thursday 05 November 2009, Frans Pop wrote:
> > On Monday 26 October 2009, Frans Pop wrote:
> > > On Tuesday 20 October 2009, Mel Gorman wrote:
> > > > I've attached a patch below that should allow us to cheat. When it's
> > > > applied, it outputs who called congestion_wait(), how long the
> > > > timeout was and how long it waited for. By comparing before and
> > > > after sleep times, we should be able to see which of the callers has
> > > > significantly changed and if it's something easily addressable.
> > >
> > > The results from this look fairly interesting (although I may be a bad
> > > judge as I don't really know what I'm looking at ;-).
> > >
> > > I've tested with two kernels:
> > > 1) 2.6.31.1: 1 test run
> > > 2) 2.6.31.1 + congestion_wait() reverts: 2 test runs
> >
> > I've taken another look at the data from this debug patch, resulting in
> > these graphs: http://people.debian.org/~fjp/tmp/kernel/congestion.pdf
> >
> > I think the graph may show the reason for the congestion_wait()
> > regression. Horizontal axis shows time, vertical axis shows number of
> > logged congestion_wait calls per type.
> 
> I'm sorry. My initial version had a skewed time axis (showed occurrences 
> instead of actual time). I've now uploaded a corrected version:
> http://people.debian.org/~fjp/tmp/kernel/congestion.pdf
> 
> I've also uploaded a second version that shows cumulative delay per type, 
> which probably gives a better insight:
> http://people.debian.org/~fjp/tmp/kernel/congestion2.pdf
> 
> For both the top chart is without the revert, the bottom one after the 
> revert.
> 

I'm looking into this at the moment. There are some definite
differences not only in the length congestion_wait() is waiting but in
what the callers are doing. I've more or less reproduced your results
locally and am slowly plodding through each caller to see what has
changed of significance. No patches yet though.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Tue, Oct 20, 2009 at 01:18:15AM +0900, Chris Mason wrote:
> On Mon, Oct 19, 2009 at 03:01:52PM +0100, Mel Gorman wrote:
> > 
> > > During the 2nd phase I see the first SKB allocation errors with a music 
> > > skip between reading commits 95.000 and 110.000.
> > > About commit 115.000 there is a very long pause during which the counter 
> > > does not increase, music stops and the desktop freezes completely. The 
> > > first 30 seconds of that freeze there is only very low disk activity (which 
> > > seems strange);
> > 
> > I'm just going to have to depend on Jens here. Jens, the congestion_wait() is
> > on BLK_RW_ASYNC after the commit. Reclaim usually writes pages asynchronously
> > but lumpy reclaim actually waits of pages to write out synchronously so
> > it's not always async.
> 
> Waiting doesn't make it synchronous from the elevator point of view ;)
> If you're using WB_SYNC_NONE, it's a async write.  WB_SYNC_ALL makes it
> a sync write.  I only see WB_SYNC_NONE in vmscan.c, so we should be
> using the async congestion wait.  (the exception is xfs which always
> does async writes).
> 

Right, reclaim always queues the pages for async IO but for lumpy reclaim,
it calls wait_on_page_writeback() but as you say, from an elevator point of
view, it's still async.

> But I'm honestly not 100% sure.  Looking back through the emails, the
> test case is doing IO on top of a whole lot of things on top of
> dm-crypt?  I just tried to figure out if dm-crypt is turning the async
> IO into sync IOs, but didn't quite make sense of it.
> 

I'm not overly sure either.

> Could you also please include which filesystems were being abused during
> the test and how?  Reading through the emails, I think you've got:
> 
> gitk being run 3 times on some FS (NFS?)
> streaming reads on NFS
> swap on dm-crypt
> 
> If other filesystems are being used, please correct me.  Also please
> include if they are on crypto or straight block device.
> 

I've attached a patch below that should allow us to cheat. When it's applied,
it outputs who called congestion_wait(), how long the timeout was and how
long it waited for. By comparing before and after sleep times, we should
be able to see which of the callers has significantly changed and if
it's something easily addressable.

> > Either way, reclaim is usually worried about writing pages but it would appear
> > after this change that a lot of read activity can also stall a process in
> > direct reclaim. What might be happening in Frans's particular case is that the
> > tasklet that allocates high-order pages for the RX buffers is getting stalled
> > by congestion caused by other processes doing reads from the filesystem.
> > While it makes sense from a congestion point of view to halt the IO, the
> > reclaim operations from direct reclaimers is getting delayed for long enough
> > to cause problems for GFP_ATOMIC.
> 
> The congestion_wait code either waits for congestion to clear or for
> a given timeout.  The part that isn't clear is if before the patch
> we waited a very short time (congestion cleared quickly) or a very long
> time (we hit the timeout or congestion cleared slowly).
> 

Using the instrumentation patch, I found with a very basic test that we
are waiting for short periods of time more often with the patch applied

      1 congestion_wait   rw=1 delay 6 timeout 25 :: before commit
      7 kswapd            congestion_wait   rw=1 delay 0 timeout 25 :: before commit
     32 kswapd            congestion_wait sync=0 delay 0 timeout 25 :: after commit
     61 kswapd            congestion_wait   rw=1 delay 1 timeout 25 :: before commit
    133 kswapd            congestion_wait sync=0 delay 1 timeout 25 :: after commit
     16 kswapd            congestion_wait   rw=1 delay 2 timeout 25 :: before commit
     70 kswapd            congestion_wait sync=0 delay 2 timeout 25 :: after commit
      1 try_to_free_pages congestion_wait sync=0 delay 2 timeout 25 :: after commit
     17 kswapd            congestion_wait   rw=1 delay 3 timeout 25 :: before commit
     28 kswapd            congestion_wait sync=0 delay 3 timeout 25 :: after commit
      1 try_to_free_pages congestion_wait sync=0 delay 3 timeout 25 :: after commit
     23 kswapd            congestion_wait   rw=1 delay 4 timeout 25 :: before commit
     16 kswapd            congestion_wait sync=0 delay 4 timeout 25 :: after commit
      5 try_to_free_pages congestion_wait sync=0 delay 4 timeout 25 :: after commit
     20 kswapd            congestion_wait   rw=1 delay 5 timeout 25 :: before commit
     18 kswapd            congestion_wait sync=0 delay 5 timeout 25 :: after commit
      3 try_to_free_pages congestion_wait sync=0 delay 5 timeout 25 :: after commit
     21 kswapd            congestion_wait   rw=1 delay 6 timeout 25 :: before commit
      8 kswapd            congestion_wait sync=0 delay 6 timeout 25 :: after commit
      2 try_to_free_pages congestion_wait sync=0 delay 6 timeout 25 :: after commit
     13 kswapd            congestion_wait   rw=1 delay 7 timeout 25 :: before commit
     12 kswapd            congestion_wait sync=0 delay 7 timeout 25 :: after commit
      2 try_to_free_pages congestion_wait sync=0 delay 7 timeout 25 :: after commit
      8 kswapd            congestion_wait   rw=1 delay 8 timeout 25 :: before commit
      7 kswapd            congestion_wait sync=0 delay 8 timeout 25 :: after commit
      9 kswapd            congestion_wait   rw=1 delay 9 timeout 25 :: before commit
      5 kswapd            congestion_wait sync=0 delay 9 timeout 25 :: after commit
      2 try_to_free_pages congestion_wait sync=0 delay 9 timeout 25 :: after commit
      4 kswapd            congestion_wait   rw=1 delay 10 timeout 25 :: before commit
      5 kswapd            congestion_wait sync=0 delay 10 timeout 25 :: after commit
      1 try_to_free_pages congestion_wait sync=0 delay 10 timeout 25 :: after commit
[... remaining output snipped ...]

The before and after commit are really 2.6.31 and 2.6.31-patch-reverted.
The first column is how many times we delayed for that length of time.
To generate the output, I just took the console log from both kernels with
a basic test, put the congestion_wait lines into two separate files and

cat congestion-*-sorted | sort -n -k5 | uniq -c

to give a count of how many times we delayed for a particular caller.

> The easiest way to tell is to just replace the congestion_wait() calls
> in direct reclaim with schedule_timeout_interruptible(10), test, then
> schedule_timeout_interruptible(HZ/20), then test again.
> 

Reclaim can also call congestion_wait() and maybe the problem isn't
within the page allocator at all but that it's indirectly affected by
timing.

> > 
> > Does this sound plausible to you? If so, what's the best way of
> > addressing this? Changing congestion_wait back to WRITE (assuming that
> > works for Frans)? Changing it to SYNC (again, assuming it actually
> > works) or a revert?
> 
> I don't think changing it to SYNC is a good plan unless we're actually
> doing sync io.  It would be better to just wait on one of the pages that
> you've sent down (or its hashed waitqueue since the page can go away).
> 

Frans, is there any chance you could apply the following patch and get
the console logs for a vanilla kernel and with the congestion patches
reverted? I'm hoping it'll be able to tell us which of the callers has
significantly changed in timing. If there is one caller that has
significantly changed, it might be enough to address just that caller.

=====

>From 757999066dc41f2e053d59589c673052fc7c1a65 Mon Sep 17 00:00:00 2001
From: Mel Gorman <mel@csn.ul.ie>
Date: Tue, 20 Oct 2009 11:01:57 +0100
Subject: [PATCH] Instrument congestion_wait

This patch instruments how long congestion_wait() really waited for a
given caller.

diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 3d3accb..fc945e0 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -10,6 +10,7 @@
 #include <linux/module.h>
 #include <linux/writeback.h>
 #include <linux/device.h>
+#include <linux/kallsyms.h>
 
 void default_unplug_io_fn(struct backing_dev_info *bdi, struct page *page)
 {
@@ -729,6 +730,11 @@ EXPORT_SYMBOL(set_bdi_congested);
  */
 long congestion_wait(int sync, long timeout)
 {
+	unsigned long jiffies_start = jiffies;
+	char *module;
+	char buf[128];
+	const char *symbol;
+	unsigned long offset, symbolsize;
 	long ret;
 	DEFINE_WAIT(wait);
 	wait_queue_head_t *wqh = &congestion_wqh[sync];
@@ -736,6 +742,13 @@ long congestion_wait(int sync, long timeout)
 	prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
 	ret = io_schedule_timeout(timeout);
 	finish_wait(wqh, &wait);
+
+	symbol = kallsyms_lookup(_RET_IP_, &symbolsize, &offset, &module, buf),
+	printk(KERN_INFO "%-20s congestion_wait sync=%d delay %lu timeout %ld\n",
+		symbol,
+		sync,
+		jiffies - jiffies_start,
+		timeout);
 	return ret;
 }
 EXPORT_SYMBOL(congestion_wait);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

Sorry for the delayed reply.

On Monday 19 October 2009, Chris Mason wrote:
> On Mon, Oct 19, 2009 at 03:01:52PM +0100, Mel Gorman wrote:
> > > During the 2nd phase I see the first SKB allocation errors with a
> > > music skip between reading commits 95.000 and 110.000.
> > > About commit 115.000 there is a very long pause during which the
> > > counter does not increase, music stops and the desktop freezes
> > > completely. The first 30 seconds of that freeze there is only very
> > > low disk activity (which seems strange);
> >
> > I'm just going to have to depend on Jens here. Jens, the
> > congestion_wait() is on BLK_RW_ASYNC after the commit. Reclaim usually
> > writes pages asynchronously but lumpy reclaim actually waits of pages
> > to write out synchronously so it's not always async.
>
> Waiting doesn't make it synchronous from the elevator point of view ;)
> If you're using WB_SYNC_NONE, it's a async write.  WB_SYNC_ALL makes it
> a sync write.  I only see WB_SYNC_NONE in vmscan.c, so we should be
> using the async congestion wait.  (the exception is xfs which always
> does async writes).
>
> But I'm honestly not 100% sure.  Looking back through the emails, the
> test case is doing IO on top of a whole lot of things on top of
> dm-crypt?  I just tried to figure out if dm-crypt is turning the async
> IO into sync IOs, but didn't quite make sense of it.
>
> Could you also please include which filesystems were being abused during
> the test and how?  Reading through the emails, I think you've got:
>
> gitk being run 3 times on some FS (NFS?)

gitk is run on an ext3 logical volume in a volume group that's on a LUKS 
encrypted partition of the local hard disk.

So it's: SATA harddisk -> dm-crypt (dmsetup) -> LVM (lvm2) -> ext3

> streaming reads on NFS

Correct. My music share is a remote (nfs4) read-only mounted ext3 
partition.

> swap on dm-crypt

Correct. Swap is another logical volume in the same volume group as 
mentioned above.

So kcrypt gets to (de)encrypt both the gitk data *and* any swapping caused 
by that [1].

> If other filesystems are being used, please correct me.  Also please
> include if they are on crypto or straight block device.

All my file systems are ext3. Nothing newfangled or exotic ;-)
There are some bind mounts involved, but I expect that's transparent.

Cheers,
FJP

[1] I've plans to move some of my data outside the encrypted volume, but 
currently everything except /boot is in the encrypted VG.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

Hi Mel,

On Wed, 2009-10-14 at 03:30 -0700, Mel Gorman wrote:
> From 5fb9f897117bf2701f9fdebe4d008dbe34358ab9 Mon Sep 17 00:00:00 2001
> From: Mel Gorman <mel@csn.ul.ie>
> Date: Wed, 14 Oct 2009 11:19:57 +0100
> Subject: [PATCH] iwlwifi: Suppress warnings related to GFP_ATOMIC allocations that do not matter
> 
> iwlwifi refills RX buffers in two ways - a direct method using GFP_ATOMIC
> and a tasklet method using GFP_KERNEL. There are a number of RX buffers and
> there are only serious issues when there are no RX buffers left. The driver
> explicitly warns when refills are failing and the buffers are low but it
> always warns when a GFP_ATOMIC allocation fails even when there is no
> packet loss as a result.


No, it does not always warn when a GFP_ATOMIC allocation fails. Please
check earlier in iwl_rx_allocate() we have:

if (rxq->free_count > RX_LOW_WATERMARK)
	priority |= __GFP_NOWARN;

So it will suppress warnings as long as we have buffers available.

We do want to see warnings if memory is below watermark and allocation
fails - your patch prevents these warnings from appearing.

> This patch specifies __GFP_NOWARN for the direct refill method that uses
> GFP_ATOMIC. To help identify where allocation failures might be coming
> from, the stack is dumped when the RX queue is dangerously low.
> 
> Signed-off-by: Mel Gorman <mel@csn.ul.ie>
> --- 
>  drivers/net/wireless/iwlwifi/iwl-rx.c |    6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/trace/postprocess/trace-pagealloc-postprocess.pl b/Documentation/trace/postprocess/trace-pagealloc-postprocess.pl
> old mode 100644
> new mode 100755
> diff --git a/drivers/net/wireless/iwlwifi/iwl-rx.c b/drivers/net/wireless/iwlwifi/iwl-rx.c
> index 8e1bb53..f91a108 100644
> --- a/drivers/net/wireless/iwlwifi/iwl-rx.c
> +++ b/drivers/net/wireless/iwlwifi/iwl-rx.c
> @@ -260,10 +260,12 @@ void iwl_rx_allocate(struct iwl_priv *priv, gfp_t priority)
>  			if (net_ratelimit())
>  				IWL_DEBUG_INFO(priv, "Failed to allocate SKB buffer.\n");
>  			if ((rxq->free_count <= RX_LOW_WATERMARK) &&
> -			    net_ratelimit())
> +			    net_ratelimit()) {
>  				IWL_CRIT(priv, "Failed to allocate SKB buffer with %s. Only %u free buffers remaining.\n",
>  					 priority == GFP_ATOMIC ?  "GFP_ATOMIC" : "GFP_KERNEL",
>  					 rxq->free_count);
> +				dump_stack();
> +			}
>  			/* We don't reschedule replenish work here -- we will
>  			 * call the restock method and if it still needs
>  			 * more buffers it will schedule replenish */
> @@ -320,7 +322,7 @@ EXPORT_SYMBOL(iwl_rx_replenish);
>  
>  void iwl_rx_replenish_now(struct iwl_priv *priv)
>  {
> -	iwl_rx_allocate(priv, GFP_ATOMIC);
> +	iwl_rx_allocate(priv, GFP_ATOMIC|__GFP_NOWARN);
>  
>  	iwl_rx_queue_restock(priv);
>  }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Mel Gorman]

On Wed, Oct 14, 2009 at 09:28:00AM -0700, reinette chatre wrote:
> Hi Mel,
> 
> On Wed, 2009-10-14 at 03:30 -0700, Mel Gorman wrote:
> > From 5fb9f897117bf2701f9fdebe4d008dbe34358ab9 Mon Sep 17 00:00:00 2001
> > From: Mel Gorman <mel@csn.ul.ie>
> > Date: Wed, 14 Oct 2009 11:19:57 +0100
> > Subject: [PATCH] iwlwifi: Suppress warnings related to GFP_ATOMIC allocations that do not matter
> > 
> > iwlwifi refills RX buffers in two ways - a direct method using GFP_ATOMIC
> > and a tasklet method using GFP_KERNEL. There are a number of RX buffers and
> > there are only serious issues when there are no RX buffers left. The driver
> > explicitly warns when refills are failing and the buffers are low but it
> > always warns when a GFP_ATOMIC allocation fails even when there is no
> > packet loss as a result.
> 
> 
> No, it does not always warn when a GFP_ATOMIC allocation fails. Please
> check earlier in iwl_rx_allocate() we have:
> 
> if (rxq->free_count > RX_LOW_WATERMARK)
> 	priority |= __GFP_NOWARN;
> 
> So it will suppress warnings as long as we have buffers available.
> 
> We do want to see warnings if memory is below watermark and allocation
> fails - your patch prevents these warnings from appearing.
> 

Yeah, the patch is balls and is not the way forward.

What is your take on GFP_ATOMIC-direct deleting the pool before the tasklet
can refill it with GFP_KERNEL? Should direct allocation be falling back to
calling with GFP_KERNEL when the pool has been depleted instead of failing?



-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

On Wed, 2009-10-14 at 09:50 -0700, Mel Gorman wrote:

> What is your take on GFP_ATOMIC-direct deleting the pool before the tasklet
> can refill it with GFP_KERNEL?

I am not sure I understand your question. We attempt to reclaim a
received buffer on every receive, and with a queue size of 256 + 64 we
assume to have a pretty big buffer to deal with cases when allocations
fail. So, technically, for us to get into this situation where we start
seeing these allocation failures there would have been more than 200
times in which GFP_ATOMIC allocations failed that we did _not_ see since
we only see those warnings when there are less than 8 free buffers
remaining. More on this below ...

>  Should direct allocation be falling back to
> calling with GFP_KERNEL when the pool has been depleted instead of failing?

This is the intention of the current implementation. In the tasklet we
run iwl_rx_replenish_now(), which attempts the GFP_ATOMIC allocations
first by calling iwl_rx_allocate() with the GFP_ATOMIC flag. No
particular action is taken when this fails (apart from the error
message), but if the buffers are running low then iwl_rx_queue_restock()
(which is also called from iwl_rx_replenish_now()) will queue work that
will do the allocation with GFP_KERNEL.

We do queue the GFP_KERNEL allocations when there are only a few buffers
remaining in the queue (8 right now) ... maybe we can make this higher?

I am not sure if this will help in what you are trying to figure out
here, but would it help to play with the numbers here? That is, in
iwl_rx_queue_restock() we have:

if (rxq->free_count <= RX_LOW_WATERMARK)
	queue_work(priv->workqueue, &priv->rx_replenish);

Would it help here to make that value higher? Maybe queue the GFP_KERNEL
allocation when there are, for example, 50 or 100 free buffers
remaining? 

Reinette


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Wednesday 14 October 2009, reinette chatre wrote:
> We do queue the GFP_KERNEL allocations when there are only a few buffers
> remaining in the queue (8 right now) ...

Are you sure of this? I have zero messages in my logs about allocation 
failures with GFP_KERNEL, but I do have plenty with "Only 0 free buffers 
remaining" with GFP_ATOMIC.

Does that indicate a bug or could they fall under the ratelimit somehow?
Or do I misunderstand the logic?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

On Wed, 2009-10-14 at 14:33 -0700, Frans Pop wrote:
> On Wednesday 14 October 2009, reinette chatre wrote:
> > We do queue the GFP_KERNEL allocations when there are only a few buffers
> > remaining in the queue (8 right now) ...
> 
> Are you sure of this? I have zero messages in my logs about allocation 
> failures with GFP_KERNEL, but I do have plenty with "Only 0 free buffers 
> remaining" with GFP_ATOMIC.

That does make sense to me. We do not expect allocations with GFP_KERNEL
to fail. Considering how I understand how things work I am considering
the following scenario:

* start with system low on available memory
* now introduce incoming traffic (causing the RX code to run)
* upon receipt of frame we attempt an allocation (to reclaim the buffer)
with GFP_ATOMIC (state: num RX buffer free > watermark)
  * this fails since memory is not available
  * num RX buffer free reduces
  * does _not_ queue replenishment of buffers with GFP_KERNEL
* repeat above until we hit the watermark (currently 8)
* upon receipt of frame we attempt an allocation (to reclaim the buffer)
with GFP_ATOMIC (state: num RX buffer free <= watermark) 
  * this fails (now user sees big warning)
  * queue replenishment of buffers with GFP_KERNEL

Essentially what I suspect could happen is that
we do attempt to replenish the buffers with GFP_KERNEL after several
failures with GFP_ATOMIC, but at that point we have already run out
completely.

One way to test this theory is to queue the GFP_KERNEL allocation
earlier (when we still have a significant number of RX buffers
available), 8 may turn out to be too small.

> Does that indicate a bug or could they fall under the ratelimit somehow?

In your kernel log I do see that the driver's error messages related to
GFP_ATOMIC are rate limited (we see many more "order-2 allocation
failure" messages than the "Failed to allocate" messages). All of these
allocation failures are from the "replenish_now" code though, which is
GFP_ATOMIC. So even though we do not see the "Failed to allocate" errors
(which are rate limited) it seems that all allocation failures are from
that (the GFP_ATOMIC) code.

Reinette


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

On Wednesday 14 October 2009, reinette chatre wrote:
> We do queue the GFP_KERNEL allocations when there are only a few buffers
> remaining in the queue (8 right now) ... maybe we can make this higher?

I've tried increasing it to 50. Here's the result for a single test:
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 25 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 48 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 48 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 48 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
__ratelimit: 1 callbacks suppressed
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
__ratelimit: 97 callbacks suppressed
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 44 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.

This is with current mainline (v2.6.32-rc4-149-ga3ccf63).

The log file timestamps don't tell much as the logging gets delayed,
so they all end up at the same time. Maybe I should enable the kernel
timestamps so we can see how far apart these failures are.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

On Wed, 2009-10-14 at 19:02 -0700, Frans Pop wrote:
> On Wednesday 14 October 2009, reinette chatre wrote:
> > We do queue the GFP_KERNEL allocations when there are only a few buffers
> > remaining in the queue (8 right now) ... maybe we can make this higher?
> 
> I've tried increasing it to 50. Here's the result for a single test:
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 25 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 48 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 48 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 48 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> __ratelimit: 1 callbacks suppressed
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> __ratelimit: 97 callbacks suppressed
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 44 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> iwlagn 0000:10:00.0: Failed to allocate SKB buffer with GFP_ATOMIC. Only 0 free buffers remaining.
> 
> This is with current mainline (v2.6.32-rc4-149-ga3ccf63).

> The log file timestamps don't tell much as the logging gets delayed,
> so they all end up at the same time. Maybe I should enable the kernel
> timestamps so we can see how far apart these failures are.

If you can get accurate timing it will be very useful. I am interested
to see how quickly it goes from "48 free buffers" to "0 free buffers".

Thank you

Reinette



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

On Thursday 15 October 2009, reinette chatre wrote:
> > The log file timestamps don't tell much as the logging gets delayed,
> > so they all end up at the same time. Maybe I should enable the kernel
> > timestamps so we can see how far apart these failures are.
>
> If you can get accurate timing it will be very useful. I am interested
> to see how quickly it goes from "48 free buffers" to "0 free buffers".

Attached the dmesg for three consecutive test runs (i.e. without 
rebooting). Not that the 2nd one includes only "0 free buffers" messages, 
even though the behavior (point where desktop freezes and music stops) 
looked similar.

Not sure if you can tell all that much from the data.

N.B. You may want to clean this up in iwlwifi code:
iwl-dev.h:#include "iwl-fh.h"
iwl-dev.h:#define RX_LOW_WATERMARK 8
iwl-fh.h:#define RX_LOW_WATERMARK 8

I.e: RX_LOW_WATERMARK is defined in iwl-dev.h even though that includes 
iwl-fh.h where it's also defined. The same may be true for other defines.

I think this gave me an incorrect result the first time I increased the 
limit as I only changed one of the two files (iwl-dev.h IIRC).

Cheers,
FJP


[-- Attachment #2: dmesg.tgz --]
[-- Type: application/x-tgz, Size: 44980 bytes --]

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

On Thu, 2009-10-15 at 12:41 -0700, Frans Pop wrote:
> On Thursday 15 October 2009, reinette chatre wrote:
> > > The log file timestamps don't tell much as the logging gets delayed,
> > > so they all end up at the same time. Maybe I should enable the kernel
> > > timestamps so we can see how far apart these failures are.
> >
> > If you can get accurate timing it will be very useful. I am interested
> > to see how quickly it goes from "48 free buffers" to "0 free buffers".
> 
> Attached the dmesg for three consecutive test runs (i.e. without 
> rebooting). Not that the 2nd one includes only "0 free buffers" messages, 
> even though the behavior (point where desktop freezes and music stops) 
> looked similar.

Thank you very much. I am studying it.


> Not sure if you can tell all that much from the data.
> 
> N.B. You may want to clean this up in iwlwifi code:
> iwl-dev.h:#include "iwl-fh.h"
> iwl-dev.h:#define RX_LOW_WATERMARK 8
> iwl-fh.h:#define RX_LOW_WATERMARK 8
> 
> I.e: RX_LOW_WATERMARK is defined in iwl-dev.h even though that includes 
> iwl-fh.h where it's also defined. The same may be true for other defines.

Sorry about that. The patch below will fix that. I will send it
separately to wireless list.

>From 7cc8e6482b359eef5ce099457037a237d355b5b1 Mon Sep 17 00:00:00 2001
From: Reinette Chatre <reinette.chatre@intel.com>
Date: Fri, 16 Oct 2009 10:11:10 -0700
Subject: [PATCH] iwlwifi: remove duplicate defines

RX_FREE_BUFFERS and RX_LOW_WATERMARK are currently defined in four places.
Based on how files are included we only need the definition in iwl-fh.h

Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Reported-by: Frans Pop <elendil@planet.nl>
---
 drivers/net/wireless/iwlwifi/iwl-3945-hw.h |    6 ------
 drivers/net/wireless/iwlwifi/iwl-3945.h    |    6 ------
 drivers/net/wireless/iwlwifi/iwl-dev.h     |    6 ------
 3 files changed, 0 insertions(+), 18 deletions(-)

diff --git a/drivers/net/wireless/iwlwifi/iwl-3945-hw.h b/drivers/net/wireless/iwlwifi/iwl-3945-hw.h
index ccdac69..6fd10d4 100644
--- a/drivers/net/wireless/iwlwifi/iwl-3945-hw.h
+++ b/drivers/net/wireless/iwlwifi/iwl-3945-hw.h
@@ -248,12 +248,6 @@ struct iwl3945_eeprom {
 #define TFD_CTL_PAD_SET(n)         (n << 28)
 #define TFD_CTL_PAD_GET(ctl)       (ctl >> 28)
 
-/*
- * RX related structures and functions
- */
-#define RX_FREE_BUFFERS 64
-#define RX_LOW_WATERMARK 8
-
 /* Sizes and addresses for instruction and data memory (SRAM) in
  * 3945's embedded processor.  Driver access is via HBUS_TARG_MEM_* regs. */
 #define IWL39_RTC_INST_LOWER_BOUND		(0x000000)
diff --git a/drivers/net/wireless/iwlwifi/iwl-3945.h b/drivers/net/wireless/iwlwifi/iwl-3945.h
index f3907c1..84fa0d7 100644
--- a/drivers/net/wireless/iwlwifi/iwl-3945.h
+++ b/drivers/net/wireless/iwlwifi/iwl-3945.h
@@ -130,12 +130,6 @@ struct iwl3945_frame {
 #define SN_TO_SEQ(ssn) (((ssn) << 4) & IEEE80211_SCTL_SEQ)
 #define MAX_SN ((IEEE80211_SCTL_SEQ) >> 4)
 
-/*
- * RX related structures and functions
- */
-#define RX_FREE_BUFFERS 64
-#define RX_LOW_WATERMARK 8
-
 #define SUP_RATE_11A_MAX_NUM_CHANNELS  8
 #define SUP_RATE_11B_MAX_NUM_CHANNELS  4
 #define SUP_RATE_11G_MAX_NUM_CHANNELS  12
diff --git a/drivers/net/wireless/iwlwifi/iwl-dev.h b/drivers/net/wireless/iwlwifi/iwl-dev.h
index 1378654..0fa0cf5 100644
--- a/drivers/net/wireless/iwlwifi/iwl-dev.h
+++ b/drivers/net/wireless/iwlwifi/iwl-dev.h
@@ -406,12 +406,6 @@ struct iwl_host_cmd {
 	u8 id;
 };
 
-/*
- * RX related structures and functions
- */
-#define RX_FREE_BUFFERS 64
-#define RX_LOW_WATERMARK 8
-
 #define SUP_RATE_11A_MAX_NUM_CHANNELS  8
 #define SUP_RATE_11B_MAX_NUM_CHANNELS  4
 #define SUP_RATE_11G_MAX_NUM_CHANNELS  12
-- 
1.5.6.3



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

[-- Attachment #1: Type: text/plain, Size: 1102 bytes --]

Hi Frans,

On Thu, 2009-10-15 at 12:41 -0700, Frans Pop wrote:
> On Thursday 15 October 2009, reinette chatre wrote:
> > > The log file timestamps don't tell much as the logging gets delayed,
> > > so they all end up at the same time. Maybe I should enable the kernel
> > > timestamps so we can see how far apart these failures are.
> >
> > If you can get accurate timing it will be very useful. I am interested
> > to see how quickly it goes from "48 free buffers" to "0 free buffers".
> 
> Attached the dmesg for three consecutive test runs (i.e. without 
> rebooting). Not that the 2nd one includes only "0 free buffers" messages, 
> even though the behavior (point where desktop freezes and music stops) 
> looked similar.
> 
> Not sure if you can tell all that much from the data.
> 

Prompted by this thread we are in process of moving allocation to paged
skb. This will definitely reduce the allocation size (from order 2 to
order 1) and hopefully help with this problem also. Could you please try
with the attached two patches? They are based on 2.6.32-rc4.

Thank you very much

Reinette






[-- Attachment #2: 0001-iwlwifi-use-paged-Rx.patch --]
[-- Type: text/x-patch, Size: 0 bytes --]

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[Frans Pop]

Sorry for the delay in replying.

On Saturday 17 October 2009, reinette chatre wrote:
> Prompted by this thread we are in process of moving allocation to paged
> skb. This will definitely reduce the allocation size (from order 2 to
> order 1) and hopefully help with this problem also. Could you please try
> with the attached two patches? They are based on 2.6.32-rc4.

Looks very good! With these patches I no longer get any SKB allocation 
errors, even during the heaviest freezes while gitk is loading. I do still 
get (long) music skips during the freezes, but that's not unexpected.
AFAICT the wireless connection is stable.

Tested on top of current mainline git: v2.6.32-rc5-81-g964fe08.

Please add, if you feel it's appropriate, my:
Reported-and-tested-by: Frans Pop <elendil@planet.nl>

Cheers,
FJP

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug #14141] order 2 page allocation failures in iwlagn

[reinette chatre]

Hi Frans,

On Tue, 2009-10-27 at 04:10 -0700, Frans Pop wrote:
> Sorry for the delay in replying.
> 
> On Saturday 17 October 2009, reinette chatre wrote:
> > Prompted by this thread we are in process of moving allocation to paged
> > skb. This will definitely reduce the allocation size (from order 2 to
> > order 1) and hopefully help with this problem also. Could you please try
> > with the attached two patches? They are based on 2.6.32-rc4.
> 
> Looks very good! With these patches I no longer get any SKB allocation 
> errors, even during the heaviest freezes while gitk is loading. I do still 
> get (long) music skips during the freezes, but that's not unexpected.
> AFAICT the wireless connection is stable.
> 
> Tested on top of current mainline git: v2.6.32-rc5-81-g964fe08.
> 
> Please add, if you feel it's appropriate, my:
> Reported-and-tested-by: Frans Pop <elendil@planet.nl>

Thank you very much for testing these patches so thoroughly. They are
both on their way upstream already so I am not able to add your
signature at this time. Since these are pretty big changes these patches
will be in 2.6.33.

Reinette


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab() (was: Re: [Bug #14265] ifconfig: page allocation failure. order:5,ode:0x8020 w/ e100)

[Karol Lewandowski]

On Thu, Oct 01, 2009 at 09:56:04PM +0200, Rafael J. Wysocki wrote:
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14265
> Subject		: ifconfig: page allocation failure. order:5, mode:0x8020 w/ e100
> Submitter	: Karol Lewandowski <karol.k.lewandowski@gmail.com>
> Date		: 2009-09-15 12:05 (17 days old)
> References	: http://marc.info/?l=linux-kernel&m=125301636509517&w=4

Guys, could anyone check if patch below helps?  I think I've finally
found culprit of all allocation failures (but I might be wrong
too... ;-)

Thanks.


commit d6849591e042bceb66f1b4513a1df6740d2ad762
Author: Karol Lewandowski <karol.k.lewandowski@gmail.com>
Date:   Wed Oct 21 21:01:20 2009 +0200

    SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab()
    
    Commit ba52270d18fb17ce2cf176b35419dab1e43fe4a3 unconditionally
    cleared __GFP_NOFAIL flag on all allocations.
    
    Preserve this flag on second attempt to allocate page (with possibly
    decreased order).
    
    This should help with bugs #14265, #14141 and similar.
    
    Signed-off-by: Karol Lewandowski <karol.k.lewandowski@gmail.com>

diff --git a/mm/slub.c b/mm/slub.c
index b627675..ac5db65 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1084,7 +1084,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 {
 	struct page *page;
 	struct kmem_cache_order_objects oo = s->oo;
-	gfp_t alloc_gfp;
+	gfp_t alloc_gfp, nofail;
 
 	flags |= s->allocflags;
 
@@ -1092,6 +1092,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	 * Let the initial higher-order allocation fail under memory pressure
 	 * so we fall-back to the minimum order allocation.
 	 */
+	nofail = flags & __GFP_NOFAIL;
 	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
 
 	page = alloc_slab_page(alloc_gfp, node, oo);
@@ -1100,8 +1101,10 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 		/*
 		 * Allocation may have failed due to fragmentation.
 		 * Try a lower order alloc if possible
+		 *
+		 * Preserve __GFP_NOFAIL flag if previous allocation failed.
 		 */
-		page = alloc_slab_page(flags, node, oo);
+		page = alloc_slab_page(flags | nofail, node, oo);
 		if (!page)
 			return NULL;
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab() (was: Re: [Bug #14265] ifconfig: page allocation failure. order:5,ode:0x8020 w/ e100)

[David Rientjes]

On Wed, 21 Oct 2009, Karol Lewandowski wrote:

> commit d6849591e042bceb66f1b4513a1df6740d2ad762
> Author: Karol Lewandowski <karol.k.lewandowski@gmail.com>
> Date:   Wed Oct 21 21:01:20 2009 +0200
> 
>     SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab()
>     
>     Commit ba52270d18fb17ce2cf176b35419dab1e43fe4a3 unconditionally
>     cleared __GFP_NOFAIL flag on all allocations.
>     

No, it clears __GFP_NOFAIL from the first allocation of oo_order(s->oo).  
If that fails (and it's easy to fail, it has __GFP_NORETRY), another 
allocation is attempted with oo_order(s->min), for which __GFP_NOFAIL 
would be preserved if that's the slab cache's allocflags.

>     Preserve this flag on second attempt to allocate page (with possibly
>     decreased order).
>     
>     This should help with bugs #14265, #14141 and similar.
>     
>     Signed-off-by: Karol Lewandowski <karol.k.lewandowski@gmail.com>
> 
> diff --git a/mm/slub.c b/mm/slub.c
> index b627675..ac5db65 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -1084,7 +1084,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
>  {
>  	struct page *page;
>  	struct kmem_cache_order_objects oo = s->oo;
> -	gfp_t alloc_gfp;
> +	gfp_t alloc_gfp, nofail;
>  
>  	flags |= s->allocflags;
>  
> @@ -1092,6 +1092,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
>  	 * Let the initial higher-order allocation fail under memory pressure
>  	 * so we fall-back to the minimum order allocation.
>  	 */
> +	nofail = flags & __GFP_NOFAIL;
>  	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
>  
>  	page = alloc_slab_page(alloc_gfp, node, oo);
> @@ -1100,8 +1101,10 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
>  		/*
>  		 * Allocation may have failed due to fragmentation.
>  		 * Try a lower order alloc if possible
> +		 *
> +		 * Preserve __GFP_NOFAIL flag if previous allocation failed.
>  		 */
> -		page = alloc_slab_page(flags, node, oo);
> +		page = alloc_slab_page(flags | nofail, node, oo);
>  		if (!page)
>  			return NULL;
>  
> 

This does nothing.  You may have missed that the lower order allocation is 
passing 'flags' (which is a union of the gfp flags passed to 
allocate_slab() based on the allocation context and the cache's 
allocflags), and not alloc_gfp where __GFP_NOFAIL is masked.

Nack.

Note: slub isn't going to be a culprit in order 5 allocation failures 
since they have kmalloc passthrough to the page allocator.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab() (was: Re: [Bug #14265] ifconfig: page allocation failure. order:5,ode:0x8020 w/ e100)

[Karol Lewandowski]

On Wed, Oct 21, 2009 at 02:06:41PM -0700, David Rientjes wrote:
> On Wed, 21 Oct 2009, Karol Lewandowski wrote:
> 
> > commit d6849591e042bceb66f1b4513a1df6740d2ad762
> > Author: Karol Lewandowski <karol.k.lewandowski@gmail.com>
> > Date:   Wed Oct 21 21:01:20 2009 +0200
> > 
> >     SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab()
> >     
> >     Commit ba52270d18fb17ce2cf176b35419dab1e43fe4a3 unconditionally
> >     cleared __GFP_NOFAIL flag on all allocations.
> >     
> 
> No, it clears __GFP_NOFAIL from the first allocation of oo_order(s->oo).  
> If that fails (and it's easy to fail, it has __GFP_NORETRY), another 
> allocation is attempted with oo_order(s->min), for which __GFP_NOFAIL 
> would be preserved if that's the slab cache's allocflags.

Right, patch is junk.

However, I haven't been able to trigger failures since I've switched
to SLAB allocator.  That patch seemed related (and wrong), but it
wasn't.

> >  		 */
> > -		page = alloc_slab_page(flags, node, oo);
> > +		page = alloc_slab_page(flags | nofail, node, oo);
> >  		if (!page)
> >  			return NULL;
> >  
> > 
> 
> This does nothing.  You may have missed that the lower order allocation is 
> passing 'flags' (which is a union of the gfp flags passed to 
> allocate_slab() based on the allocation context and the cache's 
> allocflags), and not alloc_gfp where __GFP_NOFAIL is masked.

Right, I missed that.

> Nack.
> 
> Note: slub isn't going to be a culprit in order 5 allocation failures 
> since they have kmalloc passthrough to the page allocator.

However, it might change fragmentation somewhat I guess.  This might
make problem more/less visible.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab() (was: Re: [Bug #14265] ifconfig: page allocation failure. order:5,ode:0x8020 w/ e100)

[Mel Gorman]

On Wed, Oct 21, 2009 at 11:20:34PM +0200, Karol Lewandowski wrote:
> On Wed, Oct 21, 2009 at 02:06:41PM -0700, David Rientjes wrote:
> > On Wed, 21 Oct 2009, Karol Lewandowski wrote:
> > 
> > > commit d6849591e042bceb66f1b4513a1df6740d2ad762
> > > Author: Karol Lewandowski <karol.k.lewandowski@gmail.com>
> > > Date:   Wed Oct 21 21:01:20 2009 +0200
> > > 
> > >     SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab()
> > >     
> > >     Commit ba52270d18fb17ce2cf176b35419dab1e43fe4a3 unconditionally
> > >     cleared __GFP_NOFAIL flag on all allocations.
> > >     
> > 
> > No, it clears __GFP_NOFAIL from the first allocation of oo_order(s->oo).  
> > If that fails (and it's easy to fail, it has __GFP_NORETRY), another 
> > allocation is attempted with oo_order(s->min), for which __GFP_NOFAIL 
> > would be preserved if that's the slab cache's allocflags.
> 
> Right, patch is junk.
> 
> However, I haven't been able to trigger failures since I've switched
> to SLAB allocator.  That patch seemed related (and wrong), but it
> wasn't.
> 

Interesting. Pekka, I looked for SLUB commits in the 2.6.30..2.6.31
range for patches that might affect what order of pages SLUB allocates
but didn't spot anything obvious. Can you think of any changes that
might have altered how SLUB uses memory?

> > >  		 */
> > > -		page = alloc_slab_page(flags, node, oo);
> > > +		page = alloc_slab_page(flags | nofail, node, oo);
> > >  		if (!page)
> > >  			return NULL;
> > >  
> > > 
> > 
> > This does nothing.  You may have missed that the lower order allocation is 
> > passing 'flags' (which is a union of the gfp flags passed to 
> > allocate_slab() based on the allocation context and the cache's 
> > allocflags), and not alloc_gfp where __GFP_NOFAIL is masked.
> 
> Right, I missed that.
> 
> > Nack.
> > 
> > Note: slub isn't going to be a culprit in order 5 allocation failures 
> > since they have kmalloc passthrough to the page allocator.
> 
> However, it might change fragmentation somewhat I guess.  This might
> make problem more/less visible.
> 

Did you have CONFIG_KMEMCHECK set by any chance?

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] SLUB: Don't drop __GFP_NOFAIL completely from allocate_slab() (was: Re: [Bug #14265] ifconfig: page allocation failure. order:5,ode:0x8020 w/ e100)

[Karol Lewandowski]

On Thu, Oct 22, 2009 at 11:20:14AM +0100, Mel Gorman wrote:
> On Wed, Oct 21, 2009 at 11:20:34PM +0200, Karol Lewandowski wrote:
> > > Note: slub isn't going to be a culprit in order 5 allocation failures 
> > > since they have kmalloc passthrough to the page allocator.
> > 
> > However, it might change fragmentation somewhat I guess.  This might
> > make problem more/less visible.
> > 
> 
> Did you have CONFIG_KMEMCHECK set by any chance?

No, kmemcheck (and kmemleak) was always disabled.

It's likely that's possible to trigger allocation failures with slab,
I just haven't been successful at it.  Lack of good testcase is really
problem here -- even if I can't trigger failures I can never be sure
that these wont appear in some strange moment.

BTW I'll test your patches (from another thread) shortly.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>