2.4.24 + cryptoloop: __alloc_pages: 5-order allocation failed

2.4.24 + cryptoloop: __alloc_pages: 5-order allocation failed

[Christian Kujau]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hi,

today i played around with some benchmarks i do from time to time [1]
this time with 2.4.24 and the cryptoloop patch
(patch-cryptoloop-jari-2.4.22.0, applies to 2.4.24 too) when using this
machine (powerpc)

http://www.nerdbynature.de/bits/sheep/2.4.24-benh/ver_linux

in short: i have a 700 MB file, losetup a loop device with a cipher
(aes,cast5,cast6,blowfish,serpent,twofish), create an ext3 fs on it.
then test with "tiobench" (size is only about 100MB). but the benchmarks
did not last very long, i have these messages in my syslog:

Dec 21 23:42:45 sheep kernel: __alloc_pages: 0-order allocation failed
(gfp=0x1d2/0)
Dec 21 23:42:45 sheep kernel: VM: killing process nmbd
Dec 21 23:43:07 sheep kernel: __alloc_pages: 0-order allocation failed
(gfp=0x1d2/0)
[...]
Dec 23 22:26:37 sheep kernel: __alloc_pages: 5-order allocation failed
(gfp=0x20/0)
Dec 23 22:26:39 sheep kernel: __alloc_pages: 5-order allocation failed
(gfp=0x20/0)
Dec 23 22:26:40 sheep kernel: __alloc_pages: 5-order allocation failed
(gfp=0x20/0)
Dec 23 22:26:41 sheep kernel: __alloc_pages: 5-order allocation failed
(gfp=0x20/0)
[...]
Feb 16 15:17:21 sheep kernel: __alloc_pages: 0-order allocation failed
(gfp=0x30/0)

full log excerpt on:
http://www.nerdbynature.de/bits/sheep/2.4.24/messages-alloc_pages

i am then able to sysrq-S, but not all partitions can be synced, then i
try sysrq-E / -I / -U with no luck, nothing seems to be written/read
to/from the disk, sysrq-B works :-)

i did manage to make earlier tests with different filesystems [2] and
linux 2.4.22 and have never had these messages.

Thanks for your hints,
Christian.

[1] http://www.nerdbynature.de/bench/
[2] http://www.nerdbynature.de/bench/sheep/
- --
BOFH excuse #62:

need to wrap system in aluminum foil to fix problem
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFAMXeF+A7rjkF8z0wRAnpNAJ0evZDjZJdMorl60BfYCcogGWsYlwCgnU/B
ZICw1HmIv8Hnrh0D+W29l7A=
=o7Wu
-----END PGP SIGNATURE-----

Re: 2.4.24 + cryptoloop: __alloc_pages: 5-order allocation failed

[Marcelo Tosatti]

On Tue, 17 Feb 2004, Christian Kujau wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> hi,
>
> today i played around with some benchmarks i do from time to time [1]
> this time with 2.4.24 and the cryptoloop patch
> (patch-cryptoloop-jari-2.4.22.0, applies to 2.4.24 too) when using this
> machine (powerpc)
>
> http://www.nerdbynature.de/bits/sheep/2.4.24-benh/ver_linux
>
> in short: i have a 700 MB file, losetup a loop device with a cipher
> (aes,cast5,cast6,blowfish,serpent,twofish), create an ext3 fs on it.
> then test with "tiobench" (size is only about 100MB). but the benchmarks
> did not last very long, i have these messages in my syslog:
>
> Dec 21 23:42:45 sheep kernel: __alloc_pages: 0-order allocation failed
> (gfp=0x1d2/0)
> Dec 21 23:42:45 sheep kernel: VM: killing process nmbd
> Dec 21 23:43:07 sheep kernel: __alloc_pages: 0-order allocation failed
> (gfp=0x1d2/0)
> [...]
> Dec 23 22:26:37 sheep kernel: __alloc_pages: 5-order allocation failed
> (gfp=0x20/0)
> Dec 23 22:26:39 sheep kernel: __alloc_pages: 5-order allocation failed
> (gfp=0x20/0)
> Dec 23 22:26:40 sheep kernel: __alloc_pages: 5-order allocation failed
> (gfp=0x20/0)
> Dec 23 22:26:41 sheep kernel: __alloc_pages: 5-order allocation failed
> (gfp=0x20/0)
> [...]
> Feb 16 15:17:21 sheep kernel: __alloc_pages: 0-order allocation failed
> (gfp=0x30/0)
>
> full log excerpt on:
> http://www.nerdbynature.de/bits/sheep/2.4.24/messages-alloc_pages
>
> i am then able to sysrq-S, but not all partitions can be synced, then i
> try sysrq-E / -I / -U with no luck, nothing seems to be written/read
> to/from the disk, sysrq-B works :-)
>
> i did manage to make earlier tests with different filesystems [2] and
> linux 2.4.22 and have never had these messages.

Christian, Daniel,

Please do

echo 1 > /proc/sys/vm/vm_gfp_debug

This will print additional information when the "%d-order allocation
failures" messages happen. Please post that information along with the
System.map for the running kernel.

Daniel is seeing a similar problem but running with RAID1 over loop (which
doesnt sound like a good idea to me - but should work anyway).

I suspect the problem is loop allocates its buffers in a way so that the
allocator can't sleep waiting for IO, which in turn overloads the memory
and causes the failures.

Re: 2.4.24 + cryptoloop: __alloc_pages: 5-order allocation failed

[Christian Kujau]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marcelo Tosatti wrote:
|
| Please do
|
| echo 1 > /proc/sys/vm/vm_gfp_debug
|

unfortunately i am not able to do this right now, i'll have access to
the machine on sunday i guess. so i have to wait what it gives then.

| I suspect the problem is loop allocates its buffers in a way so that the
| allocator can't sleep waiting for IO, which in turn overloads the memory
| and causes the failures.

yes, i've heard rumors too that the loop device was not so good at all
when it comes to high i/o throughput.

Thanks,
Christian.
- --
BOFH excuse #63:

not properly grounded, please bury computer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFANGeh+A7rjkF8z0wRApHdAKDHxhC2WdWbSQGROyY0vhIl0pdTUwCg0+aV
UESKUigO8UARal6xi+NzqHQ=
=nwc6
-----END PGP SIGNATURE-----

Re: 2.4.24 + cryptoloop: __alloc_pages: 5-order allocation failed

[Christian]

[sorry, took me a while]
 > Christian, Daniel,
 >
 > Please do
 >
 > echo 1 > /proc/sys/vm/vm_gfp_debug

i did so, but it revealed not much (any?) more information:

Feb 25 03:05:17 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:05:17 sheep kernel: This architecture does not implement 
dump_stack()
Feb 25 03:05:27 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:05:27 sheep kernel: This architecture does not implement 
dump_stack()
Feb 25 03:05:27 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:05:27 sheep kernel: This architecture does not implement 
dump_stack()
Feb 25 03:05:27 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:05:27 sheep kernel: This architecture does not implement 
dump_stack()
Feb 25 03:06:17 sheep syslog-ng[324]: STATS: dropped 94
Feb 25 03:07:42 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:07:42 sheep kernel: This architecture does not implement 
dump_stack()
Feb 25 03:07:52 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:07:52 sheep kernel: This architecture does not implement 
dump_stack()
Feb 25 03:07:52 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:07:52 sheep kernel: This architecture does not implement 
dump_stack()
Feb 25 03:07:52 sheep kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x30/0)
Feb 25 03:07:52 sheep kernel: This architecture does not implement 
dump_stack()

for the record:

root@sheep:~# cat /proc/sys/vm/vm_gfp_debug
1
root@sheep:~#

some thoughts of mine:
after setting vm_gfp_debug to 1 i run the benchmarks again, with a 200MB 
  testfile (within a 700MB fs, setup via cryptoloop). in the log the 
messages from above showed up, but this time the benchmark was able to 
finish, the system was *never unusable* during the test (like i was 
experienceing with former kernels too). but: i don't know if i had 
another memory consuming app running last time. even if i did, i did not 
get any OOM messages, but i could think of higher memory- / vm-pressure 
last time....

oh, and just in case i forgot last time, this is the machine:
http://nerdbynature.de/bits/sheep/2.4.24-benh/ver_linux

(maybe this is why syslog says: "
Feb 25 03:07:52 sheep kernel: This architecture does not implement 
dump_stack()"?)

Thank you,
Christian.
-- 
BOFH excuse #141:

disks spinning backwards - toggle the hemisphere jumper.