[dm-crypt] system freezes when using swap that is encrypted

[dm-crypt] system freezes when using swap that is encrypted

[Nicolas Bock]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

Hello list,

I am using 2 primary FAT partitions for swap that I encrypt with cryptsetup
on boot with a random password from /dev/urandom. I noticed that once memory
pressure is high enough for the pager to start swapping, the system freezes.
It seems as if it's only the disk that freezes however, when I am logged
into a virtual console I can still use the keyboard to switch between
different consoles and the magic sysreq keys are working. I can ping the
machine but that's about it. Anything that seems to involve disk access like
logging in, Ctrl-C'ing the high memory job, switching to graphical console,
etc, does not work. I have tried to increase kernel logging with 'dmesg -n8'
but to no avail. I do not  see any error messages when this happens. I am
using  a vanilla kernel, version 2.6.34.5. Any pointers as to what could
this be caused by or how I could track this down are greatly appreciated.

nick

[-- Attachment #2: Type: text/html, Size: 963 bytes --]

Re: [dm-crypt] system freezes when using swap that is encrypted

[dave b]

I found that I had to turn swap off...

Re: [dm-crypt] system freezes when using swap that is encrypted

[Arno Wagner]

Id this is not a hard freeze, but only an extreme slowdown,
then it probably is one of the known issues with write-barrierss.
Fortunately, the kernel folks have finally stopped twiddeling 
their thumbs and have started to do something about it.

The problem arises when writing a lot of data to a slower
device (among other things). The slowdown by the encryption
can be enough to trigger the issue. Also, doing something
like a "cat /dev/zero > ttt" tyically causes these freezes
that can take minutes or longer.

2.6.36 is alrady significantly improved, but contains
only part of the intended changes. 2.6.37 is expected
to be still better. I would recomend to try with 
2.6.36 for the moment.

Arno

On Fri, Nov 12, 2010 at 08:17:48AM -0700, Nicolas Bock wrote:
> Hello list,
> 
> I am using 2 primary FAT partitions for swap that I encrypt with cryptsetup
> on boot with a random password from /dev/urandom. I noticed that once memory
> pressure is high enough for the pager to start swapping, the system freezes.
> It seems as if it's only the disk that freezes however, when I am logged
> into a virtual console I can still use the keyboard to switch between
> different consoles and the magic sysreq keys are working. I can ping the
> machine but that's about it. Anything that seems to involve disk access like
> logging in, Ctrl-C'ing the high memory job, switching to graphical console,
> etc, does not work. I have tried to increase kernel logging with 'dmesg -n8'
> but to no avail. I do not  see any error messages when this happens. I am
> using  a vanilla kernel, version 2.6.34.5. Any pointers as to what could
> this be caused by or how I could track this down are greatly appreciated.
> 
> nick

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] system freezes when using swap that is encrypted

[Nicolas Bock]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

Thanks for the replies. I tried 2.6.36 but the result is the same. Doesn't
look like the problem was fixed.

nick


On Fri, Nov 12, 2010 at 09:51, Arno Wagner <arno@wagner.name> wrote:

> Id this is not a hard freeze, but only an extreme slowdown,
> then it probably is one of the known issues with write-barrierss.
> Fortunately, the kernel folks have finally stopped twiddeling
> their thumbs and have started to do something about it.
>
> The problem arises when writing a lot of data to a slower
> device (among other things). The slowdown by the encryption
> can be enough to trigger the issue. Also, doing something
> like a "cat /dev/zero > ttt" tyically causes these freezes
> that can take minutes or longer.
>
> 2.6.36 is alrady significantly improved, but contains
> only part of the intended changes. 2.6.37 is expected
> to be still better. I would recomend to try with
> 2.6.36 for the moment.
>
> Arno
>
> On Fri, Nov 12, 2010 at 08:17:48AM -0700, Nicolas Bock wrote:
> > Hello list,
> >
> > I am using 2 primary FAT partitions for swap that I encrypt with
> cryptsetup
> > on boot with a random password from /dev/urandom. I noticed that once
> memory
> > pressure is high enough for the pager to start swapping, the system
> freezes.
> > It seems as if it's only the disk that freezes however, when I am logged
> > into a virtual console I can still use the keyboard to switch between
> > different consoles and the magic sysreq keys are working. I can ping the
> > machine but that's about it. Anything that seems to involve disk access
> like
> > logging in, Ctrl-C'ing the high memory job, switching to graphical
> console,
> > etc, does not work. I have tried to increase kernel logging with 'dmesg
> -n8'
> > but to no avail. I do not  see any error messages when this happens. I am
> > using  a vanilla kernel, version 2.6.34.5. Any pointers as to what could
> > this be caused by or how I could track this down are greatly appreciated.
> >
> > nick
>
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt@saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
>
>
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
> arno@wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25
> 338F
> ----
> Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
>
> If it's in the news, don't worry about it.  The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
>

[-- Attachment #2: Type: text/html, Size: 3524 bytes --]

Re: [dm-crypt] system freezes when using swap that is encrypted

[Nicolas Bock]

[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]

As an aside, would it make sense to go back in kernel versions to a time
before these write-barriers? Do I need those anyway? Where they introduced
recently?

Thanks, nick


On Fri, Nov 12, 2010 at 09:51, Arno Wagner <arno@wagner.name> wrote:

> Id this is not a hard freeze, but only an extreme slowdown,
> then it probably is one of the known issues with write-barrierss.
> Fortunately, the kernel folks have finally stopped twiddeling
> their thumbs and have started to do something about it.
>
> The problem arises when writing a lot of data to a slower
> device (among other things). The slowdown by the encryption
> can be enough to trigger the issue. Also, doing something
> like a "cat /dev/zero > ttt" tyically causes these freezes
> that can take minutes or longer.
>
> 2.6.36 is alrady significantly improved, but contains
> only part of the intended changes. 2.6.37 is expected
> to be still better. I would recomend to try with
> 2.6.36 for the moment.
>
> Arno
>
> On Fri, Nov 12, 2010 at 08:17:48AM -0700, Nicolas Bock wrote:
> > Hello list,
> >
> > I am using 2 primary FAT partitions for swap that I encrypt with
> cryptsetup
> > on boot with a random password from /dev/urandom. I noticed that once
> memory
> > pressure is high enough for the pager to start swapping, the system
> freezes.
> > It seems as if it's only the disk that freezes however, when I am logged
> > into a virtual console I can still use the keyboard to switch between
> > different consoles and the magic sysreq keys are working. I can ping the
> > machine but that's about it. Anything that seems to involve disk access
> like
> > logging in, Ctrl-C'ing the high memory job, switching to graphical
> console,
> > etc, does not work. I have tried to increase kernel logging with 'dmesg
> -n8'
> > but to no avail. I do not  see any error messages when this happens. I am
> > using  a vanilla kernel, version 2.6.34.5. Any pointers as to what could
> > this be caused by or how I could track this down are greatly appreciated.
> >
> > nick
>
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt@saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
>
>
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
> arno@wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25
> 338F
> ----
> Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
>
> If it's in the news, don't worry about it.  The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
>

[-- Attachment #2: Type: text/html, Size: 3578 bytes --]

Re: [dm-crypt] system freezes when using swap that is encrypted

[Milan Broz]

On 11/12/2010 08:32 PM, Nicolas Bock wrote:
> As an aside, would it make sense to go back in kernel versions to a time before these write-barriers? Do I need those anyway? Where they introduced recently?

I don't think that barriers should be problem here, but everything is possible:-)
How is that swap configured - through /etc/crypttab? (send dmsetup table; mount )

When it freezes, can you try to get full process backtrace and sent it to me?
Just run "echo t>/proc/sysrq-trigger" or activate sysrq and do it from keybuard (alt-sysrq+t),
(maybe serial console is needed here, but it should print status to syslog)

Milan

Re: [dm-crypt] system freezes when using swap that is encrypted

[Nicolas Bock]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

I think I have ruled out now that this is a hard crash. I have 8GB of
physical memory and another 8GB as swap. I allocated 10GB in a C
program and after it had written to about  7GB or so, the system
froze. But after maybe 20 minutes the system got unstuck and had
killed the job. Since I am on amd64, I am not sure why the job got
killed, it should certainly have had enough memory to write to. While
stuck I hit the sysrq backtrace button and I have attached the output
from the logs to this email. I hope it helps.

In the meantime it sounds like I will have to live without swap until
the kernel gets up to speed and manages to swap out in under 20
minutes....

nick



On Fri, Nov 12, 2010 at 14:01, Milan Broz <mbroz@redhat.com> wrote:
>
> On 11/12/2010 08:32 PM, Nicolas Bock wrote:
>> As an aside, would it make sense to go back in kernel versions to a time before these write-barriers? Do I need those anyway? Where they introduced recently?
>
> I don't think that barriers should be problem here, but everything is possible:-)
> How is that swap configured - through /etc/crypttab? (send dmsetup table; mount )
>
> When it freezes, can you try to get full process backtrace and sent it to me?
> Just run "echo t>/proc/sysrq-trigger" or activate sysrq and do it from keybuard (alt-sysrq+t),
> (maybe serial console is needed here, but it should print status to syslog)
>
> Milan
>
>
>

[-- Attachment #2: backtrace.txt --]
[-- Type: text/plain, Size: 52539 bytes --]

Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313] mysqld        S ffff880224ef6e00     0 16329      1 0x00000000
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313]  ffff880224ef6e00 0000000000000082 0000000000000003 ffff88022fda1040
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214a33fd8
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313]  ffff880224ef6e00 ffff880214a33fd8 ffff88022fda1040 ffffffff8102740d
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8102740d>] ? 0xffffffff8102740d
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810506bc>] ? 0xffffffff810506bc
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:19 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107cebc>] ? 0xffffffff8107cebc
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ebb0>] ? 0xffffffff8134ebb0
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350042>] ? 0xffffffff81350042
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8114a1cf>] ? 0xffffffff8114a1cf
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810ca36c>] ? 0xffffffff810ca36c
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313] mysqld        S 0000000000000003     0 16330      1 0x00000000
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  ffff8802254f6b40 0000000000000082 0000000000000000 ffff88022fcd94c0
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214a25fd8
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  ffff8802254f6b40 ffff880214a25fd8 0000000000000008 0000000000000008
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810841e8>] ? 0xffffffff810841e8
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff811bfc2f>] ? 0xffffffff811bfc2f
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81028921>] ? 0xffffffff81028921
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81028e3e>] ? 0xffffffff81028e3e
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350015>] ? 0xffffffff81350015
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8102e858>] ? 0xffffffff8102e858
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350015>] ? 0xffffffff81350015
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff811c4113>] ? 0xffffffff811c4113
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81047b9e>] ? 0xffffffff81047b9e
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313] mysqld        S 0000000000000002     0 16331      1 0x00000000
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  ffff880224cf6740 0000000000000082 119fba5400000000 ffff88022fcc9480
Nov 12 20:16:35 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214a73fd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880224cf6740 ffff880214a73fd8 119fba64119fba63 119fba66119fba65
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8102bb71>] ? 0xffffffff8102bb71
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] mysqld        S 0000000000000000     0 16336      1 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021deda180 0000000000000082 ffff880200000000 ffffffff81477020
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214abbfd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021deda180 ffff880214abbfd8 000000000000e5d0 00000000810475b1
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134f46e>] ? 0xffffffff8134f46e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81046e27>] ? 0xffffffff81046e27
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b9e0b>] ? 0xffffffff810b9e0b
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b9f7e>] ? 0xffffffff810b9f7e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810baa48>] ? 0xffffffff810baa48
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810baf58>] ? 0xffffffff810baf58
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff813505ce>] ? 0xffffffff813505ce
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100324e>] ? 0xffffffff8100324e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810bac58>] ? 0xffffffff810bac58
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107b386>] ? 0xffffffff8107b386
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100324e>] ? 0xffffffff8100324e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8101a898>] ? 0xffffffff8101a898
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8104ba02>] ? 0xffffffff8104ba02
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8104ba02>] ? 0xffffffff8104ba02
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810baf2f>] ? 0xffffffff810baf2f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] mysqld        D 0000000000000003     0 16337      1 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880227e03900 0000000000000082 00000010bcd4a4d0 ffff88022fcf0840
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214abdfd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880227e03900 ffff880214abdfd8 00000000ffffffff ffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350281>] ? 0xffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134e4cc>] ? 0xffffffff8134e4cc
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103b426>] ? 0xffffffff8103b426
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134d855>] ? 0xffffffff8134d855
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107e664>] ? 0xffffffff8107e664
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81044607>] ? 0xffffffff81044607
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810cff06>] ? 0xffffffff810cff06
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c3ca>] ? 0xffffffff8109c3ca
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810d0007>] ? 0xffffffff810d0007
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c298>] ? 0xffffffff8109c298
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c36a>] ? 0xffffffff8109c36a
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810859d9>] ? 0xffffffff810859d9
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81086546>] ? 0xffffffff81086546
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810a88ed>] ? 0xffffffff810a88ed
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81085c5e>] ? 0xffffffff81085c5e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81086aec>] ? 0xffffffff81086aec
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810870d5>] ? 0xffffffff810870d5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81087ccf>] ? 0xffffffff81087ccf
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81087fab>] ? 0xffffffff81087fab
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81085b80>] ? 0xffffffff81085b80
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81081f77>] ? 0xffffffff81081f77
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108379f>] ? 0xffffffff8108379f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810838ce>] ? 0xffffffff810838ce
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107d40b>] ? 0xffffffff8107d40b
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108e609>] ? 0xffffffff8108e609
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108fcac>] ? 0xffffffff8108fcac
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff811c73b4>] ? 0xffffffff811c73b4
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100324e>] ? 0xffffffff8100324e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8101e83e>] ? 0xffffffff8101e83e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8104ba02>] ? 0xffffffff8104ba02
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b9dc8>] ? 0xffffffff810b9dc8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8135080f>] ? 0xffffffff8135080f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] mysqld        D ffff880224ec6d80     0 16338      1 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880224ec6d80 0000000000000082 00000000bcd4aa28 ffff8802175bcdc0
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214abffd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880224ec6d80 ffff880214abffd8 ffff8801bcd20f00 ffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350281>] ? 0xffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134e4cc>] ? 0xffffffff8134e4cc
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103b426>] ? 0xffffffff8103b426
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134d855>] ? 0xffffffff8134d855
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107e664>] ? 0xffffffff8107e664
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81044607>] ? 0xffffffff81044607
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810cff06>] ? 0xffffffff810cff06
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c3ca>] ? 0xffffffff8109c3ca
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810d0007>] ? 0xffffffff810d0007
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c298>] ? 0xffffffff8109c298
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c36a>] ? 0xffffffff8109c36a
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810859d9>] ? 0xffffffff810859d9
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81086546>] ? 0xffffffff81086546
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810a88ed>] ? 0xffffffff810a88ed
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81085c5e>] ? 0xffffffff81085c5e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81086aec>] ? 0xffffffff81086aec
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810870d5>] ? 0xffffffff810870d5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81087ccf>] ? 0xffffffff81087ccf
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81087fab>] ? 0xffffffff81087fab
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81085b80>] ? 0xffffffff81085b80
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81081f77>] ? 0xffffffff81081f77
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108379f>] ? 0xffffffff8108379f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810838ce>] ? 0xffffffff810838ce
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107d40b>] ? 0xffffffff8107d40b
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108e609>] ? 0xffffffff8108e609
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108fcac>] ? 0xffffffff8108fcac
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100324e>] ? 0xffffffff8100324e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8101e83e>] ? 0xffffffff8101e83e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100324e>] ? 0xffffffff8100324e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8104ba02>] ? 0xffffffff8104ba02
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b9dc8>] ? 0xffffffff810b9dc8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8135080f>] ? 0xffffffff8135080f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] mysqld        S 0000000000000002     0 16339      1 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021dedae80 0000000000000082 ffff880200000000 ffff88022fcc9480
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214a3bfd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021dedae80 ffff880214a3bfd8 00000001ffffffff 0000000000000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8101e827>] ? 0xffffffff8101e827
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8104ba02>] ? 0xffffffff8104ba02
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b9dc8>] ? 0xffffffff810b9dc8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] mysqld        S 0000000000000003     0 16341      1 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff8802256638c0 0000000000000082 0000000000000000 ffff88022fcd94c0
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214ab9fd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff8802256638c0 ffff880214ab9fd8 0000000000000002 ffffffff81052efe
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81052efe>] ? 0xffffffff81052efe
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134e368>] ? 0xffffffff8134e368
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103cf6a>] ? 0xffffffff8103cf6a
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103d144>] ? 0xffffffff8103d144
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350042>] ? 0xffffffff81350042
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c02d>] ? 0xffffffff8103c02d
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c240>] ? 0xffffffff8103c240
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] sshd          S 0000000000000000     0 16396      1 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021e2e8f80 0000000000000086 ffff880200000000 ffffffff81477020
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214a49fd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021e2e8f80 ffff880214a49fd8 0000000000000000 ffff880214a499d8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100342e>] ? 0xffffffff8100342e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134f3d0>] ? 0xffffffff8134f3d0
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100342e>] ? 0xffffffff8100342e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350281>] ? 0xffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350281>] ? 0xffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b9f7e>] ? 0xffffffff810b9f7e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810baa48>] ? 0xffffffff810baa48
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810baf58>] ? 0xffffffff810baf58
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810bb028>] ? 0xffffffff810bb028
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810bb028>] ? 0xffffffff810bb028
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810bb028>] ? 0xffffffff810bb028
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff811c4cbb>] ? 0xffffffff811c4cbb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810a7800>] ? 0xffffffff810a7800
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810a9f58>] ? 0xffffffff810a9f58
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff811c677c>] ? 0xffffffff811c677c
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8102e584>] ? 0xffffffff8102e584
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810bcde6>] ? 0xffffffff810bcde6
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810bce1c>] ? 0xffffffff810bce1c
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810bac58>] ? 0xffffffff810bac58
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81034305>] ? 0xffffffff81034305
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350281>] ? 0xffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8104482e>] ? 0xffffffff8104482e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350015>] ? 0xffffffff81350015
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103485b>] ? 0xffffffff8103485b
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810baf2f>] ? 0xffffffff810baf2f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] apache2       D ffff88022632a7c0     0 16454      1 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88022632a7c0 0000000000000082 0000001000000001 ffff88022501ac40
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214a83fd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88022632a7c0 ffff880214a83fd8 00000000ffffffff ffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350281>] ? 0xffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134e4cc>] ? 0xffffffff8134e4cc
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103b426>] ? 0xffffffff8103b426
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134d855>] ? 0xffffffff8134d855
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107e664>] ? 0xffffffff8107e664
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81044607>] ? 0xffffffff81044607
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810cff06>] ? 0xffffffff810cff06
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c3ca>] ? 0xffffffff8109c3ca
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810d0007>] ? 0xffffffff810d0007
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c298>] ? 0xffffffff8109c298
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8109c36a>] ? 0xffffffff8109c36a
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810859d9>] ? 0xffffffff810859d9
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81086546>] ? 0xffffffff81086546
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810a88ed>] ? 0xffffffff810a88ed
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81085c5e>] ? 0xffffffff81085c5e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81086aec>] ? 0xffffffff81086aec
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810870d5>] ? 0xffffffff810870d5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81087ccf>] ? 0xffffffff81087ccf
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81087fab>] ? 0xffffffff81087fab
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81085b80>] ? 0xffffffff81085b80
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81081f77>] ? 0xffffffff81081f77
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108379f>] ? 0xffffffff8108379f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810838ce>] ? 0xffffffff810838ce
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8107d40b>] ? 0xffffffff8107d40b
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108e609>] ? 0xffffffff8108e609
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108fcac>] ? 0xffffffff8108fcac
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8100324e>] ? 0xffffffff8100324e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8101e83e>] ? 0xffffffff8101e83e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103485b>] ? 0xffffffff8103485b
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8104ba02>] ? 0xffffffff8104ba02
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b9dc8>] ? 0xffffffff810b9dc8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8135080f>] ? 0xffffffff8135080f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] apache2       S 0000000000000002     0 16460  16454 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88022637d580 0000000000000082 ffffffff00000000 ffff88022fcc9480
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880214a2ffd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88022637d580 ffff880214a2ffd8 ffff88022fc28200 ffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350281>] ? 0xffffffff81350281
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b36cf>] ? 0xffffffff810b36cf
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81044607>] ? 0xffffffff81044607
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810b3ef3>] ? 0xffffffff810b3ef3
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8108fcac>] ? 0xffffffff8108fcac
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810acccc>] ? 0xffffffff810acccc
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81350015>] ? 0xffffffff81350015
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81031135>] ? 0xffffffff81031135
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c49f>] ? 0xffffffff8103c49f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810ad6ae>] ? 0xffffffff810ad6ae
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810ad807>] ? 0xffffffff810ad807
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] apache2       S ffff880227e0d940     0 16463  16454 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880227e0d940 0000000000000082 0000000000000001 ffff88022632a140
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880219017fd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880227e0d940 ffff880219017fd8 ffffffff810032ae 0000000000000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810032ae>] ? 0xffffffff810032ae
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105158b>] ? 0xffffffff8105158b
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8101e827>] ? 0xffffffff8101e827
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c49f>] ? 0xffffffff8103c49f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c564>] ? 0xffffffff8103c564
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] apache2       S 0000000000000002     0 16464  16454 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88022632a140 0000000000000082 0000000000000000 ffff88022fcc9480
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff880219019fd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88022632a140 ffff880219019fd8 0000000800010064 0000000000000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c49f>] ? 0xffffffff8103c49f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c564>] ? 0xffffffff8103c564
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] apache2       S 0000000000000002     0 16465  16454 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff8802263fcfc0 0000000000000082 0000000000000000 ffff88022fcc9480
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff88021901bfd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff8802263fcfc0 ffff88021901bfd8 0000000000000064 0000000000000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c49f>] ? 0xffffffff8103c49f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c564>] ? 0xffffffff8103c564
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] apache2       S 0000000000000002     0 16466  16454 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021de7b8c0 0000000000000082 0000000000000000 ffff88022fcc9480
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff88021901dfd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff88021de7b8c0 ffff88021901dfd8 0000000000000000 1230800112308000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c49f>] ? 0xffffffff8103c49f
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81053078>] ? 0xffffffff81053078
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c564>] ? 0xffffffff8103c564
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810028eb>] ? 0xffffffff810028eb
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] apache2       S 0000000000000002     0 16467  16454 0x00000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880225663240 0000000000000082 0000000000000000 ffff88022fcc9480
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  0000000000011f40 0000000000011f40 0000000000004000 ffff88021901ffd8
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  ffff880225663240 ffff88021901ffd8 0000000000000000 0000000000000000
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff81050954>] ? 0xffffffff81050954
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff810517c2>] ? 0xffffffff810517c2
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8105275e>] ? 0xffffffff8105275e
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21092.637313]  [<ffffffff8103c49f>] ? 0xffffffff8103c49f
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621161] console-kit-dae invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621166] console-kit-dae cpuset=/ mems_allowed=0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621169] Pid: 16376, comm: console-kit-dae Tainted: P           2.6.34.5-mujo #2
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621171] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621176]  [<ffffffff81350042>] ? 0xffffffff81350042
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621178]  [<ffffffff8107ea44>] ? 0xffffffff8107ea44
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621180]  [<ffffffff81081b9b>] ? 0xffffffff81081b9b
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621182]  [<ffffffff81350015>] ? 0xffffffff81350015
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621184]  [<ffffffff8107ebd2>] ? 0xffffffff8107ebd2
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621186]  [<ffffffff8107f1e5>] ? 0xffffffff8107f1e5
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621188]  [<ffffffff8107f27c>] ? 0xffffffff8107f27c
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621190]  [<ffffffff81082065>] ? 0xffffffff81082065
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621193]  [<ffffffff8108379f>] ? 0xffffffff8108379f
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621195]  [<ffffffff810838ce>] ? 0xffffffff810838ce
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621197]  [<ffffffff8107d40b>] ? 0xffffffff8107d40b
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621199]  [<ffffffff8108e609>] ? 0xffffffff8108e609
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621201]  [<ffffffff8107b386>] ? 0xffffffff8107b386
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621203]  [<ffffffff8122891e>] ? 0xffffffff8122891e
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621205]  [<ffffffff8108fcac>] ? 0xffffffff8108fcac
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621207]  [<ffffffff81028bd3>] ? 0xffffffff81028bd3
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621210]  [<ffffffff8101e83e>] ? 0xffffffff8101e83e
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621212]  [<ffffffff810b8f92>] ? 0xffffffff810b8f92
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621214]  [<ffffffff810b947c>] ? 0xffffffff810b947c
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621216]  [<ffffffff810adcfa>] ? 0xffffffff810adcfa
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621218]  [<ffffffff8135080f>] ? 0xffffffff8135080f
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621220] Mem-Info:
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621222] DMA per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621224] CPU    0: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621226] CPU    1: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621228] CPU    2: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621230] CPU    3: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621231] DMA32 per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621233] CPU    0: hi:  186, btch:  31 usd: 179
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621235] CPU    1: hi:  186, btch:  31 usd: 172
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621236] CPU    2: hi:  186, btch:  31 usd: 178
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621238] CPU    3: hi:  186, btch:  31 usd: 190
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621240] Normal per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621241] CPU    0: hi:  186, btch:  31 usd: 204
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621243] CPU    1: hi:  186, btch:  31 usd: 152
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621245] CPU    2: hi:  186, btch:  31 usd: 165
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621247] CPU    3: hi:  186, btch:  31 usd: 161
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621251] active_anon:1631724 inactive_anon:294103 isolated_anon:608
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621252]  active_file:66 inactive_file:57 isolated_file:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621252]  unevictable:0 dirty:0 writeback:294476 unstable:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621253]  free:10925 slab_reclaimable:1726 slab_unreclaimable:85862
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621254]  mapped:41 shmem:6 pagetables:6861 bounce:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621260] DMA free:15864kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15764kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621265] lowmem_reserve[]: 0 3255 8052 8052
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621273] DMA32 free:22500kB min:4636kB low:5792kB high:6952kB active_anon:2556464kB inactive_anon:513056kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):256kB isolated(file):0kB present:3333216kB mlocked:0kB dirty:0kB writeback:513528kB mapped:0kB shmem:0kB slab_reclaimable:396kB slab_unreclaimable:83764kB kernel_stack:8kB pagetables:7392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3708624 all_unreclaimable? no
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621278] lowmem_reserve[]: 0 0 4797 4797
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621285] Normal free:5336kB min:6832kB low:8540kB high:10248kB active_anon:3970432kB inactive_anon:663356kB active_file:232kB inactive_file:228kB unevictable:0kB isolated(anon):2176kB isolated(file):0kB present:4912640kB mlocked:0kB dirty:0kB writeback:664376kB mapped:164kB shmem:24kB slab_reclaimable:6508kB slab_unreclaimable:259684kB kernel_stack:2648kB pagetables:20052kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1323162 all_unreclaimable? no
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621291] lowmem_reserve[]: 0 0 0 0
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621294] DMA: 2*4kB 2*8kB 2*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15864kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621302] DMA32: 797*4kB 522*8kB 310*16kB 126*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 22500kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621310] Normal: 212*4kB 21*8kB 6*16kB 4*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5336kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621318] 294666 total pagecache pages
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621320] 294537 pages in swap cache
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621321] Swap cache stats: add 858701, delete 564164, find 115/160
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621323] Free swap  = 4583140kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.621324] Total swap = 8016424kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.655031] 2097151 pages RAM
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.655033] 50658 pages reserved
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.655035] 951 pages shared
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.655036] 1997374 pages non-shared
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.655039] Out of memory: kill process 15761 (bash) score 1315984 or a child
Nov 12 20:16:36 src@mujo/mujo kernel: [21438.655042] Killed process 16375 (a.out) vsz:10489500kB, anon-rss:6493524kB, file-rss:4kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064019] a.out: page allocation failure. order:0, mode:0x280da
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064024] Pid: 16375, comm: a.out Tainted: P           2.6.34.5-mujo #2
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064026] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064031]  [<ffffffff81082112>] ? 0xffffffff81082112
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064033]  [<ffffffff8108fb35>] ? 0xffffffff8108fb35
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064036]  [<ffffffff810032ee>] ? 0xffffffff810032ee
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064038]  [<ffffffff8101e83e>] ? 0xffffffff8101e83e
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064040]  [<ffffffff8107712d>] ? 0xffffffff8107712d
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064042]  [<ffffffff8134ffe5>] ? 0xffffffff8134ffe5
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064044]  [<ffffffff8102b5a4>] ? 0xffffffff8102b5a4
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064046]  [<ffffffff8134e17f>] ? 0xffffffff8134e17f
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064048]  [<ffffffff8135080f>] ? 0xffffffff8135080f
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064050] Mem-Info:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064051] DMA per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064053] CPU    0: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064055] CPU    1: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064057] CPU    2: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064059] CPU    3: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064060] DMA32 per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064062] CPU    0: hi:  186, btch:  31 usd: 180
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064064] CPU    1: hi:  186, btch:  31 usd: 160
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064065] CPU    2: hi:  186, btch:  31 usd: 197
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064067] CPU    3: hi:  186, btch:  31 usd: 184
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064068] Normal per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064070] CPU    0: hi:  186, btch:  31 usd: 191
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064072] CPU    1: hi:  186, btch:  31 usd: 157
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064074] CPU    2: hi:  186, btch:  31 usd: 179
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064076] CPU    3: hi:  186, btch:  31 usd: 169
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064080] active_anon:1631724 inactive_anon:294275 isolated_anon:416
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064081]  active_file:123 inactive_file:0 isolated_file:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064082]  unevictable:0 dirty:0 writeback:294666 unstable:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064083]  free:11018 slab_reclaimable:1726 slab_unreclaimable:85898
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064084]  mapped:41 shmem:6 pagetables:6861 bounce:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064090] DMA free:15864kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15764kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064094] lowmem_reserve[]: 0 3255 8052 8052
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064102] DMA32 free:22748kB min:4636kB low:5792kB high:6952kB active_anon:2556464kB inactive_anon:512876kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:3333216kB mlocked:0kB dirty:0kB writeback:513312kB mapped:0kB shmem:0kB slab_reclaimable:396kB slab_unreclaimable:83804kB kernel_stack:8kB pagetables:7392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5557824 all_unreclaimable? yes
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064107] lowmem_reserve[]: 0 0 4797 4797
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064114] Normal free:5460kB min:6832kB low:8540kB high:10248kB active_anon:3970432kB inactive_anon:664224kB active_file:460kB inactive_file:0kB unevictable:0kB isolated(anon):1280kB isolated(file):0kB present:4912640kB mlocked:0kB dirty:0kB writeback:665352kB mapped:164kB shmem:24kB slab_reclaimable:6508kB slab_unreclaimable:259788kB kernel_stack:2648kB pagetables:20052kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:9861963 all_unreclaimable? yes
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064119] lowmem_reserve[]: 0 0 0 0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064123] DMA: 2*4kB 2*8kB 2*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15864kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064131] DMA32: 795*4kB 526*8kB 314*16kB 127*32kB 4*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 22748kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064139] Normal: 261*4kB 20*8kB 2*16kB 4*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5460kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064148] 294839 total pagecache pages
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064149] 294710 pages in swap cache
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064151] Swap cache stats: add 859019, delete 564309, find 115/160
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064153] Free swap  = 4581868kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.064154] Total swap = 8016424kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.209827] 2097151 pages RAM
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.209830] 50658 pages reserved
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.209831] 583 pages shared
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.209832] 1997633 pages non-shared
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210426] mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210431] mysqld cpuset=/ mems_allowed=0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210434] Pid: 16337, comm: mysqld Tainted: P           2.6.34.5-mujo #2
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210436] Call Trace:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210440]  [<ffffffff81350042>] ? 0xffffffff81350042
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210443]  [<ffffffff8107ea44>] ? 0xffffffff8107ea44
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210445]  [<ffffffff81081b9b>] ? 0xffffffff81081b9b
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210447]  [<ffffffff81350015>] ? 0xffffffff81350015
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210449]  [<ffffffff8107ebd2>] ? 0xffffffff8107ebd2
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210451]  [<ffffffff8107f1e5>] ? 0xffffffff8107f1e5
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210453]  [<ffffffff8107f27c>] ? 0xffffffff8107f27c
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210455]  [<ffffffff81082065>] ? 0xffffffff81082065
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210457]  [<ffffffff8108379f>] ? 0xffffffff8108379f
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210460]  [<ffffffff810838ce>] ? 0xffffffff810838ce
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210462]  [<ffffffff8107d40b>] ? 0xffffffff8107d40b
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210464]  [<ffffffff8108e609>] ? 0xffffffff8108e609
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210466]  [<ffffffff8108fcac>] ? 0xffffffff8108fcac
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210468]  [<ffffffff811c73b4>] ? 0xffffffff811c73b4
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210470]  [<ffffffff8100324e>] ? 0xffffffff8100324e
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210472]  [<ffffffff8101e83e>] ? 0xffffffff8101e83e
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210474]  [<ffffffff8104ba02>] ? 0xffffffff8104ba02
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210476]  [<ffffffff810b9dc8>] ? 0xffffffff810b9dc8
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210479]  [<ffffffff8135080f>] ? 0xffffffff8135080f
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210481] Mem-Info:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210483] DMA per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210485] CPU    0: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210487] CPU    1: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210488] CPU    2: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210490] CPU    3: hi:    0, btch:   1 usd:   0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210492] DMA32 per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210494] CPU    0: hi:  186, btch:  31 usd: 180
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210495] CPU    1: hi:  186, btch:  31 usd: 160
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210497] CPU    2: hi:  186, btch:  31 usd: 197
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210499] CPU    3: hi:  186, btch:  31 usd: 184
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210500] Normal per-cpu:
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210502] CPU    0: hi:  186, btch:  31 usd: 191
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210504] CPU    1: hi:  186, btch:  31 usd: 157
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210506] CPU    2: hi:  186, btch:  31 usd: 179
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210508] CPU    3: hi:  186, btch:  31 usd: 170
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210512] active_anon:1631724 inactive_anon:294334 isolated_anon:352
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210513]  active_file:123 inactive_file:0 isolated_file:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210513]  unevictable:0 dirty:0 writeback:294666 unstable:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210514]  free:11018 slab_reclaimable:1726 slab_unreclaimable:85898
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210515]  mapped:41 shmem:6 pagetables:6861 bounce:0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210521] DMA free:15864kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15764kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210526] lowmem_reserve[]: 0 3255 8052 8052
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210533] DMA32 free:22748kB min:4636kB low:5792kB high:6952kB active_anon:2556464kB inactive_anon:512784kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:3333216kB mlocked:0kB dirty:0kB writeback:513312kB mapped:0kB shmem:0kB slab_reclaimable:396kB slab_unreclaimable:83804kB kernel_stack:8kB pagetables:7392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5695904 all_unreclaimable? yes
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210538] lowmem_reserve[]: 0 0 4797 4797
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210546] Normal free:5460kB min:6832kB low:8540kB high:10248kB active_anon:3970432kB inactive_anon:664552kB active_file:460kB inactive_file:0kB unevictable:0kB isolated(anon):1024kB isolated(file):0kB present:4912640kB mlocked:0kB dirty:0kB writeback:665352kB mapped:164kB shmem:24kB slab_reclaimable:6508kB slab_unreclaimable:259788kB kernel_stack:2648kB pagetables:20052kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10087179 all_unreclaimable? yes
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210551] lowmem_reserve[]: 0 0 0 0
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210554] DMA: 2*4kB 2*8kB 2*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15864kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210562] DMA32: 795*4kB 526*8kB 314*16kB 127*32kB 4*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 22748kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210570] Normal: 261*4kB 20*8kB 2*16kB 4*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5460kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210579] 294839 total pagecache pages
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210580] 294710 pages in swap cache
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210582] Swap cache stats: add 859019, delete 564309, find 115/160
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210584] Free swap  = 4606024kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.210585] Total swap = 8016424kB
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.245080] 2097151 pages RAM
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.245082] 50658 pages reserved
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.245083] 566 pages shared
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.245084] 1997649 pages non-shared
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.245087] Out of memory: kill process 15716 (gdmgreeter) score 72293 or a child
Nov 12 20:16:36 src@mujo/mujo kernel: [21445.245091] Killed process 15716 (gdmgreeter) vsz:289172kB, anon-rss:1156kB, file-rss:0kB

Re: [dm-crypt] system freezes when using swap that is encrypted

[Arno Wagner]

On Fri, Nov 12, 2010 at 08:40:43PM -0700, Nicolas Bock wrote:
> I think I have ruled out now that this is a hard crash. I have 8GB of
> physical memory and another 8GB as swap. I allocated 10GB in a C
> program and after it had written to about  7GB or so, the system
> froze. But after maybe 20 minutes the system got unstuck and had
> killed the job. Since I am on amd64, I am not sure why the job got
> killed, it should certainly have had enough memory to write to. 

I don't know what criteria lead to activation of the OOM killer,
but I use one application that allocates up to 150 MB of memory,
but never uses (accesses) it. This app runs fine in a qemu
with 16MB RAM and 200MB swap. It is possible that your process
gets killed, if it uses more memory than the larger of swap 
and main memory.

Arno


-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] system freezes when using swap that is encrypted

[Heinz Diehl]

On 13.11.2010, Arno Wagner wrote: 

> I don't know what criteria lead to activation of the OOM killer,
[....]

How do you know it gets OOM-killed?

As a workaround, one could have set "vm.overcommit_memory = 1"
in /etc/sysctl.conf in this case..

Re: [dm-crypt] system freezes when using swap that is encrypted

[Heinz Diehl]

On 13.11.2010, Heinz Diehl wrote: 

> How do you know it gets OOM-killed?

Ok, I see..

Re: [dm-crypt] system freezes when using swap that is encrypted

[Arno Wagner]

On Sat, Nov 13, 2010 at 06:01:09PM +0100, Heinz Diehl wrote:
> On 13.11.2010, Arno Wagner wrote: 
> 
> > I don't know what criteria lead to activation of the OOM killer,
> [....]
> 
> How do you know it gets OOM-killed?

I don't. By Occams Razor it is just my first suspected cause.

Reason is that (if I understand correctly) all the memory
has been allocated before the crash, so a NULL response
from malloc is not the cause. But other than an unrelated
program crash, that basically leaves the OOM killer.

AFAIK shell (vm) limits block the allocation but do
not SIGTERM/SIGKILL the process.

Arno

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] system freezes when using swap that is encrypted

[dave b]

By the way if you bother to wait long enough the system will often
'unfreeze' as the oom killer awakes up!

Re: [dm-crypt] system freezes when using swap that is encrypted

[Nicolas Bock]

The interesting thing is though that when I make the 2 swap partitions
regular swap, i.e. not encrypted, then the same program does not get
OOM killed and the system stays responsive all the way to the end of
allocating the 10GB. So it's not a problem of memory pressure, it's
rather related to the fact that the swap is encrypted.

nick

On Sat, Nov 13, 2010 at 07:53, Arno Wagner <arno@wagner.name> wrote:
> On Fri, Nov 12, 2010 at 08:40:43PM -0700, Nicolas Bock wrote:
>> I think I have ruled out now that this is a hard crash. I have 8GB of
>> physical memory and another 8GB as swap. I allocated 10GB in a C
>> program and after it had written to about  7GB or so, the system
>> froze. But after maybe 20 minutes the system got unstuck and had
>> killed the job. Since I am on amd64, I am not sure why the job got
>> killed, it should certainly have had enough memory to write to.
>
> I don't know what criteria lead to activation of the OOM killer,
> but I use one application that allocates up to 150 MB of memory,
> but never uses (accesses) it. This app runs fine in a qemu
> with 16MB RAM and 200MB swap. It is possible that your process
> gets killed, if it uses more memory than the larger of swap
> and main memory.
>
> Arno
>
>
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
> ----
> Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
>
> If it's in the news, don't worry about it.  The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
>

Re: [dm-crypt] system freezes when using swap that is encrypted

[Carlo Wood]

Hi all!

I've been a member of this list for a long long time, and
what-do-you-know, I finally see something that I can respond to :)

I have the following problem for a long time now: whenever my
system uses all memory (and would start to use swap) it freezes.
I can't even remote login anymore - although there remains a
desktop image on the monitor. Only possible option is to reboot.

I use the following configuration for my swap:

/etc/fstab:

/dev/mapper/cswap none          swap    sw,pri=1        0       0


/etc/crypttab:

cswap       /dev/md1        /dev/random swap


/dev/md1 is a 16 GB partition on my RAID 5 with three disks.

Could this be related?

On Sat, Nov 13, 2010 at 11:49:53AM -0700, Nicolas Bock wrote:
> The interesting thing is though that when I make the 2 swap partitions
> regular swap, i.e. not encrypted, then the same program does not get
> OOM killed and the system stays responsive all the way to the end of
> allocating the 10GB. So it's not a problem of memory pressure, it's
> rather related to the fact that the swap is encrypted.
> 
> nick
> 
> On Sat, Nov 13, 2010 at 07:53, Arno Wagner <arno@wagner.name> wrote:
> > On Fri, Nov 12, 2010 at 08:40:43PM -0700, Nicolas Bock wrote:
> >> I think I have ruled out now that this is a hard crash. I have 8GB of
> >> physical memory and another 8GB as swap. I allocated 10GB in a C
> >> program and after it had written to about  7GB or so, the system
> >> froze. But after maybe 20 minutes the system got unstuck and had
> >> killed the job. Since I am on amd64, I am not sure why the job got
> >> killed, it should certainly have had enough memory to write to.
> >
> > I don't know what criteria lead to activation of the OOM killer,
> > but I use one application that allocates up to 150 MB of memory,
> > but never uses (accesses) it. This app runs fine in a qemu
> > with 16MB RAM and 200MB swap. It is possible that your process
> > gets killed, if it uses more memory than the larger of swap
> > and main memory.
> >
> > Arno
> >
> >
> > --
> > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name
> > GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
> > ----
> > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
> >
> > If it's in the news, don't worry about it.  The very definition of
> > "news" is "something that hardly ever happens." -- Bruce Schneier

-- 
Carlo Wood <carlo@alinoe.com>

Re: [dm-crypt] system freezes when using swap that is encrypted

[Milan Broz]

On 11/14/2010 01:35 AM, Carlo Wood wrote:
> I use the following configuration for my swap:
> 
> /etc/fstab:
> 
> /dev/mapper/cswap none          swap    sw,pri=1        0       0
> 
> /etc/crypttab:
> 
> cswap       /dev/md1        /dev/random swap
> 
> /dev/md1 is a 16 GB partition on my RAID 5 with three disks.

Stacking devices for swap is not good idea in principle.

It should work in theory but if there is any allocation which
does not use reserved memory pools, it ends in deadlock
(waiting for itself to free some memory by swapping).

Try to set swap to individual device - if it works,
the problem is in MD RAID layer if not, it can be similar
to problem reported before.

dm-crypt uses own reserved bisosets and memory pools so
I expect that problems is in some other subsystem
(but I need to reproduce this to get some info what's going on there)

Milan

Re: [dm-crypt] system freezes when using swap that is encrypted

[Heinz Diehl]

On 14.11.2010, Nicolas Bock wrote: 

> The interesting thing is though that when I make the 2 swap partitions
> regular swap, i.e. not encrypted, then the same program does not get
> OOM killed

Just a shot in the dark: is your encrypted swap really working?
Does it show up when you do a "free -m"?

In any case, can you see something in the logs?