[Xenomai-help] Zombie user tasks

[Xenomai-help] Zombie user tasks

[Mehmet Alphan Ulusoy]

I have a problem where when I want to terminate a xenomai user space  
task, the task becomes a zombie and I can not re-run it since some of  
the resources (like mutexes) it uses are already there and can't be  
created again.

The scenario is as follows:

- A process running on the computer forks to create a child.
- The child executes the user space task using the system() call.  
(child does not return). I suppose userTask is the child of the child  
process now.
- After a while it attempts to terminate the task by calling "pkill  
-SIGTERM userTask"  using the system() call again.
- The task, not always but sooner or later -above steps are repeated  
for a number of times-, becomes a zombie.
- The user space task has a signal handler for SIGTERM and SIGINT signals.
- The signal handler is as follows:

void terminateGracefully(int sig)
{
     int err;
	    while (( err = rt_task_delete(&tid)) != 0)
	    {
	        printf("Error deleting task. Code %d.\n");
	    }
	    while (( err = rt_alarm_delete(&aid)) != 0)
	    {
	        printf("Error deleting alarm. Code %d.\n");
	    }
	    while (( err = rt_pipe_delete(&pidMon)) != 0)
	    {
	        printf("Error deleting pipe. Code %d.\n");
	    }
	    while (( err = close(sid)) != 0)
	    {
	        printf("Error deleting socket. Code %d.\n");
	    }
	}

- The signal handler is installed as follows:

signal(SIGTERM, terminateGracefully);
signal(SIGKILL, terminateGracefully);
signal(SIGINT, terminateGracefully);

- The child process which makes the system() call has a signal handler  
installed for SIGCHILD signal and calls wait() to collect their exit  
status.

signal(SIGCHLD, waitChild);

void waitChild(int sig)
{

     int status =0;

     wait(&status);

}

I've read somewhere that it's not safe to call printf() from a signal  
handler but even if I comment those lines out I can observe the same  
behavior. I don't see any error messages printed on the screen.

Is there a specific order to follow while deleting real-time entities?  
Is the system() call or "pkill -SIGTERM" the problem?

regards,

alphan

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Mehmet Alphan Ulusoy wrote:
> I have a problem where when I want to terminate a xenomai user space  
> task, the task becomes a zombie and I can not re-run it since some of  
> the resources (like mutexes) it uses are already there and can't be  
> created again.
> 
> The scenario is as follows:
> 
> - A process running on the computer forks to create a child.
> - The child executes the user space task using the system() call.  
> (child does not return). I suppose userTask is the child of the child  
> process now.
> - After a while it attempts to terminate the task by calling "pkill  
> -SIGTERM userTask"  using the system() call again.
> - The task, not always but sooner or later -above steps are repeated  
> for a number of times-, becomes a zombie.
> - The user space task has a signal handler for SIGTERM and SIGINT signals.
> - The signal handler is as follows:

You need to handle SIGCHLD in the father.

See http://en.wikipedia.org/wiki/Zombie_process

-- 
                                                 Gilles.

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Mehmet Alphan Ulusoy wrote:
> void terminateGracefully(int sig)
> {
>      int err;
> 	    while (( err = rt_task_delete(&tid)) != 0)
> 	    {
> 	        printf("Error deleting task. Code %d.\n");
> 	    }
> 	    while (( err = rt_alarm_delete(&aid)) != 0)
> 	    {
> 	        printf("Error deleting alarm. Code %d.\n");
> 	    }
> 	    while (( err = rt_pipe_delete(&pidMon)) != 0)
> 	    {
> 	        printf("Error deleting pipe. Code %d.\n");
> 	    }
> 	    while (( err = close(sid)) != 0)
> 	    {
> 	        printf("Error deleting socket. Code %d.\n");
> 	    }
> 	}

This code is seriously broken. You risk an infinite loop which may brick
your box, if ever it happens over a thread with the SCHED_FIFO policy.
All these calls are not async signal safe, so should not be called in a
signal handler. Almost the only thing permitted in a signal handler is
to call sem_post, or to set a flag used by your application when a
syscall returns EINTR.

> - The child process which makes the system() call has a signal handler  
> installed for SIGCHILD signal and calls wait() to collect their exit  
> status.

The handler for SIGCHLD should be installed in the parent process, not
the children.

> 
> signal(SIGCHLD, waitChild);
> 
> void waitChild(int sig)
> {
> 
>      int status =0;
> 
>      wait(&status);
> 
> }

That is incorrect, you may receive SIGCHLD only once for several
children termination, so, you should call wait4 in non blocking mode
until it returns EWOULDBLOCK.

Or you should ignore explicitely SIGCHLD altogether, passing SIG_IGN to
signal or sigaction, Linux will then take care correctly of terminated
children.

> Is there a specific order to follow while deleting real-time entities?  
> Is the system() call or "pkill -SIGTERM" the problem?

why using system, why not using the kill service directly?

-- 
					    Gilles.

Re: [Xenomai-help] Zombie user tasks

[Mehmet Alphan Ulusoy]

Thanks for the prompt and informative reply. After reading about  
signals and processes I have to admit that I didn't have a clear idea  
on the concept when I was posting my initial question. Nevertheless  
I've modified the code of the user space task as follows:

- signal handler sets flag when it's called.

- At the end of the initialization routine (main()):

...
while(!termFlag)
     pause();

clean();

return;

}


ensures that clean() is called only if SIGTERM is received and  
termFlag is raised by the signal handler. (termFlag is an int).

Is this safe?

By the way, in my trials I've found out that ppid of the zombie task  
is 1 (init). So does this mean that the parent of the task did not  
call wait() properly and init is the new parent of the task? Even so,  
shouldn't it be cleaned automatically by init?

-- alphan


On Thu 27 Nov 2008 21:39:50 EET, Gilles Chanteperdrix wrote:

> Mehmet Alphan Ulusoy wrote:
>> void terminateGracefully(int sig)
>> {
>>      int err;
>> 	    while (( err = rt_task_delete(&tid)) != 0)
>> 	    {
>> 	        printf("Error deleting task. Code %d.\n");
>> 	    }
>> 	    while (( err = rt_alarm_delete(&aid)) != 0)
>> 	    {
>> 	        printf("Error deleting alarm. Code %d.\n");
>> 	    }
>> 	    while (( err = rt_pipe_delete(&pidMon)) != 0)
>> 	    {
>> 	        printf("Error deleting pipe. Code %d.\n");
>> 	    }
>> 	    while (( err = close(sid)) != 0)
>> 	    {
>> 	        printf("Error deleting socket. Code %d.\n");
>> 	    }
>> 	}
>
> This code is seriously broken. You risk an infinite loop which may brick
> your box, if ever it happens over a thread with the SCHED_FIFO policy.
> All these calls are not async signal safe, so should not be called in a
> signal handler. Almost the only thing permitted in a signal handler is
> to call sem_post, or to set a flag used by your application when a
> syscall returns EINTR.
>
>> - The child process which makes the system() call has a signal handler
>> installed for SIGCHILD signal and calls wait() to collect their exit
>> status.
>
> The handler for SIGCHLD should be installed in the parent process, not
> the children.
>
>>
>> signal(SIGCHLD, waitChild);
>>
>> void waitChild(int sig)
>> {
>>
>>      int status =0;
>>
>>      wait(&status);
>>
>> }
>
> That is incorrect, you may receive SIGCHLD only once for several
> children termination, so, you should call wait4 in non blocking mode
> until it returns EWOULDBLOCK.
>
> Or you should ignore explicitely SIGCHLD altogether, passing SIG_IGN to
> signal or sigaction, Linux will then take care correctly of terminated
> children.
>
>> Is there a specific order to follow while deleting real-time entities?
>> Is the system() call or "pkill -SIGTERM" the problem?
>
> why using system, why not using the kill service directly?
>
> --
> 					    Gilles.
>
>

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Mehmet Alphan Ulusoy wrote:
> Thanks for the prompt and informative reply. After reading about  
> signals and processes I have to admit that I didn't have a clear idea  
> on the concept when I was posting my initial question. Nevertheless  
> I've modified the code of the user space task as follows:
> 
> - signal handler sets flag when it's called.
> 
> - At the end of the initialization routine (main()):
> 
> ...
> while(!termFlag)
>      pause();
> 
> clean();
> 
> return;
> 
> }
> 
> 
> ensures that clean() is called only if SIGTERM is received and  
> termFlag is raised by the signal handler. (termFlag is an int).
> 
> Is this safe?

Yes, this should be Ok.

> 
> By the way, in my trials I've found out that ppid of the zombie task  
> is 1 (init). So does this mean that the parent of the task did not  
> call wait() properly and init is the new parent of the task? Even so,  
> shouldn't it be cleaned automatically by init?

It means that the zombie task's parent died before the zombie, so the
zombie got re-parented to init. it will be cleaned automatically by init
after some time.

-- 
                                                 Gilles.

Re: [Xenomai-help] Zombie user tasks

[Mehmet Alphan Ulusoy]

ps aux says Task's state is "Zl+" and it's there for over 6 hours now,  
is init's mentioned behavior standard or somewhat optional?

If it's standard, then does this mean that there's something else going on?

On Fri 28 Nov 2008 18:23:39 EET, Gilles Chanteperdrix wrote:

> Mehmet Alphan Ulusoy wrote:
>> Thanks for the prompt and informative reply. After reading about
>> signals and processes I have to admit that I didn't have a clear idea
>> on the concept when I was posting my initial question. Nevertheless
>> I've modified the code of the user space task as follows:
>>
>> - signal handler sets flag when it's called.
>>
>> - At the end of the initialization routine (main()):
>>
>> ...
>> while(!termFlag)
>>      pause();
>>
>> clean();
>>
>> return;
>>
>> }
>>
>>
>> ensures that clean() is called only if SIGTERM is received and
>> termFlag is raised by the signal handler. (termFlag is an int).
>>
>> Is this safe?
>
> Yes, this should be Ok.
>
>>
>> By the way, in my trials I've found out that ppid of the zombie task
>> is 1 (init). So does this mean that the parent of the task did not
>> call wait() properly and init is the new parent of the task? Even so,
>> shouldn't it be cleaned automatically by init?
>
> It means that the zombie task's parent died before the zombie, so the
> zombie got re-parented to init. it will be cleaned automatically by init
> after some time.
>
> --
>                                                  Gilles.
>
>

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Mehmet Alphan Ulusoy wrote:
> ps aux says Task's state is "Zl+" and it's there for over 6 hours now,  
> is init's mentioned behavior standard or somewhat optional?

Could you try sysrq+T to see if the task cleanup could be blocked in a
xenomai cleanup handler ?


-- 
                                                 Gilles.

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Mehmet Alphan Ulusoy wrote:
> ps aux says Task's state is "Zl+" and it's there for over 6 hours now,  
> is init's mentioned behavior standard or somewhat optional?
> 
> If it's standard, then does this mean that there's something else going on?

If you are able to reproduce this issue with a simple program, could you
send it to us?

-- 
                                                 Gilles.

Re: [Xenomai-help] Zombie user tasks

[Alphan Ulusoy]

[-- Attachment #1: Type: text/plain, Size: 2863 bytes --]

I've tried sysRq+t & Alt+SysRq+T but nothing happened. I guess I  
didn't turn on the magic keys option last time I compiled the kernel.

Below you can find a code snippet that mimics the behavior of the  
process that spawns user space tasks. The user space rt task itself is  
quite long but I believe the cleanup routine that I've sent prevously  
can help. Please note that I didn't compile this code to reproduce the  
behaviour but still, this is a good representative of what the actual  
father task does.

And I've noticed that I didn't change the wait() call with the wait4()  
you've suggested. But does it matter in this case? The forking process  
always has a single child at any given time (I assume system() call  
becomes a child of the calling process), at least I believe so...

///////////////////
/////////////////// The forking process
///////////////////


#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <signal.h>

int forkResult=0;

void waitChild(int junk)
{

     int status =0;

     wait(&status);

}

int main(void)
{

     int result=0;

     signal(SIGCHLD, SIG_IGN);

     // Fork only once
     while (!forkResult)
     {

         forkResult = fork();

         // Only the child process executes here
         if (forkResult == 0)
         {

             // Install the signal handler (if set to SIG_IGN,  
system() always returns -1)
     	    signal(SIGCHLD, waitChild);

             while (1)
             {

                 // Execute the task in the background
                 result=system("/some_dir/userTask &");

                 sleep(60);

		//Terminate
		result=system("pkill -SIGTERM userTask");

		sleep(5);

		//Are we done?
		result=system("while ps -p `pgrep userTask` > /dev/null 2>&1; do  
sleep 1; done");
		

             }
         }
     }
}

/////////////////////

On Nov 28, 2008, at 7:11 PM, Gilles Chanteperdrix wrote:

> Mehmet Alphan Ulusoy wrote:
>> ps aux says Task's state is "Zl+" and it's there for over 6 hours  
>> now,
>> is init's mentioned behavior standard or somewhat optional?
>>
>> If it's standard, then does this mean that there's something else  
>> going on?
>
> If you are able to reproduce this issue with a simple program, could  
> you
> send it to us?
>
> -- 
>                                                 Gilles.
>

On Nov 28, 2008, at 7:11 PM, Gilles Chanteperdrix wrote:

> Mehmet Alphan Ulusoy wrote:
>> ps aux says Task's state is "Zl+" and it's there for over 6 hours  
>> now,
>> is init's mentioned behavior standard or somewhat optional?
>>
>> If it's standard, then does this mean that there's something else  
>> going on?
>
> If you are able to reproduce this issue with a simple program, could  
> you
> send it to us?
>
> -- 
>                                                 Gilles.
>


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2433 bytes --]

Re: [Xenomai-help] Zombie user tasks

[Alphan Ulusoy]

[-- Attachment #1.1: Type: text/plain, Size: 68066 bytes --]

Hello again,

Unfortunately I couldn't solve this issue of 'tasks turning into  
zombies after being killed' on my own. Below you may find the sysRq +  
T output. The name of the problematic task is:  "actuatorTask". This  
is a bit hard to reproduce since it does not happen all the time, i  
hope below output helps.

Also /proc/xenomai/stat says:

CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
   0  0      0          942268     0     00500080   99.9  ROOT
   0  15817  2899       2906       0     00300380    0.0  Actuator  
Aperiodic Task
   0  0      0          2275392    0     00000000    0.0  IRQ233:  
[timer]


and /proc/xenomai/sched says:

CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
   0  0       80      0          0          master    R          ROOT
   0  15817   80      0          0          master    X           
Actuator Aperiodic Task


[ 7565.309991] SysRq : Show State
[ 7565.309991]   task                PC stack   pid father
[ 7565.309991] init          S f7c2a15c     0     1      0
[ 7565.309991]        f7c2a000 00000086 006eec77 f7c2a15c c0575380  
c0126991 00000200 f7c2db64
[ 7565.309991]        006eec77 00000000 00000000 c03c8a0a c013166d  
f7c2dbe4 c0575ce8 c0575ce8
[ 7565.309991]        006eec77 c01265f0 f7c2a000 c0575380 0000000b  
00000400 c01a9b22 f7c2dbe4
[ 7565.309991] Call Trace:
[ 7565.309991]  [<c0126991>] __mod_timer+0xb1/0xf0
[ 7565.309991]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
[ 7565.309991]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309991]  [<c01265f0>] process_timeout+0x0/0x10
[ 7565.309991]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309991]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309991]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
[ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
[ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
[ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
[ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
[ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
[ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
[ 7565.309991]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
[ 7565.309991]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309991]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
[ 7565.309991]  [<c029ab23>] number+0x2d3/0x2e0
[ 7565.309991]  [<c0313538>] usb_choose_configuration+0x198/0x220
[ 7565.309991]  [<c01354cb>] hrtimer_interrupt+0x19b/0x1e0
[ 7565.309991]  [<c029b3b7>] vsnprintf+0x307/0x5e0
[ 7565.309991]  [<c014978f>] __xirq_end+0x0/0x46
[ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
[ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
[ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
[ 7565.309991]  [<c0147961>] __rcu_read_unlock+0x41/0x90
[ 7565.309991]  [<c01aea09>] __d_lookup+0xc9/0x160
[ 7565.309991]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309991]  [<c01a397a>] permission+0x6a/0x110
[ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309991]  [<c0197794>] __slab_alloc+0x74/0x510
[ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309991]  [<c01b4183>] mntput_no_expire+0x13/0x120
[ 7565.309991]  [<c01a606e>] path_walk+0x4e/0x90
[ 7565.309991]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309991]  [<c029c6c5>] copy_to_user+0x35/0x60
[ 7565.309991]  [<c019f488>] cp_new_stat64+0xf8/0x110
[ 7565.309991]  [<c01aa382>] sys_select+0xe2/0x1b0
[ 7565.309991]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309991]  =======================
[ 7565.309991] kthreadd      S f7c2a4bc     0     2      0
[ 7565.309991]        f7c2a360 00000046 f7c2a4c0 f7c2a4bc 00000000  
c0119c46 f766fde8 c047f1f0
[ 7565.309991]        00003dba f766fdc4 00000000 c0131105 00000000  
c0130f60 00000000 00000000
[ 7565.309991]        00000000 c0103db3 00000000 00000000 00000000  
00000000 00000000 00000000
[ 7565.309991] Call Trace:
[ 7565.309991]  [<c0119c46>] complete+0x76/0xb0
[ 7565.309992]  [<c0131105>] kthreadd+0x1a5/0x1d0
[ 7565.309992]  [<c0130f60>] kthreadd+0x0/0x1d0
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] ksoftirqd/0   S f7c2a81c     0     3      2
[ 7565.309992]        f7c2a6c0 00000046 c0575180 f7c2a81c c0121d9a  
00000000 00000000 00000000
[ 7565.309992]        00000000 c0122120 00000000 c0122219 fffffffc  
c0130f32 c0130ef0 00000000
[ 7565.309992]        00000000 c0103db3 f7c2df20 00000000 00000000  
00000000 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
[ 7565.309992]  [<c0122120>] ksoftirqd+0x0/0x180
[ 7565.309992]  [<c0122219>] ksoftirqd+0xf9/0x180
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] events/0      R running      0     4      2
[ 7565.309992]        f7c2aa20 00000046 c012d8c0 f7c2ab7c c04762e0  
c01315a5 00000200 f7c08100
[ 7565.309992]        f7c08108 00000000 00000000 c012e1bd 00000000  
f7c2aa20 c01312c0 f7c37fc0
[ 7565.309992]        f7c37fc0 fffffffc f7c08100 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] khelper       S f7c2aedc     0     5      2
[ 7565.309992]        f7c2ad80 00000046 c012d8c0 f7c2aedc f7f22d80  
c01315a5 00000200 f7c08180
[ 7565.309992]        f7c08188 00000000 00000000 c012e1bd 00000000  
f7c2ad80 c01312c0 f7c08188
[ 7565.309992]        f7c08188 fffffffc f7c08180 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] kblockd/0     S f7c8323c     0    52      2
[ 7565.309992]        f7c830e0 00000046 c012d8c0 f7c8323c c04762e0  
c01315a5 00000200 f7c19b80
[ 7565.309992]        f7c19b88 00000000 00000000 c012e1bd 00000000  
f7c830e0 c01312c0 f7c19b88
[ 7565.309992]        f7c19b88 fffffffc f7c19b80 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] khubd         S f7cd123c     0    61      2
[ 7565.309992]        f7cd10e0 00000046 000003e8 f7cd123c f7c9ad9c  
c01315a5 00000200 f7cddfa4
[ 7565.309992]        f7c9ad80 00000000 00000003 c0306cb4 f7cddfbc  
00000000 f7cd1108 0cb15b1c
[ 7565.309992]        00000000 00000239 f75e7d04 f7eeb21c f75e7c00  
f7c9f01c 00000000 00000002
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c0306cb4>] hub_thread+0x674/0xd10
[ 7565.309992]  [<c0117dfe>] hrtick_set+0x8e/0x1a0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c0306640>] hub_thread+0x0/0xd10
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] kseriod       S f7cd1c5c     0    64      2
[ 7565.309992]        f7cd1b00 00000046 c0197c85 f7cd1c5c 00000000  
c01315a5 00000200 f7ce3fb0
[ 7565.309992]        00000000 00000000 00000200 c0322a7f c0476160  
00000000 f7cd1b00 c03c8219
[ 7565.309992]        00000000 f7cd1b00 c01312c0 c04ae2d8 c04ae2d8  
fffffffc 00000000 c0322970
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0197c85>] kfree+0x55/0x110
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c0322a7f>] serio_thread+0x10f/0x430
[ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c0322970>] serio_thread+0x0/0x430
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] gatekeeper/0  S f7e4e4bc     0   148      2
[ 7565.309992]        f7e4e360 00000046 00000000 f7e4e4bc c057c540  
c0577b00 c0581a2c f7e61a20
[ 7565.309992]        c0581a20 c0581a2c 00000000 c0156a77 00000000  
f7e4e360 c03c8219 00000001
[ 7565.309992]        f7e4e360 c0119840 c0581a24 c0581a24 fffffffc  
c0581a20 c01569d0 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0156a77>] gatekeeper_thread+0xa7/0x130
[ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c01569d0>] gatekeeper_thread+0x0/0x130
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] pdflush       S f7e4f23c     0   152      2
[ 7565.309992]        f7e4f0e0 00000046 f7e4f0e0 f7e4f23c c03c85be  
c0119684 00000200 f7e5dfc4
[ 7565.309992]        00000000 c017d200 00000000 c017d2d4 f7e4f23c  
f7e4f0e0 00000000 00000000
[ 7565.309992]        c0485e54 f7e5ffc4 fffb6d17 fffffffc c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c85be>] preempt_schedule+0x4e/0x70
[ 7565.309992]  [<c0119684>] set_user_nice+0x114/0x170
[ 7565.309992]  [<c017d200>] pdflush+0x0/0x270
[ 7565.309992]  [<c017d2d4>] pdflush+0xd4/0x270
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] pdflush       S f7e4f59c     0   153      2
[ 7565.309992]        f7e4f440 00000046 00000000 f7e4f59c 00000000  
00000000 00000025 f7e5ffc4
[ 7565.309992]        00000000 c017d200 00000000 c017d2d4 f7e4f59c  
f7e4f440 00000000 00000000
[ 7565.309992]        f7e5dfc4 c0485e54 006edb25 fffffffc c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c017d200>] pdflush+0x0/0x270
[ 7565.309992]  [<c017d2d4>] pdflush+0xd4/0x270
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] kswapd0       S f7e4f8fc     0   154      2
[ 7565.309992]        f7e4f7a0 00000046 00000000 f7e4f8fc 00000000  
c01315a5 00000200 00000000
[ 7565.309992]        c0485340 c01312c0 00000000 c018119b 00000000  
00000000 f7eb9f74 f7e4f7c8
[ 7565.309992]        c0485de8 00000000 00000203 c0116ebb f7e4f7a0  
c0117160 c0180de0 f7e4f7c8
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c018119b>] kswapd+0x3bb/0x430
[ 7565.309992]  [<c0116ebb>] update_curr+0x6b/0x70
[ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
[ 7565.309992]  [<c0180de0>] kswapd+0x0/0x430
[ 7565.309992]  [<c011796b>] dequeue_task_fair+0x2b/0x100
[ 7565.309992]  [<c0117dfe>] hrtick_set+0x8e/0x1a0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
[ 7565.309992]  [<c0180de0>] kswapd+0x0/0x430
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] aio/0         S f7f4b23c     0   197      2
[ 7565.309992]        f7f4b0e0 00000046 c04f4338 f7f4b23c 00000000  
c01315a5 00000200 f7c19a80
[ 7565.309992]        f7c19a88 00000000 00000000 c012e1bd 00000000  
f7f4b0e0 c01312c0 f7c19a88
[ 7565.309992]        f7c19a88 fffffffc f7c19a80 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] unionfs_siod/ S f7f9415c     0   210      2
[ 7565.309992]        f7f94000 00000046 c04f4338 f7f9415c 00000000  
c01315a5 00000200 f7c190c0
[ 7565.309992]        f7c190c8 00000000 00000000 c012e1bd 00000000  
f7f94000 c01312c0 f7c190c8
[ 7565.309992]        f7c190c8 fffffffc f7c190c0 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] nfsiod        S f7f944bc     0   211      2
[ 7565.309992]        f7f94360 00000046 c04f4338 f7f944bc 00000000  
c01315a5 00000200 f7c08a80
[ 7565.309992]        f7c08a88 00000000 00000000 c012e1bd 00000000  
f7f94360 c01312c0 f7c08a88
[ 7565.309992]        f7c08a88 fffffffc f7c08a80 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] scsi_eh_0     S f7c82edc     0   874      2
[ 7565.309992]        f7c82d80 00000046 c047f254 f7c82edc f7c82d80  
c0117160 00000000 fffffffc
[ 7565.309992]        f7caa000 00000000 f7cb5fbc c02f4c7a 00000000  
c0117e30 00000000 ffffffff
[ 7565.309992]        ffffffff 00000200 f7c82d80 c0476160 00000000  
f7c82d80 c03c8219 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
[ 7565.309992]  [<c02f4c7a>] scsi_error_handler+0x4a/0x3b0
[ 7565.309992]  [<c0117e30>] hrtick_set+0xc0/0x1a0
[ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
[ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
[ 7565.309992]  [<c02f4c30>] scsi_error_handler+0x0/0x3b0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] usb-storage   S f7c838fc     0   875      2
[ 7565.309992]        f7c837a0 00000046 c04f4300 f7c838fc c0134fe2  
00000000 00000000 f7c837a0
[ 7565.309992]        7fffffff 00040000 00000000 c03c8a35 0000049b  
00000000 00989680 00000000
[ 7565.309992]        c0116e40 f7c2a6e8 f7c2a6c0 f7c837a0 f7c837a0  
7fffffff c03c94a9 f7caa39c
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0116e40>] wakeup_preempt_entity+0x70/0x80
[ 7565.309992]  [<c03c94a9>] __down_interruptible+0x69/0xe0
[ 7565.309992]  [<c0135f97>] down_interruptible+0x87/0xa0
[ 7565.309992]  [<c0320c80>] usb_stor_control_thread+0x60/0x240
[ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
[ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
[ 7565.309992]  [<c0320c20>] usb_stor_control_thread+0x0/0x240
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] kpsmoused     S f7cd04bc     0   887      2
[ 7565.309992]        f7cd0360 00000046 00000000 f7cd04bc ffffffff  
c01315a5 00000200 f7c9b880
[ 7565.309992]        f7c9b888 00000000 00000000 c012e1bd 00000000  
f7cd0360 c01312c0 f7c9b888
[ 7565.309992]        f7c9b888 fffffffc f7c9b880 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] rpciod/0      S f7cd18fc     0   898      2
[ 7565.309992]        f7cd17a0 00000046 00000000 f7cd18fc ffffffff  
c01315a5 00000200 f7c9b6c0
[ 7565.309992]        f7c9b6c8 00000000 00000000 c012e1bd 00000000  
f7cd17a0 c01312c0 f7c9b6c8
[ 7565.309992]        f7c9b6c8 fffffffc f7c9b6c0 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] kjournald     S f7e4e81c     0   902      2
[ 7565.309992]        f7e4e6c0 00000046 c0119aaf f7e4e81c 00000000  
c01315a5 00000200 f7fb8500
[ 7565.309992]        00000000 f7fb85c4 f7fb8540 c01f9da5 00000000  
00000005 f7fb8550 00000000
[ 7565.309992]        f7e4e6c0 c01312c0 f7fb8550 f7fb8550 fffffffc  
f7fb8500 c01f9b80 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] udevd         S f7c2b23c     0  1004      1
[ 7565.309992]        f7c2b0e0 00000082 808fe474 f7c2b23c f7f94d80  
f7f4fb3c c013166d 00000008
[ 7565.309992]        00000080 00000000 00000000 c03c8a35 c013166d  
c013166d f7f4fbe4 c03c8d50
[ 7565.309992]        f7461060 00000041 c01ca515 00000007 00000008  
00000080 c01a9b22 f7f4fbe4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
[ 7565.309992]  [<c01ca515>] inotify_poll+0x45/0x60
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
[ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
[ 7565.309992]  [<c017c5b9>] balance_dirty_pages_ratelimited_nr 
+0x59/0x2c0
[ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
[ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
[ 7565.309992]  [<c019304f>] shmem_getpage+0x4ef/0x960
[ 7565.309992]  [<c0183803>] do_wp_page+0x1f3/0x4c0
[ 7565.309992]  [<c017ba76>] set_page_dirty+0x46/0xd0
[ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
[ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
[ 7565.309992]  [<c01128d7>] __ipipe_handle_exception+0xe7/0x1e0
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
[ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
[ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309992]  [<c01ae2ad>] dput+0x7d/0x190
[ 7565.309992]  [<c01a3c9b>] __follow_mount+0x1b/0x70
[ 7565.309992]  [<c01a3e45>] do_lookup+0x65/0x1a0
[ 7565.309992]  [<c0195030>] shmem_permission+0x0/0x10
[ 7565.309992]  [<c01a397a>] permission+0x6a/0x110
[ 7565.309992]  [<c01a5237>] __link_path_walk+0x67/0xe50
[ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
[ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
[ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
[ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c01317d0>] remove_wait_queue+0x70/0xb0
[ 7565.309992]  [<c011f38c>] do_wait+0x3bc/0xc80
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309992]  [<c01a69e9>] do_rmdir+0xb9/0x100
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] kjournald     S f7f4a4bc     0  2073      2
[ 7565.309992]        f7f4a360 00000046 c0119aaf f7f4a4bc 00000000  
c01315a5 00000200 f745a400
[ 7565.309992]        00000000 f745a4c4 f745a440 c01f9da5 00000000  
00000005 f745a450 00000000
[ 7565.309992]        f7f4a360 c01312c0 f745a450 f745a450 fffffffc  
f745a400 c01f9b80 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] kjournald     S f7f9481c     0  2080      2
[ 7565.309992]        f7f946c0 00000046 c0119aaf f7f9481c 00000000  
c01315a5 00000200 f745a200
[ 7565.309992]        00000000 f745a2c4 f745a240 c01f9da5 00000000  
00000005 f745a250 00000000
[ 7565.309992]        f7f946c0 c01312c0 f745a250 f745a250 fffffffc  
f745a200 c01f9b80 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
[ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309992]  =======================
[ 7565.309992] dhclient3     S f7c2b8fc     0  2153      1
[ 7565.309992]        f7c2b7a0 00000086 0071a227 f7c2b8fc c0575380  
c0126991 00000200 f7633b64
[ 7565.309992]        0071a227 00000000 00000000 c03c8a0a c013166d  
f74a3a00 c0575db8 c0575db8
[ 7565.309992]        0071a227 c01265f0 f7c2b7a0 c0575380 00000006  
00000020 c01a9b22 f7633be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
[ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
[ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
[ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
[ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
[ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
[ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
[ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
[ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
[ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309992]  [<c0196f6e>] deactivate_slab+0x5e/0x180
[ 7565.309992]  [<c0197794>] __slab_alloc+0x74/0x510
[ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
[ 7565.309992]  [<c0139989>] clocksource_get_next+0x59/0x90
[ 7565.309992]  [<c0137d69>] update_wall_time+0x259/0x7d0
[ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
[ 7565.309992]  [<c0341e48>] sock_alloc_send_skb+0x168/0x1b0
[ 7565.309992]  [<c011941d>] task_rq_lock+0x2d/0x40
[ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
[ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309992]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
[ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
[ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
[ 7565.309992]  [<c0121cc8>] _local_bh_enable+0x28/0x90
[ 7565.309992]  [<c0121e45>] do_softirq+0x65/0x70
[ 7565.309992]  [<c0121f3f>] irq_exit+0x5f/0x90
[ 7565.309992]  [<c010f5a0>] smp_apic_timer_interrupt+0x30/0x70
[ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
[ 7565.309992]  [<c010318c>] restore_nocheck_notrace+0x0/0xe
[ 7565.309992]  [<c010f570>] smp_apic_timer_interrupt+0x0/0x70
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c014978f>] __xirq_end+0x0/0x46
[ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
[ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
[ 7565.309992]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
[ 7565.309992]  [<c0183803>] do_wp_page+0x1f3/0x4c0
[ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
[ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
[ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
[ 7565.309992]  [<c029c6c5>] copy_to_user+0x35/0x60
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] syslogd       S f7f95c5c     0  2294      1
[ 7565.309992]        f7f95b00 00000086 c01e6c14 f7f95c5c f7f957a0  
c01f54f6 c013166d 00000001
[ 7565.309992]        00000001 00000000 00000000 c03c8a35 00000001  
f7437be4 f7ee8000 011a92de
[ 7565.309992]        c03f2be0 00000001 f7ee8000 00000000 00000001  
00000001 c01a9b22 f7437be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
[ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
[ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
[ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
[ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
[ 7565.309992]  [<c01e7187>] ext3_mark_inode_dirty+0x37/0x50
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
[ 7565.309992]  [<c017c5b9>] balance_dirty_pages_ratelimited_nr 
+0x59/0x2c0
[ 7565.309992]  [<c01e9372>] ext3_ordered_write_end+0xe2/0x140
[ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
[ 7565.309992]  [<c0175ec5>] generic_file_buffered_write+0x1c5/0x690
[ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
[ 7565.309992]  [<c03a1f04>] unix_dgram_recvmsg+0x184/0x2a0
[ 7565.309992]  [<c01747d5>] remove_suid+0x15/0x60
[ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
[ 7565.309992]  [<c01765dd>] __generic_file_aio_write_nolock+0x24d/0x560
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c01a2ce0>] pipe_write+0x0/0x470
[ 7565.309992]  [<c019b94f>] do_sync_readv_writev+0xbf/0x100
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c019f488>] cp_new_stat64+0xf8/0x110
[ 7565.309992]  [<c029c413>] copy_from_user+0x33/0x80
[ 7565.309992]  [<c019b7de>] rw_copy_check_uvector+0x7e/0x100
[ 7565.309992]  [<c019c0aa>] do_readv_writev+0xaa/0x190
[ 7565.309992]  [<c01a2ce0>] pipe_write+0x0/0x470
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c01273b4>] sigprocmask+0x74/0x100
[ 7565.309992]  [<c0129fc9>] sys_rt_sigprocmask+0xf9/0x120
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] klogd         R running      0  2300      1
[ 7565.309992]        f7f957a0 00000082 00000000 f7f958fc 00000000  
c01315a5 00000200 f76b7f34
[ 7565.309992]        c01312c0 00000fff 0804d6a0 c011da47 f76b7f1c  
f76b7f1c 00000000 00000000
[ 7565.309992]        f76b7e80 494e9495 00000000 f7f957a0 c01312c0  
f76b7f40 f76b7f40 f7c0c300
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c011da47>] do_syslog+0x437/0x4c0
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c01dbae0>] kmsg_read+0x0/0x50
[ 7565.309992]  [<c01d2990>] proc_reg_read+0x80/0xe0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c01d2910>] proc_reg_read+0x0/0xe0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] inetd         S f7c8359c     0  2310      1
[ 7565.309992]        f7c83440 00000086 f794c000 f7c8359c f7fb8500  
c18de7a0 000fa013 00000006
[ 7565.309992]        00000020 00000000 00000000 c03c8a35 c013166d  
f7fdfbe4 f769c440 f7fc9c80
[ 7565.309992]        00000000 c036fb94 c03f2000 00000020 00000006  
00000020 c01a9b22 f7fdfbe4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c036fb94>] tcp_poll+0x14/0x150
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
[ 7565.309992]  [<c035f51a>] __nla_reserve+0x2a/0x70
[ 7565.309992]  [<c035f579>] __nla_put+0x19/0x40
[ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
[ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
[ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
[ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
[ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
[ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c0177285>] generic_file_aio_read+0x5b5/0x620
[ 7565.309992]  [<c01776ab>] filemap_fault+0x14b/0x4d0
[ 7565.309992]  [<c0183ed8>] __do_fault+0x188/0x360
[ 7565.309992]  [<c0185a6f>] handle_mm_fault+0x10f/0x670
[ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] nmbd          S f7cd015c     0  2318      1
[ 7565.309992]        f7cd0000 00000086 006edf00 f7cd015c c0575380  
c0126991 00000200 f74c1b64
[ 7565.309992]        006edf00 00000000 00000000 c03c8a0a c013166d  
f74c1be4 c0575c80 c0575c80
[ 7565.309992]        006edf00 c01265f0 f7cd0000 c0575380 0000000c  
00000800 c01a9b22 f74c1be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
[ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0369698>] ip_finish_output+0x1c8/0x2c0
[ 7565.309992]  [<c0343f8f>] __skb_clone+0x1f/0xd0
[ 7565.309992]  [<c0367fc8>] ip_cork_release+0x28/0x40
[ 7565.309992]  [<c03473fa>] skb_queue_tail+0x6a/0xb0
[ 7565.309992]  [<c034394c>] sock_def_readable+0x4c/0x80
[ 7565.309992]  [<c0342933>] sock_queue_rcv_skb+0xa3/0xe0
[ 7565.309992]  [<c0384697>] udp_queue_rcv_skb+0xc7/0x270
[ 7565.309992]  [<c0347c98>] skb_copy_and_csum_datagram+0x258/0x3c0
[ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
[ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
[ 7565.309992]  [<c03457f8>] __kfree_skb+0x8/0x80
[ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
[ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
[ 7565.309992]  [<c01986fc>] kmem_cache_free+0x7c/0xd0
[ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
[ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
[ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
[ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
[ 7565.309992]  [<c0386324>] udp_recvmsg+0x1b4/0x330
[ 7565.309992]  [<c0341425>] sock_common_recvmsg+0x45/0x70
[ 7565.309992]  [<c033f6c7>] sock_recvmsg+0xd7/0x110
[ 7565.309992]  [<c0139989>] clocksource_get_next+0x59/0x90
[ 7565.309992]  [<c0137d69>] update_wall_time+0x259/0x7d0
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c015259e>] xnarch_next_htick_shot+0x4e/0x60
[ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
[ 7565.309992]  [<c034cde7>] netdev_run_todo+0x27/0x270
[ 7565.309992]  [<c034a723>] __dev_get_by_name+0x73/0x90
[ 7565.309992]  [<c01a890f>] fasync_helper+0x7f/0x120
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01ab7ab>] __posix_lock_file+0x6b/0x570
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01acb5e>] fcntl_setlk64+0x4e/0x2b0
[ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
[ 7565.309992]  [<c01273b4>] sigprocmask+0x74/0x100
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] smbd          S f7cd0b7c     0  2320      1
[ 7565.309992]        f7cd0a20 00000082 c01bc400 f7cd0b7c f794c000  
00000000 f7fb8500 00000015
[ 7565.309992]        00100000 00000000 00000000 c03c8a35 c013166d  
f7511be4 f7fbd780 f7535a00
[ 7565.309992]        00000000 c01a21c2 00000014 00100000 00000015  
00100000 c01a9b22 f7511be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01bc400>] __mark_inode_dirty+0x30/0x1b0
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c01a21c2>] pipe_poll+0x32/0xc0
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
[ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
[ 7565.309992]  [<c01796ce>] __rmqueue+0x1e/0x1d0
[ 7565.309992]  [<c017995c>] rmqueue_bulk+0x6c/0x90
[ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
[ 7565.309992]  [<c01776ab>] filemap_fault+0x14b/0x4d0
[ 7565.309992]  [<c0183ed8>] __do_fault+0x188/0x360
[ 7565.309992]  [<c0185a6f>] handle_mm_fault+0x10f/0x670
[ 7565.309992]  [<c0197daf>] kmem_cache_alloc+0x5f/0xd0
[ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
[ 7565.309992]  [<c0197daf>] kmem_cache_alloc+0x5f/0xd0
[ 7565.309992]  [<c019d0bb>] alloc_file+0x3b/0x60
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c01a842b>] do_fcntl+0x2ab/0x310
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] smbd          S f7e4eedc     0  2330   2320
[ 7565.309992]        f7e4ed80 00000082 fffffff7 f7e4eedc 0000000e  
00000009 c01a84dd 00000000
[ 7565.309992]        00000001 00000003 f7c8a000 c01294f0 c0103145  
00000000 00000019 089f6fd8
[ 7565.309992]        00000001 00000003 bfec4fe8 0000001d 0000007b  
0000007b c0100000 0000001d
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c01a84dd>] sys_fcntl64+0x4d/0x90
[ 7565.309992]  [<c01294f0>] sys_pause+0x10/0x20
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] sshd          S f7f4b8fc     0  2335      1
[ 7565.309992]        f7f4b7a0 00000082 f794c018 f7f4b8fc f7f22d80  
c18de7a0 000fa013 00000005
[ 7565.309992]        00000010 00000000 00000000 c03c8a35 c013166d  
f74d3be4 f769d540 f7fc9b00
[ 7565.309992]        00000000 c036fb94 c03f2000 00000010 00000005  
00000010 c01a9b22 f74d3be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c036fb94>] tcp_poll+0x14/0x150
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
[ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
[ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
[ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
[ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
[ 7565.309992]  [<c01e7187>] ext3_mark_inode_dirty+0x37/0x50
[ 7565.309992]  [<c01eefe4>] __ext3_journal_stop+0x24/0x50
[ 7565.309992]  [<c01bc400>] __mark_inode_dirty+0x30/0x1b0
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
[ 7565.309992]  [<c01e9372>] ext3_ordered_write_end+0xe2/0x140
[ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
[ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
[ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
[ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
[ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
[ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
[ 7565.309992]  [<c0134df6>] hrtimer_start+0xd6/0x190
[ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
[ 7565.309992]  [<c0117e30>] hrtick_set+0xc0/0x1a0
[ 7565.309992]  [<c0195acc>] add_partial+0x4c/0xa0
[ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
[ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
[ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
[ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
[ 7565.309992]  [<c0197cbf>] kfree+0x8f/0x110
[ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
[ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
[ 7565.309992]  [<c01a288d>] pipe_release+0x7d/0xa0
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c0199ea7>] filp_close+0x47/0x80
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] proftpd       S f7c82b7c     0  2422      1
[ 7565.309992]        f7c82a20 00200086 006f461e f7c82b7c c0575380  
c0126991 00000200 f74e3b64
[ 7565.309992]        006f461e 00000000 00000000 c03c8a0a c013166d  
f74e3be4 c0575f70 c04b09d0
[ 7565.309992]        006f461e c01265f0 f7c82a20 c0575380 00000002  
00000002 c01a9b22 f74e3be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
[ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
[ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309992]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
[ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
[ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
[ 7565.309992]  [<c013501d>] ktime_get+0xd/0x30
[ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
[ 7565.309992]  [<c01bfe6a>] __find_get_block_slow+0x7a/0x130
[ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
[ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309992]  [<c01c0290>] __find_get_block+0x90/0x1c0
[ 7565.309992]  [<c01f367e>] __journal_file_buffer+0x6e/0x130
[ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
[ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
[ 7565.309992]  [<c01f367e>] __journal_file_buffer+0x6e/0x130
[ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
[ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
[ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
[ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
[ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
[ 7565.309992]  [<c01b4183>] mntput_no_expire+0x13/0x120
[ 7565.309992]  [<c01a606e>] path_walk+0x4e/0x90
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01a6e29>] __user_walk_fd+0x49/0x60
[ 7565.309992]  [<c019f6ef>] vfs_lstat_fd+0x1f/0x50
[ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
[ 7565.309992]  [<c0134df6>] hrtimer_start+0xd6/0x190
[ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
[ 7565.309992]  [<c0120fa0>] alarm_setitimer+0x30/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] cron          S f7f4b59c     0  2434      1
[ 7565.309992]        f7f4b440 00000082 01d7138e f7f4b59c 00000000  
00000200 00000000 f7fa5f60
[ 7565.309992]        f7f4b440 00000000 bfd44e04 c03c91bb 00000001  
00000000 00000001 c0134ef5
[ 7565.309992]        00000001 f7c3d740 f7659f60 fa1e6b8e 000006e2  
c0134a60 c047f254 00000001
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c91bb>] do_nanosleep+0x5b/0x90
[ 7565.309992]  [<c0134ef5>] hrtimer_nanosleep+0x45/0xb0
[ 7565.309992]  [<c0134a60>] hrtimer_wakeup+0x0/0x20
[ 7565.309992]  [<c0134fb8>] sys_nanosleep+0x58/0x60
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] xdm           S f7c824bc     0  2456      1
[ 7565.309992]        f7c82360 00000082 f794c000 f7c824bc f7fb8500  
c18de7a0 000fa016 00000005
[ 7565.309992]        00000010 00000000 00000000 c03c8a35 c0348524  
f799e900 f7444400 c0385d0f
[ 7565.309992]        00000000 c03f2060 00000010 f7444400 00000005  
00000010 c01a9b22 f7637be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0348524>] datagram_poll+0x14/0xf0
[ 7565.309992]  [<c0385d0f>] udp_poll+0xf/0x100
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0177fd2>] mempool_alloc+0x32/0x180
[ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
[ 7565.309992]  [<c017dcdf>] activate_page+0x4f/0xe0
[ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
[ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
[ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
[ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
[ 7565.309992]  [<c01c0290>] __find_get_block+0x90/0x1c0
[ 7565.309992]  [<c017e733>] lru_cache_add_active+0x43/0x80
[ 7565.309992]  [<c0185d93>] handle_mm_fault+0x433/0x670
[ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
[ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
[ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
[ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
[ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
[ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
[ 7565.309992]  [<c01eefe4>] __ext3_journal_stop+0x24/0x50
[ 7565.309992]  [<c01bc48a>] __mark_inode_dirty+0xba/0x1b0
[ 7565.309992]  [<c017007b>] __sem_open+0x23b/0x310
[ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c0177285>] generic_file_aio_read+0x5b5/0x620
[ 7565.309992]  [<c017a72a>] free_hot_cold_page+0x15a/0x1e0
[ 7565.309992]  [<c018455b>] unmap_vmas+0x2db/0x500
[ 7565.309992]  [<c01855f5>] free_pgtables+0x85/0xb0
[ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
[ 7565.309992]  [<c0187c33>] unmap_region+0xe3/0x120
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] getty         S f7f9559c     0  2468      1
[ 7565.309992]        f7f95440 00000086 00000720 f7f9559c f7c2aa20  
c1816a40 00000042 f7c89b80
[ 7565.309992]        f762b400 7fffffff 7fffffff c03c8a35 00000000  
c02ab166 c0135cee 00000001
[ 7565.309992]        001ae60e c013166d f7c89b80 f762b400 f7c89b80  
f762b400 c02b6254 00000003
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c02ab166>] vgacon_set_cursor_size+0xd6/0x1a0
[ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
[ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] getty         S f7f94b7c     0  2470      1
[ 7565.309992]        f7f94a20 00000086 00000000 f7f94b7c 00000000  
c17ec600 00000020 f7c89d80
[ 7565.309992]        f762ac00 7fffffff 7fffffff c03c8a35 f74d4c00  
00485898 c0135cee 00000008
[ 7565.309992]        0000446b c013166d f7c89d80 f762ac00 f7c89d80  
f762ac00 c02b6254 00000003
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
[ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] getty         S f7cd081c     0  2472      1
[ 7565.309992]        f7cd06c0 00000082 00000000 f7cd081c 00000000  
c17ec600 00000020 f7f51580
[ 7565.309992]        f762b800 7fffffff 7fffffff c03c8a35 f76b2480  
00485898 c0135cee 00000008
[ 7565.309992]        0000446b c013166d f7f51580 f762b800 f7f51580  
f762b800 c02b6254 00000003
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
[ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] getty         S f7f224bc     0  2474      1
[ 7565.309992]        f7f22360 00000082 00000000 f7f224bc 00000000  
c17ec600 00000020 f7c05a80
[ 7565.309992]        f762b000 7fffffff 7fffffff c03c8a35 f7f77a80  
00485898 c0135cee 00000008
[ 7565.309992]        0000446b c013166d f7c05a80 f762b000 f7c05a80  
f762b000 c02b6254 00000003
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
[ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] getty         S f7f2281c     0  2475      1
[ 7565.309992]        f7f226c0 00000086 00000000 f7f2281c 00000000  
c17ec600 00000020 f7fc9e00
[ 7565.309992]        f762a800 7fffffff 7fffffff c03c8a35 f76b2000  
00485898 c0135cee 00000008
[ 7565.309992]        0000446b c013166d f7fc9e00 f762a800 f7fc9e00  
f762a800 c02b6254 00000003
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
[ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] getty         S f7f22b7c     0  2476      1
[ 7565.309992]        f7f22a20 00000082 00000000 f7f22b7c 00000000  
c17ec600 00000020 f7c96080
[ 7565.309992]        f762a000 7fffffff 7fffffff c03c8a35 f7c56180  
00485898 c0135cee 00000008
[ 7565.309992]        0000446b c013166d f7c96080 f762a000 f7c96080  
f762a000 c02b6254 00000003
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
[ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] sshd          S f7f4a15c     0  2499   2335
[ 7565.309992]        f7f4a000 00000086 00000000 f7f4a15c c048bb60  
00000000 f7f12c10 00000008
[ 7565.309992]        00000080 00000000 00000000 c03c8a35 f7500100  
f7f12c00 c02b4f20 c02b0836
[ 7565.309992]        f768fbe4 f7f12c10 00000007 00000080 00000008  
00000080 c01a9b22 f768fbe4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c02b4f20>] normal_poll+0x0/0x150
[ 7565.309992]  [<c02b0836>] tty_poll+0x76/0x80
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c034e10d>] dev_queue_xmit+0xcd/0x400
[ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
[ 7565.309992]  [<c0352c55>] neigh_resolve_output+0xf5/0x270
[ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
[ 7565.309992]  [<c0367ec5>] ip_local_out+0x15/0x20
[ 7565.309992]  [<c036a25e>] ip_queue_xmit+0x1ae/0x320
[ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
[ 7565.309992]  [<c0121cc8>] _local_bh_enable+0x28/0x90
[ 7565.309992]  [<c037fa80>] tcp_v4_send_check+0x40/0xe0
[ 7565.309992]  [<c037a84f>] tcp_transmit_skb+0x3bf/0x7a0
[ 7565.309992]  [<c010f570>] smp_apic_timer_interrupt+0x0/0x70
[ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
[ 7565.309992]  [<c0379fbc>] tcp_init_tso_segs+0x3c/0x60
[ 7565.309992]  [<c037bfb9>] __tcp_push_pending_frames+0x139/0x750
[ 7565.309992]  [<c036f705>] sk_stream_alloc_skb+0x35/0xe0
[ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
[ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
[ 7565.309992]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
[ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
[ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
[ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
[ 7565.309992]  [<c0371082>] tcp_sendmsg+0x7a2/0xab0
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c033ea22>] sock_aio_write+0xe2/0x100
[ 7565.309992]  [<c01197b4>] try_to_wake_up+0x44/0xd0
[ 7565.309992]  [<c019ba56>] do_sync_write+0xc6/0x110
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c019c8e1>] sys_write+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] bash          S f7f4ab7c     0  2501   2499
[ 7565.309992]        f7f4aa20 00000082 f74d4ac4 f7f4ab7c f741f098  
c0185d00 c013166d ffffffea
[ 7565.309992]        f7f4aa18 0000000e 00000001 c011f60d f7e4fb00  
c02b36aa 00000000 00000001
[ 7565.309992]        00000000 f7f4ab10 00000000 00000003 0000000c  
f7f4aa20 00000001 f7541a00
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c011f60d>] do_wait+0x63d/0xc80
[ 7565.309992]  [<c02b36aa>] tty_ioctl+0x98a/0xab0
[ 7565.309992]  [<c01a288d>] pipe_release+0x7d/0xa0
[ 7565.309992]  [<c02b2d20>] tty_ioctl+0x0/0xab0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c011fcc3>] sys_wait4+0x73/0xc0
[ 7565.309992]  [<c011fd35>] sys_waitpid+0x25/0x30
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] remoteHost    S f7e4fc5c     0  2507   2501
[ 7565.309992]        f7e4fb00 00000082 f7e4fb00 f7e4fc5c 00000000  
c017af36 c0485da0 f769ea80
[ 7565.309992]        00000000 7fffffff f769ec40 c03c8a35 c0122076  
c0341b2b c16fe1e0 00000001
[ 7565.309992]        c0121c8a c0341a07 f769ea80 f769ea80 f769ea80  
00000000 c036e10f 000000d0
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
[ 7565.309992]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
[ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
[ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
[ 7565.309992]  [<c036e10f>] inet_csk_accept+0x12f/0x260
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c038ca4f>] inet_accept+0x1f/0xd0
[ 7565.309992]  [<c0340a30>] sys_accept+0xf0/0x210
[ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
[ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
[ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
[ 7565.309992]  [<c0340c16>] sys_socketcall+0xc6/0x280
[ 7565.309992]  [<c0199ea7>] filp_close+0x47/0x80
[ 7565.309992]  [<c019b5e2>] sys_close+0x82/0xf0
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] sshd          S f7f22edc     0  9291   2335
[ 7565.309992]        f7f22d80 00000086 00000000 f7f22edc c048bb60  
00000000 f7f4d410 00000008
[ 7565.309992]        00000080 00000000 00000000 c03c8a35 f7f04800  
f7f4d400 c02b4f20 c02b0836
[ 7565.309992]        f7679be4 f7f4d410 00000007 00000080 00000008  
00000080 c01a9b22 f7679be4
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c02b4f20>] normal_poll+0x0/0x150
[ 7565.309992]  [<c02b0836>] tty_poll+0x76/0x80
[ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
[ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c034e10d>] dev_queue_xmit+0xcd/0x400
[ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
[ 7565.309992]  [<c0352c55>] neigh_resolve_output+0xf5/0x270
[ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
[ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
[ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
[ 7565.309992]  [<c0367ec5>] ip_local_out+0x15/0x20
[ 7565.309992]  [<c036a25e>] ip_queue_xmit+0x1ae/0x320
[ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
[ 7565.309992]  [<c037fa80>] tcp_v4_send_check+0x40/0xe0
[ 7565.309992]  [<c037a84f>] tcp_transmit_skb+0x3bf/0x7a0
[ 7565.309992]  [<c0121f3f>] irq_exit+0x5f/0x90
[ 7565.309992]  [<c010f5a0>] smp_apic_timer_interrupt+0x30/0x70
[ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
[ 7565.309992]  [<c0379fbc>] tcp_init_tso_segs+0x3c/0x60
[ 7565.309992]  [<c037bfb9>] __tcp_push_pending_frames+0x139/0x750
[ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
[ 7565.309992]  [<c036f705>] sk_stream_alloc_skb+0x35/0xe0
[ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
[ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
[ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
[ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
[ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
[ 7565.309992]  [<c0371082>] tcp_sendmsg+0x7a2/0xab0
[ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
[ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
[ 7565.309992]  [<c033ea22>] sock_aio_write+0xe2/0x100
[ 7565.309992]  [<c01197b4>] try_to_wake_up+0x44/0xd0
[ 7565.309992]  [<c011688e>] __wake_up_common+0x3e/0x70
[ 7565.309992]  [<c019ba56>] do_sync_write+0xc6/0x110
[ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
[ 7565.309992]  [<c019c8e1>] sys_write+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] bash          S f7f238fc     0  9293   9291
[ 7565.309992]        f7f237a0 00000086 c013501d f7f238fc 09e6bb40  
00989680 c0117063 f7f04580
[ 7565.309992]        f7f4c000 7fffffff 7fffffff c03c8a35 00000200  
f7f237a0 f7f0e900 f7f0e900
[ 7565.309992]        f7f237a0 c013166d f7f04580 f7f4c000 f7f04580  
f7f4c000 c02b6254 00000003
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c013501d>] ktime_get+0xd/0x30
[ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
[ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
[ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
[ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
[ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] remoteHost    S f7c8281c     0 11799   2507
[ 7565.309992]        f7c826c0 00000086 00000200 f7c8281c c04f4300  
031703f8 c013166d f7592e84
[ 7565.309992]        f7c826b8 00000004 00000001 c011f60d f7592d80  
f7c826c0 f7c56300 00000000
[ 7565.309992]        00000000 f7c827b0 f74b6c80 00000000 00000000  
f7c826c0 00000001 f7592d80
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309992]  [<c011f60d>] do_wait+0x63d/0xc80
[ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309992]  [<c011fcc3>] sys_wait4+0x73/0xc0
[ 7565.309992]  [<c011fd35>] sys_waitpid+0x25/0x30
[ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309992]  =======================
[ 7565.309992] remoteHost    S f7c83c5c     0 11800   2507
[ 7565.309992]        f7c83b00 00000082 c03797ed f7c83c5c 00000200  
00000000 00000000 f7653d7c
[ 7565.309992]        f75a4880 f75a48e0 f75a4b14 c03c8a35 f74f4620  
c0341b2b 00000000 f74f4600
[ 7565.309992]        c0121c8a c0341a07 f7653d7c f7653d7c f7653d7c  
f75a4880 c0342564 00000000
[ 7565.309992] Call Trace:
[ 7565.309992]  [<c03797ed>] tcp_rcv_established+0x3ed/0x6d0
[ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
[ 7565.309993]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
[ 7565.309993]  [<c0121c8a>] local_bh_disable+0xa/0x20
[ 7565.309993]  [<c0341a07>] release_sock+0xc7/0xf0
[ 7565.309993]  [<c0342564>] sk_wait_data+0x84/0xd0
[ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309993]  [<c037173f>] tcp_recvmsg+0x3af/0x8a0
[ 7565.309993]  [<c0134fe2>] ktime_get_ts+0x22/0x50
[ 7565.309993]  [<c0117063>] hrtick_start_fair+0xa3/0x150
[ 7565.309993]  [<c0341425>] sock_common_recvmsg+0x45/0x70
[ 7565.309993]  [<c033f6c7>] sock_recvmsg+0xd7/0x110
[ 7565.309993]  [<c02b5324>] n_tty_receive_buf+0x2b4/0x1020
[ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309993]  [<c02b51c5>] n_tty_receive_buf+0x155/0x1020
[ 7565.309993]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
[ 7565.309993]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
[ 7565.309993]  [<c033ef82>] sockfd_lookup_light+0x32/0x60
[ 7565.309993]  [<c03406ce>] sys_recvfrom+0xce/0x160
[ 7565.309993]  [<c0117e30>] hrtick_set+0xc0/0x1a0
[ 7565.309993]  [<c0119a1b>] hrtick_resched+0x4b/0x60
[ 7565.309993]  [<c010260f>] do_notify_resume+0x6f/0x790
[ 7565.309993]  [<c02aedba>] tty_put_char+0x2a/0x40
[ 7565.309993]  [<c03ca20a>] rwsem_down_write_failed+0x2a/0x30
[ 7565.309993]  [<c0119aaf>] __wake_up+0x7f/0xd0
[ 7565.309993]  [<c0340793>] sys_recv+0x33/0x40
[ 7565.309993]  [<c0340cb5>] sys_socketcall+0x165/0x280
[ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309993]  =======================
[ 7565.309993] rt61pci       S f7f4a81c     0 15243      2
[ 7565.309993]        f7f4a6c0 00000046 c012d8c0 f7f4a81c c04762e0  
c01315a5 00000200 f7644b40
[ 7565.309993]        f7644b48 00000000 00000000 c012e1bd 00000000  
f7f4a6c0 c01312c0 f7644b48
[ 7565.309993]        f7644b48 fffffffc f7644b40 c012e130 c0130f32  
c0130ef0 00000000 00000000
[ 7565.309993] Call Trace:
[ 7565.309993]  [<c012d8c0>] run_workqueue+0x120/0x1b0
[ 7565.309993]  [<c01315a5>] prepare_to_wait+0x85/0xe0
[ 7565.309993]  [<c012e1bd>] worker_thread+0x8d/0xa0
[ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
[ 7565.309993]  [<c012e130>] worker_thread+0x0/0xa0
[ 7565.309993]  [<c0130f32>] kthread+0x42/0x70
[ 7565.309993]  [<c0130ef0>] kthread+0x0/0x70
[ 7565.309993]  [<c0103db3>] kernel_thread_helper+0x7/0x14
[ 7565.309993]  =======================
[ 7565.309993] actuatorTask  ? f759281c     0 15816      1
[ 7565.309993]        f75926c0 00000046 400000c0 f759281c f75926c0  
00000000 00003dc8 00000010
[ 7565.309993]        f75926c0 f75b1edc 00000000 c01202bc c0155590  
c04844a4 00000246 00000001
[ 7565.309993]        f75b1f90 f759276c f75927b0 00000000 00000000  
f75b1f90 f75b1f90 f7fdbe00
[ 7565.309993] Call Trace:
[ 7565.309993]  [<c01202bc>] do_exit+0x4bc/0x730
[ 7565.309993]  [<c0155590>] sigwake_event+0x0/0x140
[ 7565.309993]  [<c012055f>] do_group_exit+0x2f/0xc0
[ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309993]  =======================
[ 7565.309993] Actuator Aper S f7592b7c     0 15817      1
[ 7565.309993]        f7592a20 00000086 00000001 f7592b7c 00000001  
00000000 00000200 f7e60620
[ 7565.309993]        f7592a20 f7729fb8 00000170 c01560ad 00000010  
f7729f5c 00000001 00000006
[ 7565.309993]        00000000 f7729fb8 00000170 c015628f 00000001  
c0577b04 c0577b00 00000000
[ 7565.309993] Call Trace:
[ 7565.309993]  [<c01560ad>] xnshadow_harden+0x8d/0x1e0
[ 7565.309993]  [<c015628f>] losyscall_event+0x8f/0x170
[ 7565.309993]  [<c014a1e2>] __ipipe_dispatch_event+0xa2/0x190
[ 7565.309993]  [<c0156200>] losyscall_event+0x0/0x170
[ 7565.309993]  [<c0112731>] __ipipe_syscall_root+0x41/0x100
[ 7565.309993]  [<c010311d>] system_call+0x29/0x4a
[ 7565.309993]  =======================
[ 7565.309993] sh            S f7592edc     0 15832  11799
[ 7565.309993]        f7592d80 00000086 f7c56944 f7592edc f7f36080  
c0185d00 c013166d ffffffea
[ 7565.309993]        f7592d78 00000004 00000001 c011f60d f7593b00  
f7c56900 f7c56300 00000001
[ 7565.309993]        00000000 f7592e70 00000000 00000003 0000000c  
f7592d80 00000001 f7593b00
[ 7565.309993] Call Trace:
[ 7565.309993]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
[ 7565.309993]  [<c013166d>] add_wait_queue+0x6d/0xb0
[ 7565.309993]  [<c011f60d>] do_wait+0x63d/0xc80
[ 7565.309993]  [<c0119840>] default_wake_function+0x0/0x10
[ 7565.309993]  [<c011fcc3>] sys_wait4+0x73/0xc0
[ 7565.309993]  [<c011fd35>] sys_waitpid+0x25/0x30
[ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309993]  =======================
[ 7565.309993] sleep         S f7593c5c     0 16596  15832
[ 7565.309993]        f7593b00 00000086 4b3f0cce f7593c5c 00000000  
00000200 00000000 f7659f60
[ 7565.309993]        f7593b00 00000001 00000000 c03c91bb 00000001  
00000000 00000001 c0134ef5
[ 7565.309993]        f7fa5f61 f754c740 c0576960 86d9d6ce 000006e1  
c0134a60 c047f254 00000001
[ 7565.309993] Call Trace:
[ 7565.309993]  [<c03c91bb>] do_nanosleep+0x5b/0x90
[ 7565.309993]  [<c0134ef5>] hrtimer_nanosleep+0x45/0xb0
[ 7565.309993]  [<c0134a60>] hrtimer_wakeup+0x0/0x20
[ 7565.309993]  [<c0134fb8>] sys_nanosleep+0x58/0x60
[ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
[ 7565.309993]  =======================


On Nov 28, 2008, at 7:11 PM, Gilles Chanteperdrix wrote:

> Mehmet Alphan Ulusoy wrote:
>> ps aux says Task's state is "Zl+" and it's there for over 6 hours  
>> now,
>> is init's mentioned behavior standard or somewhat optional?
>>
>> If it's standard, then does this mean that there's something else  
>> going on?
>
> If you are able to reproduce this issue with a simple program, could  
> you
> send it to us?
>
> -- 
>                                                 Gilles.
>


[-- Attachment #1.2: Type: text/html, Size: 157224 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2433 bytes --]

Re: [Xenomai-help] Zombie user tasks

[Alphan Ulusoy]

[-- Attachment #1.1: Type: text/plain, Size: 71103 bytes --]

By the way, I've compiled my kernel w/ CONFIG_PREEMPT set. Can this be  
the problem? Because I've come across some posts from 2006 pointing  
out this.

regards,

alphan

On Dec 21, 2008, at 9:28 PM, Alphan Ulusoy wrote:

> Hello again,
>
> Unfortunately I couldn't solve this issue of 'tasks turning into  
> zombies after being killed' on my own. Below you may find the sysRq  
> + T output. The name of the problematic task is:  "actuatorTask".  
> This is a bit hard to reproduce since it does not happen all the  
> time, i hope below output helps.
>
> Also /proc/xenomai/stat says:
>
> CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>   0  0      0          942268     0     00500080   99.9  ROOT
>   0  15817  2899       2906       0     00300380    0.0  Actuator  
> Aperiodic Task
>   0  0      0          2275392    0     00000000    0.0  IRQ233:  
> [timer]
>
>
> and /proc/xenomai/sched says:
>
> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>   0  0       80      0          0          master    R          ROOT
>   0  15817   80      0          0          master    X           
> Actuator Aperiodic Task
>
>
> [ 7565.309991] SysRq : Show State
> [ 7565.309991]   task                PC stack   pid father
> [ 7565.309991] init          S f7c2a15c     0     1      0
> [ 7565.309991]        f7c2a000 00000086 006eec77 f7c2a15c c0575380  
> c0126991 00000200 f7c2db64
> [ 7565.309991]        006eec77 00000000 00000000 c03c8a0a c013166d  
> f7c2dbe4 c0575ce8 c0575ce8
> [ 7565.309991]        006eec77 c01265f0 f7c2a000 c0575380 0000000b  
> 00000400 c01a9b22 f7c2dbe4
> [ 7565.309991] Call Trace:
> [ 7565.309991]  [<c0126991>] __mod_timer+0xb1/0xf0
> [ 7565.309991]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
> [ 7565.309991]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309991]  [<c01265f0>] process_timeout+0x0/0x10
> [ 7565.309991]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309991]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309991]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
> [ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
> [ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
> [ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
> [ 7565.309991]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
> [ 7565.309991]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309991]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
> [ 7565.309991]  [<c029ab23>] number+0x2d3/0x2e0
> [ 7565.309991]  [<c0313538>] usb_choose_configuration+0x198/0x220
> [ 7565.309991]  [<c01354cb>] hrtimer_interrupt+0x19b/0x1e0
> [ 7565.309991]  [<c029b3b7>] vsnprintf+0x307/0x5e0
> [ 7565.309991]  [<c014978f>] __xirq_end+0x0/0x46
> [ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
> [ 7565.309991]  [<c0147961>] __rcu_read_unlock+0x41/0x90
> [ 7565.309991]  [<c01aea09>] __d_lookup+0xc9/0x160
> [ 7565.309991]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309991]  [<c01a397a>] permission+0x6a/0x110
> [ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309991]  [<c0197794>] __slab_alloc+0x74/0x510
> [ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309991]  [<c01b4183>] mntput_no_expire+0x13/0x120
> [ 7565.309991]  [<c01a606e>] path_walk+0x4e/0x90
> [ 7565.309991]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309991]  [<c029c6c5>] copy_to_user+0x35/0x60
> [ 7565.309991]  [<c019f488>] cp_new_stat64+0xf8/0x110
> [ 7565.309991]  [<c01aa382>] sys_select+0xe2/0x1b0
> [ 7565.309991]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309991]  =======================
> [ 7565.309991] kthreadd      S f7c2a4bc     0     2      0
> [ 7565.309991]        f7c2a360 00000046 f7c2a4c0 f7c2a4bc 00000000  
> c0119c46 f766fde8 c047f1f0
> [ 7565.309991]        00003dba f766fdc4 00000000 c0131105 00000000  
> c0130f60 00000000 00000000
> [ 7565.309991]        00000000 c0103db3 00000000 00000000 00000000  
> 00000000 00000000 00000000
> [ 7565.309991] Call Trace:
> [ 7565.309991]  [<c0119c46>] complete+0x76/0xb0
> [ 7565.309992]  [<c0131105>] kthreadd+0x1a5/0x1d0
> [ 7565.309992]  [<c0130f60>] kthreadd+0x0/0x1d0
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] ksoftirqd/0   S f7c2a81c     0     3      2
> [ 7565.309992]        f7c2a6c0 00000046 c0575180 f7c2a81c c0121d9a  
> 00000000 00000000 00000000
> [ 7565.309992]        00000000 c0122120 00000000 c0122219 fffffffc  
> c0130f32 c0130ef0 00000000
> [ 7565.309992]        00000000 c0103db3 f7c2df20 00000000 00000000  
> 00000000 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
> [ 7565.309992]  [<c0122120>] ksoftirqd+0x0/0x180
> [ 7565.309992]  [<c0122219>] ksoftirqd+0xf9/0x180
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] events/0      R running      0     4      2
> [ 7565.309992]        f7c2aa20 00000046 c012d8c0 f7c2ab7c c04762e0  
> c01315a5 00000200 f7c08100
> [ 7565.309992]        f7c08108 00000000 00000000 c012e1bd 00000000  
> f7c2aa20 c01312c0 f7c37fc0
> [ 7565.309992]        f7c37fc0 fffffffc f7c08100 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] khelper       S f7c2aedc     0     5      2
> [ 7565.309992]        f7c2ad80 00000046 c012d8c0 f7c2aedc f7f22d80  
> c01315a5 00000200 f7c08180
> [ 7565.309992]        f7c08188 00000000 00000000 c012e1bd 00000000  
> f7c2ad80 c01312c0 f7c08188
> [ 7565.309992]        f7c08188 fffffffc f7c08180 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] kblockd/0     S f7c8323c     0    52      2
> [ 7565.309992]        f7c830e0 00000046 c012d8c0 f7c8323c c04762e0  
> c01315a5 00000200 f7c19b80
> [ 7565.309992]        f7c19b88 00000000 00000000 c012e1bd 00000000  
> f7c830e0 c01312c0 f7c19b88
> [ 7565.309992]        f7c19b88 fffffffc f7c19b80 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] khubd         S f7cd123c     0    61      2
> [ 7565.309992]        f7cd10e0 00000046 000003e8 f7cd123c f7c9ad9c  
> c01315a5 00000200 f7cddfa4
> [ 7565.309992]        f7c9ad80 00000000 00000003 c0306cb4 f7cddfbc  
> 00000000 f7cd1108 0cb15b1c
> [ 7565.309992]        00000000 00000239 f75e7d04 f7eeb21c f75e7c00  
> f7c9f01c 00000000 00000002
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c0306cb4>] hub_thread+0x674/0xd10
> [ 7565.309992]  [<c0117dfe>] hrtick_set+0x8e/0x1a0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c0306640>] hub_thread+0x0/0xd10
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] kseriod       S f7cd1c5c     0    64      2
> [ 7565.309992]        f7cd1b00 00000046 c0197c85 f7cd1c5c 00000000  
> c01315a5 00000200 f7ce3fb0
> [ 7565.309992]        00000000 00000000 00000200 c0322a7f c0476160  
> 00000000 f7cd1b00 c03c8219
> [ 7565.309992]        00000000 f7cd1b00 c01312c0 c04ae2d8 c04ae2d8  
> fffffffc 00000000 c0322970
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0197c85>] kfree+0x55/0x110
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c0322a7f>] serio_thread+0x10f/0x430
> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c0322970>] serio_thread+0x0/0x430
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] gatekeeper/0  S f7e4e4bc     0   148      2
> [ 7565.309992]        f7e4e360 00000046 00000000 f7e4e4bc c057c540  
> c0577b00 c0581a2c f7e61a20
> [ 7565.309992]        c0581a20 c0581a2c 00000000 c0156a77 00000000  
> f7e4e360 c03c8219 00000001
> [ 7565.309992]        f7e4e360 c0119840 c0581a24 c0581a24 fffffffc  
> c0581a20 c01569d0 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0156a77>] gatekeeper_thread+0xa7/0x130
> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c01569d0>] gatekeeper_thread+0x0/0x130
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] pdflush       S f7e4f23c     0   152      2
> [ 7565.309992]        f7e4f0e0 00000046 f7e4f0e0 f7e4f23c c03c85be  
> c0119684 00000200 f7e5dfc4
> [ 7565.309992]        00000000 c017d200 00000000 c017d2d4 f7e4f23c  
> f7e4f0e0 00000000 00000000
> [ 7565.309992]        c0485e54 f7e5ffc4 fffb6d17 fffffffc c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c85be>] preempt_schedule+0x4e/0x70
> [ 7565.309992]  [<c0119684>] set_user_nice+0x114/0x170
> [ 7565.309992]  [<c017d200>] pdflush+0x0/0x270
> [ 7565.309992]  [<c017d2d4>] pdflush+0xd4/0x270
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] pdflush       S f7e4f59c     0   153      2
> [ 7565.309992]        f7e4f440 00000046 00000000 f7e4f59c 00000000  
> 00000000 00000025 f7e5ffc4
> [ 7565.309992]        00000000 c017d200 00000000 c017d2d4 f7e4f59c  
> f7e4f440 00000000 00000000
> [ 7565.309992]        f7e5dfc4 c0485e54 006edb25 fffffffc c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c017d200>] pdflush+0x0/0x270
> [ 7565.309992]  [<c017d2d4>] pdflush+0xd4/0x270
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] kswapd0       S f7e4f8fc     0   154      2
> [ 7565.309992]        f7e4f7a0 00000046 00000000 f7e4f8fc 00000000  
> c01315a5 00000200 00000000
> [ 7565.309992]        c0485340 c01312c0 00000000 c018119b 00000000  
> 00000000 f7eb9f74 f7e4f7c8
> [ 7565.309992]        c0485de8 00000000 00000203 c0116ebb f7e4f7a0  
> c0117160 c0180de0 f7e4f7c8
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c018119b>] kswapd+0x3bb/0x430
> [ 7565.309992]  [<c0116ebb>] update_curr+0x6b/0x70
> [ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
> [ 7565.309992]  [<c0180de0>] kswapd+0x0/0x430
> [ 7565.309992]  [<c011796b>] dequeue_task_fair+0x2b/0x100
> [ 7565.309992]  [<c0117dfe>] hrtick_set+0x8e/0x1a0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
> [ 7565.309992]  [<c0180de0>] kswapd+0x0/0x430
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] aio/0         S f7f4b23c     0   197      2
> [ 7565.309992]        f7f4b0e0 00000046 c04f4338 f7f4b23c 00000000  
> c01315a5 00000200 f7c19a80
> [ 7565.309992]        f7c19a88 00000000 00000000 c012e1bd 00000000  
> f7f4b0e0 c01312c0 f7c19a88
> [ 7565.309992]        f7c19a88 fffffffc f7c19a80 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] unionfs_siod/ S f7f9415c     0   210      2
> [ 7565.309992]        f7f94000 00000046 c04f4338 f7f9415c 00000000  
> c01315a5 00000200 f7c190c0
> [ 7565.309992]        f7c190c8 00000000 00000000 c012e1bd 00000000  
> f7f94000 c01312c0 f7c190c8
> [ 7565.309992]        f7c190c8 fffffffc f7c190c0 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] nfsiod        S f7f944bc     0   211      2
> [ 7565.309992]        f7f94360 00000046 c04f4338 f7f944bc 00000000  
> c01315a5 00000200 f7c08a80
> [ 7565.309992]        f7c08a88 00000000 00000000 c012e1bd 00000000  
> f7f94360 c01312c0 f7c08a88
> [ 7565.309992]        f7c08a88 fffffffc f7c08a80 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] scsi_eh_0     S f7c82edc     0   874      2
> [ 7565.309992]        f7c82d80 00000046 c047f254 f7c82edc f7c82d80  
> c0117160 00000000 fffffffc
> [ 7565.309992]        f7caa000 00000000 f7cb5fbc c02f4c7a 00000000  
> c0117e30 00000000 ffffffff
> [ 7565.309992]        ffffffff 00000200 f7c82d80 c0476160 00000000  
> f7c82d80 c03c8219 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
> [ 7565.309992]  [<c02f4c7a>] scsi_error_handler+0x4a/0x3b0
> [ 7565.309992]  [<c0117e30>] hrtick_set+0xc0/0x1a0
> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
> [ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
> [ 7565.309992]  [<c02f4c30>] scsi_error_handler+0x0/0x3b0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] usb-storage   S f7c838fc     0   875      2
> [ 7565.309992]        f7c837a0 00000046 c04f4300 f7c838fc c0134fe2  
> 00000000 00000000 f7c837a0
> [ 7565.309992]        7fffffff 00040000 00000000 c03c8a35 0000049b  
> 00000000 00989680 00000000
> [ 7565.309992]        c0116e40 f7c2a6e8 f7c2a6c0 f7c837a0 f7c837a0  
> 7fffffff c03c94a9 f7caa39c
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0116e40>] wakeup_preempt_entity+0x70/0x80
> [ 7565.309992]  [<c03c94a9>] __down_interruptible+0x69/0xe0
> [ 7565.309992]  [<c0135f97>] down_interruptible+0x87/0xa0
> [ 7565.309992]  [<c0320c80>] usb_stor_control_thread+0x60/0x240
> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
> [ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
> [ 7565.309992]  [<c0320c20>] usb_stor_control_thread+0x0/0x240
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] kpsmoused     S f7cd04bc     0   887      2
> [ 7565.309992]        f7cd0360 00000046 00000000 f7cd04bc ffffffff  
> c01315a5 00000200 f7c9b880
> [ 7565.309992]        f7c9b888 00000000 00000000 c012e1bd 00000000  
> f7cd0360 c01312c0 f7c9b888
> [ 7565.309992]        f7c9b888 fffffffc f7c9b880 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] rpciod/0      S f7cd18fc     0   898      2
> [ 7565.309992]        f7cd17a0 00000046 00000000 f7cd18fc ffffffff  
> c01315a5 00000200 f7c9b6c0
> [ 7565.309992]        f7c9b6c8 00000000 00000000 c012e1bd 00000000  
> f7cd17a0 c01312c0 f7c9b6c8
> [ 7565.309992]        f7c9b6c8 fffffffc f7c9b6c0 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] kjournald     S f7e4e81c     0   902      2
> [ 7565.309992]        f7e4e6c0 00000046 c0119aaf f7e4e81c 00000000  
> c01315a5 00000200 f7fb8500
> [ 7565.309992]        00000000 f7fb85c4 f7fb8540 c01f9da5 00000000  
> 00000005 f7fb8550 00000000
> [ 7565.309992]        f7e4e6c0 c01312c0 f7fb8550 f7fb8550 fffffffc  
> f7fb8500 c01f9b80 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] udevd         S f7c2b23c     0  1004      1
> [ 7565.309992]        f7c2b0e0 00000082 808fe474 f7c2b23c f7f94d80  
> f7f4fb3c c013166d 00000008
> [ 7565.309992]        00000080 00000000 00000000 c03c8a35 c013166d  
> c013166d f7f4fbe4 c03c8d50
> [ 7565.309992]        f7461060 00000041 c01ca515 00000007 00000008  
> 00000080 c01a9b22 f7f4fbe4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
> [ 7565.309992]  [<c01ca515>] inotify_poll+0x45/0x60
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
> [ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
> [ 7565.309992]  [<c017c5b9>] balance_dirty_pages_ratelimited_nr 
> +0x59/0x2c0
> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
> [ 7565.309992]  [<c019304f>] shmem_getpage+0x4ef/0x960
> [ 7565.309992]  [<c0183803>] do_wp_page+0x1f3/0x4c0
> [ 7565.309992]  [<c017ba76>] set_page_dirty+0x46/0xd0
> [ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
> [ 7565.309992]  [<c01128d7>] __ipipe_handle_exception+0xe7/0x1e0
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309992]  [<c01ae2ad>] dput+0x7d/0x190
> [ 7565.309992]  [<c01a3c9b>] __follow_mount+0x1b/0x70
> [ 7565.309992]  [<c01a3e45>] do_lookup+0x65/0x1a0
> [ 7565.309992]  [<c0195030>] shmem_permission+0x0/0x10
> [ 7565.309992]  [<c01a397a>] permission+0x6a/0x110
> [ 7565.309992]  [<c01a5237>] __link_path_walk+0x67/0xe50
> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
> [ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c01317d0>] remove_wait_queue+0x70/0xb0
> [ 7565.309992]  [<c011f38c>] do_wait+0x3bc/0xc80
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309992]  [<c01a69e9>] do_rmdir+0xb9/0x100
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] kjournald     S f7f4a4bc     0  2073      2
> [ 7565.309992]        f7f4a360 00000046 c0119aaf f7f4a4bc 00000000  
> c01315a5 00000200 f745a400
> [ 7565.309992]        00000000 f745a4c4 f745a440 c01f9da5 00000000  
> 00000005 f745a450 00000000
> [ 7565.309992]        f7f4a360 c01312c0 f745a450 f745a450 fffffffc  
> f745a400 c01f9b80 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] kjournald     S f7f9481c     0  2080      2
> [ 7565.309992]        f7f946c0 00000046 c0119aaf f7f9481c 00000000  
> c01315a5 00000200 f745a200
> [ 7565.309992]        00000000 f745a2c4 f745a240 c01f9da5 00000000  
> 00000005 f745a250 00000000
> [ 7565.309992]        f7f946c0 c01312c0 f745a250 f745a250 fffffffc  
> f745a200 c01f9b80 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309992]  =======================
> [ 7565.309992] dhclient3     S f7c2b8fc     0  2153      1
> [ 7565.309992]        f7c2b7a0 00000086 0071a227 f7c2b8fc c0575380  
> c0126991 00000200 f7633b64
> [ 7565.309992]        0071a227 00000000 00000000 c03c8a0a c013166d  
> f74a3a00 c0575db8 c0575db8
> [ 7565.309992]        0071a227 c01265f0 f7c2b7a0 c0575380 00000006  
> 00000020 c01a9b22 f7633be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
> [ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
> [ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
> [ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309992]  [<c0196f6e>] deactivate_slab+0x5e/0x180
> [ 7565.309992]  [<c0197794>] __slab_alloc+0x74/0x510
> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
> [ 7565.309992]  [<c0139989>] clocksource_get_next+0x59/0x90
> [ 7565.309992]  [<c0137d69>] update_wall_time+0x259/0x7d0
> [ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
> [ 7565.309992]  [<c0341e48>] sock_alloc_send_skb+0x168/0x1b0
> [ 7565.309992]  [<c011941d>] task_rq_lock+0x2d/0x40
> [ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309992]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
> [ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
> [ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
> [ 7565.309992]  [<c0121cc8>] _local_bh_enable+0x28/0x90
> [ 7565.309992]  [<c0121e45>] do_softirq+0x65/0x70
> [ 7565.309992]  [<c0121f3f>] irq_exit+0x5f/0x90
> [ 7565.309992]  [<c010f5a0>] smp_apic_timer_interrupt+0x30/0x70
> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
> [ 7565.309992]  [<c010318c>] restore_nocheck_notrace+0x0/0xe
> [ 7565.309992]  [<c010f570>] smp_apic_timer_interrupt+0x0/0x70
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c014978f>] __xirq_end+0x0/0x46
> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
> [ 7565.309992]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
> [ 7565.309992]  [<c0183803>] do_wp_page+0x1f3/0x4c0
> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
> [ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
> [ 7565.309992]  [<c029c6c5>] copy_to_user+0x35/0x60
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] syslogd       S f7f95c5c     0  2294      1
> [ 7565.309992]        f7f95b00 00000086 c01e6c14 f7f95c5c f7f957a0  
> c01f54f6 c013166d 00000001
> [ 7565.309992]        00000001 00000000 00000000 c03c8a35 00000001  
> f7437be4 f7ee8000 011a92de
> [ 7565.309992]        c03f2be0 00000001 f7ee8000 00000000 00000001  
> 00000001 c01a9b22 f7437be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
> [ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
> [ 7565.309992]  [<c01e7187>] ext3_mark_inode_dirty+0x37/0x50
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
> [ 7565.309992]  [<c017c5b9>] balance_dirty_pages_ratelimited_nr 
> +0x59/0x2c0
> [ 7565.309992]  [<c01e9372>] ext3_ordered_write_end+0xe2/0x140
> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
> [ 7565.309992]  [<c0175ec5>] generic_file_buffered_write+0x1c5/0x690
> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
> [ 7565.309992]  [<c03a1f04>] unix_dgram_recvmsg+0x184/0x2a0
> [ 7565.309992]  [<c01747d5>] remove_suid+0x15/0x60
> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
> [ 7565.309992]  [<c01765dd>] __generic_file_aio_write_nolock+0x24d/ 
> 0x560
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c01a2ce0>] pipe_write+0x0/0x470
> [ 7565.309992]  [<c019b94f>] do_sync_readv_writev+0xbf/0x100
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c019f488>] cp_new_stat64+0xf8/0x110
> [ 7565.309992]  [<c029c413>] copy_from_user+0x33/0x80
> [ 7565.309992]  [<c019b7de>] rw_copy_check_uvector+0x7e/0x100
> [ 7565.309992]  [<c019c0aa>] do_readv_writev+0xaa/0x190
> [ 7565.309992]  [<c01a2ce0>] pipe_write+0x0/0x470
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c01273b4>] sigprocmask+0x74/0x100
> [ 7565.309992]  [<c0129fc9>] sys_rt_sigprocmask+0xf9/0x120
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] klogd         R running      0  2300      1
> [ 7565.309992]        f7f957a0 00000082 00000000 f7f958fc 00000000  
> c01315a5 00000200 f76b7f34
> [ 7565.309992]        c01312c0 00000fff 0804d6a0 c011da47 f76b7f1c  
> f76b7f1c 00000000 00000000
> [ 7565.309992]        f76b7e80 494e9495 00000000 f7f957a0 c01312c0  
> f76b7f40 f76b7f40 f7c0c300
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c011da47>] do_syslog+0x437/0x4c0
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c01dbae0>] kmsg_read+0x0/0x50
> [ 7565.309992]  [<c01d2990>] proc_reg_read+0x80/0xe0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c01d2910>] proc_reg_read+0x0/0xe0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] inetd         S f7c8359c     0  2310      1
> [ 7565.309992]        f7c83440 00000086 f794c000 f7c8359c f7fb8500  
> c18de7a0 000fa013 00000006
> [ 7565.309992]        00000020 00000000 00000000 c03c8a35 c013166d  
> f7fdfbe4 f769c440 f7fc9c80
> [ 7565.309992]        00000000 c036fb94 c03f2000 00000020 00000006  
> 00000020 c01a9b22 f7fdfbe4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c036fb94>] tcp_poll+0x14/0x150
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
> [ 7565.309992]  [<c035f51a>] __nla_reserve+0x2a/0x70
> [ 7565.309992]  [<c035f579>] __nla_put+0x19/0x40
> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
> [ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
> [ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c0177285>] generic_file_aio_read+0x5b5/0x620
> [ 7565.309992]  [<c01776ab>] filemap_fault+0x14b/0x4d0
> [ 7565.309992]  [<c0183ed8>] __do_fault+0x188/0x360
> [ 7565.309992]  [<c0185a6f>] handle_mm_fault+0x10f/0x670
> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] nmbd          S f7cd015c     0  2318      1
> [ 7565.309992]        f7cd0000 00000086 006edf00 f7cd015c c0575380  
> c0126991 00000200 f74c1b64
> [ 7565.309992]        006edf00 00000000 00000000 c03c8a0a c013166d  
> f74c1be4 c0575c80 c0575c80
> [ 7565.309992]        006edf00 c01265f0 f7cd0000 c0575380 0000000c  
> 00000800 c01a9b22 f74c1be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
> [ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0369698>] ip_finish_output+0x1c8/0x2c0
> [ 7565.309992]  [<c0343f8f>] __skb_clone+0x1f/0xd0
> [ 7565.309992]  [<c0367fc8>] ip_cork_release+0x28/0x40
> [ 7565.309992]  [<c03473fa>] skb_queue_tail+0x6a/0xb0
> [ 7565.309992]  [<c034394c>] sock_def_readable+0x4c/0x80
> [ 7565.309992]  [<c0342933>] sock_queue_rcv_skb+0xa3/0xe0
> [ 7565.309992]  [<c0384697>] udp_queue_rcv_skb+0xc7/0x270
> [ 7565.309992]  [<c0347c98>] skb_copy_and_csum_datagram+0x258/0x3c0
> [ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
> [ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
> [ 7565.309992]  [<c03457f8>] __kfree_skb+0x8/0x80
> [ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
> [ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
> [ 7565.309992]  [<c01986fc>] kmem_cache_free+0x7c/0xd0
> [ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
> [ 7565.309992]  [<c0386324>] udp_recvmsg+0x1b4/0x330
> [ 7565.309992]  [<c0341425>] sock_common_recvmsg+0x45/0x70
> [ 7565.309992]  [<c033f6c7>] sock_recvmsg+0xd7/0x110
> [ 7565.309992]  [<c0139989>] clocksource_get_next+0x59/0x90
> [ 7565.309992]  [<c0137d69>] update_wall_time+0x259/0x7d0
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c015259e>] xnarch_next_htick_shot+0x4e/0x60
> [ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
> [ 7565.309992]  [<c034cde7>] netdev_run_todo+0x27/0x270
> [ 7565.309992]  [<c034a723>] __dev_get_by_name+0x73/0x90
> [ 7565.309992]  [<c01a890f>] fasync_helper+0x7f/0x120
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01ab7ab>] __posix_lock_file+0x6b/0x570
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01acb5e>] fcntl_setlk64+0x4e/0x2b0
> [ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
> [ 7565.309992]  [<c01273b4>] sigprocmask+0x74/0x100
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] smbd          S f7cd0b7c     0  2320      1
> [ 7565.309992]        f7cd0a20 00000082 c01bc400 f7cd0b7c f794c000  
> 00000000 f7fb8500 00000015
> [ 7565.309992]        00100000 00000000 00000000 c03c8a35 c013166d  
> f7511be4 f7fbd780 f7535a00
> [ 7565.309992]        00000000 c01a21c2 00000014 00100000 00000015  
> 00100000 c01a9b22 f7511be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01bc400>] __mark_inode_dirty+0x30/0x1b0
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c01a21c2>] pipe_poll+0x32/0xc0
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
> [ 7565.309992]  [<c01796ce>] __rmqueue+0x1e/0x1d0
> [ 7565.309992]  [<c017995c>] rmqueue_bulk+0x6c/0x90
> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
> [ 7565.309992]  [<c01776ab>] filemap_fault+0x14b/0x4d0
> [ 7565.309992]  [<c0183ed8>] __do_fault+0x188/0x360
> [ 7565.309992]  [<c0185a6f>] handle_mm_fault+0x10f/0x670
> [ 7565.309992]  [<c0197daf>] kmem_cache_alloc+0x5f/0xd0
> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
> [ 7565.309992]  [<c0197daf>] kmem_cache_alloc+0x5f/0xd0
> [ 7565.309992]  [<c019d0bb>] alloc_file+0x3b/0x60
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c01a842b>] do_fcntl+0x2ab/0x310
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] smbd          S f7e4eedc     0  2330   2320
> [ 7565.309992]        f7e4ed80 00000082 fffffff7 f7e4eedc 0000000e  
> 00000009 c01a84dd 00000000
> [ 7565.309992]        00000001 00000003 f7c8a000 c01294f0 c0103145  
> 00000000 00000019 089f6fd8
> [ 7565.309992]        00000001 00000003 bfec4fe8 0000001d 0000007b  
> 0000007b c0100000 0000001d
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c01a84dd>] sys_fcntl64+0x4d/0x90
> [ 7565.309992]  [<c01294f0>] sys_pause+0x10/0x20
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] sshd          S f7f4b8fc     0  2335      1
> [ 7565.309992]        f7f4b7a0 00000082 f794c018 f7f4b8fc f7f22d80  
> c18de7a0 000fa013 00000005
> [ 7565.309992]        00000010 00000000 00000000 c03c8a35 c013166d  
> f74d3be4 f769d540 f7fc9b00
> [ 7565.309992]        00000000 c036fb94 c03f2000 00000010 00000005  
> 00000010 c01a9b22 f74d3be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c036fb94>] tcp_poll+0x14/0x150
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
> [ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
> [ 7565.309992]  [<c01e7187>] ext3_mark_inode_dirty+0x37/0x50
> [ 7565.309992]  [<c01eefe4>] __ext3_journal_stop+0x24/0x50
> [ 7565.309992]  [<c01bc400>] __mark_inode_dirty+0x30/0x1b0
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
> [ 7565.309992]  [<c01e9372>] ext3_ordered_write_end+0xe2/0x140
> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
> [ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
> [ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
> [ 7565.309992]  [<c0134df6>] hrtimer_start+0xd6/0x190
> [ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
> [ 7565.309992]  [<c0117e30>] hrtick_set+0xc0/0x1a0
> [ 7565.309992]  [<c0195acc>] add_partial+0x4c/0xa0
> [ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
> [ 7565.309992]  [<c0197cbf>] kfree+0x8f/0x110
> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
> [ 7565.309992]  [<c01a288d>] pipe_release+0x7d/0xa0
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c0199ea7>] filp_close+0x47/0x80
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] proftpd       S f7c82b7c     0  2422      1
> [ 7565.309992]        f7c82a20 00200086 006f461e f7c82b7c c0575380  
> c0126991 00000200 f74e3b64
> [ 7565.309992]        006f461e 00000000 00000000 c03c8a0a c013166d  
> f74e3be4 c0575f70 c04b09d0
> [ 7565.309992]        006f461e c01265f0 f7c82a20 c0575380 00000002  
> 00000002 c01a9b22 f74e3be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
> [ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309992]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
> [ 7565.309992]  [<c013501d>] ktime_get+0xd/0x30
> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
> [ 7565.309992]  [<c01bfe6a>] __find_get_block_slow+0x7a/0x130
> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309992]  [<c01c0290>] __find_get_block+0x90/0x1c0
> [ 7565.309992]  [<c01f367e>] __journal_file_buffer+0x6e/0x130
> [ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
> [ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
> [ 7565.309992]  [<c01f367e>] __journal_file_buffer+0x6e/0x130
> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
> [ 7565.309992]  [<c01b4183>] mntput_no_expire+0x13/0x120
> [ 7565.309992]  [<c01a606e>] path_walk+0x4e/0x90
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01a6e29>] __user_walk_fd+0x49/0x60
> [ 7565.309992]  [<c019f6ef>] vfs_lstat_fd+0x1f/0x50
> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
> [ 7565.309992]  [<c0134df6>] hrtimer_start+0xd6/0x190
> [ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
> [ 7565.309992]  [<c0120fa0>] alarm_setitimer+0x30/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] cron          S f7f4b59c     0  2434      1
> [ 7565.309992]        f7f4b440 00000082 01d7138e f7f4b59c 00000000  
> 00000200 00000000 f7fa5f60
> [ 7565.309992]        f7f4b440 00000000 bfd44e04 c03c91bb 00000001  
> 00000000 00000001 c0134ef5
> [ 7565.309992]        00000001 f7c3d740 f7659f60 fa1e6b8e 000006e2  
> c0134a60 c047f254 00000001
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c91bb>] do_nanosleep+0x5b/0x90
> [ 7565.309992]  [<c0134ef5>] hrtimer_nanosleep+0x45/0xb0
> [ 7565.309992]  [<c0134a60>] hrtimer_wakeup+0x0/0x20
> [ 7565.309992]  [<c0134fb8>] sys_nanosleep+0x58/0x60
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] xdm           S f7c824bc     0  2456      1
> [ 7565.309992]        f7c82360 00000082 f794c000 f7c824bc f7fb8500  
> c18de7a0 000fa016 00000005
> [ 7565.309992]        00000010 00000000 00000000 c03c8a35 c0348524  
> f799e900 f7444400 c0385d0f
> [ 7565.309992]        00000000 c03f2060 00000010 f7444400 00000005  
> 00000010 c01a9b22 f7637be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0348524>] datagram_poll+0x14/0xf0
> [ 7565.309992]  [<c0385d0f>] udp_poll+0xf/0x100
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0177fd2>] mempool_alloc+0x32/0x180
> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
> [ 7565.309992]  [<c017dcdf>] activate_page+0x4f/0xe0
> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
> [ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
> [ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
> [ 7565.309992]  [<c01c0290>] __find_get_block+0x90/0x1c0
> [ 7565.309992]  [<c017e733>] lru_cache_add_active+0x43/0x80
> [ 7565.309992]  [<c0185d93>] handle_mm_fault+0x433/0x670
> [ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
> [ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
> [ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
> [ 7565.309992]  [<c01eefe4>] __ext3_journal_stop+0x24/0x50
> [ 7565.309992]  [<c01bc48a>] __mark_inode_dirty+0xba/0x1b0
> [ 7565.309992]  [<c017007b>] __sem_open+0x23b/0x310
> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c0177285>] generic_file_aio_read+0x5b5/0x620
> [ 7565.309992]  [<c017a72a>] free_hot_cold_page+0x15a/0x1e0
> [ 7565.309992]  [<c018455b>] unmap_vmas+0x2db/0x500
> [ 7565.309992]  [<c01855f5>] free_pgtables+0x85/0xb0
> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
> [ 7565.309992]  [<c0187c33>] unmap_region+0xe3/0x120
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] getty         S f7f9559c     0  2468      1
> [ 7565.309992]        f7f95440 00000086 00000720 f7f9559c f7c2aa20  
> c1816a40 00000042 f7c89b80
> [ 7565.309992]        f762b400 7fffffff 7fffffff c03c8a35 00000000  
> c02ab166 c0135cee 00000001
> [ 7565.309992]        001ae60e c013166d f7c89b80 f762b400 f7c89b80  
> f762b400 c02b6254 00000003
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c02ab166>] vgacon_set_cursor_size+0xd6/0x1a0
> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] getty         S f7f94b7c     0  2470      1
> [ 7565.309992]        f7f94a20 00000086 00000000 f7f94b7c 00000000  
> c17ec600 00000020 f7c89d80
> [ 7565.309992]        f762ac00 7fffffff 7fffffff c03c8a35 f74d4c00  
> 00485898 c0135cee 00000008
> [ 7565.309992]        0000446b c013166d f7c89d80 f762ac00 f7c89d80  
> f762ac00 c02b6254 00000003
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] getty         S f7cd081c     0  2472      1
> [ 7565.309992]        f7cd06c0 00000082 00000000 f7cd081c 00000000  
> c17ec600 00000020 f7f51580
> [ 7565.309992]        f762b800 7fffffff 7fffffff c03c8a35 f76b2480  
> 00485898 c0135cee 00000008
> [ 7565.309992]        0000446b c013166d f7f51580 f762b800 f7f51580  
> f762b800 c02b6254 00000003
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] getty         S f7f224bc     0  2474      1
> [ 7565.309992]        f7f22360 00000082 00000000 f7f224bc 00000000  
> c17ec600 00000020 f7c05a80
> [ 7565.309992]        f762b000 7fffffff 7fffffff c03c8a35 f7f77a80  
> 00485898 c0135cee 00000008
> [ 7565.309992]        0000446b c013166d f7c05a80 f762b000 f7c05a80  
> f762b000 c02b6254 00000003
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] getty         S f7f2281c     0  2475      1
> [ 7565.309992]        f7f226c0 00000086 00000000 f7f2281c 00000000  
> c17ec600 00000020 f7fc9e00
> [ 7565.309992]        f762a800 7fffffff 7fffffff c03c8a35 f76b2000  
> 00485898 c0135cee 00000008
> [ 7565.309992]        0000446b c013166d f7fc9e00 f762a800 f7fc9e00  
> f762a800 c02b6254 00000003
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] getty         S f7f22b7c     0  2476      1
> [ 7565.309992]        f7f22a20 00000082 00000000 f7f22b7c 00000000  
> c17ec600 00000020 f7c96080
> [ 7565.309992]        f762a000 7fffffff 7fffffff c03c8a35 f7c56180  
> 00485898 c0135cee 00000008
> [ 7565.309992]        0000446b c013166d f7c96080 f762a000 f7c96080  
> f762a000 c02b6254 00000003
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] sshd          S f7f4a15c     0  2499   2335
> [ 7565.309992]        f7f4a000 00000086 00000000 f7f4a15c c048bb60  
> 00000000 f7f12c10 00000008
> [ 7565.309992]        00000080 00000000 00000000 c03c8a35 f7500100  
> f7f12c00 c02b4f20 c02b0836
> [ 7565.309992]        f768fbe4 f7f12c10 00000007 00000080 00000008  
> 00000080 c01a9b22 f768fbe4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c02b4f20>] normal_poll+0x0/0x150
> [ 7565.309992]  [<c02b0836>] tty_poll+0x76/0x80
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c034e10d>] dev_queue_xmit+0xcd/0x400
> [ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
> [ 7565.309992]  [<c0352c55>] neigh_resolve_output+0xf5/0x270
> [ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
> [ 7565.309992]  [<c0367ec5>] ip_local_out+0x15/0x20
> [ 7565.309992]  [<c036a25e>] ip_queue_xmit+0x1ae/0x320
> [ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
> [ 7565.309992]  [<c0121cc8>] _local_bh_enable+0x28/0x90
> [ 7565.309992]  [<c037fa80>] tcp_v4_send_check+0x40/0xe0
> [ 7565.309992]  [<c037a84f>] tcp_transmit_skb+0x3bf/0x7a0
> [ 7565.309992]  [<c010f570>] smp_apic_timer_interrupt+0x0/0x70
> [ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
> [ 7565.309992]  [<c0379fbc>] tcp_init_tso_segs+0x3c/0x60
> [ 7565.309992]  [<c037bfb9>] __tcp_push_pending_frames+0x139/0x750
> [ 7565.309992]  [<c036f705>] sk_stream_alloc_skb+0x35/0xe0
> [ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
> [ 7565.309992]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
> [ 7565.309992]  [<c0371082>] tcp_sendmsg+0x7a2/0xab0
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c033ea22>] sock_aio_write+0xe2/0x100
> [ 7565.309992]  [<c01197b4>] try_to_wake_up+0x44/0xd0
> [ 7565.309992]  [<c019ba56>] do_sync_write+0xc6/0x110
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c019c8e1>] sys_write+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] bash          S f7f4ab7c     0  2501   2499
> [ 7565.309992]        f7f4aa20 00000082 f74d4ac4 f7f4ab7c f741f098  
> c0185d00 c013166d ffffffea
> [ 7565.309992]        f7f4aa18 0000000e 00000001 c011f60d f7e4fb00  
> c02b36aa 00000000 00000001
> [ 7565.309992]        00000000 f7f4ab10 00000000 00000003 0000000c  
> f7f4aa20 00000001 f7541a00
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c011f60d>] do_wait+0x63d/0xc80
> [ 7565.309992]  [<c02b36aa>] tty_ioctl+0x98a/0xab0
> [ 7565.309992]  [<c01a288d>] pipe_release+0x7d/0xa0
> [ 7565.309992]  [<c02b2d20>] tty_ioctl+0x0/0xab0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c011fcc3>] sys_wait4+0x73/0xc0
> [ 7565.309992]  [<c011fd35>] sys_waitpid+0x25/0x30
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] remoteHost    S f7e4fc5c     0  2507   2501
> [ 7565.309992]        f7e4fb00 00000082 f7e4fb00 f7e4fc5c 00000000  
> c017af36 c0485da0 f769ea80
> [ 7565.309992]        00000000 7fffffff f769ec40 c03c8a35 c0122076  
> c0341b2b c16fe1e0 00000001
> [ 7565.309992]        c0121c8a c0341a07 f769ea80 f769ea80 f769ea80  
> 00000000 c036e10f 000000d0
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
> [ 7565.309992]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
> [ 7565.309992]  [<c036e10f>] inet_csk_accept+0x12f/0x260
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c038ca4f>] inet_accept+0x1f/0xd0
> [ 7565.309992]  [<c0340a30>] sys_accept+0xf0/0x210
> [ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
> [ 7565.309992]  [<c0340c16>] sys_socketcall+0xc6/0x280
> [ 7565.309992]  [<c0199ea7>] filp_close+0x47/0x80
> [ 7565.309992]  [<c019b5e2>] sys_close+0x82/0xf0
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] sshd          S f7f22edc     0  9291   2335
> [ 7565.309992]        f7f22d80 00000086 00000000 f7f22edc c048bb60  
> 00000000 f7f4d410 00000008
> [ 7565.309992]        00000080 00000000 00000000 c03c8a35 f7f04800  
> f7f4d400 c02b4f20 c02b0836
> [ 7565.309992]        f7679be4 f7f4d410 00000007 00000080 00000008  
> 00000080 c01a9b22 f7679be4
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c02b4f20>] normal_poll+0x0/0x150
> [ 7565.309992]  [<c02b0836>] tty_poll+0x76/0x80
> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c034e10d>] dev_queue_xmit+0xcd/0x400
> [ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
> [ 7565.309992]  [<c0352c55>] neigh_resolve_output+0xf5/0x270
> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
> [ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
> [ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
> [ 7565.309992]  [<c0367ec5>] ip_local_out+0x15/0x20
> [ 7565.309992]  [<c036a25e>] ip_queue_xmit+0x1ae/0x320
> [ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
> [ 7565.309992]  [<c037fa80>] tcp_v4_send_check+0x40/0xe0
> [ 7565.309992]  [<c037a84f>] tcp_transmit_skb+0x3bf/0x7a0
> [ 7565.309992]  [<c0121f3f>] irq_exit+0x5f/0x90
> [ 7565.309992]  [<c010f5a0>] smp_apic_timer_interrupt+0x30/0x70
> [ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
> [ 7565.309992]  [<c0379fbc>] tcp_init_tso_segs+0x3c/0x60
> [ 7565.309992]  [<c037bfb9>] __tcp_push_pending_frames+0x139/0x750
> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
> [ 7565.309992]  [<c036f705>] sk_stream_alloc_skb+0x35/0xe0
> [ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
> [ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
> [ 7565.309992]  [<c0371082>] tcp_sendmsg+0x7a2/0xab0
> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
> [ 7565.309992]  [<c033ea22>] sock_aio_write+0xe2/0x100
> [ 7565.309992]  [<c01197b4>] try_to_wake_up+0x44/0xd0
> [ 7565.309992]  [<c011688e>] __wake_up_common+0x3e/0x70
> [ 7565.309992]  [<c019ba56>] do_sync_write+0xc6/0x110
> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
> [ 7565.309992]  [<c019c8e1>] sys_write+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] bash          S f7f238fc     0  9293   9291
> [ 7565.309992]        f7f237a0 00000086 c013501d f7f238fc 09e6bb40  
> 00989680 c0117063 f7f04580
> [ 7565.309992]        f7f4c000 7fffffff 7fffffff c03c8a35 00000200  
> f7f237a0 f7f0e900 f7f0e900
> [ 7565.309992]        f7f237a0 c013166d f7f04580 f7f4c000 f7f04580  
> f7f4c000 c02b6254 00000003
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c013501d>] ktime_get+0xd/0x30
> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] remoteHost    S f7c8281c     0 11799   2507
> [ 7565.309992]        f7c826c0 00000086 00000200 f7c8281c c04f4300  
> 031703f8 c013166d f7592e84
> [ 7565.309992]        f7c826b8 00000004 00000001 c011f60d f7592d80  
> f7c826c0 f7c56300 00000000
> [ 7565.309992]        00000000 f7c827b0 f74b6c80 00000000 00000000  
> f7c826c0 00000001 f7592d80
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309992]  [<c011f60d>] do_wait+0x63d/0xc80
> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309992]  [<c011fcc3>] sys_wait4+0x73/0xc0
> [ 7565.309992]  [<c011fd35>] sys_waitpid+0x25/0x30
> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309992]  =======================
> [ 7565.309992] remoteHost    S f7c83c5c     0 11800   2507
> [ 7565.309992]        f7c83b00 00000082 c03797ed f7c83c5c 00000200  
> 00000000 00000000 f7653d7c
> [ 7565.309992]        f75a4880 f75a48e0 f75a4b14 c03c8a35 f74f4620  
> c0341b2b 00000000 f74f4600
> [ 7565.309992]        c0121c8a c0341a07 f7653d7c f7653d7c f7653d7c  
> f75a4880 c0342564 00000000
> [ 7565.309992] Call Trace:
> [ 7565.309992]  [<c03797ed>] tcp_rcv_established+0x3ed/0x6d0
> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
> [ 7565.309993]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
> [ 7565.309993]  [<c0121c8a>] local_bh_disable+0xa/0x20
> [ 7565.309993]  [<c0341a07>] release_sock+0xc7/0xf0
> [ 7565.309993]  [<c0342564>] sk_wait_data+0x84/0xd0
> [ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309993]  [<c037173f>] tcp_recvmsg+0x3af/0x8a0
> [ 7565.309993]  [<c0134fe2>] ktime_get_ts+0x22/0x50
> [ 7565.309993]  [<c0117063>] hrtick_start_fair+0xa3/0x150
> [ 7565.309993]  [<c0341425>] sock_common_recvmsg+0x45/0x70
> [ 7565.309993]  [<c033f6c7>] sock_recvmsg+0xd7/0x110
> [ 7565.309993]  [<c02b5324>] n_tty_receive_buf+0x2b4/0x1020
> [ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309993]  [<c02b51c5>] n_tty_receive_buf+0x155/0x1020
> [ 7565.309993]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
> [ 7565.309993]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
> [ 7565.309993]  [<c033ef82>] sockfd_lookup_light+0x32/0x60
> [ 7565.309993]  [<c03406ce>] sys_recvfrom+0xce/0x160
> [ 7565.309993]  [<c0117e30>] hrtick_set+0xc0/0x1a0
> [ 7565.309993]  [<c0119a1b>] hrtick_resched+0x4b/0x60
> [ 7565.309993]  [<c010260f>] do_notify_resume+0x6f/0x790
> [ 7565.309993]  [<c02aedba>] tty_put_char+0x2a/0x40
> [ 7565.309993]  [<c03ca20a>] rwsem_down_write_failed+0x2a/0x30
> [ 7565.309993]  [<c0119aaf>] __wake_up+0x7f/0xd0
> [ 7565.309993]  [<c0340793>] sys_recv+0x33/0x40
> [ 7565.309993]  [<c0340cb5>] sys_socketcall+0x165/0x280
> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309993]  =======================
> [ 7565.309993] rt61pci       S f7f4a81c     0 15243      2
> [ 7565.309993]        f7f4a6c0 00000046 c012d8c0 f7f4a81c c04762e0  
> c01315a5 00000200 f7644b40
> [ 7565.309993]        f7644b48 00000000 00000000 c012e1bd 00000000  
> f7f4a6c0 c01312c0 f7644b48
> [ 7565.309993]        f7644b48 fffffffc f7644b40 c012e130 c0130f32  
> c0130ef0 00000000 00000000
> [ 7565.309993] Call Trace:
> [ 7565.309993]  [<c012d8c0>] run_workqueue+0x120/0x1b0
> [ 7565.309993]  [<c01315a5>] prepare_to_wait+0x85/0xe0
> [ 7565.309993]  [<c012e1bd>] worker_thread+0x8d/0xa0
> [ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
> [ 7565.309993]  [<c012e130>] worker_thread+0x0/0xa0
> [ 7565.309993]  [<c0130f32>] kthread+0x42/0x70
> [ 7565.309993]  [<c0130ef0>] kthread+0x0/0x70
> [ 7565.309993]  [<c0103db3>] kernel_thread_helper+0x7/0x14
> [ 7565.309993]  =======================
> [ 7565.309993] actuatorTask  ? f759281c     0 15816      1
> [ 7565.309993]        f75926c0 00000046 400000c0 f759281c f75926c0  
> 00000000 00003dc8 00000010
> [ 7565.309993]        f75926c0 f75b1edc 00000000 c01202bc c0155590  
> c04844a4 00000246 00000001
> [ 7565.309993]        f75b1f90 f759276c f75927b0 00000000 00000000  
> f75b1f90 f75b1f90 f7fdbe00
> [ 7565.309993] Call Trace:
> [ 7565.309993]  [<c01202bc>] do_exit+0x4bc/0x730
> [ 7565.309993]  [<c0155590>] sigwake_event+0x0/0x140
> [ 7565.309993]  [<c012055f>] do_group_exit+0x2f/0xc0
> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309993]  =======================
> [ 7565.309993] Actuator Aper S f7592b7c     0 15817      1
> [ 7565.309993]        f7592a20 00000086 00000001 f7592b7c 00000001  
> 00000000 00000200 f7e60620
> [ 7565.309993]        f7592a20 f7729fb8 00000170 c01560ad 00000010  
> f7729f5c 00000001 00000006
> [ 7565.309993]        00000000 f7729fb8 00000170 c015628f 00000001  
> c0577b04 c0577b00 00000000
> [ 7565.309993] Call Trace:
> [ 7565.309993]  [<c01560ad>] xnshadow_harden+0x8d/0x1e0
> [ 7565.309993]  [<c015628f>] losyscall_event+0x8f/0x170
> [ 7565.309993]  [<c014a1e2>] __ipipe_dispatch_event+0xa2/0x190
> [ 7565.309993]  [<c0156200>] losyscall_event+0x0/0x170
> [ 7565.309993]  [<c0112731>] __ipipe_syscall_root+0x41/0x100
> [ 7565.309993]  [<c010311d>] system_call+0x29/0x4a
> [ 7565.309993]  =======================
> [ 7565.309993] sh            S f7592edc     0 15832  11799
> [ 7565.309993]        f7592d80 00000086 f7c56944 f7592edc f7f36080  
> c0185d00 c013166d ffffffea
> [ 7565.309993]        f7592d78 00000004 00000001 c011f60d f7593b00  
> f7c56900 f7c56300 00000001
> [ 7565.309993]        00000000 f7592e70 00000000 00000003 0000000c  
> f7592d80 00000001 f7593b00
> [ 7565.309993] Call Trace:
> [ 7565.309993]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
> [ 7565.309993]  [<c013166d>] add_wait_queue+0x6d/0xb0
> [ 7565.309993]  [<c011f60d>] do_wait+0x63d/0xc80
> [ 7565.309993]  [<c0119840>] default_wake_function+0x0/0x10
> [ 7565.309993]  [<c011fcc3>] sys_wait4+0x73/0xc0
> [ 7565.309993]  [<c011fd35>] sys_waitpid+0x25/0x30
> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309993]  =======================
> [ 7565.309993] sleep         S f7593c5c     0 16596  15832
> [ 7565.309993]        f7593b00 00000086 4b3f0cce f7593c5c 00000000  
> 00000200 00000000 f7659f60
> [ 7565.309993]        f7593b00 00000001 00000000 c03c91bb 00000001  
> 00000000 00000001 c0134ef5
> [ 7565.309993]        f7fa5f61 f754c740 c0576960 86d9d6ce 000006e1  
> c0134a60 c047f254 00000001
> [ 7565.309993] Call Trace:
> [ 7565.309993]  [<c03c91bb>] do_nanosleep+0x5b/0x90
> [ 7565.309993]  [<c0134ef5>] hrtimer_nanosleep+0x45/0xb0
> [ 7565.309993]  [<c0134a60>] hrtimer_wakeup+0x0/0x20
> [ 7565.309993]  [<c0134fb8>] sys_nanosleep+0x58/0x60
> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
> [ 7565.309993]  =======================
>
>
> On Nov 28, 2008, at 7:11 PM, Gilles Chanteperdrix wrote:
>
>> Mehmet Alphan Ulusoy wrote:
>>> ps aux says Task's state is "Zl+" and it's there for over 6 hours  
>>> now,
>>> is init's mentioned behavior standard or somewhat optional?
>>>
>>> If it's standard, then does this mean that there's something else  
>>> going on?
>>
>> If you are able to reproduce this issue with a simple program,  
>> could you
>> send it to us?
>>
>> -- 
>>                                                 Gilles.
>>
>
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help


[-- Attachment #1.2: Type: text/html, Size: 157920 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2433 bytes --]

Re: [Xenomai-help] Zombie user tasks

[Alphan Ulusoy]

[-- Attachment #1.1: Type: text/plain, Size: 72703 bytes --]

I've recompiled the kernel after turning off the preemptible RCU  
option. I left CONFIG_PREEMPT as it is for the time being. Can  
preemptible RCU option be the cause ?

alphan

On Dec 21, 2008, at 10:05 PM, Alphan Ulusoy wrote:

> By the way, I've compiled my kernel w/ CONFIG_PREEMPT set. Can this  
> be the problem? Because I've come across some posts from 2006  
> pointing out this.
>
> regards,
>
> alphan
>
> On Dec 21, 2008, at 9:28 PM, Alphan Ulusoy wrote:
>
>> Hello again,
>>
>> Unfortunately I couldn't solve this issue of 'tasks turning into  
>> zombies after being killed' on my own. Below you may find the sysRq  
>> + T output. The name of the problematic task is:  "actuatorTask".  
>> This is a bit hard to reproduce since it does not happen all the  
>> time, i hope below output helps.
>>
>> Also /proc/xenomai/stat says:
>>
>> CPU  PID    MSW        CSW        PF    STAT       %CPU  NAME
>>   0  0      0          942268     0     00500080   99.9  ROOT
>>   0  15817  2899       2906       0     00300380    0.0  Actuator  
>> Aperiodic Task
>>   0  0      0          2275392    0     00000000    0.0  IRQ233:  
>> [timer]
>>
>>
>> and /proc/xenomai/sched says:
>>
>> CPU  PID    PRI      PERIOD     TIMEOUT    TIMEBASE  STAT       NAME
>>   0  0       80      0          0          master    R          ROOT
>>   0  15817   80      0          0          master    X           
>> Actuator Aperiodic Task
>>
>>
>> [ 7565.309991] SysRq : Show State
>> [ 7565.309991]   task                PC stack   pid father
>> [ 7565.309991] init          S f7c2a15c     0     1      0
>> [ 7565.309991]        f7c2a000 00000086 006eec77 f7c2a15c c0575380  
>> c0126991 00000200 f7c2db64
>> [ 7565.309991]        006eec77 00000000 00000000 c03c8a0a c013166d  
>> f7c2dbe4 c0575ce8 c0575ce8
>> [ 7565.309991]        006eec77 c01265f0 f7c2a000 c0575380 0000000b  
>> 00000400 c01a9b22 f7c2dbe4
>> [ 7565.309991] Call Trace:
>> [ 7565.309991]  [<c0126991>] __mod_timer+0xb1/0xf0
>> [ 7565.309991]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
>> [ 7565.309991]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309991]  [<c01265f0>] process_timeout+0x0/0x10
>> [ 7565.309991]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309991]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309991]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
>> [ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
>> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
>> [ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
>> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
>> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
>> [ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
>> [ 7565.309991]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309991]  [<c01088a6>] read_tsc+0x6/0x30
>> [ 7565.309991]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309991]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
>> [ 7565.309991]  [<c029ab23>] number+0x2d3/0x2e0
>> [ 7565.309991]  [<c0313538>] usb_choose_configuration+0x198/0x220
>> [ 7565.309991]  [<c01354cb>] hrtimer_interrupt+0x19b/0x1e0
>> [ 7565.309991]  [<c029b3b7>] vsnprintf+0x307/0x5e0
>> [ 7565.309991]  [<c014978f>] __xirq_end+0x0/0x46
>> [ 7565.309991]  [<c0139989>] clocksource_get_next+0x59/0x90
>> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
>> [ 7565.309991]  [<c0137d69>] update_wall_time+0x259/0x7d0
>> [ 7565.309991]  [<c0147961>] __rcu_read_unlock+0x41/0x90
>> [ 7565.309991]  [<c01aea09>] __d_lookup+0xc9/0x160
>> [ 7565.309991]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309991]  [<c01a397a>] permission+0x6a/0x110
>> [ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309991]  [<c0197794>] __slab_alloc+0x74/0x510
>> [ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309991]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309991]  [<c01b4183>] mntput_no_expire+0x13/0x120
>> [ 7565.309991]  [<c01a606e>] path_walk+0x4e/0x90
>> [ 7565.309991]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309991]  [<c029c6c5>] copy_to_user+0x35/0x60
>> [ 7565.309991]  [<c019f488>] cp_new_stat64+0xf8/0x110
>> [ 7565.309991]  [<c01aa382>] sys_select+0xe2/0x1b0
>> [ 7565.309991]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309991]  =======================
>> [ 7565.309991] kthreadd      S f7c2a4bc     0     2      0
>> [ 7565.309991]        f7c2a360 00000046 f7c2a4c0 f7c2a4bc 00000000  
>> c0119c46 f766fde8 c047f1f0
>> [ 7565.309991]        00003dba f766fdc4 00000000 c0131105 00000000  
>> c0130f60 00000000 00000000
>> [ 7565.309991]        00000000 c0103db3 00000000 00000000 00000000  
>> 00000000 00000000 00000000
>> [ 7565.309991] Call Trace:
>> [ 7565.309991]  [<c0119c46>] complete+0x76/0xb0
>> [ 7565.309992]  [<c0131105>] kthreadd+0x1a5/0x1d0
>> [ 7565.309992]  [<c0130f60>] kthreadd+0x0/0x1d0
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] ksoftirqd/0   S f7c2a81c     0     3      2
>> [ 7565.309992]        f7c2a6c0 00000046 c0575180 f7c2a81c c0121d9a  
>> 00000000 00000000 00000000
>> [ 7565.309992]        00000000 c0122120 00000000 c0122219 fffffffc  
>> c0130f32 c0130ef0 00000000
>> [ 7565.309992]        00000000 c0103db3 f7c2df20 00000000 00000000  
>> 00000000 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
>> [ 7565.309992]  [<c0122120>] ksoftirqd+0x0/0x180
>> [ 7565.309992]  [<c0122219>] ksoftirqd+0xf9/0x180
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] events/0      R running      0     4      2
>> [ 7565.309992]        f7c2aa20 00000046 c012d8c0 f7c2ab7c c04762e0  
>> c01315a5 00000200 f7c08100
>> [ 7565.309992]        f7c08108 00000000 00000000 c012e1bd 00000000  
>> f7c2aa20 c01312c0 f7c37fc0
>> [ 7565.309992]        f7c37fc0 fffffffc f7c08100 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] khelper       S f7c2aedc     0     5      2
>> [ 7565.309992]        f7c2ad80 00000046 c012d8c0 f7c2aedc f7f22d80  
>> c01315a5 00000200 f7c08180
>> [ 7565.309992]        f7c08188 00000000 00000000 c012e1bd 00000000  
>> f7c2ad80 c01312c0 f7c08188
>> [ 7565.309992]        f7c08188 fffffffc f7c08180 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] kblockd/0     S f7c8323c     0    52      2
>> [ 7565.309992]        f7c830e0 00000046 c012d8c0 f7c8323c c04762e0  
>> c01315a5 00000200 f7c19b80
>> [ 7565.309992]        f7c19b88 00000000 00000000 c012e1bd 00000000  
>> f7c830e0 c01312c0 f7c19b88
>> [ 7565.309992]        f7c19b88 fffffffc f7c19b80 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c012d8c0>] run_workqueue+0x120/0x1b0
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] khubd         S f7cd123c     0    61      2
>> [ 7565.309992]        f7cd10e0 00000046 000003e8 f7cd123c f7c9ad9c  
>> c01315a5 00000200 f7cddfa4
>> [ 7565.309992]        f7c9ad80 00000000 00000003 c0306cb4 f7cddfbc  
>> 00000000 f7cd1108 0cb15b1c
>> [ 7565.309992]        00000000 00000239 f75e7d04 f7eeb21c f75e7c00  
>> f7c9f01c 00000000 00000002
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c0306cb4>] hub_thread+0x674/0xd10
>> [ 7565.309992]  [<c0117dfe>] hrtick_set+0x8e/0x1a0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c0306640>] hub_thread+0x0/0xd10
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] kseriod       S f7cd1c5c     0    64      2
>> [ 7565.309992]        f7cd1b00 00000046 c0197c85 f7cd1c5c 00000000  
>> c01315a5 00000200 f7ce3fb0
>> [ 7565.309992]        00000000 00000000 00000200 c0322a7f c0476160  
>> 00000000 f7cd1b00 c03c8219
>> [ 7565.309992]        00000000 f7cd1b00 c01312c0 c04ae2d8 c04ae2d8  
>> fffffffc 00000000 c0322970
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0197c85>] kfree+0x55/0x110
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c0322a7f>] serio_thread+0x10f/0x430
>> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c0322970>] serio_thread+0x0/0x430
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] gatekeeper/0  S f7e4e4bc     0   148      2
>> [ 7565.309992]        f7e4e360 00000046 00000000 f7e4e4bc c057c540  
>> c0577b00 c0581a2c f7e61a20
>> [ 7565.309992]        c0581a20 c0581a2c 00000000 c0156a77 00000000  
>> f7e4e360 c03c8219 00000001
>> [ 7565.309992]        f7e4e360 c0119840 c0581a24 c0581a24 fffffffc  
>> c0581a20 c01569d0 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0156a77>] gatekeeper_thread+0xa7/0x130
>> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c01569d0>] gatekeeper_thread+0x0/0x130
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] pdflush       S f7e4f23c     0   152      2
>> [ 7565.309992]        f7e4f0e0 00000046 f7e4f0e0 f7e4f23c c03c85be  
>> c0119684 00000200 f7e5dfc4
>> [ 7565.309992]        00000000 c017d200 00000000 c017d2d4 f7e4f23c  
>> f7e4f0e0 00000000 00000000
>> [ 7565.309992]        c0485e54 f7e5ffc4 fffb6d17 fffffffc c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c85be>] preempt_schedule+0x4e/0x70
>> [ 7565.309992]  [<c0119684>] set_user_nice+0x114/0x170
>> [ 7565.309992]  [<c017d200>] pdflush+0x0/0x270
>> [ 7565.309992]  [<c017d2d4>] pdflush+0xd4/0x270
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] pdflush       S f7e4f59c     0   153      2
>> [ 7565.309992]        f7e4f440 00000046 00000000 f7e4f59c 00000000  
>> 00000000 00000025 f7e5ffc4
>> [ 7565.309992]        00000000 c017d200 00000000 c017d2d4 f7e4f59c  
>> f7e4f440 00000000 00000000
>> [ 7565.309992]        f7e5dfc4 c0485e54 006edb25 fffffffc c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c017d200>] pdflush+0x0/0x270
>> [ 7565.309992]  [<c017d2d4>] pdflush+0xd4/0x270
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] kswapd0       S f7e4f8fc     0   154      2
>> [ 7565.309992]        f7e4f7a0 00000046 00000000 f7e4f8fc 00000000  
>> c01315a5 00000200 00000000
>> [ 7565.309992]        c0485340 c01312c0 00000000 c018119b 00000000  
>> 00000000 f7eb9f74 f7e4f7c8
>> [ 7565.309992]        c0485de8 00000000 00000203 c0116ebb f7e4f7a0  
>> c0117160 c0180de0 f7e4f7c8
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c018119b>] kswapd+0x3bb/0x430
>> [ 7565.309992]  [<c0116ebb>] update_curr+0x6b/0x70
>> [ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
>> [ 7565.309992]  [<c0180de0>] kswapd+0x0/0x430
>> [ 7565.309992]  [<c011796b>] dequeue_task_fair+0x2b/0x100
>> [ 7565.309992]  [<c0117dfe>] hrtick_set+0x8e/0x1a0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
>> [ 7565.309992]  [<c0180de0>] kswapd+0x0/0x430
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] aio/0         S f7f4b23c     0   197      2
>> [ 7565.309992]        f7f4b0e0 00000046 c04f4338 f7f4b23c 00000000  
>> c01315a5 00000200 f7c19a80
>> [ 7565.309992]        f7c19a88 00000000 00000000 c012e1bd 00000000  
>> f7f4b0e0 c01312c0 f7c19a88
>> [ 7565.309992]        f7c19a88 fffffffc f7c19a80 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] unionfs_siod/ S f7f9415c     0   210      2
>> [ 7565.309992]        f7f94000 00000046 c04f4338 f7f9415c 00000000  
>> c01315a5 00000200 f7c190c0
>> [ 7565.309992]        f7c190c8 00000000 00000000 c012e1bd 00000000  
>> f7f94000 c01312c0 f7c190c8
>> [ 7565.309992]        f7c190c8 fffffffc f7c190c0 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] nfsiod        S f7f944bc     0   211      2
>> [ 7565.309992]        f7f94360 00000046 c04f4338 f7f944bc 00000000  
>> c01315a5 00000200 f7c08a80
>> [ 7565.309992]        f7c08a88 00000000 00000000 c012e1bd 00000000  
>> f7f94360 c01312c0 f7c08a88
>> [ 7565.309992]        f7c08a88 fffffffc f7c08a80 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] scsi_eh_0     S f7c82edc     0   874      2
>> [ 7565.309992]        f7c82d80 00000046 c047f254 f7c82edc f7c82d80  
>> c0117160 00000000 fffffffc
>> [ 7565.309992]        f7caa000 00000000 f7cb5fbc c02f4c7a 00000000  
>> c0117e30 00000000 ffffffff
>> [ 7565.309992]        ffffffff 00000200 f7c82d80 c0476160 00000000  
>> f7c82d80 c03c8219 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
>> [ 7565.309992]  [<c02f4c7a>] scsi_error_handler+0x4a/0x3b0
>> [ 7565.309992]  [<c0117e30>] hrtick_set+0xc0/0x1a0
>> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
>> [ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
>> [ 7565.309992]  [<c02f4c30>] scsi_error_handler+0x0/0x3b0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] usb-storage   S f7c838fc     0   875      2
>> [ 7565.309992]        f7c837a0 00000046 c04f4300 f7c838fc c0134fe2  
>> 00000000 00000000 f7c837a0
>> [ 7565.309992]        7fffffff 00040000 00000000 c03c8a35 0000049b  
>> 00000000 00989680 00000000
>> [ 7565.309992]        c0116e40 f7c2a6e8 f7c2a6c0 f7c837a0 f7c837a0  
>> 7fffffff c03c94a9 f7caa39c
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0116e40>] wakeup_preempt_entity+0x70/0x80
>> [ 7565.309992]  [<c03c94a9>] __down_interruptible+0x69/0xe0
>> [ 7565.309992]  [<c0135f97>] down_interruptible+0x87/0xa0
>> [ 7565.309992]  [<c0320c80>] usb_stor_control_thread+0x60/0x240
>> [ 7565.309992]  [<c03c8219>] schedule+0x219/0x430
>> [ 7565.309992]  [<c0119c46>] complete+0x76/0xb0
>> [ 7565.309992]  [<c0320c20>] usb_stor_control_thread+0x0/0x240
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] kpsmoused     S f7cd04bc     0   887      2
>> [ 7565.309992]        f7cd0360 00000046 00000000 f7cd04bc ffffffff  
>> c01315a5 00000200 f7c9b880
>> [ 7565.309992]        f7c9b888 00000000 00000000 c012e1bd 00000000  
>> f7cd0360 c01312c0 f7c9b888
>> [ 7565.309992]        f7c9b888 fffffffc f7c9b880 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] rpciod/0      S f7cd18fc     0   898      2
>> [ 7565.309992]        f7cd17a0 00000046 00000000 f7cd18fc ffffffff  
>> c01315a5 00000200 f7c9b6c0
>> [ 7565.309992]        f7c9b6c8 00000000 00000000 c012e1bd 00000000  
>> f7cd17a0 c01312c0 f7c9b6c8
>> [ 7565.309992]        f7c9b6c8 fffffffc f7c9b6c0 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] kjournald     S f7e4e81c     0   902      2
>> [ 7565.309992]        f7e4e6c0 00000046 c0119aaf f7e4e81c 00000000  
>> c01315a5 00000200 f7fb8500
>> [ 7565.309992]        00000000 f7fb85c4 f7fb8540 c01f9da5 00000000  
>> 00000005 f7fb8550 00000000
>> [ 7565.309992]        f7e4e6c0 c01312c0 f7fb8550 f7fb8550 fffffffc  
>> f7fb8500 c01f9b80 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] udevd         S f7c2b23c     0  1004      1
>> [ 7565.309992]        f7c2b0e0 00000082 808fe474 f7c2b23c f7f94d80  
>> f7f4fb3c c013166d 00000008
>> [ 7565.309992]        00000080 00000000 00000000 c03c8a35 c013166d  
>> c013166d f7f4fbe4 c03c8d50
>> [ 7565.309992]        f7461060 00000041 c01ca515 00000007 00000008  
>> 00000080 c01a9b22 f7f4fbe4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
>> [ 7565.309992]  [<c01ca515>] inotify_poll+0x45/0x60
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
>> [ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
>> [ 7565.309992]  [<c017c5b9>] balance_dirty_pages_ratelimited_nr 
>> +0x59/0x2c0
>> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
>> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
>> [ 7565.309992]  [<c019304f>] shmem_getpage+0x4ef/0x960
>> [ 7565.309992]  [<c0183803>] do_wp_page+0x1f3/0x4c0
>> [ 7565.309992]  [<c017ba76>] set_page_dirty+0x46/0xd0
>> [ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
>> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
>> [ 7565.309992]  [<c01128d7>] __ipipe_handle_exception+0xe7/0x1e0
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
>> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
>> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309992]  [<c01ae2ad>] dput+0x7d/0x190
>> [ 7565.309992]  [<c01a3c9b>] __follow_mount+0x1b/0x70
>> [ 7565.309992]  [<c01a3e45>] do_lookup+0x65/0x1a0
>> [ 7565.309992]  [<c0195030>] shmem_permission+0x0/0x10
>> [ 7565.309992]  [<c01a397a>] permission+0x6a/0x110
>> [ 7565.309992]  [<c01a5237>] __link_path_walk+0x67/0xe50
>> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
>> [ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
>> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
>> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c01317d0>] remove_wait_queue+0x70/0xb0
>> [ 7565.309992]  [<c011f38c>] do_wait+0x3bc/0xc80
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309992]  [<c01a69e9>] do_rmdir+0xb9/0x100
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] kjournald     S f7f4a4bc     0  2073      2
>> [ 7565.309992]        f7f4a360 00000046 c0119aaf f7f4a4bc 00000000  
>> c01315a5 00000200 f745a400
>> [ 7565.309992]        00000000 f745a4c4 f745a440 c01f9da5 00000000  
>> 00000005 f745a450 00000000
>> [ 7565.309992]        f7f4a360 c01312c0 f745a450 f745a450 fffffffc  
>> f745a400 c01f9b80 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] kjournald     S f7f9481c     0  2080      2
>> [ 7565.309992]        f7f946c0 00000046 c0119aaf f7f9481c 00000000  
>> c01315a5 00000200 f745a200
>> [ 7565.309992]        00000000 f745a2c4 f745a240 c01f9da5 00000000  
>> 00000005 f745a250 00000000
>> [ 7565.309992]        f7f946c0 c01312c0 f745a250 f745a250 fffffffc  
>> f745a200 c01f9b80 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0119aaf>] __wake_up+0x7f/0xd0
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c01f9da5>] kjournald+0x225/0x270
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c01f9b80>] kjournald+0x0/0x270
>> [ 7565.309992]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309992]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309992]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309992]  =======================
>> [ 7565.309992] dhclient3     S f7c2b8fc     0  2153      1
>> [ 7565.309992]        f7c2b7a0 00000086 0071a227 f7c2b8fc c0575380  
>> c0126991 00000200 f7633b64
>> [ 7565.309992]        0071a227 00000000 00000000 c03c8a0a c013166d  
>> f74a3a00 c0575db8 c0575db8
>> [ 7565.309992]        0071a227 c01265f0 f7c2b7a0 c0575380 00000006  
>> 00000020 c01a9b22 f7633be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
>> [ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
>> [ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
>> [ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
>> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
>> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
>> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
>> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
>> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
>> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309992]  [<c0196f6e>] deactivate_slab+0x5e/0x180
>> [ 7565.309992]  [<c0197794>] __slab_alloc+0x74/0x510
>> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
>> [ 7565.309992]  [<c0139989>] clocksource_get_next+0x59/0x90
>> [ 7565.309992]  [<c0137d69>] update_wall_time+0x259/0x7d0
>> [ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
>> [ 7565.309992]  [<c0341e48>] sock_alloc_send_skb+0x168/0x1b0
>> [ 7565.309992]  [<c011941d>] task_rq_lock+0x2d/0x40
>> [ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
>> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309992]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
>> [ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
>> [ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
>> [ 7565.309992]  [<c0121cc8>] _local_bh_enable+0x28/0x90
>> [ 7565.309992]  [<c0121e45>] do_softirq+0x65/0x70
>> [ 7565.309992]  [<c0121f3f>] irq_exit+0x5f/0x90
>> [ 7565.309992]  [<c010f5a0>] smp_apic_timer_interrupt+0x30/0x70
>> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
>> [ 7565.309992]  [<c010318c>] restore_nocheck_notrace+0x0/0xe
>> [ 7565.309992]  [<c010f570>] smp_apic_timer_interrupt+0x0/0x70
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c014978f>] __xirq_end+0x0/0x46
>> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
>> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
>> [ 7565.309992]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
>> [ 7565.309992]  [<c0183803>] do_wp_page+0x1f3/0x4c0
>> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
>> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
>> [ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
>> [ 7565.309992]  [<c029c6c5>] copy_to_user+0x35/0x60
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] syslogd       S f7f95c5c     0  2294      1
>> [ 7565.309992]        f7f95b00 00000086 c01e6c14 f7f95c5c f7f957a0  
>> c01f54f6 c013166d 00000001
>> [ 7565.309992]        00000001 00000000 00000000 c03c8a35 00000001  
>> f7437be4 f7ee8000 011a92de
>> [ 7565.309992]        c03f2be0 00000001 f7ee8000 00000000 00000001  
>> 00000001 c01a9b22 f7437be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
>> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
>> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
>> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
>> [ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
>> [ 7565.309992]  [<c01e7187>] ext3_mark_inode_dirty+0x37/0x50
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
>> [ 7565.309992]  [<c017c5b9>] balance_dirty_pages_ratelimited_nr 
>> +0x59/0x2c0
>> [ 7565.309992]  [<c01e9372>] ext3_ordered_write_end+0xe2/0x140
>> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
>> [ 7565.309992]  [<c0175ec5>] generic_file_buffered_write+0x1c5/0x690
>> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
>> [ 7565.309992]  [<c03a1f04>] unix_dgram_recvmsg+0x184/0x2a0
>> [ 7565.309992]  [<c01747d5>] remove_suid+0x15/0x60
>> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
>> [ 7565.309992]  [<c01765dd>] __generic_file_aio_write_nolock+0x24d/ 
>> 0x560
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c01a2ce0>] pipe_write+0x0/0x470
>> [ 7565.309992]  [<c019b94f>] do_sync_readv_writev+0xbf/0x100
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c019f488>] cp_new_stat64+0xf8/0x110
>> [ 7565.309992]  [<c029c413>] copy_from_user+0x33/0x80
>> [ 7565.309992]  [<c019b7de>] rw_copy_check_uvector+0x7e/0x100
>> [ 7565.309992]  [<c019c0aa>] do_readv_writev+0xaa/0x190
>> [ 7565.309992]  [<c01a2ce0>] pipe_write+0x0/0x470
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c01273b4>] sigprocmask+0x74/0x100
>> [ 7565.309992]  [<c0129fc9>] sys_rt_sigprocmask+0xf9/0x120
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] klogd         R running      0  2300      1
>> [ 7565.309992]        f7f957a0 00000082 00000000 f7f958fc 00000000  
>> c01315a5 00000200 f76b7f34
>> [ 7565.309992]        c01312c0 00000fff 0804d6a0 c011da47 f76b7f1c  
>> f76b7f1c 00000000 00000000
>> [ 7565.309992]        f76b7e80 494e9495 00000000 f7f957a0 c01312c0  
>> f76b7f40 f76b7f40 f7c0c300
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c011da47>] do_syslog+0x437/0x4c0
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c01dbae0>] kmsg_read+0x0/0x50
>> [ 7565.309992]  [<c01d2990>] proc_reg_read+0x80/0xe0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c01d2910>] proc_reg_read+0x0/0xe0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] inetd         S f7c8359c     0  2310      1
>> [ 7565.309992]        f7c83440 00000086 f794c000 f7c8359c f7fb8500  
>> c18de7a0 000fa013 00000006
>> [ 7565.309992]        00000020 00000000 00000000 c03c8a35 c013166d  
>> f7fdfbe4 f769c440 f7fc9c80
>> [ 7565.309992]        00000000 c036fb94 c03f2000 00000020 00000006  
>> 00000020 c01a9b22 f7fdfbe4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c036fb94>] tcp_poll+0x14/0x150
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
>> [ 7565.309992]  [<c035f51a>] __nla_reserve+0x2a/0x70
>> [ 7565.309992]  [<c035f579>] __nla_put+0x19/0x40
>> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
>> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
>> [ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
>> [ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
>> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
>> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c0177285>] generic_file_aio_read+0x5b5/0x620
>> [ 7565.309992]  [<c01776ab>] filemap_fault+0x14b/0x4d0
>> [ 7565.309992]  [<c0183ed8>] __do_fault+0x188/0x360
>> [ 7565.309992]  [<c0185a6f>] handle_mm_fault+0x10f/0x670
>> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] nmbd          S f7cd015c     0  2318      1
>> [ 7565.309992]        f7cd0000 00000086 006edf00 f7cd015c c0575380  
>> c0126991 00000200 f74c1b64
>> [ 7565.309992]        006edf00 00000000 00000000 c03c8a0a c013166d  
>> f74c1be4 c0575c80 c0575c80
>> [ 7565.309992]        006edf00 c01265f0 f7cd0000 c0575380 0000000c  
>> 00000800 c01a9b22 f74c1be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
>> [ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0369698>] ip_finish_output+0x1c8/0x2c0
>> [ 7565.309992]  [<c0343f8f>] __skb_clone+0x1f/0xd0
>> [ 7565.309992]  [<c0367fc8>] ip_cork_release+0x28/0x40
>> [ 7565.309992]  [<c03473fa>] skb_queue_tail+0x6a/0xb0
>> [ 7565.309992]  [<c034394c>] sock_def_readable+0x4c/0x80
>> [ 7565.309992]  [<c0342933>] sock_queue_rcv_skb+0xa3/0xe0
>> [ 7565.309992]  [<c0384697>] udp_queue_rcv_skb+0xc7/0x270
>> [ 7565.309992]  [<c0347c98>] skb_copy_and_csum_datagram+0x258/0x3c0
>> [ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
>> [ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
>> [ 7565.309992]  [<c03457f8>] __kfree_skb+0x8/0x80
>> [ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
>> [ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
>> [ 7565.309992]  [<c01986fc>] kmem_cache_free+0x7c/0xd0
>> [ 7565.309992]  [<c034860a>] skb_free_datagram+0xa/0x30
>> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
>> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
>> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
>> [ 7565.309992]  [<c0386324>] udp_recvmsg+0x1b4/0x330
>> [ 7565.309992]  [<c0341425>] sock_common_recvmsg+0x45/0x70
>> [ 7565.309992]  [<c033f6c7>] sock_recvmsg+0xd7/0x110
>> [ 7565.309992]  [<c0139989>] clocksource_get_next+0x59/0x90
>> [ 7565.309992]  [<c0137d69>] update_wall_time+0x259/0x7d0
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c015259e>] xnarch_next_htick_shot+0x4e/0x60
>> [ 7565.309992]  [<c03c8d50>] mutex_lock+0x10/0x20
>> [ 7565.309992]  [<c034cde7>] netdev_run_todo+0x27/0x270
>> [ 7565.309992]  [<c034a723>] __dev_get_by_name+0x73/0x90
>> [ 7565.309992]  [<c01a890f>] fasync_helper+0x7f/0x120
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01ab7ab>] __posix_lock_file+0x6b/0x570
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01acb5e>] fcntl_setlk64+0x4e/0x2b0
>> [ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
>> [ 7565.309992]  [<c01273b4>] sigprocmask+0x74/0x100
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] smbd          S f7cd0b7c     0  2320      1
>> [ 7565.309992]        f7cd0a20 00000082 c01bc400 f7cd0b7c f794c000  
>> 00000000 f7fb8500 00000015
>> [ 7565.309992]        00100000 00000000 00000000 c03c8a35 c013166d  
>> f7511be4 f7fbd780 f7535a00
>> [ 7565.309992]        00000000 c01a21c2 00000014 00100000 00000015  
>> 00100000 c01a9b22 f7511be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01bc400>] __mark_inode_dirty+0x30/0x1b0
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c01a21c2>] pipe_poll+0x32/0xc0
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
>> [ 7565.309992]  [<c02adef5>] extract_buf+0x75/0xe0
>> [ 7565.309992]  [<c01796ce>] __rmqueue+0x1e/0x1d0
>> [ 7565.309992]  [<c017995c>] rmqueue_bulk+0x6c/0x90
>> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
>> [ 7565.309992]  [<c01776ab>] filemap_fault+0x14b/0x4d0
>> [ 7565.309992]  [<c0183ed8>] __do_fault+0x188/0x360
>> [ 7565.309992]  [<c0185a6f>] handle_mm_fault+0x10f/0x670
>> [ 7565.309992]  [<c0197daf>] kmem_cache_alloc+0x5f/0xd0
>> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
>> [ 7565.309992]  [<c0197daf>] kmem_cache_alloc+0x5f/0xd0
>> [ 7565.309992]  [<c019d0bb>] alloc_file+0x3b/0x60
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c01a842b>] do_fcntl+0x2ab/0x310
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] smbd          S f7e4eedc     0  2330   2320
>> [ 7565.309992]        f7e4ed80 00000082 fffffff7 f7e4eedc 0000000e  
>> 00000009 c01a84dd 00000000
>> [ 7565.309992]        00000001 00000003 f7c8a000 c01294f0 c0103145  
>> 00000000 00000019 089f6fd8
>> [ 7565.309992]        00000001 00000003 bfec4fe8 0000001d 0000007b  
>> 0000007b c0100000 0000001d
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c01a84dd>] sys_fcntl64+0x4d/0x90
>> [ 7565.309992]  [<c01294f0>] sys_pause+0x10/0x20
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] sshd          S f7f4b8fc     0  2335      1
>> [ 7565.309992]        f7f4b7a0 00000082 f794c018 f7f4b8fc f7f22d80  
>> c18de7a0 000fa013 00000005
>> [ 7565.309992]        00000010 00000000 00000000 c03c8a35 c013166d  
>> f74d3be4 f769d540 f7fc9b00
>> [ 7565.309992]        00000000 c036fb94 c03f2000 00000010 00000005  
>> 00000010 c01a9b22 f74d3be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c036fb94>] tcp_poll+0x14/0x150
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
>> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
>> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
>> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
>> [ 7565.309992]  [<c01e6c14>] ext3_mark_iloc_dirty+0x174/0x330
>> [ 7565.309992]  [<c01e7187>] ext3_mark_inode_dirty+0x37/0x50
>> [ 7565.309992]  [<c01eefe4>] __ext3_journal_stop+0x24/0x50
>> [ 7565.309992]  [<c01bc400>] __mark_inode_dirty+0x30/0x1b0
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c029ab23>] number+0x2d3/0x2e0
>> [ 7565.309992]  [<c01e9372>] ext3_ordered_write_end+0xe2/0x140
>> [ 7565.309992]  [<c01b5825>] mnt_drop_write+0x95/0x180
>> [ 7565.309992]  [<c029b3b7>] vsnprintf+0x307/0x5e0
>> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
>> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
>> [ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
>> [ 7565.309992]  [<c017ab5b>] get_page_from_freelist+0x25b/0x590
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
>> [ 7565.309992]  [<c0134df6>] hrtimer_start+0xd6/0x190
>> [ 7565.309992]  [<c0117160>] __dequeue_entity+0x50/0xc0
>> [ 7565.309992]  [<c0117e30>] hrtick_set+0xc0/0x1a0
>> [ 7565.309992]  [<c0195acc>] add_partial+0x4c/0xa0
>> [ 7565.309992]  [<c0197192>] __slab_free+0x72/0x340
>> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
>> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
>> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
>> [ 7565.309992]  [<c0197cbf>] kfree+0x8f/0x110
>> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
>> [ 7565.309992]  [<c01a27de>] free_pipe_info+0xe/0x20
>> [ 7565.309992]  [<c01a288d>] pipe_release+0x7d/0xa0
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c0199ea7>] filp_close+0x47/0x80
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] proftpd       S f7c82b7c     0  2422      1
>> [ 7565.309992]        f7c82a20 00200086 006f461e f7c82b7c c0575380  
>> c0126991 00000200 f74e3b64
>> [ 7565.309992]        006f461e 00000000 00000000 c03c8a0a c013166d  
>> f74e3be4 c0575f70 c04b09d0
>> [ 7565.309992]        006f461e c01265f0 f7c82a20 c0575380 00000002  
>> 00000002 c01a9b22 f74e3be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0126991>] __mod_timer+0xb1/0xf0
>> [ 7565.309992]  [<c03c8a0a>] schedule_timeout+0x4a/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c01265f0>] process_timeout+0x0/0x10
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
>> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309992]  [<c015463f>] xntimer_start_aperiodic+0x10f/0x200
>> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
>> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
>> [ 7565.309992]  [<c013501d>] ktime_get+0xd/0x30
>> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
>> [ 7565.309992]  [<c01bfe6a>] __find_get_block_slow+0x7a/0x130
>> [ 7565.309992]  [<c01088a6>] read_tsc+0x6/0x30
>> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309992]  [<c01c0290>] __find_get_block+0x90/0x1c0
>> [ 7565.309992]  [<c01f367e>] __journal_file_buffer+0x6e/0x130
>> [ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
>> [ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
>> [ 7565.309992]  [<c01f367e>] __journal_file_buffer+0x6e/0x130
>> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
>> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
>> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
>> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
>> [ 7565.309992]  [<c01aea09>] __d_lookup+0xc9/0x160
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309992]  [<c029587a>] _atomic_dec_and_lock+0x3a/0x60
>> [ 7565.309992]  [<c01b4183>] mntput_no_expire+0x13/0x120
>> [ 7565.309992]  [<c01a606e>] path_walk+0x4e/0x90
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01a6e29>] __user_walk_fd+0x49/0x60
>> [ 7565.309992]  [<c019f6ef>] vfs_lstat_fd+0x1f/0x50
>> [ 7565.309992]  [<c01379cd>] getnstimeofday+0x3d/0xe0
>> [ 7565.309992]  [<c0134df6>] hrtimer_start+0xd6/0x190
>> [ 7565.309992]  [<c01aa382>] sys_select+0xe2/0x1b0
>> [ 7565.309992]  [<c0120fa0>] alarm_setitimer+0x30/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] cron          S f7f4b59c     0  2434      1
>> [ 7565.309992]        f7f4b440 00000082 01d7138e f7f4b59c 00000000  
>> 00000200 00000000 f7fa5f60
>> [ 7565.309992]        f7f4b440 00000000 bfd44e04 c03c91bb 00000001  
>> 00000000 00000001 c0134ef5
>> [ 7565.309992]        00000001 f7c3d740 f7659f60 fa1e6b8e 000006e2  
>> c0134a60 c047f254 00000001
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c91bb>] do_nanosleep+0x5b/0x90
>> [ 7565.309992]  [<c0134ef5>] hrtimer_nanosleep+0x45/0xb0
>> [ 7565.309992]  [<c0134a60>] hrtimer_wakeup+0x0/0x20
>> [ 7565.309992]  [<c0134fb8>] sys_nanosleep+0x58/0x60
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] xdm           S f7c824bc     0  2456      1
>> [ 7565.309992]        f7c82360 00000082 f794c000 f7c824bc f7fb8500  
>> c18de7a0 000fa016 00000005
>> [ 7565.309992]        00000010 00000000 00000000 c03c8a35 c0348524  
>> f799e900 f7444400 c0385d0f
>> [ 7565.309992]        00000000 c03f2060 00000010 f7444400 00000005  
>> 00000010 c01a9b22 f7637be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0348524>] datagram_poll+0x14/0xf0
>> [ 7565.309992]  [<c0385d0f>] udp_poll+0xf/0x100
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0177fd2>] mempool_alloc+0x32/0x180
>> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
>> [ 7565.309992]  [<c017dcdf>] activate_page+0x4f/0xe0
>> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
>> [ 7565.309992]  [<c0116066>] kunmap_atomic+0x26/0x60
>> [ 7565.309992]  [<c017ac12>] get_page_from_freelist+0x312/0x590
>> [ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
>> [ 7565.309992]  [<c01c0290>] __find_get_block+0x90/0x1c0
>> [ 7565.309992]  [<c017e733>] lru_cache_add_active+0x43/0x80
>> [ 7565.309992]  [<c0185d93>] handle_mm_fault+0x433/0x670
>> [ 7565.309992]  [<c01f3c86>] do_get_write_access+0x1e6/0x480
>> [ 7565.309992]  [<c01c03e2>] __getblk+0x22/0x220
>> [ 7565.309992]  [<c01e6eb1>] __ext3_get_inode_loc+0xe1/0x2e0
>> [ 7565.309992]  [<c01f54f6>] journal_dirty_metadata+0x76/0x120
>> [ 7565.309992]  [<c01f3312>] __ext3_journal_dirty_metadata+0x22/0x60
>> [ 7565.309992]  [<c01f3f49>] journal_get_write_access+0x29/0x40
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c01f3985>] journal_stop+0x135/0x250
>> [ 7565.309992]  [<c01eefe4>] __ext3_journal_stop+0x24/0x50
>> [ 7565.309992]  [<c01bc48a>] __mark_inode_dirty+0xba/0x1b0
>> [ 7565.309992]  [<c017007b>] __sem_open+0x23b/0x310
>> [ 7565.309992]  [<c01160b3>] kmap_atomic_prot+0x13/0xa0
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c0177285>] generic_file_aio_read+0x5b5/0x620
>> [ 7565.309992]  [<c017a72a>] free_hot_cold_page+0x15a/0x1e0
>> [ 7565.309992]  [<c018455b>] unmap_vmas+0x2db/0x500
>> [ 7565.309992]  [<c01855f5>] free_pgtables+0x85/0xb0
>> [ 7565.309992]  [<c01986c2>] kmem_cache_free+0x42/0xd0
>> [ 7565.309992]  [<c0187c33>] unmap_region+0xe3/0x120
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] getty         S f7f9559c     0  2468      1
>> [ 7565.309992]        f7f95440 00000086 00000720 f7f9559c f7c2aa20  
>> c1816a40 00000042 f7c89b80
>> [ 7565.309992]        f762b400 7fffffff 7fffffff c03c8a35 00000000  
>> c02ab166 c0135cee 00000001
>> [ 7565.309992]        001ae60e c013166d f7c89b80 f762b400 f7c89b80  
>> f762b400 c02b6254 00000003
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c02ab166>] vgacon_set_cursor_size+0xd6/0x1a0
>> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
>> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] getty         S f7f94b7c     0  2470      1
>> [ 7565.309992]        f7f94a20 00000086 00000000 f7f94b7c 00000000  
>> c17ec600 00000020 f7c89d80
>> [ 7565.309992]        f762ac00 7fffffff 7fffffff c03c8a35 f74d4c00  
>> 00485898 c0135cee 00000008
>> [ 7565.309992]        0000446b c013166d f7c89d80 f762ac00 f7c89d80  
>> f762ac00 c02b6254 00000003
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
>> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] getty         S f7cd081c     0  2472      1
>> [ 7565.309992]        f7cd06c0 00000082 00000000 f7cd081c 00000000  
>> c17ec600 00000020 f7f51580
>> [ 7565.309992]        f762b800 7fffffff 7fffffff c03c8a35 f76b2480  
>> 00485898 c0135cee 00000008
>> [ 7565.309992]        0000446b c013166d f7f51580 f762b800 f7f51580  
>> f762b800 c02b6254 00000003
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
>> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] getty         S f7f224bc     0  2474      1
>> [ 7565.309992]        f7f22360 00000082 00000000 f7f224bc 00000000  
>> c17ec600 00000020 f7c05a80
>> [ 7565.309992]        f762b000 7fffffff 7fffffff c03c8a35 f7f77a80  
>> 00485898 c0135cee 00000008
>> [ 7565.309992]        0000446b c013166d f7c05a80 f762b000 f7c05a80  
>> f762b000 c02b6254 00000003
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
>> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] getty         S f7f2281c     0  2475      1
>> [ 7565.309992]        f7f226c0 00000086 00000000 f7f2281c 00000000  
>> c17ec600 00000020 f7fc9e00
>> [ 7565.309992]        f762a800 7fffffff 7fffffff c03c8a35 f76b2000  
>> 00485898 c0135cee 00000008
>> [ 7565.309992]        0000446b c013166d f7fc9e00 f762a800 f7fc9e00  
>> f762a800 c02b6254 00000003
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
>> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] getty         S f7f22b7c     0  2476      1
>> [ 7565.309992]        f7f22a20 00000082 00000000 f7f22b7c 00000000  
>> c17ec600 00000020 f7c96080
>> [ 7565.309992]        f762a000 7fffffff 7fffffff c03c8a35 f7c56180  
>> 00485898 c0135cee 00000008
>> [ 7565.309992]        0000446b c013166d f7c96080 f762a000 f7c96080  
>> f762a000 c02b6254 00000003
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0135cee>] up+0x5e/0xa0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
>> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] sshd          S f7f4a15c     0  2499   2335
>> [ 7565.309992]        f7f4a000 00000086 00000000 f7f4a15c c048bb60  
>> 00000000 f7f12c10 00000008
>> [ 7565.309992]        00000080 00000000 00000000 c03c8a35 f7500100  
>> f7f12c00 c02b4f20 c02b0836
>> [ 7565.309992]        f768fbe4 f7f12c10 00000007 00000080 00000008  
>> 00000080 c01a9b22 f768fbe4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c02b4f20>] normal_poll+0x0/0x150
>> [ 7565.309992]  [<c02b0836>] tty_poll+0x76/0x80
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c034e10d>] dev_queue_xmit+0xcd/0x400
>> [ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
>> [ 7565.309992]  [<c0352c55>] neigh_resolve_output+0xf5/0x270
>> [ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
>> [ 7565.309992]  [<c0367ec5>] ip_local_out+0x15/0x20
>> [ 7565.309992]  [<c036a25e>] ip_queue_xmit+0x1ae/0x320
>> [ 7565.309992]  [<c0121d9a>] __do_softirq+0x6a/0xb0
>> [ 7565.309992]  [<c0121cc8>] _local_bh_enable+0x28/0x90
>> [ 7565.309992]  [<c037fa80>] tcp_v4_send_check+0x40/0xe0
>> [ 7565.309992]  [<c037a84f>] tcp_transmit_skb+0x3bf/0x7a0
>> [ 7565.309992]  [<c010f570>] smp_apic_timer_interrupt+0x0/0x70
>> [ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
>> [ 7565.309992]  [<c0379fbc>] tcp_init_tso_segs+0x3c/0x60
>> [ 7565.309992]  [<c037bfb9>] __tcp_push_pending_frames+0x139/0x750
>> [ 7565.309992]  [<c036f705>] sk_stream_alloc_skb+0x35/0xe0
>> [ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
>> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
>> [ 7565.309992]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
>> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
>> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
>> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
>> [ 7565.309992]  [<c0371082>] tcp_sendmsg+0x7a2/0xab0
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c033ea22>] sock_aio_write+0xe2/0x100
>> [ 7565.309992]  [<c01197b4>] try_to_wake_up+0x44/0xd0
>> [ 7565.309992]  [<c019ba56>] do_sync_write+0xc6/0x110
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c019c8e1>] sys_write+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] bash          S f7f4ab7c     0  2501   2499
>> [ 7565.309992]        f7f4aa20 00000082 f74d4ac4 f7f4ab7c f741f098  
>> c0185d00 c013166d ffffffea
>> [ 7565.309992]        f7f4aa18 0000000e 00000001 c011f60d f7e4fb00  
>> c02b36aa 00000000 00000001
>> [ 7565.309992]        00000000 f7f4ab10 00000000 00000003 0000000c  
>> f7f4aa20 00000001 f7541a00
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c011f60d>] do_wait+0x63d/0xc80
>> [ 7565.309992]  [<c02b36aa>] tty_ioctl+0x98a/0xab0
>> [ 7565.309992]  [<c01a288d>] pipe_release+0x7d/0xa0
>> [ 7565.309992]  [<c02b2d20>] tty_ioctl+0x0/0xab0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c011fcc3>] sys_wait4+0x73/0xc0
>> [ 7565.309992]  [<c011fd35>] sys_waitpid+0x25/0x30
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] remoteHost    S f7e4fc5c     0  2507   2501
>> [ 7565.309992]        f7e4fb00 00000082 f7e4fb00 f7e4fc5c 00000000  
>> c017af36 c0485da0 f769ea80
>> [ 7565.309992]        00000000 7fffffff f769ec40 c03c8a35 c0122076  
>> c0341b2b c16fe1e0 00000001
>> [ 7565.309992]        c0121c8a c0341a07 f769ea80 f769ea80 f769ea80  
>> 00000000 c036e10f 000000d0
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c017af36>] __alloc_pages_internal+0xa6/0x450
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
>> [ 7565.309992]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
>> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
>> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
>> [ 7565.309992]  [<c036e10f>] inet_csk_accept+0x12f/0x260
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c038ca4f>] inet_accept+0x1f/0xd0
>> [ 7565.309992]  [<c0340a30>] sys_accept+0xf0/0x210
>> [ 7565.309992]  [<c0134fe2>] ktime_get_ts+0x22/0x50
>> [ 7565.309992]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
>> [ 7565.309992]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
>> [ 7565.309992]  [<c0340c16>] sys_socketcall+0xc6/0x280
>> [ 7565.309992]  [<c0199ea7>] filp_close+0x47/0x80
>> [ 7565.309992]  [<c019b5e2>] sys_close+0x82/0xf0
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] sshd          S f7f22edc     0  9291   2335
>> [ 7565.309992]        f7f22d80 00000086 00000000 f7f22edc c048bb60  
>> 00000000 f7f4d410 00000008
>> [ 7565.309992]        00000080 00000000 00000000 c03c8a35 f7f04800  
>> f7f4d400 c02b4f20 c02b0836
>> [ 7565.309992]        f7679be4 f7f4d410 00000007 00000080 00000008  
>> 00000080 c01a9b22 f7679be4
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c02b4f20>] normal_poll+0x0/0x150
>> [ 7565.309992]  [<c02b0836>] tty_poll+0x76/0x80
>> [ 7565.309992]  [<c01a9b22>] do_select+0x3b2/0x490
>> [ 7565.309992]  [<c01aa1b0>] __pollwait+0x0/0xf0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c034e10d>] dev_queue_xmit+0xcd/0x400
>> [ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
>> [ 7565.309992]  [<c0352c55>] neigh_resolve_output+0xf5/0x270
>> [ 7565.309992]  [<c0147961>] __rcu_read_unlock+0x41/0x90
>> [ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
>> [ 7565.309992]  [<c0358eeb>] eth_header+0x2b/0xc0
>> [ 7565.309992]  [<c0367ec5>] ip_local_out+0x15/0x20
>> [ 7565.309992]  [<c036a25e>] ip_queue_xmit+0x1ae/0x320
>> [ 7565.309992]  [<c03695e9>] ip_finish_output+0x119/0x2c0
>> [ 7565.309992]  [<c037fa80>] tcp_v4_send_check+0x40/0xe0
>> [ 7565.309992]  [<c037a84f>] tcp_transmit_skb+0x3bf/0x7a0
>> [ 7565.309992]  [<c0121f3f>] irq_exit+0x5f/0x90
>> [ 7565.309992]  [<c010f5a0>] smp_apic_timer_interrupt+0x30/0x70
>> [ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
>> [ 7565.309992]  [<c0379fbc>] tcp_init_tso_segs+0x3c/0x60
>> [ 7565.309992]  [<c037bfb9>] __tcp_push_pending_frames+0x139/0x750
>> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
>> [ 7565.309992]  [<c036f705>] sk_stream_alloc_skb+0x35/0xe0
>> [ 7565.309992]  [<c0198dd8>] __kmalloc_track_caller+0x78/0x120
>> [ 7565.309992]  [<c01267eb>] lock_timer_base+0x4b/0xb0
>> [ 7565.309992]  [<c0121c8a>] local_bh_disable+0xa/0x20
>> [ 7565.309992]  [<c0341a07>] release_sock+0xc7/0xf0
>> [ 7565.309992]  [<c0122076>] local_bh_enable+0x46/0x90
>> [ 7565.309992]  [<c0371082>] tcp_sendmsg+0x7a2/0xab0
>> [ 7565.309992]  [<c01a9e08>] core_sys_select+0x208/0x340
>> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
>> [ 7565.309992]  [<c033ea22>] sock_aio_write+0xe2/0x100
>> [ 7565.309992]  [<c01197b4>] try_to_wake_up+0x44/0xd0
>> [ 7565.309992]  [<c011688e>] __wake_up_common+0x3e/0x70
>> [ 7565.309992]  [<c019ba56>] do_sync_write+0xc6/0x110
>> [ 7565.309992]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309992]  [<c01aa2ed>] sys_select+0x4d/0x1b0
>> [ 7565.309992]  [<c019c8e1>] sys_write+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] bash          S f7f238fc     0  9293   9291
>> [ 7565.309992]        f7f237a0 00000086 c013501d f7f238fc 09e6bb40  
>> 00989680 c0117063 f7f04580
>> [ 7565.309992]        f7f4c000 7fffffff 7fffffff c03c8a35 00000200  
>> f7f237a0 f7f0e900 f7f0e900
>> [ 7565.309992]        f7f237a0 c013166d f7f04580 f7f4c000 f7f04580  
>> f7f4c000 c02b6254 00000003
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c013501d>] ktime_get+0xd/0x30
>> [ 7565.309992]  [<c0117063>] hrtick_start_fair+0xa3/0x150
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c02b6254>] read_chan+0x1c4/0x6d0
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c02b2b83>] tty_read+0x83/0xb0
>> [ 7565.309992]  [<c02b6090>] read_chan+0x0/0x6d0
>> [ 7565.309992]  [<c019c411>] vfs_read+0xa1/0x130
>> [ 7565.309992]  [<c02b2b00>] tty_read+0x0/0xb0
>> [ 7565.309992]  [<c019c871>] sys_read+0x41/0x70
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] remoteHost    S f7c8281c     0 11799   2507
>> [ 7565.309992]        f7c826c0 00000086 00000200 f7c8281c c04f4300  
>> 031703f8 c013166d f7592e84
>> [ 7565.309992]        f7c826b8 00000004 00000001 c011f60d f7592d80  
>> f7c826c0 f7c56300 00000000
>> [ 7565.309992]        00000000 f7c827b0 f74b6c80 00000000 00000000  
>> f7c826c0 00000001 f7592d80
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309992]  [<c011f60d>] do_wait+0x63d/0xc80
>> [ 7565.309992]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309992]  [<c011fcc3>] sys_wait4+0x73/0xc0
>> [ 7565.309992]  [<c011fd35>] sys_waitpid+0x25/0x30
>> [ 7565.309992]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309992]  =======================
>> [ 7565.309992] remoteHost    S f7c83c5c     0 11800   2507
>> [ 7565.309992]        f7c83b00 00000082 c03797ed f7c83c5c 00000200  
>> 00000000 00000000 f7653d7c
>> [ 7565.309992]        f75a4880 f75a48e0 f75a4b14 c03c8a35 f74f4620  
>> c0341b2b 00000000 f74f4600
>> [ 7565.309992]        c0121c8a c0341a07 f7653d7c f7653d7c f7653d7c  
>> f75a4880 c0342564 00000000
>> [ 7565.309992] Call Trace:
>> [ 7565.309992]  [<c03797ed>] tcp_rcv_established+0x3ed/0x6d0
>> [ 7565.309992]  [<c03c8a35>] schedule_timeout+0x75/0xc0
>> [ 7565.309993]  [<c0341b2b>] lock_sock_nested+0xfb/0x110
>> [ 7565.309993]  [<c0121c8a>] local_bh_disable+0xa/0x20
>> [ 7565.309993]  [<c0341a07>] release_sock+0xc7/0xf0
>> [ 7565.309993]  [<c0342564>] sk_wait_data+0x84/0xd0
>> [ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309993]  [<c037173f>] tcp_recvmsg+0x3af/0x8a0
>> [ 7565.309993]  [<c0134fe2>] ktime_get_ts+0x22/0x50
>> [ 7565.309993]  [<c0117063>] hrtick_start_fair+0xa3/0x150
>> [ 7565.309993]  [<c0341425>] sock_common_recvmsg+0x45/0x70
>> [ 7565.309993]  [<c033f6c7>] sock_recvmsg+0xd7/0x110
>> [ 7565.309993]  [<c02b5324>] n_tty_receive_buf+0x2b4/0x1020
>> [ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309993]  [<c02b51c5>] n_tty_receive_buf+0x155/0x1020
>> [ 7565.309993]  [<c0113bdf>] do_page_fault+0x29f/0x6d0
>> [ 7565.309993]  [<c01345d2>] enqueue_hrtimer+0x72/0xf0
>> [ 7565.309993]  [<c033ef82>] sockfd_lookup_light+0x32/0x60
>> [ 7565.309993]  [<c03406ce>] sys_recvfrom+0xce/0x160
>> [ 7565.309993]  [<c0117e30>] hrtick_set+0xc0/0x1a0
>> [ 7565.309993]  [<c0119a1b>] hrtick_resched+0x4b/0x60
>> [ 7565.309993]  [<c010260f>] do_notify_resume+0x6f/0x790
>> [ 7565.309993]  [<c02aedba>] tty_put_char+0x2a/0x40
>> [ 7565.309993]  [<c03ca20a>] rwsem_down_write_failed+0x2a/0x30
>> [ 7565.309993]  [<c0119aaf>] __wake_up+0x7f/0xd0
>> [ 7565.309993]  [<c0340793>] sys_recv+0x33/0x40
>> [ 7565.309993]  [<c0340cb5>] sys_socketcall+0x165/0x280
>> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309993]  =======================
>> [ 7565.309993] rt61pci       S f7f4a81c     0 15243      2
>> [ 7565.309993]        f7f4a6c0 00000046 c012d8c0 f7f4a81c c04762e0  
>> c01315a5 00000200 f7644b40
>> [ 7565.309993]        f7644b48 00000000 00000000 c012e1bd 00000000  
>> f7f4a6c0 c01312c0 f7644b48
>> [ 7565.309993]        f7644b48 fffffffc f7644b40 c012e130 c0130f32  
>> c0130ef0 00000000 00000000
>> [ 7565.309993] Call Trace:
>> [ 7565.309993]  [<c012d8c0>] run_workqueue+0x120/0x1b0
>> [ 7565.309993]  [<c01315a5>] prepare_to_wait+0x85/0xe0
>> [ 7565.309993]  [<c012e1bd>] worker_thread+0x8d/0xa0
>> [ 7565.309993]  [<c01312c0>] autoremove_wake_function+0x0/0x50
>> [ 7565.309993]  [<c012e130>] worker_thread+0x0/0xa0
>> [ 7565.309993]  [<c0130f32>] kthread+0x42/0x70
>> [ 7565.309993]  [<c0130ef0>] kthread+0x0/0x70
>> [ 7565.309993]  [<c0103db3>] kernel_thread_helper+0x7/0x14
>> [ 7565.309993]  =======================
>> [ 7565.309993] actuatorTask  ? f759281c     0 15816      1
>> [ 7565.309993]        f75926c0 00000046 400000c0 f759281c f75926c0  
>> 00000000 00003dc8 00000010
>> [ 7565.309993]        f75926c0 f75b1edc 00000000 c01202bc c0155590  
>> c04844a4 00000246 00000001
>> [ 7565.309993]        f75b1f90 f759276c f75927b0 00000000 00000000  
>> f75b1f90 f75b1f90 f7fdbe00
>> [ 7565.309993] Call Trace:
>> [ 7565.309993]  [<c01202bc>] do_exit+0x4bc/0x730
>> [ 7565.309993]  [<c0155590>] sigwake_event+0x0/0x140
>> [ 7565.309993]  [<c012055f>] do_group_exit+0x2f/0xc0
>> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309993]  =======================
>> [ 7565.309993] Actuator Aper S f7592b7c     0 15817      1
>> [ 7565.309993]        f7592a20 00000086 00000001 f7592b7c 00000001  
>> 00000000 00000200 f7e60620
>> [ 7565.309993]        f7592a20 f7729fb8 00000170 c01560ad 00000010  
>> f7729f5c 00000001 00000006
>> [ 7565.309993]        00000000 f7729fb8 00000170 c015628f 00000001  
>> c0577b04 c0577b00 00000000
>> [ 7565.309993] Call Trace:
>> [ 7565.309993]  [<c01560ad>] xnshadow_harden+0x8d/0x1e0
>> [ 7565.309993]  [<c015628f>] losyscall_event+0x8f/0x170
>> [ 7565.309993]  [<c014a1e2>] __ipipe_dispatch_event+0xa2/0x190
>> [ 7565.309993]  [<c0156200>] losyscall_event+0x0/0x170
>> [ 7565.309993]  [<c0112731>] __ipipe_syscall_root+0x41/0x100
>> [ 7565.309993]  [<c010311d>] system_call+0x29/0x4a
>> [ 7565.309993]  =======================
>> [ 7565.309993] sh            S f7592edc     0 15832  11799
>> [ 7565.309993]        f7592d80 00000086 f7c56944 f7592edc f7f36080  
>> c0185d00 c013166d ffffffea
>> [ 7565.309993]        f7592d78 00000004 00000001 c011f60d f7593b00  
>> f7c56900 f7c56300 00000001
>> [ 7565.309993]        00000000 f7592e70 00000000 00000003 0000000c  
>> f7592d80 00000001 f7593b00
>> [ 7565.309993] Call Trace:
>> [ 7565.309993]  [<c0185d00>] handle_mm_fault+0x3a0/0x670
>> [ 7565.309993]  [<c013166d>] add_wait_queue+0x6d/0xb0
>> [ 7565.309993]  [<c011f60d>] do_wait+0x63d/0xc80
>> [ 7565.309993]  [<c0119840>] default_wake_function+0x0/0x10
>> [ 7565.309993]  [<c011fcc3>] sys_wait4+0x73/0xc0
>> [ 7565.309993]  [<c011fd35>] sys_waitpid+0x25/0x30
>> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309993]  =======================
>> [ 7565.309993] sleep         S f7593c5c     0 16596  15832
>> [ 7565.309993]        f7593b00 00000086 4b3f0cce f7593c5c 00000000  
>> 00000200 00000000 f7659f60
>> [ 7565.309993]        f7593b00 00000001 00000000 c03c91bb 00000001  
>> 00000000 00000001 c0134ef5
>> [ 7565.309993]        f7fa5f61 f754c740 c0576960 86d9d6ce 000006e1  
>> c0134a60 c047f254 00000001
>> [ 7565.309993] Call Trace:
>> [ 7565.309993]  [<c03c91bb>] do_nanosleep+0x5b/0x90
>> [ 7565.309993]  [<c0134ef5>] hrtimer_nanosleep+0x45/0xb0
>> [ 7565.309993]  [<c0134a60>] hrtimer_wakeup+0x0/0x20
>> [ 7565.309993]  [<c0134fb8>] sys_nanosleep+0x58/0x60
>> [ 7565.309993]  [<c0103145>] syscall_call+0x7/0xb
>> [ 7565.309993]  =======================
>>
>>
>> On Nov 28, 2008, at 7:11 PM, Gilles Chanteperdrix wrote:
>>
>>> Mehmet Alphan Ulusoy wrote:
>>>> ps aux says Task's state is "Zl+" and it's there for over 6 hours  
>>>> now,
>>>> is init's mentioned behavior standard or somewhat optional?
>>>>
>>>> If it's standard, then does this mean that there's something else  
>>>> going on?
>>>
>>> If you are able to reproduce this issue with a simple program,  
>>> could you
>>> send it to us?
>>>
>>> -- 
>>>                                                 Gilles.
>>>
>>
>> _______________________________________________
>> Xenomai-help mailing list
>> Xenomai-help@domain.hid
>> https://mail.gna.org/listinfo/xenomai-help
>


[-- Attachment #1.2: Type: text/html, Size: 158452 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2433 bytes --]

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Alphan Ulusoy wrote:
> I've recompiled the kernel after turning off the preemptible RCU  
> option. I left CONFIG_PREEMPT as it is for the time being. Can  
> preemptible RCU option be the cause ?

I do not think so. However, I have no idea what the problem could be.
Are you sure that you ran wait on the given pid? In order to be sure,
you can try and send SIGCHLD to the process father?

What would be really interesting too, is to have the process status. So,
in function sched_show_task, file kernel/sched.c, could you try and
print p->state in hexadecimal? In the dump you sent, the state is shown
as a question mark, so we do not really know in what state the task is.

You should also recompile your kernel with frame pointers, if it is not
already the case.

-- 
                                                 Gilles.

Re: [Xenomai-help] Zombie user tasks

[Mehmet Alphan Ulusoy]

On Mon 22 Dec 2008 11:48:00 EET, Gilles Chanteperdrix wrote:

> Alphan Ulusoy wrote:
>> I've recompiled the kernel after turning off the preemptible RCU
>> option. I left CONFIG_PREEMPT as it is for the time being. Can
>> preemptible RCU option be the cause ?
>
> I do not think so. However, I have no idea what the problem could be.
> Are you sure that you ran wait on the given pid? In order to be sure,
> you can try and send SIGCHLD to the process father?
>

I wait for the child processes using while ((status =  
waitpid(-1,NULL,WNOHANG)) >1); in the signal handler of the process  
father.

> What would be really interesting too, is to have the process status. So,
> in function sched_show_task, file kernel/sched.c, could you try and
> print p->state in hexadecimal? In the dump you sent, the state is shown
> as a question mark, so we do not really know in what state the task is.
>

The output of the sched says the task is 'relaxed shadow', I believe  
what you ask for is something else. I will try to make the change you  
suggested and hopefully send you the dump once the task turns into a  
zombie.

> You should also recompile your kernel with frame pointers, if it is not
> already the case.
>

OK, will do that.

thanks,

alphan

> --
>                                                  Gilles.
>
>

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Mehmet Alphan Ulusoy wrote:
> On Mon 22 Dec 2008 11:48:00 EET, Gilles Chanteperdrix wrote:
>> What would be really interesting too, is to have the process status. So,
>> in function sched_show_task, file kernel/sched.c, could you try and
>> print p->state in hexadecimal? In the dump you sent, the state is shown
>> as a question mark, so we do not really know in what state the task is.
>>
> 
> The output of the sched says the task is 'relaxed shadow', I believe  
> what you ask for is something else. I will try to make the change you  
> suggested and hopefully send you the dump once the task turns into a  
> zombie.

Unless I misunderstood something, the zombie task (called actuatorTask)
does not appear in /proc/xenomai/sched and /proc/xenomai/stat, the task
which appears there is "Actuator Aperiodic Task". actuatorTask, however,
appears in the sysrq+T output, and its state (for Linux scheduler, not
for xenomai scheduler, since for xenomai scheduler it does not even
exist) is "?", so, what I would like to know is its real state, printed
in hexadecimal. For this, you just have to print p->state using 0x%08x
as printk format.

-- 
                                                 Gilles.

Re: [Xenomai-help] Zombie user tasks

[Alphan Ulusoy]

[-- Attachment #1: Type: text/plain, Size: 73428 bytes --]

I've done the changes you suggested ( addition to sched.c file and  
fram pointers) and left the tasks running last night. Below you can  
find the dump. The hex numbers inside the brackets ( [0x00000001] )  
are newly added. And as you said in your previous post "Actuator  
Aperiodic Task" is the real-time task created and started in the  
process called "actuatorTask" which turns into a zombie. The ps aux  
output says:

root     28213  0.0  0.0      0     0 pts/0    Zl   04:21   0:02  
[actuatorTask] <defunct>


To give you a heads up, the actuator Aperiodic task is a simple  
for(;;) loop which blocks on recvfrom() on a UDP socket at every  
iteration until a packet is received. Then it writes the data it  
contains to a shared variable acquiring and releasing a mutex. any  
ideas?


regards,

alphan.



------- Dump starts here ---------


[33034.505238] SysRq : Show State
[33034.505238]   task                PC stack   pid father
[33034.505238] init          S [0x00000001] c0128feb     0     1      0
[33034.505238]        f7c2bb30 00000086 f7c2bb14 c0128feb f7c2c000  
f7c2c254 c0117a30 f7c2bb40
[33034.505238]        f7c2bb30 c012917f 00000000 00000200 f7c2bb40  
01f382e1 00000000 f7c2bb60
[33034.505238]        c03bffc7 f7f20400 f7c2bbd0 c056fbd8 c056fbd8  
01f382e1 c0128dd0 f7c2c000
[33034.505238] Call Trace:
[33034.505238]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[33034.505238]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[33034.505238]  [<c0128dd0>] ? process_timeout+0x0/0x10
[33034.505238]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505238]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<f88b4c65>] ? __ieee80211_rx+0x195/0x5e0 [mac80211]
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c033e6d1>] ? skb_dequeue+0x71/0xb0
[33034.505238]  [<f88a2482>] ? ieee80211_tasklet_handler+0x72/0x100  
[mac80211]
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c012473b>] ? _local_bh_enable+0x2b/0x80
[33034.505238]  [<c0124839>] ? __do_softirq+0xa9/0xc0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0124977>] ? irq_exit+0x27/0x80
[33034.505238]  [<c0105aad>] ? do_IRQ+0x3d/0x70
[33034.505238]  [<c010311c>] ? restore_nocheck_notrace+0x0/0xe
[33034.505238]  [<c0105a70>] ? do_IRQ+0x0/0x70
[33034.505238]  [<c014be02>] ? __xirq_end+0x0/0x45
[33034.505238]  [<c014bf4a>] ? __ipipe_walk_pipeline+0x6a/0xb0
[33034.505238]  [<c0111f41>] ? __ipipe_handle_irq+0x81/0x1a0
[33034.505238]  [<c01a9e21>] ? do_select+0x1/0x590
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c01aee28>] ? __d_lookup+0xc8/0x180
[33034.505238]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[33034.505238]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505238]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[33034.505238]  [<c01b4476>] ? mntput_no_expire+0x16/0x100
[33034.505238]  [<c01a4530>] ? path_put+0x20/0x30
[33034.505238]  [<c01a691b>] ? path_walk+0x4b/0x90
[33034.505238]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505238]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0297e73>] ? copy_to_user+0x43/0x60
[33034.505238]  [<c019ff4b>] ? cp_new_stat64+0xeb/0x100
[33034.505238]  [<c01aaac4>] sys_select+0xd4/0x190
[33034.505238]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505238]  =======================
[33034.505238] kthreadd      S [0x00000001] f7c31fa4     0     2      0
[33034.505238]        f7c31fc8 00000046 00000000 f7c31fa4 f7c2c470  
f7c2c6c4 f7c31fac c90ecb83
[33034.505238]        00001375 00000000 00000000 00000000 c04783f0  
00006e27 f7663da8 f7c31fe0
[33034.505238]        c013374a 00000000 c01335d0 00000000 00000000  
00000000 c0103d43 00000000
[33034.505238] Call Trace:
[33034.505238]  [<c013374a>] kthreadd+0x17a/0x1a0
[33034.505238]  [<c01335d0>] ? kthreadd+0x0/0x1a0
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] ksoftirqd/0   S [0x00000001] 00000000     0     3      2
[33034.505238]        f7c33fc0 00000046 c0117a30 00000000 f7c2c8e0  
f7c2cb34 f7c33fb0 c0124839
[33034.505238]        00000000 f7c33fb0 c014c146 f7c33fc0 00000000  
00000000 c0124b70 f7c33fcc
[33034.505238]        c0124c3d fffffffc f7c33fe0 c01335a2 c0133560  
00000000 00000000 00000000
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0124839>] ? __do_softirq+0xa9/0xc0
[33034.505238]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505238]  [<c0124b70>] ? ksoftirqd+0x0/0x140
[33034.505238]  [<c0124c3d>] ksoftirqd+0xcd/0x140
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] watchdog/0    S [0x00000001] 0a987ddd     0     4      2
[33034.505238]        f7c39f9c 00000046 00000000 0a987ddd f7c2cd50  
f7c2cfa4 00000073 0a98a945
[33034.505238]        00000000 00000000 00000000 f7c39f94 fffffffc  
00000000 00000000 f7c39fcc
[33034.505238]        c01471c8 f7c2cfa4 f7c2c000 03e66246 00000000  
c011c4ec 00000000 00000063
[33034.505238] Call Trace:
[33034.505238]  [<c01471c8>] watchdog+0x58/0x1c0
[33034.505238]  [<c011c4ec>] ? complete+0x6c/0x90
[33034.505238]  [<c0147170>] ? watchdog+0x0/0x1c0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] events/0      S [0x00000001] c0117a30     0     5      2
[33034.505238]        f7c3bfa4 00000046 f7c3bf7c c0117a30 f7c2d1c0  
f7c2d414 c0117a30 f7c2d1c0
[33034.505238]        f7c3bfa4 c0133b8b 00000001 00000200 f7c08140  
f7c08148 f7c3bfac f7c3bfcc
[33034.505238]        c01307f5 00000000 f7c2d1c0 c01338f0 f7c08148  
f7c08148 fffffffc f7c08140
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] khelper       S [0x00000001] c0117a30     0     6      2
[33034.505238]        f7c3dfa4 00000046 f7c3df7c c0117a30 f7c2d630  
f7c2d884 c0117a30 c9d8651e
[33034.505238]        000000d8 c0133b8b 00000001 00000200 f7c081c0  
f7c081c8 f7c3dfac f7c3dfcc
[33034.505238]        c01307f5 00000000 f7c2d630 c01338f0 f7c081c8  
f7c081c8 fffffffc f7c081c0
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] kblockd/0     S [0x00000001] c0117a30     0    54      2
[33034.505238]        f7cd9fa4 00000046 f7cd9f7c c0117a30 f7c970d0  
f7c97324 c0117a30 f7c970d0
[33034.505238]        f7cd9fa4 c0133b8b 00000001 00000200 f7c1af00  
f7c1af08 f7cd9fac f7cd9fcc
[33034.505238]        c01307f5 00000000 f7c970d0 c01338f0 f7c1af08  
f7c1af08 fffffffc f7c1af00
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] khubd         S [0x00000001] 00000002     0    63      2
[33034.505238]        f7cfdf38 00000046 00000000 00000002 f7ce5aa0  
f7ce5cf4 c0117a30 55f4defd
[33034.505238]        00000000 c0133b8b 00000001 00000200 f7cfdfa0  
f7cb8240 00000000 f7cfdfcc
[33034.505238]        c0301449 f7cfdfb8 002160e9 00000000 123e6adf  
00000000 f7c2d658 f7f3b904
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c0301449>] hub_thread+0x649/0xcb0
[33034.505238]  [<c011996e>] ? hrtick_set+0xde/0x1a0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c03bf76d>] ? schedule+0x2fd/0x570
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c011c4ec>] ? complete+0x6c/0x90
[33034.505238]  [<c0300e00>] ? hub_thread+0x0/0xcb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] kseriod       S [0x00000001] c0198add     0    66      2
[33034.505238]        f7d03f90 00000046 f7d03f84 c0198add f7ce67f0  
f7ce6a44 c0117a30 5761608e
[33034.505238]        00000000 c0133b8b 00000001 00000200 f7d03fac  
00000000 f7f388e0 f7d03fcc
[33034.505238]        c031cb4f 00000001 00000003 f7ce67f0 f7ce6a44  
00000200 00000000 f7ce67f0
[33034.505238] Call Trace:
[33034.505238]  [<c0198add>] ? kfree+0x9d/0xf0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c031cb4f>] serio_thread+0xff/0x400
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c031ca50>] ? serio_thread+0x0/0x400
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] gatekeeper/0  S [0x00000001] f7c87f78     0   150      2
[33034.505238]        f7c87f94 00000046 00000046 f7c87f78 f7c88000  
f7c88254 c0571f40 be53c9c0
[33034.505238]        0000137d 00000000 f7c87f94 c0138047 f7d81a20  
c057be60 c057be6c f7c87fcc
[33034.505238]        c0158f77 00000003 f7c88000 f7c88254 00000000  
00000001 f7c88000 c011a4d0
[33034.505238] Call Trace:
[33034.505238]  [<c0138047>] ? up+0x57/0x90
[33034.505238]  [<c0158f77>] gatekeeper_thread+0xa7/0x140
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c0158ed0>] ? gatekeeper_thread+0x0/0x140
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] pdflush       S [0x00000001] f7c65f80     0   154      2
[33034.505238]        f7c65fa4 00000046 c04ee300 f7c65f80 f7c891c0  
f7c89414 f7c65f88 0d502af9
[33034.505238]        00000000 c011a2b5 c04ee300 00000200 f7c65fbc  
00000000 c017ee20 f7c65fcc
[33034.505238]        c017eedb 0d50119b f7c891c0 00000000 00000000  
c047f094 f7dbffbc fffb6cfe
[33034.505238] Call Trace:
[33034.505238]  [<c011a2b5>] ? set_user_nice+0x105/0x160
[33034.505238]  [<c017ee20>] ? pdflush+0x0/0x230
[33034.505238]  [<c017eedb>] pdflush+0xbb/0x230
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] pdflush       S [0x00000001] f7dbff94     0   155      2
[33034.505238]        f7dbffa4 00000046 00000000 f7dbff94 f7c89630  
f7c89884 00000000 00000000
[33034.505238]        00000000 00000000 00000025 01f304ae f7dbffbc  
00000000 c017ee20 f7dbffcc
[33034.505238]        c017eedb 0d504449 f7c89630 00000000 00000000  
f7c65fbc c047f094 01f379de
[33034.505238] Call Trace:
[33034.505238]  [<c017ee20>] ? pdflush+0x0/0x230
[33034.505238]  [<c017eedb>] pdflush+0xbb/0x230
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] kswapd0       S [0x00000001] 00000000     0   156      2
[33034.505238]        f7dc1f34 00000046 00000000 00000000 f7c89aa0  
f7c89cf4 c0117a30 0d52fb0d
[33034.505238]        00000000 c0133b8b 00000001 00000200 00000000  
c047e580 c0182840 f7dc1fcc
[33034.505238]        c0182bdb f7c89ac8 f7c89ac8 f7dc1f70 c047f028  
f7dc1f70 00000000 f7c2c498
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c0182840>] ? kswapd+0x0/0x410
[33034.505238]  [<c0182bdb>] kswapd+0x39b/0x410
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c011996e>] ? hrtick_set+0xde/0x1a0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c03bf76d>] ? schedule+0x2fd/0x570
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c011c4ec>] ? complete+0x6c/0x90
[33034.505238]  [<c0182840>] ? kswapd+0x0/0x410
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] aio/0         S [0x00000001] 00000000     0   199      2
[33034.505238]        f7d01fa4 00000046 f7d01f7c 00000000 f7f551c0  
f7f55414 c0117a30 0d7079c5
[33034.505238]        00000000 c0133b8b 00000001 00000200 f7c1afc0  
f7c1afc8 f7d01fac f7d01fcc
[33034.505238]        c01307f5 00000000 f7f551c0 c01338f0 f7c1afc8  
f7c1afc8 fffffffc f7c1afc0
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] unionfs_siod/ S [0x00000001] 00000000     0   212      2
[33034.505238]        f7c83fa4 00000046 f7c83f7c 00000000 f7c94d50  
f7c94fa4 c0117a30 0d7a27f0
[33034.505238]        00000000 c0133b8b 00000001 00000200 f7c1aa40  
f7c1aa48 f7c83fac f7c83fcc
[33034.505238]        c01307f5 00000000 f7c94d50 c01338f0 f7c1aa48  
f7c1aa48 fffffffc f7c1aa40
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] nfsiod        S [0x00000001] 00000000     0   213      2
[33034.505238]        f7c5dfa4 00000046 f7c5df7c 00000000 f7c951c0  
f7c95414 c0117a30 0d7b1c2d
[33034.505238]        00000000 c0133b8b 00000001 00000200 f7c1a340  
f7c1a348 f7c5dfac f7c5dfcc
[33034.505238]        c01307f5 00000000 f7c951c0 c01338f0 f7c1a348  
f7c1a348 fffffffc f7c1a340
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] kpsmoused     S [0x00000001] ffffffff     0   878      2
[33034.505238]        f7f03fa4 00000046 00000000 ffffffff f7f91aa0  
f7f91cf4 c0117a30 5625a9d4
[33034.505238]        00000000 c0133b8b 00000001 00000200 f7c4d4c0  
f7c4d4c8 f7f03fac f7f03fcc
[33034.505238]        c01307f5 00000000 f7f91aa0 c01338f0 f7c4d4c8  
f7c4d4c8 fffffffc f7c4d4c0
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] rpciod/0      S [0x00000001] ffffffff     0   887      2
[33034.505238]        f7f09fa4 00000046 00000000 ffffffff f7f911c0  
f7f91414 c0117a30 5954fbff
[33034.505238]        00000000 c0133b8b 00000001 00000200 f7c4da00  
f7c4da08 f7f09fac f7f09fcc
[33034.505238]        c01307f5 00000000 f7f911c0 c01338f0 f7c4da08  
f7c4da08 fffffffc f7c4da00
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] kjournald     S [0x00000001] f7f0bf70     0   891      2
[33034.505238]        f7f0bf94 00000046 f7f5f440 f7f0bf70 f7f93540  
f7f93794 c0117a30 855e8900
[33034.505238]        00001e0a 00000000 00000001 00000200 f7f5f400  
00000000 f7f5f4c4 f7f0bfcc
[33034.505238]        c01f881e 00000000 00000005 f7f5f450 f7f5f440  
00000000 f7f93540 c01338f0
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c01f881e>] kjournald+0x1fe/0x240
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c01f8620>] ? kjournald+0x0/0x240
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] udevd         S [0x00000001] 00000200     0   993      1
[33034.505238]        f7fa5b30 00000082 c0133c22 00000200 f7ce5f10  
f7ce6164 f7fa5b18 d5a96edc
[33034.505238]        00001307 00000000 c0133c22 00000200 00000008  
00000080 00000000 f7fa5b60
[33034.505238]        c03bfff5 f7f830c0 f7fa5b4c c03c02f3 00000080  
00000041 f7fa5b60 c01c9c38
[33034.505238] Call Trace:
[33034.505238]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505238]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505238]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505238]  [<c03c02f3>] ? mutex_lock+0x13/0x30
[33034.505238]  [<c01c9c38>] ? inotify_poll+0x48/0x60
[33034.505238]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505238]  [<c013d146>] ? clockevents_program_event+0xb6/0x120
[33034.505238]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c017c8a7>] ? get_page_from_freelist+0x227/0x530
[33034.505238]  [<c0115cc3>] ? kmap_atomic_prot+0x43/0xb0
[33034.505238]  [<c0296228>] ? number+0x288/0x2a0
[33034.505238]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[33034.505238]  [<c017e28b>] ? balance_dirty_pages_ratelimited_nr+0x4b/ 
0x290
[33034.505238]  [<c0192c10>] ? shmem_truncate_range+0x150/0x820
[33034.505238]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c017d783>] ? set_page_dirty+0x43/0xd0
[33034.505238]  [<c0296b21>] ? vsnprintf+0x371/0x6d0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[33034.505238]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505238]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[33034.505238]  [<c01a3dd9>] ? pipe_read+0x2e9/0x380
[33034.505238]  [<c019c76c>] ? do_sync_read+0xcc/0x110
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c01aaa2f>] sys_select+0x3f/0x190
[33034.505238]  [<c019c6a0>] ? do_sync_read+0x0/0x110
[33034.505238]  [<c019d3fd>] ? sys_read+0x3d/0x70
[33034.505238]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505238]  =======================
[33034.505238] kjournald     S [0x00000001] f74b7f70     0  2019      2
[33034.505238]        f74b7f94 00000046 f7f5fc40 f74b7f70 f7f579b0  
f7f57c04 c0117a30 f7f579b0
[33034.505238]        f74b7f94 c0133b8b 00000001 00000200 f7f5fc00  
00000000 f7f5fcc4 f74b7fcc
[33034.505238]        c01f881e 00000000 00000005 f7f5fc50 f7f5fc40  
00000000 f7f579b0 c01338f0
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c01f881e>] kjournald+0x1fe/0x240
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c01f8620>] ? kjournald+0x0/0x240
[33034.505238]  [<c01335a2>] kthread+0x42/0x70
[33034.505238]  [<c0133560>] ? kthread+0x0/0x70
[33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505238]  =======================
[33034.505238] dhclient3     S [0x00000001] c0128feb     0  2097      1
[33034.505238]        f7577b30 00000082 f7577b14 c0128feb f7f56380  
f7f565d4 c0117a30 f7577b40
[33034.505238]        f7577b30 c012917f 00000000 00000200 f7577b40  
01f4ca42 00000000 f7577b60
[33034.505238]        c03bffc7 f7f7b600 f7577bd0 c04a9c8c c056fe60  
01f4ca42 c0128dd0 f7f56380
[33034.505238] Call Trace:
[33034.505238]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[33034.505238]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[33034.505238]  [<c0128dd0>] ? process_timeout+0x0/0x10
[33034.505238]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505238]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c011a3f8>] ? try_to_wake_up+0x58/0x130
[33034.505238]  [<c0108679>] ? read_tsc+0x9/0x30
[33034.505238]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[33034.505238]  [<c0137576>] ? ktime_get_ts+0x46/0x50
[33034.505238]  [<c0137591>] ? ktime_get+0x11/0x30
[33034.505238]  [<c01172cb>] ? hrtick_start_fair+0xeb/0x140
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0197e51>] ? deactivate_slab+0x61/0x160
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c01985b8>] ? __slab_alloc+0x68/0x4f0
[33034.505238]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0296228>] ? number+0x288/0x2a0
[33034.505238]  [<c011687e>] ? __wake_up_common+0x3e/0x70
[33034.505238]  [<c011c442>] ? __wake_up_sync+0x72/0xb0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c033cd65>] ? sock_def_readable+0x45/0x80
[33034.505238]  [<c0398a11>] ? unix_dgram_sendmsg+0x461/0x530
[33034.505238]  [<c0352c25>] ? __qdisc_run+0x1c5/0x220
[33034.505238]  [<c0296b21>] ? vsnprintf+0x371/0x6d0
[33034.505238]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505238]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[33034.505238]  [<c01aef04>] ? d_lookup+0x24/0x40
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[33034.505238]  [<c014a215>] ? call_rcu+0x55/0x70
[33034.505238]  [<c01874e2>] ? handle_mm_fault+0x352/0x5f0
[33034.505238]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[33034.505238]  [<c01aaac4>] sys_select+0xd4/0x190
[33034.505238]  [<c0297e73>] ? copy_to_user+0x43/0x60
[33034.505238]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505238]  =======================
[33034.505238] syslogd       S [0x00000001] 00000200     0  2238      1
[33034.505238]        f755bb30 00000086 c0133c22 00000200 f7f91630  
f7f91884 f755bbd0 f755bb2c
[33034.505238]        f755bb20 c0117a30 00000000 f755bb2c 00000001  
00000001 00000000 f755bb60
[33034.505238]        c03bfff5 f755bbd0 f7ca3b00 00000001 0155bb50  
c03eaee0 00000001 f7ca3b00
[33034.505238] Call Trace:
[33034.505238]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505238]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505238]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c0289288>] ? blk_recount_segments+0x38/0x60
[33034.505238]  [<c01e5fbe>] ? ext3_reserve_inode_write+0x5e/0x80
[33034.505238]  [<c01edcd7>] ? __ext3_journal_stop+0x27/0x50
[33034.505238]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505238]  [<c0179de1>] ? mempool_alloc+0x31/0x160
[33034.505238]  [<c0284683>] ? elv_rb_add+0x53/0x70
[33034.505238]  [<c028e087>] ? deadline_add_rq_rb+0x27/0x50
[33034.505238]  [<c028e126>] ? deadline_add_request+0x26/0x60
[33034.505238]  [<c028448b>] ? elv_insert+0xfb/0x160
[33034.505238]  [<c028454d>] ? __elv_add_request+0x5d/0xa0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0286922>] ? __make_request+0x102/0x340
[33034.505238]  [<c0285aa4>] ? generic_make_request+0x174/0x220
[33034.505238]  [<c0198b9f>] ? kmem_cache_alloc+0x5f/0xd0
[33034.505238]  [<c017991e>] ? mempool_alloc_slab+0xe/0x10
[33034.505238]  [<c0179de1>] ? mempool_alloc+0x31/0x160
[33034.505238]  [<c0108679>] ? read_tsc+0x9/0x30
[33034.505238]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[33034.505238]  [<c0137576>] ? ktime_get_ts+0x46/0x50
[33034.505238]  [<c0137591>] ? ktime_get+0x11/0x30
[33034.505238]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[33034.505238]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505238]  [<c017f819>] ? pagevec_lookup_tag+0x29/0x40
[33034.505238]  [<c019c54f>] ? do_sync_readv_writev+0xaf/0xf0
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c019cce3>] ? do_readv_writev+0x113/0x180
[33034.505238]  [<c01a3670>] ? pipe_write+0x0/0x440
[33034.505238]  [<c01aaa2f>] sys_select+0x3f/0x190
[33034.505238]  [<c0129be4>] ? sigprocmask+0x74/0x100
[33034.505238]  [<c012c7fc>] ? sys_rt_sigprocmask+0xdc/0x110
[33034.505238]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505238]  =======================
[33034.505238] klogd         R [0x00000000] running      0  2244      1
[33034.505238]        f7477ef4 00000086 f7477f6c c019c65c f7f90000  
f7f90254 c0117a30 fea8e52d
[33034.505238]        00001307 c0133b8b 00000001 00000200 f7477f18  
00000000 00000fff f7477f38
[33034.505238]        c0120713 0804d6a0 00000000 00000000 00000000  
f7f90000 c01338f0 f7477f14
[33034.505238] Call Trace:
[33034.505238]  [<c019c65c>] ? do_sync_write+0xcc/0x110
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505238]  [<c0120713>] do_syslog+0x403/0x490
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505238]  [<c01dad15>] kmsg_read+0x25/0x50
[33034.505238]  [<c01dacf0>] ? kmsg_read+0x0/0x50
[33034.505238]  [<c01d1d8c>] proc_reg_read+0x6c/0xc0
[33034.505238]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505238]  [<c01d1d20>] ? proc_reg_read+0x0/0xc0
[33034.505238]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505238]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505238]  =======================
[33034.505238] inetd         S [0x00000001] f793eb60     0  2254      1
[33034.505238]        f7fefb30 00000082 c0117a30 f793eb60 f7f939b0  
f7f93c04 f7fefc00 64b26425
[33034.505238]        00000004 00000000 00000001 00000000 00000006  
00000020 00000000 f7fefb60
[33034.505238]        c03bfff5 f7fefbd0 f7fa0440 f7f20e00 f7fefb54  
c0368657 c03ea300 00000020
[33034.505238] Call Trace:
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505238]  [<c0368657>] ? tcp_poll+0x17/0x160
[33034.505238]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505238]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505238]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[33034.505238]  [<c017c956>] ? get_page_from_freelist+0x2d6/0x530
[33034.505238]  [<c017cc50>] ? __alloc_pages_internal+0xa0/0x420
[33034.505238]  [<c0198add>] ? kfree+0x9d/0xf0
[33034.505238]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505238]  [<c02a9598>] ? extract_buf+0x88/0xf0
[33034.505238]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[33034.505238]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505238]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[33034.505238]  [<c01b1efd>] ? touch_atime+0x2d/0x110
[33034.505238]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505238]  [<c01794da>] ? filemap_fault+0x12a/0x480
[33034.505238]  [<c0176e24>] ? unlock_page+0x24/0x30
[33034.505238]  [<c018588a>] ? __do_fault+0x17a/0x340
[33034.505239]  [<c018728a>] ? handle_mm_fault+0xfa/0x5f0
[33034.505239]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[33034.505239]  [<c01aaa2f>] sys_select+0x3f/0x190
[33034.505239]  [<c0112497>] ? __ipipe_handle_exception+0xd7/0x1d0
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] nmbd          S [0x00000001] c0128feb     0  2262      1
[33034.505239]        f7fa7b30 00000086 f7fa7b14 c0128feb f7f927f0  
f7f92a44 c0117a30 f7fa7b40
[33034.505239]        f7fa7b30 c012917f 00000000 00000200 f7fa7b40  
01f37ed4 00000000 f7fa7b60
[33034.505239]        c03bffc7 f74ead00 f7fa7bd0 c056fdb8 c056fdb8  
01f37ed4 c0128dd0 f7f927f0
[33034.505239] Call Trace:
[33034.505239]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[33034.505239]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[33034.505239]  [<c0128dd0>] ? process_timeout+0x0/0x10
[33034.505239]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505239]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c0362941>] ? ip_dev_loopback_xmit+0x51/0x90
[33034.505239]  [<c0362e85>] ? ip_mc_output+0xb5/0xf0
[33034.505239]  [<c0360f3b>] ? ip_cork_release+0x2b/0x40
[33034.505239]  [<c03611ea>] ? ip_push_pending_frames+0x29a/0x370
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c037cb6d>] ? udp_push_pending_frames+0x1ad/0x3e0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c0198add>] ? kfree+0x9d/0xf0
[33034.505239]  [<c0340caa>] ? memcpy_toiovec+0x4a/0x70
[33034.505239]  [<c033ecec>] ? skb_release_data+0x5c/0x90
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c033ecec>] ? skb_release_data+0x5c/0x90
[33034.505239]  [<c033eb9c>] ? __kfree_skb+0x3c/0x90
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[33034.505239]  [<c033afc9>] ? release_sock+0xb9/0xd0
[33034.505239]  [<c037e9e4>] ? udp_recvmsg+0x184/0x300
[33034.505239]  [<c033aa23>] ? sock_common_recvmsg+0x43/0x60
[33034.505239]  [<c0338fb6>] ? sock_recvmsg+0xc6/0xf0
[33034.505239]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505239]  [<c0346179>] ? netdev_run_todo+0x29/0x270
[33034.505239]  [<c0343a0d>] ? __dev_get_by_name+0x8d/0xb0
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[33034.505239]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[33034.505239]  [<c01abeec>] ? __posix_lock_file+0x6c/0x550
[33034.505239]  [<c0385199>] ? inet_sock_destruct+0xe9/0x1f0
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[33034.505239]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[33034.505239]  [<c01aaac4>] sys_select+0xd4/0x190
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] smbd          S [0x00000001] f74e1b0c     0  2264      1
[33034.505239]        f74e1b30 00000082 f794d0e0 f74e1b0c f7f92c60  
f7f92eb4 c0117a30 f74e1c1c
[33034.505239]        f74e1b2c c0133c22 00000200 00000002 00000015  
00100000 00000000 f74e1b60
[33034.505239]        c03bfff5 f74b0180 f74e1bd0 f74b0180 f7cf5e00  
f74e1b60 c01a2b5f 00000000
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c01a2b5f>] ? pipe_poll+0x2f/0xb0
[33034.505239]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505239]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02a9598>] ? extract_buf+0x88/0xf0
[33034.505239]  [<c02a9598>] ? extract_buf+0x88/0xf0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0108679>] ? read_tsc+0x9/0x30
[33034.505239]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[33034.505239]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505239]  [<c01794da>] ? filemap_fault+0x12a/0x480
[33034.505239]  [<c013d146>] ? clockevents_program_event+0xb6/0x120
[33034.505239]  [<c0176e24>] ? unlock_page+0x24/0x30
[33034.505239]  [<c018588a>] ? __do_fault+0x17a/0x340
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c018728a>] ? handle_mm_fault+0xfa/0x5f0
[33034.505239]  [<c010f1c0>] ? smp_apic_timer_interrupt+0x0/0x70
[33034.505239]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[33034.505239]  [<c01aaa2f>] sys_select+0x3f/0x190
[33034.505239]  [<c0112497>] ? __ipipe_handle_exception+0xd7/0x1d0
[33034.505239]  [<c0111f41>] ? __ipipe_handle_irq+0x81/0x1a0
[33034.505239]  [<c01a8b5f>] ? do_fcntl+0x28f/0x300
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] smbd          S [0x00000001] f757dfb0     0  2274   2264
[33034.505239]        f757dfa8 00000086 c014c146 f757dfb0 f7c967f0  
f7c96a44 f74b0a80 1e46bbf3
[33034.505239]        00000005 00000000 0000000e f757dfb0 00000000  
00000001 00000003 f757dfb0
[33034.505239]        c012bd13 f757c000 c01030d5 00000000 00000019  
0a201fd8 00000001 00000003
[33034.505239] Call Trace:
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c012bd13>] sys_pause+0x13/0x20
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] sshd          S [0x00000001] f793eb60     0  2279      1
[33034.505239]        f74dfb30 00000082 c0117a30 f793eb60 f7c97540  
f7c97794 f74dfc00 f0367a07
[33034.505239]        00001e07 00000000 00000001 00000000 00000005  
00000010 00000000 f74dfb60
[33034.505239]        c03bfff5 f74dfbd0 f7fa1540 f75ab580 f74dfb54  
c0368657 c03ea300 00000010
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0368657>] ? tcp_poll+0x17/0x160
[33034.505239]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505239]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c01f21ff>] ? __ext3_journal_dirty_metadata+0x1f/0x50
[33034.505239]  [<c01e5aa1>] ? ext3_mark_iloc_dirty+0x181/0x350
[33034.505239]  [<c01e5fbe>] ? ext3_reserve_inode_write+0x5e/0x80
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0108679>] ? read_tsc+0x9/0x30
[33034.505239]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[33034.505239]  [<c0296228>] ? number+0x288/0x2a0
[33034.505239]  [<c0154b11>] ? xnarch_next_htick_shot+0x51/0x60
[33034.505239]  [<c013d146>] ? clockevents_program_event+0xb6/0x120
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0296b21>] ? vsnprintf+0x371/0x6d0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[33034.505239]  [<c017c956>] ? get_page_from_freelist+0x2d6/0x530
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c017c8a7>] ? get_page_from_freelist+0x227/0x530
[33034.505239]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0137365>] ? hrtimer_start+0xc5/0x180
[33034.505239]  [<c011994e>] ? hrtick_set+0xbe/0x1a0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01969a5>] ? add_partial+0x45/0x80
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c019803b>] ? __slab_free+0x5b/0x2f0
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c0198add>] ? kfree+0x9d/0xf0
[33034.505239]  [<c01a313c>] ? __free_pipe_info+0x3c/0x50
[33034.505239]  [<c01a313c>] ? __free_pipe_info+0x3c/0x50
[33034.505239]  [<c01a313c>] ? __free_pipe_info+0x3c/0x50
[33034.505239]  [<c01a3161>] ? free_pipe_info+0x11/0x20
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[33034.505239]  [<c01aaa2f>] sys_select+0x3f/0x190
[33034.505239]  [<c019dc58>] ? fput+0x18/0x20
[33034.505239]  [<c019abe7>] ? filp_close+0x47/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] proftpd       S [0x00000001] c0128feb     0  2366      1
[33034.505239]        f7487b30 00200086 f7487b14 c0128feb f7c8a380  
f7c8a5d4 c0117a30 f7487b40
[33034.505239]        f7487b30 c012917f 00000000 00000200 f7487b40  
01f3e58d 00000000 f7487b60
[33034.505239]        c03bffc7 f7487bd0 f7fa1980 c056fe40 c04ace90  
01f3e58d c0128dd0 f7c8a380
[33034.505239] Call Trace:
[33034.505239]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[33034.505239]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[33034.505239]  [<c0128dd0>] ? process_timeout+0x0/0x10
[33034.505239]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505239]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011cb92>] ? check_preempt_wakeup+0x62/0xb0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c011a3f8>] ? try_to_wake_up+0x58/0x130
[33034.505239]  [<c0108679>] ? read_tsc+0x9/0x30
[33034.505239]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[33034.505239]  [<c0137576>] ? ktime_get_ts+0x46/0x50
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01bfae7>] ? __find_get_block_slow+0x67/0x110
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01bfeeb>] ? __find_get_block+0x8b/0x1c0
[33034.505239]  [<c01f2514>] ? __journal_file_buffer+0x74/0x160
[33034.505239]  [<c01f2b92>] ? do_get_write_access+0x1d2/0x440
[33034.505239]  [<c01f2514>] ? __journal_file_buffer+0x74/0x160
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01f41ff>] ? journal_dirty_metadata+0x7f/0x120
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[33034.505239]  [<c01f28a3>] ? journal_stop+0x123/0x240
[33034.505239]  [<c01f28a3>] ? journal_stop+0x123/0x240
[33034.505239]  [<c01edcd7>] ? __ext3_journal_stop+0x27/0x50
[33034.505239]  [<c01e8df0>] ? ext3_dirty_inode+0x60/0x80
[33034.505239]  [<c01bc19c>] ? __mark_inode_dirty+0x2c/0x1a0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01aee28>] ? __d_lookup+0xc8/0x180
[33034.505239]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[33034.505239]  [<c01b4476>] ? mntput_no_expire+0x16/0x100
[33034.505239]  [<c01a4530>] ? path_put+0x20/0x30
[33034.505239]  [<c01a691b>] ? path_walk+0x4b/0x90
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c01a7646>] ? __user_walk_fd+0x46/0x60
[33034.505239]  [<c01a7646>] ? __user_walk_fd+0x46/0x60
[33034.505239]  [<c01a019e>] ? vfs_lstat_fd+0x1e/0x50
[33034.505239]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0137365>] ? hrtimer_start+0xc5/0x180
[33034.505239]  [<c01aaac4>] sys_select+0xd4/0x190
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] cron          S [0x00000001] 00000001     0  2378      1
[33034.505239]        f74c7f3c 00000082 c0117a30 00000001 f7c8b0d0  
f7c8b324 dbcd4aab 00001e04
[33034.505239]        c0478454 00000000 00000200 00000000 f74c7f5c  
f7c8b0d0 f74c7f5c f74c7f50
[33034.505239]        c03c0725 00000001 00000000 00000001 f74c7f98  
c0137462 bfea85e4 f741de41
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03c0725>] do_nanosleep+0x65/0x90
[33034.505239]  [<c0137462>] hrtimer_nanosleep+0x42/0xb0
[33034.505239]  [<c0136e50>] ? hrtimer_wakeup+0x0/0x20
[33034.505239]  [<c0137528>] sys_nanosleep+0x58/0x60
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] xdm           S [0x00000001] 00000200     0  2400      1
[33034.505239]        f7497b30 00000086 c0133c22 00000200 f7c88d50  
f7c88fa4 f7497bd0 f7497b2c
[33034.505239]        c01aa97d f798b918 f74d4580 f756d400 00000005  
00000010 00000000 f7497b60
[33034.505239]        c03bfff5 f74d4580 f7497b54 c037c7b2 00497b54  
c03ea360 00000010 f74d4580
[33034.505239] Call Trace:
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c01aa97d>] ? __pollwait+0x6d/0xe0
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c037c7b2>] ? udp_poll+0x12/0x100
[33034.505239]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505239]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[33034.505239]  [<c017c956>] ? get_page_from_freelist+0x2d6/0x530
[33034.505239]  [<c017cc50>] ? __alloc_pages_internal+0xa0/0x420
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01bfeeb>] ? __find_get_block+0x8b/0x1c0
[33034.505239]  [<c01338df>] ? wake_up_bit+0x1f/0x30
[33034.505239]  [<c01f2b92>] ? do_get_write_access+0x1d2/0x440
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01f41ff>] ? journal_dirty_metadata+0x7f/0x120
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[33034.505239]  [<c01f28a3>] ? journal_stop+0x123/0x240
[33034.505239]  [<c01f28a3>] ? journal_stop+0x123/0x240
[33034.505239]  [<c01edcd7>] ? __ext3_journal_stop+0x27/0x50
[33034.505239]  [<c01e8df0>] ? ext3_dirty_inode+0x60/0x80
[33034.505239]  [<c010311c>] ? restore_nocheck_notrace+0x0/0xe
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[33034.505239]  [<c01b1efd>] ? touch_atime+0x2d/0x110
[33034.505239]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c017c45b>] ? free_hot_cold_page+0x15b/0x1e0
[33034.505239]  [<c017c52a>] ? free_hot_page+0xa/0x10
[33034.505239]  [<c017fff4>] ? put_page+0x34/0x110
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0185e91>] ? unmap_vmas+0x291/0x480
[33034.505239]  [<c0186e46>] ? free_pgtables+0x86/0xb0
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c01893d7>] ? remove_vma+0x47/0x60
[33034.505239]  [<c01aaa2f>] sys_select+0x3f/0x190
[33034.505239]  [<c012a113>] ? sys_rt_sigaction+0x63/0xa0
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] login         S [0x00000001] f7553f2c     0  2412      1
[33034.505239]        f7553f20 00000082 f74aa140 f7553f2c f7f548e0  
f7f54b34 f7553f08 c0117a30
[33034.505239]        f7553f70 f7553f20 c0133c22 00000200 ffffffea  
f7f548d8 00000004 f7553f90
[33034.505239]        c01221e5 f7f91f10 f7553f78 c0113832 00000001  
00000000 f7f54ac8 00000000
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c01221e5>] do_wait+0x5d5/0xbd0
[33034.505239]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c012284f>] sys_wait4+0x6f/0xc0
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] getty         S [0x00000001] f7493800     0  2414      1
[33034.505239]        f760feb0 00000086 f7458dc0 f7493800 f7f55aa0  
f7f55cf4 c17ec760 2dd8d213
[33034.505239]        00000006 00000000 f760fea8 c0117a30 00000001  
f7479c00 00000000 f760fee0
[33034.505239]        c03bfff5 00000000 f760fec8 c0117a30 f760ff20  
f760fee0 c0133c22 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c02b193b>] read_chan+0x1bb/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] getty         S [0x00000001] f7493c00     0  2416      1
[33034.505239]        f7451eb0 00000082 f75508f0 f7493c00 f7f56c60  
f7f56eb4 c17ec760 2dd8d215
[33034.505239]        00000006 00000000 f7451ea8 c0117a30 00000001  
f7479400 00000000 f7451ee0
[33034.505239]        c03bfff5 00000000 f7451ec8 c0117a30 f7451f20  
f7451ee0 c0133c22 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c02b193b>] read_chan+0x1bb/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] getty         S [0x00000001] f7493e00     0  2418      1
[33034.505239]        f7539eb0 00000082 f7550630 f7493e00 f7f55630  
f7f55884 c17ec760 2dd8d217
[33034.505239]        00000006 00000000 f7539ea8 c0117a30 00000001  
f7479800 00000000 f7539ee0
[33034.505239]        c03bfff5 00000000 f7539ec8 c0117a30 f7539f20  
f7539ee0 c0133c22 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c02b193b>] read_chan+0x1bb/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] getty         S [0x00000001] f7fb5280     0  2419      1
[33034.505239]        f74e5eb0 00000086 f7d152c0 f7fb5280 f7f54000  
f7f54254 c17ec760 2dd8d219
[33034.505239]        00000006 00000000 f74e5ea8 c0117a30 00000001  
f74c5400 00000000 f74e5ee0
[33034.505239]        c03bfff5 00000000 f74e5ec8 c0117a30 f74e5f20  
f74e5ee0 c0133c22 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c02b193b>] read_chan+0x1bb/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] getty         S [0x00000001] f7fb5d00     0  2420      1
[33034.505239]        f749feb0 00000082 f7d15420 f7fb5d00 f7f57540  
f7f57794 c17ec760 2dd8d21a
[33034.505239]        00000006 f749fea4 f749fea8 c0117a30 00000001  
f75d2800 00000000 f749fee0
[33034.505239]        c03bfff5 00000000 f749fec8 c0117a30 f749ff20  
f749fee0 c0133c22 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c02b193b>] read_chan+0x1bb/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] bash          S [0x00000001] c00bca60     0  2438   2412
[33034.505239]        f7f6deb0 00000082 f7f6de90 c00bca60 f7f91f10  
f7f92164 0000000e f7f6deb4
[33034.505239]        c02a68a6 0000000e f7f6dea8 c0117a30 00000001  
f7479000 00000000 f7f6dee0
[33034.505239]        c03bfff5 f7f6decc f7f6dec8 c0117a30 f7f6df20  
f7f6dee0 c0133c22 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c02a68a6>] ? vgacon_set_cursor_size+0xc6/0x170
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c02b193b>] read_chan+0x1bb/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] sshd          S [0x00000001] 00000001     0 17390   2279
[33034.505239]        f755db30 00000082 c0117a30 00000001 f7c89f10  
f7c8a164 00000000 00000000
[33034.505239]        c0484da0 00000000 f755db28 c0117a30 00000008  
00000080 00000000 f755db60
[33034.505239]        c03bfff5 f763cc00 c02b0380 f755db60 c02abdd0  
f755dbd0 f763cc10 00000104
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c02b0380>] ? normal_poll+0x0/0x150
[33034.505239]  [<c02abdd0>] ? tty_poll+0x70/0x80
[33034.505239]  [<c01aa2b4>] do_select+0x494/0x590
[33034.505239]  [<f88b6bc1>] ? ieee80211_skb_resize+0x61/0xc0 [mac80211]
[33034.505239]  [<c01aa910>] ? __pollwait+0x0/0xe0
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c0362516>] ? ip_finish_output+0x116/0x2b0
[33034.505239]  [<c0362f24>] ? ip_output+0x64/0x80
[33034.505239]  [<c0360e38>] ? ip_local_out+0x18/0x20
[33034.505239]  [<c03630fd>] ? ip_queue_xmit+0x1bd/0x320
[33034.505239]  [<c0137576>] ? ktime_get_ts+0x46/0x50
[33034.505239]  [<c0137591>] ? ktime_get+0x11/0x30
[33034.505239]  [<c01172cb>] ? hrtick_start_fair+0xeb/0x140
[33034.505239]  [<c0378230>] ? tcp_v4_send_check+0x40/0xd0
[33034.505239]  [<c037317e>] ? tcp_transmit_skb+0x3ae/0x770
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[33034.505239]  [<c037290f>] ? tcp_init_tso_segs+0x3f/0x60
[33034.505239]  [<c0374a5a>] ? __tcp_push_pending_frames+0x2ea/0x750
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[33034.505239]  [<c033afc9>] ? release_sock+0xb9/0xd0
[33034.505239]  [<c0369aca>] ? tcp_sendmsg+0x75a/0xa40
[33034.505239]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[33034.505239]  [<c03383db>] ? sock_aio_write+0xeb/0x110
[33034.505239]  [<c019c65c>] ? do_sync_write+0xcc/0x110
[33034.505239]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505239]  [<c0124226>] ? current_fs_time+0x16/0x20
[33034.505239]  [<c01aaa2f>] sys_select+0x3f/0x190
[33034.505239]  [<c019d46d>] ? sys_write+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] bash          S [0x00000001] c0137365     0 17392  17390
[33034.505239]        f7499eb0 00000082 f7499ea8 c0137365 f7c8ac60  
f7c8aeb4 c0478454 38c98c11
[33034.505239]        00000bea 00000000 c04ee300 3965f5d5 00000001  
f763c400 00000000 f7499ee0
[33034.505239]        c03bfff5 ffffffff f7499ec8 c0117a30 f7499f20  
f7499ee0 c0133c22 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c0137365>] ? hrtimer_start+0xc5/0x180
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c02b193b>] read_chan+0x1bb/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] remoteHost    S [0x00000001] 000084d0     0 17405  17392
[33034.505239]        f76a1e34 00000086 f7f54470 000084d0 f7f54470  
f7f546c4 c047efe0 00000001
[33034.505239]        00000044 000280d0 000284d0 c047efe0 f7fa3b80  
00000000 7fffffff f76a1e64
[33034.505239]        c03bfff5 c0117a30 f7fa3b80 f76a1e4c c0124ab5  
f76a1e64 c033afc9 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[33034.505239]  [<c033afc9>] ? release_sock+0xb9/0xd0
[33034.505239]  [<c0366c69>] inet_csk_accept+0x119/0x250
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505239]  [<c038501f>] inet_accept+0x1f/0xb0
[33034.505239]  [<c033a093>] sys_accept+0xd3/0x1c0
[33034.505239]  [<c01874e2>] ? handle_mm_fault+0x352/0x5f0
[33034.505239]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfa9f>] ? preempt_schedule+0x4f/0x60
[33034.505239]  [<c011cd5b>] ? wake_up_new_task+0xbb/0xc0
[33034.505239]  [<c033a241>] sys_socketcall+0xc1/0x260
[33034.505239]  [<c019abe7>] ? filp_close+0x47/0x70
[33034.505239]  [<c019c1db>] ? sys_close+0x7b/0xf0
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] remoteHost    S [0x00000001] c76f2ef7     0 17406  17405
[33034.505239]        f769bf08 00000082 f7627780 c76f2ef7 f7f567f0  
f7f56a44 f769bef0 c7098ee0
[33034.505239]        0000137d 00000000 c0133c22 00000200 ffffffea  
f7f567e8 00000004 f769bf78
[33034.505239]        c01221e5 f7ce4470 f7627780 f7634bf8 f7f56a48  
00000000 f7f569d8 f764e7c0
[33034.505239] Call Trace:
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c01221e5>] do_wait+0x5d5/0xbd0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03bfa9f>] ? preempt_schedule+0x4f/0x60
[33034.505239]  [<c011cd5b>] ? wake_up_new_task+0xbb/0xc0
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c012284f>] sys_wait4+0x6f/0xc0
[33034.505239]  [<c01228c5>] sys_waitpid+0x25/0x30
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] remoteHost    S [0x00000001] f7fa2640     0 17407  17405
[33034.505239]        f7643ce0 00000082 f7650840 f7fa2640 f7f55f10  
f7f56164 00000000 f7fa2640
[33034.505239]        f7643cf4 c037215d c014c146 bff74101 f7643d70  
f7fa2640 f7fa26a0 f7643d10
[33034.505239]        c03bfff5 c0117a30 f7643d70 f7643cf8 c0124ab5  
f7643d10 c033afc9 00000200
[33034.505239] Call Trace:
[33034.505239]  [<c037215d>] ? tcp_rcv_established+0x40d/0x6c0
[33034.505239]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[33034.505239]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[33034.505239]  [<c033afc9>] ? release_sock+0xb9/0xd0
[33034.505239]  [<c033ba69>] sk_wait_data+0x79/0xb0
[33034.505239]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505239]  [<c036a12e>] tcp_recvmsg+0x37e/0x820
[33034.505239]  [<c033aa23>] sock_common_recvmsg+0x43/0x60
[33034.505239]  [<c0338fb6>] sock_recvmsg+0xc6/0xf0
[33034.505239]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[33034.505239]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[33034.505239]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[33034.505239]  [<c0129d8e>] ? __sigqueue_free+0x2e/0x40
[33034.505239]  [<c012a229>] ? __dequeue_signal+0xd9/0x180
[33034.505239]  [<c0338930>] ? sockfd_lookup_light+0x30/0x60
[33034.505239]  [<c0339e0d>] sys_recvfrom+0x7d/0xd0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c01874e2>] ? handle_mm_fault+0x352/0x5f0
[33034.505239]  [<c0339e92>] sys_recv+0x32/0x40
[33034.505239]  [<c033a2d8>] sys_socketcall+0x158/0x260
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] rt61pci       S [0x00000001] c0117a30     0 27579      2
[33034.505239]        f76e5fa4 00000046 f76e5f7c c0117a30 f7ce70d0  
f7ce7324 c0117a30 f7ce70d0
[33034.505239]        f76e5fa4 c0133b8b 00000001 00000200 f764ef80  
f764ef88 f76e5fac f76e5fcc
[33034.505239]        c01307f5 00000000 f7ce70d0 c01338f0 f764ef88  
f764ef88 fffffffc f764ef80
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[33034.505239]  [<c01307f5>] worker_thread+0x95/0xb0
[33034.505239]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[33034.505239]  [<c0130760>] ? worker_thread+0x0/0xb0
[33034.505239]  [<c01335a2>] kthread+0x42/0x70
[33034.505239]  [<c0133560>] ? kthread+0x0/0x70
[33034.505239]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[33034.505239]  =======================
[33034.505239] actuatorTask  ? [0x00000040] 00000001     0 28213      1
[33034.505239]        f760df54 00000046 c0198b26 00000001 f7ce48e0  
f7ce4b34 f765a540 c6b973df
[33034.505239]        0000137d 00000000 f7ce48d8 f7ce48e0 00000010  
f7ce48e0 f760ddd8 f760df98
[33034.505239]        c0122e14 c04ee300 00000000 f7ce51c0 00000001  
f760df84 f7ce4a84 f7ce4ac8
[33034.505239] Call Trace:
[33034.505239]  [<c0198b26>] ? kfree+0xe6/0xf0
[33034.505239]  [<c0122e14>] do_exit+0x494/0x710
[33034.505239]  [<c01230be>] do_group_exit+0x2e/0xb0
[33034.505239]  [<c012314f>] sys_exit_group+0xf/0x20
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] Actuator Aper S [0x00000201] f7f01f04     0 28214      1
[33034.505239]        f7f01f28 00000086 c057be64 f7f01f04 f7ce51c0  
f7ce5414 f7f01f28 ce2e19e5
[33034.505239]        0000137c 00000000 00000001 00000200 f7d80620  
f7ce51c0 c01586e0 f7f01f4c
[33034.505239]        c0158576 f7f01f6c c015891a 00000000 f7f01ebc  
00000006 00000000 c01586e0
[33034.505239] Call Trace:
[33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
[33034.505239]  [<c0158576>] xnshadow_harden+0x86/0x1f0
[33034.505239]  [<c015891a>] ? hisyscall_event+0xba/0x290
[33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
[33034.505239]  [<c015876f>] losyscall_event+0x8f/0x180
[33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
[33034.505239]  [<c014c89c>] __ipipe_dispatch_event+0x9c/0x170
[33034.505239]  [<c01122f2>] __ipipe_syscall_root+0x42/0x110
[33034.505239]  [<c01030ad>] system_call+0x29/0x4a
[33034.505239]  =======================
[33034.505239] sh            S [0x00000001] 39eff3e0     0 28229  17406
[33034.505239]        f7665f08 00000082 f74bba80 39eff3e0 f7ce4470  
f7ce46c4 f7665ef0 c0117a30
[33034.505239]        f7665f58 f7665f08 c0133c22 00000200 ffffffea  
f7ce4468 00000004 f7665f78
[33034.505239]        c01221e5 f7c2f0d0 f74bba80 f7506bfc f7ce46c8  
00000000 f7ce4658 00000000
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[33034.505239]  [<c01221e5>] do_wait+0x5d5/0xbd0
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c012284f>] sys_wait4+0x6f/0xc0
[33034.505239]  [<c01228c5>] sys_waitpid+0x25/0x30
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] sshd          R [0x00000000] running      0  9641   2279
[33034.505239]        f7523e84 00000086 c011a4db f7523e68 f7c2f540  
f7c2f798 f078bddc 71f337e7
[33034.505239]        00001e0b 00000000 00000000 00000001 00000000  
00000001 00000001 f7523e8c
[33034.505239]        c03bfa95 f7523eb0 c011c3c9 00000000 00000000  
f766e914 00000200 f766e800
[33034.505239] Call Trace:
[33034.505239]  [<c011a4db>] ? default_wake_function+0xb/0x10
[33034.505239]  [<c03bfa95>] preempt_schedule+0x45/0x60
[33034.505239]  [<c011c3c9>] __wake_up+0x99/0xa0
[33034.505239]  [<c02aabd3>] tty_wakeup+0x33/0x70
[33034.505239]  [<c02b30e5>] pty_unthrottle+0x15/0x20
[33034.505239]  [<c02b1eb1>] tty_unthrottle+0x21/0x30
[33034.505239]  [<c02afe05>] check_unthrottle+0x15/0x20
[33034.505239]  [<c02b1b5e>] read_chan+0x3de/0x680
[33034.505239]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[33034.505239]  [<c02ae0a7>] tty_read+0x77/0xb0
[33034.505239]  [<c02b1780>] ? read_chan+0x0/0x680
[33034.505239]  [<c019cfc4>] vfs_read+0x94/0x130
[33034.505239]  [<c02ae030>] ? tty_read+0x0/0xb0
[33034.505239]  [<c019d3fd>] sys_read+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] bash          R [0x00000000] running      0  9645   9641
[33034.505239]        f078bd80 f078bd78 f078bd78 c0117a30 00000031  
f078bda8 c02be55a 00000018
[33034.505239]        00000000 ffffffff c0514e38 00b28ed7 c00bd922  
00000009 c0485fa0 00025d5c
[33034.505239]        00000016 f078bdbc f078bdb8 c0117a30 c0476a58  
f078bdcc c0138047 f078bdcc
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c011fcf1>] ? release_console_sem+0x211/0x250
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c0120125>] ? vprintk+0x1f5/0x3e0
[33034.505239]  [<c0117a00>] ? hrtick_clear+0x0/0x20
[33034.505239]  [<c0120125>] ? vprintk+0x1f5/0x3e0
[33034.505239]  [<c0120a2b>] ? printk+0x13b/0x150
[33034.505239]  [<c010f6ce>] ? touch_nmi_watchdog+0x1e/0x20
[33034.505239]  [<c0104085>] ? print_trace_address+0x45/0x50
[33034.505239]  [<c01313db>] ? __kernel_text_address+0x1b/0x30
[33034.505239]  [<c0104119>] ? dump_trace+0x89/0x120
[33034.505239]  [<c01041d7>] show_trace_log_lvl+0x27/0x50
[33034.505239]  [<c01042b0>] show_stack_log_lvl+0xb0/0xe0
[33034.505239]  [<c0104b07>] show_stack+0x37/0x50
[33034.505239]  [<c0117963>] sched_show_task+0x93/0xd0
[33034.505239]  [<c011c156>] show_state_filter+0x66/0xb0
[33034.505239]  [<c02be96a>] sysrq_handle_showstate+0xa/0x10
[33034.505239]  [<c02bec4d>] __handle_sysrq+0x9d/0x180
[33034.505239]  [<c01d8d80>] ? write_sysrq_trigger+0x0/0x40
[33034.505239]  [<c01d8daf>] write_sysrq_trigger+0x2f/0x40
[33034.505239]  [<c01d1e4c>] proc_reg_write+0x6c/0xc0
[33034.505239]  [<c019ce99>] vfs_write+0x99/0x130
[33034.505239]  [<c01d1de0>] ? proc_reg_write+0x0/0xc0
[33034.505239]  [<c019d46d>] sys_write+0x3d/0x70
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] sleep         S [0x00000001] 00000001     0  9709  28229
[33034.505239]        f07c3f3c 00000082 c0117a30 00000001 f7c2f0d0  
f7c2f324 395d65ee 395d6206
[33034.505239]        00001e0b 00000000 00000200 00000000 f07c3f5c  
f7c2f0d0 f07c3f5c f07c3f50
[33034.505239]        c03c0725 00000001 00000000 00000001 f07c3f98  
c0137462 00000000 f741de40
[33034.505239] Call Trace:
[33034.505239]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[33034.505239]  [<c03c0725>] do_nanosleep+0x65/0x90
[33034.505239]  [<c0137462>] hrtimer_nanosleep+0x42/0xb0
[33034.505239]  [<c0136e50>] ? hrtimer_wakeup+0x0/0x20
[33034.505239]  [<c0137528>] sys_nanosleep+0x58/0x60
[33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
[33034.505239]  =======================
[33034.505239] Sched Debug Version: v0.07, 2.6.26.5 #5
[33034.505239] now at 33034597.126198 msecs
[33034.505239]   .sysctl_sched_latency                    : 20.000000
[33034.505239]   .sysctl_sched_min_granularity            : 4.000000
[33034.505239]   .sysctl_sched_wakeup_granularity         : 10.000000
[33034.505239]   .sysctl_sched_child_runs_first           : 0.000001
[33034.505239]   .sysctl_sched_features                   : 895
[33034.505239]
[33034.505239] cpu#0, 2000.117 MHz
[33034.505239]   .nr_running                    : 3
[33034.505239]   .load                          : 3072
[33034.505239]   .nr_switches                   : 17842168
[33034.505239]   .nr_load_updates               : 542386
[33034.505239]   .nr_uninterruptible            : 0
[33034.505239]   .jiffies                       : 32734505
[33034.505239]   .next_balance                  : 0.000000
[33034.505239]   .curr->pid                     : 9645
[33034.505239]   .clock                         : 33034505.238505
[33034.505239]   .cpu_load[0]                   : 0
[33034.505239]   .cpu_load[1]                   : 0
[33034.505239]   .cpu_load[2]                   : 57
[33034.505239]   .cpu_load[3]                   : 304
[33034.505239]   .cpu_load[4]                   : 538
[33034.505239]
[33034.505239] cfs_rq[0]:
[33034.505239]   .exec_clock                    : 1886543.312516
[33034.505239]   .MIN_vruntime                  : 2038080.995791
[33034.505239]   .min_vruntime                  : 2038100.995791
[33034.505239]   .max_vruntime                  : 2038100.995791
[33034.505239]   .spread                        : 20.000000
[33034.505239]   .spread0                       : 0.000000
[33034.505239]   .nr_running                    : 3
[33034.505239]   .load                          : 3072
[33034.505239]   .bkl_count                     : 116440
[33034.505239]   .nr_spread_over                : 17371
[33034.505239]
[33034.505239] runnable tasks:
[33034.505239]             task   PID         tree-key  switches   
prio     exec-runtime         sum-exec        sum-sleep
[33034.505239]  
----------------------------------------------------------------------------------------------------------
[33034.505239]            klogd  2244   2038080.995791      1243    
120   2038080.995791        97.397021  33015637.631792
[33034.505239]             sshd  9641   2038100.995791       143    
120   2038100.995791        86.680879     15122.725577
[33034.505239] R           bash  9645   2038080.995791        45    
120   2038080.995791         4.542348     15003.827423
[33034.505239]


----- dump ends here -----


On Dec 22, 2008, at 3:03 PM, Gilles Chanteperdrix wrote:

> Mehmet Alphan Ulusoy wrote:
>> On Mon 22 Dec 2008 11:48:00 EET, Gilles Chanteperdrix wrote:
>>> What would be really interesting too, is to have the process  
>>> status. So,
>>> in function sched_show_task, file kernel/sched.c, could you try and
>>> print p->state in hexadecimal? In the dump you sent, the state is  
>>> shown
>>> as a question mark, so we do not really know in what state the  
>>> task is.
>>>
>>
>> The output of the sched says the task is 'relaxed shadow', I believe
>> what you ask for is something else. I will try to make the change you
>> suggested and hopefully send you the dump once the task turns into a
>> zombie.
>
> Unless I misunderstood something, the zombie task (called  
> actuatorTask)
> does not appear in /proc/xenomai/sched and /proc/xenomai/stat, the  
> task
> which appears there is "Actuator Aperiodic Task". actuatorTask,  
> however,
> appears in the sysrq+T output, and its state (for Linux scheduler, not
> for xenomai scheduler, since for xenomai scheduler it does not even
> exist) is "?", so, what I would like to know is its real state,  
> printed
> in hexadecimal. For this, you just have to print p->state using 0x%08x
> as printk format.
>
> -- 
>                                                 Gilles.
>


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2433 bytes --]

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Alphan Ulusoy wrote:
> I've done the changes you suggested ( addition to sched.c file and  
> fram pointers) and left the tasks running last night. Below you can  
> find the dump. The hex numbers inside the brackets ( [0x00000001] )  
> are newly added. And as you said in your previous post "Actuator  
> Aperiodic Task" is the real-time task created and started in the  
> process called "actuatorTask" which turns into a zombie. The ps aux  
> output says:
> 
> root     28213  0.0  0.0      0     0 pts/0    Zl   04:21   0:02  
> [actuatorTask] <defunct>
> 
> 
> To give you a heads up, the actuator Aperiodic task is a simple  
> for(;;) loop which blocks on recvfrom() on a UDP socket at every  
> iteration until a packet is received. Then it writes the data it  
> contains to a shared variable acquiring and releasing a mutex. any  
> ideas?

If you mean that you have a sample code with which I would be able to
reproduce the problem without requiring special hardware, I am very
interested.

> 
> 
> regards,
> 
> alphan.
> 
> 
> 
> ------- Dump starts here ---------
> 
> 
> [33034.505238] SysRq : Show State
> [33034.505238]   task                PC stack   pid father
> (...)
> [33034.505238] gatekeeper/0  S [0x00000001] f7c87f78     0   150      2
> [33034.505238]        f7c87f94 00000046 00000046 f7c87f78 f7c88000  
> f7c88254 c0571f40 be53c9c0
> [33034.505238]        0000137d 00000000 f7c87f94 c0138047 f7d81a20  
> c057be60 c057be6c f7c87fcc
> [33034.505238]        c0158f77 00000003 f7c88000 f7c88254 00000000  
> 00000001 f7c88000 c011a4d0
> [33034.505238] Call Trace:
> [33034.505238]  [<c0138047>] ? up+0x57/0x90
> [33034.505238]  [<c0158f77>] gatekeeper_thread+0xa7/0x140
> [33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
> [33034.505238]  [<c0158ed0>] ? gatekeeper_thread+0x0/0x140
> [33034.505238]  [<c01335a2>] kthread+0x42/0x70
> [33034.505238]  [<c0133560>] ? kthread+0x0/0x70
> [33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14

So, the gatekeeper is in TASK_INTERRUPTIBLE state, waiting for a thread
to call "wake_up_interruptible_sync" in xnshadow_harden.


> [33034.505239] actuatorTask  ? [0x00000040] 00000001     0 28213      1
> [33034.505239]        f760df54 00000046 c0198b26 00000001 f7ce48e0  
> f7ce4b34 f765a540 c6b973df
> [33034.505239]        0000137d 00000000 f7ce48d8 f7ce48e0 00000010  
> f7ce48e0 f760ddd8 f760df98
> [33034.505239]        c0122e14 c04ee300 00000000 f7ce51c0 00000001  
> f760df84 f7ce4a84 f7ce4ac8
> [33034.505239] Call Trace:
> [33034.505239]  [<c0198b26>] ? kfree+0xe6/0xf0
> [33034.505239]  [<c0122e14>] do_exit+0x494/0x710
> [33034.505239]  [<c01230be>] do_group_exit+0x2e/0xb0
> [33034.505239]  [<c012314f>] sys_exit_group+0xf/0x20
> [33034.505239]  [<c01030d5>] syscall_call+0x7/0xb

actuatorTask in TASK_DEAD state. So, it is officially dead.

> [33034.505239]  =======================
> [33034.505239] Actuator Aper S [0x00000201] f7f01f04     0 28214      1
> [33034.505239]        f7f01f28 00000086 c057be64 f7f01f04 f7ce51c0  
> f7ce5414 f7f01f28 ce2e19e5
> [33034.505239]        0000137c 00000000 00000001 00000200 f7d80620  
> f7ce51c0 c01586e0 f7f01f4c
> [33034.505239]        c0158576 f7f01f6c c015891a 00000000 f7f01ebc  
> 00000006 00000000 c01586e0
> [33034.505239] Call Trace:
> [33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
> [33034.505239]  [<c0158576>] xnshadow_harden+0x86/0x1f0
> [33034.505239]  [<c015891a>] ? hisyscall_event+0xba/0x290
> [33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
> [33034.505239]  [<c015876f>] losyscall_event+0x8f/0x180
> [33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
> [33034.505239]  [<c014c89c>] __ipipe_dispatch_event+0x9c/0x170
> [33034.505239]  [<c01122f2>] __ipipe_syscall_root+0x42/0x110
> [33034.505239]  [<c01030ad>] system_call+0x29/0x4a

So, "Actuator Aperiodic Task" is blocked inside xnshadow_harden, in
state TASK_INTERRUPTIBLE | TASK_ATOMICSWITCH. This state is
transitional, and, in fact, I think you should not even be able to
observe it. The fact you got it two traces would be impossible, so I
think the tasks are blocked in these states, and what we observe here is
the harden machinery being jamed. When this happens, could you try to
hit sysrq+T again to see if you get the same states ?


-- 
                                                 Gilles.

Re: [Xenomai-help] Zombie user tasks

[Alphan Ulusoy]

[-- Attachment #1: Type: text/plain, Size: 77156 bytes --]

Below you can find the dump that I've just taken with sysRq+T. I left  
the computer as it is for 5+ hours since sending the previous post in  
the morning. Or would you like me to restart everything all over and  
send the new sysRq+T output when this happens again?

-- dump starts here --

[54744.929081] SysRq : Show State
[54744.929153]   task                PC stack   pid father
[54744.929218] init          S [0x00000001] c0128feb     0     1      0
[54744.929288]        f7c2bb30 00000086 f7c2bb14 c0128feb f7c2c000  
f7c2c254 c0117a30 f7c2bb40
[54744.929367]        f7c2bb30 c012917f 00000000 00000200 f7c2bb40  
033ec79d 00000000 f7c2bb60
[54744.929493]        c03bffc7 f7f20400 f7c2bbd0 c056fc00 c056fc00  
033ec79d c0128dd0 f7c2c000
[54744.929624] Call Trace:
[54744.929733]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[54744.929804]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.929873]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[54744.929939]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[54744.930001]  [<c0128dd0>] ? process_timeout+0x0/0x10
[54744.930001]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930001]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930001]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930001]  [<c011cb92>] ? check_preempt_wakeup+0x62/0xb0
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c011a3f8>] ? try_to_wake_up+0x58/0x130
[54744.930001]  [<c011a4db>] ? default_wake_function+0xb/0x10
[54744.930001]  [<c013390b>] ? autoremove_wake_function+0x1b/0x50
[54744.930001]  [<c011687e>] ? __wake_up_common+0x3e/0x70
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[54744.930001]  [<c012fd8b>] ? insert_work+0x4b/0x70
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c013032d>] ? __queue_work+0x5d/0x90
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c013068c>] ? queue_work+0x4c/0x70
[54744.930001]  [<f88ae37e>] ? ieee80211_sta_rx_mgmt+0xae/0xf0  
[mac80211]
[54744.930001]  [<f88b3ff3>] ? ieee80211_invoke_rx_handlers 
+0x1223/0x1330 [mac80211]
[54744.930001]  [<c010311c>] ? restore_nocheck_notrace+0x0/0xe
[54744.930001]  [<c0105a70>] ? do_IRQ+0x0/0x70
[54744.930001]  [<c014bf4a>] ? __ipipe_walk_pipeline+0x6a/0xb0
[54744.930001]  [<f88b43a4>] ? __ieee80211_rx_handle_packet 
+0x2a4/0x610 [mac80211]
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c01aee28>] ? __d_lookup+0xc8/0x180
[54744.930001]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[54744.930001]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930001]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[54744.930001]  [<c01b4476>] ? mntput_no_expire+0x16/0x100
[54744.930001]  [<c01a4530>] ? path_put+0x20/0x30
[54744.930001]  [<c01a691b>] ? path_walk+0x4b/0x90
[54744.930001]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930001]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0297e73>] ? copy_to_user+0x43/0x60
[54744.930001]  [<c019ff4b>] ? cp_new_stat64+0xeb/0x100
[54744.930001]  [<c01aaac4>] sys_select+0xd4/0x190
[54744.930001]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930001]  =======================
[54744.930001] kthreadd      S [0x00000001] f7c31fa4     0     2      0
[54744.930001]        f7c31fc8 00000046 00000000 f7c31fa4 f7c2c470  
f7c2c6c4 f7c31fac c90ecb83
[54744.930001]        00001375 00000000 00000000 00000000 c04783f0  
00006e27 f7663da8 f7c31fe0
[54744.930001]        c013374a 00000000 c01335d0 00000000 00000000  
00000000 c0103d43 00000000
[54744.930001] Call Trace:
[54744.930001]  [<c013374a>] kthreadd+0x17a/0x1a0
[54744.930001]  [<c01335d0>] ? kthreadd+0x0/0x1a0
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] ksoftirqd/0   S [0x00000001] 00000000     0     3      2
[54744.930001]        f7c33fc0 00000046 c0117a30 00000000 f7c2c8e0  
f7c2cb34 f7c33fb0 c0124839
[54744.930001]        00000000 f7c33fb0 c014c146 f7c33fc0 00000000  
00000000 c0124b70 f7c33fcc
[54744.930001]        c0124c3d fffffffc f7c33fe0 c01335a2 c0133560  
00000000 00000000 00000000
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0124839>] ? __do_softirq+0xa9/0xc0
[54744.930001]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930001]  [<c0124b70>] ? ksoftirqd+0x0/0x140
[54744.930001]  [<c0124c3d>] ksoftirqd+0xcd/0x140
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] watchdog/0    S [0x00000001] 0a987ddd     0     4      2
[54744.930001]        f7c39f9c 00000046 00000000 0a987ddd f7c2cd50  
f7c2cfa4 00000073 0a98a945
[54744.930001]        00000000 00000000 00000000 f7c39f94 fffffffc  
00000000 00000000 f7c39fcc
[54744.930001]        c01471c8 f7c2cfa4 f7c2c000 03e66246 00000000  
c011c4ec 00000000 00000063
[54744.930001] Call Trace:
[54744.930001]  [<c01471c8>] watchdog+0x58/0x1c0
[54744.930001]  [<c011c4ec>] ? complete+0x6c/0x90
[54744.930001]  [<c0147170>] ? watchdog+0x0/0x1c0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] events/0      S [0x00000001] c0117a30     0     5      2
[54744.930001]        f7c3bfa4 00000046 f7c3bf7c c0117a30 f7c2d1c0  
f7c2d414 c0117a30 f7c2d1c0
[54744.930001]        f7c3bfa4 c0133b8b 00000001 00000200 f7c08140  
f7c08148 f7c3bfac f7c3bfcc
[54744.930001]        c01307f5 00000000 f7c2d1c0 c01338f0 f7c08148  
f7c08148 fffffffc f7c08140
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] khelper       S [0x00000001] c0117a30     0     6      2
[54744.930001]        f7c3dfa4 00000046 f7c3df7c c0117a30 f7c2d630  
f7c2d884 c0117a30 c9d8651e
[54744.930001]        000000d8 c0133b8b 00000001 00000200 f7c081c0  
f7c081c8 f7c3dfac f7c3dfcc
[54744.930001]        c01307f5 00000000 f7c2d630 c01338f0 f7c081c8  
f7c081c8 fffffffc f7c081c0
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] kblockd/0     S [0x00000001] c0117a30     0    54      2
[54744.930001]        f7cd9fa4 00000046 f7cd9f7c c0117a30 f7c970d0  
f7c97324 c0117a30 f7c970d0
[54744.930001]        f7cd9fa4 c0133b8b 00000001 00000200 f7c1af00  
f7c1af08 f7cd9fac f7cd9fcc
[54744.930001]        c01307f5 00000000 f7c970d0 c01338f0 f7c1af08  
f7c1af08 fffffffc f7c1af00
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] khubd         S [0x00000001] 00000002     0    63      2
[54744.930001]        f7cfdf38 00000046 00000000 00000002 f7ce5aa0  
f7ce5cf4 c0117a30 55f4defd
[54744.930001]        00000000 c0133b8b 00000001 00000200 f7cfdfa0  
f7cb8240 00000000 f7cfdfcc
[54744.930001]        c0301449 f7cfdfb8 002160e9 00000000 123e6adf  
00000000 f7c2d658 f7f3b904
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c0301449>] hub_thread+0x649/0xcb0
[54744.930001]  [<c011996e>] ? hrtick_set+0xde/0x1a0
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c03bf76d>] ? schedule+0x2fd/0x570
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c011c4ec>] ? complete+0x6c/0x90
[54744.930001]  [<c0300e00>] ? hub_thread+0x0/0xcb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] kseriod       S [0x00000001] c0198add     0    66      2
[54744.930001]        f7d03f90 00000046 f7d03f84 c0198add f7ce67f0  
f7ce6a44 c0117a30 5761608e
[54744.930001]        00000000 c0133b8b 00000001 00000200 f7d03fac  
00000000 f7f388e0 f7d03fcc
[54744.930001]        c031cb4f 00000001 00000003 f7ce67f0 f7ce6a44  
00000200 00000000 f7ce67f0
[54744.930001] Call Trace:
[54744.930001]  [<c0198add>] ? kfree+0x9d/0xf0
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c031cb4f>] serio_thread+0xff/0x400
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c031ca50>] ? serio_thread+0x0/0x400
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] gatekeeper/0  S [0x00000001] f7c87f78     0   150      2
[54744.930001]        f7c87f94 00000046 00000046 f7c87f78 f7c88000  
f7c88254 c0571f40 be53c9c0
[54744.930001]        0000137d 00000000 f7c87f94 c0138047 f7d81a20  
c057be60 c057be6c f7c87fcc
[54744.930001]        c0158f77 00000003 f7c88000 f7c88254 00000000  
00000001 f7c88000 c011a4d0
[54744.930001] Call Trace:
[54744.930001]  [<c0138047>] ? up+0x57/0x90
[54744.930001]  [<c0158f77>] gatekeeper_thread+0xa7/0x140
[54744.930001]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930001]  [<c0158ed0>] ? gatekeeper_thread+0x0/0x140
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] pdflush       S [0x00000001] f7c65f80     0   154      2
[54744.930001]        f7c65fa4 00000046 c04ee300 f7c65f80 f7c891c0  
f7c89414 f7c65f88 0d502af9
[54744.930001]        00000000 c011a2b5 c04ee300 00000200 f7c65fbc  
00000000 c017ee20 f7c65fcc
[54744.930001]        c017eedb 0d50119b f7c891c0 00000000 00000000  
c047f094 f7dbffbc fffb6cfe
[54744.930001] Call Trace:
[54744.930001]  [<c011a2b5>] ? set_user_nice+0x105/0x160
[54744.930001]  [<c017ee20>] ? pdflush+0x0/0x230
[54744.930001]  [<c017eedb>] pdflush+0xbb/0x230
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] pdflush       S [0x00000001] f7dbff94     0   155      2
[54744.930001]        f7dbffa4 00000046 00000000 f7dbff94 f7c89630  
f7c89884 00000000 00000000
[54744.930001]        00000000 00000000 00000025 033e496d f7dbffbc  
00000000 c017ee20 f7dbffcc
[54744.930001]        c017eedb 0d504449 f7c89630 00000000 00000000  
f7c65fbc c047f094 033ebe9d
[54744.930001] Call Trace:
[54744.930001]  [<c017ee20>] ? pdflush+0x0/0x230
[54744.930001]  [<c017eedb>] pdflush+0xbb/0x230
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] kswapd0       S [0x00000001] 00000000     0   156      2
[54744.930001]        f7dc1f34 00000046 00000000 00000000 f7c89aa0  
f7c89cf4 c0117a30 0d52fb0d
[54744.930001]        00000000 c0133b8b 00000001 00000200 00000000  
c047e580 c0182840 f7dc1fcc
[54744.930001]        c0182bdb f7c89ac8 f7c89ac8 f7dc1f70 c047f028  
f7dc1f70 00000000 f7c2c498
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c0182840>] ? kswapd+0x0/0x410
[54744.930001]  [<c0182bdb>] kswapd+0x39b/0x410
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c011996e>] ? hrtick_set+0xde/0x1a0
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c03bf76d>] ? schedule+0x2fd/0x570
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c011c4ec>] ? complete+0x6c/0x90
[54744.930001]  [<c0182840>] ? kswapd+0x0/0x410
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] aio/0         S [0x00000001] 00000000     0   199      2
[54744.930001]        f7d01fa4 00000046 f7d01f7c 00000000 f7f551c0  
f7f55414 c0117a30 0d7079c5
[54744.930001]        00000000 c0133b8b 00000001 00000200 f7c1afc0  
f7c1afc8 f7d01fac f7d01fcc
[54744.930001]        c01307f5 00000000 f7f551c0 c01338f0 f7c1afc8  
f7c1afc8 fffffffc f7c1afc0
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] unionfs_siod/ S [0x00000001] 00000000     0   212      2
[54744.930001]        f7c83fa4 00000046 f7c83f7c 00000000 f7c94d50  
f7c94fa4 c0117a30 0d7a27f0
[54744.930001]        00000000 c0133b8b 00000001 00000200 f7c1aa40  
f7c1aa48 f7c83fac f7c83fcc
[54744.930001]        c01307f5 00000000 f7c94d50 c01338f0 f7c1aa48  
f7c1aa48 fffffffc f7c1aa40
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] nfsiod        S [0x00000001] 00000000     0   213      2
[54744.930001]        f7c5dfa4 00000046 f7c5df7c 00000000 f7c951c0  
f7c95414 c0117a30 0d7b1c2d
[54744.930001]        00000000 c0133b8b 00000001 00000200 f7c1a340  
f7c1a348 f7c5dfac f7c5dfcc
[54744.930001]        c01307f5 00000000 f7c951c0 c01338f0 f7c1a348  
f7c1a348 fffffffc f7c1a340
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] kpsmoused     S [0x00000001] ffffffff     0   878      2
[54744.930001]        f7f03fa4 00000046 00000000 ffffffff f7f91aa0  
f7f91cf4 c0117a30 5625a9d4
[54744.930001]        00000000 c0133b8b 00000001 00000200 f7c4d4c0  
f7c4d4c8 f7f03fac f7f03fcc
[54744.930001]        c01307f5 00000000 f7f91aa0 c01338f0 f7c4d4c8  
f7c4d4c8 fffffffc f7c4d4c0
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930001]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930001]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930001]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930001]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930001]  [<c01335a2>] kthread+0x42/0x70
[54744.930001]  [<c0133560>] ? kthread+0x0/0x70
[54744.930001]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930001]  =======================
[54744.930001] rpciod/0      S [0x00000001] ffffffff     0   887      2
[54744.930001]        f7f09fa4 00000046 00000000 ffffffff f7f911c0  
f7f91414 c0117a30 5954fbff
[54744.930001]        00000000 c0133b8b 00000001 00000200 f7c4da00  
f7c4da08 f7f09fac f7f09fcc
[54744.930001]        c01307f5 00000000 f7f911c0 c01338f0 f7c4da08  
f7c4da08 fffffffc f7c4da00
[54744.930001] Call Trace:
[54744.930001]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930002]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930002]  [<c01335a2>] kthread+0x42/0x70
[54744.930002]  [<c0133560>] ? kthread+0x0/0x70
[54744.930002]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930002]  =======================
[54744.930002] kjournald     S [0x00000001] f7f0bf70     0   891      2
[54744.930002]        f7f0bf94 00000046 f7f5f440 f7f0bf70 f7f93540  
f7f93794 c0117a30 32c7c186
[54744.930002]        000031ca c0133b8b 00000001 00000200 f7f5f400  
00000000 f7f5f4c4 f7f0bfcc
[54744.930002]        c01f881e 00000000 00000005 f7f5f450 f7f5f440  
00000000 f7f93540 c01338f0
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930002]  [<c01f881e>] kjournald+0x1fe/0x240
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c01f8620>] ? kjournald+0x0/0x240
[54744.930002]  [<c01335a2>] kthread+0x42/0x70
[54744.930002]  [<c0133560>] ? kthread+0x0/0x70
[54744.930002]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930002]  =======================
[54744.930002] udevd         S [0x00000001] 00000200     0   993      1
[54744.930002]        f7fa5b30 00000082 c0133c22 00000200 f7ce5f10  
f7ce6164 f7fa5b18 d5a96edc
[54744.930002]        00001307 00000000 c0133c22 00000200 00000008  
00000080 00000000 f7fa5b60
[54744.930002]        c03bfff5 f7f830c0 f7fa5b4c c03c02f3 00000080  
00000041 f7fa5b60 c01c9c38
[54744.930002] Call Trace:
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c03c02f3>] ? mutex_lock+0x13/0x30
[54744.930002]  [<c01c9c38>] ? inotify_poll+0x48/0x60
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c013d146>] ? clockevents_program_event+0xb6/0x120
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c017c8a7>] ? get_page_from_freelist+0x227/0x530
[54744.930002]  [<c0115cc3>] ? kmap_atomic_prot+0x43/0xb0
[54744.930002]  [<c0296228>] ? number+0x288/0x2a0
[54744.930002]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[54744.930002]  [<c017e28b>] ? balance_dirty_pages_ratelimited_nr+0x4b/ 
0x290
[54744.930002]  [<c0192c10>] ? shmem_truncate_range+0x150/0x820
[54744.930002]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c017d783>] ? set_page_dirty+0x43/0xd0
[54744.930002]  [<c0296b21>] ? vsnprintf+0x371/0x6d0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[54744.930002]  [<c01a3dd9>] ? pipe_read+0x2e9/0x380
[54744.930002]  [<c019c76c>] ? do_sync_read+0xcc/0x110
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c019c6a0>] ? do_sync_read+0x0/0x110
[54744.930002]  [<c019d3fd>] ? sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] kjournald     S [0x00000001] f74b7f70     0  2019      2
[54744.930002]        f74b7f94 00000046 f7f5fc40 f74b7f70 f7f579b0  
f7f57c04 c0117a30 f7f579b0
[54744.930002]        f74b7f94 c0133b8b 00000001 00000200 f7f5fc00  
00000000 f7f5fcc4 f74b7fcc
[54744.930002]        c01f881e 00000000 00000005 f7f5fc50 f7f5fc40  
00000000 f7f579b0 c01338f0
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930002]  [<c01f881e>] kjournald+0x1fe/0x240
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c01f8620>] ? kjournald+0x0/0x240
[54744.930002]  [<c01335a2>] kthread+0x42/0x70
[54744.930002]  [<c0133560>] ? kthread+0x0/0x70
[54744.930002]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930002]  =======================
[54744.930002] dhclient3     S [0x00000001] c0128feb     0  2097      1
[54744.930002]        f7577b30 00000082 f7577b14 c0128feb f7f56380  
f7f565d4 c0117a30 f7577b40
[54744.930002]        f7577b30 c012917f 00000000 00000200 f7577b40  
0340ca72 00000000 f7577b60
[54744.930002]        c03bffc7 f7f7b600 f7577bd0 c056fde0 c056fde0  
0340ca72 c0128dd0 f7f56380
[54744.930002] Call Trace:
[54744.930002]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[54744.930002]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[54744.930002]  [<c0128dd0>] ? process_timeout+0x0/0x10
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a3f8>] ? try_to_wake_up+0x58/0x130
[54744.930002]  [<c0108679>] ? read_tsc+0x9/0x30
[54744.930002]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[54744.930002]  [<c0137576>] ? ktime_get_ts+0x46/0x50
[54744.930002]  [<c0137591>] ? ktime_get+0x11/0x30
[54744.930002]  [<c01172cb>] ? hrtick_start_fair+0xeb/0x140
[54744.930002]  [<c01176e4>] ? enqueue_task_fair+0x34/0x220
[54744.930002]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0296228>] ? number+0x288/0x2a0
[54744.930002]  [<c011687e>] ? __wake_up_common+0x3e/0x70
[54744.930002]  [<c011c442>] ? __wake_up_sync+0x72/0xb0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c033cd65>] ? sock_def_readable+0x45/0x80
[54744.930002]  [<c0398a11>] ? unix_dgram_sendmsg+0x461/0x530
[54744.930002]  [<c0352c25>] ? __qdisc_run+0x1c5/0x220
[54744.930002]  [<c0296b21>] ? vsnprintf+0x371/0x6d0
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[54744.930002]  [<c01aef04>] ? d_lookup+0x24/0x40
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[54744.930002]  [<c014a215>] ? call_rcu+0x55/0x70
[54744.930002]  [<c01874e2>] ? handle_mm_fault+0x352/0x5f0
[54744.930002]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[54744.930002]  [<c01aaac4>] sys_select+0xd4/0x190
[54744.930002]  [<c0297e73>] ? copy_to_user+0x43/0x60
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] syslogd       S [0x00000001] 00000200     0  2238      1
[54744.930002]        f755bb30 00000086 c0133c22 00000200 f7f91630  
f7f91884 f755bbd0 f755bb2c
[54744.930002]        f755bb20 c0117a30 00000000 f755bb2c 00000001  
00000001 00000000 f755bb60
[54744.930002]        c03bfff5 f755bbd0 f7ca3b00 00000001 0155bbe0  
c03eaee0 00000001 f7ca3b00
[54744.930002] Call Trace:
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c01e5aa1>] ? ext3_mark_iloc_dirty+0x181/0x350
[54744.930002]  [<c01e5fbe>] ? ext3_reserve_inode_write+0x5e/0x80
[54744.930002]  [<c01edcd7>] ? __ext3_journal_stop+0x27/0x50
[54744.930002]  [<c01e8df0>] ? ext3_dirty_inode+0x60/0x80
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[54744.930002]  [<c01f28a3>] ? journal_stop+0x123/0x240
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c017e28b>] ? balance_dirty_pages_ratelimited_nr+0x4b/ 
0x290
[54744.930002]  [<c01e81ea>] ? ext3_ordered_write_end+0xda/0x140
[54744.930002]  [<c0177e89>] ? generic_file_buffered_write+0x189/0x600
[54744.930002]  [<c0176888>] ? remove_suid+0x18/0x60
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[54744.930002]  [<c0178531>] ? __generic_file_aio_write_nolock 
+0x231/0x540
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[54744.930002]  [<c0129d8e>] ? __sigqueue_free+0x2e/0x40
[54744.930002]  [<c012a229>] ? __dequeue_signal+0xd9/0x180
[54744.930002]  [<c01020f4>] ? setup_sigcontext+0xf4/0x190
[54744.930002]  [<c0108679>] ? read_tsc+0x9/0x30
[54744.930002]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[54744.930002]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0137365>] ? hrtimer_start+0xc5/0x180
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c0123a65>] ? alarm_setitimer+0x35/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] klogd         R [0x00000000] running      0  2244      1
[54744.930002]        f7477ef4 00000086 f7477f6c c019c65c f7f90000  
f7f90254 c0117a30 7ad1992f
[54744.930002]        00001e0b 00000000 00000001 00000200 f7477f18  
00000000 00000fff f7477f38
[54744.930002]        c0120713 0804d6a0 00000000 00000000 00000000  
f7f90000 c01338f0 f7477f14
[54744.930002] Call Trace:
[54744.930002]  [<c019c65c>] ? do_sync_write+0xcc/0x110
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0120713>] do_syslog+0x403/0x490
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c01dad15>] kmsg_read+0x25/0x50
[54744.930002]  [<c01dacf0>] ? kmsg_read+0x0/0x50
[54744.930002]  [<c01d1d8c>] proc_reg_read+0x6c/0xc0
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c01d1d20>] ? proc_reg_read+0x0/0xc0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] inetd         S [0x00000001] f793eb60     0  2254      1
[54744.930002]        f7fefb30 00000082 c0117a30 f793eb60 f7f939b0  
f7f93c04 f7fefc00 64b26425
[54744.930002]        00000004 00000000 00000001 00000000 00000006  
00000020 00000000 f7fefb60
[54744.930002]        c03bfff5 f7fefbd0 f7fa0440 f7f20e00 f7fefb54  
c0368657 c03ea300 00000020
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0368657>] ? tcp_poll+0x17/0x160
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[54744.930002]  [<c017c956>] ? get_page_from_freelist+0x2d6/0x530
[54744.930002]  [<c017cc50>] ? __alloc_pages_internal+0xa0/0x420
[54744.930002]  [<c0198add>] ? kfree+0x9d/0xf0
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c02a9598>] ? extract_buf+0x88/0xf0
[54744.930002]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[54744.930002]  [<c01b1efd>] ? touch_atime+0x2d/0x110
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c01794da>] ? filemap_fault+0x12a/0x480
[54744.930002]  [<c0176e24>] ? unlock_page+0x24/0x30
[54744.930002]  [<c018588a>] ? __do_fault+0x17a/0x340
[54744.930002]  [<c018728a>] ? handle_mm_fault+0xfa/0x5f0
[54744.930002]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c0112497>] ? __ipipe_handle_exception+0xd7/0x1d0
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] nmbd          S [0x00000001] c0128feb     0  2262      1
[54744.930002]        f7fa7b30 00000086 f7fa7b14 c0128feb f7f927f0  
f7f92a44 c0117a30 f7fa7b40
[54744.930002]        f7fa7b30 c012917f 00000000 00000200 f7fa7b40  
033ed470 00000000 f7fa7b60
[54744.930002]        c03bffc7 f74ead00 f7fa7bd0 c056fc68 c056fc68  
033ed470 c0128dd0 f7f927f0
[54744.930002] Call Trace:
[54744.930002]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[54744.930002]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[54744.930002]  [<c0128dd0>] ? process_timeout+0x0/0x10
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c01338df>] ? wake_up_bit+0x1f/0x30
[54744.930002]  [<c01f2b92>] ? do_get_write_access+0x1d2/0x440
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01f41ff>] ? journal_dirty_metadata+0x7f/0x120
[54744.930002]  [<c01f21ff>] ? __ext3_journal_dirty_metadata+0x1f/0x50
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c019803b>] ? __slab_free+0x5b/0x2f0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01bfae7>] ? __find_get_block_slow+0x67/0x110
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c033eb9c>] ? __kfree_skb+0x3c/0x90
[54744.930002]  [<c033eb9c>] ? __kfree_skb+0x3c/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01bfeeb>] ? __find_get_block+0x8b/0x1c0
[54744.930002]  [<c01f2514>] ? __journal_file_buffer+0x74/0x160
[54744.930002]  [<c01f2b92>] ? do_get_write_access+0x1d2/0x440
[54744.930002]  [<c01f2514>] ? __journal_file_buffer+0x74/0x160
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01f41ff>] ? journal_dirty_metadata+0x7f/0x120
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[54744.930002]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[54744.930002]  [<c01abeec>] ? __posix_lock_file+0x6c/0x550
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[54744.930002]  [<c01abbd1>] ? locks_free_lock+0x31/0x50
[54744.930002]  [<c01aaac4>] sys_select+0xd4/0x190
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] smbd          S [0x00000001] f74e1b0c     0  2264      1
[54744.930002]        f74e1b30 00000082 f794d0e0 f74e1b0c f7f92c60  
f7f92eb4 c0117a30 f74e1c1c
[54744.930002]        f74e1b2c c0133c22 00000200 00000002 00000015  
00100000 00000000 f74e1b60
[54744.930002]        c03bfff5 f74b0180 f74e1bd0 f74b0180 f7cf5e00  
f74e1b60 c01a2b5f 00000000
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c01a2b5f>] ? pipe_poll+0x2f/0xb0
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02a9598>] ? extract_buf+0x88/0xf0
[54744.930002]  [<c02a9598>] ? extract_buf+0x88/0xf0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0108679>] ? read_tsc+0x9/0x30
[54744.930002]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c01794da>] ? filemap_fault+0x12a/0x480
[54744.930002]  [<c013d146>] ? clockevents_program_event+0xb6/0x120
[54744.930002]  [<c0176e24>] ? unlock_page+0x24/0x30
[54744.930002]  [<c018588a>] ? __do_fault+0x17a/0x340
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c018728a>] ? handle_mm_fault+0xfa/0x5f0
[54744.930002]  [<c010f1c0>] ? smp_apic_timer_interrupt+0x0/0x70
[54744.930002]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c0112497>] ? __ipipe_handle_exception+0xd7/0x1d0
[54744.930002]  [<c0111f41>] ? __ipipe_handle_irq+0x81/0x1a0
[54744.930002]  [<c01a8b5f>] ? do_fcntl+0x28f/0x300
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] smbd          S [0x00000001] f757dfb0     0  2274   2264
[54744.930002]        f757dfa8 00000086 c014c146 f757dfb0 f7c967f0  
f7c96a44 f74b0a80 1e46bbf3
[54744.930002]        00000005 00000000 0000000e f757dfb0 00000000  
00000001 00000003 f757dfb0
[54744.930002]        c012bd13 f757c000 c01030d5 00000000 00000019  
0a201fd8 00000001 00000003
[54744.930002] Call Trace:
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c012bd13>] sys_pause+0x13/0x20
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] sshd          S [0x00000001] f793eb60     0  2279      1
[54744.930002]        f74dfb30 00000082 c0117a30 f793eb60 f7c97540  
f7c97794 f74dfc00 f0367a07
[54744.930002]        00001e07 00000000 00000001 00000000 00000005  
00000010 00000000 f74dfb60
[54744.930002]        c03bfff5 f74dfbd0 f7fa1540 f75ab580 f74dfb54  
c0368657 c03ea300 00000010
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0368657>] ? tcp_poll+0x17/0x160
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c01f21ff>] ? __ext3_journal_dirty_metadata+0x1f/0x50
[54744.930002]  [<c01e5aa1>] ? ext3_mark_iloc_dirty+0x181/0x350
[54744.930002]  [<c01e5fbe>] ? ext3_reserve_inode_write+0x5e/0x80
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0108679>] ? read_tsc+0x9/0x30
[54744.930002]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[54744.930002]  [<c0296228>] ? number+0x288/0x2a0
[54744.930002]  [<c0154b11>] ? xnarch_next_htick_shot+0x51/0x60
[54744.930002]  [<c013d146>] ? clockevents_program_event+0xb6/0x120
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0296b21>] ? vsnprintf+0x371/0x6d0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[54744.930002]  [<c017c956>] ? get_page_from_freelist+0x2d6/0x530
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c017c8a7>] ? get_page_from_freelist+0x227/0x530
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0137365>] ? hrtimer_start+0xc5/0x180
[54744.930002]  [<c011994e>] ? hrtick_set+0xbe/0x1a0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01969a5>] ? add_partial+0x45/0x80
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c019803b>] ? __slab_free+0x5b/0x2f0
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c0198add>] ? kfree+0x9d/0xf0
[54744.930002]  [<c01a313c>] ? __free_pipe_info+0x3c/0x50
[54744.930002]  [<c01a313c>] ? __free_pipe_info+0x3c/0x50
[54744.930002]  [<c01a313c>] ? __free_pipe_info+0x3c/0x50
[54744.930002]  [<c01a3161>] ? free_pipe_info+0x11/0x20
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c019dc58>] ? fput+0x18/0x20
[54744.930002]  [<c019abe7>] ? filp_close+0x47/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] proftpd       S [0x00000001] c0128feb     0  2366      1
[54744.930002]        f7487b30 00200086 f7487b14 c0128feb f7c8a380  
f7c8a5d4 c0117a30 f7487b40
[54744.930002]        f7487b30 c012917f 00000000 00000200 f7487b40  
033f27c1 00000000 f7487b60
[54744.930002]        c03bffc7 f7487bd0 f7fa1980 c056ffa8 f766948c  
033f27c1 c0128dd0 f7c8a380
[54744.930002] Call Trace:
[54744.930002]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[54744.930002]  [<c03bffc7>] schedule_timeout+0x47/0xc0
[54744.930002]  [<c0128dd0>] ? process_timeout+0x0/0x10
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[54744.930002]  [<c012fd8b>] ? insert_work+0x4b/0x70
[54744.930002]  [<c0108679>] ? read_tsc+0x9/0x30
[54744.930002]  [<c0139cfb>] ? getnstimeofday+0x3b/0xe0
[54744.930002]  [<c0137576>] ? ktime_get_ts+0x46/0x50
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01bfae7>] ? __find_get_block_slow+0x67/0x110
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01bfeeb>] ? __find_get_block+0x8b/0x1c0
[54744.930002]  [<c01f2514>] ? __journal_file_buffer+0x74/0x160
[54744.930002]  [<c01f2b92>] ? do_get_write_access+0x1d2/0x440
[54744.930002]  [<c01f2514>] ? __journal_file_buffer+0x74/0x160
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01f41ff>] ? journal_dirty_metadata+0x7f/0x120
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[54744.930002]  [<c01f28a3>] ? journal_stop+0x123/0x240
[54744.930002]  [<c01f28a3>] ? journal_stop+0x123/0x240
[54744.930002]  [<c01edcd7>] ? __ext3_journal_stop+0x27/0x50
[54744.930002]  [<c01e8df0>] ? ext3_dirty_inode+0x60/0x80
[54744.930002]  [<c01bc19c>] ? __mark_inode_dirty+0x2c/0x1a0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01aee28>] ? __d_lookup+0xc8/0x180
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[54744.930002]  [<c01b4476>] ? mntput_no_expire+0x16/0x100
[54744.930002]  [<c01a4530>] ? path_put+0x20/0x30
[54744.930002]  [<c01a691b>] ? path_walk+0x4b/0x90
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c01a7646>] ? __user_walk_fd+0x46/0x60
[54744.930002]  [<c01a7646>] ? __user_walk_fd+0x46/0x60
[54744.930002]  [<c01a019e>] ? vfs_lstat_fd+0x1e/0x50
[54744.930002]  [<c01369e5>] ? enqueue_hrtimer+0x75/0xf0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0137365>] ? hrtimer_start+0xc5/0x180
[54744.930002]  [<c01aaac4>] sys_select+0xd4/0x190
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] cron          S [0x00000001] 00000001     0  2378      1
[54744.930002]        f74c7f3c 00000082 c0117a30 00000001 f7c8b0d0  
f7c8b324 f2672ea2 000031c5
[54744.930002]        c0478454 00000000 00000200 00000000 f74c7f5c  
f7c8b0d0 f74c7f5c f74c7f50
[54744.930002]        c03c0725 00000001 00000000 00000001 f74c7f98  
c0137462 bfea85e4 f741de41
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03c0725>] do_nanosleep+0x65/0x90
[54744.930002]  [<c0137462>] hrtimer_nanosleep+0x42/0xb0
[54744.930002]  [<c0136e50>] ? hrtimer_wakeup+0x0/0x20
[54744.930002]  [<c0137528>] sys_nanosleep+0x58/0x60
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] xdm           S [0x00000001] 00000200     0  2400      1
[54744.930002]        f7497b30 00000086 c0133c22 00000200 f7c88d50  
f7c88fa4 f7497bd0 f7497b2c
[54744.930002]        c01aa97d f798b918 f74d4580 f756d400 00000005  
00000010 00000000 f7497b60
[54744.930002]        c03bfff5 f74d4580 f7497b54 c037c7b2 00497b54  
c03ea360 00000010 f74d4580
[54744.930002] Call Trace:
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c01aa97d>] ? __pollwait+0x6d/0xe0
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c037c7b2>] ? udp_poll+0x12/0x100
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c0115c48>] ? kunmap_atomic+0x38/0x70
[54744.930002]  [<c017c956>] ? get_page_from_freelist+0x2d6/0x530
[54744.930002]  [<c017cc50>] ? __alloc_pages_internal+0xa0/0x420
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01bfeeb>] ? __find_get_block+0x8b/0x1c0
[54744.930002]  [<c01338df>] ? wake_up_bit+0x1f/0x30
[54744.930002]  [<c01f2b92>] ? do_get_write_access+0x1d2/0x440
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01f41ff>] ? journal_dirty_metadata+0x7f/0x120
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c011c39c>] ? __wake_up+0x6c/0xa0
[54744.930002]  [<c01f28a3>] ? journal_stop+0x123/0x240
[54744.930002]  [<c01f28a3>] ? journal_stop+0x123/0x240
[54744.930002]  [<c01edcd7>] ? __ext3_journal_stop+0x27/0x50
[54744.930002]  [<c01e8df0>] ? ext3_dirty_inode+0x60/0x80
[54744.930002]  [<c010311c>] ? restore_nocheck_notrace+0x0/0xe
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01b5a31>] ? mnt_drop_write+0x81/0x170
[54744.930002]  [<c01b1efd>] ? touch_atime+0x2d/0x110
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c017c45b>] ? free_hot_cold_page+0x15b/0x1e0
[54744.930002]  [<c017c52a>] ? free_hot_page+0xa/0x10
[54744.930002]  [<c017fff4>] ? put_page+0x34/0x110
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0185e91>] ? unmap_vmas+0x291/0x480
[54744.930002]  [<c0186e46>] ? free_pgtables+0x86/0xb0
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c01893d7>] ? remove_vma+0x47/0x60
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c012a113>] ? sys_rt_sigaction+0x63/0xa0
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] login         S [0x00000001] f7553f2c     0  2412      1
[54744.930002]        f7553f20 00000082 f74aa140 f7553f2c f7f548e0  
f7f54b34 f7553f08 c0117a30
[54744.930002]        f7553f70 f7553f20 c0133c22 00000200 ffffffea  
f7f548d8 00000004 f7553f90
[54744.930002]        c01221e5 f7f91f10 f7553f78 c0113832 00000001  
00000000 f7f54ac8 00000000
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c01221e5>] do_wait+0x5d5/0xbd0
[54744.930002]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c012284f>] sys_wait4+0x6f/0xc0
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] getty         S [0x00000001] f7493800     0  2414      1
[54744.930002]        f760feb0 00000086 f7458dc0 f7493800 f7f55aa0  
f7f55cf4 c17ec760 2dd8d213
[54744.930002]        00000006 00000000 f760fea8 c0117a30 00000001  
f7479c00 00000000 f760fee0
[54744.930002]        c03bfff5 00000000 f760fec8 c0117a30 f760ff20  
f760fee0 c0133c22 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c02b193b>] read_chan+0x1bb/0x680
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02ae0a7>] tty_read+0x77/0xb0
[54744.930002]  [<c02b1780>] ? read_chan+0x0/0x680
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c02ae030>] ? tty_read+0x0/0xb0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] getty         S [0x00000001] f7493c00     0  2416      1
[54744.930002]        f7451eb0 00000082 f75508f0 f7493c00 f7f56c60  
f7f56eb4 c17ec760 2dd8d215
[54744.930002]        00000006 00000000 f7451ea8 c0117a30 00000001  
f7479400 00000000 f7451ee0
[54744.930002]        c03bfff5 00000000 f7451ec8 c0117a30 f7451f20  
f7451ee0 c0133c22 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c02b193b>] read_chan+0x1bb/0x680
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02ae0a7>] tty_read+0x77/0xb0
[54744.930002]  [<c02b1780>] ? read_chan+0x0/0x680
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c02ae030>] ? tty_read+0x0/0xb0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] getty         S [0x00000001] f7493e00     0  2418      1
[54744.930002]        f7539eb0 00000082 f7550630 f7493e00 f7f55630  
f7f55884 c17ec760 2dd8d217
[54744.930002]        00000006 00000000 f7539ea8 c0117a30 00000001  
f7479800 00000000 f7539ee0
[54744.930002]        c03bfff5 00000000 f7539ec8 c0117a30 f7539f20  
f7539ee0 c0133c22 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c02b193b>] read_chan+0x1bb/0x680
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02ae0a7>] tty_read+0x77/0xb0
[54744.930002]  [<c02b1780>] ? read_chan+0x0/0x680
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c02ae030>] ? tty_read+0x0/0xb0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] getty         S [0x00000001] f7fb5280     0  2419      1
[54744.930002]        f74e5eb0 00000086 f7d152c0 f7fb5280 f7f54000  
f7f54254 c17ec760 2dd8d219
[54744.930002]        00000006 00000000 f74e5ea8 c0117a30 00000001  
f74c5400 00000000 f74e5ee0
[54744.930002]        c03bfff5 00000000 f74e5ec8 c0117a30 f74e5f20  
f74e5ee0 c0133c22 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c02b193b>] read_chan+0x1bb/0x680
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02ae0a7>] tty_read+0x77/0xb0
[54744.930002]  [<c02b1780>] ? read_chan+0x0/0x680
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c02ae030>] ? tty_read+0x0/0xb0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] getty         S [0x00000001] f7fb5d00     0  2420      1
[54744.930002]        f749feb0 00000082 f7d15420 f7fb5d00 f7f57540  
f7f57794 c17ec760 2dd8d21a
[54744.930002]        00000006 f749fea4 f749fea8 c0117a30 00000001  
f75d2800 00000000 f749fee0
[54744.930002]        c03bfff5 00000000 f749fec8 c0117a30 f749ff20  
f749fee0 c0133c22 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c02b193b>] read_chan+0x1bb/0x680
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02ae0a7>] tty_read+0x77/0xb0
[54744.930002]  [<c02b1780>] ? read_chan+0x0/0x680
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c02ae030>] ? tty_read+0x0/0xb0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] bash          S [0x00000001] c00bca60     0  2438   2412
[54744.930002]        f7f6deb0 00000082 f7f6de90 c00bca60 f7f91f10  
f7f92164 0000000e f7f6deb4
[54744.930002]        c02a68a6 0000000e f7f6dea8 c0117a30 00000001  
f7479000 00000000 f7f6dee0
[54744.930002]        c03bfff5 f7f6decc f7f6dec8 c0117a30 f7f6df20  
f7f6dee0 c0133c22 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c02a68a6>] ? vgacon_set_cursor_size+0xc6/0x170
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c02b193b>] read_chan+0x1bb/0x680
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02ae0a7>] tty_read+0x77/0xb0
[54744.930002]  [<c02b1780>] ? read_chan+0x0/0x680
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c02ae030>] ? tty_read+0x0/0xb0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] sshd          S [0x00000001] 00000001     0 17390   2279
[54744.930002]        f755db30 00000082 c0117a30 00000001 f7c89f10  
f7c8a164 00000000 00000000
[54744.930002]        c0484da0 00000000 f755db28 c0117a30 00000008  
00000080 00000000 f755db60
[54744.930002]        c03bfff5 f763cc00 c02b0380 f755db60 c02abdd0  
f755dbd0 f763cc10 00000104
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c02b0380>] ? normal_poll+0x0/0x150
[54744.930002]  [<c02abdd0>] ? tty_poll+0x70/0x80
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<f88b6bc1>] ? ieee80211_skb_resize+0x61/0xc0 [mac80211]
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c0362516>] ? ip_finish_output+0x116/0x2b0
[54744.930002]  [<c0362f24>] ? ip_output+0x64/0x80
[54744.930002]  [<c0360e38>] ? ip_local_out+0x18/0x20
[54744.930002]  [<c03630fd>] ? ip_queue_xmit+0x1bd/0x320
[54744.930002]  [<c0137576>] ? ktime_get_ts+0x46/0x50
[54744.930002]  [<c0137591>] ? ktime_get+0x11/0x30
[54744.930002]  [<c01172cb>] ? hrtick_start_fair+0xeb/0x140
[54744.930002]  [<c0378230>] ? tcp_v4_send_check+0x40/0xd0
[54744.930002]  [<c037317e>] ? tcp_transmit_skb+0x3ae/0x770
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[54744.930002]  [<c037290f>] ? tcp_init_tso_segs+0x3f/0x60
[54744.930002]  [<c0374a5a>] ? __tcp_push_pending_frames+0x2ea/0x750
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[54744.930002]  [<c033afc9>] ? release_sock+0xb9/0xd0
[54744.930002]  [<c0369aca>] ? tcp_sendmsg+0x75a/0xa40
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c03383db>] ? sock_aio_write+0xeb/0x110
[54744.930002]  [<c019c65c>] ? do_sync_write+0xcc/0x110
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c0124226>] ? current_fs_time+0x16/0x20
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c019d46d>] ? sys_write+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] bash          S [0x00000001] c0137365     0 17392  17390
[54744.930002]        f7499eb0 00000082 f7499ea8 c0137365 f7c8ac60  
f7c8aeb4 c0478454 38c98c11
[54744.930002]        00000bea 00000000 c04ee300 3965f5d5 00000001  
f763c400 00000000 f7499ee0
[54744.930002]        c03bfff5 ffffffff f7499ec8 c0117a30 f7499f20  
f7499ee0 c0133c22 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c0137365>] ? hrtimer_start+0xc5/0x180
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c02b193b>] read_chan+0x1bb/0x680
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c02ae0a7>] tty_read+0x77/0xb0
[54744.930002]  [<c02b1780>] ? read_chan+0x0/0x680
[54744.930002]  [<c019cfc4>] vfs_read+0x94/0x130
[54744.930002]  [<c02ae030>] ? tty_read+0x0/0xb0
[54744.930002]  [<c019d3fd>] sys_read+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] remoteHost    S [0x00000001] 000084d0     0 17405  17392
[54744.930002]        f76a1e34 00000086 f7f54470 000084d0 f7f54470  
f7f546c4 c047efe0 00000001
[54744.930002]        00000044 000280d0 000284d0 c047efe0 f7fa3b80  
00000000 7fffffff f76a1e64
[54744.930002]        c03bfff5 c0117a30 f7fa3b80 f76a1e4c c0124ab5  
f76a1e64 c033afc9 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[54744.930002]  [<c033afc9>] ? release_sock+0xb9/0xd0
[54744.930002]  [<c0366c69>] inet_csk_accept+0x119/0x250
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c038501f>] inet_accept+0x1f/0xb0
[54744.930002]  [<c033a093>] sys_accept+0xd3/0x1c0
[54744.930002]  [<c01874e2>] ? handle_mm_fault+0x352/0x5f0
[54744.930002]  [<c0113832>] ? do_page_fault+0x2c2/0x670
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfa9f>] ? preempt_schedule+0x4f/0x60
[54744.930002]  [<c011cd5b>] ? wake_up_new_task+0xbb/0xc0
[54744.930002]  [<c033a241>] sys_socketcall+0xc1/0x260
[54744.930002]  [<c019abe7>] ? filp_close+0x47/0x70
[54744.930002]  [<c019c1db>] ? sys_close+0x7b/0xf0
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] remoteHost    S [0x00000001] c76f2ef7     0 17406  17405
[54744.930002]        f769bf08 00000082 f7627780 c76f2ef7 f7f567f0  
f7f56a44 f769bef0 c7098ee0
[54744.930002]        0000137d 00000000 c0133c22 00000200 ffffffea  
f7f567e8 00000004 f769bf78
[54744.930002]        c01221e5 f7ce4470 f7627780 f7634bf8 f7f56a48  
00000000 f7f569d8 f764e7c0
[54744.930002] Call Trace:
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c01221e5>] do_wait+0x5d5/0xbd0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfa9f>] ? preempt_schedule+0x4f/0x60
[54744.930002]  [<c011cd5b>] ? wake_up_new_task+0xbb/0xc0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c012284f>] sys_wait4+0x6f/0xc0
[54744.930002]  [<c01228c5>] sys_waitpid+0x25/0x30
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] remoteHost    S [0x00000001] f7fa2640     0 17407  17405
[54744.930002]        f7643ce0 00000082 f7650840 f7fa2640 f7f55f10  
f7f56164 00000000 f7fa2640
[54744.930002]        f7643cf4 c037215d c014c146 bff74101 f7643d70  
f7fa2640 f7fa26a0 f7643d10
[54744.930002]        c03bfff5 c0117a30 f7643d70 f7643cf8 c0124ab5  
f7643d10 c033afc9 00000200
[54744.930002] Call Trace:
[54744.930002]  [<c037215d>] ? tcp_rcv_established+0x40d/0x6c0
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[54744.930002]  [<c033afc9>] ? release_sock+0xb9/0xd0
[54744.930002]  [<c033ba69>] sk_wait_data+0x79/0xb0
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c036a12e>] tcp_recvmsg+0x37e/0x820
[54744.930002]  [<c033aa23>] sock_common_recvmsg+0x43/0x60
[54744.930002]  [<c0338fb6>] sock_recvmsg+0xc6/0xf0
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[54744.930002]  [<c019950a>] ? kmem_cache_free+0x8a/0xd0
[54744.930002]  [<c029101d>] ? _atomic_dec_and_lock+0x2d/0x50
[54744.930002]  [<c0129d8e>] ? __sigqueue_free+0x2e/0x40
[54744.930002]  [<c012a229>] ? __dequeue_signal+0xd9/0x180
[54744.930002]  [<c0338930>] ? sockfd_lookup_light+0x30/0x60
[54744.930002]  [<c0339e0d>] sys_recvfrom+0x7d/0xd0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0185204>] ? do_wp_page+0x1e4/0x490
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c01874e2>] ? handle_mm_fault+0x352/0x5f0
[54744.930002]  [<c0339e92>] sys_recv+0x32/0x40
[54744.930002]  [<c033a2d8>] sys_socketcall+0x158/0x260
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] rt61pci       S [0x00000001] c0117a30     0 27579      2
[54744.930002]        f76e5fa4 00000046 f76e5f7c c0117a30 f7ce70d0  
f7ce7324 c0117a30 f7ce70d0
[54744.930002]        f76e5fa4 c0133b8b 00000001 00000200 f764ef80  
f764ef88 f76e5fac f76e5fcc
[54744.930002]        c01307f5 00000000 f7ce70d0 c01338f0 f764ef88  
f764ef88 fffffffc f764ef80
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133b8b>] ? prepare_to_wait+0x7b/0xb0
[54744.930002]  [<c01307f5>] worker_thread+0x95/0xb0
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c0130760>] ? worker_thread+0x0/0xb0
[54744.930002]  [<c01335a2>] kthread+0x42/0x70
[54744.930002]  [<c0133560>] ? kthread+0x0/0x70
[54744.930002]  [<c0103d43>] kernel_thread_helper+0x7/0x14
[54744.930002]  =======================
[54744.930002] actuatorTask  ? [0x00000040] 00000001     0 28213      1
[54744.930002]        f760df54 00000046 c0198b26 00000001 f7ce48e0  
f7ce4b34 f765a540 c6b973df
[54744.930002]        0000137d 00000000 f7ce48d8 f7ce48e0 00000010  
f7ce48e0 f760ddd8 f760df98
[54744.930002]        c0122e14 c04ee300 00000000 f7ce51c0 00000001  
f760df84 f7ce4a84 f7ce4ac8
[54744.930002] Call Trace:
[54744.930002]  [<c0198b26>] ? kfree+0xe6/0xf0
[54744.930002]  [<c0122e14>] do_exit+0x494/0x710
[54744.930002]  [<c01230be>] do_group_exit+0x2e/0xb0
[54744.930002]  [<c012314f>] sys_exit_group+0xf/0x20
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] Actuator Aper S [0x00000201] f7f01f04     0 28214      1
[54744.930002]        f7f01f28 00000086 c057be64 f7f01f04 f7ce51c0  
f7ce5414 f7f01f28 ce2e19e5
[54744.930002]        0000137c 00000000 00000001 00000200 f7d80620  
f7ce51c0 c01586e0 f7f01f4c
[54744.930002]        c0158576 f7f01f6c c015891a 00000000 f7f01ebc  
00000006 00000000 c01586e0
[54744.930002] Call Trace:
[54744.930002]  [<c01586e0>] ? losyscall_event+0x0/0x180
[54744.930002]  [<c0158576>] xnshadow_harden+0x86/0x1f0
[54744.930002]  [<c015891a>] ? hisyscall_event+0xba/0x290
[54744.930002]  [<c01586e0>] ? losyscall_event+0x0/0x180
[54744.930002]  [<c015876f>] losyscall_event+0x8f/0x180
[54744.930002]  [<c01586e0>] ? losyscall_event+0x0/0x180
[54744.930002]  [<c014c89c>] __ipipe_dispatch_event+0x9c/0x170
[54744.930002]  [<c01122f2>] __ipipe_syscall_root+0x42/0x110
[54744.930002]  [<c01030ad>] system_call+0x29/0x4a
[54744.930002]  =======================
[54744.930002] sh            S [0x00000001] 33a349f2     0 28229  17406
[54744.930002]        f7665f08 00000082 f74bba80 33a349f2 f7ce4470  
f7ce46c4 f7665ef0 c0117a30
[54744.930002]        f7665f58 f7665f08 c0133c22 00000200 ffffffea  
f7ce4468 00000004 f7665f78
[54744.930002]        c01221e5 f7c2df10 f74bba80 f7506bfc f7ce46c8  
00000000 f7ce4658 00000000
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0133c22>] ? add_wait_queue+0x62/0xa0
[54744.930002]  [<c01221e5>] do_wait+0x5d5/0xbd0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c012284f>] sys_wait4+0x6f/0xc0
[54744.930002]  [<c01228c5>] sys_waitpid+0x25/0x30
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] sshd          S [0x00000001] 00000001     0  9641   2279
[54744.930002]        f7523b30 00000086 c0117a30 00000001 f7c2f540  
f7c2f794 00000000 42bf7612
[54744.930002]        000031ca 00000000 f7523b28 c0117a30 00000008  
00000080 00000000 f7523b60
[54744.930002]        c03bfff5 f766ec00 c02b0380 f7523b60 c02abdd0  
f7523bd0 f766ec10 00000104
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03bfff5>] schedule_timeout+0x75/0xc0
[54744.930002]  [<c02b0380>] ? normal_poll+0x0/0x150
[54744.930002]  [<c02abdd0>] ? tty_poll+0x70/0x80
[54744.930002]  [<c01aa2b4>] do_select+0x494/0x590
[54744.930002]  [<c01aa910>] ? __pollwait+0x0/0xe0
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c011a4d0>] ? default_wake_function+0x0/0x10
[54744.930002]  [<c0362516>] ? ip_finish_output+0x116/0x2b0
[54744.930002]  [<c0362f24>] ? ip_output+0x64/0x80
[54744.930002]  [<c0360e38>] ? ip_local_out+0x18/0x20
[54744.930002]  [<c03630fd>] ? ip_queue_xmit+0x1bd/0x320
[54744.930002]  [<c0378230>] ? tcp_v4_send_check+0x40/0xd0
[54744.930002]  [<c0117ac0>] ? add_preempt_count+0x10/0x80
[54744.930002]  [<c0128feb>] ? lock_timer_base+0x4b/0xa0
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c012917f>] ? __mod_timer+0xaf/0xf0
[54744.930002]  [<c0129234>] ? mod_timer+0x14/0x30
[54744.930002]  [<c033b10f>] ? sk_reset_timer+0xf/0x20
[54744.930002]  [<c0372c9b>] ? tcp_event_new_data_sent+0x7b/0xa0
[54744.930002]  [<c0374a5a>] ? __tcp_push_pending_frames+0x2ea/0x750
[54744.930002]  [<c014c146>] ? __ipipe_restore_root+0x16/0x20
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c0124ab5>] ? local_bh_enable+0x35/0x90
[54744.930002]  [<c033afc9>] ? release_sock+0xb9/0xd0
[54744.930002]  [<c0369aca>] ? tcp_sendmsg+0x75a/0xa40
[54744.930002]  [<c01aa59a>] core_sys_select+0x1ea/0x320
[54744.930002]  [<c03383db>] ? sock_aio_write+0xeb/0x110
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c019c65c>] ? do_sync_write+0xcc/0x110
[54744.930002]  [<c011c31a>] ? hrtick_resched+0x5a/0x70
[54744.930002]  [<c01338f0>] ? autoremove_wake_function+0x0/0x50
[54744.930002]  [<c0124226>] ? current_fs_time+0x16/0x20
[54744.930002]  [<c01aaa2f>] sys_select+0x3f/0x190
[54744.930002]  [<c019d46d>] ? sys_write+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] bash          R [0x00000000] running      0  9645   9641
[54744.930002]        f078bd80 f078bd78 f078bd78 c0117a30 00000031  
f078bda8 c02be570 00000018
[54744.930002]        00000000 ffffffff c05276b7 001b2340 c00baae2  
00000009 c0485fa0 000385db
[54744.930002]        00000016 f078bdbc f078bdb8 c0117a30 c0476a58  
f078bdcc c0138075 f078bdcc
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c011fcf1>] ? release_console_sem+0x211/0x250
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c012027a>] ? vprintk+0x34a/0x3e0
[54744.930002]  [<c0117a00>] ? hrtick_clear+0x0/0x20
[54744.930002]  [<c012027a>] ? vprintk+0x34a/0x3e0
[54744.930002]  [<c0120a2b>] ? printk+0x13b/0x150
[54744.930002]  [<c010f6ce>] ? touch_nmi_watchdog+0x1e/0x20
[54744.930002]  [<c0104085>] ? print_trace_address+0x45/0x50
[54744.930002]  [<c01313db>] ? __kernel_text_address+0x1b/0x30
[54744.930002]  [<c0104119>] ? dump_trace+0x89/0x120
[54744.930002]  [<c01041d7>] show_trace_log_lvl+0x27/0x50
[54744.930002]  [<c01042b0>] show_stack_log_lvl+0xb0/0xe0
[54744.930002]  [<c0104b07>] show_stack+0x37/0x50
[54744.930002]  [<c0117963>] sched_show_task+0x93/0xd0
[54744.930002]  [<c011c156>] show_state_filter+0x66/0xb0
[54744.930002]  [<c02be96a>] sysrq_handle_showstate+0xa/0x10
[54744.930002]  [<c02bec4d>] __handle_sysrq+0x9d/0x180
[54744.930002]  [<c01d8d80>] ? write_sysrq_trigger+0x0/0x40
[54744.930002]  [<c01d8daf>] write_sysrq_trigger+0x2f/0x40
[54744.930002]  [<c01d1e4c>] proc_reg_write+0x6c/0xc0
[54744.930002]  [<c019ce99>] vfs_write+0x99/0x130
[54744.930002]  [<c01d1de0>] ? proc_reg_write+0x0/0xc0
[54744.930002]  [<c019d46d>] sys_write+0x3d/0x70
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] sleep         S [0x00000001] 00000001     0 31350  28229
[54744.930002]        f76bbf3c 00000082 c0117a30 00000001 f7c2df10  
f7c2e164 3310b0a6 3310ae2c
[54744.930002]        000031ca 00000000 00000200 00000000 f76bbf5c  
f7c2df10 f76bbf5c f76bbf50
[54744.930002]        c03c0725 00000001 00000000 00000001 f76bbf98  
c0137462 00000000 f741de41
[54744.930002] Call Trace:
[54744.930002]  [<c0117a30>] ? sub_preempt_count+0x10/0x90
[54744.930002]  [<c03c0725>] do_nanosleep+0x65/0x90
[54744.930002]  [<c0137462>] hrtimer_nanosleep+0x42/0xb0
[54744.930002]  [<c0136e50>] ? hrtimer_wakeup+0x0/0x20
[54744.930002]  [<c0137528>] sys_nanosleep+0x58/0x60
[54744.930002]  [<c01030d5>] syscall_call+0x7/0xb
[54744.930002]  =======================
[54744.930002] Sched Debug Version: v0.07, 2.6.26.5 #5
[54744.930002] now at 54745022.308533 msecs
[54744.930002]   .sysctl_sched_latency                    : 20.000000
[54744.930002]   .sysctl_sched_min_granularity            : 4.000000
[54744.930002]   .sysctl_sched_wakeup_granularity         : 10.000000
[54744.930002]   .sysctl_sched_child_runs_first           : 0.000001
[54744.930002]   .sysctl_sched_features                   : 895
[54744.930002]
[54744.930002] cpu#0, 2000.117 MHz
[54744.930002]   .nr_running                    : 2
[54744.930002]   .load                          : 2048
[54744.930002]   .nr_switches                   : 23030001
[54744.930002]   .nr_load_updates               : 910582
[54744.930002]   .nr_uninterruptible            : 0
[54744.930002]   .jiffies                       : 54444929
[54744.930002]   .next_balance                  : 0.000000
[54744.930002]   .curr->pid                     : 9645
[54744.930002]   .clock                         : 54744929.094552
[54744.930002]   .cpu_load[0]                   : 1024
[54744.930002]   .cpu_load[1]                   : 858
[54744.930002]   .cpu_load[2]                   : 1501
[54744.930002]   .cpu_load[3]                   : 1603
[54744.930002]   .cpu_load[4]                   : 1360
[54744.930002]
[54744.930002] cfs_rq[0]:
[54744.930002]   .exec_clock                    : 4399658.298055
[54744.930002]   .MIN_vruntime                  : 4095824.863250
[54744.930002]   .min_vruntime                  : 4095844.863250
[54744.930002]   .max_vruntime                  : 4095824.863250
[54744.930002]   .spread                        : 0.000000
[54744.930002]   .spread0                       : 0.000000
[54744.930002]   .nr_running                    : 2
[54744.930002]   .load                          : 2048
[54744.930002]   .bkl_count                     : 116440
[54744.930002]   .nr_spread_over                : 35924
[54744.930002]
[54744.930002] runnable tasks:
[54744.930002]             task   PID         tree-key  switches   
prio     exec-runtime         sum-exec        sum-sleep
[54744.930002]  
----------------------------------------------------------------------------------------------------------
[54744.930002]            klogd  2244   4095824.863250      1897    
120   4095824.863250       113.055790  54725912.854089
[54744.930002] R           bash  9645   4095980.959130       121    
120   4095980.959130       375.568117  21724808.541901
[54744.930002]

-- dump ends here --

On Dec 23, 2008, at 12:53 PM, Gilles Chanteperdrix wrote:

> Alphan Ulusoy wrote:
>> I've done the changes you suggested ( addition to sched.c file and
>> fram pointers) and left the tasks running last night. Below you can
>> find the dump. The hex numbers inside the brackets ( [0x00000001] )
>> are newly added. And as you said in your previous post "Actuator
>> Aperiodic Task" is the real-time task created and started in the
>> process called "actuatorTask" which turns into a zombie. The ps aux
>> output says:
>>
>> root     28213  0.0  0.0      0     0 pts/0    Zl   04:21   0:02
>> [actuatorTask] <defunct>
>>
>>
>> To give you a heads up, the actuator Aperiodic task is a simple
>> for(;;) loop which blocks on recvfrom() on a UDP socket at every
>> iteration until a packet is received. Then it writes the data it
>> contains to a shared variable acquiring and releasing a mutex. any
>> ideas?
>
> If you mean that you have a sample code with which I would be able to
> reproduce the problem without requiring special hardware, I am very
> interested.
>
>>
>>
>> regards,
>>
>> alphan.
>>
>>
>>
>> ------- Dump starts here ---------
>>
>>
>> [33034.505238] SysRq : Show State
>> [33034.505238]   task                PC stack   pid father
>> (...)
>> [33034.505238] gatekeeper/0  S [0x00000001] f7c87f78     0    
>> 150      2
>> [33034.505238]        f7c87f94 00000046 00000046 f7c87f78 f7c88000
>> f7c88254 c0571f40 be53c9c0
>> [33034.505238]        0000137d 00000000 f7c87f94 c0138047 f7d81a20
>> c057be60 c057be6c f7c87fcc
>> [33034.505238]        c0158f77 00000003 f7c88000 f7c88254 00000000
>> 00000001 f7c88000 c011a4d0
>> [33034.505238] Call Trace:
>> [33034.505238]  [<c0138047>] ? up+0x57/0x90
>> [33034.505238]  [<c0158f77>] gatekeeper_thread+0xa7/0x140
>> [33034.505238]  [<c011a4d0>] ? default_wake_function+0x0/0x10
>> [33034.505238]  [<c0158ed0>] ? gatekeeper_thread+0x0/0x140
>> [33034.505238]  [<c01335a2>] kthread+0x42/0x70
>> [33034.505238]  [<c0133560>] ? kthread+0x0/0x70
>> [33034.505238]  [<c0103d43>] kernel_thread_helper+0x7/0x14
>
> So, the gatekeeper is in TASK_INTERRUPTIBLE state, waiting for a  
> thread
> to call "wake_up_interruptible_sync" in xnshadow_harden.
>
>
>> [33034.505239] actuatorTask  ? [0x00000040] 00000001     0  
>> 28213      1
>> [33034.505239]        f760df54 00000046 c0198b26 00000001 f7ce48e0
>> f7ce4b34 f765a540 c6b973df
>> [33034.505239]        0000137d 00000000 f7ce48d8 f7ce48e0 00000010
>> f7ce48e0 f760ddd8 f760df98
>> [33034.505239]        c0122e14 c04ee300 00000000 f7ce51c0 00000001
>> f760df84 f7ce4a84 f7ce4ac8
>> [33034.505239] Call Trace:
>> [33034.505239]  [<c0198b26>] ? kfree+0xe6/0xf0
>> [33034.505239]  [<c0122e14>] do_exit+0x494/0x710
>> [33034.505239]  [<c01230be>] do_group_exit+0x2e/0xb0
>> [33034.505239]  [<c012314f>] sys_exit_group+0xf/0x20
>> [33034.505239]  [<c01030d5>] syscall_call+0x7/0xb
>
> actuatorTask in TASK_DEAD state. So, it is officially dead.
>
>> [33034.505239]  =======================
>> [33034.505239] Actuator Aper S [0x00000201] f7f01f04     0  
>> 28214      1
>> [33034.505239]        f7f01f28 00000086 c057be64 f7f01f04 f7ce51c0
>> f7ce5414 f7f01f28 ce2e19e5
>> [33034.505239]        0000137c 00000000 00000001 00000200 f7d80620
>> f7ce51c0 c01586e0 f7f01f4c
>> [33034.505239]        c0158576 f7f01f6c c015891a 00000000 f7f01ebc
>> 00000006 00000000 c01586e0
>> [33034.505239] Call Trace:
>> [33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
>> [33034.505239]  [<c0158576>] xnshadow_harden+0x86/0x1f0
>> [33034.505239]  [<c015891a>] ? hisyscall_event+0xba/0x290
>> [33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
>> [33034.505239]  [<c015876f>] losyscall_event+0x8f/0x180
>> [33034.505239]  [<c01586e0>] ? losyscall_event+0x0/0x180
>> [33034.505239]  [<c014c89c>] __ipipe_dispatch_event+0x9c/0x170
>> [33034.505239]  [<c01122f2>] __ipipe_syscall_root+0x42/0x110
>> [33034.505239]  [<c01030ad>] system_call+0x29/0x4a
>
> So, "Actuator Aperiodic Task" is blocked inside xnshadow_harden, in
> state TASK_INTERRUPTIBLE | TASK_ATOMICSWITCH. This state is
> transitional, and, in fact, I think you should not even be able to
> observe it. The fact you got it two traces would be impossible, so I
> think the tasks are blocked in these states, and what we observe  
> here is
> the harden machinery being jamed. When this happens, could you try to
> hit sysrq+T again to see if you get the same states ?
>
>
> -- 
>                                                 Gilles.
>


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2433 bytes --]

Re: [Xenomai-help] Zombie user tasks

[Gilles Chanteperdrix]

Alphan Ulusoy wrote:
> Below you can find the dump that I've just taken with sysRq+T. I left  
> the computer as it is for 5+ hours since sending the previous post in  
> the morning. Or would you like me to restart everything all over and  
> send the new sysRq+T output when this happens again?

No, it is perfect, I mean, it shows exactly what I suspected. We just
need to understand how it happens. A small example which would allow me
to reproduce the bug would still be welcome.

-- 
					    Gilles.

Re: [Xenomai-help] Zombie user tasks

[Alphan Ulusoy]

[-- Attachment #1: Type: text/plain, Size: 983 bytes --]

I've downloaded and installed Xenomai 2.4.6.1 on a 2.6.27.10 kernel. I  
wanted to be sure before posting: the system ran for 13+ hours without  
crashing or anything. I guess the problem was the bug in the  
gatekeeper code you mentioned. Thanks a lot for pointing me in the  
right direction and helping me solve this issue! (I was reluctant to  
upgrade due to the problem with the I-pipe patch included in the 2.4.6  
release)


-- 
alphan




On Dec 23, 2008, at 2:49 PM, Gilles Chanteperdrix wrote:

> Alphan Ulusoy wrote:
>> Ok I will do that and post my findings. I use the adeos-ipipe-2.6.26-
>> x86-2.0-15.patch. Is upgrading xenomai and i-pipe enough or would you
>> recommend upgrading the kernel too (if there're any relevant changes
>> in the new releases) ?
>
> Please use the patch in the v2.4.6.1 release. I think it is for linux
> 2.6.26.7 and later, so, you will have to upgrade the kernel too.
>
> -- 
>                                                 Gilles.
>


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2433 bytes --]