Stale XFS mount for Kernel 2.6.25.14

Stale XFS mount for Kernel 2.6.25.14

[Ngo, Andrew]

Hi,  
 
This is my first attempt to report a stale mount problem.  If this is
not the relevant mail list, please let me know.  I suspect that my
problem is xfs related, thus I select this mailing list.
 
I have a system that recently upgraded it's kernel to 2.6.25.14.  The
system's previous kernel was 2.6.18.-8.el5.  With the previous kernel, I
have no problem with executing the following commands when the system is
in operation.  Note that the /mnt/mountpoint is mounted on a XFS file
system.
 
mount -o rw,remount /mnt/mountpoint
mount -o ro,remount /mnt/mountpoint
 
With kernel 2.6.25.14, however, the mount command may be hang when
changing from rw to ro.  When this happen, system is very sluggish.
Even the "shutdown -r now" command is not taken.  The only way to get
out of this hang is by manually pressing the power button
 
Your sugguestion is greatly appreciated.
 
Andy Ngo


[[HTML alternate version deleted]]

Re: Stale XFS mount for Kernel 2.6.25.14

[Dave Chinner]

On Sun, Oct 12, 2008 at 10:10:54PM -0400, Ngo, Andrew wrote:
> Hi,  
>  
> This is my first attempt to report a stale mount problem.  If this is
> not the relevant mail list, please let me know.  I suspect that my
> problem is xfs related, thus I select this mailing list.
>  
> I have a system that recently upgraded it's kernel to 2.6.25.14.  The
> system's previous kernel was 2.6.18.-8.el5.  With the previous kernel, I
> have no problem with executing the following commands when the system is
> in operation.  Note that the /mnt/mountpoint is mounted on a XFS file
> system.
>  
> mount -o rw,remount /mnt/mountpoint
> mount -o ro,remount /mnt/mountpoint
>  
> With kernel 2.6.25.14, however, the mount command may be hang when
> changing from rw to ro.  When this happen, system is very sluggish.

That's because the remount,ro is actually flushing all the dirty
data to disk in 2.6.25. There is also a check to catch when the
remount,ro fails to clean everything that is dirty - that's probably
what is triggering. Do you have an oops in the dmesg log when
mount hangs?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

RE: Stale XFS mount for Kernel 2.6.25.14

[Ngo, Andrew]

No.  There is no oops in the /var/log/messages when the mount command
hangs.

Andy Ngo 

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 13, 2008 12:00 AM
To: Ngo, Andrew
Cc: xfs@oss.sgi.com; Johnson, Je
Subject: Re: Stale XFS mount for Kernel 2.6.25.14

On Sun, Oct 12, 2008 at 10:10:54PM -0400, Ngo, Andrew wrote:
> Hi,
>  
> This is my first attempt to report a stale mount problem.  If this is 
> not the relevant mail list, please let me know.  I suspect that my 
> problem is xfs related, thus I select this mailing list.
>  
> I have a system that recently upgraded it's kernel to 2.6.25.14.  The 
> system's previous kernel was 2.6.18.-8.el5.  With the previous kernel,

> I have no problem with executing the following commands when the 
> system is in operation.  Note that the /mnt/mountpoint is mounted on a

> XFS file system.
>  
> mount -o rw,remount /mnt/mountpoint
> mount -o ro,remount /mnt/mountpoint
>  
> With kernel 2.6.25.14, however, the mount command may be hang when 
> changing from rw to ro.  When this happen, system is very sluggish.

That's because the remount,ro is actually flushing all the dirty data to
disk in 2.6.25. There is also a check to catch when the remount,ro fails
to clean everything that is dirty - that's probably what is triggering.
Do you have an oops in the dmesg log when mount hangs?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

RE: Stale XFS mount for Kernel 2.6.25.14

[Ngo, Andrew]

Dave,

Do you know of a solution to this problem?  Currently this is a major
impact for my system.  Thanks in advance.

Andy Ngo

-----Original Message-----
From: Ngo, Andrew 
Sent: Monday, October 13, 2008 8:07 AM
To: 'Dave Chinner'
Cc: xfs@oss.sgi.com; Johnson, Je; Ngo, Andrew
Subject: RE: Stale XFS mount for Kernel 2.6.25.14

No.  There is no oops in the /var/log/messages when the mount command
hangs.

Andy Ngo 

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com]
Sent: Monday, October 13, 2008 12:00 AM
To: Ngo, Andrew
Cc: xfs@oss.sgi.com; Johnson, Je
Subject: Re: Stale XFS mount for Kernel 2.6.25.14

On Sun, Oct 12, 2008 at 10:10:54PM -0400, Ngo, Andrew wrote:
> Hi,
>  
> This is my first attempt to report a stale mount problem.  If this is 
> not the relevant mail list, please let me know.  I suspect that my 
> problem is xfs related, thus I select this mailing list.
>  
> I have a system that recently upgraded it's kernel to 2.6.25.14.  The 
> system's previous kernel was 2.6.18.-8.el5.  With the previous kernel,

> I have no problem with executing the following commands when the 
> system is in operation.  Note that the /mnt/mountpoint is mounted on a

> XFS file system.
>  
> mount -o rw,remount /mnt/mountpoint
> mount -o ro,remount /mnt/mountpoint
>  
> With kernel 2.6.25.14, however, the mount command may be hang when 
> changing from rw to ro.  When this happen, system is very sluggish.

That's because the remount,ro is actually flushing all the dirty data to
disk in 2.6.25. There is also a check to catch when the remount,ro fails
to clean everything that is dirty - that's probably what is triggering.
Do you have an oops in the dmesg log when mount hangs?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

Re: Stale XFS mount for Kernel 2.6.25.14

[Dave Chinner]

[Andrew, Please don't top post replies - it makes it
completely impossible to folow the thread. ]

> -----Original Message-----
On Mon, Oct 13, 2008 at 10:04:27PM -0400, Ngo, Andrew wrote:
On Mon, Oct 13, 2008 at 10:04:27PM -0400, Ngo, Andrew wrote:
> 
> Dave,
> 
> Do you know of a solution to this problem?  Currently this is a major
> impact for my system.  Thanks in advance.

I don't know what the problem is, so I certainly don't have
a solution for you. You'll need to provide a whole lot more detail
about your system and problem for us to make any sense of it.
Such as:

What hardware do you have (including storage)?

What's the storage structure (DM, MD, iSCSI, etc)?

Output of 'xfs_info <mtpt>'?

What are you really trying to acheive with the remount command?

Is the filesystem busy at the time the ro,remount is run?
(e.g. any large background writes occurring?)

Does the problem go away if you do:

# xfs_freeze -f <mtpt>
# xfs_freeze -u <mtpt>
# mount -o ro,remount <mtpt>

Or does the first freeze command trigger the same problem?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

RE: Stale XFS mount for Kernel 2.6.25.14

[Ngo, Andrew]

 
What hardware do you have (including storage)?

[Andrew Ngo]Mother board: H8DA3-2
            Processor: Quad Core Opteron
            Storage: Quad SAS drives

What's the storage structure (DM, MD, iSCSI, etc)?

Output of 'xfs_info <mtpt>'?

[Andrew Ngo]xfs_info /mtpt
meta-data=/dev/sda3 isize=256 agcount=16 agsize=672219 blks
         =          sectsz=512 attr=0
data     =          bsize=4096 blocks=10755504, imaxpct=25
         =          sunit=0; swidth=0 blks, unwritten=1
naming   = version2 bsize=4096
log      = internal bsize=4096 blocks=525, version=1
                    sectsz=512  sunit=0 blocks
realtime = none     extsz=65536 blocks=0 rtextents=0

What are you really trying to acheive with the remount command?

[Andrew Ngo]This server hosts the patch directory for the developers.
When the developers are not patching software, the file system is ro.
When the developers needs to patch the software, the file system is
changed to rw.

Is the filesystem busy at the time the ro,remount is run?
(e.g. any large background writes occurring?)

[Andrew Ngo]Yes.  
Note that I make a script that performs the "mount -o rw,remount <mtpt>"
and "mount -o ro,remount <mnpt>".  When the system is not being used, I
can finish the loop of 100 mount operations; however, when the system is
being used, a couple of the above mount commands, even manually, will
cause the system to hang.

Does the problem go away if you do:

# xfs_freeze -f <mtpt>
# xfs_freeze -u <mtpt>
# mount -o ro,remount <mtpt>

[Andrew Ngo] When the mount command hangs, the above commands continue
the hang the system.

Or does the first freeze command trigger the same problem?
[Andrew Ngo] Yes, the xfs_freeze -f command hang the system, just like
the stale mount command does.

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 13, 2008 10:16 PM
To: Ngo, Andrew
Cc: v9fs-developer@lists.sourceforge.net; xfs@oss.sgi.com; Johnson, Je
Subject: Re: Stale XFS mount for Kernel 2.6.25.14

[Andrew, Please don't top post replies - it makes it completely
impossible to folow the thread. ]

> -----Original Message-----
On Mon, Oct 13, 2008 at 10:04:27PM -0400, Ngo, Andrew wrote:
On Mon, Oct 13, 2008 at 10:04:27PM -0400, Ngo, Andrew wrote:
> 
> Dave,
> 
> Do you know of a solution to this problem?  Currently this is a major 
> impact for my system.  Thanks in advance.

I don't know what the problem is, so I certainly don't have a solution
for you. You'll need to provide a whole lot more detail about your
system and problem for us to make any sense of it.
Such as:

What hardware do you have (including storage)?

What's the storage structure (DM, MD, iSCSI, etc)?

Output of 'xfs_info <mtpt>'?

What are you really trying to acheive with the remount command?

Is the filesystem busy at the time the ro,remount is run?
(e.g. any large background writes occurring?)

Does the problem go away if you do:

# xfs_freeze -f <mtpt>
# xfs_freeze -u <mtpt>
# mount -o ro,remount <mtpt>

Or does the first freeze command trigger the same problem?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

Re: Stale XFS mount for Kernel 2.6.25.14

[Dave Chinner]

On Mon, Oct 13, 2008 at 11:40:17PM -0400, Ngo, Andrew wrote:
> What's the storage structure (DM, MD, iSCSI, etc)?
> 
> Output of 'xfs_info <mtpt>'?
> 
> [Andrew Ngo]xfs_info /mtpt
> meta-data=/dev/sda3 isize=256 agcount=16 agsize=672219 blks
>          =          sectsz=512 attr=0
> data     =          bsize=4096 blocks=10755504, imaxpct=25
>          =          sunit=0; swidth=0 blks, unwritten=1
> naming   = version2 bsize=4096
> log      = internal bsize=4096 blocks=525, version=1
>                     sectsz=512  sunit=0 blocks
> realtime = none     extsz=65536 blocks=0 rtextents=0

I think your cut-n-paste has dropped characters there - the log
cannot be that small - it must be at least 10MB in size
(2560 blocks, IIRC). Can you check this, please?

> What are you really trying to acheive with the remount command?
> 
> [Andrew Ngo]This server hosts the patch directory for the developers.
> When the developers are not patching software, the file system is ro.
> When the developers needs to patch the software, the file system is
> changed to rw.

Ok. So when a software update needs to be done, the filesystem
is made writable, correct? But once the update is done, it
then gets made RO again? how much data gets written when an
update is run?

> Is the filesystem busy at the time the ro,remount is run?
> (e.g. any large background writes occurring?)
> 
> [Andrew Ngo]Yes.  

So you try to make a busy, dirty filesystem read only while an
application is still writing to it?

> Note that I make a script that performs the "mount -o rw,remount <mtpt>"
> and "mount -o ro,remount <mnpt>".  When the system is not being used, I
> can finish the loop of 100 mount operations; however, when the system is
> being used, a couple of the above mount commands, even manually, will
> cause the system to hang.
> 
> Does the problem go away if you do:
> 
> # xfs_freeze -f <mtpt>
> # xfs_freeze -u <mtpt>
> # mount -o ro,remount <mtpt>
> 
> [Andrew Ngo] When the mount command hangs, the above commands continue
> the hang the system.

I meant using that sequence of commands instead of just a single
"mount -o ro,remount <mtpt>" to make the fs RO. They won't magically
fix anything once something is broken.

> Or does the first freeze command trigger the same problem?
> [Andrew Ngo] Yes, the xfs_freeze -f command hang the system, just like
> the stale mount command does.

Is that before or after a remount has already hung?

Next time you hang the system, please attach the output that
appears in your syslog from:

# echo t > /proc/sysrq-trigger

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

RE: Stale XFS mount for Kernel 2.6.25.14

[Ngo, Andrew]

Dave,

Here is the response to your questions.  Thanks...

> I think your cut-n-paste has dropped characters there - the log cannot
be that small - it 
> must be at least 10MB in size (2560 blocks, IIRC). Can you check this,
please?

[Andrew Ngo]xfs_info /mtpt
meta-data=/dev/sda3 isize=256 agcount=16 agsize=672219 blks
         =          sectsz=512 attr=0
data     =          bsize=4096 blocks=10755504, imaxpct=25
         =          sunit=0 swidth=0 blks, unwritten=1
naming   = version2 bsize=4096
log      = internal bsize=4096 blocks=5251, version=1
         =          sectsz=512  sunit=0 blks
realtime = none     extsz=65536 blocks=0 rtextents=0 

If the log size is too small, please advise on how to configure the log
to a larger size. 

> Ok. So when a software update needs to be done, the filesystem is made
writable, correct? 
> But once the update is done, it then gets made RO again? how much data
gets written when an 
> update is run?
[Andrew Ngo]It depends on the size of the patch.  

> So you try to make a busy, dirty filesystem read only while an
application is still writing 
> to it?

[Andrew Ngo]Actually, we only perform the mount ro after we patch the
softwarfe.  At that time, we have full control of the screen which means
that the system is not busy; but I don't know if the file system is
still busy under the cover.

When the file system is actually busy, the mount command should return
saying the partition is busy.  For us, it is not returning anything.  It
just hangs.

> Is that before or after a remount has already hung?
[Andrew Ngo]The xfs_freeze -f command hangs the system after the remount
has already hung.

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Tuesday, October 14, 2008 1:54 AM
To: Ngo, Andrew
Cc: v9fs-developer@lists.sourceforge.net; xfs@oss.sgi.com; Johnson, Je
Subject: Re: Stale XFS mount for Kernel 2.6.25.14

On Mon, Oct 13, 2008 at 11:40:17PM -0400, Ngo, Andrew wrote:
> What's the storage structure (DM, MD, iSCSI, etc)?
> 
> Output of 'xfs_info <mtpt>'?
> 
> [Andrew Ngo]xfs_info /mtpt
> meta-data=/dev/sda3 isize=256 agcount=16 agsize=672219 blks
>          =          sectsz=512 attr=0
> data     =          bsize=4096 blocks=10755504, imaxpct=25
>          =          sunit=0; swidth=0 blks, unwritten=1
> naming   = version2 bsize=4096
> log      = internal bsize=4096 blocks=525, version=1
>                     sectsz=512  sunit=0 blocks
> realtime = none     extsz=65536 blocks=0 rtextents=0

I think your cut-n-paste has dropped characters there - the log cannot
be that small - it must be at least 10MB in size (2560 blocks, IIRC).
Can you check this, please?

> What are you really trying to acheive with the remount command?
> 
> [Andrew Ngo]This server hosts the patch directory for the developers.
> When the developers are not patching software, the file system is ro.
> When the developers needs to patch the software, the file system is 
> changed to rw.

Ok. So when a software update needs to be done, the filesystem is made
writable, correct? But once the update is done, it then gets made RO
again? how much data gets written when an update is run?

> Is the filesystem busy at the time the ro,remount is run?
> (e.g. any large background writes occurring?)
> 
> [Andrew Ngo]Yes.  

So you try to make a busy, dirty filesystem read only while an
application is still writing to it?

> Note that I make a script that performs the "mount -o rw,remount
<mtpt>"
> and "mount -o ro,remount <mnpt>".  When the system is not being used, 
> I can finish the loop of 100 mount operations; however, when the 
> system is being used, a couple of the above mount commands, even 
> manually, will cause the system to hang.
> 
> Does the problem go away if you do:
> 
> # xfs_freeze -f <mtpt>
> # xfs_freeze -u <mtpt>
> # mount -o ro,remount <mtpt>
> 
> [Andrew Ngo] When the mount command hangs, the above commands continue

> the hang the system.

I meant using that sequence of commands instead of just a single "mount
-o ro,remount <mtpt>" to make the fs RO. They won't magically fix
anything once something is broken.

> Or does the first freeze command trigger the same problem?
> [Andrew Ngo] Yes, the xfs_freeze -f command hang the system, just like

> the stale mount command does.

Is that before or after a remount has already hung?

Next time you hang the system, please attach the output that appears in
your syslog from:

# echo t > /proc/sysrq-trigger

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

ngo@lmco.com>,
	v9fs-developer@lists.sourceforge.net, xfs@oss.sgi.com,
	"Johnson, Je" <je.johnson@lmco.com>
Message-ID: <20081014055348.GL10716@disturbed>
Received: from mailgw3b.lmco.com (mg_filter)
	by mailgw3b.lmco.com with mailout; Tue, 14 Oct 2008 01:54:09 %z
Received: from ipmail05.adl2.internode.on.net
	by mailgw3b.lmco.com with ESMTP id m9E5s5Mf015764; Tue, 14 Oct
2008 01:54:05 -0400 (EDT)
Received: from ppp121-44-93-99.lns10.syd6.internode.on.net (HELO
disturbed) ([121.44.93.99])
  by ipmail05.adl2.internode.on.net with ESMTP; 14 Oct 2008 16:23:49
+1030
Received: from dave by disturbed with local (Exim 4.69)
	(envelope-from <david@fromorbit.com>)
	id 1Kpcqq-0000dZ-Lm; Tue, 14 Oct 2008 16:53:48 +1100
References:
<8604545CB7815D419F5FF108D3E434BA3BD626@emss04m05.us.lmco.com>
<20081013035939.GB10716@disturbed>
<8604545CB7815D419F5FF108D3E434BA3BD628@emss04m05.us.lmco.com>
<8604545CB7815D419F5FF108D3E434BA3BD62F@emss04m05.us.lmco.com>
<20081014021534.GI10716@disturbed>
<8604545CB7815D419F5FF108D3E434BA3BD631@emss04m05.us.lmco.com>
Subject: Re: Stale XFS mount for Kernel 2.6.25.14
To: "Ngo, Andrew" <andrew.ngo@lmco.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
X-IronPort-AV: E=Sophos;i="4.33,407,1220193000"; 
   d="scan'208";a="229396226"
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApoEAOfP80h5LF1j/2dsb2JhbAC9MoFs
X-MessageGate-spam: 116
X-MessageGate-virus: 1

On Mon, Oct 13, 2008 at 11:40:17PM -0400, Ngo, Andrew wrote:
> What's the storage structure (DM, MD, iSCSI, etc)?
> 
> Output of 'xfs_info <mtpt>'?
> 
> [Andrew Ngo]xfs_info /mtpt
> meta-data=/dev/sda3 isize=256 agcount=16 agsize=672219 blks
>          =          sectsz=512 attr=0
> data     =          bsize=4096 blocks=10755504, imaxpct=25
>          =          sunit=0; swidth=0 blks, unwritten=1
> naming   = version2 bsize=4096
> log      = internal bsize=4096 blocks=525, version=1
>                     sectsz=512  sunit=0 blocks
> realtime = none     extsz=65536 blocks=0 rtextents=0

I think your cut-n-paste has dropped characters there - the log cannot
be that small - it must be at least 10MB in size (2560 blocks, IIRC).
Can you check this, please?

> What are you really trying to acheive with the remount command?
> 
> [Andrew Ngo]This server hosts the patch directory for the developers.
> When the developers are not patching software, the file system is ro.
> When the developers needs to patch the software, the file system is 
> changed to rw.

Ok. So when a software update needs to be done, the filesystem is made
writable, correct? But once the update is done, it then gets made RO
again? how much data gets written when an update is run?

> Is the filesystem busy at the time the ro,remount is run?
> (e.g. any large background writes occurring?)
> 
> [Andrew Ngo]Yes.  

So you try to make a busy, dirty filesystem read only while an
application is still writing to it?

> Note that I make a script that performs the "mount -o rw,remount
<mtpt>"
> and "mount -o ro,remount <mnpt>".  When the system is not being used, 
> I can finish the loop of 100 mount operations; however, when the 
> system is being used, a couple of the above mount commands, even 
> manually, will cause the system to hang.
> 
> Does the problem go away if you do:
> 
> # xfs_freeze -f <mtpt>
> # xfs_freeze -u <mtpt>
> # mount -o ro,remount <mtpt>
> 
> [Andrew Ngo] When the mount command hangs, the above commands continue

> the hang the system.

I meant using that sequence of commands instead of just a single "mount
-o ro,remount <mtpt>" to make the fs RO. They won't magically fix
anything once something is broken.

> Or does the first freeze command

RE: Stale XFS mount for Kernel 2.6.25.14

[Ngo, Andrew]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

Hi, Dave,

Here is the captured /var/log/messages when the system is hung by the
mount command on a xfs file system and I issued the 'echo w >
/proc/sysrq-trigger` command.

Note that I may have run multiple times of the above command, and that I
was also issuing the xfs_freeze commands while the mount is hung.  I
hope that these commands does not corrupt the /var/log/message that you
want to examine.

Thanks...
Andy Ngo

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 20, 2008 7:08 PM
To: Ngo, Andrew
Cc: Johnson, Je
Subject: Re: FW: Stale XFS mount for Kernel 2.6.25.14

On Mon, Oct 20, 2008 at 06:46:19PM -0400, Ngo, Andrew wrote:
>  
> Hi, Dave,
> 
> Just wondering do you have any more insight to the mount problem we 
> are having?  I posted my response to your question on Oct 14, and have

> not heard from you.

Been busy at my day job - I haven't had much time to look into this
further yet.

FWIW, w.r.t to the question about xfs_freeze -f hanging the system, I
meant doing that instead of remounting read-only, not after the remount
has already hung. Can you try that and see?

Also, the output of 'echo w > /proc/sysrq-trigger` once the system is in
the hung state will tell me a lot about what has gone wrong.

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com



[-- Attachment #2: messages.tar.gz --]
[-- Type: application/x-gzip, Size: 150993 bytes --]

Re: Stale XFS mount for Kernel 2.6.25.14

[Dave Chinner]

On Mon, Nov 03, 2008 at 03:49:55PM -0500, Ngo, Andrew wrote:
> Hi, Dave,
> 
> Here is the captured /var/log/messages when the system is hung by the
> mount command on a xfs file system and I issued the 'echo w >
> /proc/sysrq-trigger` command.
> 
> Note that I may have run multiple times of the above command, and that I
> was also issuing the xfs_freeze commands while the mount is hung.  I
> hope that these commands does not corrupt the /var/log/message that you
> want to examine.

Summary:

It appears you have a hung NFS exported ext3 filesystem that
pdflush is stuck on holding while a superblock lock and then
sync, remount and freeze all get stuck on.

pdflush:

Nov  3 18:27:24 4003a6 kernel: pdflush       D ffffffffffffffff     0   498      2
Nov  3 18:27:24 4003a6 kernel:  ffff810246037dc0 0000000000000046 00000000ffffffff ffffffff8024413b
Nov  3 18:27:24 4003a6 kernel:  0000000000000004 ffff810247326440 ffff8102472342c0 ffff8102473267a0
Nov  3 18:27:24 4003a6 kernel:  0000000646037d60 ffffffff80226c12 0000000000000000 ffffffff80226a88
Nov  3 18:27:24 4003a6 kernel: Call Trace:
Nov  3 18:27:24 4003a6 kernel:  [<ffffffff8024413b>] bit_waitqueue+0x10/0xa3
Nov  3 18:27:24 4003a6 kernel:  [<ffffffff80226c12>] dequeue_entity+0x1b/0xd3
Nov  3 18:27:24 4003a6 kernel:  [<ffffffff80226a88>] __dequeue_entity+0x25/0x69
Nov  3 18:27:24 4003a6 kernel:  [<ffffffff80226aef>] set_next_entity+0x23/0x73
Nov  3 18:27:24 4003a6 kernel:  [<ffffffff804c4be3>] __down_read+0x71/0x8b
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8028d68e>] sync_supers+0x39/0xa4
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026ad66>] pdflush+0x0/0x1f7
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026a969>] wb_kupdate+0x2d/0x112
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026ad66>] pdflush+0x0/0x1f7
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026aeb7>] pdflush+0x151/0x1f7
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026a93c>] wb_kupdate+0x0/0x112
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff802440ea>] kthread+0x47/0x75
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8022e2e4>] schedule_tail+0x28/0x5d
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8020cac8>] child_rip+0xa/0x12
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff802440a3>] kthread+0x0/0x75
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8020cabe>] child_rip+0x0/0x12

Is stuck in sync_supers() here:

408         list_for_each_entry(sb, &super_blocks, s_list) {
409                 if (sb->s_dirt) {
410                         sb->s_count++;
411                         spin_unlock(&sb_lock);
412    >>>>>>>>             down_read(&sb->s_umount);
413                         write_super(sb);
414                         up_read(&sb->s_umount);


The xfs_io (freeze) command:

Nov  3 18:27:25 4003a6 kernel: xfs_io        D ffff810446c1ad80     0 30903  30883
Nov  3 18:27:25 4003a6 kernel:  ffff810178c8ddb8 0000000000000082 0000000100792e1c 00007fca13fad000
Nov  3 18:27:25 4003a6 kernel:  ffff81042ac01740 ffff81023483f660 ffff810150566c70 ffff81023483f9c0
Nov  3 18:27:25 4003a6 kernel:  0000000503c19970 ffff810301a2a1e8 00007fca13fad000 0000000000000002
Nov  3 18:27:25 4003a6 kernel: Call Trace:
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026efc7>] zone_statistics+0x3f/0x93
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026436f>] find_lock_page+0x1f/0x7f
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff804c4be3>] __down_read+0x71/0x8b
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8028d837>] get_super+0x44/0x8d
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff802aa347>] freeze_bdev+0x28/0x80
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80226a88>] __dequeue_entity+0x25/0x69
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80337363>] xfs_ioctl+0x5a1/0x656
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80273044>] handle_mm_fault+0x360/0x728
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff803359c8>] xfs_file_ioctl+0x28/0x58
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80296dcd>] vfs_ioctl+0x21/0x6b
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8029705f>] do_vfs_ioctl+0x248/0x261
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff802970c9>] sys_ioctl+0x51/0x70
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80

is stuck here:

493         list_for_each_entry(sb, &super_blocks, s_list) {
494                 if (sb->s_bdev == bdev) {
495                         sb->s_count++;
496                         spin_unlock(&sb_lock);
497   >>>>>>                down_read(&sb->s_umount);
498                         if (sb->s_root)
499                                 return sb;
500                         up_read(&sb->s_umount);

Which is probably the same place as pdflush.

The mount command:

Nov  3 18:27:25 4003a6 kernel: mount         D ffff8104460f7c00     0  8431   4967
Nov  3 18:27:25 4003a6 kernel:  ffff810283a5fdc8 0000000000000082 ffff810447045480 ffff810446c1ad80
Nov  3 18:27:25 4003a6 kernel:  ffff810447045480 ffff810441872b30 ffff8103a97869f0 ffff810441872e90
Nov  3 18:27:25 4003a6 kernel:  0000000683a5fe68 ffffffff80294ad0 000000090505097a ffff81031edc1008
Nov  3 18:27:25 4003a6 kernel: Call Trace:
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80294ad0>] __link_path_walk+0xc45/0xd64
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026436f>] find_lock_page+0x1f/0x7f
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff804c4b51>] __down_write_nested+0x71/0x8b
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff802a05aa>] do_remount+0x6f/0xca
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff802a1127>] do_mount+0x14b/0x1db
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff802a1241>] sys_mount+0x8a/0xcd
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80

is stuck on a down_write here:

1548 >>>>>   down_write(&sb->s_umount);
1549         if (flags & MS_BIND)
1550                 err = change_mount_flags(nd->path.mnt, flags);
1551         else
1552                 err = do_remount_sb(sb, flags, data, 0);
1553         if (!err)
1554                 nd->path.mnt->mnt_flags = mnt_flags;
1555         up_write(&sb->s_umount);

Which implies that the s_umount semaphore has leaked as none of the
blocked threads are holding that semaphore. Hmmm - later on it appears
that some nfsds get stuck doing write_inode_now on an I_SYNC flag:

Nov  3 18:28:18 4003a6 kernel: nfsd          D ffff810246cbdb70     0  5370      2
Nov  3 18:28:18 4003a6 kernel:  ffff8104345d3c80 0000000000000046 ffff8104345d3c30 ffff8104345d3d90
Nov  3 18:28:18 4003a6 kernel:  0000000000000000 ffff810434bd8180 ffff8104411e5320 ffff810434bd84e0
Nov  3 18:28:18 4003a6 kernel:  0000000000000001 ffffffff8026a133 ffff810246cbdb70 ffff8100754192a0
Nov  3 18:28:18 4003a6 kernel: Call Trace:
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8026a133>] write_cache_pages+0x261/0x2b6
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff804c3e0e>] schedule_timeout+0x8a/0xad
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88031426>] :jbd:journal_stop+0x92/0x18d
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff802a5b83>] __writeback_single_inode+0x17f/0x274
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff802a5cf1>] write_inode_now+0x79/0xbd
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88c4b982>] :nfsd:nfsd_setattr+0x3df/0x40c
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88c5202f>] :nfsd:nfsd3_proc_setattr+0x8d/0x99
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88c47245>] :nfsd:nfsd_dispatch+0xde/0x1b6
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88b8d145>] :sunrpc:svc_process+0x3fe/0x6f1
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff804c4b84>] __down_read+0x12/0x8b
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88c4767c>] :nfsd:nfsd+0x0/0x2b4
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88c47819>] :nfsd:nfsd+0x19d/0x2b4
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8022e2e4>] schedule_tail+0x28/0x5d
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8020cac8>] child_rip+0xa/0x12
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88c4767c>] :nfsd:nfsd+0x0/0x2b4

That makes me think there's already a process doing writeback
holding the s_umount semaphore, but rather than being stuck asleep
is spinning so is not showing up in the "blocked thread" traces. If
there a CPU running at 100% once this problem has triggered?

Also, whatever it is might show up in an all-process stack dump
rather than just block processes.  Can you issue a 'echo t >
/proc/sysrq-trigger" to show all the processes in the machine

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

RE: Stale XFS mount for Kernel 2.6.25.14

[Ngo, Andrew]

[-- Attachment #1: Type: text/plain, Size: 11010 bytes --]

Dave,

>From time to time I am still experiencing the stale mount.  From this
last email, you want me to issue the 'echo t > /proc/sysrq-trigger'
command to show all the processes in the machine.  Here is the capture
of the /var/log/messages when I run the command. Please let me know what
you find out.  Thanks...

Andy Ngo

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Tuesday, November 04, 2008 1:06 AM
To: Ngo, Andrew
Cc: v9fs-developer@lists.sourceforge.net; xfs@oss.sgi.com; Johnson, Je
Subject: Re: Stale XFS mount for Kernel 2.6.25.14

On Mon, Nov 03, 2008 at 03:49:55PM -0500, Ngo, Andrew wrote:
> Hi, Dave,
> 
> Here is the captured /var/log/messages when the system is hung by the 
> mount command on a xfs file system and I issued the 'echo w > 
> /proc/sysrq-trigger` command.
> 
> Note that I may have run multiple times of the above command, and that

> I was also issuing the xfs_freeze commands while the mount is hung.  I

> hope that these commands does not corrupt the /var/log/message that 
> you want to examine.

Summary:

It appears you have a hung NFS exported ext3 filesystem that pdflush is
stuck on holding while a superblock lock and then sync, remount and
freeze all get stuck on.

pdflush:

Nov  3 18:27:24 4003a6 kernel: pdflush       D ffffffffffffffff     0
498      2
Nov  3 18:27:24 4003a6 kernel:  ffff810246037dc0 0000000000000046
00000000ffffffff ffffffff8024413b Nov  3 18:27:24 4003a6 kernel:
0000000000000004 ffff810247326440 ffff8102472342c0 ffff8102473267a0 Nov
3 18:27:24 4003a6 kernel:  0000000646037d60 ffffffff80226c12
0000000000000000 ffffffff80226a88 Nov  3 18:27:24 4003a6 kernel: Call
Trace:
Nov  3 18:27:24 4003a6 kernel:  [<ffffffff8024413b>]
bit_waitqueue+0x10/0xa3 Nov  3 18:27:24 4003a6 kernel:
[<ffffffff80226c12>] dequeue_entity+0x1b/0xd3 Nov  3 18:27:24 4003a6
kernel:  [<ffffffff80226a88>] __dequeue_entity+0x25/0x69 Nov  3 18:27:24
4003a6 kernel:  [<ffffffff80226aef>] set_next_entity+0x23/0x73 Nov  3
18:27:24 4003a6 kernel:  [<ffffffff804c4be3>] __down_read+0x71/0x8b Nov
3 18:27:25 4003a6 kernel:  [<ffffffff8028d68e>] sync_supers+0x39/0xa4
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026ad66>] pdflush+0x0/0x1f7
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026a969>]
wb_kupdate+0x2d/0x112 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff8026ad66>] pdflush+0x0/0x1f7 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff8026aeb7>] pdflush+0x151/0x1f7 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff8026a93c>] wb_kupdate+0x0/0x112 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff802440ea>] kthread+0x47/0x75 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff8022e2e4>] schedule_tail+0x28/0x5d Nov  3 18:27:25 4003a6
kernel:  [<ffffffff8020cac8>] child_rip+0xa/0x12 Nov  3 18:27:25 4003a6
kernel:  [<ffffffff802440a3>] kthread+0x0/0x75 Nov  3 18:27:25 4003a6
kernel:  [<ffffffff8020cabe>] child_rip+0x0/0x12

Is stuck in sync_supers() here:

408         list_for_each_entry(sb, &super_blocks, s_list) {
409                 if (sb->s_dirt) {
410                         sb->s_count++;
411                         spin_unlock(&sb_lock);
412    >>>>>>>>             down_read(&sb->s_umount);
413                         write_super(sb);
414                         up_read(&sb->s_umount);


The xfs_io (freeze) command:

Nov  3 18:27:25 4003a6 kernel: xfs_io        D ffff810446c1ad80     0
30903  30883
Nov  3 18:27:25 4003a6 kernel:  ffff810178c8ddb8 0000000000000082
0000000100792e1c 00007fca13fad000 Nov  3 18:27:25 4003a6 kernel:
ffff81042ac01740 ffff81023483f660 ffff810150566c70 ffff81023483f9c0 Nov
3 18:27:25 4003a6 kernel:  0000000503c19970 ffff810301a2a1e8
00007fca13fad000 0000000000000002 Nov  3 18:27:25 4003a6 kernel: Call
Trace:
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff8026efc7>]
zone_statistics+0x3f/0x93 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff8026436f>] find_lock_page+0x1f/0x7f Nov  3 18:27:25 4003a6
kernel:  [<ffffffff804c4be3>] __down_read+0x71/0x8b Nov  3 18:27:25
4003a6 kernel:  [<ffffffff8028d837>] get_super+0x44/0x8d Nov  3 18:27:25
4003a6 kernel:  [<ffffffff802aa347>] freeze_bdev+0x28/0x80 Nov  3
18:27:25 4003a6 kernel:  [<ffffffff80226a88>] __dequeue_entity+0x25/0x69
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80337363>]
xfs_ioctl+0x5a1/0x656 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff80273044>] handle_mm_fault+0x360/0x728 Nov  3 18:27:25 4003a6
kernel:  [<ffffffff803359c8>] xfs_file_ioctl+0x28/0x58 Nov  3 18:27:25
4003a6 kernel:  [<ffffffff80296dcd>] vfs_ioctl+0x21/0x6b Nov  3 18:27:25
4003a6 kernel:  [<ffffffff8029705f>] do_vfs_ioctl+0x248/0x261 Nov  3
18:27:25 4003a6 kernel:  [<ffffffff802970c9>] sys_ioctl+0x51/0x70 Nov  3
18:27:25 4003a6 kernel:  [<ffffffff8020bc9b>]
system_call_after_swapgs+0x7b/0x80

is stuck here:

493         list_for_each_entry(sb, &super_blocks, s_list) {
494                 if (sb->s_bdev == bdev) {
495                         sb->s_count++;
496                         spin_unlock(&sb_lock);
497   >>>>>>                down_read(&sb->s_umount);
498                         if (sb->s_root)
499                                 return sb;
500                         up_read(&sb->s_umount);

Which is probably the same place as pdflush.

The mount command:

Nov  3 18:27:25 4003a6 kernel: mount         D ffff8104460f7c00     0
8431   4967
Nov  3 18:27:25 4003a6 kernel:  ffff810283a5fdc8 0000000000000082
ffff810447045480 ffff810446c1ad80 Nov  3 18:27:25 4003a6 kernel:
ffff810447045480 ffff810441872b30 ffff8103a97869f0 ffff810441872e90 Nov
3 18:27:25 4003a6 kernel:  0000000683a5fe68 ffffffff80294ad0
000000090505097a ffff81031edc1008 Nov  3 18:27:25 4003a6 kernel: Call
Trace:
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80294ad0>]
__link_path_walk+0xc45/0xd64 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff8026436f>] find_lock_page+0x1f/0x7f Nov  3 18:27:25 4003a6
kernel:  [<ffffffff804c4b51>] __down_write_nested+0x71/0x8b Nov  3
18:27:25 4003a6 kernel:  [<ffffffff802a05aa>] do_remount+0x6f/0xca Nov
3 18:27:25 4003a6 kernel:  [<ffffffff802a1127>] do_mount+0x14b/0x1db Nov
3 18:27:25 4003a6 kernel:  [<ffffffff802a1241>] sys_mount+0x8a/0xcd Nov
3 18:27:25 4003a6 kernel:  [<ffffffff8020bc9b>]
system_call_after_swapgs+0x7b/0x80

is stuck on a down_write here:

1548 >>>>>   down_write(&sb->s_umount);
1549         if (flags & MS_BIND)
1550                 err = change_mount_flags(nd->path.mnt, flags);
1551         else
1552                 err = do_remount_sb(sb, flags, data, 0);
1553         if (!err)
1554                 nd->path.mnt->mnt_flags = mnt_flags;
1555         up_write(&sb->s_umount);

Which implies that the s_umount semaphore has leaked as none of the
blocked threads are holding that semaphore. Hmmm - later on it appears
that some nfsds get stuck doing write_inode_now on an I_SYNC flag:

Nov  3 18:28:18 4003a6 kernel: nfsd          D ffff810246cbdb70     0
5370      2
Nov  3 18:28:18 4003a6 kernel:  ffff8104345d3c80 0000000000000046
ffff8104345d3c30 ffff8104345d3d90 Nov  3 18:28:18 4003a6 kernel:
0000000000000000 ffff810434bd8180 ffff8104411e5320 ffff810434bd84e0 Nov
3 18:28:18 4003a6 kernel:  0000000000000001 ffffffff8026a133
ffff810246cbdb70 ffff8100754192a0 Nov  3 18:28:18 4003a6 kernel: Call
Trace:
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8026a133>]
write_cache_pages+0x261/0x2b6 Nov  3 18:28:18 4003a6 kernel:
[<ffffffff8023addd>] lock_timer_base+0x26/0x4b Nov  3 18:28:18 4003a6
kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf Nov  3 18:28:18
4003a6 kernel:  [<ffffffff804c3e0e>] schedule_timeout+0x8a/0xad Nov  3
18:28:18 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88031426>]
:jbd:journal_stop+0x92/0x18d Nov  3 18:28:18 4003a6 kernel:
[<ffffffff802a5b83>] __writeback_single_inode+0x17f/0x274
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff802a5cf1>]
write_inode_now+0x79/0xbd Nov  3 18:28:18 4003a6 kernel:
[<ffffffff88c4b982>] :nfsd:nfsd_setattr+0x3df/0x40c Nov  3 18:28:18
4003a6 kernel:  [<ffffffff88c5202f>] :nfsd:nfsd3_proc_setattr+0x8d/0x99
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff88c47245>]
:nfsd:nfsd_dispatch+0xde/0x1b6 Nov  3 18:28:18 4003a6 kernel:
[<ffffffff88b8d145>] :sunrpc:svc_process+0x3fe/0x6f1 Nov  3 18:28:18
4003a6 kernel:  [<ffffffff804c4b84>] __down_read+0x12/0x8b Nov  3
18:28:18 4003a6 kernel:  [<ffffffff88c4767c>] :nfsd:nfsd+0x0/0x2b4 Nov
3 18:28:18 4003a6 kernel:  [<ffffffff88c47819>] :nfsd:nfsd+0x19d/0x2b4
Nov  3 18:28:18 4003a6 kernel:  [<ffffffff8022e2e4>]
schedule_tail+0x28/0x5d Nov  3 18:28:18 4003a6 kernel:
[<ffffffff8020cac8>] child_rip+0xa/0x12 Nov  3 18:28:18 4003a6 kernel:
[<ffffffff88c4767c>] :nfsd:nfsd+0x0/0x2b4

That makes me think there's already a process doing writeback holding
the s_umount semaphore, but rather than being stuck asleep is spinning
so is not showing up in the "blocked thread" traces. If there a CPU
running at 100% once this problem has triggered?

Also, whatever it is might show up in an all-process stack dump rather
than just block processes.  Can you issue a 'echo t >
/proc/sysrq-trigger" to show all the processes in the machine

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

          up_read(&sb->s_umount);

Which is probably the same place as pdflush.

The mount command:

Nov  3 18:27:25 4003a6 kernel: mount         D ffff8104460f7c00     0
8431   4967
Nov  3 18:27:25 4003a6 kernel:  ffff810283a5fdc8 0000000000000082
ffff810447045480 ffff810446c1ad80 Nov  3 18:27:25 4003a6 kernel:
ffff810447045480 ffff810441872b30 ffff8103a97869f0 ffff810441872e90 Nov
3 18:27:25 4003a6 kernel:  0000000683a5fe68 ffffffff80294ad0
000000090505097a ffff81031edc1008 Nov  3 18:27:25 4003a6 kernel: Call
Trace:
Nov  3 18:27:25 4003a6 kernel:  [<ffffffff80294ad0>]
__link_path_walk+0xc45/0xd64 Nov  3 18:27:25 4003a6 kernel:
[<ffffffff8026436f>] find_lock_page+0x1f/0x7f Nov  3 18:27:25 4003a6
kernel:  [<ffffffff804c4b51>] __down_write_nested+0x71/0x8b Nov  3
18:27:25 4003a6 kernel:  [<ffffffff802a05aa>] do_remount+0x6f/0xca Nov
3 18:27:25 4003a6 kernel:  [<ffffffff802a1127>] do_mount+0x14b/0x1db Nov
3 18:27:25 4003a6 kernel:  [<ffffffff802a1241>] sys_mount+0x8a/0xcd Nov
3 18:27:25 4003a6 kernel:  [<ffffffff8020bc9b>]
system_call_after_swapgs+0x7b/0x80

is stuck on a down_write here:

1548 >>>>>   down_write(&sb->s_umount);
1549         if (flags & MS_BIND)
1550                 err = change_mount_flags(nd->path.mnt, flags);
1551         else
1552                 err = do_remount_sb(sb, flags, data, 0);
1553         if (!err)
1554                 nd->path.mnt->mnt_flags = mnt_flags;
1555         up_write(&sb->s_umount);

Which implies that the s_umount semaphore has leaked as none of the
blocked threads are holding that semaphore. Hmmm - later on it appears
that some nfsds get stuck doing write_inode_now on an I_SYNC flag:

Nov  3 18:28:18 4003a6 kernel: nfsd          D ffff810246cbdb70     0
5370      2
Nov  3 18:28:18 4003a6 kernel:  ffff8104345d3c80 0000000000000046
ffff8104345d3c30 ffff8104345d3d90 Nov  3 18:28:18 4003a6 kernel:
0000000000000000 ffff810434bd8180 ffff8104411e5320 ffff

[-- Attachment #2: messages.captued.with_sysrq-trigger --]
[-- Type: application/octet-stream, Size: 211929 bytes --]

4580a4581,7320
> Jan 14 18:18:36 4003a6 xinetd[5538]: EXIT: tftp status=0 pid=7423 duration=908(sec)
> Jan 14 18:19:00 4003a6 kernel: s_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b8ff18     0  6292      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81042ae3dca8 0000000000000082 ffff8102e3088780 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8103ff4df300 ffff8103808fa970 ffff8102e3196140 ffff8103808facd0
> Jan 14 18:19:00 4003a6 kernel:  00000005ae802540 0000000100000000 ffff81042ae3dea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b91fb8     0  6325      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102ae9b7ca8 0000000000000082 ffff81042b494740 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8104259bcd80 ffff81030f8335e0 ffff8104089cc400 ffff81030f833940
> Jan 14 18:19:00 4003a6 kernel:  00000006ae802540 0000000100000000 ffff8102ae9b7ea8 0000000080267b8c
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b940f8     0  6352      1
> Jan 14 18:19:00 4003a6 kernel:  ffff810263111ca8 0000000000000082 ffff8103388dedc0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff81040682a8c0 ffff810447334d70 ffff81024512e300 ffff8104473350d0
> Jan 14 18:19:00 4003a6 kernel:  00000007ae802540 0000000100000000 ffff810263111ea8 0000000000000000
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80228fee>] __wake_up+0x38/0x4e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024b31d>] futex_wake+0xef/0x101
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b965c0     0  6726      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102e3117ca8 0000000000000082 ffff8103388c12c0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8102465fc9c0 ffff8103531fd5e0 ffff81026301a140 ffff8103531fd940
> Jan 14 18:19:00 4003a6 kernel:  00000001ae802540 0000000100000000 ffff8102e3117ea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b98800     0  6743      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81025a27bca8 0000000000000082 ffff8104090b6d00 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8102465fc1c0 ffff81040916b5a0 ffff810235deacf0 ffff81040916b900
> Jan 14 18:19:00 4003a6 kernel:  00000001ae802540 0000000100000000 ffff81025a27bea8 000000005a27bc78
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80228fee>] __wake_up+0x38/0x4e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024b31d>] futex_wake+0xef/0x101
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b9a818     0  6756      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102e313fca8 0000000000000082 ffff810400d66dc0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8104449a15c0 ffff8104095a13a0 ffff810270a98400 ffff8104095a1700
> Jan 14 18:19:00 4003a6 kernel:  00000004ae802540 0000000100000000 ffff8102e313fea8 00000000e313fc78
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b9ca30     0  6777      1
> Jan 14 18:19:00 4003a6 kernel:  ffff810263eb1ca8 0000000000000082 0000000000000000 00000000ffffffff
> Jan 14 18:19:00 4003a6 kernel:  ffff810238560348 ffff810409d94040 ffff8102af9f6280 ffff810409d943a0
> Jan 14 18:19:00 4003a6 kernel:  00000006428c8840 ffffffffffffff0d ffffffff80274c48 0000000000000010
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80274c48>] find_vma+0x1b/0x5d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6b9f008     0  6839      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81042bf3dca8 0000000000000082 ffff810427501280 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff810242549c80 ffff8102714c8d30 ffff810279af16a0 ffff8102714c9090
> Jan 14 18:19:00 4003a6 kernel:  00000002ae802540 0000000100000000 ffff81042bf3dea8 000000002bf3dc78
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6ba1870     0  6940      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8104330ffca8 0000000000000082 ffff8104190059c0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8103becac0c0 ffff8103809a36a0 ffff8102374d8c70 ffff8103809a3a00
> Jan 14 18:19:00 4003a6 kernel:  00000006ae802540 0000000100000000 ffff8104330ffea8 00000000330ffc78
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6ba38e8     0  6946      1
> Jan 14 18:19:00 4003a6 kernel:  ffff810400df1ca8 0000000000000082 ffff81042e4673c0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff810237c98dc0 ffff81041c5816a0 ffff810219dcec70 ffff81041c581a00
> Jan 14 18:19:00 4003a6 kernel:  00000001ae802540 0000000100000000 ffff810400df1ea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6ba5dd0     0  6984      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81028b187ca8 0000000000000082 ffff810409592780 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  0000000000000246 ffff8103e0814340 ffff810209483620 ffff8103e08146a0
> Jan 14 18:19:00 4003a6 kernel:  0000000700000002 0000000000000000 0000000000000001 ffff8102428c88a0
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5066>] __down_read+0x84/0x8b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6ba8078     0  7005      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102e3001ca8 0000000000000082 ffff8103388c06c0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff810239dc45c0 ffff8103f51d2af0 ffff81023f97e380 ffff8103f51d2e50
> Jan 14 18:19:00 4003a6 kernel:  00000001ae802540 0000000100000000 ffff8102e3001ea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024b31d>] futex_wake+0xef/0x101
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6baa3b8     0  7108      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102e30d5ca8 0000000000000082 ffff8104274db2c0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff81042ae1a4c0 ffff8104472ef560 ffff810447370a70 ffff8104472ef8c0
> Jan 14 18:19:00 4003a6 kernel:  0000000400000246 ffff8102e30d4000 0000000000000000 00000000ffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bac4f8     0  7154      1
> Jan 14 18:19:00 4003a6 kernel:  ffff810417849ca8 0000000000000082 ffff81030f868700 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8102428e31c0 ffff8102af9f6280 ffff810400d84db0 ffff8102af9f65e0
> Jan 14 18:19:00 4003a6 kernel:  00000001ae802540 0000000100000000 ffff810417849ea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bae9d8     0  7391      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102af97dca8 0000000000000082 ffff810400d80080 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8104064118c0 ffff810294460300 ffff81042087ad70 ffff810294460660
> Jan 14 18:19:00 4003a6 kernel:  00000004ae802540 0000000100000000 ffff8102af97dea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bb0d58     0  7443      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8103f51efca8 0000000000000082 ffff8104078ba080 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff810237c985c0 ffff81041c583460 ffff8103809a36a0 ffff81041c5837c0
> Jan 14 18:19:00 4003a6 kernel:  0000000100000246 ffff8103f51ee000 00000000ef0a0b98 0000000000000000
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8037539b>] plist_add+0x43/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bb2d50     0  7447      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81038093fca8 0000000000000082 ffff81040f040080 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff81024499f780 ffff8102af8b60c0 ffff81041c5816a0 ffff8102af8b6420
> Jan 14 18:19:00 4003a6 kernel:  00000000ae802540 0000000100000000 ffff81038093fea8 000000008093fc78
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80228fee>] __wake_up+0x38/0x4e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024b31d>] futex_wake+0xef/0x101
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bb4d48     0  7459      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102715dbca8 0000000000000082 ffff8103ff512c80 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff810236d36900 ffff81042bf512e0 ffff81023f8a8c70 ffff81042bf51640
> Jan 14 18:19:00 4003a6 kernel:  00000001ae802540 0000000100000000 ffff8102715dbea8 00000000715dbc78
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bb7030     0  7497      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81042bf31ca8 0000000000000082 ffff8104190053c0 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff810270baa500 ffff810428c84cb0 ffff81020c8c89b0 ffff810428c85010
> Jan 14 18:19:00 4003a6 kernel:  00000007ae802540 0000000100000000 ffff81042bf31ea8 ffff81042bf31d20
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c123>] native_flush_tlb_others+0xa1/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bb92c8     0  7545      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102ae8adca8 0000000000000082 ffff81030f869900 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff81023952acc0 ffff81029441c340 ffff8101f6086200 ffff81029441c6a0
> Jan 14 18:19:00 4003a6 kernel:  00000001ae802540 0000000100000000 ffff8102ae8adea8 00000000ae8adc78
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bbbc20     0  7899      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81042aec3ca8 0000000000000082 ffff8103388e9380 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff81030f9aabc0 ffff81044732a440 ffff81025045c140 ffff81044732a7a0
> Jan 14 18:19:00 4003a6 kernel:  00000007ae802540 0000000100000000 ffff81042aec3ea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024b31d>] futex_wake+0xef/0x101
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bbdd08     0  7966      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102af9a7ca8 0000000000000082 ffff8104090d4780 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff81040682aac0 ffff810447370a70 ffff81023f906bf0 ffff810447370dd0
> Jan 14 18:19:00 4003a6 kernel:  00000007ae802540 0000000100000000 ffff8102af9a7ea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000f6bc1700     0  8622      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81025a3ffca8 0000000000000082 ffff81040653cd40 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff810234863540 ffff81025a2575a0 ffff8104079381c0 ffff81025a257900
> Jan 14 18:19:00 4003a6 kernel:  00000000ae802540 0000000100000000 ffff81025a3ffea8 0000000046dcc740
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024b31d>] futex_wake+0xef/0x101
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000080a4910     0 11441      1
> Jan 14 18:19:00 4003a6 kernel:  ffff810263045ca8 0000000000000082 ffff81035305b940 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff81041c484d40 ffff81042e6e2180 ffff8104094eb520 ffff81042e6e24e0
> Jan 14 18:19:00 4003a6 kernel:  00000007000009df 0000000100000000 ffff810263045ea8 00000000275a62c0
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027520b>] find_extend_vma+0x16/0x59
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024aaee>] get_futex_key+0x82/0x14e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS S 00000000080a7120     0 13076      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8102505cdca8 0000000000000082 ffff810400c71340 ffffffff80440d2b
> Jan 14 18:19:00 4003a6 kernel:  ffff8103389e3300 ffff8103f50132a0 ffff81040943a040 ffff8103f5013600
> Jan 14 18:19:00 4003a6 kernel:  00000007ae802540 0000000100000000 ffff8102505cdea8 ffff8102505cdd20
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ae80>] futex_wait+0x1eb/0x318
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b667>] do_sync_readv_writev+0xc0/0x107
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024b31d>] futex_wake+0xef/0x101
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024be66>] do_futex+0x85/0x98c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b4f0d>] compat_do_readv_writev+0x234/0x257
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024ccb0>] compat_sys_futex+0xf8/0x116
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028bf4d>] vfs_write+0x121/0x136
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sspmt_Ramdisk S 0000000000000000     0  4162  24652
> Jan 14 18:19:00 4003a6 kernel:  ffff810263013db8 0000000000000086 ffff8102385fdff0 ffff8101d2d4da28
> Jan 14 18:19:00 4003a6 kernel:  00000000ffd45ff4 ffff810409516180 ffff81023f8a8c70 ffff8104095164e0
> Jan 14 18:19:00 4003a6 kernel:  000000000661e8e8 80000001a11a4065 ffff8101d2d4da28 00003ffffffff000
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80272660>] copy_page_range+0x594/0x644
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c750e>] do_page_fault+0x43b/0x7f6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80254d5a>] compat_sys_wait4+0x21/0xae
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80372870>] __put_user_4+0x20/0x30
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802322d4>] do_fork+0x169/0x236
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223865>] ia32_ptregs_common+0x25/0x4c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sshd          S ffff8101f5cf9d70     0 11708   5520
> Jan 14 18:19:00 4003a6 kernel:  ffff8101f5cf9be8 0000000000000086 ffff8101f5cf9ed8 ffffffff8044764d
> Jan 14 18:19:00 4003a6 kernel:  ffffffff88059c30 ffff8102348c32a0 ffff81020c848d70 ffff8102348c3600
> Jan 14 18:19:00 4003a6 kernel:  0000000200000246 ffffffff8044415c ffff81020c9b68c0 ffffffff80442781
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044764d>] __scm_send+0x113/0x296
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044415c>] skb_queue_tail+0x17/0x3e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80442781>] sock_def_readable+0x10/0x5d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a828b>] unix_stream_sendmsg+0x27e/0x341
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80244398>] prepare_to_wait+0x34/0x5d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a7b10>] unix_stream_recvmsg+0x23c/0x538
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc3a>] dput+0x26/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e784>] sock_aio_read+0x104/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029f52d>] mntput_no_expire+0x1c/0x7f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802922ec>] permission+0xdd/0x10a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5616>] lock_kernel+0x1b/0x37
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028e1e6>] chrdev_open+0x0/0x177
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c01f>] vfs_read+0xbd/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sshd          S 0000000000000200     0 11715  11708
> Jan 14 18:19:00 4003a6 kernel:  ffff8101d2cc9a18 0000000000000082 ffff8102448f17c0 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  ffff8101d2cc9a18 ffff81020c848d70 ffff810263e37360 ffff81020c8490d0
> Jan 14 18:19:00 4003a6 kernel:  0000000400000286 ffffffff8057d902 000000000000eded 0000000000000286
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80228fee>] __wake_up+0x38/0x4e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803bf807>] tty_poll+0x5f/0x6d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 3 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8047e0bf>] __tcp_push_pending_frames+0x722/0x7df
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80473ca4>] tcp_sendmsg+0x947/0xa57
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e668>] sock_aio_write+0xf8/0x110
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80228fee>] __wake_up+0x38/0x4e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b777>] do_sync_write+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236ffb>] current_fs_time+0x1e/0x24
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803be716>] tty_ldisc_deref+0x62/0x75
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029840b>] sys_select+0xc1/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c416>] sys_write+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: tcsh          S 0000000000000000     0 11716  11715
> Jan 14 18:19:00 4003a6 kernel:  ffff8101f4c8bf28 0000000000000086 0000000000000000 ffffffff80273394
> Jan 14 18:19:00 4003a6 kernel:  80000001bc46d065 ffff810238420d30 ffff8104470ac930 ffff810238421090
> Jan 14 18:19:00 4003a6 kernel:  0000000219d60200 0000000100ce5cc2 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023d46a>] sys_rt_sigsuspend+0xb8/0xd9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023db7f>] sys_rt_sigprocmask+0x50/0xcc
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bfb7>] ptregscall_common+0x67/0xb0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: su            S 0000000000000000     0 13581  11716
> Jan 14 18:19:00 4003a6 kernel:  ffff8101d0d29e88 0000000000000082 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff810238da1320 ffffffff805c2da0 ffff810238da1680
> Jan 14 18:19:00 4003a6 kernel:  0000000000000008 0000000100ce6380 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba5c>] sys_rt_sigaction+0x55/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: bash          S 0000000000000000     0 13665  13581
> Jan 14 18:19:00 4003a6 kernel:  ffff8101ee9e1e88 0000000000000082 000000000000000e ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff810239da6bb0 ffff8102ae8092e0 ffff810239da6f10
> Jan 14 18:19:00 4003a6 kernel:  0000000700000008 ffffe20005a9f1b8 ffff8101f5ca2728 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296e08>] vfs_ioctl+0x5c/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sshd          S 0000000000000000     0 25050   5520
> Jan 14 18:19:00 4003a6 kernel:  ffff8102af805be8 0000000000000082 0000000000000000 ffffffff8044764d
> Jan 14 18:19:00 4003a6 kernel:  0000000000000080 ffff8102ae8015e0 ffff8104470ac930 ffff8102ae801940
> Jan 14 18:19:00 4003a6 kernel:  0000000200000246 0000000100e51d5d ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044764d>] __scm_send+0x113/0x296
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80244398>] prepare_to_wait+0x34/0x5d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a7b10>] unix_stream_recvmsg+0x23c/0x538
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc3a>] dput+0x26/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e784>] sock_aio_read+0x104/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029f52d>] mntput_no_expire+0x1c/0x7f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802922ec>] permission+0xdd/0x10a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5616>] lock_kernel+0x1b/0x37
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028e1e6>] chrdev_open+0x0/0x177
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c01f>] vfs_read+0xbd/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sshd          S 0000000000000200     0 25097  25050
> Jan 14 18:19:00 4003a6 kernel:  ffff8101d2d9fa18 0000000000000086 0000000000000000 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  000000005bc3ef05 ffff810239da89b0 ffff8104071dd520 ffff810239da8d10
> Jan 14 18:19:00 4003a6 kernel:  0000000700000000 0000000000000000 ffff8102480d6788 ffff8102428c8840
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80228fee>] __wake_up+0x38/0x4e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803bf807>] tty_poll+0x5f/0x6d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 3 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440e84>] sk_reset_timer+0xf/0x19
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8047e0bf>] __tcp_push_pending_frames+0x722/0x7df
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80473ca4>] tcp_sendmsg+0x947/0xa57
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e668>] sock_aio_write+0xf8/0x110
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b777>] do_sync_write+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236ffb>] current_fs_time+0x1e/0x24
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803be716>] tty_ldisc_deref+0x62/0x75
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029840b>] sys_select+0xc1/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c416>] sys_write+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: tcsh          S 0000000000000000     0 25099  25097
> Jan 14 18:19:00 4003a6 kernel:  ffff81042bee7db8 0000000000000086 0000000000000000 0000000000000001
> Jan 14 18:19:00 4003a6 kernel:  ffff81042bee7d48 ffff810409412d30 ffff8104470ad220 ffff810409413090
> Jan 14 18:19:00 4003a6 kernel:  0000000334862000 0000000100e53227 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802443e5>] add_wait_queue+0x24/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803c408a>] read_chan+0x3ba/0x67c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803bf66c>] tty_ldisc_ref_wait+0xe/0x9b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803c11b7>] tty_read+0x71/0xc7
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: suCopy.sh     S 0000000000000000     0 27863      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81019d05fe88 0000000000000086 0000000000000000 ffff81019d05ff58
> Jan 14 18:19:00 4003a6 kernel:  00000000ffffffff ffff810219c50240 ffff81023f938400 ffff810219c505a0
> Jan 14 18:19:00 4003a6 kernel:  0000000000000008 0000000400eb96f6 0000000000000002 00000000ffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: mount         R  running task        0 27866  27863
> Jan 14 18:19:00 4003a6 kernel:  ffff8101ee92dd78 0000000000000086 ffff8101e3d4dbe0 ffffffff8021c471
> Jan 14 18:19:00 4003a6 kernel:  0000000000000080 ffff81023f938400 ffff81042b585260 ffff81023f938768
> Jan 14 18:19:00 4003a6 kernel:  0000000400000286 ffff8101eadc77d0 0000000000000000 ffff8101fe572b10
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c471>] smp_send_reschedule+0x4a/0x4f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804726fb>] tcp_poll+0x0/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029be76>] shrink_dcache_sb+0xc7/0xe6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022e678>] __cond_resched+0x1c/0x43
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022fed3>] cond_resched_lock+0x40/0x5a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029be6c>] shrink_dcache_sb+0xbd/0xe6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028d194>] do_remount_sb+0x4e/0x151
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802a05ba>] do_remount+0x7f/0xca
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802a1127>] do_mount+0x14b/0x1db
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802a1241>] sys_mount+0x8a/0xcd
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: su            S 0000000000000000     0  4187      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8101badbbe88 0000000000000082 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff81024512ebf0 ffff8104470ac930 ffff81024512ef50
> Jan 14 18:19:00 4003a6 kernel:  0000000200000008 0000000100ed3e86 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba5c>] sys_rt_sigaction+0x55/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: bash          S 0000000000000000     0  4347   4187
> Jan 14 18:19:00 4003a6 kernel:  ffff8101f5c93e88 0000000000000086 ffff8101f5c93e08 ffffffff80226316
> Jan 14 18:19:00 4003a6 kernel:  0000000300000000 ffff81023a1ab2e0 ffff8104091bf260 ffff81023a1ab640
> Jan 14 18:19:00 4003a6 kernel:  00000006000000ff 0000000000000000 0000000000000292 ffffffff804c53ab
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80226316>] __wake_up_common+0x41/0x74
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c53ab>] __down+0x99/0xd8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296e08>] vfs_ioctl+0x5c/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: mount         D ffff8104468fec00     0  4761   4347
> Jan 14 18:19:00 4003a6 kernel:  ffff810420815dc8 0000000000000086 ffff810447045480 ffff810446c94d80
> Jan 14 18:19:00 4003a6 kernel:  ffff810447045480 ffff8104071c12a0 ffff810219cca2c0 ffff8104071c1600
> Jan 14 18:19:00 4003a6 kernel:  0000000520815e68 ffffffff80294ad0 000000090505097a ffff81041c441008
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80294ad0>] __link_path_walk+0xc45/0xd64
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8026436f>] find_lock_page+0x1f/0x7f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4fc1>] __down_write_nested+0x71/0x8b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802a05aa>] do_remount+0x6f/0xca
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802a1127>] do_mount+0x14b/0x1db
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802a1241>] sys_mount+0x8a/0xcd
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: su            S 0000000000000000     0 13449  13665
> Jan 14 18:19:00 4003a6 kernel:  ffff810271449e88 0000000000000086 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff81042aefa4c0 ffff81023f938400 ffff81042aefa820
> Jan 14 18:19:00 4003a6 kernel:  0000000600000008 ffffe20009d40cf0 ffffe200088c9298 ffffffff8027096a
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027096a>] do_wp_page+0x1ce/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba5c>] sys_rt_sigaction+0x55/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: tcsh          S 00007fff50486b90     0 13531  13449
> Jan 14 18:19:00 4003a6 kernel:  ffff8102af9b3f28 0000000000000086 ffff81027151aa28 ffffffff80273394
> Jan 14 18:19:00 4003a6 kernel:  800000019da76065 ffff8102714ae1c0 ffff81020c9af560 ffff8102714ae520
> Jan 14 18:19:00 4003a6 kernel:  0000000071511600 0000000000000cd8 ffff81025a2d5018 ffffffff802300af
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c750e>] do_page_fault+0x43b/0x7f6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023d46a>] sys_rt_sigsuspend+0xb8/0xd9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023db7f>] sys_rt_sigprocmask+0x50/0xcc
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bfb7>] ptregscall_common+0x67/0xb0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: patch.sh      S 0000000000000000     0   338  13531
> Jan 14 18:19:00 4003a6 kernel:  ffff8101eb13fe88 0000000000000082 ffff8101eb13fe28 ffffffff8029b76a
> Jan 14 18:19:00 4003a6 kernel:  ffff8101eb13fe18 ffff81019ac74440 ffff8102e2d6e440 ffff81019ac747a0
> Jan 14 18:19:00 4003a6 kernel:  000000020f1a01c0 00000352eb13fe48 0000000000000001 ffff81040f1a01e0
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029b76a>] d_lookup+0x1b/0x3c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba77>] sys_rt_sigaction+0x70/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: suCopy.sh     S 0000000000000000     0   856    338
> Jan 14 18:19:00 4003a6 kernel:  ffff810238429e88 0000000000000086 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff81019c1249f0 ffff81040814e080 ffff81019c124d50
> Jan 14 18:19:00 4003a6 kernel:  0000000000000008 ffffe200059ecfd8 ffff8101bad4e620 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba77>] sys_rt_sigaction+0x70/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: su            S 0000000000000000     0 12584   5142
> Jan 14 18:19:00 4003a6 kernel:  ffff81019c1a5e88 0000000000000082 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff81019d86a440 ffff81042bf0f2a0 ffff81019d86a7a0
> Jan 14 18:19:00 4003a6 kernel:  0000000400000008 ffffe20008ee4e10 ffff8104095364a8 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba5c>] sys_rt_sigaction+0x55/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: ksh           S 0000000000000000     0 12795  12584
> Jan 14 18:19:00 4003a6 kernel:  ffff81026307fe88 0000000000000086 00000000007eaf40 ffff81026307ff58
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff81044725b2a0 ffff8102e3196a30 ffff81044725b600
> Jan 14 18:19:00 4003a6 kernel:  0000000500000012 00000004e3196a30 0000000000000011 0000000000000012
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba77>] sys_rt_sigaction+0x70/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: bash          S 0000000000000000     0 12935  12795
> Jan 14 18:19:00 4003a6 kernel:  ffff8103f0575e88 0000000000000082 000000000000000e ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff8102e3196a30 ffff81042b5a52a0 ffff8102e3196d90
> Jan 14 18:19:00 4003a6 kernel:  0000000500000008 ffffe200059d1470 ffff81026316a600 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296e08>] vfs_ioctl+0x5c/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sonsym_amd_li S 0000000000000000     0 19412  30433
> Jan 14 18:19:00 4003a6 kernel:  ffff8103f5161b38 0000000000000086 ffff8103f5161b00 ffff8103f5161ac8
> Jan 14 18:19:00 4003a6 kernel:  ffff8104079381f8 ffff81042bfd6a70 ffff81023f938400 ffff81042bfd6dd0
> Jan 14 18:19:00 4003a6 kernel:  000000048071b000 0000000000000046 ffff81000100f080 0000000000000246
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029785f>] do_sys_poll+0x255/0x324
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 3 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80226c12>] dequeue_entity+0x1b/0xd3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80226a88>] __dequeue_entity+0x25/0x69
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020a8c2>] __switch_to+0xf0/0x2a1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c3df4>] thread_return+0x3d/0xa2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440e4e>] lock_sock_nested+0xa0/0xab
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80472bae>] tcp_ioctl+0x118/0x124
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296dcd>] vfs_ioctl+0x21/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029705f>] do_vfs_ioctl+0x248/0x261
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802970c9>] sys_ioctl+0x51/0x70
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b739c>] do_ioctl32_pointer+0xb/0xd
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b7290>] compat_sys_ioctl+0x2f9/0x33e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298769>] sys_poll+0x35/0x84
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: oracle        S ffff81042bf61df8     0 19517      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81042bf61bc8 0000000000000082 ffff810235ded5a0 ffffffff80449daa
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff810270b6ed70 ffff81024512e300 ffff810270b6f0d0
> Jan 14 18:19:00 4003a6 kernel:  00000001071058c0 ffffffff80475aa9 00000000e9536322 ffffffff80478723
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80449daa>] dev_hard_start_xmit+0x1d2/0x244
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80475aa9>] tcp_cong_avoid+0xe/0x1d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80478723>] tcp_ack+0x1742/0x190b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804417f0>] sk_wait_data+0x81/0xc1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80474118>] tcp_recvmsg+0x364/0x7b6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804406eb>] sock_common_recvmsg+0x30/0x45
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e784>] sock_aio_read+0x104/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c01f>] vfs_read+0xbd/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: Xvnc          S 0000000000000000     0 19590      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8101eb0dfa18 0000000000000086 0000000000000000 ffffffff80916700
> Jan 14 18:19:00 4003a6 kernel:  00000000ffffffff ffff81019d5badb0 ffffffff805c2da0 ffff81019d5bb110
> Jan 14 18:19:00 4003a6 kernel:  00000000472395c8 0000000101031882 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a6584>] unix_poll+0x0/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 5 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8047445a>] tcp_recvmsg+0x6a6/0x7b6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804406eb>] sock_common_recvmsg+0x30/0x45
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246223>] enqueue_hrtimer+0x5c/0x63
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802466f6>] hrtimer_start+0xf9/0x10b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020b703>] do_notify_resume+0x708/0x7e6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802461be>] __remove_hrtimer+0x4b/0x54
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bb37>] sys_rt_sigreturn+0x342/0x37b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802984a4>] sys_select+0x15a/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: mwm           S 0000000000000000     0 19807      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8101bad47b38 0000000000000082 0000000000000001 ffff8101bad47ac8
> Jan 14 18:19:00 4003a6 kernel:  ffff81019d5bade8 ffff810239cc9220 ffff8104089cccf0 ffff810239cc9580
> Jan 14 18:19:00 4003a6 kernel:  00000006c77a0000 ffff81019d5bade8 ffff81019d5badb0 ffff8102480b6700
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80227f48>] enqueue_task_fair+0x24/0x3a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802264f3>] enqueue_task+0x4d/0x58
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297597>] free_poll_entry+0x11/0x1a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802443e5>] add_wait_queue+0x24/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029785f>] do_sys_poll+0x255/0x324
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044104c>] sock_alloc_send_skb+0x77/0x1d2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a7cf7>] unix_stream_recvmsg+0x423/0x538
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804460f1>] memcpy_fromiovec+0x36/0x66
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044415c>] skb_queue_tail+0x17/0x3e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80442781>] sock_def_readable+0x10/0x5d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a828b>] unix_stream_sendmsg+0x27e/0x341
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e668>] sock_aio_write+0xf8/0x110
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8026efc7>] zone_statistics+0x3f/0x93
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b777>] do_sync_write+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a7ec4>] unix_ioctl+0xb8/0xc1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e4e0>] sock_ioctl+0x1da/0x1fe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296dcd>] vfs_ioctl+0x21/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029705f>] do_vfs_ioctl+0x248/0x261
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298769>] sys_poll+0x35/0x84
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cat           ? ffff81019ad1f650     0 19812  19412
> Jan 14 18:19:00 4003a6 kernel:  ffff81019c07bf08 0000000000000046 ffff81042afb3c88 0000000000000056
> Jan 14 18:19:00 4003a6 kernel:  0000000000000056 ffff81019ad1f660 ffff81041c580db0 ffff81019ad1f9c0
> Jan 14 18:19:00 4003a6 kernel:  0000000000004d64 0000000000000000 0000000000000006 00000000000000c9
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a033>] filp_close+0x5d/0x65
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236202>] do_exit+0x676/0x67a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236275>] do_group_exit+0x6f/0x8a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sqlplus       S ffff810212c8cd70     0 24933  12935
> Jan 14 18:19:00 4003a6 kernel:  ffff81026305ddb8 0000000000000086 0000000000000296 0000000000000001
> Jan 14 18:19:00 4003a6 kernel:  ffff81026305dd48 ffff8103f51fcaf0 ffff8104473a0300 ffff8103f51fce50
> Jan 14 18:19:00 4003a6 kernel:  00000006e2ca0800 0000000000000000 ffff810237d7f480 ffff8102e2ca0800
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803c3b39>] n_tty_receive_buf+0xdcc/0xe24
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802443e5>] add_wait_queue+0x24/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803c408a>] read_chan+0x3ba/0x67c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803bf66c>] tty_ldisc_ref_wait+0xe/0x9b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803c11b7>] tty_read+0x71/0xc7
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: oracle        S ffff810250589f50     0 25218  24933
> Jan 14 18:19:00 4003a6 kernel:  ffff810250589cf8 0000000000000086 ffff8103f51fcaf0 ffff8102480c6700
> Jan 14 18:19:00 4003a6 kernel:  ffff810250589c68 ffff8103e08b7260 ffff8103f51fcaf0 ffff8103e08b75c0
> Jan 14 18:19:00 4003a6 kernel:  0000000650589c88 ffffffff802264f3 00000000ffffffff ffff8102480c6700
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802264f3>] enqueue_task+0x4d/0x58
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6bd>] try_to_wake_up+0x1ae/0x1bf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ae68>] del_timer_sync+0xc/0x16
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291587>] pipe_wait+0x66/0x8d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c44f2>] mutex_lock+0xd/0x1e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291da7>] pipe_read+0x31e/0x397
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: oracle        S ffffffff805d78e8     0  4718      1
> Jan 14 18:19:00 4003a6 kernel:  ffff810270afdcc8 0000000000000082 ffff810270afdde4 0000000180227efe
> Jan 14 18:19:00 4003a6 kernel:  ffff810270afde20 ffff8102e2d6e440 ffff810270a5f520 ffff8102e2d6e7a0
> Jan 14 18:19:00 4003a6 kernel:  0000000470afdc58 ffffffff80227f48 0000000000000292 ffff810236c24300
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80227f48>] enqueue_task_fair+0x24/0x3a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8033ebaf>] sys_semtimedop+0x515/0x604
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236ffb>] current_fs_time+0x1e/0x24
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269c6b>] set_page_dirty+0x3a/0x82
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8026ab48>] set_page_dirty_balance+0x10/0x47
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270789>] __do_fault+0x3c4/0x3d7
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023bbfa>] lock_task_sighand+0x2b/0x50
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023e7a4>] getrusage+0x1cb/0x1e6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ffc8>] sys_times+0xe7/0xfa
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: xterm         R  running task        0  8017      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81019d15df70 0000000000000086 ffffffff8043e4e0 ffff810246184d18
> Jan 14 18:19:00 4003a6 kernel:  ffff810407912740 ffff81021000ab30 ffff8101974d2180 ffff81021000ae98
> Jan 14 18:19:00 4003a6 kernel:  0000000480296dcd ffff810407912740 00007fffb8272c8c 0000000000000003
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e4e0>] sock_ioctl+0x1da/0x1fe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029705f>] do_vfs_ioctl+0x248/0x261
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c080>] vfs_read+0x11e/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802970c9>] sys_ioctl+0x51/0x70
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bd05>] sysret_careful+0xd/0x10
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: su            S 0000000000000000     0  8044   8017
> Jan 14 18:19:00 4003a6 kernel:  ffff81042aff7e88 0000000000000082 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff810279af0db0 ffff810234887320 ffff810279af1110
> Jan 14 18:19:00 4003a6 kernel:  0000000500000008 ffffe200085fd3b0 ffff8102af897b30 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba5c>] sys_rt_sigaction+0x55/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: tcsh          S 00007fff805ee730     0  8050   8044
> Jan 14 18:19:00 4003a6 kernel:  ffff81025057ff28 0000000000000082 ffff810406436608 ffffffff80273394
> Jan 14 18:19:00 4003a6 kernel:  8000000196bc6065 ffff810409d0acb0 ffff8104473a0300 ffff810409d0b010
> Jan 14 18:19:00 4003a6 kernel:  000000061c54b840 0000000000000cd8 ffff810408985018 ffffffff802300af
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c750e>] do_page_fault+0x43b/0x7f6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023d46a>] sys_rt_sigsuspend+0xb8/0xd9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023db7f>] sys_rt_sigprocmask+0x50/0xcc
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bfb7>] ptregscall_common+0x67/0xb0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: su            S 0000000000000000     0 10014   9169
> Jan 14 18:19:00 4003a6 kernel:  ffff8103f0587e88 0000000000000086 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff81027157c440 ffff81044510ad30 ffff81027157c7a0
> Jan 14 18:19:00 4003a6 kernel:  0000000600000008 ffffe200089fb1f0 ffffe2000a1a9468 ffffffff8027096a
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027096a>] do_wp_page+0x1ce/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba5c>] sys_rt_sigaction+0x55/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: bash          S 0000000000000000     0 10184  10014
> Jan 14 18:19:00 4003a6 kernel:  ffff8102af849e88 0000000000000082 ffff8102af849e08 ffffffff80226316
> Jan 14 18:19:00 4003a6 kernel:  0000000300000000 ffff8102e2d50340 ffff8102145cac30 ffff8102e2d506a0
> Jan 14 18:19:00 4003a6 kernel:  00000003000000ff 0000000000000000 0000000000000292 ffffffff804c53ab
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80226316>] __wake_up_common+0x41/0x74
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c53ab>] __down+0x99/0xd8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296e08>] vfs_ioctl+0x5c/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: dhcpd         S 0000000000000100     0 10567      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8101d0d41a18 0000000000000082 0000000000000000 ffffffff80916700
> Jan 14 18:19:00 4003a6 kernel:  ffff81024595ee80 ffff81019d968bf0 ffff81040895f460 ffff81019d968f50
> Jan 14 18:19:00 4003a6 kernel:  00000007480ca998 0000000101030c60 ffffffffffffffef ffff810237dcd440
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80446d14>] datagram_poll+0x0/0xc0
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80446d2c>] datagram_poll+0x18/0xc0
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802443e5>] add_wait_queue+0x24/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 3 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80446de0>] skb_free_datagram+0xc/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804848f6>] raw_recvmsg+0x146/0x15f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804406eb>] sock_common_recvmsg+0x30/0x45
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043ed11>] sock_recvmsg+0xf0/0x10f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043fa5b>] move_addr_to_user+0x5d/0x78
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043fdab>] sys_recvfrom+0xe1/0x120
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80446d14>] datagram_poll+0x0/0xc0
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029840b>] sys_select+0xc1/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sonsym_amd_li S 0000000000000000     0 11575  30433
> Jan 14 18:19:00 4003a6 kernel:  ffff810263073b38 0000000000000086 0000000000000001 ffff810263073ac8
> Jan 14 18:19:00 4003a6 kernel:  ffff8104079381f8 ffff8102e2d6ed30 ffff8104091bf260 ffff8102e2d6f090
> Jan 14 18:19:00 4003a6 kernel:  00000006806fb000 ffff8104079381f8 ffff8104079381c0 ffff810001011700
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804726fb>] tcp_poll+0x0/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029785f>] do_sys_poll+0x255/0x324
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 3 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80226c12>] dequeue_entity+0x1b/0xd3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80226a88>] __dequeue_entity+0x25/0x69
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80227efe>] enqueue_entity+0x118/0x13e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80227f48>] enqueue_task_fair+0x24/0x3a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802264f3>] enqueue_task+0x4d/0x58
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440e4e>] lock_sock_nested+0xa0/0xab
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80472bae>] tcp_ioctl+0x118/0x124
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296dcd>] vfs_ioctl+0x21/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029705f>] do_vfs_ioctl+0x248/0x261
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802970c9>] sys_ioctl+0x51/0x70
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b739c>] do_ioctl32_pointer+0xb/0xd
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b7290>] compat_sys_ioctl+0x2f9/0x33e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298769>] sys_poll+0x35/0x84
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: oracle        S ffff810263e6bdf8     0 11632      1
> Jan 14 18:19:00 4003a6 kernel:  ffff810263e6bbc8 0000000000000082 ffff81028caae6c0 ffff81028caae6c0
> Jan 14 18:19:00 4003a6 kernel:  ffffffff80475046 ffff81040f10cd30 ffff8103389c4af0 ffff81040f10d090
> Jan 14 18:19:00 4003a6 kernel:  00000006fb562ac4 ffffffff80478723 0000000000000000 fb562ac400000001
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80475046>] tcp_rtt_estimator+0xcf/0x113
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80478723>] tcp_ack+0x1742/0x190b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804417f0>] sk_wait_data+0x81/0xc1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80474118>] tcp_recvmsg+0x364/0x7b6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804406eb>] sock_common_recvmsg+0x30/0x45
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e784>] sock_aio_read+0x104/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269c6b>] set_page_dirty+0x3a/0x82
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c01f>] vfs_read+0xbd/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: Xvnc          S 0000000000000040     0 11660      1
> Jan 14 18:19:00 4003a6 kernel:  ffff81028cafda18 0000000000000086 ffff81028cafd9e0 ffffffff80916700
> Jan 14 18:19:00 4003a6 kernel:  ffff8104424300c0 ffff81042be94b30 ffff8102428dd520 ffff81042be94e90
> Jan 14 18:19:00 4003a6 kernel:  00000006471696e8 000000010102f0e0 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804726fb>] tcp_poll+0x0/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 5 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8047445a>] tcp_recvmsg+0x6a6/0x7b6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804406eb>] sock_common_recvmsg+0x30/0x45
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246223>] enqueue_hrtimer+0x5c/0x63
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802466f6>] hrtimer_start+0xf9/0x10b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020b703>] do_notify_resume+0x708/0x7e6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802461be>] __remove_hrtimer+0x4b/0x54
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bb37>] sys_rt_sigreturn+0x342/0x37b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802984a4>] sys_select+0x15a/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: mwm           S 0000000000000000     0 11965      1
> Jan 14 18:19:00 4003a6 kernel:  ffff8101974d5b38 0000000000000086 0000000000000001 ffff8101974d5ac8
> Jan 14 18:19:00 4003a6 kernel:  ffff81042be94b68 ffff810219c8eb70 ffff81042be94b30 ffff810219c8eed0
> Jan 14 18:19:00 4003a6 kernel:  000000038072b000 ffff81042be94b68 ffff81042be94b30 ffff810001041700
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80227f48>] enqueue_task_fair+0x24/0x3a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802264f3>] enqueue_task+0x4d/0x58
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297597>] free_poll_entry+0x11/0x1a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802443e5>] add_wait_queue+0x24/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029785f>] do_sys_poll+0x255/0x324
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044104c>] sock_alloc_send_skb+0x77/0x1d2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a7cf7>] unix_stream_recvmsg+0x423/0x538
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804460f1>] memcpy_fromiovec+0x36/0x66
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044415c>] skb_queue_tail+0x17/0x3e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80442781>] sock_def_readable+0x10/0x5d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a828b>] unix_stream_sendmsg+0x27e/0x341
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e668>] sock_aio_write+0xf8/0x110
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8026efc7>] zone_statistics+0x3f/0x93
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b777>] do_sync_write+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a7ec4>] unix_ioctl+0xb8/0xc1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e4e0>] sock_ioctl+0x1da/0x1fe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80296dcd>] vfs_ioctl+0x21/0x6b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029705f>] do_vfs_ioctl+0x248/0x261
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298769>] sys_poll+0x35/0x84
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cat           ? ffff8102e3197310     0 12161  11575
> Jan 14 18:19:00 4003a6 kernel:  ffff8102af809f08 0000000000000046 ffff810270afeec8 0000000000000056
> Jan 14 18:19:00 4003a6 kernel:  0000000000000056 ffff8102e3197320 ffff810263e36a70 ffff8102e3197680
> Jan 14 18:19:00 4003a6 kernel:  0000000000002f81 0000000000000000 0000000000000006 0000000000000065
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a033>] filp_close+0x5d/0x65
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236202>] do_exit+0x676/0x67a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236275>] do_group_exit+0x6f/0x8a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: tail          S 00007f0a83a07c60     0 14348  10184
> Jan 14 18:19:00 4003a6 kernel:  ffff810279ac5eb8 0000000000000082 00ff81042af06bb0 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  ffff810200000000 ffff81042af06bb0 ffff8102471ca970 ffff81042af06f10
> Jan 14 18:19:00 4003a6 kernel:  0000000579ac5e58 ffffffff8036fd3b ffff810279ac5ed8 ffff8102480b36a8
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8036fd3b>] rb_insert_color+0x61/0xda
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246223>] enqueue_hrtimer+0x5c/0x63
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802466f6>] hrtimer_start+0xf9/0x10b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c47c0>] do_nanosleep+0x55/0x95
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024675c>] hrtimer_nanosleep+0x54/0xd2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802462c1>] hrtimer_wakeup+0x0/0x21
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246826>] sys_nanosleep+0x4c/0x62
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sshd          S ffff8102504c5d70     0 14762   5520
> Jan 14 18:19:00 4003a6 kernel:  ffff8102504c5be8 0000000000000082 0000000000000282 ffffffff80244411
> Jan 14 18:19:00 4003a6 kernel:  ffff8102504c5bd0 ffff81042bfd7360 ffff81042e5decf0 ffff81042bfd76c0
> Jan 14 18:19:00 4003a6 kernel:  00000007504c5c08 ffffffff802975c9 0000000000000202 ffffffff80446ccc
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80244411>] remove_wait_queue+0x1c/0x2f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802975c9>] poll_freewait+0x29/0x6a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80446ccc>] __skb_recv_datagram+0x1f2/0x219
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80446300>] memcpy_toiovec+0x36/0x66
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80244398>] prepare_to_wait+0x34/0x5d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c44a8>] mutex_unlock+0x3/0xb
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a7b10>] unix_stream_recvmsg+0x23c/0x538
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8026efc7>] zone_statistics+0x3f/0x93
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e784>] sock_aio_read+0x104/0x11c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c01f>] vfs_read+0xbd/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: su            S 0000000000000000     0 14765   8050
> Jan 14 18:19:00 4003a6 kernel:  ffff8102e3195e88 0000000000000082 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff8104070f8d70 ffff8102374d9560 ffff8104070f90d0
> Jan 14 18:19:00 4003a6 kernel:  0000000500000008 ffffe2000ccd2ef8 ffff8102715a8508 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba5c>] sys_rt_sigaction+0x55/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sshd          S 0000000000000800     0 14807  14762
> Jan 14 18:19:00 4003a6 kernel:  ffff81025a249a18 0000000000000082 ffff81025a249b74 ffffffff8049e4f0
> Jan 14 18:19:00 4003a6 kernel:  ffff81025a2499d0 ffff81042aefb6a0 ffff810245838040 ffff81042aefba00
> Jan 14 18:19:00 4003a6 kernel:  00000006480c4108 ffff810445c29800 ffff81025a249b68 0000000000000292
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8049e4f0>] xfrm_policy_lookup+0x186/0x1cc
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80454e1a>] flow_cache_lookup+0x2f0/0x343
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8049e36a>] xfrm_policy_lookup+0x0/0x1cc
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c4212>] schedule_timeout+0x1e/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802443e5>] add_wait_queue+0x24/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802443e5>] add_wait_queue+0x24/0x34
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 last message repeated 3 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044677f>] skb_copy_datagram_iovec+0x4b/0x215
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8047445a>] tcp_recvmsg+0x6a6/0x7b6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8026efc7>] zone_statistics+0x3f/0x93
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e4f6>] sock_ioctl+0x1f0/0x1fe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029840b>] sys_select+0xc1/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804a8de3>] unix_shutdown+0xc4/0x130
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: tcsh          S ffff810001005e60     0 14812  14807
> Jan 14 18:19:00 4003a6 kernel:  ffff81028cbe7cf8 0000000000000086 ffff81028cbe7cc0 0000000001192d38
> Jan 14 18:19:00 4003a6 kernel:  ffff81000001fdf8 ffff8104209ac480 ffff81023f8a8c70 ffff8104209ac7e0
> Jan 14 18:19:00 4003a6 kernel:  00000000ffffffff 0000000101031e5a ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291587>] pipe_wait+0x66/0x8d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c44f2>] mutex_lock+0xd/0x1e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291da7>] pipe_read+0x31e/0x397
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: ssrag_RaidAge S 0000000000000000     0 14849  24961
> Jan 14 18:19:00 4003a6 kernel:  ffff8101b7c47db8 0000000000000082 0000000000000000 ffff81019d88f018
> Jan 14 18:19:00 4003a6 kernel:  00000000ffc032a4 ffff810238d532a0 ffff8104470ac930 ffff810238d53600
> Jan 14 18:19:00 4003a6 kernel:  0000000205a75f58 000000010102dfd9 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8027096a>] do_wp_page+0x1ce/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80254d5a>] compat_sys_wait4+0x21/0xae
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80372870>] __put_user_4+0x20/0x30
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c750e>] do_page_fault+0x43b/0x7f6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803702f3>] __up_read+0x19/0x7f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5709>] error_exit+0x0/0x51
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: ssrag_RaidMan S 0000000000000000     0 14852  14849
> Jan 14 18:19:00 4003a6 kernel:  ffff8101ee95be88 0000000000000086 ffff8101ee95be28 ffffffff8029b76a
> Jan 14 18:19:00 4003a6 kernel:  ffff8101ee95be18 ffff8101974d2180 ffff8102af9689b0 ffff8101974d24e0
> Jan 14 18:19:00 4003a6 kernel:  000000042a0c15c0 00003df90cd02ad8 0000000000000001 ffff81042a0c15e0
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029b76a>] d_lookup+0x1b/0x3c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023ba77>] sys_rt_sigaction+0x70/0x8c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: bash          R  running task        0 14933  14765
> Jan 14 18:19:00 4003a6 kernel:  ffff810271473bc8 ffffffff80370f05 3830333730663035 3338333033333337
> Jan 14 18:19:00 4003a6 kernel:  ffff810271473b00 ffff810200000000 ffff8102471bd2a0 ffffffff8020bc9b
> Jan 14 18:19:00 4003a6 kernel:  ffff813134373635 ffffffff805642ac ffffffff805642ac ffffffff805642ac
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80370f05>] number+0x106/0x1f9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80370f05>] number+0x106/0x1f9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80232f85>] __call_console_drivers+0x59/0x67
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80233322>] release_console_sem+0x183/0x19c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80251799>] kallsyms_lookup+0x49/0x80
> Jan 14 18:19:00 4003a6 last message repeated 2 times
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020d60b>] printk_address+0xea/0xf6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022719a>] __update_rq_clock+0x1a/0xd0
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022719a>] __update_rq_clock+0x1a/0xd0
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020d6e9>] print_trace_address+0x11/0x20
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80242449>] __kernel_text_address+0x1a/0x26
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020d17c>] dump_trace+0x2c8/0x317
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020d20b>] show_trace+0x40/0x57
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020d302>] _show_stack+0xe0/0xef
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022f044>] show_state_filter+0x50/0x9a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802c636f>] proc_reg_open+0x0/0x69
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803cffba>] __handle_sysrq+0x98/0x128
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802cbf5c>] write_sysrq_trigger+0x28/0x30
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802c6120>] proc_reg_write+0x76/0x91
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028bed9>] vfs_write+0xad/0x136
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c416>] sys_write+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sleep         S 0000000000000001     0 15242  30655
> Jan 14 18:19:00 4003a6 kernel:  ffff8103f518beb8 0000000000000086 00ff8104453055e0 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  ffff810000000000 ffff8104453055e0 ffff8101975a2af0 ffff810445305940
> Jan 14 18:19:00 4003a6 kernel:  00000000f518be58 ffffffff80246894 ffff8103f518bed8 ffff81000100e6a8
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246894>] ktime_get_ts+0x17/0x48
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246223>] enqueue_hrtimer+0x5c/0x63
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802466f6>] hrtimer_start+0xf9/0x10b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c47c0>] do_nanosleep+0x55/0x95
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024675c>] hrtimer_nanosleep+0x54/0xd2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802462c1>] hrtimer_wakeup+0x0/0x21
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246826>] sys_nanosleep+0x4c/0x62
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sleep         S 0000000000000000     0 15263  30654
> Jan 14 18:19:00 4003a6 kernel:  ffff81019c1d3eb8 0000000000000086 0000000000000000 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  ffff810000000000 ffff81019745ca30 ffff8104470ac930 ffff81019745cd90
> Jan 14 18:19:00 4003a6 kernel:  000000029c1d3e58 000000010102f4f7 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c47c0>] do_nanosleep+0x55/0x95
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024675c>] hrtimer_nanosleep+0x54/0xd2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802462c1>] hrtimer_wakeup+0x0/0x21
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246826>] sys_nanosleep+0x4c/0x62
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sleep         S 0000000000000001     0 15320  30574
> Jan 14 18:19:00 4003a6 kernel:  ffff81019d847eb8 0000000000000086 00ff81020c856480 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  ffff810000000000 ffff81020c856480 ffff8101ef5ea180 ffff81020c8567e0
> Jan 14 18:19:00 4003a6 kernel:  000000039d847e58 ffffffff8036fd8c ffff81019d847ed8 ffff81000103e6a8
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8036fd8c>] rb_insert_color+0xb2/0xda
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246223>] enqueue_hrtimer+0x5c/0x63
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802466f6>] hrtimer_start+0xf9/0x10b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c47c0>] do_nanosleep+0x55/0x95
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024675c>] hrtimer_nanosleep+0x54/0xd2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802462c1>] hrtimer_wakeup+0x0/0x21
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246826>] sys_nanosleep+0x4c/0x62
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: S95make_dhcp  S ffff81028b137f50     0 15568  24959
> Jan 14 18:19:00 4003a6 kernel:  ffff81028b137cf8 0000000000000086 ffff81000001fdf0 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  ffff81000001fdf8 ffff810263fb0970 ffff810239d88180 ffff810263fb0cd0
> Jan 14 18:19:00 4003a6 kernel:  00000003ffffffff 0000000000000001 00000000ffffffff ffff810236460930
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291587>] pipe_wait+0x66/0x8d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c44f2>] mutex_lock+0xd/0x1e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291da7>] pipe_read+0x31e/0x397
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023b8f3>] do_sigaction+0x6c/0x146
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: service       S 0000000000000000     0 15864  14852
> Jan 14 18:19:00 4003a6 kernel:  ffff81027145be88 0000000000000086 000000000000000e ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff810409f60c70 ffff8102452ba400 ffff810409f60fd0
> Jan 14 18:19:00 4003a6 kernel:  0000000100000008 ffffe2000596a3f0 ffff8101bad6e5c8 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: nfs           R  running task        0 15894  15864
> Jan 14 18:19:00 4003a6 kernel:  ffff8101ef8dff70 0000000000000082 ffffffff802300af 00007f7c1645e780
> Jan 14 18:19:00 4003a6 kernel:  0000000000000286 ffff8102394cd4a0 ffff8102e2d6f620 ffff8102394cd808
> Jan 14 18:19:00 4003a6 kernel:  00000003802322d4 ffff8101ef8dff58 00007fff1e474120 0000000000000004
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291eb4>] do_pipe+0x94/0xf4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023be78>] recalc_sigpending+0xe/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bd05>] sysret_careful+0xd/0x10
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sleep         S 0000000000000001     0 15966    856
> Jan 14 18:19:00 4003a6 kernel:  ffff8101bacc5eb8 0000000000000082 00ff810239c603c0 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  ffff810000000000 ffff810239c603c0 ffff81033894e040 ffff810239c60720
> Jan 14 18:19:00 4003a6 kernel:  00000001bacc5e58 ffffffff80246894 ffff8101bacc5ed8 ffff81000101e6a8
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246894>] ktime_get_ts+0x17/0x48
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246223>] enqueue_hrtimer+0x5c/0x63
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802466f6>] hrtimer_start+0xf9/0x10b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c47c0>] do_nanosleep+0x55/0x95
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024675c>] hrtimer_nanosleep+0x54/0xd2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802462c1>] hrtimer_wakeup+0x0/0x21
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80246826>] sys_nanosleep+0x4c/0x62
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: ssrdm_rsync_f S ffff81019d02df50     0 15998   4162
> Jan 14 18:19:00 4003a6 kernel:  ffff81019d02dcf8 0000000000000082 ffff810248004c70 0000000001c10600
> Jan 14 18:19:00 4003a6 kernel:  ffff810248004c78 ffff8102364644c0 ffff81028b1ac2c0 ffff810236464820
> Jan 14 18:19:00 4003a6 kernel:  00000006ffffffff 0000000000000001 00000000ffffffff ffff81025a2ef700
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291587>] pipe_wait+0x66/0x8d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c44f2>] mutex_lock+0xd/0x1e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291da7>] pipe_read+0x31e/0x397
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022fc3b>] check_preempt_wakeup+0x90/0xcc
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023b8f3>] do_sigaction+0x6c/0x146
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: hacheck       S ffff810417977f50     0 16029  31056
> Jan 14 18:19:00 4003a6 kernel:  ffff810417977cf8 0000000000000082 0000000000000000 0000000000438ca2
> Jan 14 18:19:00 4003a6 kernel:  ffff810248004c78 ffff81042af5b260 ffff8102e318ebf0 ffff81042af5b5c0
> Jan 14 18:19:00 4003a6 kernel:  00000006ffffffff 0000000101031c95 ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291587>] pipe_wait+0x66/0x8d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c44f2>] mutex_lock+0xd/0x1e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291da7>] pipe_read+0x31e/0x397
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rsync         S 00000000ffffffea     0 16046  32191
> Jan 14 18:19:00 4003a6 kernel:  ffff8102505dba18 0000000000000082 ffff8102448f17c0 ffffffff80916700
> Jan 14 18:19:00 4003a6 kernel:  ffff8102505e8440 ffff8102ae808100 ffff8102ae84f420 ffff8102ae808460
> Jan 14 18:19:00 4003a6 kernel:  000000078b1ac2c0 ffffffff80269904 0000000000000005 ffff8102ae808100
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80449daa>] dev_hard_start_xmit+0x1d2/0x244
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8044c258>] dev_queue_xmit+0x236/0x262
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8046d0f9>] ip_queue_xmit+0x2c2/0x315
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803475fd>] dummy_socket_recvmsg+0x0/0x3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029b6f3>] __d_lookup+0xa8/0x104
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440e84>] sk_reset_timer+0xf/0x19
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8047e0bf>] __tcp_push_pending_frames+0x722/0x7df
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80473ca4>] tcp_sendmsg+0x947/0xa57
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297ddb>] do_select+0x457/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6bd>] try_to_wake_up+0x1ae/0x1bf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023bce1>] __sigqueue_alloc+0x3b/0x69
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80226a88>] __dequeue_entity+0x25/0x69
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80370026>] rb_erase+0x1c8/0x2aa
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020a8c2>] __switch_to+0xf0/0x2a1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80244411>] remove_wait_queue+0x1c/0x2f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802359c8>] do_wait+0x914/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802984a4>] sys_select+0x15a/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: hacheck       S ffff81020089df50     0 16052  26763
> Jan 14 18:19:00 4003a6 kernel:  ffff81020089dcf8 0000000000000082 ffff810248004c70 0000000001b1c3c3
> Jan 14 18:19:00 4003a6 kernel:  ffff810248004c78 ffff81023a9ccaf0 ffff81019d93c100 ffff81023a9cce50
> Jan 14 18:19:00 4003a6 kernel:  00000004ffffffff 0000000000000001 00000000ffffffff ffff8102e2cfa7c8
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291587>] pipe_wait+0x66/0x8d
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c44f2>] mutex_lock+0xd/0x1e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291da7>] pipe_read+0x31e/0x397
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80269904>] __alloc_pages+0x78/0x30c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028b883>] do_sync_read+0xc9/0x10c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023b8f3>] do_sigaction+0x6c/0x146
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c00c>] vfs_read+0xaa/0x132
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c3a8>] sys_read+0x45/0x6e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: hacheck       R  running task        0 16053  26764
> Jan 14 18:19:00 4003a6 kernel:  ffff8101f5c11f70 ffffffff804c3df4 00000000496d04f4 000000001536ca98
> Jan 14 18:19:00 4003a6 kernel:  ffff8102630ee608 ffff8101ee950240 ffff8102471ca970 ffff8101ee9505a8
> Jan 14 18:19:00 4003a6 kernel:  00000005ee950240 ffff8103ff5f0fe0 ffffffff804c750e ffff810446c73d80
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c3df4>] thread_return+0x3d/0xa2
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c750e>] do_page_fault+0x43b/0x7f6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803702f3>] __up_read+0x19/0x7f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020c237>] retint_careful+0xd/0x21
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: hacheck       S 0000000000000000     0 16059  16029
> Jan 14 18:19:00 4003a6 kernel:  ffff81025a3f7e88 0000000000000086 0000000000000000 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff8102e2cc4140 ffff81042af5b260 ffff8102e2cc44a0
> Jan 14 18:19:00 4003a6 kernel:  0000000600000008 ffffe200086b3030 ffff8104410e55c8 ffffffff80270bf2
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80270bf2>] do_wp_page+0x456/0x4e8
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: ifconfig      R  running task        0 16060  16059
> Jan 14 18:19:00 4003a6 kernel:  ffff8102715bdea8 0000000000000046 ffff8104274f1400 0000000000000001
> Jan 14 18:19:00 4003a6 kernel:  0000000000000001 ffff810409f60380 ffff8102e318ebf0 ffff810409f606e8
> Jan 14 18:19:00 4003a6 kernel:  0000000600000000 0000000000000001 0000000000000000 ffff8104274f1400
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80291285>] pipe_release+0x7e/0x89
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc3a>] dput+0x26/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028c79d>] __fput+0x14f/0x17e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022e678>] __cond_resched+0x1c/0x43
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c3ece>] _cond_resched+0x2d/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80234c2e>] put_files_struct+0x6b/0xc4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235df5>] do_exit+0x269/0x67a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236275>] do_group_exit+0x6f/0x8a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: S95make_dhcp  S ffff810001005e60     0 16061  15568
> Jan 14 18:19:00 4003a6 kernel:  ffff8101f4c4fe88 0000000000000086 ffff8101f4c4fe50 ffffffff8021c2fd
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff8102101f6cf0 ffff8104190b6100 ffff8102101f7050
> Jan 14 18:19:00 4003a6 kernel:  0000000000000008 0000000101031e5a ffffffffffffffff ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8021c2fd>] flush_tlb_page+0x61/0x7c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: grep          R  running task        0 16062  16059
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff8104449d32c0 ffffffff8028a033 ffff81025a36f380
> Jan 14 18:19:00 4003a6 kernel:  0000000000000001 0000000000000001 0000000000000286 ffff810447045480
> Jan 14 18:19:00 4003a6 kernel:  ffff810400c98200 0000000000000000 ffff8102e318ebf0 0000000000000000
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a033>] filp_close+0x5d/0x65
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5581>] _write_lock_irq+0xf/0x10
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235e93>] do_exit+0x307/0x67a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236275>] do_group_exit+0x6f/0x8a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: snmpget       S 0000000000000008     0 16063  16061
> Jan 14 18:19:00 4003a6 kernel:  ffff8101d2481a28 0000000000000082 0000000000000001 ffff8101d2481a38
> Jan 14 18:19:00 4003a6 kernel:  ffff81044505e000 ffff8102429b8080 ffff8101f6198930 ffff8102429b83e0
> Jan 14 18:19:00 4003a6 kernel:  0000000237037b58 00000000000380ba ffff810446c60e98 ffffffff804c544f
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c544f>] _read_lock_bh+0x9/0x19
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80485958>] udp_push_pending_frames+0x2ca/0x345
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80486be9>] udp_sendmsg+0x4e5/0x5e7
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043ee12>] sock_sendmsg+0xe2/0xff
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029de3f>] touch_atime+0xbe/0x103
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b47c5>] compat_core_sys_select+0x18a/0x227
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043f87b>] sys_sendto+0xf3/0x127
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273044>] handle_mm_fault+0x360/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802b5b3b>] compat_sys_select+0xa3/0x149
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80223574>] cstar_do_call+0x1b/0x65
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rsync         R  running task        0 16064  16046
> Jan 14 18:19:00 4003a6 kernel:  ffff8103bec5d6c0 ffff8103bec5d6c0 ffff810445d47580 0000000000000360
> Jan 14 18:19:00 4003a6 kernel:  ffff810445d47580 0000000000000080 0000000000000286 ffff81044685b3c0
> Jan 14 18:19:00 4003a6 kernel:  ffff8102af886e00 0000000000000000 ffff81028b1ac2c0 0000000000000000
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5581>] _write_lock_irq+0xf/0x10
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235e93>] do_exit+0x307/0x67a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236275>] do_group_exit+0x6f/0x8a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: tcsh          S 00007fff09b78970     0 16065  14812
> Jan 14 18:19:00 4003a6 kernel:  ffff81019c033f28 0000000000000082 ffff8101edccda28 ffffffff80273394
> Jan 14 18:19:00 4003a6 kernel:  800000039d367065 ffff8101ef5ea180 ffff8104209ac480 ffff8101ef5ea4e0
> Jan 14 18:19:00 4003a6 kernel:  000000003f9f4280 00000000000005f0 ffff81019d8bf018 ffffffff802300af
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80273394>] handle_mm_fault+0x6b0/0x728
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c750e>] do_page_fault+0x43b/0x7f6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023d46a>] sys_rt_sigsuspend+0xb8/0xd9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023db7f>] sys_rt_sigprocmask+0x50/0xcc
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bfb7>] ptregscall_common+0x67/0xb0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: hostname      R  running task        0 16066  16065
> Jan 14 18:19:00 4003a6 kernel:  ffff81019ac7bc68 0000000000000082 ffffffff803474df ffff8101f600d710
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff81019d93c100 ffff81023f8a8c70 ffff81019d93c468
> Jan 14 18:19:00 4003a6 kernel:  0000000000000074 ffff8101edcac009 ffff81034a839aec ffffffffffffff10
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803474df>] dummy_inode_permission+0x0/0x3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80370898>] memcmp+0xb/0x22
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029b6f3>] __d_lookup+0xa8/0x104
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022e678>] __cond_resched+0x1c/0x43
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c3ece>] _cond_resched+0x2d/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc2b>] dput+0x17/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80292385>] path_put_conditional+0x16/0x3e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80294bb6>] __link_path_walk+0xd2b/0xd64
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc3a>] dput+0x26/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80294c4c>] path_walk+0x5d/0xb9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80294f14>] do_path_lookup+0x158/0x1d0
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029591f>] __path_lookup_intent_open+0x56/0x97
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80295c41>] open_namei+0x7a/0x66b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028e68f>] cp_new_stat+0xe5/0xfd
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a35e>] do_filp_open+0x1c/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a0ac>] get_unused_fd_flags+0x71/0x115
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a3c0>] do_sys_open+0x46/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: hacheck       R  running task        0 16067  16052
> Jan 14 18:19:00 4003a6 kernel:  ffff810270a99f70 0000000000000082 ffffffff802300af 00007fa7d72eb780
> Jan 14 18:19:00 4003a6 kernel:  0000000000000286 ffff81025045c140 ffff8102009c1360 ffff81025045c4a8
> Jan 14 18:19:00 4003a6 kernel:  00000000802322d4 ffff810270a99f58 00007fffdf2fea60 0000000000000008
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802300af>] wake_up_new_task+0xad/0xb5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bd05>] sysret_careful+0xd/0x10
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: ssrdm_rsync_f S 0000000000000000     0 16068  15998
> Jan 14 18:19:00 4003a6 kernel:  ffff81042be9fe88 0000000000000086 ffff81042be9fe28 ffffffff8029b76a
> Jan 14 18:19:00 4003a6 kernel:  ffff81042be9fe18 ffff81026301aa30 ffff81023f938400 ffff81026301ad90
> Jan 14 18:19:00 4003a6 kernel:  00000004e09433c0 00003ec5095ac630 0000000000000001 ffff8103e09433e0
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029b76a>] d_lookup+0x1b/0x3c
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023598a>] do_wait+0x8d6/0x9a4
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235ae2>] sys_wait4+0x8a/0xa1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: sed           R  running task        0 16070  16068
> Jan 14 18:19:00 4003a6 kernel:  ffff81025a303c68 0000000000000082 ffff81025a303ea8 ffff810446c94d80
> Jan 14 18:19:00 4003a6 kernel:  ffff810446d86d80 ffff8103e0815520 ffff810238421620 ffff8103e0815888
> Jan 14 18:19:00 4003a6 kernel:  0000000246c94d80 ffff81025a303c78 ffff81044685b1c0 ffff81042ae441ff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc3a>] dput+0x26/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029b6f3>] __d_lookup+0xa8/0x104
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022e678>] __cond_resched+0x1c/0x43
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c3ece>] _cond_resched+0x2d/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc2b>] dput+0x17/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80292385>] path_put_conditional+0x16/0x3e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80294bb6>] __link_path_walk+0xd2b/0xd64
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80294c4c>] path_walk+0x5d/0xb9
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80294f14>] do_path_lookup+0x158/0x1d0
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029591f>] __path_lookup_intent_open+0x56/0x97
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80295c41>] open_namei+0x7a/0x66b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028e989>] vfs_stat_fd+0x1b/0x4a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a35e>] do_filp_open+0x1c/0x38
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a0ac>] get_unused_fd_flags+0x71/0x115
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a3c0>] do_sys_open+0x46/0xc3
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rsync         R  running task        0 16071    658
> Jan 14 18:19:00 4003a6 kernel:  ffff81028b0f7a18 0000000000000082 41000010410000a0 4000003041000020
> Jan 14 18:19:00 4003a6 kernel:  400000a0400000e0 ffff8102ae84f420 ffff81041780d2a0 ffff8102ae84f780
> Jan 14 18:19:00 4003a6 kernel:  00000007410000c0 4000001041000020 0000000000000005 ffff810270b24168
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c427e>] schedule_timeout+0x8a/0xad
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023aca4>] process_timeout+0x0/0x5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297d81>] do_select+0x3fd/0x461
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80298269>] __pollwait+0x0/0xe1
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8022a6ce>] default_wake_function+0x0/0xe
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8046d0f9>] ip_queue_xmit+0x2c2/0x315
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff88152e99>] :forcedeth:nv_start_xmit_optimized+0x495/0x4d6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8026efc7>] zone_statistics+0x3f/0x93
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023addd>] lock_timer_base+0x26/0x4b
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8023af22>] __mod_timer+0xb0/0xbf
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440e84>] sk_reset_timer+0xf/0x19
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8047e0bf>] __tcp_push_pending_frames+0x722/0x7df
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c54f6>] _spin_lock_bh+0x9/0x1f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80440d2b>] release_sock+0x13/0x96
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80473ca4>] tcp_sendmsg+0x947/0xa57
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8043e668>] sock_aio_write+0xf8/0x110
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80297fa0>] core_sys_select+0x1bb/0x263
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8024421d>] autoremove_wake_function+0x0/0x2e
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802984a4>] sys_select+0x15a/0x183
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: date          R  running task        0 16073  24960
> Jan 14 18:19:00 4003a6 kernel:  ffff8101eb0f3d28 ffff8101eb0f3d6c 0000000000000000 ffff8101eb0f3d70
> Jan 14 18:19:00 4003a6 kernel:  0000000000000001 ffffffff802f7ba5 000000018020c228 0000000000001000
> Jan 14 18:19:00 4003a6 kernel:  0000000000000292 ffff8101eb0f3dc8 0000000000000005 ffff8101eb0f3de8
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802f7ba5>] xfs_bmap_search_extents+0x5b/0xe6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802fdfa9>] xfs_bmapi+0x26e/0xf89
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803702f3>] __up_read+0x19/0x7f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80314887>] xfs_iunlock+0x57/0x79
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8032ce6c>] xfs_free_eofblocks+0xb2/0x213
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc3a>] dput+0x26/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a033>] filp_close+0x5d/0x65
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5581>] _write_lock_irq+0xf/0x10
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235e93>] do_exit+0x307/0x67a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236275>] do_group_exit+0x6f/0x8a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: ifconfig      R  running task        0 16074  16067
> Jan 14 18:19:00 4003a6 kernel:  ffff81019d8dff70 0000000000000086 ffff81034a9292f0 0000000031d3597b
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 ffff8102009c1360 ffff8101f6198930 ffff8102009c16c8
> Jan 14 18:19:00 4003a6 kernel:  000000023bdb2800 00000000fffffff2 0000000000000007 ffff8101ed53d380
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bd05>] sysret_careful+0xd/0x10
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: rdserver.XBOS R  running task        0 16075  30239
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 0000000001200011 0000000000000246 ffff8102504a2000
> Jan 14 18:19:00 4003a6 kernel:  ffff810246cd86c0 0000000000000000 0000000000000000 0000000000000000
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: nfs           R  running task        0 16076  15894
> Jan 14 18:19:00 4003a6 kernel:  0000000000000000 00000000ffffffff 00000000006f9380 00007fff1e473850
> Jan 14 18:19:00 4003a6 kernel:  00007fff1e4738c0 0000000000000000 0000000000000246 00007f7c1645e780
> Jan 14 18:19:00 4003a6 kernel:  ffffffffffffffc0 0000000000003ecc 0000000000000038 ffffffffffffffff
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: Sched Debug Version: v0.07, 2.6.25.14 #1
> Jan 14 18:19:00 4003a6 kernel: now at 17282168.206823 msecs
> Jan 14 18:19:00 4003a6 kernel:   .sysctl_sched_latency                    : 80.000000
> Jan 14 18:19:00 4003a6 kernel:   .sysctl_sched_min_granularity            : 16.000000
> Jan 14 18:19:00 4003a6 kernel:   .sysctl_sched_wakeup_granularity         : 20.000000
> Jan 14 18:19:00 4003a6 kernel:   .sysctl_sched_batch_wakeup_granularity   : 40.000000
> Jan 14 18:19:00 4003a6 kernel:   .sysctl_sched_child_runs_first           : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .sysctl_sched_features                   : 15
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#0, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 6
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 6144
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 31130467
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17282052
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : -1591
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949348
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949368
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 12726
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 17856362.890245
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 12900660.669318
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307515.521711
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 44659538
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 3955114
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 8
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 6144
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 6144
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 6135
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 6027
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 5780
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 3932105.042613
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 1024
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 915
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 97334.666576
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 2458156.257369
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 14940657.595102
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 14940924.262990
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 266.667888
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 6
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 6144
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 41619
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 926504.744454
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2898342.449032
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 14904
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 59
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:  ssifg_ArciRecrC 27573  14940737.595102     13630   120  14940737.595102       962.318266  13194320.603374
> Jan 14 18:19:00 4003a6 kernel:             java  1027  14940817.595102    128416   120  14940817.595102     23643.142291  13099956.427897
> Jan 14 18:19:00 4003a6 kernel: R           java 12726  14941031.206772      1666   120  14941031.206772      2988.031953     34122.517070
> Jan 14 18:19:00 4003a6 kernel:           expect 25704  14940657.595102     20113   120  14940657.595102       776.611591  12316547.565722
> Jan 14 18:19:00 4003a6 kernel:         hostname 16066  14940897.595102         3   120  14940897.595102        10.369949         0.000000
> Jan 14 18:19:00 4003a6 kernel:          hacheck 16067  14940924.262990         2   120  14940924.262990         9.719064         0.000000
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#1, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 4
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 4096
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 38365314
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17281822
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : -1397
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949348
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949310
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 16073
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 17911531.595164
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 11878891.858124
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307340.896471
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 44235525
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 4924600
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 4096
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 4288
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 4366
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 4452
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 4840
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 4872674.780692
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 1024
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 1887
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 1.612972
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 109695.470616
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.208495
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 2697787.900002
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 17587277.649672
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 17587384.316338
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 106.666666
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 4
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 4096
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 40610
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 1.786435
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 942886.238083
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3791696.391025
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 893353.941993
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 15589
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 35
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:  sspmt_NtpPm_amd 24960  17587357.649672      9131   120  17587357.649672      2248.762306  13204797.049445
> Jan 14 18:19:00 4003a6 kernel:             java 30703  17587277.649672   3130190   120  17587277.649672   1547073.570323  10784095.849521
> Jan 14 18:19:00 4003a6 kernel: R           date 16073  17587277.653916         7   120  17587277.653916        39.787606         0.000000
> Jan 14 18:19:00 4003a6 kernel:    rdserver.XBOS 16075  17587384.316338         0   120  17587384.316338         0.000000         0.000000
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#2, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 5
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 5120
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 38015880
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17281726
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : -1604
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949349
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949283
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 3300
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 17914546.754043
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 12081246.098899
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307337.933098
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 44517653
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 4760055
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 4096
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 3808
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 4054
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 4177
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 4201
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 4680314.976742
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 1024
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 1581
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 110939.524178
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 2703838.059435
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 16244095.567722
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 16244175.565837
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 79.998115
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 5
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 5120
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 65922
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 968373.302162
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3572600.099710
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 674257.650678
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 16631
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 66
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:  slogger_amd_lin 29783  16244095.567722  11290644   120  16244095.567722     28218.412033   4016120.346522
> Jan 14 18:19:00 4003a6 kernel:  slogger_amd_lin 29784  16244095.572483  11290232   120  16244095.572483     28459.969911   4047586.535499
> Jan 14 18:19:00 4003a6 kernel: R           java  3300  16244089.535316    221880   120  16244089.535316     64040.413267  12957072.394905
> Jan 14 18:19:00 4003a6 kernel:              sed 16070  16244169.078022         7   120  16244169.078022        19.266816         0.000000
> Jan 14 18:19:00 4003a6 kernel:         ifconfig 16074  16244175.565837         5   120  16244175.565837        19.281597         0.000000
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#3, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 2
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 2048
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 47765620
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17281632
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : -2141
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949349
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949235
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 16076
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 17965258.503496
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 12251666.677509
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307336.902280
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 46095315
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 4650820
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 2048
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 2048
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 2047
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 1990
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 1818
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 4532016.242361
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 2048
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 1151
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 1.905207
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 109585.977211
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 2730706.095459
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 44950
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 993265.083169
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 19497461.456447
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3407171.317888
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 19497461.456447
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 508828.868856
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 2
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 2048
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 23779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 114
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:              nfs 15894  19497461.456447        18   120  19497461.456447       135.315610      1979.859961
> Jan 14 18:19:00 4003a6 kernel: R            nfs 16076  19497457.804637         0   120  19497457.804637         0.311881         0.000000
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#4, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 6
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 6397
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 36610554
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17281523
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : -3173
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949349
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949195
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 14933
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 17777411.083400
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 13144901.094473
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307515.849181
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 49749464
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 3847345
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 3072
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 2686
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 2425
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 2122
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 1894
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 3713853.114485
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 2868334.631803
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 2868334.631803
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 2
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 3072
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 133
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 21.023079
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.436086
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 123008.146954
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 60.082018
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 2600099.065594
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 8851908.950967
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 8851908.950967
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 1024
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 54761
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 734768.866481
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 8996718.441946
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 2868414.631803
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 8996878.296809
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 159.854863
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : -29927.817229
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 5
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 5373
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 13274
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 58
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:            klogd  5127   8996718.441946       338   120   8996718.441946         9.275270  17864523.423532
> Jan 14 18:19:00 4003a6 kernel:                X  7209   8996798.300296    142387   119   8996798.300296      8714.327915  17864494.934939
> Jan 14 18:19:00 4003a6 kernel:             java 10397   8851908.950967     19412   120   8851908.950967     24915.177624    433324.758073
> Jan 14 18:19:00 4003a6 kernel:            mount 27866   8996878.296809     91894   120   8996878.296809   1421226.200778         0.000000
> Jan 14 18:19:00 4003a6 kernel:            xterm  8017   8996798.441946       631   120   8996798.441946       173.251433    119502.335263
> Jan 14 18:19:00 4003a6 kernel: R           bash 14933   8996799.387381        84   120   8996799.387381       167.130400     13631.618861
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#5, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 2
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 2048
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 40969983
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17281612
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : -3736
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949349
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949425
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 16053
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 17848663.155093
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 12748807.837629
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307516.064610
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 49743214
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 4195086
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 2048
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 2064
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 2128
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 2264
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 2441
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 4012138.375793
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 1024
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 185
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 106.621491
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.828228
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 100518.528164
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 1
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.011243
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 2838466.759016
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 11328634.534658
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 11328634.534658
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 2
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 2048
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 65379
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 836122.262044
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3093660.184892
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 195317.735860
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 22779
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 236
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:  sssep_ErrorProc 30138  11328634.534658   5868344   120  11328634.534658    279812.126054  12481874.626567
> Jan 14 18:19:00 4003a6 kernel: R        hacheck 16053  11328634.532033        38   120  11328634.532033        75.163250         0.029867
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#6, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 4
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 4096
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 153217997
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17281342
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : -7038
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949349
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949210
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 16062
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 18232409.994014
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 12184009.400516
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307338.632147
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 54295059
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 5417899
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 1024
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 1535
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 1865
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 2000
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 2271
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 4552649.628483
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 1024
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 98
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 126.217497
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.234136
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 110549.215850
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 52.308082
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 3371020.635587
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 8663893.271925
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 8664053.271925
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 160.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 4
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 4096
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 65671
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 936595.523152
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 3517200.219460
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 618857.770428
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 17682
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 357
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:             java 31094   8663973.271925    467980   120   8663973.271925     13165.292777  12925895.726681
> Jan 14 18:19:00 4003a6 kernel:            slapd 30740   8663893.271925   1365195   120   8663893.271925     59736.577795  13072867.732932
> Jan 14 18:19:00 4003a6 kernel:         ifconfig 16060   8664053.271925        80   120   8664053.271925        85.676611         0.000000
> Jan 14 18:19:00 4003a6 kernel: R           grep 16062   8664031.696300        41   120   8664031.696300         3.551059       124.927439
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cpu#7, 1800.081 MHz
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 3
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 3072
> Jan 14 18:19:00 4003a6 kernel:   .nr_switches                   : 282208745
> Jan 14 18:19:00 4003a6 kernel:   .nr_load_updates               : 17281254
> Jan 14 18:19:00 4003a6 kernel:   .nr_uninterruptible            : 20682
> Jan 14 18:19:00 4003a6 kernel:   .jiffies                       : 4311949350
> Jan 14 18:19:00 4003a6 kernel:   .next_balance                  : 4311.949243
> Jan 14 18:19:00 4003a6 kernel:   .curr->pid                     : 16064
> Jan 14 18:19:00 4003a6 kernel:   .clock                         : 17986239.256096
> Jan 14 18:19:00 4003a6 kernel:   .idle_clock                    : 2781601.332349
> Jan 14 18:19:00 4003a6 kernel:   .prev_clock_raw                : 17307347.260738
> Jan 14 18:19:00 4003a6 kernel:   .clock_warps                   : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_overflows               : 63056889
> Jan 14 18:19:00 4003a6 kernel:   .clock_underflows              : 10695003
> Jan 14 18:19:00 4003a6 kernel:   .clock_deep_idle_events        : 0
> Jan 14 18:19:00 4003a6 kernel:   .clock_max_delta               : 0.999848
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[0]                   : 3072
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[1]                   : 3460
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[2]                   : 3574
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[3]                   : 3961
> Jan 14 18:19:00 4003a6 kernel:   .cpu_load[4]                   : 4843
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 13827361.728150
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 1
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 1024
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 2
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 2420.559315
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 7.137281
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 35691.061126
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 37.031507
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 12766853.985154
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 11059011.999455
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 11059013.819265
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 1.819810
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 3
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 3072
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 80380
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 15.512259
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 0
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: cfs_rq
> Jan 14 18:19:00 4003a6 kernel:   .exec_clock                    : 971880.304338
> Jan 14 18:19:00 4003a6 kernel:   .MIN_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .min_vruntime                  : 12769443.124835
> Jan 14 18:19:00 4003a6 kernel:   .max_vruntime                  : 0.000001
> Jan 14 18:19:00 4003a6 kernel:   .spread                        : 0.000000
> Jan 14 18:19:00 4003a6 kernel:   .spread0                       : 9871100.675803
> Jan 14 18:19:00 4003a6 kernel:   .nr_running                    : 0
> Jan 14 18:19:00 4003a6 kernel:   .load                          : 0
> Jan 14 18:19:00 4003a6 kernel:   .bkl_count                     : 7866
> Jan 14 18:19:00 4003a6 kernel:   .nr_spread_over                : 13
> Jan 14 18:19:00 4003a6 kernel: 
> Jan 14 18:19:00 4003a6 kernel: runnable tasks:
> Jan 14 18:19:00 4003a6 kernel:             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Jan 14 18:19:00 4003a6 kernel: ----------------------------------------------------------------------------------------------------------
> Jan 14 18:19:00 4003a6 kernel:    rdserver.XBOS   858  11059013.819265  18178664   120  11059013.819265   1946926.355979    740523.458392
> Jan 14 18:19:00 4003a6 kernel: R          rsync 16064  11059018.015230         6   120  11059018.015230         0.376429         0.460058
> Jan 14 18:19:00 4003a6 kernel:            rsync 16071  11059011.999455        33   120  11059011.999455        34.019923         1.421904
> Jan 14 18:19:00 4003a6 kernel: 

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Stale XFS mount for Kernel 2.6.25.14

[Dave Chinner]

On Wed, Jan 14, 2009 at 06:50:24PM -0500, Ngo, Andrew wrote:
> Dave,
> 
> From time to time I am still experiencing the stale mount.  From this
> last email, you want me to issue the 'echo t > /proc/sysrq-trigger'
> command to show all the processes in the machine.  Here is the capture
> of the /var/log/messages when I run the command. Please let me know what
> you find out.  Thanks...

> Jan 14 18:19:00 4003a6 kernel: date          R  running task        0 16073  24960
> Jan 14 18:19:00 4003a6 kernel:  ffff8101eb0f3d28 ffff8101eb0f3d6c 0000000000000000 ffff8101eb0f3d70
> Jan 14 18:19:00 4003a6 kernel:  0000000000000001 ffffffff802f7ba5 000000018020c228 0000000000001000
> Jan 14 18:19:00 4003a6 kernel:  0000000000000292 ffff8101eb0f3dc8 0000000000000005 ffff8101eb0f3de8
> Jan 14 18:19:00 4003a6 kernel: Call Trace:
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802f7ba5>] xfs_bmap_search_extents+0x5b/0xe6
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff802fdfa9>] xfs_bmapi+0x26e/0xf89
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff803702f3>] __up_read+0x19/0x7f
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80314887>] xfs_iunlock+0x57/0x79
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8032ce6c>] xfs_free_eofblocks+0xb2/0x213
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8029bc3a>] dput+0x26/0xd5
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8028a033>] filp_close+0x5d/0x65
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff804c5581>] _write_lock_irq+0xf/0x10
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80235e93>] do_exit+0x307/0x67a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff80236275>] do_group_exit+0x6f/0x8a
> Jan 14 18:19:00 4003a6 kernel:  [<ffffffff8020bc9b>] system_call_after_swapgs+0x7b/0x80

That's the only stack trace that has any XFS references in it.
Do you have a 'date' process stuck on your machine or was this just
a "lucky" occurrence?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs