Re: [REGRESSION]: hibernate/sleep regression w/ bisection

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Tejun Heo]

On Tue, Nov 01, 2011 at 07:48:19AM -0500, Andrew Watts wrote:
> Hi.
> 
> Hibernate/sleep (echo disk/mem > /sys/power/state) has presented problems
> for me starting with 2.6.39. Kernel 2.6.37.6 was the last completely bug-free
> version I used (I skipped the 2.6.38 branch entirely).
> 
> The symptoms are that upon resume (from sleep/hibernate) there is no video
> nor any keyboard input with the exception of sysrq.
> 
> It has been a frustrating bug to hunt down because it is not easily
> reproduced; sometimes the bug doesn't pop up until after a long sequence of
> hibernate/sleep cycles.
> 
> I successfully bisected the problem to: 8ee294cd9def000.
> 
> =======
> Commit: 8ee294cd9def0004887da7f44b80563493b0a097
> Author: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> Date:   Mon Nov 15 01:39:57 2010 -0800
> Input: serio - convert to common workqueue instead of a thread
> =======
> 
> Backing out 8ee294cd9def000 (which requires reversing part of
> 1d64b655dc083df also) fixes this particular problem on 2.6.39.4,
> 3.0.8, and 3.1.
> 
> Unfortunately, in 3.0.8 and 3.1 I have other suspend/hibernate problems
> that I will investigate next and detail in a different regression report.

Does the following patch fix the problem?

Thanks.

diff --git a/drivers/input/serio/serio.c b/drivers/input/serio/serio.c
index ba70058..95aeebd 100644
--- a/drivers/input/serio/serio.c
+++ b/drivers/input/serio/serio.c
@@ -949,6 +949,7 @@ static int serio_resume(struct device *dev)
 	 * deal with it.
 	 */
 	serio_queue_event(serio, NULL, SERIO_RECONNECT_PORT);
+	queue_work(system_long_wq, &serio_event_work);
 
 	return 0;
 }

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Tue, Nov 01, 2011 at 09:42:35AM -0700, Tejun Heo wrote:
> 
> Does the following patch fix the problem?
> 
> Thanks.

Unfortunately, it does not fix the problem.

~ Andy

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Tejun Heo]

On Tue, Nov 01, 2011 at 02:15:16PM -0500, Andrew Watts wrote:
> On Tue, Nov 01, 2011 at 09:42:35AM -0700, Tejun Heo wrote:
> > 
> > Does the following patch fix the problem?
> > 
> > Thanks.
> 
> Unfortunately, it does not fix the problem.

Odd... how about the following one?  Thanks.

diff --git a/drivers/input/serio/serio.c b/drivers/input/serio/serio.c
index ba70058..14d560a 100644
--- a/drivers/input/serio/serio.c
+++ b/drivers/input/serio/serio.c
@@ -299,9 +299,8 @@ static int serio_queue_event(void *object, struct module *owner,
 	event->owner = owner;
 
 	list_add_tail(&event->node, &serio_event_list);
-	queue_work(system_long_wq, &serio_event_work);
-
 out:
+	queue_work(system_long_wq, &serio_event_work);
 	spin_unlock_irqrestore(&serio_event_lock, flags);
 	return retval;
 }

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Tue, Nov 01, 2011 at 01:19:02PM -0700, Tejun Heo wrote:
> 
> Odd... how about the following one?  Thanks.
> 

No, the 2nd patch doesn't work either. However, as I mentioned, reversing 
8ee294cd9def000 (and part of 1d64b655dc083df) does fix it.

~ Andy

PS I noticed your posts make it to linux-pm@lists.linux-foundation.org.
   Mine never arrive there. Is that list closed?

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Dmitry Torokhov]

Hi Andrew,

On Tue, Nov 01, 2011 at 07:48:19AM -0500, Andrew Watts wrote:
> Hi.
> 
> Hibernate/sleep (echo disk/mem > /sys/power/state) has presented problems
> for me starting with 2.6.39. Kernel 2.6.37.6 was the last completely bug-free
> version I used (I skipped the 2.6.38 branch entirely).
> 
> The symptoms are that upon resume (from sleep/hibernate) there is no video
> nor any keyboard input with the exception of sysrq.

If SysRq is working that means that i8042, atkbd and input core are
working properly.

> 
> It has been a frustrating bug to hunt down because it is not easily
> reproduced; sometimes the bug doesn't pop up until after a long sequence of
> hibernate/sleep cycles.
> 
> I successfully bisected the problem to: 8ee294cd9def000.
> 
> =======
> Commit: 8ee294cd9def0004887da7f44b80563493b0a097
> Author: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> Date:   Mon Nov 15 01:39:57 2010 -0800
> Input: serio - convert to common workqueue instead of a thread
> =======
> 
> Backing out 8ee294cd9def000 (which requires reversing part of
> 1d64b655dc083df also) fixes this particular problem on 2.6.39.4,
> 3.0.8, and 3.1.

I am not sure how this commit could cause your "no video" issue...
Are you hibernating from text console or X? Can you try plugging in USB
keyboard and see if you are getting input? What about ssh into the box?

Thanks.

-- 
Dmitry

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Tue, Nov 01, 2011 at 10:46:58PM -0700, Dmitry Torokhov wrote:
> 
> I am not sure how this commit could cause your "no video" issue...
> Are you hibernating from text console or X? Can you try plugging in USB
> keyboard and see if you are getting input? What about ssh into the box?
> 
> Thanks.
> 
> -- 
> Dmitry

Hi Dmitry et al.

Yes, I was hibernating from X. I do not have problems when I hibernate from
console.

I conducted the following exercise: a script runs a 20-hibernate loop that
saves dmesg output to individual files and pings my httpd server (on lo)
after every resume.

>From X, with the vanilla 2.6.39.4 kernel, I had 5 bad resumes out of 20
(the script continues though). My httpd server is pinged after all 20 
resumes and my dmesg is saved to a file. So, even after the bad resumes,
loopback networking and HD IO are functional.

The five problem resumes all have a similar warning in dmesg (see below).
I am copying Rafael Wysocki (since I can't seem to post to linux-pm@) 
and David Airlie (given the 'GPU lockup CP stall' message). [Please see 
thread https://lkml.org/lkml/2011/11/1/126 for complete history]

The same loop test conducted either a) from console or b) from X after
reversing commits 8ee294cd9def000 & 1d64b655dc083df encounters no problems
(all 20 resumes succeed) and there are no GPU lockup messages in my saved
dmesg output.

System specs:
P4 + ATI 9100IGP

~ Andy

PS. My messages do not arrive to linux-pm@lists.linux-foundation.org. Does
    posting require subscription?

-----------------------------

[  243.626231] radeon 0000:01:05.0: GPU lockup CP stall for more than 82318msec
[  243.626236] ------------[ cut here ]------------
[  243.626252] WARNING: at drivers/gpu/drm/radeon/radeon_fence.c:246 radeon_fence_wait+0x2c3/0x311()
[  243.626256] Hardware name: Pavilion zv5000
[  243.626259] GPU lockup (waiting for 0x00000F8C last fence id 0x00000F8A)
[  243.626262] Modules linked in: fan snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss fuse pl2303 snd_atiixp snd_ac97_codec ac97_bus snd_pcm pcmcia snd_timer snd ohci_hcd shpchp processor yenta_socket pcmcia_rsrc ssb video thermal wmi battery button ac ehci_hcd thermal_sys pcmcia_core firewire_ohci snd_page_alloc
[  243.626311] Pid: 1646, comm: X Tainted: G   M        2.6.39.4 #19
[  243.626314] Call Trace:
[  243.626325]  [<c1033164>] warn_slowpath_common+0x67/0x8e
[  243.626331]  [<c12c082a>] ? radeon_fence_wait+0x2c3/0x311
[  243.626335]  [<c12c082a>] ? radeon_fence_wait+0x2c3/0x311
[  243.626339]  [<c1033207>] warn_slowpath_fmt+0x2e/0x30
[  243.626344]  [<c12c082a>] radeon_fence_wait+0x2c3/0x311
[  243.626352]  [<c104b12a>] ? wake_up_bit+0x62/0x62
[  243.626358]  [<c12c0e37>] radeon_sync_obj_wait+0xc/0xe
[  243.626363]  [<c12908be>] ttm_bo_wait+0xa1/0x108
[  243.626371]  [<c12d6e7b>] radeon_gem_wait_idle_ioctl+0x76/0xc4
[  243.626378]  [<c127e62e>] drm_ioctl+0x1c2/0x42c
[  243.626383]  [<c12d6e05>] ? radeon_gem_set_tiling_ioctl+0x8e/0x8e
[  243.626389]  [<c109ad9a>] ? perf_pmu_enable+0x1a/0x21
[  243.626395]  [<c1026eed>] ? update_curr+0x164/0x24d
[  243.626403]  [<c11f5171>] ? rb_erase+0x16e/0x27a
[  243.626411]  [<c1007ba8>] ? __switch_to_xtra+0xf7/0x11d
[  243.626417]  [<c103015e>] ? set_next_entity+0xad/0xc1
[  243.626422]  [<c127e46c>] ? drm_version+0x8a/0x8a
[  243.626428]  [<c10e288e>] do_vfs_ioctl+0x79/0x54b
[  243.626436]  [<c158c95a>] ? schedule+0x29a/0x614
[  243.626441]  [<c10e2dcb>] sys_ioctl+0x6b/0x70
[  243.626446]  [<c1593813>] sysenter_do_call+0x12/0x22
[  243.626449] ---[ end trace 17a119d9361222c9 ]---
[  243.633615] radeon 0000:01:05.0: GPU reset succeed
[  243.671128] radeon 0000:01:05.0: WB disabled
[  243.671170] [drm] radeon: ring at 0x00000000D2001000
[  243.671189] [drm] ring test succeeded in 0 usecs
[  243.671200] [drm] ib test succeeded in 1 usecs
[  243.704033] ohci_hcd 0000:00:13.1: auto-stop root hub
[  243.704048] ohci_hcd 0000:00:13.0: auto-stop root hub

-----------------------------

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Tue, Nov 01, 2011 at 10:46:58PM -0700, Dmitry Torokhov wrote:
> 
> Are you hibernating from text console or X? Can you try plugging in USB
> keyboard and see if you are getting input? What about ssh into the box?

Forgot to answer a few of your questions. Plugging in a USB keyboard does
not help. And for now I cannot test whether the sshd daemon is reponsive
though it would seem that it would indeed be OK.

~ Andy

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Dmitry Torokhov]

On Wed, Nov 02, 2011 at 11:02:09AM -0500, Andrew Watts wrote:
> On Tue, Nov 01, 2011 at 10:46:58PM -0700, Dmitry Torokhov wrote:
> > 
> > Are you hibernating from text console or X? Can you try plugging in USB
> > keyboard and see if you are getting input? What about ssh into the box?
> 
> Forgot to answer a few of your questions. Plugging in a USB keyboard does
> not help.

OK, it looks like you do have input, what you do not have is X
functioning.

I am not sure why radeon reports lockup with that patch; serio uses
system_long_wq to schedule its long-playing works and the only other
user of system_long_wq seems to be sata, but then concurrency management
should take care of them sharing the workqueue.

Tejun, do you have any ideas here?

Thanks.

-- 
Dmitry

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Wed, Nov 02, 2011 at 09:31:09AM -0700, Dmitry Torokhov wrote:
> 
> OK, it looks like you do have input, what you do not have is X
> functioning.

The extent of my tests on input were to notice that the LED for caps
lock does not toggle when pressing "caps lock" after a bad resume. I
have also tried to blindly type "reboot" and such in what should be an
active xterm.

~ Andy

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Dmitry Torokhov]

On Wednesday, November 02, 2011 10:01:00 AM Andrew Watts wrote:
> On Wed, Nov 02, 2011 at 09:31:09AM -0700, Dmitry Torokhov wrote:
> > OK, it looks like you do have input, what you do not have is X
> > functioning.
> 
> The extent of my tests on input were to notice that the LED for caps
> lock does not toggle when pressing "caps lock" after a bad resume. I
> have also tried to blindly type "reboot" and such in what should be an
> active xterm.

When in X it is X that is responsible for LED switching. Also input
processing beyond the kernel won't be happening if X is hosed.

-- 
Dmitry

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Tejun Heo]

Hello,

On Wed, Nov 02, 2011 at 09:31:09AM -0700, Dmitry Torokhov wrote:
> On Wed, Nov 02, 2011 at 11:02:09AM -0500, Andrew Watts wrote:
> > On Tue, Nov 01, 2011 at 10:46:58PM -0700, Dmitry Torokhov wrote:
> > > 
> > > Are you hibernating from text console or X? Can you try plugging in USB
> > > keyboard and see if you are getting input? What about ssh into the box?
> > 
> > Forgot to answer a few of your questions. Plugging in a USB keyboard does
> > not help.
> 
> OK, it looks like you do have input, what you do not have is X
> functioning.
> 
> I am not sure why radeon reports lockup with that patch; serio uses
> system_long_wq to schedule its long-playing works and the only other
> user of system_long_wq seems to be sata, but then concurrency management
> should take care of them sharing the workqueue.
> 
> Tejun, do you have any ideas here?

Not at all.  Dang, I was feeling pretty sure that it was some missing
queue_work somewhere.  So, input is working fine but X locks up
because of that change?  That's just odd.

Andrew, just in case, can you please double check that you're
reverting only that single commit (not all the ones afterwards) and
the test results?  If they're all okay, can you please trigger the
problem, wait a while, trigger sysrq-t and attach the kernel log?

Thank you.

-- 
tejun

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Thu, Nov 03, 2011 at 08:59:56AM -0700, Tejun Heo wrote:
> Andrew, just in case, can you please double check that you're
> reverting only that single commit (not all the ones afterwards) and
> the test results?  If they're all okay, can you please trigger the
> problem, wait a while, trigger sysrq-t and attach the kernel log?

Hi Tejun.

I am only reverting 8ee294cd9def0004887da7f44b80563493b0a097 though I first
revert part of 1d64b655dc083df5c5ac39945ccbbc6532903bf1 so it reverts 
cleanly. 

I am very sure of my test results; normally I get 4-5 bad resumes on a 20
hibernate loop and get no bad resumes with the commit reverted. I have
been very thorough in my testing to ensure I am not generating false
positives or false negatives (repeating the loops many times).

Below is the output you asked for (sysrq t).

Thanks.

~ Andy

======================


kernel: [  345.226523] SysRq : Show State
kernel: [  345.226531]   task                PC stack   pid father
kernel: [  345.226535] init            S f4a47b68  5488     1      0 0x00000000
kernel: [  345.226765] kthreadd        S 00000000  7172     2      0 0x00000000
kernel: [  345.226819] ksoftirqd/0     S 0000003f  7472     3      2 0x00000000
kernel: [  345.226887] kworker/u:0     S 00000000  6404     5      2 0x00000000
kernel: [  345.226956] rcu_kthread     S c10ce899  7324     6      2 0x00000000
kernel: [  345.227032] watchdog/0      S 0000000b  7616     7      2 0x00000000
kernel: [  345.227096] cpuset          S f5c55f58  7696     8      2 0x00000000
kernel: [  345.227160] khelper         S f5c57f58  7696     9      2 0x00000000
kernel: [  345.227224] kworker/u:1     S 00000000  5800    10      2 0x00000000
kernel: [  345.227284] netns           S f5c69f58  7660    13      2 0x00000000
kernel: [  345.227348] sync_supers     S f5ca5fbc  7712   227      2 0x00000000
kernel: [  345.227425] bdi-default     S f5ca7e70  7392   229      2 0x00000000
kernel: [  345.227507] kblockd         S f5cd7f58  7660   231      2 0x00000000
kernel: [  345.227509] ata_sff         S f5cf1f58  7660   318      2 0x00000000
kernel: [  345.227509] khubd           S c13f0020  5844   328      2 0x00000000
kernel: [  345.227509] md              S f5ce1f58  7660   335      2 0x00000000
kernel: [  345.227509] cfg80211        S f5cfdf58  7660   345      2 0x00000000
kernel: [  345.227509] kworker/0:1     S f49ae800  6248   346      2 0x00000000
kernel: [  345.227509] rpciod          S f5ce5f58  7660   444      2 0x00000000
kernel: [  345.227509] khungtaskd      S 00000000  7564   461      2 0x00000000
kernel: [  345.227509] kswapd0         S f5d25ea8  7444   467      2 0x00000000
kernel: [  345.227509] fsnotify_mark   S 00000000  7668   532      2 0x00000000
kernel: [  345.227509] nfsiod          S f5f05f58  7660   544      2 0x00000000
kernel: [  345.227509] crypto          S f5f19f58  7660   549      2 0x00000000
kernel: [  345.227509] ttm_swap        S f5f13f58  7660   628      2 0x00000000
kernel: [  345.227509] kworker/0:2     S 000ce000  6776   768      2 0x00000000
kernel: [  345.227509] kpsmoused       S f577ff58  7660   776      2 0x00000000
kernel: [  345.227509] kjournald       S 0000004f  6524   834      2 0x00000000
kernel: [  345.227509] udevd           S f500de68  6192   866      1 0x00000000
kernel: [  345.227509] kworker/0:3     S f5115f20  6860  1043      2 0x00000000
kernel: [  345.227509] pccardd         S f52dff24  7344  1048      2 0x00000000
kernel: [  345.227509] pccardd         S f530bf24  7344  1050      2 0x00000000
kernel: [  345.227509] flush-3:0       S c10d787b  6432  1116      2 0x00000000
kernel: [  345.227509] syslogd         S 00000000  6164  1167      1 0x00000000
kernel: [  345.227509] klogd           S 00000058  6612  1171      1 0x00000000
kernel: [  345.227509] inetd           S 00000000  6440  1283      1 0x00000000
kernel: [  345.227509] sshd            S 00000000  6332  1286      1 0x00000000
kernel: [  345.227509] acpid           S f539daf4  6248  1293      1 0x00000000
kernel: [  345.227509] dbus-daemon     S 00000000  6356  1301      1 0x00000000
kernel: [  345.227509] hald            S f5c649e0  6180  1306      1 0x00000000
kernel: [  345.227509] hald            S f77d82c4  7076  1308      1 0x00000000
kernel: [  345.227509] hald-runner     S c11f5171  6356  1307   1306 0x00000000
kernel: [  345.227509] hald-addon-inpu S 0222270e  6356  1342   1307 0x00000000
kernel: [  345.227509] hald-addon-gene S 00000010  6592  1344   1307 0x00000000
kernel: [  345.227509] hald-addon-acpi S 00000001  6572  1356   1307 0x00000000
kernel: [  345.227509] crond           S f53cdf40  6632  1363      1 0x00000000
kernel: [  345.227509] atd             S f53d1fb4  6532  1365      1 0x00000000
kernel: [  345.227509] sendmail        S f53f1f40  6268  1368      1 0x00000000
kernel: [  345.227509] sendmail        S f517c800  6176  1371      1 0x00000000
kernel: [  345.227509] udevd           S f53dde8c  6592  1377    866 0x00000000
kernel: [  345.227509] udevd           S f57a8000  6924  1378    866 0x00000000
kernel: [  345.227509] vnstatd         S 00000000  6556  1421      1 0x00000000
kernel: [  345.227509] auditd          S f5cb3d90  6584  1427      1 0x00000000
kernel: [  345.227509] auditd          S c1029e3c  6072  1428      1 0x00000000
kernel: [  345.227509] kauditd         S f481bf2c  7440  1430      2 0x00000000
kernel: [  345.227509] bash            S f5009b80  5844  1613      1 0x00000080
kernel: [  345.227509] agetty          S f53e3e04  6748  1614      1 0x00000080
kernel: [  345.227509] agetty          S f53e5e04  6748  1615      1 0x00000080
kernel: [  345.227509] agetty          S f4819e04  6748  1616      1 0x00000080
kernel: [  345.227509] agetty          S f53f3e04  6704  1617      1 0x00000080
kernel: [  345.227509] agetty          S f4971e04  6404  1618      1 0x00000080
kernel: [  345.227509] startx          S f5ddb700  6308  1633   1613 0x00000080
kernel: [  345.227509] xinit           S f49b5f9c  6592  1649   1633 0x00000080
kernel: [  345.227509] X               S f49d5944  5468  1650   1649 0x00400080
kernel: [  345.227509] sh              S f51e7180  6480  1658   1649 0x00000080
kernel: [  345.227509] xscreensaver    S f53f1f40  5800  1664   1658 0x00000080
kernel: [  345.227509] ssh-agent       S 00000000  6440  1667      1 0x00000080
kernel: [  345.227509] dbus-launch     S 00000000  6440  1672      1 0x00000080
kernel: [  345.227509] dbus-daemon     S 00000000  6356  1674      1 0x00000080
kernel: [  345.227509] xfce4-session   S f4a0daf4  5800  1676   1658 0x00000080
kernel: [  345.227509] xfconfd         S 00000033  5728  1678      1 0x00000080
kernel: [  345.227509] xfwm4           S c11f5234  5896  1683      1 0x00000080
kernel: [  345.227509] xfce4-panel     S c11f5234  5800  1685      1 0x00000080
kernel: [  345.227509] xfce4-panel     S 00000000  7360  1745      1 0x00000080
kernel: [  345.227509] Thunar          S f53f1f40  5800  1687      1 0x00000080
kernel: [  345.227509] xfsettingsd     S 00000010  6592  1688      1 0x00000080
kernel: [  345.227509] xfdesktop       S f53f1f40  5800  1690      1 0x00000080
kernel: [  345.227509] gkrellm         S f29e981d  5728  1694      1 0x00000080
kernel: [  345.227509] tilda           S 00000010  5876  1696      1 0x00000080
kernel: [  345.227509] tilda           S 00000000  7360  1720      1 0x00000080
kernel: [  345.227509] autocutsel      S c11f5234  6200  1698      1 0x00000080
kernel: [  345.227509] orage           S c11f5171  5728  1700      1 0x00000080
kernel: [  345.227509] gam_server      S 00000246  5728  1702      1 0x00000080
kernel: [  345.227509] devilspie       S f4aafaf4  6312  1706      1 0x00000080
kernel: [  345.227509] xterm           S 00000000  5728  1710      1 0x00000080
kernel: [  345.227509] xfce4-settings- S f4ad1b00  6356  1715      1 0x00000080
kernel: [  345.227509] xfce4-menu-plug S 00200246  6268  1717   1685 0x00000080
kernel: [  345.227509] gnome-pty-helpe S 00000003  6532  1718   1696 0x00000080
kernel: [  345.227509] bash            S f4ae1e04  6704  1719   1696 0x00000080
kernel: [  345.227509] tail            S f3c03fb4  5800  1724   1710 0x00000080
kernel: [  345.227509] gvfsd           S f3c07af0  5800  1726      1 0x00000080
kernel: [  345.227509] xfce4-weather-p S f4aa3b68  5800  1729   1685 0x00000080
kernel: [  345.227509] xfce4-notes-plu S f3c05af4  5800  1730   1685 0x00000080
kernel: [  345.227509] xfce4-battery-p S f4aed630  6180  1731   1685 0x00000080
kernel: [  345.227509] xfce4-mixer-plu S 00000000  6200  1732   1685 0x00000080
kernel: [  345.227509] task0           S 00000000  6484  1740   1685 0x00000080
kernel: [  345.227509] gvfs-fuse-daemo S c1029e3c  7176  1736      1 0x00000080
kernel: [  345.227509] gvfs-fuse-daemo S f4aebdc4  6944  1737      1 0x00000080
kernel: [  345.227509] gvfs-fuse-daemo S f4aeef04  7268  1738      1 0x00000080
kernel: [  345.227509] gvfs-fuse-daemo S 00000000  6592  1739      1 0x00000080
kernel: [  345.227509] xterm           S f3c69aa0  6160  1744      1 0x00000080
kernel: [  345.227509] bash            S f3c0cb00  6876  1747   1744 0x00000080
kernel: [  345.227509] bash            S 7f1c0300  5728  1763   1747 0x00000080
kernel: [  345.227509] flush-0:18      S f3d13f24  7616  1809      2 0x00000080
kernel: [  345.227509] kworker/u:2     S 00000000  6768  1810      2 0x00000080
kernel: [  345.227509] kworker/u:3     S 00000000  6552  1811      2 0x00000080
kernel: [  345.227509] kworker/u:4     S 00000000  6768  1812      2 0x00000080
kernel: [  345.227509] kworker/u:5     S 00000000  6768  1813      2 0x00000080
kernel: [  345.227509] kworker/u:6     S 00000000  6768  1814      2 0x00000080
kernel: [  345.227509] kworker/u:7     S 00000000  7124  1815      2 0x00000080
kernel: [  345.227509] kworker/u:8     S 00000000  6864  1816      2 0x00000080
kernel: [  345.227509] kworker/u:9     S 00000000  6356  1817      2 0x00000080
kernel: [  345.227509] kworker/u:10    S 00000000  6864  1818      2 0x00000080
kernel: [  345.227509] kworker/u:11    S 00000000  6600  1819      2 0x00000080
kernel: [  345.227509] kworker/u:12    S 00000000  7356  1820      2 0x00000080
kernel: [  345.227509] kworker/u:13    S 00000000  6356  1821      2 0x00000080
kernel: [  345.227509] kworker/u:14    S 00000000  7356  1822      2 0x00000080
kernel: [  345.227509] kworker/u:15    S 00000000  7356  1823      2 0x00000080
kernel: [  345.227509] kworker/u:16    S 00000000  7136  1824      2 0x00000080
kernel: [  345.227509] kworker/u:17    S 00000000  6600  1825      2 0x00000080
kernel: [  345.227509] kworker/u:18    S 00000000  7356  1826      2 0x00000080
kernel: [  345.227509] kworker/u:19    S 00000000  7136  1827      2 0x00000080
kernel: [  345.227509] kworker/u:20    S 00000000  6600  1828      2 0x00000080
kernel: [  345.227509] kworker/u:21    S 00000000  6356  1829      2 0x00000080
kernel: [  345.227509] kworker/u:22    S 00000000  7136  1830      2 0x00000080
kernel: [  345.227509] kworker/u:23    S 00000000  6864  1831      2 0x00000080
kernel: [  345.227509] kworker/u:24    S 00000000  7120  1832      2 0x00000080
kernel: [  345.227509] kworker/u:25    S 00000000  6768  1833      2 0x00000080
kernel: [  345.227509] kworker/u:26    S 00000000  7356  1834      2 0x00000080
kernel: [  345.227509] kworker/u:27    S 00000000  6768  1835      2 0x00000080
kernel: [  345.227509] kworker/u:28    S 00000000  6404  1836      2 0x00000080
kernel: [  345.227509] kworker/u:29    S 00000000  7224  1837      2 0x00000080
kernel: [  345.227509] kworker/u:30    S 00000000  6404  1838      2 0x00000080
kernel: [  345.227509] kworker/u:31    S 00000000  6768  1839      2 0x00000080
kernel: [  345.227509] kworker/u:32    S 00000000  6568  1840      2 0x00000080
kernel: [  345.227509] kworker/u:33    S 00000000  6768  1841      2 0x00000080
kernel: [  345.227509] kworker/u:34    S 00000000  6404  1842      2 0x00000080
kernel: [  345.227509] kworker/u:35    S 00000000  7248  1850      2 0x00000080
kernel: [  345.227509] kworker/u:36    S 00000000  6404  1851      2 0x00000080
kernel: [  345.227509] kworker/u:37    S 00000000  6768  1852      2 0x00000080
kernel: [  345.227509] kworker/u:38    S 00000000  6568  1853      2 0x00000080
kernel: [  345.227509] kworker/u:39    S 00000000  7356  1854      2 0x00000080
kernel: [  345.227509] kworker/u:40    S 00000000  7356  1855      2 0x00000080
kernel: [  345.227509] kworker/u:41    S 00000000  7672  1864      2 0x00000080

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Tejun Heo]

Hello, Andrew.

On Thu, Nov 03, 2011 at 01:46:19PM -0500, Andrew Watts wrote:
> On Thu, Nov 03, 2011 at 08:59:56AM -0700, Tejun Heo wrote:
> > Andrew, just in case, can you please double check that you're
> > reverting only that single commit (not all the ones afterwards) and
> > the test results?  If they're all okay, can you please trigger the
> > problem, wait a while, trigger sysrq-t and attach the kernel log?
> 
> Hi Tejun.
> 
> I am only reverting 8ee294cd9def0004887da7f44b80563493b0a097 though I first
> revert part of 1d64b655dc083df5c5ac39945ccbbc6532903bf1 so it reverts 
> cleanly. 
> 
> I am very sure of my test results; normally I get 4-5 bad resumes on a 20
> hibernate loop and get no bad resumes with the commit reverted. I have
> been very thorough in my testing to ensure I am not generating false
> positives or false negatives (repeating the loops many times).

Thanks.  Yeah, this is odd & interesting.

> Below is the output you asked for (sysrq t).
> 
> Thanks.
> 
> ~ Andy
> 
> ======================
> 
> 
> kernel: [  345.226523] SysRq : Show State
> kernel: [  345.226531]   task                PC stack   pid father
> kernel: [  345.226535] init            S f4a47b68  5488     1      0 0x00000000
> kernel: [  345.226765] kthreadd        S 00000000  7172     2      0 0x00000000

This is missing all stack traces.  Can you please turn on
FRAME_POINTER and adjust log level such that all stack dumps are
included too?

Thank you.

-- 
tejun

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On Thu, Nov 03, 2011 at 02:39:59PM -0700, Tejun Heo wrote:
> 
> This is missing all stack traces.  Can you please turn on
> FRAME_POINTER and adjust log level such that all stack dumps are
> included too?

Hi Tejun.

Sorry about that. My syslogd was sending things to different files and
being unfamiliar with sysrq-t output I thought I was sending it all.

Here's a new sysrq-t snapshot I took about 30 seconds after a bad resume.
Please let me know if I am still missing something you've asked for. 

Hope this sheds some light on what is going on.

~ Andy


[-- Attachment #2: tejun.txt --]
[-- Type: text/plain, Size: 86975 bytes --]

SysRq : Show State
  task                PC stack   pid father
init            S f53c3ad0  5380     1      0 0x00000000
 f5c35ab8 00000086 c183e0b4 f53c3ad0 f5c35ad0 c183e0b4 00000000 f5c35a68
 c11f5f74 f5c35ab8 f5c38000 f5c35a70 c104e0c6 f5c38000 00000066 f5c35ad0
 f5c35aa0 00000296 004c4b3e 00000000 f5c35ad0 00000292 00000000 00000296
Call Trace:
 [<c11f5f74>] ? timerqueue_add+0x54/0xb0
 [<c104e0c6>] ? enqueue_hrtimer+0x1e/0x4d
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d7eb>] schedule_hrtimeout_range_clock+0xab/0x110
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e3c2a>] do_select+0x556/0x665
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c10f5d0e>] ? __find_get_block+0x59/0x140
 [<c10a4297>] ? find_get_page+0x1d/0x88
 [<c10f5e27>] ? __getblk+0x32/0x33d
 [<c11430a2>] ? do_get_write_access+0x25c/0x36c
 [<c1128ae5>] ? __ext3_get_inode_loc+0xa7/0x2a7
 [<c1135a19>] ? __ext3_journal_dirty_metadata+0x19/0x3f
 [<c104b126>] ? wake_up_bit+0x5e/0x62
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c1030480>] ? dequeue_task_fair+0x180/0x255
 [<c10dd2fa>] ? __follow_mount_rcu+0x3b/0xb5
 [<c10dda39>] ? do_lookup+0x184/0x233
 [<c11998fe>] ? security_inode_exec_permission+0x1d/0x24
 [<c10dda39>] ? do_lookup+0x184/0x233
 [<c10e44c0>] core_sys_select+0x135/0x217
 [<c10de003>] ? putname+0x25/0x33
 [<c10de003>] ? putname+0x25/0x33
 [<c10e1402>] ? user_path_at+0x3f/0x62
 [<c11f9c47>] ? copy_to_user+0x2e/0x119
 [<c10d8cad>] ? cp_new_stat64+0xde/0xf0
 [<c1053354>] ? ktime_get_ts+0xb5/0xd8
 [<c10e4727>] sys_select+0x2c/0xb3
 [<c1593813>] sysenter_do_call+0x12/0x22
kthreadd        S 00000000  7172     2      0 0x00000000
 f5c41fcc 00000046 00000000 00000000 00000000 00800711 f4a0bf08 f5c41f7c
 f5c41fcc f5c41fcc f5c38c50 8854d338 00000057 f5c38c50 00000000 00000000
 00000000 c104ad92 f4a0bf08 00000000 00000000 0000007b 0000007b 00000000
Call Trace:
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593db8>] ? common_interrupt+0x38/0x38
 [<c104ad8c>] kthreadd+0xaa/0xb0
 [<c104ace2>] ? kthread_worker_fn+0xf7/0xf7
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
ksoftirqd/0     S 00000060  7576     3      2 0x00000000
 f5c43fa8 00000046 de370bf1 00000060 de462780 00000060 de462780 f5c43f58
 de370bf1 f5c43fa8 f5c398a0 00000020 00000008 f5c398a0 00000246 00000286
 00000046 00000000 f5c43fa8 c10381bb f5c398a0 00000000 f5c43fa8 00000000
Call Trace:
 [<c10381bb>] ? __do_softirq+0xc3/0x187
 [<c102007b>] ? gup_huge_pmd+0x4f/0xba
 [<c1038310>] run_ksoftirqd+0x91/0xb7
 [<c103827f>] ? __do_softirq+0x187/0x187
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:0     S 00000000  6552     5      2 0x00000000
 f5c4bf88 00000046 00000296 00000000 00000000 f51fa300 00000246 f5c4bf38
 c1050c1f f5c4bf88 f5c3b140 a26a3b4e 00000057 f5c3b140 f5c07380 00000286
 f51fa308 f51fa308 f51fa308 f5c07380 f5c4bf88 c10464da f5c4bf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
rcu_kthread     S 00000202  7444     6      2 0x00000000
 f5c4df90 00000046 00000000 00000202 c193f600 00021039 f5c4df44 f5c4df40
 c158d121 f5c4df90 f5c3bd90 b48ecaf9 00000063 f5c3bd90 00000286 00000000
 00200200 00021039 c193f600 c103e185 f5c3bd90 ffffffff ffffffff 00000000
Call Trace:
 [<c158d121>] ? schedule_timeout+0x101/0x211
 [<c103e185>] ? sys_gettid+0x13/0x13
 [<c1084869>] rcu_kthread+0x4e/0xaf
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c108481b>] ? rcu_process_callbacks+0x4f/0x4f
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
watchdog/0      S c15983e0  7732     7      2 0x00000000
 f5c4ffac 00000046 c183e0b4 c15983e0 c1840140 4365d43c 0000011a f5c4ff5c
 c104fd22 f5c4ffac f5c3c9e0 a1bfb89e 00000065 f5c3c9e0 00000000 ffffffff
 ffffffff a1bf83e9 00000065 a1bf83e9 00000065 00000000 00000001 00000000
Call Trace:
 [<c104fd22>] ? T.334+0xe/0x1a6
 [<c104ffc3>] ? sched_clock_cpu+0x7c/0xcf
 [<c10819f9>] ? watchdog_enable+0x15e/0x15e
 [<c10819f9>] ? watchdog_enable+0x15e/0x15e
 [<c10819f9>] ? watchdog_enable+0x15e/0x15e
 [<c1081a7f>] watchdog+0x86/0xb1
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
cpuset          S f5c55f58  7696     8      2 0x00000000
 f5c55f84 00000046 0112a880 f5c55f58 c102a472 00000000 0112a365 f5c55f34
 00000000 f5c55f84 f5c3d630 0112a365 00000000 f5c3d630 00000000 00000000
 010b438f 00000000 00000000 00000000 c1598420 f5c3d630 00000000 f5c55f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
khelper         S f5c57f58  7696     9      2 0x00000000
 f5c57f84 00000046 0112a880 f5c57f58 c102a472 00000000 0112a365 f5c57f34
 00000000 f5c57f84 f5c3e280 0112a365 00000000 f5c3e280 00000000 00000000
 0129c584 00000000 00000000 00000000 c1598420 f5c3e280 00000000 f5c57f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:1     S 00000000  5800    10      2 0x00000000
 f5c5bf88 00000046 00000296 00000000 00000000 f4bff000 00000246 f5c5bf38
 c1050c1f f5c5bf88 f5c3eed0 017a4d6b 00000001 f5c3eed0 f5c07780 00000286
 f4bff008 f4bff008 f4bff008 f5c07780 f5c5bf88 c10464da f5c5bfbc 00000086
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
netns           S f5c69f58  7696    13      2 0x00000000
 f5c69f84 00000046 0112a880 f5c69f58 c102a472 00000000 0112a365 f5c69f34
 00000000 f5c69f84 f5c618a0 0112a365 00000000 f5c618a0 00000000 00000000
 01854b63 00000000 00000000 00000000 c1598420 f5c618a0 00000000 f5c69f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
sync_supers     S f5cbdfbc  7712   227      2 0x00000000
 f5cbdfb0 00000046 f5c3eed0 f5cbdfbc c158c95a f5cbdf50 c1831400 f5cbdf60
 00000000 f5cbdfb0 f5cc0000 7079df72 0000005c f5cc0000 01b9ea4a 00000000
 f5cc0000 f5c35f28 f5c35f64 00000000 f5cbdf80 c102fb31 f5cbdfa8 c10265f3
Call Trace:
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c102fb31>] ? default_wake_function+0xb/0xd
 [<c10265f3>] ? __wake_up_common+0x40/0x63
 [<c10b761e>] ? bdi_sched_wait+0xc/0xc
 [<c10b763e>] bdi_sync_supers+0x20/0x35
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
bdi-default     S f5cbfe70  7392   229      2 0x00000000
 f5cbfedc 00000046 00000001 f5cbfe70 c1029e3c f5cc251c c183ae60 f5cbfe8c
 c103015e f5cbfedc f5cc18a0 d432dc83 00000057 f5cc18a0 f5cbfea8 00030222
 d842a1c5 f5cc18a0 d842a1c5 f5cc18a0 f4a5c9e0 f5cbff30 c158c95a ae763538
Call Trace:
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c158d11a>] schedule_timeout+0xfa/0x211
 [<c103e185>] ? sys_gettid+0x13/0x13
 [<c10b8040>] bdi_forker_thread+0x163/0x37f
 [<c10f131f>] ? wb_do_writeback+0x1aa/0x1aa
 [<c10b7edd>] ? bdi_debug_stats_show+0x12a/0x12a
 [<c10b7edd>] ? bdi_debug_stats_show+0x12a/0x12a
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kblockd         S f5cbbf58  7660   231      2 0x00000000
 f5cbbf84 00000046 0112a880 f5cbbf58 c102a472 00000000 01bf7999 f5cbbf34
 00000000 f5cbbf84 f5cc3140 01bf7084 00000000 f5cc3140 00000000 00000000
 1b30fb4e 00000000 00000000 00000000 c1598420 f5cc3140 00000000 f5cbbf6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
ata_sff         S f5cf1f58  7660   318      2 0x00000000
 f5cf1f84 00000046 0112a880 f5cf1f58 c102a472 00000000 05c33ed1 f5cf1f34
 00000000 f5cf1f84 f5c624f0 05c3366b 00000000 f5c624f0 00000000 00000000
 25a917a5 00000000 00000000 00000000 c1598420 f5c624f0 00000000 f5cf1f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
khubd           S c13f0020  5880   328      2 0x00000000
 f5cfff00 00000046 5d353134 c13f0020 00013a10 f4a51310 00000004 f5cffeb0
 f5cffec0 f5cfff00 f5cc24f0 d432e7f3 00000057 f5cc24f0 c1865650 f5204000
 f5cffec8 c158c4d9 c17a45c0 f5cffed4 f5cffeec c133525b c17a45c0 c1770eab
Call Trace:
 [<c13f0020>] ? usb_free_coherent+0xc/0x27
 [<c158c4d9>] ? printk+0x28/0x2a
 [<c133525b>] ? __dev_printk+0x38/0x72
 [<c13f3d1b>] hub_thread+0xdd4/0x10a8
 [<c1020002>] ? __virt_addr_valid+0x3e/0x68
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c13f2f47>] ? hub_disconnect+0xa1/0xa1
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
md              S f5ce1f58  7660   335      2 0x00000000
 f5ce1f84 00000046 0112a880 f5ce1f58 c102a472 00000000 05d280a2 f5ce1f34
 00000000 f5ce1f84 f5c649e0 05d27744 00000000 f5c649e0 00000000 00000000
 27b15239 00000000 00000000 00000000 c1598420 f5c649e0 00000000 f5ce1f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
cfg80211        S f5cfdf58  7660   345      2 0x00000000
 f5cfdf84 00000046 0112a880 f5cfdf58 c102a472 00000000 05e79665 f5cfdf34
 00000000 f5cfdf84 f5d03d90 05e04da8 00000000 f5d03d90 00000000 00000000
 28e36376 00000000 00000000 00000000 c1598420 f5d03d90 00000000 f5cfdf6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/0:1     S f5d09f20  6860   346      2 0x00000000
 f5d09f88 00000046 00000720 f5d09f20 c122308c f4a7d400 f5f18400 f5d09f38
 f5d09f50 f5d09f88 f5d049e0 00000007 00000000 f5d049e0 c1228a68 00000000
 00000002 f5f18594 f5f18594 f5ca66c0 f5d09f88 c10464da f5d09f64 c104694b
Call Trace:
 [<c122308c>] ? get_color+0x1a/0x100
 [<c1228a68>] ? bit_putcs+0x41e/0x41e
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1223172>] ? get_color+0x100/0x100
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
rpciod          S f5ce5f58  7660   444      2 0x00000000
 f5ce5f84 00000046 0112a880 f5ce5f58 c102a472 00000000 0a23fc4a f5ce5f34
 00000000 f5ce5f84 f5c60000 0a23ecf5 00000000 f5c60000 00000000 00000000
 347d944e 00000000 00000000 00000000 c1598420 f5c60000 00000000 f5ce5f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
khungtaskd      S 00000000  7564   461      2 0x00000000
 f5d0bf1c 00000046 00000000 00000000 00000000 367e65f4 f4a118a0 f5d0becc
 00000000 f5d0bf1c f5d024f0 ae063402 00000054 f5d024f0 f5d0bf0c c1030550
 00000000 00000000 00000001 f5c38034 c183ae60 00000001 f5d0befc c1029e3c
Call Trace:
 [<c1030550>] ? dequeue_task_fair+0x250/0x255
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c158d11a>] schedule_timeout+0xfa/0x211
 [<c103e185>] ? sys_gettid+0x13/0x13
 [<c158d271>] schedule_timeout_interruptible+0x14/0x23
 [<c108145b>] watchdog+0x3a/0x1ba
 [<c1081421>] ? hung_task_panic+0x11/0x11
 [<c1081421>] ? hung_task_panic+0x11/0x11
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kswapd0         S f5d25ea8  7444   467      2 0x00000000
 f5d25f08 00000046 f5cc24f0 f5d25ea8 f5d25ee4 f5d25ef8 c193f600 f5d25eb8
 f5d25ec4 f5d25f08 f5d018a0 11d9adb8 00000000 f5d018a0 00000000 00000002
 f5d25edc c10a73a0 00000002 00000000 0003bc9c c1874f20 00000002 f5d25f08
Call Trace:
 [<c10a73a0>] ? zone_watermark_ok_safe+0x35/0x3c
 [<c10aeadb>] ? sleeping_prematurely+0xb2/0xe5
 [<c10b197e>] kswapd+0x848/0x909
 [<c1026cee>] ? check_preempt_curr+0x6d/0x7e
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c10b1136>] ? mem_cgroup_shrink_node_zone+0xdd/0xdd
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
fsnotify_mark   S 00000000  7668   532      2 0x00000000
 f5de3f84 00000046 f5cc3d90 00000000 00000000 f5c38000 f5c3bd90 f5de3f34
 00030222 f5de3f84 f5cc3d90 297d5c40 00000007 f5cc3d90 f5de3fbc 00000282
 f51d4b88 f5c05c00 f6fa0a80 f5de3f74 c10ce522 c110067e f6fa4420 f53a1120
Call Trace:
 [<c10ce522>] ? kmem_cache_free+0x98/0xc6
 [<c110067e>] ? inotify_free_mark+0xf/0x11
 [<c110067e>] ? inotify_free_mark+0xf/0x11
 [<c10ff297>] fsnotify_mark_destroy+0xbd/0xed
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c10ff1da>] ? fsnotify_duplicate_mark+0x21/0x21
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
nfsiod          S f5f05f58  7660   544      2 0x00000000
 f5f05f84 00000046 0112a880 f5f05f58 c102a472 00000000 0c1c22e8 f5f05f34
 00000000 f5f05f84 f5d03140 0c1c1955 00000000 f5d03140 00000000 00000000
 406b4d3e 00000000 00000000 00000000 c1598420 f5d03140 00000000 f5f05f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
crypto          S f5f1bf58  7660   549      2 0x00000000
 f5f1bf84 00000046 0112a880 f5f1bf58 c102a472 00000000 0c245561 f5f1bf34
 00000000 f5f1bf84 f5d06280 0c244ef5 00000000 f5d06280 00000000 00000000
 410460d1 00000000 00000000 00000000 c1598420 f5d06280 00000000 f5f1bf6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
ttm_swap        S f5f15f58  7660   628      2 0x00000000
 f5f15f84 00000046 0112a880 f5f15f58 c102a472 00000000 15e6af05 f5f15f34
 00000000 f5f15f84 f5c63d90 15e69fdf 00000000 f5c63d90 00000000 00000000
 4f4e984b 00000000 00000000 00000000 c1598420 f5c63d90 00000000 f5f15f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/0:2     S 000ec000  7144   768      2 0x00000000
 f578df88 00000046 c18528a0 000ec000 f578df3c 00000004 00000000 f578df38
 00000000 f578df88 f5c60c50 c970c873 00000054 f5c60c50 00000000 f57801c0
 c183dd00 f578df50 c1852918 f57801c0 f578df88 c10464da f578df64 c104694b
Call Trace:
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1268348>] ? do_blank_screen+0x1ee/0x1ee
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kpsmoused       S f5725f58  7660   776      2 0x00000000
 f5725f84 00000046 0112a880 f5725f58 c102a472 00000000 83a41e53 f5725f34
 00000000 f5725f84 f5c66ed0 83a41658 00000000 f5c66ed0 00000000 00000000
 698ff881 00000000 00000000 00000000 c1598420 f5c66ed0 00000000 f5725f6c
Call Trace:
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c102df09>] ? T.1453+0x48/0x54
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046c11>] rescuer_thread+0x1cc/0x20c
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c1046a45>] ? idle_worker_timeout+0x58/0x58
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kjournald       S 00000059  6524   834      2 0x00000000
 f578ff78 00000046 00000000 00000059 2d0ecba3 00000059 f53e9e40 f578ff28
 f57c5a94 f578ff78 f57b0000 f578ff18 f578ff20 f57b0000 00000000 3234382e
 5d333330 f57c0020 f578ff60 c10265f3 00000744 00000001 00000003 f57c5a40
Call Trace:
 [<c10265f3>] ? __wake_up_common+0x40/0x63
 [<c1146c6d>] kjournald+0x20a/0x337
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c1146a63>] ? commit_timeout+0xa/0xa
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
udevd           S f500de68  6192   866      1 0x00000000
 f500decc 00000086 f53cf2f0 f500de68 0000000a f500deb4 f500deb0 f500de7c
 f500de74 f500decc f57b5630 d7724d1c 00000057 f57b5630 00000040 00000000
 00000000 bf9bc4a0 f500deb8 00000000 f500dea8 00000001 00000000 00000000
Call Trace:
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c11018c1>] ? ep_send_events_proc+0x5c/0xf6
 [<c1101f19>] ? ep_scan_ready_list+0xed/0x108
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c11021dc>] sys_epoll_wait+0x297/0x2fa
 [<c1468b03>] ? sys_recv+0x36/0x38
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1593813>] sysenter_do_call+0x12/0x22
pccardd         S f5309f24  7344  1052      2 0x00000000
 f5309f84 00000046 f5d018a0 f5309f24 f5309f30 f5309f74 f5148c50 f5309f34
 00000056 f5309f84 f5148c50 d432fc14 00000057 f5148c50 00200200 fffb8eff
 c193f600 c103e185 f5148c50 ffffffff ffffffff 00000000 00000000 00000000
Call Trace:
 [<c103e185>] ? sys_gettid+0x13/0x13
 [<f8366d9e>] pccardd+0x1df/0x33c [pcmcia_core]
 [<f8366bbf>] ? pcmcia_register_socket+0x2b5/0x2b5 [pcmcia_core]
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
pccardd         S f5079f24  7344  1054      2 0x00000000
 f5079f84 00000046 f5148c50 f5079f24 f5079f30 f5079f74 f514bd90 f5079f34
 00000056 f5079f84 f514bd90 d432ff6e 00000057 f514bd90 c1026cee f514a4f0
 00000000 00000292 f5079f6c 00000292 ffffffff 00000000 00000001 01000000
Call Trace:
 [<c1026cee>] ? check_preempt_curr+0x6d/0x7e
 [<f8366d9e>] pccardd+0x1df/0x33c [pcmcia_core]
 [<f8366bbf>] ? pcmcia_register_socket+0x2b5/0x2b5 [pcmcia_core]
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
flush-3:0       S 00000000  6432  1118      2 0x00000000
 f52e3f84 00000046 000003f3 00000000 00000000 00000000 f52e3f24 f52e3f34
 0000000d f52e3f84 f5176280 c10aac6f 0000000d f5176280 00000000 f52e3f84
 c10f1217 00000000 00000000 f5729b64 f5729b04 c103e185 000018e0 00000000
Call Trace:
 [<c10aac6f>] ? global_dirty_limits+0x23/0xb8
 [<c10f1217>] ? wb_do_writeback+0xa2/0x1aa
 [<c103e185>] ? sys_gettid+0x13/0x13
 [<c10f1430>] bdi_writeback_thread+0x111/0x1ca
 [<c10f131f>] ? wb_do_writeback+0x1aa/0x1aa
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
syslogd         S 00000000  6164  1169      1 0x00000000
 f57e1ab8 00000086 00000002 00000000 00000000 01280000 c1831400 f57e1a68
 f57c5a00 f57e1ab8 f51749e0 d8afc9e1 00000057 f51749e0 c19ece40 f57e1aac
 c1128ae5 f5a412b0 f5a19930 f5896000 f57e1aa0 c1135a19 f57e1a98 f5a19930
Call Trace:
 [<c1128ae5>] ? __ext3_get_inode_loc+0xa7/0x2a7
 [<c1135a19>] ? __ext3_journal_dirty_metadata+0x19/0x3f
 [<c10f5d0e>] ? __find_get_block+0x59/0x140
 [<c11288b4>] ? ext3_mark_iloc_dirty+0x1c2/0x34c
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c10f5e27>] ? __getblk+0x32/0x33d
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c11430a2>] ? do_get_write_access+0x25c/0x36c
 [<c14eba78>] ? unix_dgram_poll+0x22/0x154
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e3c2a>] do_select+0x556/0x665
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c11430a2>] ? do_get_write_access+0x25c/0x36c
 [<c1128ae5>] ? __ext3_get_inode_loc+0xa7/0x2a7
 [<c1135a19>] ? __ext3_journal_dirty_metadata+0x19/0x3f
 [<c104b126>] ? wake_up_bit+0x5e/0x62
 [<c1141f59>] ? journal_stop+0x1dc/0x313
 [<c1141f59>] ? journal_stop+0x1dc/0x313
 [<c1129250>] ? ext3_dirty_inode+0x54/0xa8
 [<c1128560>] ? ext3_journal_dirty_data+0x12/0x37
 [<c10f62b7>] ? __block_commit_write+0x66/0x8b
 [<c10f0b3b>] ? __mark_inode_dirty+0x24/0x18f
 [<c10a44ea>] ? unlock_page+0x46/0x49
 [<c112b1f9>] ? ext3_ordered_write_end+0x118/0x1a8
 [<c1128585>] ? ext3_journal_dirty_data+0x37/0x37
 [<c10a3d39>] ? generic_file_buffered_write+0x15d/0x1d6
 [<c10a5ea2>] ? __generic_file_aio_write+0x1dc/0x4a7
 [<c1040506>] ? __sigqueue_free+0x2c/0x2f
 [<c1008466>] ? save_i387_fxsave+0x74/0x81
 [<c10e44c0>] core_sys_select+0x135/0x217
 [<c10083e8>] ? restore_i387_fxsave+0x71/0x7b
 [<c1008644>] ? restore_i387_xstate+0xbe/0x1dd
 [<c1036dde>] ? do_setitimer+0x159/0x182
 [<c11f9c47>] ? copy_to_user+0x2e/0x119
 [<c10e4727>] sys_select+0x2c/0xb3
 [<c1593813>] sysenter_do_call+0x12/0x22
klogd           S 0000004d  6532  1173      1 0x00000000
 f5399ef4 00000086 f5399ea4 0000004d f5939600 f5399e84 00000000 f5399ea4
 f4a18000 f5399ef4 f5170000 0001cea7 f5399e30 f5170000 00000000 00000000
 f5399f50 00000001 00000000 00000000 00000000 f5399ee0 00000000 f50d8780
Call Trace:
 [<c1034516>] do_syslog+0x392/0x3ea
 [<c1199e26>] ? security_file_permission+0x8a/0x8c
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c111c38c>] ? kmsg_poll+0x3b/0x3b
 [<c111c3b3>] kmsg_read+0x27/0x50
 [<c1113c4e>] proc_reg_read+0x39/0x4c
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c1113c15>] ? proc_reg_write+0x4c/0x4c
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
inetd           S 00000000  6440  1285      1 0x00000000
 f53a9ab8 00000086 00000000 00000000 00000000 00000000 f5170000 f53a9a68
 00390020 f53a9ab8 f5d00c50 d4337f22 00000057 f5d00c50 00000002 00000000
 00000000 00e40080 f50d9098 f50d9c78 f53a9ab8 c10cdd69 c11430a2 00000002
Call Trace:
 [<c10cdd69>] ? kmem_cache_alloc+0x9d/0xf5
 [<c11430a2>] ? do_get_write_access+0x25c/0x36c
 [<c11ed958>] ? cfq_init_prio_data+0x63/0xf1
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c10cdd69>] ? kmem_cache_alloc+0x9d/0xf5
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e3c2a>] do_select+0x556/0x665
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
last message repeated 3 times
 [<c10d2dd8>] ? T.941+0x3d/0x67
 [<c10d2e54>] ? mem_cgroup_charge_common+0x52/0x60
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c127048f>] ? extract_buf+0x83/0x109
 [<c1030480>] ? dequeue_task_fair+0x180/0x255
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c10aba4a>] ? ra_submit+0x21/0x27
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c10e44c0>] core_sys_select+0x135/0x217
 [<c10a44ea>] ? unlock_page+0x46/0x49
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c10bccd5>] ? handle_pte_fault+0x7a/0x604
 [<c10c20c2>] ? unlink_anon_vmas+0x41/0x97
 [<c10bd2da>] ? handle_mm_fault+0x7b/0xb8
 [<c1590895>] ? do_page_fault+0x191/0x3dc
 [<c10406a7>] ? do_sigaction+0x125/0x156
 [<c10bf057>] ? remove_vma+0x44/0x53
 [<c11f9c47>] ? copy_to_user+0x2e/0x119
 [<c10e4727>] sys_select+0x2c/0xb3
 [<c1002bc6>] ? do_notify_resume+0x4d/0x67
 [<c1593813>] sysenter_do_call+0x12/0x22
sshd            S 00000000  6440  1288      1 0x00000000
 f53abab8 00200086 00000000 00000000 00000000 00000000 f5d00c50 f53aba68
 00390021 f53abab8 f5d05630 d433829a 00000057 f5d05630 00000002 00000000
 00000000 00e40084 f5a4c340 f591f9a0 00000000 f53abad8 c11430a2 00000002
Call Trace:
 [<c11430a2>] ? do_get_write_access+0x25c/0x36c
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c1128ae5>] ? __ext3_get_inode_loc+0xa7/0x2a7
 [<c1135a19>] ? __ext3_journal_dirty_metadata+0x19/0x3f
 [<c10f5d0e>] ? __find_get_block+0x59/0x140
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e3c2a>] do_select+0x556/0x665
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c10f5d0e>] ? __find_get_block+0x59/0x140
 [<c10f5e27>] ? __getblk+0x32/0x33d
 [<c11430a2>] ? do_get_write_access+0x25c/0x36c
 [<c1128ae5>] ? __ext3_get_inode_loc+0xa7/0x2a7
 [<c1135a19>] ? __ext3_journal_dirty_metadata+0x19/0x3f
 [<c104b126>] ? wake_up_bit+0x5e/0x62
 [<c1141f59>] ? journal_stop+0x1dc/0x313
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c1141f59>] ? journal_stop+0x1dc/0x313
 [<c11f51e6>] ? rb_erase+0x1e3/0x27a
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c10e44c0>] core_sys_select+0x135/0x217
 [<c10a44ea>] ? unlock_page+0x46/0x49
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c10bccd5>] ? handle_pte_fault+0x7a/0x604
 [<c10c20c2>] ? unlink_anon_vmas+0x41/0x97
 [<c10bd2da>] ? handle_mm_fault+0x7b/0xb8
 [<c1590895>] ? do_page_fault+0x191/0x3dc
 [<c10bf057>] ? remove_vma+0x44/0x53
 [<c10bf057>] ? remove_vma+0x44/0x53
 [<c10e4727>] sys_select+0x2c/0xb3
 [<c1002bc6>] ? do_notify_resume+0x4d/0x67
 [<c1593813>] sysenter_do_call+0x12/0x22
acpid           S f530e750  6072  1295      1 0x00000000
 f52e1b50 00000086 00000001 f530e750 00000000 f578a000 f5173140 f52e1b00
 ffffffff f52e1b50 f5d00000 d4dc3330 00000057 f5d00000 000000d0 00011200
 f5734180 00011210 f52e1b3c f52e1b1c c10a627b f52e1b50 c10a636b f578a228
Call Trace:
 [<c10a627b>] ? mempool_alloc_slab+0xe/0x10
 [<c10a636b>] ? mempool_alloc+0x3d/0xc5
 [<c11ef27a>] ? changed_ioprio+0x53/0x53
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c11e1ee3>] ? drive_stat_acct+0xab/0x110
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e4101>] do_sys_poll+0x3c8/0x459
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c11430a2>] ? do_get_write_access+0x25c/0x36c
 [<c1128ae5>] ? __ext3_get_inode_loc+0xa7/0x2a7
 [<c1135a19>] ? __ext3_journal_dirty_metadata+0x19/0x3f
 [<c104b126>] ? wake_up_bit+0x5e/0x62
 [<c1141f59>] ? journal_stop+0x1dc/0x313
 [<c1141f59>] ? journal_stop+0x1dc/0x313
 [<c1129250>] ? ext3_dirty_inode+0x54/0xa8
 [<c1128560>] ? ext3_journal_dirty_data+0x12/0x37
 [<c10f62b7>] ? __block_commit_write+0x66/0x8b
 [<c10f0b3b>] ? __mark_inode_dirty+0x24/0x18f
 [<c10a44ea>] ? unlock_page+0x46/0x49
 [<c112b1f9>] ? ext3_ordered_write_end+0x118/0x1a8
 [<c1128585>] ? ext3_journal_dirty_data+0x37/0x37
 [<c10a3d39>] ? generic_file_buffered_write+0x15d/0x1d6
 [<c10a5ea2>] ? __generic_file_aio_write+0x1dc/0x4a7
 [<c10d598f>] ? do_sync_write+0x9b/0xc8
 [<c10d8cad>] ? cp_new_stat64+0xde/0xf0
 [<c10fe58d>] ? fsnotify+0x1a0/0x297
 [<c10d5f02>] ? vfs_write+0xea/0x14f
 [<c10d58f4>] ? do_sync_readv_writev+0xbe/0xbe
 [<c10e42ed>] sys_poll+0x5a/0xb6
 [<c1593813>] sysenter_do_call+0x12/0x22
dbus-daemon     S 00000057  6356  1303      1 0x00000000
 f53b1b50 00000086 000ee1df 00000057 00000000 c1838e60 f5c3bd90 f53b1b00
 d47b9921 f53b1b50 f5c65630 e3cd33e6 00000060 f5c65630 c1838e60 00000057
 c183e0d8 f53b1b1c c105815e 00000000 00000057 f53b1b68 c104e50b f53b1b44
Call Trace:
 [<c105815e>] ? tick_program_event+0x1b/0x21
 [<c104e50b>] ? hrtimer_interrupt+0x161/0x24c
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c1015f4b>] ? smp_apic_timer_interrupt+0x46/0x77
 [<c158e45d>] ? apic_timer_interrupt+0x31/0x38
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e4101>] do_sys_poll+0x3c8/0x459
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
last message repeated 4 times
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c147021f>] ? memcpy_fromiovec+0x3f/0x60
 [<c14ebcc1>] ? scm_destroy+0xd/0x44
 [<c14ec11e>] ? unix_stream_sendmsg+0x30c/0x31c
 [<c1468666>] ? sock_sendmsg+0xcd/0xf5
 [<c100d5f3>] ? p4_pmu_enable_all+0x36/0x44
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c146ff58>] ? verify_iovec+0x44/0x92
 [<c1469110>] ? sys_sendmsg+0x149/0x253
 [<c105722a>] ? clockevents_program_event+0x80/0x10a
 [<c105815e>] ? tick_program_event+0x1b/0x21
 [<c104e50b>] ? hrtimer_interrupt+0x161/0x24c
 [<c1038054>] ? irq_exit+0x2f/0x7d
 [<c1468fa4>] ? sys_recvmsg+0x36/0x59
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c1053354>] ? ktime_get_ts+0xb5/0xd8
 [<c10e42ed>] sys_poll+0x5a/0xb6
 [<c1049fed>] ? sys_clock_gettime+0x83/0x96
 [<c1593813>] sysenter_do_call+0x12/0x22
hald            S f5294044  6248  1308      1 0x00000000
 f53b5b50 00000086 c183e0b4 f5294044 f53b5b68 c183e0b4 00000000 f53b5b00
 c11f5f74 f53b5b50 f514e280 f53b5b08 c104e0c6 f514e280 00000067 f53b5b68
 f53b5b38 00000282 01ba796e 00000000 00000000 00000060 00000000 00000282
Call Trace:
 [<c11f5f74>] ? timerqueue_add+0x54/0xb0
 [<c104e0c6>] ? enqueue_hrtimer+0x1e/0x4d
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d7eb>] schedule_hrtimeout_range_clock+0xab/0x110
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e4101>] do_sys_poll+0x3c8/0x459
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
last message repeated 12 times
 [<c11f63a8>] ? put_dec+0x87/0x8f
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c103075c>] ? __enqueue_rt_entity+0x207/0x218
 [<c1469162>] ? sys_sendmsg+0x19b/0x253
 [<c100d5f3>] ? p4_pmu_enable_all+0x36/0x44
 [<c1015865>] ? lapic_next_event+0x16/0x1a
 [<c105722a>] ? clockevents_program_event+0x80/0x10a
 [<c105801a>] ? tick_dev_program_event+0x3c/0x135
 [<c104e171>] ? __run_hrtimer+0x7c/0x189
 [<c105815e>] ? tick_program_event+0x1b/0x21
 [<c1053354>] ? ktime_get_ts+0xb5/0xd8
 [<c10e42ed>] sys_poll+0x5a/0xb6
 [<c1593813>] sysenter_do_call+0x12/0x22
hald            S f77c5c24  7076  1310      1 0x00000000
 f53b9e50 00000086 c158c95a f77c5c24 c1874f20 c1874f20 f514e280 f53b9e00
 f53b9e04 f53b9e50 f514c9e0 d433936c 00000057 f514c9e0 f514c9e0 f514c9e0
 f51d0994 c183ae60 00000001 f53b9e2c c10424b9 f53b9efc f514cd44 f514cd44
Call Trace:
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c1590704>] ? spurious_fault+0xc3/0xc3
 [<c10dc135>] pipe_wait+0x46/0x5d
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c10dc8dd>] pipe_read+0x265/0x3bb
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c10d5a57>] do_sync_read+0x9b/0xc8
 [<c10d5ae1>] ? rw_verify_area+0x5d/0x101
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c10d59bc>] ? do_sync_write+0xc8/0xc8
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
hald-runner     S c11f5171  6424  1309   1308 0x00000000
 f53b7b50 00000086 c183e0b4 c11f5171 f5294044 c183e0b4 f514e280 f53b7b00
 c183e0b4 f53b7b50 f514d630 d4576448 00000057 f514d630 f53b7b24 c104dfa0
 f53b7b38 00000282 01304a6e 00000000 f53b7b68 00000292 c183e0ac f53b7b44
Call Trace:
 [<c11f5171>] ? rb_erase+0x16e/0x27a
 [<c104dfa0>] ? __remove_hrtimer+0x24/0x81
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e4101>] do_sys_poll+0x3c8/0x459
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c102fb31>] ? default_wake_function+0xb/0xd
 [<c10e3601>] ? pollwake+0x4f/0x56
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c10265f3>] ? __wake_up_common+0x40/0x63
 [<c14ebcc1>] ? scm_destroy+0xd/0x44
 [<c14ec11e>] ? unix_stream_sendmsg+0x30c/0x31c
 [<c1468666>] ? sock_sendmsg+0xcd/0xf5
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c146ff58>] ? verify_iovec+0x44/0x92
 [<c1469110>] ? sys_sendmsg+0x149/0x253
 [<c104292b>] ? get_signal_to_deliver+0x87/0x3ac
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c10bccd5>] ? handle_pte_fault+0x7a/0x604
 [<c10bd2da>] ? handle_mm_fault+0x7b/0xb8
 [<c1468fa4>] ? sys_recvmsg+0x36/0x59
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c146937c>] ? sys_socketcall+0xd5/0x289
 [<c10e42ed>] sys_poll+0x5a/0xb6
 [<c1593813>] sysenter_do_call+0x12/0x22
hald-addon-inpu S 02222764  6356  1344   1309 0x00000000
 f53bbb50 00000086 c105722a 02222764 00000000 000f0742 c1831400 f53bbb00
 c1838e60 f53bbb50 f57b0c50 87e6da26 00000065 f57b0c50 c104e171 00000000
 01053453 c1838e60 0000004e c183e0d8 f53bbb28 c105815e 00000000 0000004e
Call Trace:
 [<c105722a>] ? clockevents_program_event+0x80/0x10a
 [<c104e171>] ? __run_hrtimer+0x7c/0x189
 [<c105815e>] ? tick_program_event+0x1b/0x21
 [<c104e50b>] ? hrtimer_interrupt+0x161/0x24c
 [<c105815e>] ? tick_program_event+0x1b/0x21
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c1038054>] ? irq_exit+0x2f/0x7d
 [<c1015f4b>] ? smp_apic_timer_interrupt+0x46/0x77
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e4101>] do_sys_poll+0x3c8/0x459
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
last message repeated 7 times
 [<c11f51e6>] ? rb_erase+0x1e3/0x27a
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c1015865>] ? lapic_next_event+0x16/0x1a
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c100d5f3>] ? p4_pmu_enable_all+0x36/0x44
 [<c1199e26>] ? security_file_permission+0x8a/0x8c
 [<c10d5ae1>] ? rw_verify_area+0x5d/0x101
 [<c10d60cc>] ? vfs_read+0x89/0x14c
 [<c1411eb6>] ? evdev_write+0x9c/0x9c
 [<c10e42ed>] sys_poll+0x5a/0xb6
 [<c1593813>] sysenter_do_call+0x12/0x22
hald-addon-gene S 00000010  6572  1346   1309 0x00000000
 f53bfb50 00000086 c183e0b4 00000010 f5294044 c183e0b4 f57b0c50 f53bfb00
 c183e0b4 f53bfb50 f57b3d90 d433a08e 00000057 f57b3d90 f53bfb24 c104dfa0
 f53bfb38 00000282 017d783e 00000000 f53bfb68 00000292 c183e0ac f53bfb44
Call Trace:
 [<c104dfa0>] ? __remove_hrtimer+0x24/0x81
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e4101>] do_sys_poll+0x3c8/0x459
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c10ce1aa>] ? __kmalloc_track_caller+0xeb/0x18d
 [<c146b6ec>] ? sock_wfree+0x3f/0x46
 [<c14ebe0a>] ? unix_destruct_scm+0xab/0xb3
 [<c146d3d5>] ? skb_release_data+0x63/0x86
 [<c146d052>] ? __kfree_skb+0x38/0x7b
 [<c14eca98>] ? scm_recv+0x2c/0x139
 [<c146d052>] ? __kfree_skb+0x38/0x7b
 [<c14ece26>] ? unix_stream_recvmsg+0x281/0x4b1
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c1030480>] ? dequeue_task_fair+0x180/0x255
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c10a511b>] ? filemap_fault+0x8f/0x389
 [<c104292b>] ? get_signal_to_deliver+0x87/0x3ac
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c10bccd5>] ? handle_pte_fault+0x7a/0x604
 [<c10d8cad>] ? cp_new_stat64+0xde/0xf0
 [<c10bd2da>] ? handle_mm_fault+0x7b/0xb8
 [<c1590895>] ? do_page_fault+0x191/0x3dc
 [<c1469364>] ? sys_socketcall+0xbd/0x289
 [<c10e4374>] do_restart_poll+0x2b/0x42
 [<c103f71e>] sys_restart_syscall+0x11/0x13
 [<c1593813>] sysenter_do_call+0x12/0x22
hald-addon-acpi S 00000001  6616  1357   1309 0x00000000
 f539bd64 00000086 f5a0ed00 00000001 f539bd20 c10dda39 f57b3d90 f539bd14
 f5880f10 f539bd64 f57b18a0 d433a490 00000057 f57b18a0 c1026eed c1026eed
 f539bdf4 f539bee3 0000000c f20c7a7c 00000003 ffffe80c ffffffff 00000056
Call Trace:
 [<c10dda39>] ? do_lookup+0x184/0x233
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c158d15d>] schedule_timeout+0x13d/0x211
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c14ecf14>] unix_stream_recvmsg+0x36f/0x4b1
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c1467036>] sock_aio_read+0x14a/0x16d
 [<c10d5a57>] do_sync_read+0x9b/0xc8
 [<c10d5ae1>] ? rw_verify_area+0x5d/0x101
 [<c10d617e>] vfs_read+0x13b/0x14c
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
crond           S f4a1a000  6556  1365      1 0x00000000
 f53b3ef0 00000086 c183e0b4 f4a1a000 f53b3f40 c183e0b4 00000000 f53b3ea0
 c11f5f74 f53b3ef0 f57b6ed0 556da3fa 00000065 f57b6ed0 00000066 f53b3f40
 f53b3ed8 00000282 0000c350 00000000 f53b3ec0 00000001 00000000 00000282
Call Trace:
 [<c11f5f74>] ? timerqueue_add+0x54/0xb0
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d8f3>] do_nanosleep+0x81/0xab
 [<c104ebb3>] hrtimer_nanosleep+0x9c/0x11e
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c104ec8c>] sys_nanosleep+0x57/0x5b
 [<c1593813>] sysenter_do_call+0x12/0x22
atd             S f53c1fb4  6532  1367      1 0x00000000
 f53c1f28 00000086 c183e0b4 f53c1fb4 f53c1f54 c183e0b4 00000000 f53c1ed8
 c11f5f74 f53c1f28 f57b6280 d433abd7 00000057 f57b6280 0000034d f53c1f54
 f53c1f10 00000296 00000000 00000000 f57b65e4 00000292 00000000 00000296
Call Trace:
 [<c11f5f74>] ? timerqueue_add+0x54/0xb0
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d8f3>] do_nanosleep+0x81/0xab
 [<c158d95c>] hrtimer_nanosleep_restart+0x3f/0x7a
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c103f71e>] sys_restart_syscall+0x11/0x13
 [<c1593813>] sysenter_do_call+0x12/0x22
sendmail        S f53c3ad0  6244  1370      1 0x00000000
 f53c3ab8 00200086 c183e0b4 f53c3ad0 f53c3ad0 c183e0b4 00000000 f53c3a68
 c11f5f74 f53c3ab8 f57b3140 ded80853 00000064 f57b3140 00000066 f53c3ad0
 f53c3aa0 00200296 004c4b3e 00000000 f53c3ad0 00200292 00000000 00200296
Call Trace:
 [<c11f5f74>] ? timerqueue_add+0x54/0xb0
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d7eb>] schedule_hrtimeout_range_clock+0xab/0x110
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e3c2a>] do_select+0x556/0x665
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c127048f>] ? extract_buf+0x83/0x109
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c10cdcf0>] ? kmem_cache_alloc+0x24/0xf5
 [<c10cdde8>] ? kmem_cache_alloc_trace+0x27/0xf0
 [<c10e7b52>] ? inode_init_always+0xef/0x183
 [<c11f63a8>] ? put_dec+0x87/0x8f
 [<c11f6702>] ? number+0x352/0x366
 [<c1119e0a>] ? devinfo_show+0x22/0x68
 [<c1113984>] ? proc_reg_open+0x61/0xe1
 [<c1199c8c>] ? security_dentry_open+0x65/0x67
 [<c10e44c0>] core_sys_select+0x135/0x217
 [<c10ec9ed>] ? seq_printf+0x27/0x4a
 [<c1119f8b>] ? loadavg_proc_show+0xb0/0xc2
 [<c10fe76b>] ? __fsnotify_inode_delete+0x8/0xd
 [<c111372b>] ? proc_destroy_inode+0x10/0x12
 [<c10e7a4c>] ? destroy_inode+0x26/0x3d
 [<c10e7dc5>] ? evict+0x9a/0xff
 [<c10e4e8b>] ? d_free+0x37/0x49
 [<c10e6c24>] ? dput+0x107/0x1b3
 [<c10b6592>] ? kzfree+0x22/0x25
 [<c1053354>] ? ktime_get_ts+0xb5/0xd8
 [<c10e4727>] sys_select+0x2c/0xb3
 [<c1593813>] sysenter_do_call+0x12/0x22
sendmail        S f50ea000  6176  1373      1 0x00000000
 f53cdfa4 00200086 080950d1 f50ea000 f53cdfac c1590895 f57b3140 f53cdf54
 f57b4c10 f53cdfa4 f57b49e0 d433b2f3 00000057 f57b49e0 00000008 00000004
 00000005 00000000 00000000 bf950b68 bf950b94 00000000 f53cdf9c f53cdf8c
Call Trace:
 [<c1590895>] ? do_page_fault+0x191/0x3dc
 [<c1001eaf>] ? restore_sigcontext+0xc1/0xd9
 [<c10402ce>] sys_pause+0x13/0x1a
 [<c1593813>] sysenter_do_call+0x12/0x22
udevd           S f53bde8c  6592  1379    866 0x00000000
 f53bdecc 00000086 f53bdd84 f53bde8c bf9bb92c 00000000 09c71835 f53bde7c
 f53bdeb4 f53bdecc f5cc5630 d758b9ec 00000057 f5cc5630 00000000 f53bdf48
 c14687bd bf9bc1d4 00000000 00000000 f53bdea8 00000001 00000000 00000000
Call Trace:
 [<c14687bd>] ? sys_sendto+0xf8/0x132
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c1102ab2>] ? signalfd_poll+0x63/0x6d
 [<c11018c1>] ? ep_send_events_proc+0x5c/0xf6
 [<c1101f19>] ? ep_scan_ready_list+0xed/0x108
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c11021dc>] sys_epoll_wait+0x297/0x2fa
 [<c146882d>] ? sys_send+0x36/0x38
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c10bfe18>] ? sys_munmap+0x3a/0x41
 [<c1593813>] sysenter_do_call+0x12/0x22
udevd           S f5cc49e0  6924  1380    866 0x00000000
 f53d1ecc 00000086 00000000 f5cc49e0 f53d1e64 f53d1fb4 ffffffff f53d1e7c
 f53d1eb8 f53d1ecc f5cc49e0 d433bb4d 00000057 f5cc49e0 f52ffe18 f53d1ec4
 f5cc4c94 f53d1f7c f53d1fb4 f5cc49e0 f5cc49e0 77c33025 f5cc4d44 f500b180
Call Trace:
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c11018c1>] ? ep_send_events_proc+0x5c/0xf6
 [<c10bd2da>] ? handle_mm_fault+0x7b/0xb8
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c11021dc>] sys_epoll_wait+0x297/0x2fa
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1593813>] sysenter_do_call+0x12/0x22
vnstatd         S 00000000  6288  1423      1 0x00000000
 f4801ef0 00000086 c183e0b4 00000000 f4801f40 c183e0b4 00000000 f4801ea0
 c11f5f74 f4801ef0 f5175630 a2be3f8d 00000064 f5175630 00000065 f4801f40
 f4801ed8 00000282 0000c350 00000000 f4801ec4 c10ac018 00000000 00000282
Call Trace:
 [<c11f5f74>] ? timerqueue_add+0x54/0xb0
 [<c10ac018>] ? __put_single_page+0x14/0x17
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d8f3>] do_nanosleep+0x81/0xab
 [<c104ebb3>] hrtimer_nanosleep+0x9c/0x11e
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c104ec8c>] sys_nanosleep+0x57/0x5b
 [<c1593813>] sysenter_do_call+0x12/0x22
auditd          S f5170c50  6584  1429      1 0x00000000
 f4805ecc 00000086 c183e0b4 f5170c50 f4805ee4 c183e0b4 00000000 f4805e7c
 c11f5f74 f4805ecc f5170c50 d433bf54 00000057 f5170c50 00000065 f4805ee4
 f4805eb4 00000292 038f9b16 00000000 f5170c50 00000292 00000000 00000292
Call Trace:
 [<c11f5f74>] ? timerqueue_add+0x54/0xb0
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d7eb>] schedule_hrtimeout_range_clock+0xab/0x110
 [<c104de53>] ? update_rmtp+0x62/0x62
 [<c104eb11>] ? hrtimer_start_range_ns+0x21/0x27
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c11021dc>] sys_epoll_wait+0x297/0x2fa
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1593813>] sysenter_do_call+0x12/0x22
auditd          S c1029e3c  6248  1430      1 0x00000000
 f4807e04 00000086 f4807d94 c1029e3c f514802c c183ae60 f4807db0 f4807db4
 0007ba34 f4807e04 f5176ed0 d433bf54 00000057 f5176ed0 00030222 86cb0d60
 f5176ed0 86cb0d60 f5176ed0 f5170c50 f4807e54 c158c95a 00000003 00000000
Call Trace:
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c10594d8>] futex_wait_queue_me+0xac/0xc8
 [<c1059efc>] ? futex_wait_setup+0x32/0x98
 [<c105a0be>] futex_wait+0x15c/0x22c
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c105a29c>] ? futex_wake+0xc7/0xe0
 [<c11f9c47>] ? copy_to_user+0x2e/0x119
 [<c105b0b4>] do_futex+0xd6/0x827
 [<c10f51d8>] ? sys_fstatfs+0x22/0x28
 [<c105b869>] sys_futex+0x64/0xfb
 [<c1593813>] sysenter_do_call+0x12/0x22
kauditd         S f480bf2c  7476  1432      2 0x00000000
 f480bf8c 00000046 f5176280 f480bf2c f480bf30 f480bf7c f5c66280 f480bf3c
 f51db000 f480bf8c f5c66280 d4336124 00000057 f5c66280 f6f9de20 f5071500
 00004401 c146d052 00004482 f5071300 00000000 00000001 00000003 c183fd00
Call Trace:
 [<c146d052>] ? __kfree_skb+0x38/0x7b
 [<c106fdd7>] kauditd_thread+0x103/0x126
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c106fcd4>] ? audit_printk_skb+0x5b/0x5b
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
bash            S f504e100  5800  1615      1 0x00000080
 f480df1c 00000086 f5148364 f504e100 f480deac c119919e f480df1c f480decc
 00000000 f480df1c f5148000 d433bf54 00000057 f5148000 fffb9dbc f5396088
 f53e6034 74529065 c183ae60 d8cf2c78 f5cc6efc f514802c f53e6000 f5148364
Call Trace:
 [<c119919e>] ? security_task_wait+0xf/0x11
 [<c1254bd7>] ? tty_check_change+0xa4/0xa4
 [<c1035c41>] do_wait+0x1aa/0x1ee
 [<c1035ced>] sys_wait4+0x68/0xb1
 [<c10347ca>] ? get_files_struct+0x11/0x11
 [<c1035d5d>] sys_waitpid+0x27/0x29
 [<c1593813>] sysenter_do_call+0x12/0x22
agetty          S f4809e04  6748  1616      1 0x00000080
 f4809e64 00000086 f5148000 f4809e04 00000004 f4809e54 f514a4f0 f4809e14
 00000056 f4809e64 f514a4f0 d433bf54 00000057 f514a4f0 f4809e2c c10424b9
 f4809efc f514a854 f514a854 0804c300 f4809e64 c104261d f4809e98 c158d15d
Call Trace:
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c158d15d>] ? schedule_timeout+0x13d/0x211
 [<c158d15d>] schedule_timeout+0x13d/0x211
 [<c1045e91>] ? start_flush_work+0x14/0xd6
 [<c1046ec9>] ? flush_work+0x13/0x2c
 [<c1257c26>] n_tty_read+0x35d/0x641
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1253a03>] tty_read+0x71/0x9e
 [<c12578c9>] ? n_tty_open+0x79/0x79
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c1253992>] ? tty_del_file+0x31/0x31
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
agetty          S f4803e04  6728  1617      1 0x00000080
 f4803e64 00000086 f514a4f0 f4803e04 00000000 f4803e54 f514eed0 f4803e14
 00000056 f4803e64 f514eed0 d433bf54 00000057 f514eed0 f4803e2c c10424b9
 f4803efc f514f234 f514f234 0804c300 f4803e64 c104261d 00d3abbf c183ae20
Call Trace:
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c158d15d>] schedule_timeout+0x13d/0x211
 [<c1045e91>] ? start_flush_work+0x14/0xd6
 [<c1046ec9>] ? flush_work+0x13/0x2c
 [<c1257c26>] n_tty_read+0x35d/0x641
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1253a03>] tty_read+0x71/0x9e
 [<c12578c9>] ? n_tty_open+0x79/0x79
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c1253992>] ? tty_del_file+0x31/0x31
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
agetty          S f53f3e04  6748  1618      1 0x00000080
 f53f3e64 00000086 f514eed0 f53f3e04 00000000 f53f3e54 f514b140 f53f3e14
 00000056 f53f3e64 f514b140 d433bf54 00000057 f514b140 f53f3e2c c10424b9
 f53f3efc f514b4a4 f514b4a4 0804c300 f53f3e64 c104261d 00d3ae56 c183ae20
Call Trace:
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c158d15d>] schedule_timeout+0x13d/0x211
 [<c1045e91>] ? start_flush_work+0x14/0xd6
 [<c1046ec9>] ? flush_work+0x13/0x2c
 [<c1257c26>] n_tty_read+0x35d/0x641
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1253a03>] tty_read+0x71/0x9e
 [<c12578c9>] ? n_tty_open+0x79/0x79
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c1253992>] ? tty_del_file+0x31/0x31
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
agetty          S f53f5e04  6244  1619      1 0x00000080
 f53f5e64 00000086 f514b140 f53f5e04 00000000 f53f5e54 f51498a0 f53f5e14
 00000056 f53f5e64 f51498a0 d433bf54 00000057 f51498a0 f53f5e2c c10424b9
 f53f5efc f5149c04 f5149c04 0804c300 f53f5e64 c104261d 00d3d8f6 c183ae20
Call Trace:
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c158d15d>] schedule_timeout+0x13d/0x211
 [<c1045e91>] ? start_flush_work+0x14/0xd6
 [<c1046ec9>] ? flush_work+0x13/0x2c
 [<c1257c26>] n_tty_read+0x35d/0x641
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1253a03>] tty_read+0x71/0x9e
 [<c12578c9>] ? n_tty_open+0x79/0x79
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c1253992>] ? tty_del_file+0x31/0x31
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
agetty          S f4965e04  6404  1620      1 0x00000080
 f4965e64 00000086 f51498a0 f4965e04 00000000 f4965e54 f57b24f0 f4965e14
 00000056 f4965e64 f57b24f0 d433bf54 00000057 f57b24f0 f4965e2c c10424b9
 f4965efc f57b2854 f57b2854 0804c300 f4965e64 c104261d 00d3feb8 c183ae20
Call Trace:
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c158d15d>] schedule_timeout+0x13d/0x211
 [<c1045e91>] ? start_flush_work+0x14/0xd6
 [<c1046ec9>] ? flush_work+0x13/0x2c
 [<c1257c26>] n_tty_read+0x35d/0x641
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1253a03>] tty_read+0x71/0x9e
 [<c12578c9>] ? n_tty_open+0x79/0x79
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c1253992>] ? tty_del_file+0x31/0x31
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
startx          S f5ddb180  6392  1635   1615 0x00000080
 f49bbf1c 00000086 f5cc7234 f5ddb180 f49bbeac c119919e f49bbf1c f49bbecc
 00000000 f49bbf1c f5cc6ed0 d433bf54 00000057 f5cc6ed0 fffb9e3c f499c084
 f53e6234 74497065 c183ae60 ecc4532b f5173dbc f5cc6efc f53e6200 f5cc7234
Call Trace:
 [<c119919e>] ? security_task_wait+0xf/0x11
 [<c10bd2da>] ? handle_mm_fault+0x7b/0xb8
 [<c1035c41>] do_wait+0x1aa/0x1ee
 [<c1035ced>] sys_wait4+0x68/0xb1
 [<c10347ca>] ? get_files_struct+0x11/0x11
 [<c1035d5d>] sys_waitpid+0x27/0x29
 [<c1593813>] sysenter_do_call+0x12/0x22
xinit           S f4999f9c  6592  1651   1635 0x00000080
 f4999f34 00000086 b7621ff4 f4999f9c f4999ec4 c119919e f4999f34 f4999ee4
 00000000 f4999f34 f5173d90 d433bf54 00000057 f5173d90 00000000 f5173178
 c183ae20 f51740f4 90bccc61 f51718cc f5173dbc f5173d90 f4999f28 14f4e2c9
Call Trace:
 [<c119919e>] ? security_task_wait+0xf/0x11
 [<c158cefa>] ? wait_for_common+0x80/0xeb
 [<c1035c41>] do_wait+0x1aa/0x1ee
 [<c1035ced>] sys_wait4+0x68/0xb1
 [<c10347ca>] ? get_files_struct+0x11/0x11
 [<c1593813>] sysenter_do_call+0x12/0x22
X               S f499b944  5800  1652   1651 0x00400080
 f499b9a8 00003086 00000000 f499b944 c100d4a4 00000000 00000000 f499b958
 00000000 f499b9a8 f5173140 d7857c56 00000057 f5173140 d8b69880 00000057
 00000001 00000000 f499b9b4 c104dd89 000f4240 00000000 00000000 f499ba68
Call Trace:
 [<c100d4a4>] ? x86_pmu_enable+0x1c9/0x239
 [<c104dd89>] ? hrtimer_forward+0x159/0x18c
 [<c1291301>] ttm_bo_wait_unreserved+0x5f/0x106
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c129145f>] ttm_bo_reserve_locked+0xb7/0xe1
 [<c1292c27>] ttm_bo_reserve+0x26/0x95
 [<c12c3c97>] radeon_crtc_do_set_base+0xbd/0x6d2
 [<c1038054>] ? irq_exit+0x2f/0x7d
 [<c1015f4b>] ? smp_apic_timer_interrupt+0x46/0x77
 [<c158e45d>] ? apic_timer_interrupt+0x31/0x38
 [<c12c42e7>] radeon_crtc_set_base+0x1b/0x1d
 [<c12c430d>] radeon_crtc_mode_set+0x24/0xdd7
 [<c12c3961>] ? radeon_crtc_dpms+0x161/0x34d
 [<c1279c57>] drm_crtc_helper_set_mode+0x32c/0x48b
 [<c1279e2f>] drm_helper_resume_force_mode+0x79/0x23e
 [<c12ace10>] radeon_gpu_reset+0x84/0x98
 [<c12c0838>] radeon_fence_wait+0x2d1/0x311
 [<c104b12a>] ? wake_up_bit+0x62/0x62
 [<c12c0e37>] radeon_sync_obj_wait+0xc/0xe
 [<c12908be>] ttm_bo_wait+0xa1/0x108
 [<c12d6e7b>] radeon_gem_wait_idle_ioctl+0x76/0xc4
 [<c127e62e>] drm_ioctl+0x1c2/0x42c
 [<c12d6e05>] ? radeon_gem_set_tiling_ioctl+0x8e/0x8e
 [<c127e46c>] ? drm_version+0x8a/0x8a
 [<c10e288e>] do_vfs_ioctl+0x79/0x54b
 [<c10d5f02>] ? vfs_write+0xea/0x14f
 [<c10e2dcb>] sys_ioctl+0x6b/0x70
 [<c1593813>] sysenter_do_call+0x12/0x22
xterm           S f49c9aa0  5800  1660   1651 0x00000080
 f49c9ab8 00000086 00000000 f49c9aa0 c158d15d ffffffff f4a124f0 f49c9a68
 448db3e0 f49c9ab8 f51718a0 d6a88ced 00000057 f51718a0 00000000 00000003
 00000001 f5173174 00000000 c183dd00 f49c9ae8 00000000 f49c9a90 c102fb4e
Call Trace:
 [<c158d15d>] ? schedule_timeout+0x13d/0x211
 [<c102fb4e>] ? wake_up_process+0xf/0x11
 [<c158cefa>] ? wait_for_common+0x80/0xeb
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c125a383>] ? put_ldisc+0x3f/0x85
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e3c2a>] do_select+0x556/0x665
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
last message repeated 2 times
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c146a5bf>] ? sock_alloc_send_pskb+0x16d/0x293
 [<c10ce1db>] ? __kmalloc_track_caller+0x11c/0x18d
 [<c146a5bf>] ? sock_alloc_send_pskb+0x16d/0x293
 [<c146dc60>] ? __alloc_skb+0x53/0xf8
 [<c146a5bf>] ? sock_alloc_send_pskb+0x16d/0x293
 [<c11f9d5f>] ? _copy_from_user+0x2d/0x10e
 [<c14ecb61>] ? scm_recv+0xf5/0x139
 [<c14ebcc1>] ? scm_destroy+0xd/0x44
 [<c14ec11e>] ? unix_stream_sendmsg+0x30c/0x31c
 [<c14ece26>] ? unix_stream_recvmsg+0x281/0x4b1
 [<c10e44c0>] core_sys_select+0x135/0x217
 [<c10d5a57>] ? do_sync_read+0x9b/0xc8
 [<c10d5ae1>] ? rw_verify_area+0x5d/0x101
 [<c1253a0d>] ? tty_read+0x7b/0x9e
 [<c10e4727>] sys_select+0x2c/0xb3
 [<c1593813>] sysenter_do_call+0x12/0x22
twm             S 0000004b  5800  1665   1660 0x00000080
 f49d1b50 00000086 c1838e60 0000004b c183e0d8 f49d1aec f51718a0 f49d1b00
 0000004b f49d1b50 f5cc0c50 d433bf54 00000057 f5cc0c50 dea68c3b 0000004b
 00000000 dea68c3b 0000004b deb5bec0 0000004b 00000046 f49d0000 00000000
Call Trace:
 [<c1038054>] ? irq_exit+0x2f/0x7d
 [<c1015f4b>] ? smp_apic_timer_interrupt+0x46/0x77
 [<c104e50b>] ? hrtimer_interrupt+0x161/0x24c
 [<c158d839>] schedule_hrtimeout_range_clock+0xf9/0x110
 [<c104af67>] ? add_wait_queue+0x1d/0x21
 [<c10e3552>] ? __pollwait+0x59/0xb9
 [<c158d862>] schedule_hrtimeout_range+0x12/0x14
 [<c10e346d>] poll_schedule_timeout+0x28/0x3f
 [<c10e4101>] do_sys_poll+0x3c8/0x459
 [<c10e34f9>] ? poll_freewait+0x5a/0x5a
 [<c10e35b2>] ? __pollwait+0xb9/0xb9
 [<c109ad9a>] ? perf_pmu_enable+0x1a/0x21
 [<c102a472>] ? enqueue_task_fair+0x4d4/0x5ae
 [<c146a5bf>] ? sock_alloc_send_pskb+0x16d/0x293
 [<c10ce1db>] ? __kmalloc_track_caller+0x11c/0x18d
 [<c1026eed>] ? update_curr+0x164/0x24d
 [<c146a5bf>] ? sock_alloc_send_pskb+0x16d/0x293
 [<c1030550>] ? dequeue_task_fair+0x250/0x255
 [<c1029e3c>] ? __dequeue_entity+0x31/0x35
 [<c103015e>] ? set_next_entity+0xad/0xc1
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c10424b9>] ? __dequeue_signal+0x13/0xfe
 [<c104261d>] ? dequeue_signal+0x79/0x136
 [<c104292b>] ? get_signal_to_deliver+0x87/0x3ac
 [<c10021c4>] ? do_signal+0x5c/0xa11
 [<c10d5a57>] ? do_sync_read+0x9b/0xc8
 [<c10d5ae1>] ? rw_verify_area+0x5d/0x101
 [<c10e4374>] do_restart_poll+0x2b/0x42
 [<c103f71e>] sys_restart_syscall+0x11/0x13
 [<c1593813>] sysenter_do_call+0x12/0x22
bash            S f49dc580  6548  1670   1660 0x00000080
 f4a05f1c 00000086 f4a10364 f49dc580 f4a05eac c119919e f4a05f1c f4a05ecc
 00000000 f4a05f1c f4a10000 d433bf54 00000057 f4a10000 fffb9a2c f49f2090
 f49ef434 72d7d065 c183ae60 58f1783d f4a1251c f4a1002c f49ef400 f4a10364
Call Trace:
 [<c119919e>] ? security_task_wait+0xf/0x11
 [<c1254bd7>] ? tty_check_change+0xa4/0xa4
 [<c1035c41>] do_wait+0x1aa/0x1ee
 [<c1035ced>] sys_wait4+0x68/0xb1
 [<c10347ca>] ? get_files_struct+0x11/0x11
 [<c1035d5d>] sys_waitpid+0x27/0x29
 [<c1593813>] sysenter_do_call+0x12/0x22
kworker/0:0     S 00000004  7076  1720      2 0x00000080
 f49d3f88 00000046 c12798cd 00000004 f49d3f3c 00000004 00000000 f49d3f38
 00000000 f49d3f88 f4a118a0 00000000 00000246 f4a118a0 00000000 f50f5e40
 c183dd00 f49d3f50 c1852918 f50f5e40 f49d3f88 c10464da f49d3fbc f4a118a0
Call Trace:
 [<c12798cd>] ? output_poll_execute+0x143/0x17b
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c1268348>] ? do_blank_screen+0x1ee/0x1ee
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
bash            S 7f1c0300  5728  1723   1670 0x00000080
 f4a39e64 00000086 00008a31 7f1c0300 01000415 1a131100 170f1200 f4a39e14
 00009600 f4a39e64 f4a124f0 d6b49857 00000057 f4a124f0 f4a39e6c c1259d29
 c183ae20 c183ae60 f4a124f0 c1598420 c183ae20 c1598420 f4a39e48 c1026cee
Call Trace:
 [<c1259d29>] ? tty_mode_ioctl+0x2e4/0x3d7
 [<c1026cee>] ? check_preempt_curr+0x6d/0x7e
 [<c158d15d>] schedule_timeout+0x13d/0x211
 [<c102fb31>] ? default_wake_function+0xb/0xd
 [<c10265f3>] ? __wake_up_common+0x40/0x63
 [<c1045e91>] ? start_flush_work+0x14/0xd6
 [<c1046ec9>] ? flush_work+0x13/0x2c
 [<c1257c26>] n_tty_read+0x35d/0x641
 [<c125bb9b>] ? pty_write+0x54/0x59
 [<c102fb26>] ? try_to_wake_up+0xfe/0xfe
 [<c1253a03>] tty_read+0x71/0x9e
 [<c12578c9>] ? n_tty_open+0x79/0x79
 [<c10d60cc>] vfs_read+0x89/0x14c
 [<c11f9c47>] ? copy_to_user+0x2e/0x119
 [<c1253992>] ? tty_del_file+0x31/0x31
 [<c10d623d>] sys_read+0x3d/0x6b
 [<c1593813>] sysenter_do_call+0x12/0x22
kworker/u:2     S 00000000  6964  1744      2 0x00000080
 ed31df88 00000046 00000296 00000000 00000000 f4bff280 00000246 ed31df38
 c1050c1f ed31df88 f4a13140 b1816d44 00000057 f4a13140 f5124cc0 00000282
 f4bff288 f4bff288 f4bff288 f5124cc0 ed31df88 c10464da ed31df64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:3     S 00000000  6768  1745      2 0x00000080
 ed31ff88 00000046 00000296 00000000 00000000 f51fa7c0 00000246 ed31ff38
 c1050c1f ed31ff88 f4a149e0 a26a4315 00000057 f4a149e0 f53f0c80 00000286
 f51fa7c8 f51fa7c8 f51fa7c8 f53f0c80 ed31ff88 c10464da ed31ffbc f4a149e0
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:4     S 00000000  6404  1746      2 0x00000080
 ed321f88 00000046 00000296 00000000 00000000 f4bff080 00000246 ed321f38
 c1050c1f ed321f88 f4a10c50 a452838d 00000057 f4a10c50 f53f0e40 00000282
 f4bff088 f4bff088 f4bff088 f53f0e40 ed321f88 c10464da ed321fbc f4a10c50
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:5     S 00000000  6768  1747      2 0x00000080
 f4aaff88 00000046 00000296 00000000 00000000 f4bff140 00000246 f4aaff38
 c1050c1f f4aaff88 f4a15630 a452950f 00000057 f4a15630 f5035240 00000282
 f4bff148 f4bff148 f4bff148 f5035240 f4aaff88 c10464da f4aaff64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:6     S 00000000  6768  1748      2 0x00000080
 f4aabf88 00000046 00000296 00000000 00000000 f4bff0c0 00000246 f4aabf38
 c1050c1f f4aabf88 f4a16280 a4528dbe 00000057 f4a16280 f50357c0 00000286
 f4bff0c8 f4bff0c8 f4bff0c8 f50357c0 f4aabf88 c10464da f4aabf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:7     S 00000000  7120  1749      2 0x00000080
 f4a95f88 00000046 00000296 00000000 00000000 f4bff100 00000246 f4a95f38
 c1050c1f f4a95f88 f4a16ed0 013eb6b5 00000001 f4a16ed0 f5035e80 00000286
 f4bff108 f4bff108 f4bff108 f5035e80 f4a95f88 c10464da f4a95f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:8     S 00000000  6864  1750      2 0x00000080
 f4a93f88 00000046 00000296 00000000 00000000 f4bff040 00000246 f4a93f38
 c1050c1f f4a93f88 f5cc6280 a26a4aa1 00000057 f5cc6280 f5035bc0 00000282
 f4bff048 f4bff048 f4bff048 f5035bc0 f4a93f88 c10464da f4a93f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:9     S 00000000  7120  1751      2 0x00000080
 f4a8ff88 00000046 00000296 00000000 00000000 f4bff240 00000246 f4a8ff38
 c1050c1f f4a8ff88 f5c3a4f0 b1816683 00000057 f5c3a4f0 f5c07340 00000282
 f4bff248 f4bff248 f4bff248 f5c07340 f4a8ff88 c10464da f4a8ff64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:10    S 00000000  7356  1752      2 0x00000080
 f4a8bf88 00000046 00000296 00000000 00000000 f4bff200 00000246 f4a8bf38
 c1050c1f f4a8bf88 f5c63140 b1815fc7 00000057 f5c63140 f50ab600 00000286
 f4bff208 f4bff208 f4bff208 f50ab600 f4a8bf88 c10464da f4a8bf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:11    S 00000000  6356  1753      2 0x00000080
 f4a7ff88 00000046 00000296 00000000 00000000 f4bff180 00000246 f4a7ff38
 c1050c1f f4a7ff88 f51724f0 b1813911 00000057 f51724f0 f50ab8c0 00000282
 f4bff188 f4bff188 f4bff188 f50ab8c0 f4a7ff88 c10464da f4a7ff64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:12    S 00000000  7356  1754      2 0x00000080
 f4a67f88 00000046 00000296 00000000 00000000 f4bff1c0 00000246 f4a67f38
 c1050c1f f4a67f88 f5d06ed0 b1814195 00000057 f5d06ed0 f50f5980 00000286
 f4bff1c8 f4bff1c8 f4bff1c8 f50f5980 f4a67f88 c10464da f4a67f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:13    S 00000000  6552  1755      2 0x00000080
 f4a61f88 00000046 00000296 00000000 00000000 f51fa680 00000246 f4a61f38
 c1050c1f f4a61f88 f4b60000 9c65ef86 00000057 f4b60000 f50f5300 00000282
 f51fa688 f51fa688 f51fa688 f50f5300 f4a61f88 c10464da f4a61f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:14    S 00000000  7356  1756      2 0x00000080
 f4a53f88 00000046 00000296 00000000 00000000 f51faa40 00000246 f4a53f38
 c1050c1f f4a53f88 f4b60c50 9c65cf58 00000057 f4b60c50 f50f5800 00000282
 f51faa48 f51faa48 f51faa48 f50f5800 f4a53f88 c10464da f4a53f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:15    S 00000000  7136  1757      2 0x00000080
 f4a4bf88 00000046 00000296 00000000 00000000 f50aba00 00000246 f4a4bf38
 c1050c1f f4a4bf88 f4b618a0 9c64f96f 00000057 f4b618a0 f50f5100 00000286
 f50aba08 f50aba08 f50aba08 f50f5100 f4a4bf88 c10464da f4a4bf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:16    S 00000000  7356  1758      2 0x00000080
 f4a45f88 00000046 00000296 00000000 00000000 f51facc0 00000246 f4a45f38
 c1050c1f f4a45f88 f4b624f0 9c64fbbb 00000057 f4b624f0 f53d9c80 00000286
 f51facc8 f51facc8 f51facc8 f53d9c80 f4a45f88 c10464da f4a45f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:17    S 00000000  6600  1759      2 0x00000080
 f4a3bf88 00000046 00000296 00000000 00000000 f51fac00 00000246 f4a3bf38
 c1050c1f f4a3bf88 f4b63140 9c656ee1 00000057 f4b63140 f53d9f40 00000286
 f51fac08 f51fac08 f51fac08 f53d9f40 f4a3bf88 c10464da f4a3bf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:18    S 00000000  7356  1760      2 0x00000080
 f4a31f88 00000046 00000296 00000000 00000000 f51fa140 00000246 f4a31f38
 c1050c1f f4a31f88 f4b63d90 9c6540aa 00000057 f4b63d90 f53d9400 00000282
 f51fa148 f51fa148 f51fa148 f53d9400 f4a31f88 c10464da f4a31f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:19    S 00000000  7356  1761      2 0x00000080
 f4a23f88 00000046 00000296 00000000 00000000 f50ab240 00000246 f4a23f38
 c1050c1f f4a23f88 f4b649e0 9c657726 00000057 f4b649e0 f57c99c0 00000282
 f50ab248 f50ab248 f50ab248 f57c99c0 f4a23f88 c10464da f4a23f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:20    S 00000000  7136  1762      2 0x00000080
 f4a07f88 00000046 00000296 00000000 00000000 f51fa240 00000246 f4a07f38
 c1050c1f f4a07f88 f4b65630 9c64f776 00000057 f4b65630 f5124280 00000282
 f51fa248 f51fa248 f51fa248 f5124280 f4a07f88 c10464da f4a07f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:21    S 00000000  6356  1763      2 0x00000080
 f4a03f88 00000046 00000296 00000000 00000000 f50ec500 00000246 f4a03f38
 c1050c1f f4a03f88 f4b66280 89f9f948 00000057 f4b66280 f5124580 00000286
 f50ec508 f50ec508 f50ec508 f5124580 f4a03f88 c10464da f4a03f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:22    S 00000000  6964  1764      2 0x00000080
 f49fbf88 00000046 00000296 00000000 00000000 f50351c0 00000246 f49fbf38
 c1050c1f f49fbf88 f4b66ed0 8fefd5fc 00000057 f4b66ed0 f5124680 00000286
 f50351c8 f50351c8 f50351c8 f5124680 f49fbf88 c10464da f49fbf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:23    S 00000000  6768  1765      2 0x00000080
 f49f5f88 00000046 00000296 00000000 00000000 f4bff440 00000246 f49f5f38
 c1050c1f f49f5f88 f4ab8000 d246586f 00000057 f4ab8000 f5124180 00000282
 f4bff448 f4bff448 f4bff448 f5124180 f49f5f88 c10464da f49f5f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:24    S 00000000  6768  1766      2 0x00000080
 f49cbf88 00000046 00000296 00000000 00000000 f4bff480 00000246 f49cbf38
 c1050c1f f49cbf88 f4ab8c50 01490187 00000001 f4ab8c50 f5124b00 00000282
 f4bff488 f4bff488 f4bff488 f5124b00 f49cbf88 c10464da f49cbf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:25    S 00000000  6404  1767      2 0x00000080
 ecf0ff88 00000046 00000296 00000000 00000000 f5035c40 00000246 ecf0ff38
 c1050c1f ecf0ff88 f4ab98a0 9327ce97 00000057 f4ab98a0 f53f0380 00000282
 f5035c48 f5035c48 f5035c48 f53f0380 ecf0ff88 c10464da ecf0ff64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:26    S 00000000  7248  1768      2 0x00000080
 f4aa1f88 00000046 00000296 00000000 00000000 f5035400 00000246 f4aa1f38
 c1050c1f f4aa1f88 f4aba4f0 89f9f948 00000057 f4aba4f0 f53f0ec0 00000286
 f5035408 f5035408 f5035408 f53f0ec0 f4aa1f88 c10464da f4aa1f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:27    S 00000000  6568  1769      2 0x00000080
 f4aa3f88 00000046 00000296 00000000 00000000 f4bff2c0 00000246 f4aa3f38
 c1050c1f f4aa3f88 f4abb140 b18155c9 00000057 f4abb140 f50ec040 00000286
 f4bff2c8 f4bff2c8 f4bff2c8 f50ec040 f4aa3f88 c10464da f4aa3f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:28    S 00000000  6404  1770      2 0x00000080
 f4aa5f88 00000046 00000296 00000000 00000000 f57c94c0 00000246 f4aa5f38
 c1050c1f f4aa5f88 f4abbd90 a2984e4e 00000057 f4abbd90 f50ecc00 00000286
 f57c94c8 f57c94c8 f57c94c8 f50ecc00 f4aa5f88 c10464da f4aa5f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:29    S 00000000  7356  1771      2 0x00000080
 f4aa7f88 00000046 00000296 00000000 00000000 f57c9a80 00000246 f4aa7f38
 c1050c1f f4aa7f88 f4abc9e0 a2983c1a 00000057 f4abc9e0 f50ec200 00000282
 f57c9a88 f57c9a88 f57c9a88 f50ec200 f4aa7f88 c10464da f4aa7f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:30    S 00000000  7120  1772      2 0x00000080
 f4a9bf88 00000046 00000296 00000000 00000000 f50ec3c0 00000246 f4a9bf38
 c1050c1f f4a9bf88 f4abd630 94b5cbac 00000057 f4abd630 f50ec480 00000286
 f50ec3c8 f50ec3c8 f50ec3c8 f50ec480 f4a9bf88 c10464da f4a9bf64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:31    S 00000000  7356  1773      2 0x00000080
 f4a9df88 00000046 00000296 00000000 00000000 f53f08c0 00000246 f4a9df38
 c1050c1f f4a9df88 f4abe280 a298363d 00000057 f4abe280 f50ece00 00000286
 f53f08c8 f53f08c8 f53f08c8 f50ece00 f4a9df88 c10464da f4a9df64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:32    S 00000000  6768  1774      2 0x00000080
 f4a9ff88 00000046 00000296 00000000 00000000 f50ec380 00000246 f4a9ff38
 c1050c1f f4a9ff88 f4abeed0 8f0344ea 00000057 f4abeed0 f50ecd40 00000282
 f50ec388 f50ec388 f50ec388 f50ecd40 f4a9ff88 c10464da f4a9ff64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:33    S 00000000  6864  1775      2 0x00000080
 f4a85f88 00000046 00000296 00000000 00000000 f53f0900 00000246 f4a85f38
 c1050c1f f4a85f88 f4a58000 a2982d61 00000057 f4a58000 f50ec300 00000286
 f53f0908 f53f0908 f53f0908 f50ec300 f4a85f88 c10464da f4a85f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:34    S 00000000  6404  1776      2 0x00000080
 f4a87f88 00000046 00000296 00000000 00000000 f4bff300 00000246 f4a87f38
 c1050c1f f4a87f88 f4a58c50 d2463928 00000057 f4a58c50 f50ec840 00000286
 f4bff308 f4bff308 f4bff308 f50ec840 f4a87f88 c10464da f4a87f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/0:3     S 000ef000  7672  1777      2 0x00000080
 f4a2df88 00000046 c18528a0 000ef000 f4a2df3c 00000004 00000000 f4a2df38
 00000000 f4a2df88 f4a598a0 a08b84d0 00000057 f4a598a0 00000000 f50ec7c0
 c183dd00 f4a2df50 c1852918 f50ec7c0 f4a2df88 c10464da f4a2dfbc f4a598a0
Call Trace:
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c1268348>] ? do_blank_screen+0x1ee/0x1ee
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:35    S 00000000  7356  1783      2 0x00000080
 f4b41f88 00000046 00000296 00000000 00000000 f4bff340 00000246 f4b41f38
 c1050c1f f4b41f88 f4a5b140 d2464186 00000057 f4a5b140 f4bff540 00000282
 f4bff348 f4bff348 f4bff348 f4bff540 f4b41f88 c10464da f4b41f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:36    S 00000000  7356  1784      2 0x00000080
 f4a41f88 00000046 00000296 00000000 00000000 f4bff380 00000246 f4a41f38
 c1050c1f f4a41f88 f4a5bd90 d246605f 00000057 f4a5bd90 f4bff5c0 00000282
 f4bff388 f4bff388 f4bff388 f4bff5c0 f4a41f88 c10464da f4a41f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:37    S 00000000  7356  1785      2 0x00000080
 f4a2ff88 00000046 00000296 00000000 00000000 f4bff3c0 00000246 f4a2ff38
 c1050c1f f4a2ff88 f4a5a4f0 d24647bd 00000057 f4a5a4f0 f4bff640 00000286
 f4bff3c8 f4bff3c8 f4bff3c8 f4bff640 f4a2ff88 c10464da f4a2ff64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:38    S 00000000  7356  1786      2 0x00000080
 f4a09f88 00000046 00000296 00000000 00000000 f4bff400 f5c398a0 f4a09f38
 c1050c1f f4a09f88 f4a5d630 d453d2bf 00000057 f4a5d630 f4bff6c0 00000286
 f4bff408 f4bff408 f4bff408 f4bff6c0 f4a09f88 c10464da f4a09f64 c104694b
Call Trace:
 [<c1050c1f>] ? async_run_entry_fn+0xbe/0x14f
 [<c10464da>] ? process_one_work+0x173/0x366
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1050b61>] ? async_schedule+0xf/0xf
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:39    S c183ae20  7372  1787      2 0x00000080
 f4a0bf88 00000046 c1598420 c183ae20 c1598420 f4a0bf28 c1026cee f4a0bf38
 00000000 f4a0bf88 f4a5e280 92daeea9 00000057 f4a5e280 00000001 01ffffff
 00000001 f4bff7c0 c19406e0 f4bff7c0 f4a0bf58 c102fb4e f4a0bf64 c104694b
Call Trace:
 [<c1026cee>] ? check_preempt_curr+0x6d/0x7e
 [<c102fb4e>] ? wake_up_process+0xf/0x11
 [<c104694b>] ? start_worker+0x20/0x23
 [<c1047bdc>] ? manage_workers+0x2a5/0x314
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
kworker/u:40    S 00000000  7672  1788      2 0x00000080
 f4a0df88 00000046 00000000 00000000 f4a5e280 f4a5e2ac f4a0df34 f4a0df38
 8a803bad f4a0df88 f4a5eed0 89f9f948 00000057 f4a5eed0 c158c95a f4a0df50
 c1026cee f4a0df50 c10475d6 f4a5e280 f4a0df6c 00000082 f4a0dfbc f4a5eed0
Call Trace:
 [<c158c95a>] ? schedule+0x29a/0x614
 [<c1026cee>] ? check_preempt_curr+0x6d/0x7e
 [<c10475d6>] ? wq_worker_waking_up+0xb/0x26
 [<c1047df7>] worker_thread+0x1ac/0x323
 [<c1047c4b>] ? manage_workers+0x314/0x314
 [<c104adf9>] kthread+0x67/0x69
 [<c104ad92>] ? kthreadd+0xb0/0xb0
 [<c1593dbe>] kernel_thread_helper+0x6/0xd
Sched Debug Version: v0.10, 2.6.39.4 #19
ktime                                   : 436994.538124
sched_clk                               : 436808.124081
cpu_clk                                 : 436986.131748
jiffies                                 : 136985
sched_clock_stable                      : 0

sysctl_sched
  .sysctl_sched_latency                    : 6.000000
  .sysctl_sched_min_granularity            : 0.750000
  .sysctl_sched_wakeup_granularity         : 1.000000
  .sysctl_sched_child_runs_first           : 0
  .sysctl_sched_features                   : 7279
  .sysctl_sched_tunable_scaling            : 1 (logaritmic)

cpu#0, 2667.072 MHz
  .nr_running                    : 0
  .load                          : 0
  .nr_switches                   : 67740
  .nr_load_updates               : 32576
  .nr_uninterruptible            : 0
  .next_balance                  : 0.000000
  .curr->pid                     : 0
  .clock                         : 436505.393605
  .cpu_load[0]                   : 1024
  .cpu_load[1]                   : 768
  .cpu_load[2]                   : 448
  .cpu_load[3]                   : 240
  .cpu_load[4]                   : 124
  .yld_count                     : 159
  .sched_switch                  : 0
  .sched_count                   : 67985
  .sched_goidle                  : 13458
  .ttwu_count                    : 4
  .ttwu_local                    : 4
  .bkl_count                     : 0

cfs_rq[0]:/
  .exec_clock                    : 18313.884391
  .MIN_vruntime                  : 0.000001
  .min_vruntime                  : 17397.193868
  .max_vruntime                  : 0.000001
  .spread                        : 0.000000
  .spread0                       : 0.000000
  .nr_spread_over                : 1709
  .nr_running                    : 0
  .load                          : 0

runnable tasks:
            task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
----------------------------------------------------------------------------------------------------------

SysRq : Emergency Sync
Emergency Sync complete

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Tejun Heo]

(cc'ing David Airlie and dri-devel)

Hello, the original thread can be read from

  http://thread.gmane.org/gmane.linux.kernel/1209587

Full sysrq-t output at

  http://article.gmane.org/gmane.linux.kernel/1211256

So, the problem is that after a seemingly unreated update to input
serio driver (convert to use workqueue), X seems to lock up
sporadically across suspend/resume cycles.

I went through the full sysrq-t output but couldn't spot anything
suspicious w/ anything else.  No worker is stuck and nobody is waiting
for flush to finish.

Stack trace for X follows.

> X               S f499b944  5800  1652   1651 0x00400080
>  f499b9a8 00003086 00000000 f499b944 c100d4a4 00000000 00000000 f499b958
>  00000000 f499b9a8 f5173140 d7857c56 00000057 f5173140 d8b69880 00000057
>  00000001 00000000 f499b9b4 c104dd89 000f4240 00000000 00000000 f499ba68
> Call Trace:
>  [<c1291301>] ttm_bo_wait_unreserved+0x5f/0x106
>  [<c129145f>] ttm_bo_reserve_locked+0xb7/0xe1
>  [<c1292c27>] ttm_bo_reserve+0x26/0x95
>  [<c12c3c97>] radeon_crtc_do_set_base+0xbd/0x6d2
>  [<c12c42e7>] radeon_crtc_set_base+0x1b/0x1d
>  [<c12c430d>] radeon_crtc_mode_set+0x24/0xdd7
>  [<c1279c57>] drm_crtc_helper_set_mode+0x32c/0x48b
>  [<c1279e2f>] drm_helper_resume_force_mode+0x79/0x23e
>  [<c12ace10>] radeon_gpu_reset+0x84/0x98
>  [<c12c0838>] radeon_fence_wait+0x2d1/0x311
>  [<c12c0e37>] radeon_sync_obj_wait+0xc/0xe
>  [<c12908be>] ttm_bo_wait+0xa1/0x108
>  [<c12d6e7b>] radeon_gem_wait_idle_ioctl+0x76/0xc4
>  [<c127e62e>] drm_ioctl+0x1c2/0x42c
>  [<c10e288e>] do_vfs_ioctl+0x79/0x54b
>  [<c10e2dcb>] sys_ioctl+0x6b/0x70
>  [<c1593813>] sysenter_do_call+0x12/0x22

Do you guys have any ideas what's going on?  It seems to be waiting
for bo->reserved to go zero.  Is it possible that someone there is
forgetting to properly kick a work item after resume causing the wait
to stall?

Andrew, can you please kill the X server after the hang and see
whether that brings the system back?  I think sshd should still work
and if not you can write a script to kill the X server after 30secs
after resume (and kill that script if resume succeeds).

Thank you.

-- 
tejun

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Jerome Glisse]

On Fri, Nov 04, 2011 at 09:14:31AM -0700, Tejun Heo wrote:
> (cc'ing David Airlie and dri-devel)
> 
> Hello, the original thread can be read from
> 
>   http://thread.gmane.org/gmane.linux.kernel/1209587
> 
> Full sysrq-t output at
> 
>   http://article.gmane.org/gmane.linux.kernel/1211256
> 
> So, the problem is that after a seemingly unreated update to input
> serio driver (convert to use workqueue), X seems to lock up
> sporadically across suspend/resume cycles.
> 
> I went through the full sysrq-t output but couldn't spot anything
> suspicious w/ anything else.  No worker is stuck and nobody is waiting
> for flush to finish.
> 
> Stack trace for X follows.
> 
> > X               S f499b944  5800  1652   1651 0x00400080
> >  f499b9a8 00003086 00000000 f499b944 c100d4a4 00000000 00000000 f499b958
> >  00000000 f499b9a8 f5173140 d7857c56 00000057 f5173140 d8b69880 00000057
> >  00000001 00000000 f499b9b4 c104dd89 000f4240 00000000 00000000 f499ba68
> > Call Trace:
> >  [<c1291301>] ttm_bo_wait_unreserved+0x5f/0x106
> >  [<c129145f>] ttm_bo_reserve_locked+0xb7/0xe1
> >  [<c1292c27>] ttm_bo_reserve+0x26/0x95
> >  [<c12c3c97>] radeon_crtc_do_set_base+0xbd/0x6d2
> >  [<c12c42e7>] radeon_crtc_set_base+0x1b/0x1d
> >  [<c12c430d>] radeon_crtc_mode_set+0x24/0xdd7
> >  [<c1279c57>] drm_crtc_helper_set_mode+0x32c/0x48b
> >  [<c1279e2f>] drm_helper_resume_force_mode+0x79/0x23e
> >  [<c12ace10>] radeon_gpu_reset+0x84/0x98
> >  [<c12c0838>] radeon_fence_wait+0x2d1/0x311
> >  [<c12c0e37>] radeon_sync_obj_wait+0xc/0xe
> >  [<c12908be>] ttm_bo_wait+0xa1/0x108
> >  [<c12d6e7b>] radeon_gem_wait_idle_ioctl+0x76/0xc4
> >  [<c127e62e>] drm_ioctl+0x1c2/0x42c
> >  [<c10e288e>] do_vfs_ioctl+0x79/0x54b
> >  [<c10e2dcb>] sys_ioctl+0x6b/0x70
> >  [<c1593813>] sysenter_do_call+0x12/0x22
> 
> Do you guys have any ideas what's going on?  It seems to be waiting
> for bo->reserved to go zero.  Is it possible that someone there is
> forgetting to properly kick a work item after resume causing the wait
> to stall?
> 
> Andrew, can you please kill the X server after the hang and see
> whether that brings the system back?  I think sshd should still work
> and if not you can write a script to kill the X server after 30secs
> after resume (and kill that script if resume succeeds).
> 
> Thank you.
> 

Ok so issue is funny, it should happen without the serio change, i guess
this other change make it just more likely. So here is my theory
radeon_gem_wait_idle_ioctl is call on the scanout buffer it reserve this
buffer. It wait for it to go idle, for some reasone the GPU is either
lockup or not yet fully resume or in some other state (see below for
more suposition).

At that point the gpu reset is call, which reset the gpu and then
restore it, to restore it need to reserve the scanout buffer and
bang you stuck. As the scanout buffer is already reserve by the
wait ioctl.

Thing is i don't know what would be a good solution to this, we could
set some flag to say that we are in reset phase and test if scanout
buffer are already reserve not try to reserve them again in the
restore after gpu reset path.


The GPU lockup is weird, can we get a dmesg on resume when the lockup
happen ? I am really not sure what happen here.

Cheers,
Jerome

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Fri, Nov 04, 2011 at 09:14:31AM -0700, Tejun Heo wrote:
> Andrew, can you please kill the X server after the hang and see
> whether that brings the system back?  I think sshd should still work
> and if not you can write a script to kill the X server after 30secs
> after resume (and kill that script if resume succeeds).

Hi Tejun & Jerome.

Tejun: My script killed X after a small wait from a bad resume. I am returned 
to the funcioning calling console. I can then re-start X and all subsystems
seem to working fine.

Jerome: I posted a post-bad-resume dmesg snippet with a GPU lockup warning
and trace here:

   http://article.gmane.org/gmane.linux.kernel/1210057

Did you need to see more?

Thanks.

~ Andy

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Jerome Glisse]

On Fri, Nov 4, 2011 at 1:34 PM, Andrew Watts <akwatts@ymail.com> wrote:
> On Fri, Nov 04, 2011 at 09:14:31AM -0700, Tejun Heo wrote:
>> Andrew, can you please kill the X server after the hang and see
>> whether that brings the system back?  I think sshd should still work
>> and if not you can write a script to kill the X server after 30secs
>> after resume (and kill that script if resume succeeds).
>
> Hi Tejun & Jerome.
>
> Tejun: My script killed X after a small wait from a bad resume. I am returned
> to the funcioning calling console. I can then re-start X and all subsystems
> seem to working fine.
>
> Jerome: I posted a post-bad-resume dmesg snippet with a GPU lockup warning
> and trace here:
>
>   http://article.gmane.org/gmane.linux.kernel/1210057
>
> Did you need to see more?
>
> Thanks.
>
> ~ Andy
>
>

I need full dmesg

Cheers,
Jerome

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Jerome Glisse]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

On Fri, Nov 04, 2011 at 12:34:58PM -0500, Andrew Watts wrote:
> On Fri, Nov 04, 2011 at 09:14:31AM -0700, Tejun Heo wrote:
> > Andrew, can you please kill the X server after the hang and see
> > whether that brings the system back?  I think sshd should still work
> > and if not you can write a script to kill the X server after 30secs
> > after resume (and kill that script if resume succeeds).
> 
> Hi Tejun & Jerome.
> 
> Tejun: My script killed X after a small wait from a bad resume. I am returned 
> to the funcioning calling console. I can then re-start X and all subsystems
> seem to working fine.
> 
> Jerome: I posted a post-bad-resume dmesg snippet with a GPU lockup warning
> and trace here:
> 
>    http://article.gmane.org/gmane.linux.kernel/1210057
> 
> Did you need to see more?
> 
> Thanks.
> 
> ~ Andy
> 

Attached patch might hide the issue.

Cheers,
Jerome

[-- Attachment #2: 0001-drm-radeon-kms-make-sure-GPU-is-idle-and-working-at-.patch --]
[-- Type: text/plain, Size: 928 bytes --]

>From a13af3d0bff9e093160da63f22c6b44e436aa570 Mon Sep 17 00:00:00 2001
From: Jerome Glisse <jglisse@redhat.com>
Date: Fri, 4 Nov 2011 13:41:37 -0400
Subject: [PATCH] drm/radeon/kms: make sure GPU is idle and working at end of
 resume

This might hide some GPU lockup issue deadlock on resume.

Signed-off-by: Jerome Glisse <jglisse@redhat.com>
---
 drivers/gpu/drm/radeon/radeon_device.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/gpu/drm/radeon/radeon_device.c b/drivers/gpu/drm/radeon/radeon_device.c
index c33bc91..1158e55 100644
--- a/drivers/gpu/drm/radeon/radeon_device.c
+++ b/drivers/gpu/drm/radeon/radeon_device.c
@@ -947,6 +947,7 @@ int radeon_resume_kms(struct drm_device *dev)
 	list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
 		drm_helper_connector_dpms(connector, DRM_MODE_DPMS_ON);
 	}
+	radeon_fence_wait_last(rdev);
 	return 0;
 }
 
-- 
1.7.6.4


[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

[-- Attachment #1: Type: text/plain, Size: 242 bytes --]

On Fri, Nov 04, 2011 at 01:44:01PM -0400, Jerome Glisse wrote:
> 
> Attached patch might hide the issue.
> 
> Cheers,
> Jerome

Unfortunately, the patch did not help.

Attached is the full hibernate->resume dmesg requested.

Thanks.

~ Andy


[-- Attachment #2: bad-dmesg.txt.gz --]
[-- Type: application/x-gunzip, Size: 3950 bytes --]

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

[-- Attachment #1: Type: text/plain, Size: 213 bytes --]

On Fri, Nov 04, 2011 at 01:35:53PM -0400, Jerome Glisse wrote:
> 
> I need full dmesg
> 
> Cheers,
> Jerome

Hi. Just noticed I attached a gzip of my dmesg last time by mistake.

Here goes as text/plain.

~ Andy


[-- Attachment #2: bad-dmesg.txt --]
[-- Type: text/plain, Size: 14635 bytes --]

[  237.030115] PM: Marking nosave pages: 000000000009f000 - 0000000000100000
[  237.030121] PM: Basic memory bitmaps created
[  237.030123] PM: Syncing filesystems ... done.
[  237.046109] Freezing user space processes ... (elapsed 0.01 seconds) done.
[  237.057089] Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
[  237.068083] PM: Preallocating image memory... done (allocated 83937 pages)
[  237.154418] PM: Allocated 335748 kbytes in 0.08 seconds (4196.85 MB/s)
[  237.154422] Suspending console(s) (use no_console_suspend to debug)
[  237.154847] hub 4-0:1.0: hub_suspend
[  237.154858] usb usb4: bus suspend
[  237.154863] ohci_hcd 0000:02:07.0: suspend root hub
[  237.154891] hub 3-0:1.0: hub_suspend
[  237.154896] usb usb3: bus suspend
[  237.154900] ohci_hcd 0000:00:13.1: suspend root hub
[  237.154927] hub 2-0:1.0: hub_suspend
[  237.154932] usb usb2: bus suspend
[  237.154936] ohci_hcd 0000:00:13.0: suspend root hub
[  237.154969] hub 1-0:1.0: hub_suspend
[  237.154974] usb usb1: bus suspend
[  237.154978] ehci_hcd 0000:02:07.2: suspend root hub
[  237.156423] hub 5-0:1.0: hub_suspend
[  237.156429] usb usb5: bus suspend
[  237.156433] ohci_hcd 0000:02:07.1: suspend root hub
[  237.209815] i8042 aux 00:06: wake-up capability disabled by ACPI
[  237.213576] i8042 kbd 00:05: wake-up capability enabled by ACPI
[  237.213699] b43-pci-bridge 0000:02:02.0: PCI INT A disabled
[  237.213748] ACPI handle has no context!
[  237.234057] ATI IXP AC97 controller 0000:00:14.5: PCI INT B disabled
[  237.234108] ATIIXP_IDE 0000:00:14.1: PCI INT A disabled
[  237.244424] PM: freeze of devices complete after 89.751 msecs
[  237.244831] PM: late freeze of devices complete after 0.401 msecs
[  237.244951] ACPI: Preparing to enter system sleep state S4
[  237.270225] PM: Saving platform NVS memory
[  237.270273] hibernation debug: Waiting for 5 seconds.
[  242.272850] ACPI: Waking up from system sleep state S4
[  242.275175] ATIIXP_IDE 0000:00:14.1: restoring config space at offset 0x1 (was 0xa000001, writing 0xa000005)
[  242.286031] firewire_ohci 0000:02:00.0: BAR 0: set to [mem 0xd0208000-0xd02087ff] (PCI address [0xd0208000-0xd02087ff])
[  242.286038] firewire_ohci 0000:02:00.0: BAR 1: set to [mem 0xd0200000-0xd0203fff] (PCI address [0xd0200000-0xd0203fff])
[  242.286049] firewire_ohci 0000:02:00.0: restoring config space at offset 0xf (was 0x4020100, writing 0x402010a)
[  242.286068] firewire_ohci 0000:02:00.0: restoring config space at offset 0x3 (was 0x0, writing 0x4008)
[  242.286075] firewire_ohci 0000:02:00.0: restoring config space at offset 0x1 (was 0x2100000, writing 0x2100016)
[  242.286105] b43-pci-bridge 0000:02:02.0: restoring config space at offset 0x1 (was 0x2, writing 0x6)
[  242.286617] PM: early restore of devices complete after 11.560 msecs
[  242.319891] ohci_hcd 0000:00:13.0: lost power
[  242.319919] ohci_hcd 0000:00:13.1: lost power
[  242.319946] ATIIXP_IDE 0000:00:14.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[  242.319964] ATI IXP AC97 controller 0000:00:14.5: PCI INT B -> GSI 5 (level, low) -> IRQ 5
[  242.323512] [drm] AGP mode requested: 8
[  242.323517] agpgart-ati 0000:00:00.0: AGP 3.0 bridge
[  242.323535] agpgart-ati 0000:00:00.0: putting AGP V3 device into 8x mode
[  242.323553] radeon 0000:01:05.0: putting AGP V3 device into 8x mode
[  242.323603] radeon 0000:01:05.0: GTT: 32M 0xD2000000 - 0xD3FFFFFF
[  242.333997] b43-pci-bridge 0000:02:02.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[  242.334041] ohci_hcd 0000:02:07.0: lost power
[  242.334060] ohci_hcd 0000:02:07.1: lost power
[  242.334411] radeon 0000:01:05.0: WB disabled
[  242.334450] [drm] radeon: ring at 0x00000000D2001000
[  242.334469] [drm] ring test succeeded in 0 usecs
[  242.334497] [drm] ib test succeeded in 0 usecs
[  242.446359] ohci_hcd 0000:00:13.0: OHCI controller state
[  242.446364] ohci_hcd 0000:00:13.0: OHCI 1.0, NO legacy support registers
[  242.446369] ohci_hcd 0000:00:13.0: control 0x083 HCFS=operational CBSR=3
[  242.446373] ohci_hcd 0000:00:13.0: cmdstatus 0x00000 SOC=0
[  242.446377] ohci_hcd 0000:00:13.0: intrstatus 0x00000004 SF
[  242.446382] ohci_hcd 0000:00:13.0: intrenable 0x8000005a MIE RHSC UE RD WDH
[  242.446390] ohci_hcd 0000:00:13.0: hcca frame #0005
[  242.446395] ohci_hcd 0000:00:13.0: roothub.a 02000203 POTPGT=2 NPS NDP=3(3)
[  242.446399] ohci_hcd 0000:00:13.0: roothub.b 00000000 PPCM=0000 DR=0000
[  242.446403] ohci_hcd 0000:00:13.0: roothub.status 00008000 DRWE
[  242.446408] ohci_hcd 0000:00:13.0: roothub.portstatus [0] 0x00000100 PPS
[  242.446412] ohci_hcd 0000:00:13.0: roothub.portstatus [1] 0x00000100 PPS
[  242.446416] ohci_hcd 0000:00:13.0: roothub.portstatus [2] 0x00000100 PPS
[  242.446419] ohci_hcd 0000:00:13.0: restart complete
[  242.446423] usb usb2: root hub lost power or was reset
[  242.446426] ohci_hcd 0000:00:13.0: suspend root hub
[  242.450451] ohci_hcd 0000:00:13.1: OHCI controller state
[  242.450455] ohci_hcd 0000:00:13.1: OHCI 1.0, NO legacy support registers
[  242.450459] ohci_hcd 0000:00:13.1: control 0x083 HCFS=operational CBSR=3
[  242.450463] ohci_hcd 0000:00:13.1: cmdstatus 0x00000 SOC=0
[  242.450467] ohci_hcd 0000:00:13.1: intrstatus 0x00000004 SF
[  242.450471] ohci_hcd 0000:00:13.1: intrenable 0x8000005a MIE RHSC UE RD WDH
[  242.450479] ohci_hcd 0000:00:13.1: hcca frame #0005
[  242.450484] ohci_hcd 0000:00:13.1: roothub.a 02000203 POTPGT=2 NPS NDP=3(3)
[  242.450488] ohci_hcd 0000:00:13.1: roothub.b 00000000 PPCM=0000 DR=0000
[  242.450492] ohci_hcd 0000:00:13.1: roothub.status 00008000 DRWE
[  242.450496] ohci_hcd 0000:00:13.1: roothub.portstatus [0] 0x00000100 PPS
[  242.450500] ohci_hcd 0000:00:13.1: roothub.portstatus [1] 0x00000100 PPS
[  242.450504] ohci_hcd 0000:00:13.1: roothub.portstatus [2] 0x00000100 PPS
[  242.450507] ohci_hcd 0000:00:13.1: restart complete
[  242.450509] usb usb3: root hub lost power or was reset
[  242.450513] ohci_hcd 0000:00:13.1: suspend root hub
[  242.480589] ohci_hcd 0000:02:07.0: OHCI controller state
[  242.480593] ohci_hcd 0000:02:07.0: OHCI 1.0, NO legacy support registers
[  242.480597] ohci_hcd 0000:02:07.0: control 0x283 RWC HCFS=operational CBSR=3
[  242.480601] ohci_hcd 0000:02:07.0: cmdstatus 0x00000 SOC=0
[  242.480605] ohci_hcd 0000:02:07.0: intrstatus 0x00000004 SF
[  242.480609] ohci_hcd 0000:02:07.0: intrenable 0x8000005a MIE RHSC UE RD WDH
[  242.480616] ohci_hcd 0000:02:07.0: hcca frame #001f
[  242.480620] ohci_hcd 0000:02:07.0: roothub.a 0f000203 POTPGT=15 NPS NDP=3(3)
[  242.480624] ohci_hcd 0000:02:07.0: roothub.b 00000000 PPCM=0000 DR=0000
[  242.480627] ohci_hcd 0000:02:07.0: roothub.status 00008000 DRWE
[  242.480632] ohci_hcd 0000:02:07.0: roothub.portstatus [0] 0x00000100 PPS
[  242.480636] ohci_hcd 0000:02:07.0: roothub.portstatus [1] 0x00000100 PPS
[  242.480640] ohci_hcd 0000:02:07.0: roothub.portstatus [2] 0x00000100 PPS
[  242.480642] ohci_hcd 0000:02:07.0: restart complete
[  242.480645] usb usb4: root hub lost power or was reset
[  242.480648] ohci_hcd 0000:02:07.0: suspend root hub
[  242.510670] ohci_hcd 0000:02:07.1: OHCI controller state
[  242.510674] ohci_hcd 0000:02:07.1: OHCI 1.0, NO legacy support registers
[  242.510678] ohci_hcd 0000:02:07.1: control 0x283 RWC HCFS=operational CBSR=3
[  242.510682] ohci_hcd 0000:02:07.1: cmdstatus 0x00000 SOC=0
[  242.510686] ohci_hcd 0000:02:07.1: intrstatus 0x00000004 SF
[  242.510689] ohci_hcd 0000:02:07.1: intrenable 0x8000005a MIE RHSC UE RD WDH
[  242.510696] ohci_hcd 0000:02:07.1: hcca frame #001f
[  242.510700] ohci_hcd 0000:02:07.1: roothub.a 0f000202 POTPGT=15 NPS NDP=2(2)
[  242.510704] ohci_hcd 0000:02:07.1: roothub.b 00000000 PPCM=0000 DR=0000
[  242.510708] ohci_hcd 0000:02:07.1: roothub.status 00008000 DRWE
[  242.510712] ohci_hcd 0000:02:07.1: roothub.portstatus [0] 0x00000100 PPS
[  242.510716] ohci_hcd 0000:02:07.1: roothub.portstatus [1] 0x00000100 PPS
[  242.510718] ohci_hcd 0000:02:07.1: restart complete
[  242.510721] usb usb5: root hub lost power or was reset
[  242.510724] ohci_hcd 0000:02:07.1: suspend root hub
[  242.510778] firewire_core: skipped bus generations, destroying all nodes
[  242.510792] usb usb2: usb resume
[  242.510797] ohci_hcd 0000:00:13.0: resume root hub
[  242.510803] usb usb3: usb resume
[  242.510807] ohci_hcd 0000:00:13.1: resume root hub
[  242.510818] usb usb1: root hub lost power or was reset
[  242.510826] ehci_hcd 0000:02:07.2: reset command 0010006 (park)=0 ithresh=1 period=512 Reset HALT
[  242.510834] ehci_hcd 0000:02:07.2: MWI active
[  242.510840] ehci_hcd 0000:02:07.2: ...powerup ports...
[  242.510856] usb usb4: usb resume
[  242.510860] ohci_hcd 0000:02:07.0: wakeup root hub
[  242.510865] usb usb5: usb resume
[  242.541947] ohci_hcd 0000:02:07.1: resume root hub
[  242.541956] usb usb1: usb resume
[  242.541960] ehci_hcd 0000:02:07.2: resume root hub after power loss
[  242.570028] hub 3-0:1.0: hub_reset_resume
[  242.570032] hub 3-0:1.0: trying to enable port power on non-switchable hub
[  242.570045] hub 2-0:1.0: hub_reset_resume
[  242.570047] hub 2-0:1.0: trying to enable port power on non-switchable hub
[  242.570054] hub 4-0:1.0: hub_reset_resume
[  242.570057] hub 4-0:1.0: trying to enable port power on non-switchable hub
[  242.571026] ehci_hcd 0000:02:07.2: port 2 low speed --> companion
[  242.601023] hub 5-0:1.0: hub_reset_resume
[  242.601026] hub 5-0:1.0: trying to enable port power on non-switchable hub
[  242.662162] ehci_hcd 0000:02:07.2: port 3 full speed --> companion
[  242.662170] ehci_hcd 0000:02:07.2: GetStatus port:3 status 003801 0  ACK POWER OWNER sig=j CONNECT
[  242.662192] ehci_hcd 0000:02:07.2: GetStatus port:2 status 003402 0  ACK POWER OWNER sig=k CSC
[  242.671059] ohci_hcd 0000:02:07.0: GetStatus roothub.portstatus [1] = 0x00010101 CSC PPS CCS
[  242.671064] hub 4-0:1.0: port 2: status 0101 change 0001
[  242.673022] hub 1-0:1.0: hub_reset_resume
[  242.673025] hub 1-0:1.0: enabling power on all ports
[  242.702027] ohci_hcd 0000:02:07.1: GetStatus roothub.portstatus [0] = 0x00010301 CSC LSDA PPS CCS
[  242.702032] hub 5-0:1.0: port 1: status 0301 change 0001
[  242.743895] i8042 kbd 00:05: wake-up capability disabled by ACPI
[  242.746578] hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
[  242.751037] hda: UDMA/100 mode selected
[  242.774059] usb 4-2: reset-resume
[  242.803444] usb 5-1: reset-resume
[  242.836028] ohci_hcd 0000:02:07.0: GetStatus roothub.portstatus [1] = 0x00100103 PRSC PPS PES CCS
[  242.887035] usb 4-2: reset full speed USB device number 3 using ohci_hcd
[  242.951026] ohci_hcd 0000:02:07.0: GetStatus roothub.portstatus [1] = 0x00100103 PRSC PPS PES CCS
[  243.011046] firewire_core: rediscovered device fw0
[  243.024045] pl2303 4-2:1.0: no reset_resume for driver pl2303?
[  243.024047] pl2303 4-2:1.0: forced unbind
[  243.024165] pl2303 ttyUSB0: pl2303 converter now disconnected from ttyUSB0
[  243.024180] pl2303 4-2:1.0: device disconnected
[  243.230025] ohci_hcd 0000:02:07.1: GetStatus roothub.portstatus [0] = 0x00100303 PRSC LSDA PPS PES CCS
[  243.281032] usb 5-1: reset low speed USB device number 2 using ohci_hcd
[  243.494027] ohci_hcd 0000:02:07.1: GetStatus roothub.portstatus [0] = 0x00100303 PRSC LSDA PPS PES CCS
[  243.574073] PM: restore of devices complete after 1254.411 msecs
[  243.574095] pl2303 4-2:1.0: usb_probe_interface
[  243.574098] pl2303 4-2:1.0: usb_probe_interface - got id
[  243.574104] pl2303 4-2:1.0: pl2303 converter detected
[  243.606085] usb 4-2: pl2303 converter now attached to ttyUSB0
[  243.606236] PM: Image restored successfully.
[  243.606239] Restarting tasks ... done.
[  243.606443] PM: Basic memory bitmaps freed
[  243.606452] video LNXVIDEO:00: Restoring backlight state
[  243.606484] hub 3-0:1.0: state 7 ports 3 chg 0000 evt 0000
[  243.606489] hub 2-0:1.0: state 7 ports 3 chg 0000 evt 0000
[  243.606493] hub 4-0:1.0: state 7 ports 3 chg 0004 evt 0004
[  243.606502] hub 4-0:1.0: port 2, status 0103, change 0000, 12 Mb/s
[  243.606507] hub 1-0:1.0: state 7 ports 5 chg 0000 evt 0000
[  243.606510] hub 5-0:1.0: state 7 ports 2 chg 0002 evt 0002
[  243.606515] hub 5-0:1.0: port 1, status 0303, change 0000, 1.5 Mb/s
[  243.626231] radeon 0000:01:05.0: GPU lockup CP stall for more than 82318msec
[  243.626236] ------------[ cut here ]------------
[  243.626252] WARNING: at drivers/gpu/drm/radeon/radeon_fence.c:246 radeon_fence_wait+0x2c3/0x311()
[  243.626256] Hardware name: Pavilion zv5000
[  243.626259] GPU lockup (waiting for 0x00000F8C last fence id 0x00000F8A)
[  243.626262] Modules linked in: fan snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss fuse pl2303 snd_atiixp snd_ac97_codec ac97_bus snd_pcm pcmcia snd_timer snd ohci_hcd shpchp processor yenta_socket pcmcia_rsrc ssb video thermal wmi battery button ac ehci_hcd thermal_sys pcmcia_core firewire_ohci snd_page_alloc
[  243.626311] Pid: 1646, comm: X Tainted: G   M        2.6.39.4 #19
[  243.626314] Call Trace:
[  243.626325]  [<c1033164>] warn_slowpath_common+0x67/0x8e
[  243.626331]  [<c12c082a>] ? radeon_fence_wait+0x2c3/0x311
[  243.626335]  [<c12c082a>] ? radeon_fence_wait+0x2c3/0x311
[  243.626339]  [<c1033207>] warn_slowpath_fmt+0x2e/0x30
[  243.626344]  [<c12c082a>] radeon_fence_wait+0x2c3/0x311
[  243.626352]  [<c104b12a>] ? wake_up_bit+0x62/0x62
[  243.626358]  [<c12c0e37>] radeon_sync_obj_wait+0xc/0xe
[  243.626363]  [<c12908be>] ttm_bo_wait+0xa1/0x108
[  243.626371]  [<c12d6e7b>] radeon_gem_wait_idle_ioctl+0x76/0xc4
[  243.626378]  [<c127e62e>] drm_ioctl+0x1c2/0x42c
[  243.626383]  [<c12d6e05>] ? radeon_gem_set_tiling_ioctl+0x8e/0x8e
[  243.626389]  [<c109ad9a>] ? perf_pmu_enable+0x1a/0x21
[  243.626395]  [<c1026eed>] ? update_curr+0x164/0x24d
[  243.626403]  [<c11f5171>] ? rb_erase+0x16e/0x27a
[  243.626411]  [<c1007ba8>] ? __switch_to_xtra+0xf7/0x11d
[  243.626417]  [<c103015e>] ? set_next_entity+0xad/0xc1
[  243.626422]  [<c127e46c>] ? drm_version+0x8a/0x8a
[  243.626428]  [<c10e288e>] do_vfs_ioctl+0x79/0x54b
[  243.626436]  [<c158c95a>] ? schedule+0x29a/0x614
[  243.626441]  [<c10e2dcb>] sys_ioctl+0x6b/0x70
[  243.626446]  [<c1593813>] sysenter_do_call+0x12/0x22
[  243.626449] ---[ end trace 17a119d9361222c9 ]---
[  243.633615] radeon 0000:01:05.0: GPU reset succeed
[  243.671128] radeon 0000:01:05.0: WB disabled
[  243.671170] [drm] radeon: ring at 0x00000000D2001000
[  243.671189] [drm] ring test succeeded in 0 usecs
[  243.671200] [drm] ib test succeeded in 1 usecs
[  243.704033] ohci_hcd 0000:00:13.1: auto-stop root hub
[  243.704048] ohci_hcd 0000:00:13.0: auto-stop root hub

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Tejun Heo]

On Fri, Nov 04, 2011 at 06:08:20PM -0500, Andrew Watts wrote:
> On Fri, Nov 04, 2011 at 01:35:53PM -0400, Jerome Glisse wrote:
> > 
> > I need full dmesg
> > 
> > Cheers,
> > Jerome
> 
> Hi. Just noticed I attached a gzip of my dmesg last time by mistake.
> 
> Here goes as text/plain.

Out of curiosity, did it get somewhere?

Thanks.

-- 
tejun

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Jerome Glisse]

On Mon, Nov 21, 2011 at 4:09 PM, Tejun Heo <tj@kernel.org> wrote:
> On Fri, Nov 04, 2011 at 06:08:20PM -0500, Andrew Watts wrote:
>> On Fri, Nov 04, 2011 at 01:35:53PM -0400, Jerome Glisse wrote:
>> >
>> > I need full dmesg
>> >
>> > Cheers,
>> > Jerome
>>
>> Hi. Just noticed I attached a gzip of my dmesg last time by mistake.
>>
>> Here goes as text/plain.
>
> Out of curiosity, did it get somewhere?
>
> Thanks.
>
> --
> tejun
>

Yeah i am pretty sure i know what's going wrong but there is no easy
fix, there is a deadlock in the kernel when gem wait ioctl is call on
scanout buffer and that a gpu reset happen

Cheers,
Jerome

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

On Mon, Nov 21, 2011 at 04:29:23PM -0500, Jerome Glisse wrote:
> On Mon, Nov 21, 2011 at 4:09 PM, Tejun Heo <tj@kernel.org> wrote:
> >
> > Out of curiosity, did it get somewhere?
> >
> > Thanks.
> >
> > --
> > tejun
> >
> 
> Yeah i am pretty sure i know what's going wrong but there is no easy
> fix, there is a deadlock in the kernel when gem wait ioctl is call on
> scanout buffer and that a gpu reset happen
> 
> Cheers,
> Jerome

Tejun, I was about to send a similar email asking about this; many thanks
for sharing my interest.

Jerome, sounds encouraging that you think you know what the issue is. What
are the next steps? Please let me know how I can assist on my end (code 
validation, etc.).

~ Andy

Re: [REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

Hello all.

I wanted to follow-up on a very peculiar yet highly reproducible bug
involving suspend/resume, radeon, and a seemingly unrelated patch to
input serio.

The last comment in the thread was from Jerome Glisse saying it was a tough
bug to fix. Has anyone had any good ideas on how to fix it since?

Thanks!

~ Andy


On Fri, Nov 04, 2011 at 09:14:31AM -0700, Tejun Heo wrote:
> (cc'ing David Airlie and dri-devel)
> 
> Hello, the original thread can be read from
> 
>   http://thread.gmane.org/gmane.linux.kernel/1209587
> 
> Full sysrq-t output at
> 
>   http://article.gmane.org/gmane.linux.kernel/1211256
> 
> So, the problem is that after a seemingly unreated update to input
> serio driver (convert to use workqueue), X seems to lock up
> sporadically across suspend/resume cycles.
> 
> I went through the full sysrq-t output but couldn't spot anything
> suspicious w/ anything else.  No worker is stuck and nobody is waiting
> for flush to finish.
> 
> Stack trace for X follows.
> 
> > X               S f499b944  5800  1652   1651 0x00400080
> >  f499b9a8 00003086 00000000 f499b944 c100d4a4 00000000 00000000 f499b958
> >  00000000 f499b9a8 f5173140 d7857c56 00000057 f5173140 d8b69880 00000057
> >  00000001 00000000 f499b9b4 c104dd89 000f4240 00000000 00000000 f499ba68
> > Call Trace:
> >  [<c1291301>] ttm_bo_wait_unreserved+0x5f/0x106
> >  [<c129145f>] ttm_bo_reserve_locked+0xb7/0xe1
> >  [<c1292c27>] ttm_bo_reserve+0x26/0x95
> >  [<c12c3c97>] radeon_crtc_do_set_base+0xbd/0x6d2
> >  [<c12c42e7>] radeon_crtc_set_base+0x1b/0x1d
> >  [<c12c430d>] radeon_crtc_mode_set+0x24/0xdd7
> >  [<c1279c57>] drm_crtc_helper_set_mode+0x32c/0x48b
> >  [<c1279e2f>] drm_helper_resume_force_mode+0x79/0x23e
> >  [<c12ace10>] radeon_gpu_reset+0x84/0x98
> >  [<c12c0838>] radeon_fence_wait+0x2d1/0x311
> >  [<c12c0e37>] radeon_sync_obj_wait+0xc/0xe
> >  [<c12908be>] ttm_bo_wait+0xa1/0x108
> >  [<c12d6e7b>] radeon_gem_wait_idle_ioctl+0x76/0xc4
> >  [<c127e62e>] drm_ioctl+0x1c2/0x42c
> >  [<c10e288e>] do_vfs_ioctl+0x79/0x54b
> >  [<c10e2dcb>] sys_ioctl+0x6b/0x70
> >  [<c1593813>] sysenter_do_call+0x12/0x22
> 
> Do you guys have any ideas what's going on?  It seems to be waiting
> for bo->reserved to go zero.  Is it possible that someone there is
> forgetting to properly kick a work item after resume causing the wait
> to stall?
> 
> Andrew, can you please kill the X server after the hang and see
> whether that brings the system back?  I think sshd should still work
> and if not you can write a script to kill the X server after 30secs
> after resume (and kill that script if resume succeeds).
> 
> Thank you.
> 
> -- 
> tejun

[REGRESSION]: hibernate/sleep regression w/ bisection

[Andrew Watts]

Hi.

Hibernate/sleep (echo disk/mem > /sys/power/state) has presented problems
for me starting with 2.6.39. Kernel 2.6.37.6 was the last completely bug-free
version I used (I skipped the 2.6.38 branch entirely).

The symptoms are that upon resume (from sleep/hibernate) there is no video
nor any keyboard input with the exception of sysrq.

It has been a frustrating bug to hunt down because it is not easily
reproduced; sometimes the bug doesn't pop up until after a long sequence of
hibernate/sleep cycles.

I successfully bisected the problem to: 8ee294cd9def000.

=======
Commit: 8ee294cd9def0004887da7f44b80563493b0a097
Author: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Date:   Mon Nov 15 01:39:57 2010 -0800
Input: serio - convert to common workqueue instead of a thread
=======

Backing out 8ee294cd9def000 (which requires reversing part of
1d64b655dc083df also) fixes this particular problem on 2.6.39.4,
3.0.8, and 3.1.

Unfortunately, in 3.0.8 and 3.1 I have other suspend/hibernate problems
that I will investigate next and detail in a different regression report.

~ Andy

System specs: P4/Radeon GPU