* 2.6.38.x: system hangs, iowait ~75, no resource-hogs, no logs
@ 2011-04-07 22:21 Thomas Sattler
2011-04-07 23:14 ` Kelly Anderson
0 siblings, 1 reply; 3+ messages in thread
From: Thomas Sattler @ 2011-04-07 22:21 UTC (permalink / raw)
To: Linux Kernel Mailing List
Hi there ...
I observed three system hangs since 2.6.38 (two times 2.6.38,
once 2.6.38.2). I have no idea what triggers the problem, I
cannot find the cause myself. I tried these:
top:
~~~~
- no cpu-hogg, still a bit free RAM, lots of free SWAP
- iowait is constantly high at ~75%, I don't know why
dmesg, /var/log/*:
~~~~~~~~~~~~~~~~~~
Nothing in 'dmesg', nothing in the logfiles.
pkill / iotop / ps:
~~~~~~~~~~~~~~~~~~~
they all hang quite soon, no reaction on Ctrl-C. While 'ps'
shows some lines of output, 'pkill' and 'iotop' just hang.
vmstat 1:
~~~~~~~~~
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 4 72 421008 54560 326652 0 0 0 0 518 1293 25 9 0 66
0 4 72 421008 54560 326652 0 0 0 0 311 535 13 4 0 83
2 4 72 420208 54560 326652 0 0 0 0 450 882 18 9 0 73
0 4 72 421008 54560 326652 0 0 0 0 377 914 19 5 0 76
(I collected about 90 lines, there's not much to be seen.)
I run 'strace -o dump command' with iotop, ps and su.
(And lost a tty to each of them.)
$ tail dump_*
==> dump_iotop <==
fstat64(9, {st_mode=S_IFREG|0644, st_size=2952, ...}) = 0
mmap2(NULL, 2952, PROT_READ, MAP_SHARED, 9, 0) = 0xb6d80000
_llseek(9, 2952, [2952], SEEK_SET) = 0
munmap(0xb6d80000, 2952) = 0
close(9) = 0
open("/proc/5904/cmdline", O_RDONLY|O_LARGEFILE) = 9
fstat64(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fstat64(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0xb6d80000
read(9,
==> dump_ps <==
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
stat64("/proc/5904", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/5904/stat", O_RDONLY) = 6
read(6, "5904 (thunderbird-bin) D 5900 58"..., 1023) = 242
close(6) = 0
open("/proc/5904/status", O_RDONLY) = 6
read(6, "Name:\tthunderbird-bin\nState:\tD ("..., 1023) = 836
close(6) = 0
open("/proc/5904/cmdline", O_RDONLY) = 6
read(6,
==> dump_su <==
stat64("/home/tsattler/.pam_environment", 0xbffdb3e0) = -1 ENOENT (No such
file or directory)
access("/usr/bin/xauth", X_OK) = 0
setuid32(1000) = 0
chdir("/home/tsattler") = 0
close(3) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0xb75f3728) = 25247
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], NULL, 8) = 0
rt_sigaction(SIGTERM, {0x8049ce0, [], 0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [ALRM TERM], NULL, 8) = 0
waitpid(-1,
'top' showed constantly increasing sysload, I rebooted
(via Alt-SysRq-[S][U][B]) at a load of about 25-30.
My system is as follows:
~~~~~~~~~~~~~~~~~~~~~~~~
IBM T41p, 1.5GB RAM, 4GB SWAP (nearly unused), two HDs
160/80GB. Gentoo Linux, kernel 2.6.38.2 (vanilla sources).
I compiled the kernel myself. I skipped 2.6.37 but run
2.6.36.3 with sabayon's fourth version of a backport of
sched-automated-per-session-task-groups.patch for more
than two months without any problems.
CPU-Speed is 1700MHz (hardware) but I run the machine
at 1400MHz most of the time as the fans become loud
when running at full speed.
** Please CC: me, as I'm not on the list. **
Thomas
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: 2.6.38.x: system hangs, iowait ~75, no resource-hogs, no logs
2011-04-07 22:21 2.6.38.x: system hangs, iowait ~75, no resource-hogs, no logs Thomas Sattler
@ 2011-04-07 23:14 ` Kelly Anderson
2011-04-12 22:37 ` Thomas Sattler
0 siblings, 1 reply; 3+ messages in thread
From: Kelly Anderson @ 2011-04-07 23:14 UTC (permalink / raw)
To: Thomas Sattler; +Cc: Linux Kernel Mailing List
On 04/07/11 16:21, Thomas Sattler wrote:
> Hi there ...
>
> I observed three system hangs since 2.6.38 (two times 2.6.38,
> once 2.6.38.2). I have no idea what triggers the problem, I
> cannot find the cause myself. I tried these:
>
> top:
> ~~~~
> - no cpu-hogg, still a bit free RAM, lots of free SWAP
> - iowait is constantly high at ~75%, I don't know why
>
> dmesg, /var/log/*:
> ~~~~~~~~~~~~~~~~~~
> Nothing in 'dmesg', nothing in the logfiles.
>
> pkill / iotop / ps:
> ~~~~~~~~~~~~~~~~~~~
> they all hang quite soon, no reaction on Ctrl-C. While 'ps'
> shows some lines of output, 'pkill' and 'iotop' just hang.
>
> vmstat 1:
> ~~~~~~~~~
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 4 72 421008 54560 326652 0 0 0 0 518 1293 25 9 0 66
> 0 4 72 421008 54560 326652 0 0 0 0 311 535 13 4 0 83
> 2 4 72 420208 54560 326652 0 0 0 0 450 882 18 9 0 73
> 0 4 72 421008 54560 326652 0 0 0 0 377 914 19 5 0 76
> (I collected about 90 lines, there's not much to be seen.)
>
> I run 'strace -o dump command' with iotop, ps and su.
> (And lost a tty to each of them.)
>
> $ tail dump_*
> ==> dump_iotop<==
> fstat64(9, {st_mode=S_IFREG|0644, st_size=2952, ...}) = 0
> mmap2(NULL, 2952, PROT_READ, MAP_SHARED, 9, 0) = 0xb6d80000
> _llseek(9, 2952, [2952], SEEK_SET) = 0
> munmap(0xb6d80000, 2952) = 0
> close(9) = 0
> open("/proc/5904/cmdline", O_RDONLY|O_LARGEFILE) = 9
> fstat64(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> fstat64(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0xb6d80000
> read(9,
> ==> dump_ps<==
> stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
> stat64("/proc/5904", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
> open("/proc/5904/stat", O_RDONLY) = 6
> read(6, "5904 (thunderbird-bin) D 5900 58"..., 1023) = 242
> close(6) = 0
> open("/proc/5904/status", O_RDONLY) = 6
> read(6, "Name:\tthunderbird-bin\nState:\tD ("..., 1023) = 836
> close(6) = 0
> open("/proc/5904/cmdline", O_RDONLY) = 6
> read(6,
> ==> dump_su<==
> stat64("/home/tsattler/.pam_environment", 0xbffdb3e0) = -1 ENOENT (No such
> file or directory)
> access("/usr/bin/xauth", X_OK) = 0
> setuid32(1000) = 0
> chdir("/home/tsattler") = 0
> close(3) = 0
> clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0xb75f3728) = 25247
> rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], NULL, 8) = 0
> rt_sigaction(SIGTERM, {0x8049ce0, [], 0}, NULL, 8) = 0
> rt_sigprocmask(SIG_UNBLOCK, [ALRM TERM], NULL, 8) = 0
> waitpid(-1,
>
> 'top' showed constantly increasing sysload, I rebooted
> (via Alt-SysRq-[S][U][B]) at a load of about 25-30.
>
> My system is as follows:
> ~~~~~~~~~~~~~~~~~~~~~~~~
> IBM T41p, 1.5GB RAM, 4GB SWAP (nearly unused), two HDs
> 160/80GB. Gentoo Linux, kernel 2.6.38.2 (vanilla sources).
>
> I compiled the kernel myself. I skipped 2.6.37 but run
> 2.6.36.3 with sabayon's fourth version of a backport of
> sched-automated-per-session-task-groups.patch for more
> than two months without any problems.
>
> CPU-Speed is 1700MHz (hardware) but I run the machine
> at 1400MHz most of the time as the fans become loud
> when running at full speed.
>
> ** Please CC: me, as I'm not on the list. **
>
> Thomas
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
Try the patch found in "Linux 2.6.38 freeze because of
sound/core/pcm_lib.c commit"
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: 2.6.38.x: system hangs, iowait ~75, no resource-hogs, no logs
2011-04-07 23:14 ` Kelly Anderson
@ 2011-04-12 22:37 ` Thomas Sattler
0 siblings, 0 replies; 3+ messages in thread
From: Thomas Sattler @ 2011-04-12 22:37 UTC (permalink / raw)
To: Linux Kernel Mailing List; +Cc: Kelly Anderson
On 04/08/11 01:14, Kelly Anderson wrote:
> Try the patch found in "Linux 2.6.38 freeze because of
> sound/core/pcm_lib.c commit"
I applied https://lkml.org/lkml/diff/2011/4/1/69/1 but that
did not (completely?) solve my problem:
While running firefox (>50 open Tabs), khugepaged suddenly
run at 99.99% IO (according to 'iotop'). X became unusable
within seconds, but the system itself was still running.
I was able to switch to tty1 where everything was quite
normal, except khugepaged beeing constantly above 90%.
Switching back to X showed a frozen windowmanager with
empty windows. Back to tty1 I was able to kill X and
start a new X session, which behaved as usual.
khugepaged kept the system busy for another 10min so I
rebooted the machine. (This time there was no problem
running 'pkill' and 'iotop' several times.)
My current kernel is configured like this:
$ zgrep HUGE /proc/config.gz
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
** Please CC: me, as I'm not on the list. **
Thomas
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-04-12 22:37 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-04-07 22:21 2.6.38.x: system hangs, iowait ~75, no resource-hogs, no logs Thomas Sattler
2011-04-07 23:14 ` Kelly Anderson
2011-04-12 22:37 ` Thomas Sattler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox