2.6.26.3 mount process looping on ext3 rw remount

2.6.26.3 mount process looping on ext3 rw remount

[Marc Haber]

Hi,

on my laptop, I am a heavy user of laptop-mode and suspend-to-disk.
Sometimes (maybe once out of ten), when disconnecting or connecting
external power, I have a mount process called by laptopmode busy
looping and taking all available CPU:

  ├─acpid,4680 -c /etc/acpi/events
  │   └─lm_ac_adapter.s,29317 /etc/acpi/actions/lm_ac_adapter.sh
  │       └─laptop_mode,29318 /usr/sbin/laptop_mode auto
  │           └─laptop-mode,29386 /usr/share/laptop-mode-tools/modules/laptop-mode
  │               └─laptop-mode,29390 /usr/share/laptop-mode-tools/modules/laptop-mode
  │                   └─laptop-mode,29409 /usr/share/laptop-mode-tools/modules/laptop-mode
  │                       └─mount,29410 /dev/mapper/usr /mnt/usr -t ext3 -o remount,rw,commit=600

this is what top says
Cpu(s):  2.7%us, 96.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  1.3%hi,  0.0%si, 0.0%st
29410 root      20   0  2068  684  564 R 92.7  0.0  87:37.71 mount

The mount process does not react to SIGKILL, stracing the looping
process doesn't give any output, and the strace gets stuck and does
not react to Ctrl-C. A SIGKILL works for the strace process, though.

To me as a layman this looks like the mount process gets stuck
somewhere in kernel land. I currently have the issue with 2.6.26.3.

In the situation of plugging and/or unplugging the power, the notebook
used to completely freeze in the time when 2.6.24 and 2.6.25.$SMALL
were in use, with 2.6.25.$HIGH and 2.6.26 I haven't hat these freezes
any more. However, I am now plagued with the hanging mount processes.

Any ideas?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: 2.6.26.3 mount process looping on ext3 rw remount

[Andrew Morton]

On Sat, 23 Aug 2008 10:42:28 +0200 Marc Haber <mh+linux-kernel@zugschlus.de> wrote:

> Hi,
> 
> on my laptop, I am a heavy user of laptop-mode and suspend-to-disk.
> Sometimes (maybe once out of ten), when disconnecting or connecting
> external power, I have a mount process called by laptopmode busy
> looping and taking all available CPU:
> 
>   ______acpid,4680 -c /etc/acpi/events
>   ___   ______lm_ac_adapter.s,29317 /etc/acpi/actions/lm_ac_adapter.sh
>   ___       ______laptop_mode,29318 /usr/sbin/laptop_mode auto
>   ___           ______laptop-mode,29386 /usr/share/laptop-mode-tools/modules/laptop-mode
>   ___               ______laptop-mode,29390 /usr/share/laptop-mode-tools/modules/laptop-mode
>   ___                   ______laptop-mode,29409 /usr/share/laptop-mode-tools/modules/laptop-mode
>   ___                       ______mount,29410 /dev/mapper/usr /mnt/usr -t ext3 -o remount,rw,commit=600
> 
> this is what top says
> Cpu(s):  2.7%us, 96.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  1.3%hi,  0.0%si, 0.0%st
> 29410 root      20   0  2068  684  564 R 92.7  0.0  87:37.71 mount
> 
> The mount process does not react to SIGKILL, stracing the looping
> process doesn't give any output, and the strace gets stuck and does
> not react to Ctrl-C. A SIGKILL works for the strace process, though.
> 
> To me as a layman this looks like the mount process gets stuck
> somewhere in kernel land. I currently have the issue with 2.6.26.3.
> 
> In the situation of plugging and/or unplugging the power, the notebook
> used to completely freeze in the time when 2.6.24 and 2.6.25.$SMALL
> were in use, with 2.6.25.$HIGH and 2.6.26 I haven't hat these freezes
> any more. However, I am now plagued with the hanging mount processes.

Yes, it's hung in the kernel.

Please try to get a kernel profile while it's happening.  oprofile
maybe, or just the plain old timer-based profiler.  There's some info
in Documentation/basic_profiling.txt.

> Any ideas?

The profile will tell us where it got stuck.


Actually, a simple alternative is to hit sysrq-P five or ten times. 
Most of the resulting stack traces will point back at where the CPU is
stuck.

This gets a bit hit-or-miss if you have multiple CPUs, because the
sysrq-p trace can land on the wrong CPU.  We recently added a sysrq-l
which will generate a trace on all CPUS.

I think we might recently have broken the sysrq output: some info which
should be coming out isn't.  Altering the logging priority (dmesg -n 7)
might help with that.

Re: 2.6.26.3 mount process looping on ext3 rw remount

[Marc Haber]

On Mon, Aug 25, 2008 at 06:35:28PM -0700, Andrew Morton wrote:
> Please try to get a kernel profile while it's happening.  oprofile
> maybe, or just the plain old timer-based profiler.  There's some info
> in Documentation/basic_profiling.txt.

I'll boot my next kernel with profile=2.

> Actually, a simple alternative is to hit sysrq-P five or ten times. 
> Most of the resulting stack traces will point back at where the CPU is
> stuck.

Where are they written to? "P" in a sysrqd telnet session does not
result in anything being written to the kernel log or to dmesg.

> This gets a bit hit-or-miss if you have multiple CPUs, because the
> sysrq-p trace can land on the wrong CPU.  We recently added a sysrq-l
> which will generate a trace on all CPUS.

"l" results in the SysRq HELP being written to the kernel log, so I
guess that "recently" means "not yet in 2.6.26.3".

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: 2.6.26.3 mount process looping on ext3 rw remount

[Andrew Morton]

On Fri, 5 Sep 2008 13:39:02 +0200 Marc Haber <mh+linux-kernel@zugschlus.de> wrote:

> On Mon, Aug 25, 2008 at 06:35:28PM -0700, Andrew Morton wrote:
> > Please try to get a kernel profile while it's happening.  oprofile
> > maybe, or just the plain old timer-based profiler.  There's some info
> > in Documentation/basic_profiling.txt.
> 
> I'll boot my next kernel with profile=2.
> 
> > Actually, a simple alternative is to hit sysrq-P five or ten times. 
> > Most of the resulting stack traces will point back at where the CPU is
> > stuck.
> 
> Where are they written to? "P" in a sysrqd telnet session does not
> result in anything being written to the kernel log or to dmesg.

alt-sysrq-p from the attached keyboard.

When operating remotely, use `echo p > /proc/sysrq-trigger'.

The output will be in /var/log/messages and is accessible via
`dmesg -s 1000000'.

Please avoid email client wordwrapping when sending these logs.  99% of
people do this and it makes the text very hard to read.

> > This gets a bit hit-or-miss if you have multiple CPUs, because the
> > sysrq-p trace can land on the wrong CPU.  We recently added a sysrq-l
> > which will generate a trace on all CPUS.
> 
> "l" results in the SysRq HELP being written to the kernel log, so I
> guess that "recently" means "not yet in 2.6.26.3".

ok