uninteruptable sleep

uninteruptable sleep

[Trevor Nichols]

Hi all,

Since upgrading to the latest stable (2.4.3) kernel, I've noticed that
randomly some processes are going into an uninteruptable sleep and not
waking up at all.

It's happened to nautilus and today just happened to mozilla also.
Another common related problem is the load averages go up to n + "normal"
where n is the number of processes that have gone uninteruptable sleep.
This is making me think it's a kernel related problem.

I had one time where nautilus with 9 [presumably forked] processes of
itself go this way, causing load averages to go 9+, however the system
doesn't appear to be straining or strugling under that much load.

The previous kernel version that I was using (2.4.1) did not have this
problem.

One last thing, if this turns out to be a non-kernel problem, the
processes that *do* get stuck, are unkillable - even by root with SIGKILL.
Is there any way for it to be able to? :)  So far I have to reboot each
time it happens.


Best regards,
Trevor Nichols.


ps please CC replies to my address. thanks.

Re: uninteruptable sleep

[Stephen E. Clark]

That happened to me with 2.4.2-ac28 when I tried using DRM.
I also got the following messages in syslog.

/var/log/messages.1:Mar 31 12:15:04 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:15:04 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:15:15 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:15:15 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:15:16 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:15:40 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:16:18 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:16:31 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:16:32 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:16:45 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:16:45 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:16:48 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!
/var/log/messages.1:Mar 31 12:16:49 joker kernel:
[drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!/

So I turned off DRI in X 4.0.3

HTH
Steve

Trevor Nichols wrote:
> 
> Hi all,
> 
> Since upgrading to the latest stable (2.4.3) kernel, I've noticed that
> randomly some processes are going into an uninteruptable sleep and not
> waking up at all.
> 
> It's happened to nautilus and today just happened to mozilla also.
> Another common related problem is the load averages go up to n + "normal"
> where n is the number of processes that have gone uninteruptable sleep.
> This is making me think it's a kernel related problem.
> 
> I had one time where nautilus with 9 [presumably forked] processes of
> itself go this way, causing load averages to go 9+, however the system
> doesn't appear to be straining or strugling under that much load.
> 
> The previous kernel version that I was using (2.4.1) did not have this
> problem.
> 
> One last thing, if this turns out to be a non-kernel problem, the
> processes that *do* get stuck, are unkillable - even by root with SIGKILL.
> Is there any way for it to be able to? :)  So far I have to reboot each
> time it happens.
> 
> Best regards,
> Trevor Nichols.
> 
> ps please CC replies to my address. thanks.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: uninteruptable sleep

[Alan Cox]

> One last thing, if this turns out to be a non-kernel problem, the
> processes that *do* get stuck, are unkillable - even by root with SIGKILL.
> Is there any way for it to be able to? :)  So far I have to reboot each
> time it happens.

Its a kernel bug if it gets stuck like this. You need to provide more info
though - what file system, what devices, how much memory. Also ps can give you
the wait address of a process stuck in 'D' state which is valuable for debug

Re: uninteruptable sleep

[Trevor Nichols]

> Its a kernel bug if it gets stuck like this. You need to provide more info
> though - what file system, what devices, how much memory. Also ps can give you
> the wait address of a process stuck in 'D' state which is valuable for debug

System specs:
Pentium 200 MMX
80MB RAM

2 IDE Drives:
SAMSUNG SV0844D 8.4GB
WDC AC21200H 1.2GB

All partitions are ext2 filesytems.

ps xl:
  F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME  COMMAND
040  1000  1230     1   9   0 24320    4 down_w D    ?          0:00  /home/data/mozilla/obj/dist/bin/mozi

[I'm not exactly sure how to get the wait address if it isn't shown above]

Other stuff:

Creative SB AWE64 PnP
16MB Voodoo 3 2000 and a 2MB S3 Virge display
RealTek RTL-8029 NIC
Sony CRX100E Burner

I'm running X in a dual-head configuration using the above 2 cards.
That's all I can think of at this time.

Thanks,
Trevor Nichols.

Re: uninteruptable sleep

[Manfred Spraul]

> ps xl:
>   F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
> 040 1000 1230 1 9 0 24320 4 down_w D ? 0:00
>           /home/data/mozilla/obj/dist/bin/mozi
>
down_w

Perhaps down_write_failed()? 2.4.3 converted the mmap semaphore to a
rw-sem.
Did you compile sysrq into your kernel? Then enable it with

#echo 1 > /proc/sys/kernel/sysrq
and press <Alt>+<SysRQ>+'t'

It prints the complete back trace, not just one function name

--
    Manfred

Re: uninteruptable sleep

[J Sloan]

Trevor Nichols wrote:

> > Its a kernel bug if it gets stuck like this. You need to provide more info
> > though - what file system, what devices, how much memory. Also ps can give you
> > the wait address of a process stuck in 'D' state which is valuable for debug
>
> ps xl:
>   F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME  COMMAND
> 040  1000  1230     1   9   0 24320    4 down_w D    ?          0:00  /home/data/mozilla/obj/dist/bin/mozi
>
> [I'm not exactly sure how to get the wait address if it isn't shown above]
>

Try this:

ps -eo pid,stat,pcpu,nwchan,wchan=WIDE-WCHAN-COLUMN -o args


cu

Jup

Re: uninteruptable sleep

[Trevor Nichols]

> Did you compile sysrq into your kernel?

I haven't yet.  I'll enable it and see if I can trigger it next time I
reboot again.


> ps -eo pid,stat,pcpu,nwchan,wchan=WIDE-WCHAN-COLUMN -o args

1230 D     0.0 105cc1 down_write_failed /home/data/mozilla/obj/dist/bin/mozilla-bin


Hopefully that helps a bit more.

-Trev

Re: uninteruptable sleep

[christophe barbe]

This problem seems to be related with the recent post from David Howells <dhowells@cambridge.redhat.com> with the subject "rw_semaphore bug".

Christophe

On mar, 03 avr 2001 18:40:53 Manfred Spraul wrote:
> > ps xl:
> >   F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
> > 040 1000 1230 1 9 0 24320 4 down_w D ? 0:00
> >           /home/data/mozilla/obj/dist/bin/mozi
> >
> down_w
> 
> Perhaps down_write_failed()? 2.4.3 converted the mmap semaphore to a
> rw-sem.
> Did you compile sysrq into your kernel? Then enable it with
> 
> #echo 1 > /proc/sys/kernel/sysrq
> and press <Alt>+<SysRQ>+'t'
> 
> It prints the complete back trace, not just one function name
> 
> --
>     Manfred
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
-- 
Christophe Barbé
Software Engineer
Lineo High Availability Group
42-46, rue Médéric
92110 Clichy - France
phone (33).1.41.40.02.12
fax (33).1.41.40.02.01
www.lineo.com

Re: uninteruptable sleep

[andersg]

On Wed, Apr 04, 2001 at 08:39:19AM +0930, Trevor Nichols wrote:

> > ps -eo pid,stat,pcpu,nwchan,wchan=WIDE-WCHAN-COLUMN -o args
> 
> 1230 D     0.0 105cc1 down_write_failed /home/data/mozilla/obj/dist/bin/mozilla-bin

My mysql-server got stuck in down_write_failed today too.
SMP dual PentiumIII system with no swap. I can provide more info at request
and is willing to do more bug-hunting if that is needed.

-- 

//anders/g

Re: uninteruptable sleep

[Christian Pernegger]

> Its a kernel bug if it gets stuck like this. You need to provide more info
> though - what file system, what devices, how much memory. Also ps can give you
> the wait address of a process stuck in 'D' state which is valuable for debug

Let's see if I'm getting this right, processes in D state should be killable?

I do not know if this is related, but I had two occurrences of those within
the last 48 hours, albeit on 2.2.18. 

1. starting tin (1.4.1) as a user - nothing happened, but the ssh session froze.
   Same on second try and a third with mutt (1.2.5) The three processes ended up
   D and unkillable by root. A few seconds later the sytem became totally
   unresponsive, with the kernel spewing 'VM: do_try_to_free_pages faild for
   cupsd' (sp?) at top speed... reboot.

2. The cups (1.1.4) usb backend ended up in this state after I did a
   'rmmod printer; insmod printer'

Regards
	Christian


Kernel: linux-2.2.18 + raid-2.2.17-A0

System: Pentium III 600 w/ 256MB RAM

hard disks: 
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: IBM      Model: DDRS-34560D      Rev: DC1B
  Type:   Direct-Access                    ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: YAMAHA   Model: CRW4260          Rev: 1.0q
  Type:   CD-ROM                           ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 08 Lun: 00
  Vendor: QUANTUM  Model: ATLAS_V_18_WLS   Rev: 0230
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 09 Lun: 00
  Vendor: QUANTUM  Model: ATLAS_V_18_WLS   Rev: 0230
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 10 Lun: 00
  Vendor: QUANTUM  Model: ATLAS_V_18_WLS   Rev: 0230
  Type:   Direct-Access                    ANSI SCSI revision: 03

RAID info:
Personalities : [raid5] 
read_ahead 1024 sectors
md0 : active raid5 sdd1[2] sdc1[1] sdb1[0] 35069824 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>

Layout of RAID disks (sdb-sdd):
Disk /dev/sdb: 64 heads, 32 sectors, 17510 cylinders
Units = cylinders of 2048 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdb1           387     17510  17534976   fd  Linux raid autodetect
/dev/sdb3           130       386    263168   83  Linux
/dev/sdb4             1       129    132095+  82  Linux swap

lspci -vvv:
00:00.0 Host bridge: Intel Corporation 440BX/ZX - 82443BX/ZX Host bridge (rev 03)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX - 82443BX/ZX AGP bridge (rev 03)
00:07.0 ISA bridge: Intel Corporation 82371AB PIIX4 ISA (rev 02)
00:07.1 IDE interface: Intel Corporation 82371AB PIIX4 IDE (rev 01)
00:07.2 USB Controller: Intel Corporation 82371AB PIIX4 USB (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB PIIX4 ACPI (rev 02)
00:08.0 VGA compatible controller: Cirrus Logic GD 5430/40 [Alpine] (rev 48)
00:09.0 SCSI storage controller: Adaptec 7892A (rev 02)
00:0b.0 Ethernet controller: 3Com Corporation 3c905C-TX [Fast Etherlink] (rev 74)
jesus:/raid/home/chris# lspci -vvv
00:00.0 Host bridge: Intel Corporation 440BX/ZX - 82443BX/ZX Host bridge (rev 03)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR+
        Latency: 64 set
        Region 0: Memory at d8000000 (32-bit, prefetchable)
        Capabilities: [a0] AGP version 1.0
                Status: RQ=31 SBA+ 64bit- FW- Rate=21
                Command: RQ=0 SBA- AGP- 64bit- FW- Rate=

00:01.0 PCI bridge: Intel Corporation 440BX/ZX - 82443BX/ZX AGP bridge (rev 03) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 64 set
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: fff00000-000fffff
        Prefetchable memory behind bridge: fff00000-000fffff
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B+

00:07.0 ISA bridge: Intel Corporation 82371AB PIIX4 ISA (rev 02)
        Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0 set

00:07.1 IDE interface: Intel Corporation 82371AB PIIX4 IDE (rev 01) (prog-if 80 [Master])
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Region 4: I/O ports at f000 [disabled]

00:07.2 USB Controller: Intel Corporation 82371AB PIIX4 USB (rev 01) (prog-if 00 [UHCI])
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 64 set
        Interrupt: pin D routed to IRQ 15
        Region 4: I/O ports at e000

00:07.3 Bridge: Intel Corporation 82371AB PIIX4 ACPI (rev 02)
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-

00:08.0 VGA compatible controller: Cirrus Logic GD 5430/40 [Alpine] (rev 48) (prog-if 00 [VGA])
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at dc000000 (32-bit, prefetchable)
        Expansion ROM at dd000000 [disabled]

00:09.0 SCSI storage controller: Adaptec 7892A (rev 02)
        Subsystem: Adaptec: Unknown device e2a0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 40 min, 25 max, 64 set, cache line size 08
        Interrupt: pin A routed to IRQ 10
        BIST result: 00
        Region 0: I/O ports at e400
        Region 1: Memory at e0001000 (64-bit, non-prefetchable)
        Expansion ROM at de000000 [disabled]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- AuxPwr- DSI- D1- D2- PME-
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

00:0b.0 Ethernet controller: 3Com Corporation 3c905C-TX [Fast Etherlink] (rev 74)
        Subsystem: 3Com Corporation: Unknown device 1000
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 10 min, 10 max, 64 set, cache line size 08
        Interrupt: pin A routed to IRQ 15
        Region 0: I/O ports at e800
        Region 1: Memory at e0000000 (32-bit, non-prefetchable)
        Expansion ROM at df000000 [disabled]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- AuxPwr- DSI- D1+ D2+ PME+
                Status: D0 PME-Enable+ DSel=0 DScale=2 PME-

Re: uninteruptable sleep

[Anton Blanchard]

> That happened to me with 2.4.2-ac28 when I tried using DRM.
> I also got the following messages in syslog.
> 
> /var/log/messages.1:Mar 31 12:15:04 joker kernel:
> [drm:r128_do_wait_for_fifo] *ERROR* r128_do_wait_for_fifo failed!

You need to replace down(...->mmap_sem), up(...->mmap_sem) with
down_write(...), up_write(...) in the X11 r128 drm kernel module.

Anton