Re: [2.6.21.1] totally hanging altough not crashed

Folkert van Heusden
2007-07-20T21:01:49Z

> > One of my systems running 2.6.21.1 on a P4 with HT and 2GB of ram
> > occasionally crashes. Not with an oops or panic, it just suddenly stops
> > doing anything. It still lets you ping and it still forwards ip traffic
> > but you can't login: not via ssh and not on the console - pressing enter
> > brings the cursor to the next line and that's it. It is NOT swapping at
> > all, in fact alt+sysrq+m says that there's still plenty of memory and
> > swap available:
>
> Lots of processes are in uninterruptible wait at
> start_this_handle+0x208/0x365. Can you find out what line of code
> that matches? (There are a few of those waits in that function.)

I did some more investigating and found:

folkert@muur:~$ grep -A 1 "Call Trace" www/ast.txt | grep -v -e "^--" -v -e "Call Trace:" | cut -d " " -f 2- | genstats | more
1 358 54.82% 1.75 [<c120f326>] schedule_timeout+0x8c/0x8e
2 88 13.48% 7.42 [<c10c23a0>] start_this_handle+0x208/0x365
3 63 9.65% 10.10 [<c120f2e0>] schedule_timeout+0x46/0x8e
4 36 5.51% 17.69 [<c1020868>] do_wait+0x2c4/0x396
5 36 5.51% 16.81 [<c1098cbd>] inotify_read+0x8d/0x1b5
6 17 2.60% 37.59 [<c1074f52>] pipe_wait+0x8a/0xab
7 13 1.99% 2.69 [<c102d22e>] worker_thread+0x130/0x165
8 11 1.68% 55.73 [<c1210307>] do_nanosleep+0x42/0x70
9 7 1.07% 92.29 [<c120f5e9>] __mutex_lock_slowpath+0xac/0x28c
10 7 1.07% 90.71 [<c101f9fc>] do_exit+0x253/0x428
11 2 0.31% 145.50 [<c1138101>] write_chan+0x15b/0x1d6
12 2 0.31% 3.50 [<c104de2a>] watchdog+0x47/0x55
13 2 0.31% 3.00 [<c10224f6>] ksoftirqd+0x85/0x98
14 2 0.31% 2.50 [<c1018e44>] migration_thread+0x8a/0x10f
15 1 0.15% 617.00 [<c10c7d56>] log_wait_commit+0xb5/0x121
16 1 0.15% 562.00 [<c1029e46>] sys_pause+0x14/0x1b
17 1 0.15% 225.00 [<f8a58ae1>] schluffen+0xad/0xaf [zaptel]
18 1 0.15% 63.00 [<c1029efc>] sys_rt_sigsuspend+0xaf/0xcf
19 1 0.15% 23.00 [<c10c75ff>] kjournald+0x1fd/0x207
20 1 0.15% 22.00 [<c10c4b6f>] journal_commit_transaction+0x242/0xcd3
21 1 0.15% 17.00 [<c105ab1b>] kswapd+0xf7/0x10b
22 1 0.15% 16.00 [<c1191647>] serio_thread+0xfa/0xff
23 1 0.15% 15.00 [<c11730e7>] hub_thread+0xe6/0xe8

(please ignore the fourth column)

As I'm not entirely sure that this an innocent to be in for a kernel I
put this point into gdb:

(gdb) p schedule_timeout
$1 = {long int (long int)} 0xc121a8b4 <schedule_timeout>
(gdb) l *0xC120F3B2
0xc120f3b2 is in __xfrm_state_insert (net/xfrm/xfrm_hash.h:49).
44 static inline unsigned __xfrm_src_hash(xfrm_address_t *daddr,
45 xfrm_address_t *saddr,
46 unsigned short family,
47 unsigned int hmask)
48 {
49 unsigned int h = family;
50 switch (family) {
51 case AF_INET:
52 h ^= __xfrm4_daddr_saddr_hash(daddr, saddr);
53 break;

I very much hope it is of any help.

Folkert van Heusden

--
www.vanheusden.com/multitail - win een vlaai van multivlaai! zorg
ervoor dat multitail opgenomen wordt in Fedora Core, AIX, Solaris of
HP/UX en win een vlaai naar keuze