* Scheduling in interrupt BUG.
@ 2001-05-14 11:18 Marcell GAL
2001-05-14 12:32 ` Scheduling in interrupt BUG. [Patch] Michal Ostrowski
2001-05-14 13:04 ` Scheduling in interrupt BUG Andrew Morton
0 siblings, 2 replies; 6+ messages in thread
From: Marcell GAL @ 2001-05-14 11:18 UTC (permalink / raw)
To: linux-kernel, Paul Mackerras, Michal Ostrowski
Hi Guys,
Once upon a time on my
x86 UP box, UP kernel 2.4.4, (64M ram, 260M swap)
http://home.sch.bme.hu/~cell/.config
I hit a reproducable "Scheduling in interrupt" BUG.
Also reproduced with 128M ram and low memory pressure
(first I suspected it is related to swapping)
Running lots of pppd version 2.4.0 (pppoe) sessions almost at the same
time.
(before the crash the pppoe sessions work fine)
It crashed after 89 sessions, 473 another time.. (depending
on the phase of Jupiter moons I guess .. I still have to verify this),
usually much before memory is exhausted (30k mem/pppd process).
To do this you have to patch ppp_generic.c
http://x-dsl.hu/~cell/ppp_generic_hash/, because
otherwise we hit 'NULL ptr in all_ppp_units list'
BUG much _more likely_ than this 'sched.c line 709 thingy'..
EIP: c010faa4 <schedule+388/394> <===== sched.c schedule(), line 709:
which is ~ printk("Scheduling in interrupt");BUG();
Trace:
0xc01ddac5 <__lock_sock+53>: movl $0x0,0x1c(%esp,1)
0xc01ddacd <__lock_sock+61>: mov %ebx,0x20(%esp,1)
0xc01ddad1 <__lock_sock+65>: movl $0x0,0x24(%esp,1)
0xc01ddad9 <__lock_sock+73>: movl $0x0,0x28(%esp,1)
0xc01ddae1 <__lock_sock+81>: lea 0x1c(%esp,1),%esi
0xc01ddae5 <__lock_sock+85>: lea 0x34(%edi),%eax
0xc01ddae8 <__lock_sock+88>: mov %esi,%edx
0xc01ddaea <__lock_sock+90>: call 0xc0110598
<add_wait_queue_exclusive>
0xc01ddaef <__lock_sock+95>: nop
0xc01ddaf0 <__lock_sock+96>: movl $0x2,(%ebx)
0xc01ddaf6 <__lock_sock+102>: decl 0xc02f75ec
0xc01ddafc <__lock_sock+108>: call 0xc010f71c <schedule>
*****************
0xc01ddb01 <__lock_sock+113>: incl 0xc02f75ec
0xc01ddb07 <__lock_sock+119>: cmpl $0x0,0x30(%edi)
0xc01ddb0b <__lock_sock+123>: jne 0xc01ddaf0 <__lock_sock+96>
-----
0xc01a315c <pppoe_backlog_rcv>: push %esi
0xc01a315d <pppoe_backlog_rcv+1>: push %ebx
0xc01a315e <pppoe_backlog_rcv+2>: mov 0xc(%esp,1),%ebx
0xc01a3162 <pppoe_backlog_rcv+6>: incl 0xc02f75ec
0xc01a3168 <pppoe_backlog_rcv+12>: cmpl $0x0,0x30(%ebx)
0xc01a316c <pppoe_backlog_rcv+16>:
je 0xc01a3177 <pppoe_backlog_rcv+27>
0xc01a316e <pppoe_backlog_rcv+18>: push %ebx
0xc01a316f <pppoe_backlog_rcv+19>: call 0xc01dda90
<__lock_sock> ************
0xc01a3174 <pppoe_backlog_rcv+24>: add $0x4,%esp
0xc01a3177 <pppoe_backlog_rcv+27>: movl $0x1,0x30(%ebx)
0xc01a317e <pppoe_backlog_rcv+34>: decl 0xc02f75ec
0xc01a3184 <pppoe_backlog_rcv+40>: mov 0x10(%esp,1),%eax
--------
0xc01ddb2c <__release_sock>: push %esi
0xc01ddb2d <__release_sock+1>: push %ebx
0xc01ddb2e <__release_sock+2>: mov 0xc(%esp,1),%esi
0xc01ddb32 <__release_sock+6>: mov 0xb8(%esi),%eax
0xc01ddb38 <__release_sock+12>: movl $0x0,0xbc(%esi)
0xc01ddb42 <__release_sock+22>: movl $0x0,0xb8(%esi)
0xc01ddb4c <__release_sock+32>: lea 0x0(%esi,1),%esi
0xc01ddb50 <__release_sock+36>: mov (%eax),%ebx
0xc01ddb52 <__release_sock+38>: movl $0x0,(%eax)
0xc01ddb58 <__release_sock+44>: push %eax
0xc01ddb59 <__release_sock+45>: push %esi
0xc01ddb5a <__release_sock+46>: mov 0x31c(%esi),%eax
0xc01ddb60 <__release_sock+52>: call *%eax ********************
0xc01ddb62 <__release_sock+54>: mov %ebx,%eax
0xc01ddb64 <__release_sock+56>: add $0x8,%esp
0xc01ddb67 <__release_sock+59>: test %eax,%eax
int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
lock_sock(sk);
pppoe_rcv_core(sk, skb);
release_sock(sk);
return 0;
}
What else should I check? How can we fix it?
PPPoE is more and more frequently used nowadays because of ADSL
services. I think this can effect its stability (guess which direction
;-)
even with one session (though probably not that bad as with many
sessions).
Have a nice week:
Cell
--
Alice? WTFIA?
^ permalink raw reply [flat|nested] 6+ messages in thread* Scheduling in interrupt BUG. [Patch]
2001-05-14 11:18 Scheduling in interrupt BUG Marcell GAL
@ 2001-05-14 12:32 ` Michal Ostrowski
2001-05-14 20:21 ` Marcell Gal
2001-05-14 20:45 ` Andi Kleen
2001-05-14 13:04 ` Scheduling in interrupt BUG Andrew Morton
1 sibling, 2 replies; 6+ messages in thread
From: Michal Ostrowski @ 2001-05-14 12:32 UTC (permalink / raw)
To: Marcell GAL, linux-kernel, paulus, David S. Miller
Marcell GAL writes:
> Hi Guys,
>
> Once upon a time on my
> x86 UP box, UP kernel 2.4.4, (64M ram, 260M swap)
> http://home.sch.bme.hu/~cell/.config
> I hit a reproducable "Scheduling in interrupt" BUG.
> Also reproduced with 128M ram and low memory pressure
> (first I suspected it is related to swapping)
> Running lots of pppd version 2.4.0 (pppoe) sessions almost at the same
> time.
> (before the crash the pppoe sessions work fine)
> It crashed after 89 sessions, 473 another time.. (depending
> on the phase of Jupiter moons I guess .. I still have to verify this),
> usually much before memory is exhausted (30k mem/pppd process).
> To do this you have to patch ppp_generic.c
> http://x-dsl.hu/~cell/ppp_generic_hash/, because
> otherwise we hit 'NULL ptr in all_ppp_units list'
> BUG much _more likely_ than this 'sched.c line 709 thingy'..
>
> EIP: c010faa4 <schedule+388/394> <===== sched.c schedule(), line 709:
> which is ~ printk("Scheduling in interrupt");BUG();
>From what I've seen, you have a call chain consisting of:
__release_sock -> pppoe_backlog_rcv -> __lock_sock
This is going to be bad, because when __release_sock calls
sk->backlog_rcv, lock.users is still non-zero and thus the lock_sock
operation will block (leading to deadlock). This problem is fixed
with the patch that I've added below.
You're seeing the "Scheduling in interrupt" message because the
combined effect of the various spin_lock/unlock calls in release_sock
and __release_sock at the time of the call to sk->backlog_rcv is to
increase the local bh count.
Having looked at the code for locking sockets I am concerned that the
locking procedures for tcp may be wrong. __release_sock releases the
socket spinlock before calling sk->backlog_rcv() (== tcp_v4_do_rcv),
however the comments at the top of tcp_v4_do_rcv() assert that the
socket's spinlock is held (which is definitely not the case).
Anybody care to comment on this?
Michal Ostrowski
mostrows@speakeasy.net
--- drivers/net/pppoe.c~ Tue Mar 6 22:44:35 2001
+++ drivers/net/pppoe.c Mon May 14 08:24:06 2001
@@ -5,7 +5,7 @@
* PPPoE --- PPP over Ethernet (RFC 2516)
*
*
- * Version: 0.6.5
+ * Version: 0.6.6
*
* 030700 : Fixed connect logic to allow for disconnect.
* 270700 : Fixed potential SMP problems; we must protect against
@@ -19,6 +19,7 @@
* 051000 : Initialization cleanup.
* 111100 : Fix recvmsg.
* 050101 : Fix PADT procesing.
+ * 140501 : pppoe_backlog_rcv must call bh_lock_sock, not lock_sock.
*
* Author: Michal Ostrowski <mostrows@styx.uwaterloo.ca>
* Contributors:
@@ -384,9 +385,9 @@
***********************************************************************/
int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
- lock_sock(sk);
+ bh_lock_sock(sk);
pppoe_rcv_core(sk, skb);
- release_sock(sk);
+ bh_unlock_sock(sk);
return 0;
}
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Scheduling in interrupt BUG. [Patch]
2001-05-14 12:32 ` Scheduling in interrupt BUG. [Patch] Michal Ostrowski
@ 2001-05-14 20:21 ` Marcell Gal
2001-05-14 20:24 ` Michal Ostrowski
2001-05-14 20:45 ` Andi Kleen
1 sibling, 1 reply; 6+ messages in thread
From: Marcell Gal @ 2001-05-14 20:21 UTC (permalink / raw)
To: mostrows; +Cc: linux-kernel, paulus, David S. Miller
Hi,
This patch solved the problem. Should be ready for inclusion in 2.4.
No more 'Scheduling in interrupt' under those conditions.
Thanx for the thoughts, solution and the amazing speed.
You guys are doing a really great job!
I hope we can get the earlier mentioned NULL ptr in all_ppp_units list
straight
soon. (I have a simple workaround - the mentioned hash, that even improves
speed,
but I a real fix would be more satisfaction. The relevant part of
ppp_generic.c
is so simple that it's really strange it is not correct.. ).
thanx:
Cell
Michal Ostrowski wrote:
> Anybody care to comment on this?
> mostrows@speakeasy.net
--- linuxold/drivers/net/pppoe.c Mon May 14 22:06:44 2001
+++ linux/drivers/net/pppoe.c Mon May 14 22:11:25 2001
@@ -4,9 +4,9 @@
* PPPoX --- Generic PPP encapsulation socket family
* PPPoE --- PPP over Ethernet (RFC 2516)
*
*
- * Version: 0.6.5
+ * Version: 0.6.6
*
* 030700 : Fixed connect logic to allow for disconnect.
* 270700 : Fixed potential SMP problems; we must protect against
* simultaneous invocation of ppp_input
@@ -18,8 +18,9 @@
* in pppoe_release.
* 051000 : Initialization cleanup.
* 111100 : Fix recvmsg.
* 050101 : Fix PADT procesing.
+ * 140501 : pppoe_backlog_rcv must call bh_lock_sock, not lock_sock.
*
* Author: Michal Ostrowski <mostrows@styx.uwaterloo.ca>
* Contributors:
* Arnaldo Carvalho de Melo <acme@xconectiva.com.br>
@@ -383,11 +384,11 @@
*
***********************************************************************/
int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
- lock_sock(sk);
+ bh_lock_sock(sk);
pppoe_rcv_core(sk, skb);
- release_sock(sk);
+ bh_unlock_sock(sk);
return 0;
}
--
You'll never see all the places, or read all the books, but fortunately,
they're not all recommended.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Scheduling in interrupt BUG. [Patch]
2001-05-14 20:21 ` Marcell Gal
@ 2001-05-14 20:24 ` Michal Ostrowski
0 siblings, 0 replies; 6+ messages in thread
From: Michal Ostrowski @ 2001-05-14 20:24 UTC (permalink / raw)
To: Marcell Gal; +Cc: linux-kernel
Marcell Gal writes:
> Hi,
>
> This patch solved the problem. Should be ready for inclusion in 2.4.
> No more 'Scheduling in interrupt' under those conditions.
> Thanx for the thoughts, solution and the amazing speed.
> You guys are doing a really great job!
>
Alexey pointed out a much nicer/correct/elegant/efficient
solution to this problem and I think that that's the way to go.
New patch below.
Michal Ostrowski
mostrows@speakeasy.net
--- drivers/net/pppoe.c~ Tue Mar 6 22:44:35 2001
+++ drivers/net/pppoe.c Mon May 14 14:10:49 2001
@@ -5,7 +5,7 @@
* PPPoE --- PPP over Ethernet (RFC 2516)
*
*
- * Version: 0.6.5
+ * Version: 0.6.6
*
* 030700 : Fixed connect logic to allow for disconnect.
* 270700 : Fixed potential SMP problems; we must protect against
@@ -19,6 +19,7 @@
* 051000 : Initialization cleanup.
* 111100 : Fix recvmsg.
* 050101 : Fix PADT procesing.
+ * 140501 : Use pppoe_rcv_core to handle all backlog. (Alexey)
*
* Author: Michal Ostrowski <mostrows@styx.uwaterloo.ca>
* Contributors:
@@ -376,22 +377,6 @@
return ret;
}
-
-/************************************************************************
- *
- * Receive wrapper called in process context.
- *
- ***********************************************************************/
-int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
-{
- lock_sock(sk);
- pppoe_rcv_core(sk, skb);
- release_sock(sk);
- return 0;
-}
-
-
-
/************************************************************************
*
* Receive a PPPoE Discovery frame.
@@ -481,7 +466,7 @@
sk->protocol = PX_PROTO_OE;
sk->family = PF_PPPOX;
- sk->backlog_rcv = pppoe_backlog_rcv;
+ sk->backlog_rcv = pppoe_rcv_core;
sk->next = NULL;
sk->pprev = NULL;
sk->state = PPPOX_NONE;
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Scheduling in interrupt BUG. [Patch]
2001-05-14 12:32 ` Scheduling in interrupt BUG. [Patch] Michal Ostrowski
2001-05-14 20:21 ` Marcell Gal
@ 2001-05-14 20:45 ` Andi Kleen
1 sibling, 0 replies; 6+ messages in thread
From: Andi Kleen @ 2001-05-14 20:45 UTC (permalink / raw)
To: mostrows; +Cc: Marcell GAL, linux-kernel, paulus, David S. Miller
On Mon, May 14, 2001 at 08:32:33AM -0400, Michal Ostrowski wrote:
> Having looked at the code for locking sockets I am concerned that the
> locking procedures for tcp may be wrong. __release_sock releases the
> socket spinlock before calling sk->backlog_rcv() (== tcp_v4_do_rcv),
> however the comments at the top of tcp_v4_do_rcv() assert that the
> socket's spinlock is held (which is definitely not the case).
>
> Anybody care to comment on this?
Looks ok for me.
The user socket lock (lock.users>0) is held while __release_sock runs,
which is also sufficient to protect it as all new packets will go into backlog.
The spinlock comment only applies to bottom halves.
-Andi
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Scheduling in interrupt BUG.
2001-05-14 11:18 Scheduling in interrupt BUG Marcell GAL
2001-05-14 12:32 ` Scheduling in interrupt BUG. [Patch] Michal Ostrowski
@ 2001-05-14 13:04 ` Andrew Morton
1 sibling, 0 replies; 6+ messages in thread
From: Andrew Morton @ 2001-05-14 13:04 UTC (permalink / raw)
To: Marcell GAL; +Cc: linux-kernel, Paul Mackerras, Michal Ostrowski
Marcell GAL wrote:
>
> int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
> {
> lock_sock(sk);
> pppoe_rcv_core(sk, skb);
> release_sock(sk);
> return 0;
> }
>
The backlog_rcv() method is called inside local_bh_disable()
and so cannot call lock_sock(). Definitely a bug in pppoe.
It looks like pppoe_backlog_rcv() should be using bh_lock_sock().
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2001-05-14 20:45 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2001-05-14 11:18 Scheduling in interrupt BUG Marcell GAL
2001-05-14 12:32 ` Scheduling in interrupt BUG. [Patch] Michal Ostrowski
2001-05-14 20:21 ` Marcell Gal
2001-05-14 20:24 ` Michal Ostrowski
2001-05-14 20:45 ` Andi Kleen
2001-05-14 13:04 ` Scheduling in interrupt BUG Andrew Morton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox