From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michal Ostrowski Date: Mon, 19 Oct 2009 03:34:06 +0000 Subject: Re: kernel panic in latest vanilla stable, while using nameif with Message-Id: List-Id: References: <200910190002.39937.denys@visp.net.lb> In-Reply-To: <200910190002.39937.denys@visp.net.lb> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: Denys Fedoryschenko Cc: netdev , linux-ppp@vger.kernel.org, paulus@samba.org, mostrows@earthlink.net Here's my theory on this after an inital look... Looking at the oops report and disassembly of the actual module binary that caused the oops, one can deduce that: Execution was in pppoe_flush_dev(). =A0%ebx contained the pointer "struct pppox_sock *po", which is what we faulted on, excuting "cmp %eax, 0x190(%eb= x)". %ebx value was 0xffffffff (hence we got "NULL pointer dereference at 0x18f"= ). At this point "i" (stored in %esi) is 15 (valid), meaning that we got a val= ue of 0xffffffff in pn->hash_table[i]. >From this I'd hypothesize that the combination of dev_put() and release_soc= k() may have allowed us to free "pn". =A0At the bottom of the loop we alreayd recognize that since locks are dropped we're responsible for handling invalidation of objects, and perhaps that should be extended to "pn" as wel= l. -- Michal Ostrowski mostrows@gmail.com --- =A0drivers/net/pppoe.c | =A0 86 ++++++++++++++++++++++++++---- -------------------- =A01 files changed, 45 insertions(+), 41 deletions(-) diff --git a/drivers/net/pppoe.c b/drivers/net/pppoe.c index 7cbf6f9..720c4ea 100644 --- a/drivers/net/pppoe.c +++ b/drivers/net/pppoe.c @@ -296,6 +296,7 @@ static void pppoe_flush_dev(struct net_device *dev) =A0 =A0 =A0 =A0BUG_ON(dev =3D NULL); +restart: =A0 =A0 =A0 =A0pn =3D pppoe_pernet(dev_net(dev)); =A0 =A0 =A0 =A0if (!pn) /* already freed */ =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return; @@ -303,48 +304,51 @@ static void pppoe_flush_dev(struct net_device *dev) =A0 =A0 =A0 =A0write_lock_bh(&pn->hash_lock); =A0 =A0 =A0 =A0for (i =3D 0; i < PPPOE_HASH_SIZE; i++) { =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0struct pppox_sock *po =3D pn->hash_table[i]; + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0struct sock *sk; - =A0 =A0 =A0 =A0 =A0 =A0 =A0 while (po !=3D NULL) { - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct sock *sk; - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (po->pppoe_dev !=3D dev) { - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 po =3D po->ne= xt; - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 continue; - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sk =3D sk_pppox(po); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_lock(&flush_lock); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 po->pppoe_dev =3D NULL; - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock(&flush_lock); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 dev_put(dev); - - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* We always grab the socket = lock, followed by the - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* hash_lock, in that order= . =A0Since we should - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* hold the sock lock while= doing any unbinding, - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* we need to release the l= ock we're holding. - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* Hold a reference to the = sock so it doesn't disappear - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* as we're jumping between= locks. - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0*/ + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0while (po && po->pppoe_dev !=3D dev) { + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0po =3D po->next; + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sock_hold(sk); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (po =3D NULL) { + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0continue; + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 write_unlock_bh(&pn->hash_loc= k); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 lock_sock(sk); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sk =3D sk_pppox(po); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (sk->sk_state & (PPPOX_CON= NECTED | PPPOX_BOUND)) { - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pppox_unbind_= sock(sk); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sk->sk_state = =3D PPPOX_ZOMBIE; - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sk->sk_state_= change(sk); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0spin_lock(&flush_lock); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0po->pppoe_dev =3D NULL; + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0spin_unlock(&flush_lock); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 release_sock(sk); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sock_put(sk); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0dev_put(dev); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Restart scan at the beginn= ing of this hash chain. - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* While the lock was dropp= ed the chain contents may - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* have changed. - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0*/ - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 write_lock_bh(&pn->hash_lock); - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 po =3D pn->hash_table[i]; - =A0 =A0 =A0 =A0 =A0 =A0 =A0 } + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* We always grab the socket lock, followe= d by the + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * hash_lock, in that order. =A0Since we s= hould + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * hold the sock lock while doing any unbi= nding, + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * we need to release the lock we're holdi= ng. + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Hold a reference to the sock so it does= n't disappear + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * as we're jumping between locks. + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */ + + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sock_hold(sk); + + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0write_unlock_bh(&pn->hash_lock); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0lock_sock(sk); + + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (sk->sk_state & (PPPOX_CONNECTED | PPPO= X_BOUND)) { + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0pppox_unbind_sock(sk); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sk->sk_state =3D PPPOX_ZOM= BIE; + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sk->sk_state_change(sk); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} + + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0release_sock(sk); + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sock_put(sk); + + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* Restart the flush process from the begi= nning. =A0While + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * the lock was dropped the chain contents= may have + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * changed, and sock_put may have made thi= ngs go away. + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */ + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0goto restart; =A0 =A0 =A0 =A0} =A0 =A0 =A0 =A0write_unlock_bh(&pn->hash_lock); =A0} -- 1.6.3.3 On Sun, Oct 18, 2009 at 4:02 PM, Denys Fedoryschenko wr= ote: > > I have server running as pppoe NAS. > Tried to rename customers without dropping pppd connections first, got pa= nic > after few seconds. > Panic triggerable at 2.6.30.4 and 2.6.31.4 > pppoe users running on eth2 > pppoe flags: > =A01457 root =A0 =A0 /usr/sbin/pppoe-server -I eth2 -k -L 172.16.1.1 -R > 172.16.1.2 -N 253 -C gpzone -S gpzone > > > Commands sequence that i think triggered that: > > ip link set eth0 down > ip link set eth1 down > ip link set eth2 down > nameif etherx 00:16:76:8D:83:BA > nameif eth0 00:19:e0:72:4a:37 > nameif eth1 00:19:e0:72:4a:4b > > ip addr flush dev eth0 > ip addr flush dev eth1 > ip addr add X.X.X.X/29 dev eth0 > ip addr add 192.168.2.177/24 dev eth0 > ip addr add 192.168.0.1/32 dev eth1 > ip addr add 127.0.0.0/8 dev lo > #ip link set eth0 up > ip link set eth0 up > ip link set eth1 up > ip link set lo up > ip route add 0.0.0.0/0 via X.X.X.X > > > [ =A0103.428591] r8169: eth0: link up > [ =A0103.430360] r8169: eth1: link up > [ =A0113.361528] BUG: unable to handle kernel > NULL pointer dereference > at 0000018f > [ =A0113.361717] IP: > [] pppoe_device_event+0x80/0x12c [pppoe] > [ =A0113.361853] *pdpt =3D 000000003411a001 > *pde =3D 0000000000000000 > Oct 18 23:59:40 194.146.153.93 > [ =A0113.362012] Oops: 0000 [#1] > SMP > Oct 18 23:59:40 194.146.153.93 > [ =A0113.362166] last sysfs file: /sys/devices/virtual/vc/vcs3/dev > [ =A0113.362246] Modules linked in: > netconsole > configfs > act_skbedit > sch_ingress > sch_prio > cls_flow > cls_u32 > em_meta > cls_basic > xt_dscp > xt_DSCP > ipt_REJECT > ts_bm > xt_string > xt_hl > ifb > cls_fw > sch_tbf > sch_htb > act_ipt > act_mirred > xt_MARK > pppoe > pppox > ppp_generic > slhc > xt_TCPMSS > xt_mark > xt_tcpudp > iptable_mangle > iptable_nat > nf_nat > rtc_cmos > nf_conntrack_ipv4 > rtc_core > nf_conntrack > rtc_lib > nf_defrag_ipv4 > iptable_filter > ip_tables > x_tables > 8021q > garp > stp > llc > loop > sata_sil > pata_atiixp > pata_acpi > ata_generic > libata > 8139cp > usb_storage > mtdblock > mtd_blkdevs > mtd > sr_mod > cdrom > tulip > r8169 > sky2 > via_velocity > via_rhine > sis900 > ne2k_pci > 8390 > skge > tg3 > libphy > 8139too > e1000 > e100 > usbhid > ohci_hcd > uhci_hcd > ehci_hcd > usbcore > nls_base > Oct 18 23:59:40 194.146.153.93 > [ =A0113.362344] > [ =A0113.362344] Pid: 2858, comm: pppd Not tainted (2.6.31.4-build-0047 #= 7) > [ =A0113.362344] EIP: 0060:[] EFLAGS: 00010286 CPU: 0 > [ =A0113.362344] EIP is at pppoe_device_event+0x80/0x12c [pppoe] > [ =A0113.362344] EAX: f4fbe000 EBX: ffffffff ECX: f6cea5a0 EDX: f7403680 > [ =A0113.362344] ESI: 0000000f EDI: f6cea5e0 EBP: f4145e34 ESP: f4145e1c > [ =A0113.362344] =A0DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 > [ =A0113.362344] Process pppd (pid: 2858, ti=F4145000 task=F4112ff0 > task.ti=F4145000) > [ =A0113.362344] Stack: > [ =A0113.362344] =A0f4fbe220 > f4fbe000 > f6cea5a0 > f886a430 > fffffff5 > 00000000 > f4145e54 > c01422b3 > Oct 18 23:59:40 194.146.153.93 > [ =A0113.362344] <0> > f4fbe000 > 00000009 > f8a457d8 > f4fbe000 > f8850190 > 00001091 > f4145e64 > c0142361 > Oct 18 23:59:40 194.146.153.93 > [ =A0113.362344] <0> > ffffffff > 00000000 > f4145e74 > c029ffbf > f4fbe000 > 000010d0 > f4145e90 > c029fa70 > Oct 18 23:59:40 194.146.153.93 > [ =A0113.362344] Call Trace: > [ =A0113.362344] =A0[] ? notifier_call_chain+0x2b/0x4a > [ =A0113.362344] =A0[] ? raw_notifier_call_chain+0xc/0xe > [ =A0113.362344] =A0[] ? dev_close+0x4c/0x8c > [ =A0113.362344] =A0[] ? dev_change_flags+0xa5/0x158 > [ =A0113.362344] =A0[] ? devinet_ioctl+0x21a/0x503 > [ =A0113.362344] =A0[] ? inet_ioctl+0x8d/0xa6 > [ =A0113.362344] =A0[] ? sock_ioctl+0x1c8/0x1ec > [ =A0113.362344] =A0[] ? sock_ioctl+0x0/0x1ec > [ =A0113.362344] =A0[] ? vfs_ioctl+0x22/0x69 > [ =A0113.362344] =A0[] ? do_vfs_ioctl+0x41f/0x459 > [ =A0113.362344] =A0[] ? sys_send+0x18/0x1a > [ =A0113.362344] =A0[] ? do_page_fault+0x242/0x26f > [ =A0113.362344] =A0[] ? sys_ioctl+0x2c/0x45 > [ =A0113.362344] =A0[] ? syscall_call+0x7/0xb > [ =A0113.362344] Code: > c9 > 00 > 00 > 00 > 89 > c7 > 31 > f6 > 83 > c7 > 40 > 89 > f8 > e8 > cc > 60 > a9 > c7 > 8b > 45 > ec > 05 > 20 > 02 > 00 > 00 > 89 > 45 > e8 > 8b > 4d > f0 > 8b > 1c > b1 > e9 > 8c > 00 > 00 > 00 > 8b > 45 > ec > Oct 18 23:59:40 194.146.153.93 > 83 > 90 > 01 > 00 > 00 > 74 > 08 > 8b > 9b > 8c > 01 > 00 > 00 > eb > 79 > b8 > c0 > a6 > 86 > f8 > Oct 18 23:59:40 194.146.153.93 > [ =A0113.362344] EIP: [] > pppoe_device_event+0x80/0x12c [pppoe] > SS:ESP 0068:f4145e1c > [ =A0113.362344] CR2: 000000000000018f > [ =A0113.373124] ---[ end trace f6fe64a307e97f3b ]--- > [ =A0113.373203] Kernel panic - not syncing: Fatal exception in interrupt > [ =A0113.373286] Pid: 2858, comm: pppd Tainted: G =A0 =A0 =A0D =A0 =A02.6= .31.4-build-0047 > #7 > [ =A0113.373379] Call Trace: > [ =A0113.373479] =A0[] ? printk+0xf/0x11 > [ =A0113.373561] =A0[] panic+0x39/0xd9 > [ =A0113.373656] =A0[] oops_end+0x8b/0x9a > [ =A0113.373727] =A0[] no_context+0x13d/0x147 > [ =A0113.373800] =A0[] __bad_area_nosemaphore+0x113/0x11b > [ =A0113.373881] =A0[] ? sock_alloc_send_pskb+0x8b/0x24a > [ =A0113.373959] =A0[] ? __wake_up_sync_key+0x3b/0x45 > [ =A0113.374030] =A0[] ? irq_exit+0x39/0x5c > [ =A0113.374107] =A0[] ? do_IRQ+0x80/0x96 > [ =A0113.374183] =A0[] ? common_interrupt+0x29/0x30 > [ =A0113.374259] =A0[] bad_area_nosemaphore+0xd/0x10 > [ =A0113.374348] =A0[] do_page_fault+0x114/0x26f > [ =A0113.374526] =A0[] ? do_page_fault+0x0/0x26f > [ =A0113.374605] =A0[] error_code+0x66/0x6c > [ =A0113.374683] =A0[] ? tcp_v4_send_ack+0xa3/0x10e > [ =A0113.374764] =A0[] ? do_page_fault+0x0/0x26f > [ =A0113.374850] =A0[] ? pppoe_device_event+0x80/0x12c [pppoe] > [ =A0113.374928] =A0[] notifier_call_chain+0x2b/0x4a > [ =A0113.375012] =A0[] raw_notifier_call_chain+0xc/0xe > [ =A0113.375097] =A0[] dev_close+0x4c/0x8c > [ =A0113.375169] =A0[] dev_change_flags+0xa5/0x158 > [ =A0113.375239] =A0[] devinet_ioctl+0x21a/0x503 > [ =A0113.375318] =A0[] inet_ioctl+0x8d/0xa6 > [ =A0113.375411] =A0[] sock_ioctl+0x1c8/0x1ec > [ =A0113.375491] =A0[] ? sock_ioctl+0x0/0x1ec > [ =A0113.375574] =A0[] vfs_ioctl+0x22/0x69 > [ =A0113.375653] =A0[] do_vfs_ioctl+0x41f/0x459 > [ =A0113.375734] =A0[] ? sys_send+0x18/0x1a > [ =A0113.375813] =A0[] ? do_page_fault+0x242/0x26f > [ =A0113.375884] =A0[] sys_ioctl+0x2c/0x45 > [ =A0113.375960] =A0[] syscall_call+0x7/0xb > [ =A0113.376041] Rebooting in 5 seconds..