Re: [Bugme-new] [Bug 27742] New: PPP over SSH tunnel triggers OOPS

Re: [Bugme-new] [Bug 27742] New: PPP over SSH tunnel triggers OOPS

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Fri, 28 Jan 2011 21:58:49 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=27742
> 
>            Summary: PPP over SSH tunnel triggers OOPS
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 2.6.30 ?
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: acme@ghostprotocols.net
>         ReportedBy: ktk@bigfoot.com
>         Regression: Yes
> 
> 
> Created an attachment (id=45412)
>  --> (https://bugzilla.kernel.org/attachment.cgi?id=45412)
> Hand-copied OOPS from 2.6.37 kernel
> 
> When creating a VPN connection by using PPP tunneled over SSH, the kernel will
> OOPS when certain traffic patterns are encountered.  (See attached OOPS)
> 
> I first created such a VPN connection using kernel 2.6.33, which is affected. 
> Kernel 2.6.27 is not affected.  I have not attempted to binary-search for the
> exact commit, but am merely guessing it is in kernel 2.6.30 (as a variety of
> ppp-related commits appear in the changelog there).
> 
> The VPN tunnel is established by invoking 'pppd' with the 'pty' parameter set
> to invoke "ssh remotehost.com pppd" which establishes a SSH tunnel over IPv4 to
> the remote host and then invokes the remote pppd to handle the other end of the
> point-to-point VPN.
> 
> Reproducing this bug is not easy.  With the ppp-ssh-ppp tunnel open, I have
> tried triggering the OOPs by sending PINGs, rsync-ing files in both directions,
> opening interactive SSH connections.  Nothing seems to trigger the OOPS except
> one: running Mozilla Thunderbird on the remote end; it opens several IMAP
> connections over the tunnel simultaneously.  Typically, the OOPS will occur
> within 1 or 2 seconds of invoking Thunderbird.
> 
> When the OOPS occurs, usually the console will be scrolling wildly with OOPS
> after OOPS, making copying impossible.  It has taken me two months of repeated
> tries to get one OOPS that remained on-screen and could be copied.  The kernel
> is in a hard-run state when the OOPS occurs; nothing gets logged to syslog, the
> keyboard is unresponsive (magic sysrq key does nothing).
> 

> skb_over_panic: text:c12a354f len:847 put:847 head:f57e8c00 data:f57e8c00 tail:0xf57e8f4f end:0xf57e8e80 dev:<NULL>
> kernel BUG at net/core/skbuff.c:127!
> invalid opcode: 0000 [#1] SMP
> last sysfs file: /sys/devices/virtual/net/ppp0/flags
> Modules linked in:
> 
> Pid: 0, comm: swapper Not tainted 2.6.37 #1 0KH290/OptiPlex GX620
> EIP: 0060:[<c1330110>] EFLAGS: 00010282 CPU: 0
> EIP is at skb_put+0x82/0x84
> EAX: 00000089 EBX: f57e8f4f ECX: c151579c EDX: 00000046
> ESI: 00000000 EDI: c1530760 EBP: f67bb384 ESP: f6409d50
>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> Process swapper (pid: 0, ti=f6408000 task=c15114a0 task.ti=c1502000)
> Stack:
>  c14d4590 c12a354f 0000034f 0000034f f57e8c00 f57e8c00 f57e8f4f f57e8e80
>  c14d2509 f67bb380 f454db80 c12a354f 000005e0 00000244 f5f944c2 0000034f
>  c1390a0a f67bb3d4 f4648380 f67bb394 f67bb3a4 00000202 f454db80 f67bb000
> Call Trace:
>  [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
>  [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
>  [<c1390a0a>] ? tcp_manip_pkt+0xad/0xcb
>  [<c12a36d4>] ? ppp_start_xmit+0xf9/0x175
>  [<c133a496>] ? dev_hard_start_xmit+0x2a4/0x5c3
>  [<c1347cad>] ? sch_direct_xmit+0xb9/0x184
>  [<c134c663>] ? nf_iterate+0x52/0x76
>  [<c1362d56>] ? ip_finish_output+0x0/0x294
>  [<c133a88e>] ? dev_queue_xmit+0xd9/0x3b0
>  [<c1362d56>] ? ip_finish_output+0x0/0x294
>  [<c1362f32>] ? ip_finish_output+0x1dc/0x294
>  [<c1362d56>] ? ip_finish_output+0x0/0x294
>  [<c1360d66>] ? ip_forward_finish+0x36/0x42
>  [<c135f8a4>] ? ip_rcv_finish+0x42/0x323
>  [<c13384ac>] ? __netif_receive_skb+0x225/0x299
>  [<c1048b56>] ? getnstimeofday+0x42/0xe8
>  [<c13386ab>] ? netif_receive_skb+0x41/0x64
>  [<c1339356>] ? dev_gro_receive+0x146/0x1dd
>  [<c133955e>] ? napi_gro_receive+0xa5/0xb3
>  [<c129ae2f>] ? tg3_poll_wor+0x5df/0xaca
>  [<c1007340>] ? nommu_sync_single_for_device+0x0/0x1
>  [<c129b3e2>] ? tg3_poll+0x43/0x19a
>  [<c133893b>] ? net_rx_action+0x6c/0xf4
>  [<c1031eb5>] ? __do_softirq+0x77/0xf0
>  [<c1031e3e>] ? __do_softirq+0x0/0xf0
>  <IRQ>
>  [<c1031fe6>] ? irq_exit+0x5d/0x5f
> 

Re: [Bugme-new] [Bug 27742] New: PPP over SSH tunnel triggers OOPS

[David Miller]

From: Andrew Morton <akpm@linux-foundation.org>
Date: Fri, 28 Jan 2011 14:32:38 -0800

>> skb_over_panic: text:c12a354f len:847 put:847 head:f57e8c00 data:f57e8c00 tail:0xf57e8f4f end:0xf57e8e80 dev:<NULL>
>> kernel BUG at net/core/skbuff.c:127!
...
>> Pid: 0, comm: swapper Not tainted 2.6.37 #1 0KH290/OptiPlex GX620
>> EIP: 0060:[<c1330110>] EFLAGS: 00010282 CPU: 0
>> EIP is at skb_put+0x82/0x84
...
>> Call Trace:
>>  [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
>>  [<c12a354f>] ? ppp_xmit_process+0x45a/0x4e6
>>  [<c1390a0a>] ? tcp_manip_pkt+0xad/0xcb
>>  [<c12a36d4>] ? ppp_start_xmit+0xf9/0x175

I took a quick look at this, I can surmise that we have a packet we
are trying to compress (that's the only way I see in the
ppp_xmit_process() code paths that we can get an skb_put() call so
large).

And we can see from the skb_over_panic message that we have an SKB
which was allocated with 640 bytes of space, but we are trying to
"put" 847 bytes into it which is too large and overflows.

Can you run with the following debugging patch and see what it prints
out when this happens?

diff --git a/drivers/net/ppp_generic.c b/drivers/net/ppp_generic.c
index 9f6d670..06c6ea7 100644
--- a/drivers/net/ppp_generic.c
+++ b/drivers/net/ppp_generic.c
@@ -1093,6 +1093,15 @@ pad_compress_skb(struct ppp *ppp, struct sk_buff *skb)
 	if (len > 0 && (ppp->flags & SC_CCP_UP)) {
 		kfree_skb(skb);
 		skb = new_skb;
+#if 1
+		if (len > (skb->end - skb->tail)) {
+			printk(KERN_ERR "pad_compress_skb: Compression overflow ["
+			       "new_skb_size(%d) compressor_skb_size(%d) "
+			       "hard_header_len(%d) len(%d)]\n",
+			       new_skb_size, compressor_skb_size,
+			       ppp->dev->hard_header_len, len);
+		}
+#endif
 		skb_put(skb, len);
 		skb_pull(skb, 2);	/* pull off A/C bytes */
 	} else if (len == 0) {
@@ -1179,6 +1188,9 @@ ppp_send_frame(struct ppp *ppp, struct sk_buff *skb)
 			/* didn't compress */
 			kfree_skb(new_skb);
 		} else {
+#if 1
+			unsigned int orig_skb_len = skb->len;
+#endif
 			if (cp[0] & SL_TYPE_COMPRESSED_TCP) {
 				proto = PPP_VJC_COMP;
 				cp[0] &= ~SL_TYPE_COMPRESSED_TCP;
@@ -1188,6 +1200,13 @@ ppp_send_frame(struct ppp *ppp, struct sk_buff *skb)
 			}
 			kfree_skb(skb);
 			skb = new_skb;
+#if 1
+			if (len > (skb->end - skb->tail)) {
+				printk(KERN_ERR "slhc_compress_skb: Compression overflow ["
+				       "skb->len(%u) hard_header_len(%d) len(%d)]\n",
+				       orig_skb_len, ppp->dev->hard_header_len, len);
+			}
+#endif
 			cp = skb_put(skb, len + 2);
 			cp[0] = 0;
 			cp[1] = proto;