Re: [REGRESSION] tcp/ipv4: kernel panic because of (possible) division by zero

Yuchung Cheng
2015-12-22T02:11:18Z

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

On Mon, Dec 21, 2015 at 12:25 PM, Oleksandr Natalenko
<oleksandr@natalenko.name> wrote:
> Commit 3759824da87b30ce7a35b4873b62b0ba38905ef5 (tcp: PRR uses CRB mode by
> default and SS mode conditionally) introduced changes to net/ipv4/tcp_input.c
> tcp_cwnd_reduction() that, possibly, cause division by zero, and therefore,
> kernel panic in interrupt handler [1].
>
> Reverting 3759824da87b30ce7a35b4873b62b0ba38905ef5 seems to fix the issue.
>
> I'm able to reproduce the issue on 4.3.0–4.3.3 once per several day
> (occasionally).
>
> What could be done to help in debugging this issue?
Do you have ECN enabled (i.e. sysctl net.ipv4.tcp_ecn > 0)?

If so I suspect an ACK carrying ECE during CA_Loss causes entering CWR
state w/o calling tcp_init_cwnd_reduct() to set tp->prior_cwnd. Can
you try this debug / quick-fix patch and send me the error message if
any?

>
> Regards,
> Oleksandr.
>
> [1] http://i.piccy.info/
> i9/6f5cb187c4ff282d189f78c63f95af43/1450729403/283985/951663/panic.jpg

[-- Attachment #2: 0001-tcp-debug-tcp_cwnd_reduction-div0.patch --]
[-- Type: application/octet-stream, Size: 1350 bytes --]

From 00c80bddae93d712d1e51e021266c0067d2e130e Mon Sep 17 00:00:00 2001
From: Yuchung Cheng <ycheng@google.com>
Date: Mon, 21 Dec 2015 17:37:51 -0800
Subject: [PATCH] tcp: debug tcp_cwnd_reduction div0

It's been reported tcp_cwnd_reduction() may have div0 bug b/c
tp->prior_cwnd == 0. To avoid this, the best option is to simply
use packet conservation to send packets in cwnd reduction states.
Also add some pr_err to debug this.

Signed-off-by: Yuchung Cheng <ycheng@google.com>
---
net/ipv4/tcp_input.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 7b1fddc..c27fa84 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -2478,8 +2478,12 @@ static void tcp_cwnd_reduction(struct sock *sk, const int prior_unsacked,
int newly_acked_sacked = prior_unsacked -
(tp->packets_out - tp->sacked_out);

+ if (WARN_ON(!tp->prior_cwnd))
+ pr_err(" cwr prior_cwnd0 %d cwnd %u %d %d 0x%x\n",
+ inet_csk(sk)->icsk_ca_state, tp->snd_cwnd,
+ tp->prr_out, tp->prr_delivered, flag);
tp->prr_delivered += newly_acked_sacked;
- if (delta < 0) {
+ if (delta < 0 && tp->prior_cwnd) {
u64 dividend = (u64)tp->snd_ssthresh * tp->prr_delivered +
tp->prior_cwnd - 1;
sndcnt = div_u64(dividend, tp->prior_cwnd) - tp->prr_out;
--
2.6.0.rc2.230.g3dd15c0