oops in tcp_xmit_retransmit_queue() w/ v2.6.32.15

Tejun Heo
2010-07-08T08:22:15Z

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

Hello,

We've been seeing oops in tcp_xmit_retransmit_queue() w/ 2.6.32.15.
Please see the attached photoshoot. This is happening on a HPC
cluster and very interestingly caused by one particular job. How long
it takes isn't clear yet (at least more than a day) but when it
happens it happens on a lot of machines in relatively short time.

With a bit of disassemblying, I've found that the oops is happening
during tcp_for_write_queue_from() because the skb->next points to
NULL.

void tcp_xmit_retransmit_queue(struct sock *sk)
{
...
if (tp->retransmit_skb_hint) {
skb = tp->retransmit_skb_hint;
last_lost = TCP_SKB_CB(skb)->end_seq;
if (after(last_lost, tp->retransmit_high))
last_lost = tp->retransmit_high;
} else {
skb = tcp_write_queue_head(sk);
last_lost = tp->snd_una;
}

=> tcp_for_write_queue_from(skb, sk) {
__u8 sacked = TCP_SKB_CB(skb)->sacked;

if (skb == tcp_send_head(sk))
break;
/* we could do better than to assign each time */
if (hole == NULL)

This can happen for one of the following reasons,

1. tp->retransmit_skb_hint is NULL and tcp_write_queue_head() is NULL
too. ie. tcp_xmit_retransmit_queue() is called on an empty write
queue for some reason.

2. tp->retransmit_skb_hint is pointing to a skb which is not on the
write_queue. ie. somebody forgot to update hint while removing the
skb from the write queue.

3. The hint is pointing to a skb on the list but the list itself is
corrupt.

I added some debug code and the crash is happening when
tp->retransmit_skb_hint is not NULL but tp->retransmit_skb_hint->next
is NULL. So, #1 is out; unfortunately, I didn't have debug code in
place to discern between #2 and #3.

Does anything ring a bell? This is a production system and debugging
affects quite a number of people. I can put debug code in to discern
between #2 and #3 but I'm basically shooting in the dark and it would
be great if someone has a better idea.

Thanks.

--
tejun

[-- Attachment #2: oops.jpg --]
[-- Type: image/jpeg, Size: 92585 bytes --]