raw ipv6 broken in 2.4.19

raw ipv6 broken in 2.4.19

[Martin Josefsson]

Hi Alexey,

raw ipv6 doesn't work in 2.4.19, I've traced it to part of a patch that
went in between -pre7 and -pre8 back in April 23. Reverting the patch
below makes it work again in two machines. (part of changeset 1.383.17.3
in marcelos BK tree)

The description of that part in the changeset is:

"IPv6 raw had missing sk->filter handling and rawv6_rcv missing some
checksum processing."

The symptoms were that ping6 didn't work, it complained about:

ping: recvmsg: No route to host

but icmp echo-requests were sent out and icmp echo-replies were
recieved.
Ip6InDiscards increased for each icmp echo-reply recieved, but
Ip6InDelivers also increased for each packet recieved.

And traceroute6 returned bogus addresses most of the time (it was either
the correct address or a bogus one but always the same bogus address
independent of which ip the response came from)

Tested against Linux and OpenBSD with the same results.


--- 1.8/net/ipv6/raw.c	Thu Mar 14 00:46:57 2002
+++ 1.9/net/ipv6/raw.c	Tue Apr 23 04:13:30 2002
@@ -278,6 +278,16 @@
 
 static inline int rawv6_rcv_skb(struct sock * sk, struct sk_buff * skb)
 {
+#if defined(CONFIG_FILTER)
+	if (sk->filter && skb->ip_summed != CHECKSUM_UNNECESSARY) {
+		if ((unsigned short)csum_fold(skb_checksum(skb, 0, skb->len, skb->csum))) {
+			IP6_INC_STATS_BH(Ip6InDiscards);
+			kfree_skb(skb);
+			return 0;
+		}
+		skb->ip_summed = CHECKSUM_UNNECESSARY;
+	}
+#endif
 	/* Charge it to the socket. */
 	if (sock_queue_rcv_skb(sk,skb)<0) {
 		IP6_INC_STATS_BH(Ip6InDiscards);
@@ -298,9 +308,33 @@
  */
 int rawv6_rcv(struct sock *sk, struct sk_buff *skb)
 {
+	if (!sk->tp_pinfo.tp_raw.checksum)
+		skb->ip_summed = CHECKSUM_UNNECESSARY;
+
+	if (skb->ip_summed != CHECKSUM_UNNECESSARY) {
+		if (skb->ip_summed == CHECKSUM_HW) {
+			skb->ip_summed = CHECKSUM_UNNECESSARY;
+			if (csum_ipv6_magic(&skb->nh.ipv6h->saddr,
+					    &skb->nh.ipv6h->daddr,
+					    skb->len, sk->num, skb->csum)) {
+				NETDEBUG(if (net_ratelimit()) printk(KERN_DEBUG "raw v6 hw csum failure.\n"));
+				skb->ip_summed = CHECKSUM_NONE;
+			}
+		}
+		if (skb->ip_summed == CHECKSUM_NONE)
+			skb->csum = ~csum_ipv6_magic(&skb->nh.ipv6h->saddr,
+						     &skb->nh.ipv6h->daddr,
+						     skb->len, sk->num, 0);
+	}
+
 	if (sk->protinfo.af_inet.hdrincl) {
-		__skb_push(skb, skb->nh.raw - skb->data);
-		skb->h.raw = skb->nh.raw;
+		if (skb->ip_summed != CHECKSUM_UNNECESSARY &&
+		    (unsigned short)csum_fold(skb_checksum(skb, 0, skb->len, skb->csum))) {
+			IP6_INC_STATS_BH(Ip6InDiscards);
+			kfree_skb(skb);
+			return 0;
+		}
+		skb->ip_summed = CHECKSUM_UNNECESSARY;
 	}
 
 	rawv6_rcv_skb(sk, skb);
@@ -339,7 +373,17 @@
   		msg->msg_flags |= MSG_TRUNC;
   	}
 
-	err = skb_copy_datagram_iovec(skb, 0, msg->msg_iov, copied);
+	if (skb->ip_summed==CHECKSUM_UNNECESSARY) {
+		err = skb_copy_datagram_iovec(skb, 0, msg->msg_iov, copied);
+	} else if (msg->msg_flags&MSG_TRUNC) {
+		if ((unsigned short)csum_fold(skb_checksum(skb, 0, skb->len, skb->csum)))
+			goto csum_copy_err;
+		err = skb_copy_datagram_iovec(skb, 0, msg->msg_iov, copied);
+	} else {
+		err = skb_copy_and_csum_datagram_iovec(skb, 0, msg->msg_iov);
+		if (err == -EINVAL)
+			goto csum_copy_err;
+	}
 	if (err)
 		goto out_free;
 
@@ -366,6 +410,27 @@
 	skb_free_datagram(sk, skb);
 out:
 	return err;
+
+csum_copy_err:
+	/* Clear queue. */
+	if (flags&MSG_PEEK) {
+		int clear = 0;
+		spin_lock_irq(&sk->receive_queue.lock);
+		if (skb == skb_peek(&sk->receive_queue)) {
+			__skb_unlink(skb, &sk->receive_queue);
+			clear = 1;
+		}
+		spin_unlock_irq(&sk->receive_queue.lock);
+		if (clear)
+			kfree_skb(skb);
+	}
+
+	/* Error for blocking case is chosen to masquerade
+	   as some normal condition.
+	 */
+	err = (flags&MSG_DONTWAIT) ? -EAGAIN : -EHOSTUNREACH;
+	IP6_INC_STATS_USER(Ip6InDiscards);
+	goto out_free;
 }
 
 /*


-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> raw ipv6 doesn't work in 2.4.19,

Seems, more information is required because of:

root@mops:~ # cat /proc/version
Linux version 2.4.20-pre1-mops (root@mops) (gcc version 3.0.2 20010905 (ASPLinux 7.2 3.0.1-3)) #3370 Втр Авг 6 18:18:59 MSD 2002
root@mops:~ # traceroute6 sex
traceroute to sex.inr.ac.ru (3ffe:2400:0:5:202:b3ff:fe90:d922) from 3ffe:2400:0:4:240:d0ff:fe16:a83d, 30 hops max, 16 byte packets
 1  dust.inr.ac.ru (3ffe:2400:0:4:2c0:95ff:fee7:f562)  0.412 ms  0.195 ms  0.143 ms
 2  sex.inr.ac.ru (3ffe:2400:0:5:202:b3ff:fe90:d922)  0.216 ms  0.199 ms  0.207 ms
root@mops:~ # ping6 sex
PING sex(sex.inr.ac.ru) 56 data bytes
64 bytes from sex.inr.ac.ru: icmp_seq=1 ttl=63 time=0.218 ms
64 bytes from sex.inr.ac.ru: icmp_seq=2 ttl=63 time=0.229 ms
64 bytes from sex.inr.ac.ru: icmp_seq=3 ttl=63 time=0.206 ms
64 bytes from sex.inr.ac.ru: icmp_seq=4 ttl=63 time=0.223 ms

--- sex ping statistics ---
4 packets transmitted, 4 received, 0% loss, time 3030ms
rtt min/avg/max/mdev = 0.206/0.219/0.229/0.008 ms
root@mops:~ #

So, please, make binary tcpdumps and tell me version of iputils,
which you use and make straces of ping.

Alexey

Re: raw ipv6 broken in 2.4.19

[Martin Josefsson]

On Mon, 2002-08-12 at 21:20, kuznet@ms2.inr.ac.ru wrote: 
> Hello!
> 
> > raw ipv6 doesn't work in 2.4.19,
> 
> Seems, more information is required because of:
> So, please, make binary tcpdumps and tell me version of iputils,
> which you use and make straces of ping.

I think I've narrowed it down to what's causing the problem now.
I didn't think far enough before.

It seems to be the tulip driver, if I use it things stop working but if
I use an old ISA ne2k clone it works fine.
The NIC is a D-Link DFE-570TX (quad tulip with "Digital DS21143 Tulip
rev 65" for those who don't know)

Both the tulip-driver in vanilla 2.4.19 (without NAPI patch) and the
tulip-NAPI-011103 driver causes this.

Sorry for pointing fingers at the ipv6 code, it was the most logical
solution for my brain (and the fact that I found something to back out
that made it work again :)

I think the strace and tcpdumps are uninteressing now so I'm not
attaching them, the tcpdump looked perfectly normal to me.
But here a small piece of the strace:

sendto(3, "\200\0\0\0\304\2\3\0Z\16X=\331%\17\0\10\t\n\v\f\r\16\17"..., 64, 0, {sin_family=AF_INET6, sin6_port=htons(58), inet_pton(AF_INET6, "3ffe:200:3d:1:2d0:b7ff:fe3f:b7b", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 64
recvmsg(3, 0xbfffea80, 0)               = -1 EHOSTUNREACH (No route to host)
recvmsg(3, 0xbfffe800, MSG_ERRQUEUE|MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily unavailable)
write(2, "ping: recvmsg: No route to host\n", 32ping: recvmsg: No route to host
) = 32

I'm going to take a look at the driver now but I don't really know what
to look for.

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> It seems to be the tulip driver,

Apparently, it corrupts IPv6 packets. I have seen this bug,
it was cured by tuning pci settings in bios setup to be more conservative.
The most intersting thing was that only ICMP IPv6 were corrupted. :-)

> attaching them, the tcpdump looked perfectly normal to me.

I do not think so, packets should have wrong checksum.

Alexey

Re: raw ipv6 broken in 2.4.19

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 1052 bytes --]

On Tue, 2002-08-13 at 06:04, kuznet@ms2.inr.ac.ru wrote:
> Hello!
> 
> > It seems to be the tulip driver,
> 
> Apparently, it corrupts IPv6 packets. I have seen this bug,
> it was cured by tuning pci settings in bios setup to be more conservative.
> The most intersting thing was that only ICMP IPv6 were corrupted. :-)

Eek!

I have this problem in two machines, one SMP and one UP, both with
i440BX chipset. I've tried setting the BIOS settings as conservative as
possible in one machine but it didn't help. Will try the other one a
little later.

I've heard something about a PCI posting bug in the tulip hardware?
could this have anything to do with that?

> > attaching them, the tcpdump looked perfectly normal to me.
> 
> I do not think so, packets should have wrong checksum.

I've attached a small dump. captured while trying to ping an OpenBSD
machine.

I also found that ping6 -s 37 and lower works fine but -s 38 and higher
doesn't.

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

[-- Attachment #2: dumpfile --]
[-- Type: application/octet-stream, Size: 2836 bytes --]

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> I've heard something about a PCI posting bug in the tulip hardware?
> could this have anything to do with that?

No ideas. Eventually I moved that card to an ancient Neptunand it works
flawlessly there.


> I've attached a small dump. captured while trying to ping an OpenBSD

It looks boringly correct, indeed. Was it really taken at faulting machine?

Alexey

Re: raw ipv6 broken in 2.4.19

[Martin Josefsson]

On Tue, 2002-08-13 at 15:30, kuznet@ms2.inr.ac.ru wrote:
> Hello!
> 
> > I've heard something about a PCI posting bug in the tulip hardware?
> > could this have anything to do with that?
> 
> No ideas. Eventually I moved that card to an ancient Neptunand it works
> flawlessly there.

I'm going to test it with a shitty i820 chipset later. Asus P3C-D and
the D-Link DFE570-TX is a really bad combination (thats why I don't use
that board anymore :), having ~90% cpu idle while maxing at ~75Mbit/s
routed is not fun. One NIC active at once is ok, but two or more just
destroys performance. But I've never had that problem with 440BX
chipsets.
 
> > I've attached a small dump. captured while trying to ping an OpenBSD
> 
> It looks boringly correct, indeed. Was it really taken at faulting machine?

Yes it was.

If the packets recieved is really corrupted wouldn't something have
complained before that patch was merged in 2.4.19-pre8 ?
if the ipv6 header was corrupt it would have been dropped?
if the icmp packet was corrupt, wouldn't ping6 have complained?

But then on the other side, it works fine with other NIC's...
And the machine answers icmp echo-requests without any problems with the
tulip. Nothing's beeing discarded then.

I think I'll just go with reverting that patch for now.

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> If the packets recieved is really corrupted wouldn't something have
> complained before that patch was merged in 2.4.19-pre8 ?

No. We simply forgot to verify checksum on raw packets. :-)


> if the ipv6 header was corrupt it would have been dropped?

No, ipv6 has no checksum on header. Corruptions there will remain
unnoticed.


> if the icmp packet was corrupt, wouldn't ping6 have complained?

Provided wrong bits are in data area, it would dump wrong bits
comparing them to original. But if it is in timestamp or in header,
it also would remain unnoticed before 2.4.19.

Very strange. We have similar phenomenon reported with TCP, by the way.
So, I have to assume that checksumming routine is wrong and does some shit
sort of relying on an uninitialized data.

Alexey

Re: raw ipv6 broken in 2.4.19

[Martin Josefsson]

On Tue, 2002-08-13 at 15:58, kuznet@ms2.inr.ac.ru wrote: 

> Very strange. We have similar phenomenon reported with TCP, by the way.
> So, I have to assume that checksumming routine is wrong and does some shit
> sort of relying on an uninitialized data.

I've added some debug printk's and found out that it's the call to
csum_fold that fails in skb_copy_and_csum_datagram_iovec.

skb_copy_and_csum_datagram_iovec is called from:

net/ipv6/raw.c:rawv6_recvmsg()
	if (skb->ip_summed==CHECKSUM_UNNECESSARY) {
		err = skb_copy_datagram_iovec(skb, 0, msg->msg_iov, copied);
	} else if (msg->msg_flags&MSG_TRUNC) {
		if ((unsigned short)csum_fold(skb_checksum(skb, 0, skb->len, skb->csum)))
			goto csum_copy_err;
		err = skb_copy_datagram_iovec(skb, 0, msg->msg_iov, copied);
	} else {
		err = skb_copy_and_csum_datagram_iovec(skb, 0, msg->msg_iov);
		      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
		if (err == -EINVAL)
			goto csum_copy_err;
	}

This obviosly works with my old ISA ne2k clone but not with the tulip.
And the tcpdump looked ok

Does anyone else have an idea or suggestion I can try?

Alexey, when you had checksum problems, did you see invalid checksums in
tcpdump?

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> Alexey, when you had checksum problems, did you see invalid checksums in
> tcpdump?

No. It is dubiously similar to your case. It is the sibj
"Linux TCP problem while talking to hostme.bkbits.net" in linux-kernel.
It is mistique not less than your case. :-)

> Does anyone else have an idea or suggestion I can try?

Well, let's look what exactly checksum routines calculate on the steps.
Maybe, this will give a clue.

Actually, tulip is really different of another cards, this creature
generates badly aligned packets. F.e. it is possible a checksumming routine
folds some bytes beyond end of frame. :-) I have lots of verious tulips here,
but have never seen such a shit. :-)

Alexey

Re: raw ipv6 broken in 2.4.19

[Martin Josefsson]

On Tue, 2002-08-13 at 22:06, kuznet@ms2.inr.ac.ru wrote:
> Hello!
> 
> > Alexey, when you had checksum problems, did you see invalid checksums in
> > tcpdump?
> 
> No. It is dubiously similar to your case. It is the sibj
> "Linux TCP problem while talking to hostme.bkbits.net" in linux-kernel.
> It is mistique not less than your case. :-)
> 
> > Does anyone else have an idea or suggestion I can try?
> 
> Well, let's look what exactly checksum routines calculate on the steps.
> Maybe, this will give a clue.

Just tell me what you want me to dig out from the checksumming stuff.

> Actually, tulip is really different of another cards, this creature
> generates badly aligned packets. F.e. it is possible a checksumming routine
> folds some bytes beyond end of frame. :-) I have lots of verious tulips here,
> but have never seen such a shit. :-)

hehe, this isn't a clone, it's a quad with "Digital DS21143 Tulip rev
65" chips (from tulip driver) behind a "PCI bridge: Digital Equipment
Corporation DECchip 21152" (from lspci)

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> 	What about the known problem with csum_partial called
> with zero length.

Maybe it is known to you, but not to me.


>     (addr&2!=0 => bug)

Sorry? What do you mean? Checksumming routines must not have any alignment
constraints, they used with arbitrary combinations of alignments.

Alexey

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> Just tell me what you want me to dig out from the checksumming stuff.

First of all, value returned by routine. Apparently, it is wrong.

Then arguments (pointers, lengths).

Alexey

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> > Sorry? What do you mean? Checksumming routines must not have any alignment
> > constraints, they used with arbitrary combinations of alignments.
> 
> 	Do you believe, they have :) See the 2nd URL,

Well, if this is only in combination with zero length, it is not a disaster.

Alexey

Re: raw ipv6 broken in 2.4.19

[Martin Josefsson]

On Wed, 2002-08-14 at 03:38, Julian Anastasov wrote:
> 
> 	Hello,
> 
> On Wed, 14 Aug 2002 kuznet@ms2.inr.ac.ru wrote:
> 
> > Well, if this is only in combination with zero length, it is not a disaster.
> 
> 	May be. It seems there were no such callers but they
> appeared. I'm not sure a comment can avoid further problems.
> As for the fix, we don't see often lengths of 0 and 1 for
> unaligned addrs, it is in the slow path in csum_partial. IMHO,
> it will not hurt if we apply the fix instead of auditing everything.

I applied the patch and now it's working fine, both ping6 and
traceroute6 seems to be working fine.

I'll leave it to you guys to debate this :)

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: raw ipv6 broken in 2.4.19

[Julian Anastasov]

	Hello,

On 13 Aug 2002, Martin Josefsson wrote:

> On Tue, 2002-08-13 at 15:58, kuznet@ms2.inr.ac.ru wrote:
>
> > Very strange. We have similar phenomenon reported with TCP, by the way.
> > So, I have to assume that checksumming routine is wrong and does some shit
> > sort of relying on an uninitialized data.
>
> I've added some debug printk's and found out that it's the call to
> csum_fold that fails in skb_copy_and_csum_datagram_iovec.

	What about the known problem with csum_partial called
with zero length. IIRC, on CONFIG_X86_USE_PPRO_CHECKSUM compilation
this function depends on the data alignment (addr&2!=0 => bug) - calling 
csum_partial with zero length in 2.4+ is a bug, checks should be added
in the caller. Can this be problem with skb_copy_and_csum_datagram_iovec? 
I see that net/ipv6/raw.c provides 0 as hlen.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: raw ipv6 broken in 2.4.19

[Julian Anastasov]

	Hello,

On Wed, 14 Aug 2002 kuznet@ms2.inr.ac.ru wrote:

> > 	What about the known problem with csum_partial called
> > with zero length.
>
> Maybe it is known to you, but not to me.

	Hm, some links:

http://marc.theaimsgroup.com/?l=linux-kernel&m=99489362903633&w=2
http://marc.theaimsgroup.com/?l=linux-virtual-server&m=100332800916641&w=2

> >     (addr&2!=0 => bug)
>
> Sorry? What do you mean? Checksumming routines must not have any alignment
> constraints, they used with arbitrary combinations of alignments.

	Do you believe, they have :) See the 2nd URL, there is
a fix tested from Wensong (old LVS problem).

> Alexey

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: raw ipv6 broken in 2.4.19

[Julian Anastasov]

	Hello,

On Wed, 14 Aug 2002 kuznet@ms2.inr.ac.ru wrote:

> Well, if this is only in combination with zero length, it is not a disaster.

	May be. It seems there were no such callers but they
appeared. I'm not sure a comment can avoid further problems.
As for the fix, we don't see often lengths of 0 and 1 for
unaligned addrs, it is in the slow path in csum_partial. IMHO,
it will not hurt if we apply the fix instead of auditing everything.

> Alexey

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: raw ipv6 broken in 2.4.19

[Donald Becker]

On Wed, 14 Aug 2002 kuznet@ms2.inr.ac.ru wrote:

> Actually, tulip is really different of another cards, this creature
> generates badly aligned packets. F.e. it is possible a checksumming routine
> folds some bytes beyond end of frame. :-) I have lots of verious tulips here,
> but have never seen such a shit. :-)

I'm not certain what you mean by this: there are many NIC chips that have
alignment requirements similar to the Tulip design.

Note that the Tulip is no more likely than other cards to trigger
motherboard PCI implementation bugs.  It's just that there are far more
NIC chip types supported by the Tulip driver than other drivers.

-- 
Donald Becker				becker@scyld.com
Scyld Computing Corporation		http://www.scyld.com
410 Severn Ave. Suite 210		Second Generation Beowulf Clusters
Annapolis MD 21403			410-990-9993

Re: raw ipv6 broken in 2.4.19

[kuznet]

Hello!

> I'm not certain what you mean by this: there are many NIC chips that have
> alignment requirements similar to the Tulip design.

Comparing to _his_ ones.


> Note that the Tulip is no more likely than other cards to trigger
> motherboard PCI implementation bugs.

Donald, it was brain attack. When speaking about alignment we have
already switched to investigation of our checksumming routines.

Alexey