IPv6 forwarding to TAP-interface with ip6tables fails

IPv6 forwarding to TAP-interface with ip6tables fails

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 4451 bytes --]

Hello, list.


I wonder if the following problem is a bug in netfilter or I'm just
doing something wrong, but it wasn't always like that.
Problem started probably with some kernel update on router machine,
since I can't trace any network layout updates since the point where it
still worked, and complete downgrade is quite a bad solution.


What I'm trying to do is to connect with ssh and/or ping6 from
2001:470:1f0b:11de::13 to 2001:470:1f0b:11de::21.
And it works at first (and worked fine before), but stops after some
time, which doesn't seem constant or relative to amount of data being
forwarded.

"tcpdump -i lan" shows a lot of tcp retransmission packets or icmp6 (in
case of ping6), but "tcpdump -i vde" shows none of them (while showing
all of them while the connection is still working).

I've tried setting -j LOG to FORWARD chain of ip6tables and it always
(both while the connection works and not) shows lines like this:
  Dec 19 14:09:48 kern.warn<4> kernel[-]@damnation: IN=lan OUT=vde
    SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
    DST=2001:0470:1f0b:11de:0000:0000:0000:0021
    LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6
    TYPE=128 CODE=0 ID=29524 SEQ=8 

So, I assume the packet is lost somewhere in netfilter, but I've no
idea how to tell why it's getting discarded like that.

Strangely enough (for me), when I type "ping6 2001:0470:1f0b:11de::21"
on router machine it gives "Destination unreachable: Address
unreachable" for the first packet and then it works for the rest of
them from both router and remote machine in "lan" network (whole
forwarding thing starts working).
When I stop this ping, situation repeats after some time.

IPv4 forwarding works fine over the same link, but it's NAT there.


Note that aforementioned "vde" is a tun/tap interface of Virtual
Distributed Ethernet link, but tcpdump should show the packet on tap
interface (vde) regardless of other software, monitoring it, so it's
not a VDE-related bug, right?
That's the first question that's bothering me.

It worked a few months ago but I haven't used it actively since then,
and now it doesn't. Same IPs, but different software versions (see below).

Is something wrong with the setup?
Are there any settings that can induce aforementioned behavior I should
check, apart from the ones, included in this mail?

And if not...
Is there any way to debug the problem futher, apart of inserting
(enabling) debug code in the kernel?
Does it look like a bug in netfilter forwarding, should I post it to
bugzilla?


Thanks in advance for any answers / suggestions.


ip addr:
  2: lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
      link/ether 00:22:15:3a:ef:7d brd ff:ff:ff:ff:ff:ff
      inet 192.168.0.10/29 scope global lan
      inet6 2001:470:1f0b:11de::10/125 scope global 
         valid_lft forever preferred_lft forever
      inet6 2001:470:1f0b:11de::1/125 scope global 
         valid_lft forever preferred_lft forever
      inet6 fe80::222:15ff:fe3a:ef7d/64 scope link 
         valid_lft forever preferred_lft forever
  7: vde: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
      link/ether b2:f5:e7:68:fd:9e brd ff:ff:ff:ff:ff:ff
      inet 192.168.0.20/28 scope global vde
      inet6 2001:470:1f0b:11de::20/125 scope global 
         valid_lft forever preferred_lft forever

ip -6 route:
  2001:470:1f0b:11de::/125 dev lan  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
  2001:470:1f0b:11de::10/125 dev lan  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
  2001:470:1f0b:11de::20/125 dev vde  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
  fe80::/64 dev lan  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
  fe80::/64 dev vde  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295

ip6tables has following rules (at the start of FORWARD chain):
  -I FORWARD -i vde -j ACCEPT
  -I FORWARD -i lan -j ACCEPT

Software:
  Linux damnation 2.6.32-gentoo-fg.mf_master #1 SMP Sun Dec 13 22:35:05
    YEKT 2009 x86_64 Intel(R) Xeon(R) CPU L5420 @ 2.50GHz GenuineIntel
    GNU/Linux (happened with .31 as well, will try official .32.2 today)
  iptables-1.4.5
  vde-2.2.3 (shouldn't be relevant?)

IPv6 forwarding is enabled in sysctl.


-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: IPv6 forwarding to TAP-interface with ip6tables fails

[Pascal Hambourg]

Hello,

Mike Kazantsev a écrit :
> 
> I wonder if the following problem is a bug in netfilter or I'm just
> doing something wrong, but it wasn't always like that.
> Problem started probably with some kernel update on router machine,
[...]
> I've tried setting -j LOG to FORWARD chain of ip6tables and it always
> (both while the connection works and not) shows lines like this:
>   Dec 19 14:09:48 kern.warn<4> kernel[-]@damnation: IN=lan OUT=vde
>     SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
>     DST=2001:0470:1f0b:11de:0000:0000:0000:0021
>     LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6
>     TYPE=128 CODE=0 ID=29524 SEQ=8 
> 
> So, I assume the packet is lost somewhere in netfilter, but I've no
> idea how to tell why it's getting discarded like that.

Have you tried to track the packet in the mangle/POSTROUTING chain too,
either with the LOG target in that chain or with the TRACE target in the
raw/PREROUTING chain ?

> Strangely enough (for me), when I type "ping6 2001:0470:1f0b:11de::21"
> on router machine it gives "Destination unreachable: Address
> unreachable"

This indicates that the neighbour discovery for that address failed,
either because the ICMPv6 neighbour solicitation request could not be
sent or the ICMPv6 neighbour advertisement reply was not received. If
neighbour discovery fails, packets cannot be sent or forwarded to that
destination.

> ip addr:
>   2: lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
[...]
>       inet6 2001:470:1f0b:11de::10/125 scope global 
[...]
>   7: vde: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
[...]
>       inet6 2001:470:1f0b:11de::20/125 scope global 

Regardless of your problem, 2001:470:1f0b:11de::10 and
2001:470:1f0b:11de::20 are the prefix addresses of
2001:470:1f0b:11de::10/25 and 2001:470:1f0b:11de::20/25 and have a
special status : they are anycast addresses meaning "any router
connected to that prefix". So I guess that they should not be assigned
to an interface.

> ip6tables has following rules (at the start of FORWARD chain):
>   -I FORWARD -i vde -j ACCEPT
>   -I FORWARD -i lan -j ACCEPT

ICMPv6 packets involved in neighbour discovery go through the INPUT and
OUTPUT chains. What do these chains contain ? Connection tracking of
these ICMPv6 types changed in kernel 2.6.29.

You wrote in the subject "with ip6tables". Do you mean that it works
without ip6tables, without ip6_tables loaded or without ip6table_filter
loaded, or with all chains empty and with ACCEPT default policy ?

Re: IPv6 forwarding to TAP-interface fails

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 7437 bytes --]

On Sat, 19 Dec 2009 15:09:08 +0100
Pascal Hambourg <pascal.mail@plouf.fr.eu.org> wrote:

> > So, I assume the packet is lost somewhere in netfilter, but I've no
> > idea how to tell why it's getting discarded like that.
> 
> Have you tried to track the packet in the mangle/POSTROUTING chain
> too, either with the LOG target in that chain or with the TRACE
> target in the raw/PREROUTING chain ?

Didn't know about it, very handy way to check what's happening.
Thanks. Pity I've missed it before.

As for the results - it seems that everything is passing through
netfilter just fine, more about it below.


> > Strangely enough (for me), when I type "ping6
> > 2001:0470:1f0b:11de::21" on router machine it gives "Destination
> > unreachable: Address unreachable"
> 
> This indicates that the neighbour discovery for that address failed,
> either because the ICMPv6 neighbour solicitation request could not be
> sent or the ICMPv6 neighbour advertisement reply was not received. If
> neighbour discovery fails, packets cannot be sent or forwarded to that
> destination.

Yes, that seem to be the cause of a problem, but then I don't seem to
understand why router doesn't send these requests.


I've set TRACE to packets with 2001:470:1f0b:11de::22 as a destination
in raw.PREROUTING and -j LOG to INPUT and OUTPUT chains of filter table.

On VM start tcpdump (with "ip6" filter) shows these:
  IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
  IP6 :: > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::22, length 24
  IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28

Then I start the simple ping6 from lan network to vde (VM):
  IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 1, length 64
  
  TRACE: raw:PREROUTING:policy:2 IN=lan OUT= MAC=00:22:15:3a:ef:7d:00:17:c4:10:7f:4f:86:dd
   SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
   DST=2001:0470:1f0b:11de:0000:0000:0000:0022
   LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
  TRACE: mangle:PREROUTING:policy:1 IN=lan OUT= MAC=00:22:15:3a:ef:7d:00:17:c4:10:7f:4f:86:dd
   SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
   DST=2001:0470:1f0b:11de:0000:0000:0000:0022
   LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
  TRACE: mangle:FORWARD:policy:1 IN=lan OUT=vde
   SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
   DST=2001:0470:1f0b:11de:0000:0000:0000:0022
   LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
  TRACE: filter:FORWARD:policy:1 IN=lan OUT=vde
   SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
   DST=2001:0470:1f0b:11de:0000:0000:0000:0022
   LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
  TRACE: mangle:POSTROUTING:policy:1 IN= OUT=vde
   SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
   DST=2001:0470:1f0b:11de:0000:0000:0000:0022
   LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1

  IP6 2001:470:1f0b:11de::22 > ff02::1:ff00:21: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::21, length 32
  
  IN=vde OUT= MAC=33:33:ff:00:00:21:00:16:3e:16:38:41:86:dd
   SRC=2001:0470:1f0b:11de:0000:0000:0000:0022
   DST=ff02:0000:0000:0000:0000:0001:ff00:0021
   LEN=72 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=135 CODE=0
  
  IP6 2001:470:1f0b:11de::21 > 2001:470:1f0b:11de::22: ICMP6, neighbor advertisement, tgt is 2001:470:1f0b:11de::21, length 32
  
  IN= OUT=vde
   SRC=2001:0470:1f0b:11de:0000:0000:0000:0021
   DST=2001:0470:1f0b:11de:0000:0000:0000:0022
   LEN=72 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=136 CODE=0
  
  
  IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 1, length 64

...and so on until tcpdump just stops, while traces show the same thing.


On lan interface, unlike vde, neighbor requests seem to be passed quite
frequently:

  ...
  IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 5, length 64
  IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 5, length 64
  IP6 fe80::222:15ff:fe3a:ef7d > 2001:470:1f0b:11de::13: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::13, length 32
  IP6 2001:470:1f0b:11de::13 > fe80::222:15ff:fe3a:ef7d: ICMP6, neighbor advertisement, tgt is 2001:470:1f0b:11de::13, length 24
  IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 6, length 64
  IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 6, length 64
  ...


And when I type "ping6 2001:470:1f0b:11de::22" on router itself I can
see neighbor solicitation in tcpdump and pings from lan start working
again, although not for long (if I ping on router):

  [complete silence]
  IP6 2001:470:1f0b:11de::21 > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::22, length 32
  IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::21: ICMP6, neighbor advertisement, tgt is 2001:470:1f0b:11de::22, length 32
  IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 135, length 64
  IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 135, length 64
  ...


So I guess that pinpoints the problem as "router doesn't send neighbor
solicitation requests along with forwarded packets", thanks to your
suggestion.

I have to admit that "neighbor solicitation", although basic IPv6
mechanism, is a obscure to me, since I haven't done necessary research
on the topic before setting the network up, and that's quite a stupid
mistake I'm going to fix right now ;)

Guess that gives me something to search for and debug further, so I'll
probably be able to either fix it or come up with another, more
specific, question.
Thanks for your help.


> Regardless of your problem, 2001:470:1f0b:11de::10 and
> 2001:470:1f0b:11de::20 are the prefix addresses of
> 2001:470:1f0b:11de::10/25 and 2001:470:1f0b:11de::20/25 and have a
> special status : they are anycast addresses meaning "any router
> connected to that prefix". So I guess that they should not be assigned
> to an interface.

While that worked before, I agree that it's a bad practice, and I've
fixed it now, although it doesn't seem to affect this particular
problem indeed.

For the above tests I've changed ...::10 to 11, ...::20 to 21
and ...::21 to 22.


> ICMPv6 packets involved in neighbour discovery go through the INPUT
> and OUTPUT chains. What do these chains contain ? Connection tracking
> of these ICMPv6 types changed in kernel 2.6.29.

These (as well as the rest of ICMPv6) were unrestricted in all chains,
but for the sake of this test (dumps/traces posted above) I've set
everything to ACCEPT explicitly.


> You wrote in the subject "with ip6tables". Do you mean that it works
> without ip6tables, without ip6_tables loaded or without
> ip6table_filter loaded, or with all chains empty and with ACCEPT
> default policy ?

My mistake, sorry, ip6tables tool probably have nothing to do with it,
it's just that the whole forwarding concept is closely associated with
iptables in my mind ;)

ACCEPT everywhere doesn't affect the test, and above results acquired
with just that setup.


-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[SOLVED] Re: IPv6 forwarding to TAP-interface fails

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]

On Mon, 21 Dec 2009 11:46:48 +0500
Mike Kazantsev <mk.fraggod@gmail.com> wrote:

> Then I start the simple ping6 from lan network to vde (VM):
...
> 
>   IP6 2001:470:1f0b:11de::22 > ff02::1:ff00:21: ICMP6, neighbor
> solicitation, who has 2001:470:1f0b:11de::21, length 32 
>   IN=vde OUT= MAC=33:33:ff:00:00:21:00:16:3e:16:38:41:86:dd
>    SRC=2001:0470:1f0b:11de:0000:0000:0000:0022
>    DST=ff02:0000:0000:0000:0000:0001:ff00:0021
>    LEN=72 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=135 CODE=0
>   
>   IP6 2001:470:1f0b:11de::21 > 2001:470:1f0b:11de::22: ICMP6,
> neighbor advertisement, tgt is 2001:470:1f0b:11de::21, length 32 
>   IN= OUT=vde
>    SRC=2001:0470:1f0b:11de:0000:0000:0000:0021
>    DST=2001:0470:1f0b:11de:0000:0000:0000:0022
>    LEN=72 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=136 CODE=0
>   

Just looking at these lines as I pasted them actually helped: ICMP6
neighbor solicitation request was sent from 22 to 21, while the packets
were originated from 21.
Made me wondering about what kind of backwards magic is at work here...


> ip addr:
...
>   7: vde: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
>       link/ether b2:f5:e7:68:fd:9e brd ff:ff:ff:ff:ff:ff
>       inet 192.168.0.20/28 scope global vde
>       inet6 2001:470:1f0b:11de::20/125 scope global
>          valid_lft forever preferred_lft forever

And here's the problem: no link-local address on the interface.
"ip addr flush dev vde" in init.d script setting up interface did that,
that was the mistake.

With this address added (or rather preserved), router is able to send
neighbor solicitation requests from it and everything works fine.

Anyway, strange thing is that, apparently, VM on the other side is able
to send these requests without link-local address, using global one
instead:

  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
      inet6 2001:470:1f0b:11de::22/125 scope global
         valid_lft forever preferred_lft forever

Guess older kernel (2.6.30.4) there is the cause of it, and that's also
why it worked before with pretty much the same setup.

Thanks again to Pascal for pointing me in the right direction.


-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [SOLVED] Re: IPv6 forwarding to TAP-interface fails

[Pascal Hambourg]

Mike Kazantsev a écrit :
> 
> And here's the problem: no link-local address on the interface.

Good catch. I didn't notice it.

> With this address added (or rather preserved), router is able to send
> neighbor solicitation requests from it and everything works fine.
> 
> Anyway, strange thing is that, apparently, VM on the other side is able
> to send these requests without link-local address, using global one
> instead:
[...]
> Guess older kernel (2.6.30.4) there is the cause of it, and that's also
> why it worked before with pretty much the same setup.

I observed that when sending a locally generated packet, Linux uses the
source address (which is one one of its own addresses) of the packet as
the source address in the neighbour solicitation, whereas when
forwarding a packet it uses the link-local address attached to the
output interface. So there may be trouble when forwarding a packet and
the output interface has no link-local address. The tcpdump trace in
your previous message shows a neighbour solicitation packet with
unspecified source address :

> IP6 :: > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::22, length 24

This may explain the different results with pings from the LAN and from
the router, and also why VM does not have the problem even though it has
no link-local address.