bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Jan Stancek]

Hi,

I have a host A running 3.14.0-rc5 with one KVM guest, where network
is set up to use bridge:
# brctl show
bridge name     bridge id               STP enabled     interfaces
br1             8000.0023aeed1a00       no              eno1
                                                        vnet0
virbr0          8000.000000000000       yes

I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6
neighbor solicitation packets to KVM guest and as result KVM guest doesn't
respond with neighbor advertisement.

I was trying to narrow down conditions when this happens and eventually
came to following packet sent by other host B on same network:

Ethernet II, Src: Hewlett-_47:93:1c (00:21:5a:47:93:1c), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
    Destination: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
        Address: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: Hewlett-_47:93:1c (00:21:5a:47:93:1c)
        Address: Hewlett-_47:93:1c (00:21:5a:47:93:1c)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: 100:0:600:0:78fb:100:: (100:0:600:0:78fb:100::), Dst: ff02::1 (ff02::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 32
    Next header: IPv6 hop-by-hop option (0)
    Hop limit: 1
    Source: 100:0:600:0:78fb:100:: (100:0:600:0:78fb:100::)
    Destination: ff02::1 (ff02::1)
    Hop-by-Hop Option
        Next header: ICMPv6 (58)
        Length: 0 (8 bytes)
        IPv6 Option (Router Alert)
            Type: Router Alert (5)
            Length: 2
            Router Alert: MLD (0)
        IPv6 Option (Pad1)
            Type: Pad1 (0)
            Pad1
        IPv6 Option (Pad1)
            Type: Pad1 (0)
            Pad1
Internet Control Message Protocol v6
    Type: Multicast Listener Query (130)
    Code: 0
    Checksum: 0xf9c5 [correct]
    Maximum Response Delay [ms]: 1000
    Reserved: 0000
    Multicast Address: :: (::)

The reason I think this packet is related is because when I send same exact
packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh.
solicitation packets to KVM guest.

My current way to reproduce this is:
0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c
   guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29
1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP
   using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/
   with parameters:
   --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029
   -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c
   --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29
   tcpdump running on guest can see both solicit and advertisement packets
2. wait ~5 minutes
3. host B sends Multicast Listener Query packet described above
4. tcpdump running on guest is no longer seeing any neigh solicit packets

I also found out that issue goes away if I do any of following:
1. disable CONFIG_BRIDGE_IGMP_SNOOPING=y
2. echo 0 > /sys/devices/virtual/net/br1/bridge/multicast_snooping
3. make following change:
diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c
index ef66365..8ccc0bf 100644
--- a/net/bridge/br_multicast.c
+++ b/net/bridge/br_multicast.c
@@ -1562,8 +1562,8 @@ static int br_multicast_ipv6_rcv(struct net_bridge *br,
                return 0;

        /* Prevent flooding this packet if there is no listener present */
-       if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr))
-               BR_INPUT_SKB_CB(skb)->mrouters_only = 1;
+/*     if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr))
+               BR_INPUT_SKB_CB(skb)->mrouters_only = 1;*/

        if (ip6h->nexthdr != IPPROTO_HOPOPTS ||
            ip6h->payload_len == 0)

Change above makes me suspect following patch:
  commit 3c3769e63301fd92fcaf51870c371583dd0282ce
  Author: Linus Lüssing <linus.luessing@web.de>
  Date:   Wed Sep 4 02:13:39 2013 +0200
    bridge: apply multicast snooping to IPv6 link-local, too

Please let me know if I can assist in any way to help resolve this.

Regards,
Jan

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Linus Lüssing]

[-- Attachment #1: Type: text/plain, Size: 2376 bytes --]

Hi Jan,

On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote:
> I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6
> neighbor solicitation packets to KVM guest and as result KVM guest doesn't
> respond with neighbor advertisement.

Hm, okay, that's not supposed to happen.

> The reason I think this packet is related is because when I send same exact
> packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh.
> solicitation packets to KVM guest.

Yes, the MLD query is kicking the multicast snooping into gear. If
there's never a query, then snooping is basically disabled
(compare: "bridge: disable snooping if there is no querier").

> 
> My current way to reproduce this is:
> 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c
>    guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29
> 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP
>    using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/
>    with parameters:
>    --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029
>    -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c
>    --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29
>    tcpdump running on guest can see both solicit and advertisement packets
> 2. wait ~5 minutes
> 3. host B sends Multicast Listener Query packet described above
> 4. tcpdump running on guest is no longer seeing any neigh solicit packets

Just to clarify, host B is behind eno1 and vnet0 is directly
connected to the interface of the guest, no additional bridge or
anything else on top of that, right?

Would it be possible for you to upload the tcpdumps from host B
(or if you can't tcpdump on host B, then capturing on eno1)
and the guest somewhere and saying at which time/packet in the dumps
it stops working (probably ~10 seconds after the query). Filtering
for ICMPv6 should be sufficient. 

What I'm curious about is, whether the guest receives
the MLD query and responds with an MLD report. I suspect that
either the bridge doesn't get an MLD report and therefore is
shutting down the according port or there's a bug in parsing the
MLD report in the bridge code.


Thanks for the detailed report so far!

Cheers, Linus

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Vlad Yasevich]

On 03/03/2014 04:27 PM, Linus Lüssing wrote:
> Hi Jan,
> 
> On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote:
>> I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6
>> neighbor solicitation packets to KVM guest and as result KVM guest doesn't
>> respond with neighbor advertisement.
> 
> Hm, okay, that's not supposed to happen.
> 
>> The reason I think this packet is related is because when I send same exact
>> packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh.
>> solicitation packets to KVM guest.
> 
> Yes, the MLD query is kicking the multicast snooping into gear. If
> there's never a query, then snooping is basically disabled
> (compare: "bridge: disable snooping if there is no querier").
> 
>>
>> My current way to reproduce this is:
>> 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c
>>    guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29
>> 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP
>>    using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/
>>    with parameters:
>>    --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029
>>    -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c
>>    --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29
>>    tcpdump running on guest can see both solicit and advertisement packets
>> 2. wait ~5 minutes
>> 3. host B sends Multicast Listener Query packet described above
>> 4. tcpdump running on guest is no longer seeing any neigh solicit packets
> 
> Just to clarify, host B is behind eno1 and vnet0 is directly
> connected to the interface of the guest, no additional bridge or
> anything else on top of that, right?
> 
> Would it be possible for you to upload the tcpdumps from host B
> (or if you can't tcpdump on host B, then capturing on eno1)
> and the guest somewhere and saying at which time/packet in the dumps
> it stops working (probably ~10 seconds after the query). Filtering
> for ICMPv6 should be sufficient. 
> 
> What I'm curious about is, whether the guest receives
> the MLD query and responds with an MLD report. I suspect that
> either the bridge doesn't get an MLD report and therefore is
> shutting down the according port or there's a bug in parsing the
> MLD report in the bridge code.
> 

I did notice a minor issue in the bridge code.  The following
code:
       /* Prevent flooding this packet if there is no listener present */
        if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr))
                BR_INPUT_SKB_CB(skb)->mrouters_only = 1;

        if (ip6h->nexthdr != IPPROTO_HOPOPTS ||
            ip6h->payload_len == 0)
                return 0;

will mark most multicast traffic is mrouters_only.  The two
statement should be probably be reversed.  However, that's shouldn't
cause the reported problem.

-vlad

> 
> Thanks for the detailed report so far!
> 
> Cheers, Linus
> 

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Linus Lüssing]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

On Mon, Mar 03, 2014 at 04:40:40PM -0500, Vlad Yasevich wrote:
> I did notice a minor issue in the bridge code.  The following
> code:
>        /* Prevent flooding this packet if there is no listener present */
>         if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr))
>                 BR_INPUT_SKB_CB(skb)->mrouters_only = 1;
> 
>         if (ip6h->nexthdr != IPPROTO_HOPOPTS ||
>             ip6h->payload_len == 0)
>                 return 0;
> 
> will mark most multicast traffic is mrouters_only.  The two
> statement should be probably be reversed.  However, that's shouldn't
> cause the reported problem.

Reversing the order of these two if-clauses would reintroduce the issue
this commit tried to address, I think:
"bridge: prevent flooding IPv6 packets that do not have a listener"

Besides, I don't quite see what minor issue you are refering to,
would you mind being a little more verbose?

Cheers, Linus

PS: mrouters_only has a kind of confusing naming... for
MLD/IGMP packets it means sending to multicast routers only, there
the name fits. But for non-MLD/IGMP packets it means something
else since "bridge: Only flood unregistered groups to routers"
(and I went along with it with "bridge: prevent flooding IPv6 packets
that do not have a listener"), there it means dropping the skb if
there is no router or matching listener.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Jan Stancek]

----- Original Message -----
> From: "Linus Lüssing" <linus.luessing@web.de>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> Sent: Monday, 3 March, 2014 10:27:59 PM
> Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> 
> Hi Jan,
> 
> On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote:
> > I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6
> > neighbor solicitation packets to KVM guest and as result KVM guest doesn't
> > respond with neighbor advertisement.
> 
> Hm, okay, that's not supposed to happen.
> 
> > The reason I think this packet is related is because when I send same exact
> > packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh.
> > solicitation packets to KVM guest.
> 
> Yes, the MLD query is kicking the multicast snooping into gear. If
> there's never a query, then snooping is basically disabled
> (compare: "bridge: disable snooping if there is no querier").
> 
> > 
> > My current way to reproduce this is:
> > 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c /
> > 00:21:5a:47:93:1c
> >    guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29
> > 1. host B is sending neigh solicit packets every 5 seconds with KVM guest
> > IP
> >    using ns6 from ipv6toolkit:
> >    http://www.si6networks.com/tools/ipv6toolkit/
> >    with parameters:
> >    --src-address=2620:52:0:1040:221:5aff:fe47:931c
> >    --dst-address=ff02::1:ff00:0029
> >    -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c
> >    --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29
> >    tcpdump running on guest can see both solicit and advertisement packets
> > 2. wait ~5 minutes
> > 3. host B sends Multicast Listener Query packet described above
> > 4. tcpdump running on guest is no longer seeing any neigh solicit packets
> 
> Just to clarify, host B is behind eno1 and vnet0 is directly
> connected to the interface of the guest, no additional bridge or
> anything else on top of that, right?

Yes, host B should be behind eno1 (All hosts are remote to me).
There should be only single bridge on host A. Host A has 3 more
interfaces but those are all down.

# cat /etc/sysconfig/network-scripts/ifcfg-eno1
DEVICE=eno1
ONBOOT=yes
BRIDGE=br1
HWADDR=00:23:ae:ed:1a:00

# cat /etc/sysconfig/network-scripts/ifcfg-br1
DEVICE=br1
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Bridge
DELAY=0

There is also bridge on host B. I assume that doesn't matter
but I could set up host B without bridge if needed.

> 
> Would it be possible for you to upload the tcpdumps from host B
> (or if you can't tcpdump on host B, then capturing on eno1)
> and the guest somewhere and saying at which time/packet in the dumps
> it stops working (probably ~10 seconds after the query). Filtering
> for ICMPv6 should be sufficient.

Here are tcpdumps from hostA, hostB and guest (on hostA):
http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces1/

I didn't apply any filter, because that multicast query wasn't showing up
for some reason when I tried to filter by icmp6.

What I did:
1. started tcpdump on all systems
2. send 3 neigh. solicit from hostB manually with couple seconds in between
3. send multicast listener query from hostB manually
4. send 5 neigh. solicit from hostB manually with couple seconds in between

hostA.cap
tcpdump -i eno1 -w hostA.cap
  frame 124, 125 -> OK
  frame 217, 218 -> OK
  frame 291, 292 -> OK
  frame 373 -> Multicast Listener Query
  frame 484 -> no reply?
  frame 572 -> no reply?
  frame 665 -> no reply?

hostB.cap
tcpdump -i br0 -w hostB.cap
  frame 106, 108 -> OK
  frame 214, 216 -> OK
  frame 300, 302 -> OK
  frame 396 -> Multicast Listener Query
  frame 523 -> no reply?
  frame 623 -> no reply?
  frame 730 -> no reply?

guest.cap
tcpdump -i eth0 -w guest.cap
  frame 89, 90 -> OK
  frame 181, 182 -> OK
  frame 254, 255 -> OK
  frame 334 -> Multicast Listener Query
  no more neigh. solicit packets

> 
> What I'm curious about is, whether the guest receives
> the MLD query and responds with an MLD report. I suspect that
> either the bridge doesn't get an MLD report and therefore is
> shutting down the according port or there's a bug in parsing the
> MLD report in the bridge code.

I'm no expert in this area, but shouldn't neigh. solicit packets
be forwarded to all ports regardless of any/no MLD reports?

Regards,
Jan

> 
> 
> Thanks for the detailed report so far!
> 
> Cheers, Linus
> 

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Linus Lüssing]

[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]

Hi Jan,

On Mon, Mar 03, 2014 at 05:45:49PM -0500, Jan Stancek wrote:
> There is also bridge on host B. I assume that doesn't matter
> but I could set up host B without bridge if needed.

It can matter, but in this case it doesn't :).

> > What I'm curious about is, whether the guest receives
> > the MLD query and responds with an MLD report. I suspect that
> > either the bridge doesn't get an MLD report and therefore is
> > shutting down the according port or there's a bug in parsing the
> > MLD report in the bridge code.
> 
> I'm no expert in this area, but shouldn't neigh. solicit packets
> be forwarded to all ports regardless of any/no MLD reports?

That's the beauty of IPv6 Neighbor Discovery using these neat
solicited-node multicast addresses :). With IPv4 and ARP
requests there's no other way than flooding. But for IPv6 we know
in advance behind which bridge port someone interested in the
neighbor solicitation message might be (assuming MLD is working,
properly), allowing us to save bandwidth.

In this case, MLD is not working properly, the main issue is the
following:

Host B sends broken MLD queries, the source address should be an
IPv6 link-local one, not "100:0:600:0:78fb:100::". MLDv2 mandates
this (see RFC3810, section 5.1.14.: "Source Addresses for
Queries").

Though I couldn't find that requirement for MLDv1, Linux ignores
MLDv1 queries with a non-link-local source address, too (see
net/ipv6/mcast.c, igmp6_event_query() ). So Linux never sends an
MLD report in reply to these broken queries.


The second "minor" but in this case fatal issue is, that the
bridge code doesn't have this link-local-src check, therefore
kicking the snooping into gear even though it shouldn't because we
don't have a _working_ querier.

I'm going to make a patch for the bridge code adding this sanity
check.


For the broken query, ok, it's your manually crafted query. But
did you see a query with such a bogus source address "in the
wild", too? (I'm curious how urgent this sanity check is)

Cheers, Linus

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Jan Stancek]

----- Original Message -----
> From: "Linus Lüssing" <linus.luessing@web.de>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> Sent: Tuesday, 4 March, 2014 1:00:41 AM
> Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> 
> Hi Jan,
> 
> On Mon, Mar 03, 2014 at 05:45:49PM -0500, Jan Stancek wrote:
> > There is also bridge on host B. I assume that doesn't matter
> > but I could set up host B without bridge if needed.
> 
> It can matter, but in this case it doesn't :).
> 
> > > What I'm curious about is, whether the guest receives
> > > the MLD query and responds with an MLD report. I suspect that
> > > either the bridge doesn't get an MLD report and therefore is
> > > shutting down the according port or there's a bug in parsing the
> > > MLD report in the bridge code.
> > 
> > I'm no expert in this area, but shouldn't neigh. solicit packets
> > be forwarded to all ports regardless of any/no MLD reports?
> 
> That's the beauty of IPv6 Neighbor Discovery using these neat
> solicited-node multicast addresses :). With IPv4 and ARP
> requests there's no other way than flooding. But for IPv6 we know
> in advance behind which bridge port someone interested in the
> neighbor solicitation message might be (assuming MLD is working,
> properly), allowing us to save bandwidth.
> 
> In this case, MLD is not working properly, the main issue is the
> following:
> 
> Host B sends broken MLD queries, the source address should be an
> IPv6 link-local one, not "100:0:600:0:78fb:100::". MLDv2 mandates
> this (see RFC3810, section 5.1.14.: "Source Addresses for
> Queries").
> 
> Though I couldn't find that requirement for MLDv1, Linux ignores
> MLDv1 queries with a non-link-local source address, too (see
> net/ipv6/mcast.c, igmp6_event_query() ). So Linux never sends an
> MLD report in reply to these broken queries.
> 
> 
> The second "minor" but in this case fatal issue is, that the
> bridge code doesn't have this link-local-src check, therefore
> kicking the snooping into gear even though it shouldn't because we
> don't have a _working_ querier.
> 
> I'm going to make a patch for the bridge code adding this sanity
> check.
> 
> 
> For the broken query, ok, it's your manually crafted query. But
> did you see a query with such a bogus source address "in the
> wild", too? (I'm curious how urgent this sanity check is)

It's real packet I managed to capture during one such occurrence.
I'm sending it with small C program over raw socket, but it's byte
by byte exact copy of what I captured with tcpdump previously.

I'm not sure how that packet came to existence. Based on IPv6 address
it came from host B, but all host B was doing at the time
was running RHEL6 with couple qemu-kvm instances. KVM guests were
set up to use bridge, so I'm assuming if any of them crafted
this packet, source IPv6 address would be different.

Regards,
Jan

> 
> Cheers, Linus
> 

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Linus Lüssing]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

Hi Jan,

On Tue, Mar 04, 2014 at 03:02:36AM -0500, Jan Stancek wrote:
> > For the broken query, ok, it's your manually crafted query. But
> > did you see a query with such a bogus source address "in the
> > wild", too? (I'm curious how urgent this sanity check is)
> 
> It's real packet I managed to capture during one such occurrence.
> I'm sending it with small C program over raw socket, but it's byte
> by byte exact copy of what I captured with tcpdump previously.
> 
> I'm not sure how that packet came to existence. Based on IPv6 address
> it came from host B, but all host B was doing at the time
> was running RHEL6 with couple qemu-kvm instances. KVM guests were
> set up to use bridge, so I'm assuming if any of them crafted
> this packet, source IPv6 address would be different.
> 

Ah, okay. Can you check whether it maybe came from the querier
code in the Linux bridge on host B? Is
"cat /sys/class/net/br0/bridge/multicast_querier" 1? Can you
isolate host B and disable any multicast router daemon on it? Then
check again, if you still see these queries. What kernel version
is running on host B?

Cheers, Linus

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Jan Stancek]

----- Original Message -----
> From: "Linus Lüssing" <linus.luessing@web.de>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> Sent: Tuesday, 4 March, 2014 11:52:54 AM
> Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> 
> Hi Jan,
> 
> On Tue, Mar 04, 2014 at 03:02:36AM -0500, Jan Stancek wrote:
> > > For the broken query, ok, it's your manually crafted query. But
> > > did you see a query with such a bogus source address "in the
> > > wild", too? (I'm curious how urgent this sanity check is)
> > 
> > It's real packet I managed to capture during one such occurrence.
> > I'm sending it with small C program over raw socket, but it's byte
> > by byte exact copy of what I captured with tcpdump previously.
> > 
> > I'm not sure how that packet came to existence. Based on IPv6 address
> > it came from host B, but all host B was doing at the time
> > was running RHEL6 with couple qemu-kvm instances. KVM guests were
> > set up to use bridge, so I'm assuming if any of them crafted
> > this packet, source IPv6 address would be different.
> > 
> 
> Ah, okay. Can you check whether it maybe came from the querier
> code in the Linux bridge on host B? Is
> "cat /sys/class/net/br0/bridge/multicast_querier" 1?

# cat /sys/class/net/br0/bridge/multicast_querier
cat: /sys/class/net/br0/bridge/multicast_querier: No such file or directory

> Can you isolate host B and disable any multicast router daemon on it? Then
> check again, if you still see these queries.

Besides those cases where I sent it by myself, I haven't seen host B send
that query for couple days now.

> What kernel version is running on host B?

2.6.32-279.42.1.el6.x86_64
It's a RHEL6.3.z kernel.

> Where does Linux use :: for queries?

I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I see
packets like these on my network every ~125 seconds:

No.     Time        Source                Destination           Protocol Length Info
  22675 1334.751135 ::                    ff02::1               ICMPv6   86     Multicast Listener Query

Internet Control Message Protocol v6
    Type: Multicast Listener Query (130)
    Code: 0
    Checksum: 0x7ac1 [correct]
    Maximum Response Delay [ms]: 1000
    Reserved: 0000
    Multicast Address: :: (::)

Regards,
Jan

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Linus Lüssing]

[-- Attachment #1: Type: text/plain, Size: 3129 bytes --]

On Tue, Mar 04, 2014 at 06:06:29AM -0500, Jan Stancek wrote:
> 
> 
> ----- Original Message -----
> > From: "Linus Lüssing" <linus.luessing@web.de>
> > To: "Jan Stancek" <jstancek@redhat.com>
> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> > Sent: Tuesday, 4 March, 2014 11:52:54 AM
> > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> > 
> > Hi Jan,
> > 
> > On Tue, Mar 04, 2014 at 03:02:36AM -0500, Jan Stancek wrote:
> > > > For the broken query, ok, it's your manually crafted query. But
> > > > did you see a query with such a bogus source address "in the
> > > > wild", too? (I'm curious how urgent this sanity check is)
> > > 
> > > It's real packet I managed to capture during one such occurrence.
> > > I'm sending it with small C program over raw socket, but it's byte
> > > by byte exact copy of what I captured with tcpdump previously.
> > > 
> > > I'm not sure how that packet came to existence. Based on IPv6 address
> > > it came from host B, but all host B was doing at the time
> > > was running RHEL6 with couple qemu-kvm instances. KVM guests were
> > > set up to use bridge, so I'm assuming if any of them crafted
> > > this packet, source IPv6 address would be different.
> > > 
> > 
> > Ah, okay. Can you check whether it maybe came from the querier
> > code in the Linux bridge on host B? Is
> > "cat /sys/class/net/br0/bridge/multicast_querier" 1?
> 
> # cat /sys/class/net/br0/bridge/multicast_querier
> cat: /sys/class/net/br0/bridge/multicast_querier: No such file or directory

Ok, that's not present on such old kernels, there it is actually
enabled by default.

> 
> > Can you isolate host B and disable any multicast router daemon on it? Then
> > check again, if you still see these queries.
> 
> Besides those cases where I sent it by myself, I haven't seen host B send
> that query for couple days now.
> 
> > What kernel version is running on host B?
> 
> 2.6.32-279.42.1.el6.x86_64
> It's a RHEL6.3.z kernel.
> 
> > Where does Linux use :: for queries?
> 
> I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I see
> packets like these on my network every ~125 seconds:
> 
> No.     Time        Source                Destination           Protocol Length Info
>   22675 1334.751135 ::                    ff02::1               ICMPv6   86     Multicast Listener Query

It's probably the bridge on this ancient kernel, you might want to
backport the following patch:

"bridge: Use IPv6 link-local address for multicast listener queries"
(fe29ec41aaa51902aebd63658dfb04fe6fea8be5)

And it's follow-up fixes:

"bridge: Fix possibly wrong MLD queries' ethernet source address"
(a7bff75b087e7a355838a32efe61707cfa73c194)
"bridge: check return value of ipv6_dev_get_saddr()"
(d1d81d4c3dd886d5fa25a2c4fa1e39cb89613712)

If these patches on host B xor the sanity check I just submitted applied
on your host A / VM host fix your issue, then they might be worth
considering for the stable queue.

Cheers, Linus

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Jan Stancek]

----- Original Message -----
> From: "Linus Lüssing" <linus.luessing@web.de>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> Sent: Tuesday, 4 March, 2014 10:37:57 PM
> Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> > I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I
> > see
> > packets like these on my network every ~125 seconds:
> > 
> > No.     Time        Source                Destination           Protocol
> > Length Info
> >   22675 1334.751135 ::                    ff02::1               ICMPv6   86
> >   Multicast Listener Query
> 
> It's probably the bridge on this ancient kernel, you might want to
> backport the following patch:

Hi,

The example above was from box, which turned out not be running Linux.

> 
> If these patches on host B xor the sanity check I just submitted applied
> on your host A / VM host fix your issue, then they might be worth
> considering for the stable queue.

Is it necessary to apply same patch also on any (older) kernel running in VM?

I applied your sanity check patch [1] on host A only and I wasn't able to reproduce
this issue any longer when I sent this malformed packet from host B:
  Ethernet II, Src: Hewlett-_47:93:1c (00:21:5a:47:93:1c), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
  Internet Protocol Version 6, Src: 100:0:600:0:78fb:100:: (100:0:600:0:78fb:100::), Dst: ff02::1 (ff02::1)
  Internet Control Message Protocol v6


I hand-crafted one new packet from malformed one used in previous tests.
I modified source address from :: to host B link-scope address and changed
dst address from ff02::1 to ff02::1:ffaa:aaaa

Ethernet II, Src: Hewlett-_47:93:1c (00:21:5a:47:93:1c), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
Internet Protocol Version 6, Src: fe80::221:5aff:fe47:931c (fe80::221:5aff:fe47:931c), Dst: ff02::1:ffaa:aaaa (ff02::1:ffaa:aaaa)
Internet Control Message Protocol v6
    Type: Multicast Listener Query (130)
    Code: 0
    Checksum: 0xe365 [correct]
    Maximum Response Delay [ms]: 1000
    Reserved: 0000
    Multicast Address: :: (::)

When I sent it from host B, guest on host A hit same issue - it stopped seeing
neighbor solicitation packets. host A was running 3.14.0-rc5 kernel with
patch [1] applied.

Then I updated kernel on guest to 3.14.0-rc5 + patch [1], but result was the same.
Here is trace from guest: http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/
host B was sending neigh solicit every 5 seconds:
  frame 134, 135 -> OK
  frame 216, 217 -> OK
  frame 329, 330 -> OK
  frame 432, 433 -> OK
  frame 459 -> new hand-crafted packet
  after 35 second mark, guest doesn't see any longer neigh solicit packets

Regards,
Jan

[1] bridge: multicast: add sanity check for query source addresses

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Linus Lüssing]

On Wed, Mar 05, 2014 at 07:10:07AM -0500, Jan Stancek wrote:
> 
> 
> ----- Original Message -----
> > From: "Linus Lüssing" <linus.luessing@web.de>
> > To: "Jan Stancek" <jstancek@redhat.com>
> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> > Sent: Tuesday, 4 March, 2014 10:37:57 PM
> > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> > > I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I
> > > see
> > > packets like these on my network every ~125 seconds:
> > > 
> > > No.     Time        Source                Destination           Protocol
> > > Length Info
> > >   22675 1334.751135 ::                    ff02::1               ICMPv6   86
> > >   Multicast Listener Query
> > 
> > It's probably the bridge on this ancient kernel, you might want to
> > backport the following patch:
> 
> Hi,
> 
> The example above was from box, which turned out not be running Linux.

Okay, yeah, there's somehow the misconception flying around, that
a snooping switch could use "::" as a source address. Even the
informational RFC4541 writes about "proxy IGMP queries"
disregarding the source address requirements for MLDv1/2, feeding
this misconception. I just skimmed RFC4605 ("IGMP/MLD Proxying")
briefly, but couldn't find any note for "0.0.0.0" or "::" there.
But I think we could see with your practical example, that "::"
MLD query source addresses should be considered broken.

> 
> > 
> > If these patches on host B xor the sanity check I just submitted applied
> > on your host A / VM host fix your issue, then they might be worth
> > considering for the stable queue.
> 
> Is it necessary to apply same patch also on any (older) kernel running in VM?

The source address patches are only required on boxes where the
internal MLD querier code of the bridge is used. Afaik tell the
source address patch is present on all kernels since v2.6.38 and
all kernels listed on the kernel.org frontpage, except 2.6.34.15
and 2.6.32.61.

For the sanity check patch, it should probably be added to any
kernel having a bridge (though in your particular scenario and
what we are currently debugging at, having it on host A should
be sufficient).

> I hand-crafted one new packet from malformed one used in previous tests.
> I modified source address from :: to host B link-scope address and changed
> dst address from ff02::1 to ff02::1:ffaa:aaaa

Okay, again according to your capture the guest is receiving the
MLD query on its interface but does not react with an MLD report.

Two things I'd like to know:

Is using the link-scope address as a source and "ff02::1" as the
destination address for the MLD query work for you?

Is using the link-scope address as a source and "ff02::1:ff00:29"
as the destination address for the MLD query "work" for you (do
we see an MLD report from the guest and keep on seeing neighbor
solicitations from host B then?).

For the latter, I don't see anything in particular filtering these
for a general MLD query wrong destination address in the IPv6
code from igmp6_event_query() on. But I suspect that the query
doesn't even get that far on the kernel of the guest, as it is not
listening on ff02::1:ffaa:aaaa. Therefore the test with
"ff02::1:ff00:29", an address the guest is listening on, would be
interesting.

If that works, then I'm going to make a patch ignore General MLD
Queries without ff02::1 as their destination address, too.


Hm, looking at more checks in igmp6_event_query(), I'm currently
wondering whether we should only enable the snooping behaviour in
the bridge when receiving a General MLD Query, so one with "::" in
the multicast field of the MLD message, instead of activating it
upon a Multicast-Address-Specific Query, too. That'd seem more
sane to me, I'm going to make a patch for that tomorrow.

Cheers, Linus

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Jan Stancek]

----- Original Message -----
> From: "Linus Lüssing" <linus.luessing@web.de>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> Sent: Wednesday, 5 March, 2014 3:27:07 PM
> Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

<snip>

> 
> > I hand-crafted one new packet from malformed one used in previous tests.
> > I modified source address from :: to host B link-scope address and changed
> > dst address from ff02::1 to ff02::1:ffaa:aaaa
> 
> Okay, again according to your capture the guest is receiving the
> MLD query on its interface but does not react with an MLD report.
> 
> Two things I'd like to know:
> 
> Is using the link-scope address as a source and "ff02::1" as the
> destination address for the MLD query work for you?

Yes, I could not trigger it with such query:
http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1.cap
  frame 795 -> query
  frame 1040 -> MLD report from guest
  ~20 seconds later
  frame 1507, 1508 -> neigh solicit/advert
  frame 1580, 1581 -> neigh solicit/advert

> 
> Is using the link-scope address as a source and "ff02::1:ff00:29"
> as the destination address for the MLD query "work" for you (do
> we see an MLD report from the guest and keep on seeing neighbor
> solicitations from host B then?).

Yes, this also worked (though I received 2 reports):
http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1_ff0029.cap
  frame 446 -> query
  frame 448 -> MLD report from guest
  frame 465 -> MLD report from guest
  frame 689, 690 -> neigh solicit/advert
  frame 760, 761 -> neigh solicit/advert
  ...

Both host and guest were running 3.14.0-rc5 with your sanity check patch.

Regards,
Jan

> 
> For the latter, I don't see anything in particular filtering these
> for a general MLD query wrong destination address in the IPv6
> code from igmp6_event_query() on. But I suspect that the query
> doesn't even get that far on the kernel of the guest, as it is not
> listening on ff02::1:ffaa:aaaa. Therefore the test with
> "ff02::1:ff00:29", an address the guest is listening on, would be
> interesting.
> 
> If that works, then I'm going to make a patch ignore General MLD
> Queries without ff02::1 as their destination address, too.
> 
> 
> Hm, looking at more checks in igmp6_event_query(), I'm currently
> wondering whether we should only enable the snooping behaviour in
> the bridge when receiving a General MLD Query, so one with "::" in
> the multicast field of the MLD message, instead of activating it
> upon a Multicast-Address-Specific Query, too. That'd seem more
> sane to me, I'm going to make a patch for that tomorrow.
> 
> Cheers, Linus
> 

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Linus Lüssing]

Hi Jan,

Hope your bridge-snooping related issues are resolved with the two
new patches present in net. If not just let me/us know. Thanks
again for the detailed and helpful reporting!

Cheers, Linus


On Wed, Mar 05, 2014 at 09:57:52AM -0500, Jan Stancek wrote:
> 
> 
> 
> 
> ----- Original Message -----
> > From: "Linus Lüssing" <linus.luessing@web.de>
> > To: "Jan Stancek" <jstancek@redhat.com>
> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> > Sent: Wednesday, 5 March, 2014 3:27:07 PM
> > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> 
> <snip>
> 
> > 
> > > I hand-crafted one new packet from malformed one used in previous tests.
> > > I modified source address from :: to host B link-scope address and changed
> > > dst address from ff02::1 to ff02::1:ffaa:aaaa
> > 
> > Okay, again according to your capture the guest is receiving the
> > MLD query on its interface but does not react with an MLD report.
> > 
> > Two things I'd like to know:
> > 
> > Is using the link-scope address as a source and "ff02::1" as the
> > destination address for the MLD query work for you?
> 
> Yes, I could not trigger it with such query:
> http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1.cap
>   frame 795 -> query
>   frame 1040 -> MLD report from guest
>   ~20 seconds later
>   frame 1507, 1508 -> neigh solicit/advert
>   frame 1580, 1581 -> neigh solicit/advert
> 
> > 
> > Is using the link-scope address as a source and "ff02::1:ff00:29"
> > as the destination address for the MLD query "work" for you (do
> > we see an MLD report from the guest and keep on seeing neighbor
> > solicitations from host B then?).
> 
> Yes, this also worked (though I received 2 reports):
> http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1_ff0029.cap
>   frame 446 -> query
>   frame 448 -> MLD report from guest
>   frame 465 -> MLD report from guest
>   frame 689, 690 -> neigh solicit/advert
>   frame 760, 761 -> neigh solicit/advert
>   ...
> 
> Both host and guest were running 3.14.0-rc5 with your sanity check patch.
> 
> Regards,
> Jan
> 
> > 
> > For the latter, I don't see anything in particular filtering these
> > for a general MLD query wrong destination address in the IPv6
> > code from igmp6_event_query() on. But I suspect that the query
> > doesn't even get that far on the kernel of the guest, as it is not
> > listening on ff02::1:ffaa:aaaa. Therefore the test with
> > "ff02::1:ff00:29", an address the guest is listening on, would be
> > interesting.
> > 
> > If that works, then I'm going to make a patch ignore General MLD
> > Queries without ff02::1 as their destination address, too.
> > 
> > 
> > Hm, looking at more checks in igmp6_event_query(), I'm currently
> > wondering whether we should only enable the snooping behaviour in
> > the bridge when receiving a General MLD Query, so one with "::" in
> > the multicast field of the MLD message, instead of activating it
> > upon a Multicast-Address-Specific Query, too. That'd seem more
> > sane to me, I'm going to make a patch for that tomorrow.
> > 
> > Cheers, Linus
> > 

Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest

[Jan Stancek]

----- Original Message -----
> From: "Linus Lüssing" <linus.luessing@web.de>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org
> Sent: Wednesday, 12 March, 2014 5:37:40 AM
> Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
> 
> Hi Jan,
> 
> Hope your bridge-snooping related issues are resolved with the two
> new patches present in net. If not just let me/us know. Thanks
> again for the detailed and helpful reporting!

Hi,

looks good, I wasn't able to reproduce this issue after I applied
patches you posted recently.

Regards,
Jan

> 
> Cheers, Linus
> 
> 
> On Wed, Mar 05, 2014 at 09:57:52AM -0500, Jan Stancek wrote:
> > 
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Linus Lüssing" <linus.luessing@web.de>
> > > To: "Jan Stancek" <jstancek@redhat.com>
> > > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>,
> > > bridge@lists.linux-foundation.org
> > > Sent: Wednesday, 5 March, 2014 3:27:07 PM
> > > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM
> > > guest
> > 
> > <snip>
> > 
> > > 
> > > > I hand-crafted one new packet from malformed one used in previous
> > > > tests.
> > > > I modified source address from :: to host B link-scope address and
> > > > changed
> > > > dst address from ff02::1 to ff02::1:ffaa:aaaa
> > > 
> > > Okay, again according to your capture the guest is receiving the
> > > MLD query on its interface but does not react with an MLD report.
> > > 
> > > Two things I'd like to know:
> > > 
> > > Is using the link-scope address as a source and "ff02::1" as the
> > > destination address for the MLD query work for you?
> > 
> > Yes, I could not trigger it with such query:
> > http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1.cap
> >   frame 795 -> query
> >   frame 1040 -> MLD report from guest
> >   ~20 seconds later
> >   frame 1507, 1508 -> neigh solicit/advert
> >   frame 1580, 1581 -> neigh solicit/advert
> > 
> > > 
> > > Is using the link-scope address as a source and "ff02::1:ff00:29"
> > > as the destination address for the MLD query "work" for you (do
> > > we see an MLD report from the guest and keep on seeing neighbor
> > > solicitations from host B then?).
> > 
> > Yes, this also worked (though I received 2 reports):
> > http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1_ff0029.cap
> >   frame 446 -> query
> >   frame 448 -> MLD report from guest
> >   frame 465 -> MLD report from guest
> >   frame 689, 690 -> neigh solicit/advert
> >   frame 760, 761 -> neigh solicit/advert
> >   ...
> > 
> > Both host and guest were running 3.14.0-rc5 with your sanity check patch.
> > 
> > Regards,
> > Jan
> > 
> > > 
> > > For the latter, I don't see anything in particular filtering these
> > > for a general MLD query wrong destination address in the IPv6
> > > code from igmp6_event_query() on. But I suspect that the query
> > > doesn't even get that far on the kernel of the guest, as it is not
> > > listening on ff02::1:ffaa:aaaa. Therefore the test with
> > > "ff02::1:ff00:29", an address the guest is listening on, would be
> > > interesting.
> > > 
> > > If that works, then I'm going to make a patch ignore General MLD
> > > Queries without ff02::1 as their destination address, too.
> > > 
> > > 
> > > Hm, looking at more checks in igmp6_event_query(), I'm currently
> > > wondering whether we should only enable the snooping behaviour in
> > > the bridge when receiving a General MLD Query, so one with "::" in
> > > the multicast field of the MLD message, instead of activating it
> > > upon a Multicast-Address-Specific Query, too. That'd seem more
> > > sane to me, I'm going to make a patch for that tomorrow.
> > > 
> > > Cheers, Linus
> > > 
>