IPv4 multicast and mac-vlans acting weird on 3.0.4+

IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Ben Greear]

This is on a hacked 3.0.4 kernel...

I am seeing an issue where an IPv4 mcast receiver will not receive
a 1473 or larger byte mcast message, but will receive a 1472.  The difference
being that 1473 ends up being two packets on the wire.  It works on
802.1Q VLANs, VETH interfaces and real Ethernet.  It does not work
on a mac-vlan hanging off the VETH.

I see packets received on the macvlan in tshark, and they appear correct.  No
obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.

So, I'm about to go digging into the code, but if anyone has any
suggestions for places to look, please let me know!

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 09:46 -0700, Ben Greear a écrit :
> This is on a hacked 3.0.4 kernel...
> 
> I am seeing an issue where an IPv4 mcast receiver will not receive
> a 1473 or larger byte mcast message, but will receive a 1472.  The difference
> being that 1473 ends up being two packets on the wire.  It works on
> 802.1Q VLANs, VETH interfaces and real Ethernet.  It does not work
> on a mac-vlan hanging off the VETH.
> 
> I see packets received on the macvlan in tshark, and they appear correct.  No
> obvious errors in the macvlan port stats or netstat -s,
> and the 'ss' tool doesn't appear to support UDP sockets at all.
> 
> So, I'm about to go digging into the code, but if anyone has any
> suggestions for places to look, please let me know!
> 

Well, problem is defragmentation and macvlan cooperation.

Multicast messages are broadcasted on all macvlan ports.

But IP defrag will probably deliver a single final frame.

We probably need to handle defrag in macvlan before broadcasting to all
ports.

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Ben Greear]

On 10/05/2011 12:54 PM, Eric Dumazet wrote:
> Le mercredi 05 octobre 2011 à 09:46 -0700, Ben Greear a écrit :
>> This is on a hacked 3.0.4 kernel...
>>
>> I am seeing an issue where an IPv4 mcast receiver will not receive
>> a 1473 or larger byte mcast message, but will receive a 1472.  The difference
>> being that 1473 ends up being two packets on the wire.  It works on
>> 802.1Q VLANs, VETH interfaces and real Ethernet.  It does not work
>> on a mac-vlan hanging off the VETH.
>>
>> I see packets received on the macvlan in tshark, and they appear correct.  No
>> obvious errors in the macvlan port stats or netstat -s,
>> and the 'ss' tool doesn't appear to support UDP sockets at all.
>>
>> So, I'm about to go digging into the code, but if anyone has any
>> suggestions for places to look, please let me know!
>>
>
> Well, problem is defragmentation and macvlan cooperation.
>
> Multicast messages are broadcasted on all macvlan ports.
>
> But IP defrag will probably deliver a single final frame.
>
> We probably need to handle defrag in macvlan before broadcasting to all
> ports.

I see packets get to this code in ip_input.c (line 467 or so),
and that printk is mine of course.

	if ((dev && strcmp(dev->name, "rddVR10#0") == 0) ||
	    (dev && strcmp(dev->name, "rddVR10") == 0)) {
		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
		       dev->name, skb->len);
	}

	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
		       ip_rcv_finish);

But, the macvlan packets never make it to the ip_rcv_finish method.

I do see a big and a little packet entering this code.

I have no firewall rules that I'm aware of, though there
is some conn-track logic (though not associated with the
mac-vlan interface):

[root@lec2010-ath9k-1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


And, I just reproduced the problem on vanilla linus top-of-tree (3.1.0-rc9).


Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 13:09 -0700, Ben Greear a écrit :
> On 10/05/2011 12:54 PM, Eric Dumazet wrote:
> > Le mercredi 05 octobre 2011 à 09:46 -0700, Ben Greear a écrit :
> >> This is on a hacked 3.0.4 kernel...
> >>
> >> I am seeing an issue where an IPv4 mcast receiver will not receive
> >> a 1473 or larger byte mcast message, but will receive a 1472.  The difference
> >> being that 1473 ends up being two packets on the wire.  It works on
> >> 802.1Q VLANs, VETH interfaces and real Ethernet.  It does not work
> >> on a mac-vlan hanging off the VETH.
> >>
> >> I see packets received on the macvlan in tshark, and they appear correct.  No
> >> obvious errors in the macvlan port stats or netstat -s,
> >> and the 'ss' tool doesn't appear to support UDP sockets at all.
> >>
> >> So, I'm about to go digging into the code, but if anyone has any
> >> suggestions for places to look, please let me know!
> >>
> >
> > Well, problem is defragmentation and macvlan cooperation.
> >
> > Multicast messages are broadcasted on all macvlan ports.
> >
> > But IP defrag will probably deliver a single final frame.
> >
> > We probably need to handle defrag in macvlan before broadcasting to all
> > ports.
> 
> I see packets get to this code in ip_input.c (line 467 or so),
> and that printk is mine of course.
> 
> 	if ((dev && strcmp(dev->name, "rddVR10#0") == 0) ||
> 	    (dev && strcmp(dev->name, "rddVR10") == 0)) {
> 		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
> 		       dev->name, skb->len);
> 	}
> 
> 	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
> 		       ip_rcv_finish);
> 
> But, the macvlan packets never make it to the ip_rcv_finish method.
> 
> I do see a big and a little packet entering this code.
> 
> I have no firewall rules that I'm aware of, though there
> is some conn-track logic (though not associated with the
> mac-vlan interface):

Say you have 10 vlans on your eth0, how many times do you want one
incoming multicast frame being delivered to your application listening
on 0.0.0.0:port ?

10 or 1 ?

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Ben Greear]

On 10/05/2011 01:17 PM, Eric Dumazet wrote:
> Le mercredi 05 octobre 2011 à 13:09 -0700, Ben Greear a écrit :
>> On 10/05/2011 12:54 PM, Eric Dumazet wrote:
>>> Le mercredi 05 octobre 2011 à 09:46 -0700, Ben Greear a écrit :
>>>> This is on a hacked 3.0.4 kernel...
>>>>
>>>> I am seeing an issue where an IPv4 mcast receiver will not receive
>>>> a 1473 or larger byte mcast message, but will receive a 1472.  The difference
>>>> being that 1473 ends up being two packets on the wire.  It works on
>>>> 802.1Q VLANs, VETH interfaces and real Ethernet.  It does not work
>>>> on a mac-vlan hanging off the VETH.
>>>>
>>>> I see packets received on the macvlan in tshark, and they appear correct.  No
>>>> obvious errors in the macvlan port stats or netstat -s,
>>>> and the 'ss' tool doesn't appear to support UDP sockets at all.
>>>>
>>>> So, I'm about to go digging into the code, but if anyone has any
>>>> suggestions for places to look, please let me know!
>>>>
>>>
>>> Well, problem is defragmentation and macvlan cooperation.
>>>
>>> Multicast messages are broadcasted on all macvlan ports.
>>>
>>> But IP defrag will probably deliver a single final frame.
>>>
>>> We probably need to handle defrag in macvlan before broadcasting to all
>>> ports.
>>
>> I see packets get to this code in ip_input.c (line 467 or so),
>> and that printk is mine of course.
>>
>> 	if ((dev&&  strcmp(dev->name, "rddVR10#0") == 0) ||
>> 	    (dev&&  strcmp(dev->name, "rddVR10") == 0)) {
>> 		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
>> 		       dev->name, skb->len);
>> 	}
>>
>> 	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
>> 		       ip_rcv_finish);
>>
>> But, the macvlan packets never make it to the ip_rcv_finish method.
>>
>> I do see a big and a little packet entering this code.
>>
>> I have no firewall rules that I'm aware of, though there
>> is some conn-track logic (though not associated with the
>> mac-vlan interface):
>
> Say you have 10 vlans on your eth0, how many times do you want one
> incoming multicast frame being delivered to your application listening
> on 0.0.0.0:port ?

How would it work for two Ethernet devices on the same LAN?  I'd
say that mac-vlans should mimic that case.

And in my case, I'm binding hard to a device & IP address,
so my app should get it once regardless.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 13:19 -0700, Ben Greear a écrit :
> On 10/05/2011 01:17 PM, Eric Dumazet wrote:
> > Le mercredi 05 octobre 2011 à 13:09 -0700, Ben Greear a écrit :
> >> On 10/05/2011 12:54 PM, Eric Dumazet wrote:
> >>> Le mercredi 05 octobre 2011 à 09:46 -0700, Ben Greear a écrit :
> >>>> This is on a hacked 3.0.4 kernel...
> >>>>
> >>>> I am seeing an issue where an IPv4 mcast receiver will not receive
> >>>> a 1473 or larger byte mcast message, but will receive a 1472.  The difference
> >>>> being that 1473 ends up being two packets on the wire.  It works on
> >>>> 802.1Q VLANs, VETH interfaces and real Ethernet.  It does not work
> >>>> on a mac-vlan hanging off the VETH.
> >>>>
> >>>> I see packets received on the macvlan in tshark, and they appear correct.  No
> >>>> obvious errors in the macvlan port stats or netstat -s,
> >>>> and the 'ss' tool doesn't appear to support UDP sockets at all.
> >>>>
> >>>> So, I'm about to go digging into the code, but if anyone has any
> >>>> suggestions for places to look, please let me know!
> >>>>
> >>>
> >>> Well, problem is defragmentation and macvlan cooperation.
> >>>
> >>> Multicast messages are broadcasted on all macvlan ports.
> >>>
> >>> But IP defrag will probably deliver a single final frame.
> >>>
> >>> We probably need to handle defrag in macvlan before broadcasting to all
> >>> ports.
> >>
> >> I see packets get to this code in ip_input.c (line 467 or so),
> >> and that printk is mine of course.
> >>
> >> 	if ((dev&&  strcmp(dev->name, "rddVR10#0") == 0) ||
> >> 	    (dev&&  strcmp(dev->name, "rddVR10") == 0)) {
> >> 		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
> >> 		       dev->name, skb->len);
> >> 	}
> >>
> >> 	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
> >> 		       ip_rcv_finish);
> >>
> >> But, the macvlan packets never make it to the ip_rcv_finish method.
> >>
> >> I do see a big and a little packet entering this code.
> >>
> >> I have no firewall rules that I'm aware of, though there
> >> is some conn-track logic (though not associated with the
> >> mac-vlan interface):
> >
> > Say you have 10 vlans on your eth0, how many times do you want one
> > incoming multicast frame being delivered to your application listening
> > on 0.0.0.0:port ?
> 
> How would it work for two Ethernet devices on the same LAN?  I'd
> say that mac-vlans should mimic that case.
> 
> And in my case, I'm binding hard to a device & IP address,
> so my app should get it once regardless.
> 

OK, but before frame being delivered to your app, it must be
re-assembled by net/ipv4/inet_fragment.c & net/ipv4/ip_fragment.c
machinery.

This machinery uses :

static int ip4_frag_match(struct inet_frag_queue *q, void *a)
{
        struct ipq *qp;
        struct ip4_create_arg *arg = a;

        qp = container_of(q, struct ipq, q);
        return  qp->id == arg->iph->id &&
                        qp->saddr == arg->iph->saddr &&
                        qp->daddr == arg->iph->daddr &&
                        qp->protocol == arg->iph->protocol &&
                        qp->user == arg->user;
}

All frames broadcasted (because of multicast code in macvlan) on vlans
have same saddr/daddr/protocol (and user).

So kernel will discard all redundant copies of frames and deliver one
copy only to upper stack.

Check commit 7736d33f4262d437c5 (packet: Add pre-defragmentation support
for ipv4 fanouts) for a possible hint :

We could perform the re-assembly in macvlan code, before doing the
"broadcast the frame on all ports" part.

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Ben Greear]

On 10/05/2011 01:31 PM, Eric Dumazet wrote:
> Le mercredi 05 octobre 2011 à 13:19 -0700, Ben Greear a écrit :
>> On 10/05/2011 01:17 PM, Eric Dumazet wrote:
>>> Le mercredi 05 octobre 2011 à 13:09 -0700, Ben Greear a écrit :
>>>> On 10/05/2011 12:54 PM, Eric Dumazet wrote:
>>>>> Le mercredi 05 octobre 2011 à 09:46 -0700, Ben Greear a écrit :
>>>>>> This is on a hacked 3.0.4 kernel...
>>>>>>
>>>>>> I am seeing an issue where an IPv4 mcast receiver will not receive
>>>>>> a 1473 or larger byte mcast message, but will receive a 1472.  The difference
>>>>>> being that 1473 ends up being two packets on the wire.  It works on
>>>>>> 802.1Q VLANs, VETH interfaces and real Ethernet.  It does not work
>>>>>> on a mac-vlan hanging off the VETH.
>>>>>>
>>>>>> I see packets received on the macvlan in tshark, and they appear correct.  No
>>>>>> obvious errors in the macvlan port stats or netstat -s,
>>>>>> and the 'ss' tool doesn't appear to support UDP sockets at all.
>>>>>>
>>>>>> So, I'm about to go digging into the code, but if anyone has any
>>>>>> suggestions for places to look, please let me know!
>>>>>>
>>>>>
>>>>> Well, problem is defragmentation and macvlan cooperation.
>>>>>
>>>>> Multicast messages are broadcasted on all macvlan ports.
>>>>>
>>>>> But IP defrag will probably deliver a single final frame.
>>>>>
>>>>> We probably need to handle defrag in macvlan before broadcasting to all
>>>>> ports.
>>>>
>>>> I see packets get to this code in ip_input.c (line 467 or so),
>>>> and that printk is mine of course.
>>>>
>>>> 	if ((dev&&   strcmp(dev->name, "rddVR10#0") == 0) ||
>>>> 	    (dev&&   strcmp(dev->name, "rddVR10") == 0)) {
>>>> 		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
>>>> 		       dev->name, skb->len);
>>>> 	}
>>>>
>>>> 	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
>>>> 		       ip_rcv_finish);
>>>>
>>>> But, the macvlan packets never make it to the ip_rcv_finish method.
>>>>
>>>> I do see a big and a little packet entering this code.
>>>>
>>>> I have no firewall rules that I'm aware of, though there
>>>> is some conn-track logic (though not associated with the
>>>> mac-vlan interface):
>>>
>>> Say you have 10 vlans on your eth0, how many times do you want one
>>> incoming multicast frame being delivered to your application listening
>>> on 0.0.0.0:port ?
>>
>> How would it work for two Ethernet devices on the same LAN?  I'd
>> say that mac-vlans should mimic that case.
>>
>> And in my case, I'm binding hard to a device&  IP address,
>> so my app should get it once regardless.
>>
>
> OK, but before frame being delivered to your app, it must be
> re-assembled by net/ipv4/inet_fragment.c&  net/ipv4/ip_fragment.c
> machinery.
>
> This machinery uses :
>
> static int ip4_frag_match(struct inet_frag_queue *q, void *a)
> {
>          struct ipq *qp;
>          struct ip4_create_arg *arg = a;
>
>          qp = container_of(q, struct ipq, q);
>          return  qp->id == arg->iph->id&&
>                          qp->saddr == arg->iph->saddr&&
>                          qp->daddr == arg->iph->daddr&&
>                          qp->protocol == arg->iph->protocol&&
>                          qp->user == arg->user;
> }
>
> All frames broadcasted (because of multicast code in macvlan) on vlans
> have same saddr/daddr/protocol (and user).

Wouldn't you have the same problem with two real Ethernet interfaces on
the same LAN, or two 802.1Q devices for that matter?  The addrs will all
be the same in that case too?

Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'),
I still see the problem with mcast.

 From what you describe, I am thinking I may be hitting a different
issue.  Any ideas on how to figure out why exactly the NF_HOOK isn't
calling the ip_rcv_finish method?

> So kernel will discard all redundant copies of frames and deliver one
> copy only to upper stack.

> Check commit 7736d33f4262d437c5 (packet: Add pre-defragmentation support
> for ipv4 fanouts) for a possible hint :
>
> We could perform the re-assembly in macvlan code, before doing the
> "broadcast the frame on all ports" part.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 13:56 -0700, Ben Greear a écrit :

> Wouldn't you have the same problem with two real Ethernet interfaces on
> the same LAN, or two 802.1Q devices for that matter?  The addrs will all
> be the same in that case too?
> 

Usually multicast is coupled with routing.

A JOIN message from your app wont be sent on all interfaces...

But yes, we might have a similar issue with regular vlans.

Probably nobody noticed yet. Just say no to fragments :)

> Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'),
> I still see the problem with mcast.
> 

Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for
unicast messages. Please test following patch.

>  From what you describe, I am thinking I may be hitting a different
> issue.  Any ideas on how to figure out why exactly the NF_HOOK isn't
> calling the ip_rcv_finish method?
> 

Really I believe I tried to explain the thing already...

ip_local_deliver() -> ip_defrag() :


[PATCH] macvlan: dont send frames on DOWN devices

Reported-by: Ben Greear <greearb@candelatech.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index b100c90..94a0282 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -145,7 +145,8 @@ static void macvlan_broadcast(struct sk_buff *skb,
 		hlist_for_each_entry_rcu(vlan, n, &port->vlan_hash[i], hlist) {
 			if (vlan->dev == src || !(vlan->mode & mode))
 				continue;
-
+			if (!(vlan->dev->flags & IFF_UP))
+				continue;
 			nskb = skb_clone(skb, GFP_ATOMIC);
 			err = macvlan_broadcast_one(nskb, vlan, eth,
 					 mode == MACVLAN_MODE_BRIDGE);

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Ben Greear]

On 10/05/2011 02:36 PM, Eric Dumazet wrote:
> Le mercredi 05 octobre 2011 à 13:56 -0700, Ben Greear a écrit :
>
>> Wouldn't you have the same problem with two real Ethernet interfaces on
>> the same LAN, or two 802.1Q devices for that matter?  The addrs will all
>> be the same in that case too?
>>
>
> Usually multicast is coupled with routing.
>
> A JOIN message from your app wont be sent on all interfaces...

It will be if you open two sockets and bind each one of them
to a network device, at least as far as I can tell.

>
> But yes, we might have a similar issue with regular vlans.
>
> Probably nobody noticed yet. Just say no to fragments :)

Heh, it's regression testing time..we're trying all the weird
stuff this week :)

>> Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'),
>> I still see the problem with mcast.
>>
>
> Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for
> unicast messages. Please test following patch.
>
>>    From what you describe, I am thinking I may be hitting a different
>> issue.  Any ideas on how to figure out why exactly the NF_HOOK isn't
>> calling the ip_rcv_finish method?
>>
>
> Really I believe I tried to explain the thing already...
>
> ip_local_deliver() ->  ip_defrag() :

It seems that netfilter is reporting the pkt as NF_STOLEN, probably
because of the nf_ct_ipv4_gather_frags (which ends up calling ip_defrag)
logic in nf_defrag_ipv4.c, line 86 or so.  I'm adding more debugging
to verify this.

I'll try out your patch below shortly.

Thanks,
Ben

>
>
> [PATCH] macvlan: dont send frames on DOWN devices
>
> Reported-by: Ben Greear<greearb@candelatech.com>
> Signed-off-by: Eric Dumazet<eric.dumazet@gmail.com>
> ---
> diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
> index b100c90..94a0282 100644
> --- a/drivers/net/macvlan.c
> +++ b/drivers/net/macvlan.c
> @@ -145,7 +145,8 @@ static void macvlan_broadcast(struct sk_buff *skb,
>   		hlist_for_each_entry_rcu(vlan, n,&port->vlan_hash[i], hlist) {
>   			if (vlan->dev == src || !(vlan->mode&  mode))
>   				continue;
> -
> +			if (!(vlan->dev->flags&  IFF_UP))
> +				continue;
>   			nskb = skb_clone(skb, GFP_ATOMIC);
>   			err = macvlan_broadcast_one(nskb, vlan, eth,
>   					 mode == MACVLAN_MODE_BRIDGE);
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Ben Greear]

On 10/05/2011 02:52 PM, Ben Greear wrote:
> On 10/05/2011 02:36 PM, Eric Dumazet wrote:
>> Le mercredi 05 octobre 2011 à 13:56 -0700, Ben Greear a écrit :
>>
>>> Wouldn't you have the same problem with two real Ethernet interfaces on
>>> the same LAN, or two 802.1Q devices for that matter? The addrs will all
>>> be the same in that case too?
>>>
>>
>> Usually multicast is coupled with routing.
>>
>> A JOIN message from your app wont be sent on all interfaces...
>
> It will be if you open two sockets and bind each one of them
> to a network device, at least as far as I can tell.
>
>>
>> But yes, we might have a similar issue with regular vlans.
>>
>> Probably nobody noticed yet. Just say no to fragments :)
>
> Heh, it's regression testing time..we're trying all the weird
> stuff this week :)
>
>>> Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'),
>>> I still see the problem with mcast.
>>>
>>
>> Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for
>> unicast messages. Please test following patch.
>>
>>> From what you describe, I am thinking I may be hitting a different
>>> issue. Any ideas on how to figure out why exactly the NF_HOOK isn't
>>> calling the ip_rcv_finish method?
>>>
>>
>> Really I believe I tried to explain the thing already...
>>
>> ip_local_deliver() -> ip_defrag() :
>
> It seems that netfilter is reporting the pkt as NF_STOLEN, probably
> because of the nf_ct_ipv4_gather_frags (which ends up calling ip_defrag)
> logic in nf_defrag_ipv4.c, line 86 or so. I'm adding more debugging
> to verify this.

Ok, this is definitely the problem.  Also, even if you have a single
mac-vlan, you will still have this problem because the underlying device
will get a copy first.  So, your patch doesn't solve my particular problem,
but it does appear to be correct.

If someone wants to cook up macvlan-ip-defrag patch I'll be happy
to test it.  But, as far as I can tell, this problem can happen on
any two interfaces.  The reason that some of mine work (.1q vlans)
and macvlan didn't is probably because those were separated by
some virtual network links that imparted extra delay...so the
vlan consumed all its fragments and passed the complete pkt up
the stack before the mac-vlan ever saw the initial frame.

With this in mind, it seems that using multiple udp multicast
sockets bound to specific devices is fundamentally broken for
fragmented packets.

I have no pressing need for this feature, so now that I better understand
the problem I can just document it and move on to other things.

Thanks for all the help.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

[PATCH net-next] macvlan: handle fragmented multicast frames

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 15:35 -0700, Ben Greear a écrit :

> If someone wants to cook up macvlan-ip-defrag patch I'll be happy
> to test it.  But, as far as I can tell, this problem can happen on
> any two interfaces.  The reason that some of mine work (.1q vlans)
> and macvlan didn't is probably because those were separated by
> some virtual network links that imparted extra delay...so the
> vlan consumed all its fragments and passed the complete pkt up
> the stack before the mac-vlan ever saw the initial frame.
> 
> With this in mind, it seems that using multiple udp multicast
> sockets bound to specific devices is fundamentally broken for
> fragmented packets.
> 
> I have no pressing need for this feature, so now that I better understand
> the problem I can just document it and move on to other things.
> 
> Thanks for all the help.
> 

Please test following patch (note I had no time to test it, sorry !)

Based on net-next tree, might apply on 3.0 kernel...

[PATCH net-next] macvlan: handle fragmented multicast frames

Fragmented multicast frames are delivered to a single macvlan port,
because ip defrag logic considers other samples are redundant.

Implement a defrag step before trying to send the multicast frame.

Reported-by: Ben Greear <greearb@candelatech.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
 drivers/net/macvlan.c  |    3 +++
 include/net/ip.h       |    9 +++++++++
 net/ipv4/ip_fragment.c |   36 ++++++++++++++++++++++++++++++++++++
 net/packet/af_packet.c |   39 +--------------------------------------

diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index b100c90..40366eb 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -169,6 +169,9 @@ static rx_handler_result_t macvlan_handle_frame(struct sk_buff **pskb)
 
 	port = macvlan_port_get_rcu(skb->dev);
 	if (is_multicast_ether_addr(eth->h_dest)) {
+		skb = ip_check_defrag(skb, IP_DEFRAG_MACVLAN);
+		if (!skb)
+			return RX_HANDLER_CONSUMED;
 		src = macvlan_hash_lookup(port, eth->h_source);
 		if (!src)
 			/* frame comes from an external address */
diff --git a/include/net/ip.h b/include/net/ip.h
index aa76c7a..c7e066a 100644
--- a/include/net/ip.h
+++ b/include/net/ip.h
@@ -406,9 +406,18 @@ enum ip_defrag_users {
 	IP_DEFRAG_VS_OUT,
 	IP_DEFRAG_VS_FWD,
 	IP_DEFRAG_AF_PACKET,
+	IP_DEFRAG_MACVLAN,
 };
 
 int ip_defrag(struct sk_buff *skb, u32 user);
+#ifdef CONFIG_INET
+struct sk_buff *ip_check_defrag(struct sk_buff *skb, u32 user);
+#else
+static inline struct sk_buff *ip_check_defrag(struct sk_buff *skb, u32 user)
+{
+	return skb;
+}
+#endif
 int ip_frag_mem(struct net *net);
 int ip_frag_nqueues(struct net *net);
 
diff --git a/net/ipv4/ip_fragment.c b/net/ipv4/ip_fragment.c
index 0e0ab98..763589a 100644
--- a/net/ipv4/ip_fragment.c
+++ b/net/ipv4/ip_fragment.c
@@ -682,6 +682,42 @@ int ip_defrag(struct sk_buff *skb, u32 user)
 }
 EXPORT_SYMBOL(ip_defrag);
 
+struct sk_buff *ip_check_defrag(struct sk_buff *skb, u32 user)
+{
+	const struct iphdr *iph;
+	u32 len;
+
+	if (skb->protocol != htons(ETH_P_IP))
+		return skb;
+
+	if (!pskb_may_pull(skb, sizeof(struct iphdr)))
+		return skb;
+
+	iph = ip_hdr(skb);
+	if (iph->ihl < 5 || iph->version != 4)
+		return skb;
+	if (!pskb_may_pull(skb, iph->ihl*4))
+		return skb;
+	iph = ip_hdr(skb);
+	len = ntohs(iph->tot_len);
+	if (skb->len < len || len < (iph->ihl * 4))
+		return skb;
+
+	if (ip_is_fragment(ip_hdr(skb))) {
+		skb = skb_share_check(skb, GFP_ATOMIC);
+		if (skb) {
+			if (pskb_trim_rcsum(skb, len))
+				return skb;
+			memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
+			if (ip_defrag(skb, user))
+				return NULL;
+			skb->rxhash = 0;
+		}
+	}
+	return skb;
+}
+EXPORT_SYMBOL(ip_check_defrag);
+
 #ifdef CONFIG_SYSCTL
 static int zero;
 
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 25e68f5..ff9eed7 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1213,43 +1213,6 @@ static struct sock *fanout_demux_cpu(struct packet_fanout *f, struct sk_buff *sk
 	return f->arr[cpu % num];
 }
 
-static struct sk_buff *fanout_check_defrag(struct sk_buff *skb)
-{
-#ifdef CONFIG_INET
-	const struct iphdr *iph;
-	u32 len;
-
-	if (skb->protocol != htons(ETH_P_IP))
-		return skb;
-
-	if (!pskb_may_pull(skb, sizeof(struct iphdr)))
-		return skb;
-
-	iph = ip_hdr(skb);
-	if (iph->ihl < 5 || iph->version != 4)
-		return skb;
-	if (!pskb_may_pull(skb, iph->ihl*4))
-		return skb;
-	iph = ip_hdr(skb);
-	len = ntohs(iph->tot_len);
-	if (skb->len < len || len < (iph->ihl * 4))
-		return skb;
-
-	if (ip_is_fragment(ip_hdr(skb))) {
-		skb = skb_share_check(skb, GFP_ATOMIC);
-		if (skb) {
-			if (pskb_trim_rcsum(skb, len))
-				return skb;
-			memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
-			if (ip_defrag(skb, IP_DEFRAG_AF_PACKET))
-				return NULL;
-			skb->rxhash = 0;
-		}
-	}
-#endif
-	return skb;
-}
-
 static int packet_rcv_fanout(struct sk_buff *skb, struct net_device *dev,
 			     struct packet_type *pt, struct net_device *orig_dev)
 {
@@ -1268,7 +1231,7 @@ static int packet_rcv_fanout(struct sk_buff *skb, struct net_device *dev,
 	case PACKET_FANOUT_HASH:
 	default:
 		if (f->defrag) {
-			skb = fanout_check_defrag(skb);
+			skb = ip_check_defrag(skb, IP_DEFRAG_AF_PACKET);
 			if (!skb)
 				return 0;
 		}

Re: IPv4 multicast and mac-vlans acting weird on 3.0.4+

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 23:36 +0200, Eric Dumazet a écrit :
> Le mercredi 05 octobre 2011 à 13:56 -0700, Ben Greear a écrit :
> 
> > Wouldn't you have the same problem with two real Ethernet interfaces on
> > the same LAN, or two 802.1Q devices for that matter?  The addrs will all
> > be the same in that case too?
> > 
> 
> Usually multicast is coupled with routing.
> 
> A JOIN message from your app wont be sent on all interfaces...
> 
> But yes, we might have a similar issue with regular vlans.
> 
> Probably nobody noticed yet. Just say no to fragments :)
> 
> > Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'),
> > I still see the problem with mcast.
> > 
> 
> Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for
> unicast messages. Please test following patch.
> 
> >  From what you describe, I am thinking I may be hitting a different
> > issue.  Any ideas on how to figure out why exactly the NF_HOOK isn't
> > calling the ip_rcv_finish method?
> > 
> 
> Really I believe I tried to explain the thing already...
> 
> ip_local_deliver() -> ip_defrag() :
> 
> 
> [PATCH] macvlan: dont send frames on DOWN devices
> 
> Reported-by: Ben Greear <greearb@candelatech.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---
> diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
> index b100c90..94a0282 100644
> --- a/drivers/net/macvlan.c
> +++ b/drivers/net/macvlan.c
> @@ -145,7 +145,8 @@ static void macvlan_broadcast(struct sk_buff *skb,
>  		hlist_for_each_entry_rcu(vlan, n, &port->vlan_hash[i], hlist) {
>  			if (vlan->dev == src || !(vlan->mode & mode))
>  				continue;
> -
> +			if (!(vlan->dev->flags & IFF_UP))
> +				continue;
>  			nskb = skb_clone(skb, GFP_ATOMIC);
>  			err = macvlan_broadcast_one(nskb, vlan, eth,
>  					 mode == MACVLAN_MODE_BRIDGE);
> 

This one is not needed.

When a port is down, its not in vlan_hash[] table anymore.

(Not sure why we perform the IFF_UP test for unicast frames.)

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

[Ben Greear]

On 10/06/2011 01:28 PM, Eric Dumazet wrote:
> Le mercredi 05 octobre 2011 à 15:35 -0700, Ben Greear a écrit :
>
>> If someone wants to cook up macvlan-ip-defrag patch I'll be happy
>> to test it.  But, as far as I can tell, this problem can happen on
>> any two interfaces.  The reason that some of mine work (.1q vlans)
>> and macvlan didn't is probably because those were separated by
>> some virtual network links that imparted extra delay...so the
>> vlan consumed all its fragments and passed the complete pkt up
>> the stack before the mac-vlan ever saw the initial frame.
>>
>> With this in mind, it seems that using multiple udp multicast
>> sockets bound to specific devices is fundamentally broken for
>> fragmented packets.
>>
>> I have no pressing need for this feature, so now that I better understand
>> the problem I can just document it and move on to other things.
>>
>> Thanks for all the help.
>>
>
> Please test following patch (note I had no time to test it, sorry !)

I hope to test this soon, but lots of other things piled up all
at once, so might be a few days.

Thanks,
Ben

>
> Based on net-next tree, might apply on 3.0 kernel...
>
> [PATCH net-next] macvlan: handle fragmented multicast frames
>
> Fragmented multicast frames are delivered to a single macvlan port,
> because ip defrag logic considers other samples are redundant.
>
> Implement a defrag step before trying to send the multicast frame.


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

[Ben Greear]

On 10/06/2011 01:28 PM, Eric Dumazet wrote:
> Le mercredi 05 octobre 2011 à 15:35 -0700, Ben Greear a écrit :
>
>> If someone wants to cook up macvlan-ip-defrag patch I'll be happy
>> to test it.  But, as far as I can tell, this problem can happen on
>> any two interfaces.  The reason that some of mine work (.1q vlans)
>> and macvlan didn't is probably because those were separated by
>> some virtual network links that imparted extra delay...so the
>> vlan consumed all its fragments and passed the complete pkt up
>> the stack before the mac-vlan ever saw the initial frame.
>>
>> With this in mind, it seems that using multiple udp multicast
>> sockets bound to specific devices is fundamentally broken for
>> fragmented packets.
>>
>> I have no pressing need for this feature, so now that I better understand
>> the problem I can just document it and move on to other things.
>>
>> Thanks for all the help.
>>
>
> Please test following patch (note I had no time to test it, sorry !)
>
> Based on net-next tree, might apply on 3.0 kernel...
>
> [PATCH net-next] macvlan: handle fragmented multicast frames
>
> Fragmented multicast frames are delivered to a single macvlan port,
> because ip defrag logic considers other samples are redundant.
>
> Implement a defrag step before trying to send the multicast frame.

I applied this to Linus' top-of-tree this morning and it does appear
to fix the problem for mac-vlans.

I do see this error, but I doubt it has anything to do with your
patch:

device eth0 entered promiscuous mode
device rddVR10 entered promiscuous mode
ADDRCONF(NETDEV_CHANGE): rddVR1b: link becomes ready

================================================
[ BUG: lock held when returning to user space! ]
------------------------------------------------
ip/3452 is leaving the kernel with locks still held!
1 lock held by ip/3452:
  #0:  (rcu_read_lock){.+.+..}, at: [<f8c5336f>] rcu_read_lock+0x0/0x26 [ipv6]
ADDRCONF(NETDEV_CHANGE): rddVR4b: link becomes ready
ADDRCONF(NETDEV_CHANGE): rddVR5b: link becomes ready


I have no idea why it doesn't print out a more useful stack
trace.  It seems repeatable (2 of 2 reboots so far).  I'm
configuring a pretty complex virtual network, with veth devices,
xorp instances running ipv4 and ipv6 routing protocols, etc.

This is a clean upstream kernel with no outside patches aside from your
own.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

[Eric Dumazet]

Le lundi 10 octobre 2011 à 09:27 -0700, Ben Greear a écrit :

> I applied this to Linus' top-of-tree this morning and it does appear
> to fix the problem for mac-vlans.
> 

Thanks for testing

> I do see this error, but I doubt it has anything to do with your
> patch:
> 
> device eth0 entered promiscuous mode
> device rddVR10 entered promiscuous mode
> ADDRCONF(NETDEV_CHANGE): rddVR1b: link becomes ready
> 
> ================================================
> [ BUG: lock held when returning to user space! ]
> ------------------------------------------------
> ip/3452 is leaving the kernel with locks still held!
> 1 lock held by ip/3452:
>   #0:  (rcu_read_lock){.+.+..}, at: [<f8c5336f>] rcu_read_lock+0x0/0x26 [ipv6]
> ADDRCONF(NETDEV_CHANGE): rddVR4b: link becomes ready
> ADDRCONF(NETDEV_CHANGE): rddVR5b: link becomes ready
> 
> 
> I have no idea why it doesn't print out a more useful stack
> trace.  It seems repeatable (2 of 2 reboots so far).  I'm
> configuring a pretty complex virtual network, with veth devices,
> xorp instances running ipv4 and ipv6 routing protocols, etc.
> 

Do you have LOCKDEP enabled ?

> This is a clean upstream kernel with no outside patches aside from your
> own.

Hmm, it seems we have an rcu_read_unlock() missing...

Any idea what was done by this "ip" command ?

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

[Ben Greear]

On 10/10/2011 09:41 AM, Eric Dumazet wrote:
> Le lundi 10 octobre 2011 à 09:27 -0700, Ben Greear a écrit :
>
>> I applied this to Linus' top-of-tree this morning and it does appear
>> to fix the problem for mac-vlans.
>>
>
> Thanks for testing
>
>> I do see this error, but I doubt it has anything to do with your
>> patch:
>>
>> device eth0 entered promiscuous mode
>> device rddVR10 entered promiscuous mode
>> ADDRCONF(NETDEV_CHANGE): rddVR1b: link becomes ready
>>
>> ================================================
>> [ BUG: lock held when returning to user space! ]
>> ------------------------------------------------
>> ip/3452 is leaving the kernel with locks still held!
>> 1 lock held by ip/3452:
>>    #0:  (rcu_read_lock){.+.+..}, at: [<f8c5336f>] rcu_read_lock+0x0/0x26 [ipv6]
>> ADDRCONF(NETDEV_CHANGE): rddVR4b: link becomes ready
>> ADDRCONF(NETDEV_CHANGE): rddVR5b: link becomes ready
>>
>>
>> I have no idea why it doesn't print out a more useful stack
>> trace.  It seems repeatable (2 of 2 reboots so far).  I'm
>> configuring a pretty complex virtual network, with veth devices,
>> xorp instances running ipv4 and ipv6 routing protocols, etc.
>>
>
> Do you have LOCKDEP enabled ?

Yes, as far as I can tell:
[greearb@build-32 linux-2.6.p4s]$ grep LOCKDEP .config
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_LOCKDEP=y

And it doesn't appear to have turned itself off:

[root@lec2010-ath9k-1 ~]# dmesg|grep lockdep
	RCU lockdep checking is enabled.
lockdep: fixing up alternatives.
[root@lec2010-ath9k-1 ~]#

I looked through the kernel debug section of the config, and it
seems normal enough...

But, after this splat, if I run sysrq-d, then it says sysrq is off,
maybe because the splat disabled it?

SysRq : Show Locks Held
INFO: lockdep is turned off.

sysrq-l does show backtraces, so the backtrace logic in general
seems to work fine.

>
>> This is a clean upstream kernel with no outside patches aside from your
>> own.
>
> Hmm, it seems we have an rcu_read_unlock() missing...
>
> Any idea what was done by this "ip" command ?

No, it's called multiple times by my user-space control logic.  Basically,
it configures around 30 interfaces, some GRE, veth, mac-vlans, .1q vlans, normal ethernet, etc.
Also, I have some ipv6 addrs configured on many of them.

And, setting up routing rules, for ipv4 and ipv6 for the virtual routers.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 06 Oct 2011 22:28:31 +0200

> [PATCH net-next] macvlan: handle fragmented multicast frames
> 
> Fragmented multicast frames are delivered to a single macvlan port,
> because ip defrag logic considers other samples are redundant.
> 
> Implement a defrag step before trying to send the multicast frame.
> 
> Reported-by: Ben Greear <greearb@candelatech.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>

Applied to net-next, thanks.