Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Mark Smith]

Hi,

Is it valid to add a macvlan interface to a bridge? I've been having
some trouble with inbound unicast traffic not being forwarded into or
across the bridge, yet inbound broadcast or outbound unicast traffic
being delivered across the bridge correctly.

My setup has been as follows:

o  One physical ethernet interface, purely used to "host" macvlan
interfaces i.e. no IP address, not added to the bridge.

o  Quite a number of macvlan interfaces (I've found the limit of
99 :-) ).

o  Most of those macvlan interfaces are used by individual
instances of roaring penguin pppoe-server. This has worked fine.

o  One of the macvlan interfaces is in a bridge instance, with the
other interface in the bridge being a tap interface. Attached to the
tap interface is guest virtual host, also running a pppoe server.

This bridged macvlan setup seemed to be working ok, as I was seeing
incoming broadcast traffic and outgoing unicast traffic. My full setup
wasn't working correctly, so I spent quite a bit of time investigating
other possible causes. I finally came back around to the bridged macvlan
interface, and then noticed that only incoming unicast traffic wasn't
being bridged/forwarded to the device behind the tap interface.
Bridging the tap interface with another real physical interface resolved
the issue.

I've had a look at the dev.c file in 2.6.27, and my very naive guess
is that as the handle_bridge() call is before the handle_macvlan() call,
because the incoming real physical interface is not part of the bridge,
the incoming unicast packet is being dropped, before the macvlan code
gets a look at it.

Should what I'm doing be working or possible? If not, could something
be added to the kernel to prevent macvlan interfaces being added to
bridge instances, to stop other people spending time trying to do what
I've tried to do?

Thanks,
Mark.

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Patrick McHardy]

Mark Smith wrote:
> Hi,
> 
> Is it valid to add a macvlan interface to a bridge? I've been having
> some trouble with inbound unicast traffic not being forwarded into or
> across the bridge, yet inbound broadcast or outbound unicast traffic
> being delivered across the bridge correctly.
> 
> My setup has been as follows:
> 
> o  One physical ethernet interface, purely used to "host" macvlan
> interfaces i.e. no IP address, not added to the bridge.
> 
> o  Quite a number of macvlan interfaces (I've found the limit of
> 99 :-) ).

Actually there's no limit of 99. The only limiting factor is
the hash size off 256.

> o  Most of those macvlan interfaces are used by individual
> instances of roaring penguin pppoe-server. This has worked fine.
> 
> o  One of the macvlan interfaces is in a bridge instance, with the
> other interface in the bridge being a tap interface. Attached to the
> tap interface is guest virtual host, also running a pppoe server.
> 
> This bridged macvlan setup seemed to be working ok, as I was seeing
> incoming broadcast traffic and outgoing unicast traffic. My full setup
> wasn't working correctly, so I spent quite a bit of time investigating
> other possible causes. I finally came back around to the bridged macvlan
> interface, and then noticed that only incoming unicast traffic wasn't
> being bridged/forwarded to the device behind the tap interface.
> Bridging the tap interface with another real physical interface resolved
> the issue.
> 
> I've had a look at the dev.c file in 2.6.27, and my very naive guess
> is that as the handle_bridge() call is before the handle_macvlan() call,
> because the incoming real physical interface is not part of the bridge,
> the incoming unicast packet is being dropped, before the macvlan code
> gets a look at it.

Yes, that seems correct.

> Should what I'm doing be working or possible? If not, could something
> be added to the kernel to prevent macvlan interfaces being added to
> bridge instances, to stop other people spending time trying to do what
> I've tried to do?

Unfortunately one of both combinations will not work, no matter
what we do. The bridge code could issue a warning when adding
a bridge on top of a macvlan device, but there's no clean indication
that something is a macvlan device besides dev->rtnl_link_ops->kind
being "macvlan".

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Ben Greear]

Patrick McHardy wrote:

>> Should what I'm doing be working or possible? If not, could something
>> be added to the kernel to prevent macvlan interfaces being added to
>> bridge instances, to stop other people spending time trying to do what
>> I've tried to do?
> 
> Unfortunately one of both combinations will not work, no matter
> what we do. The bridge code could issue a warning when adding
> a bridge on top of a macvlan device, but there's no clean indication
> that something is a macvlan device besides dev->rtnl_link_ops->kind
> being "macvlan".

A mac-vlan can't really be PROMISC either since it only receives
BCAST frames and packets destined for it's own MAC, so I can't see
how it could work in a bridge....

user-space brctl could check for the driver string returned by
ethtool API to check if no one wants any additional cruft in
the kernel...


You might try using a pair of VETH interfaces to bridge between
your host and virtual host.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Mark Smith]

Hi,

On Fri, 05 Dec 2008 09:43:25 -0800
Ben Greear <greearb@candelatech.com> wrote:

> Patrick McHardy wrote:
> 
> >> Should what I'm doing be working or possible? If not, could something
> >> be added to the kernel to prevent macvlan interfaces being added to
> >> bridge instances, to stop other people spending time trying to do what
> >> I've tried to do?
> > 
> > Unfortunately one of both combinations will not work, no matter
> > what we do. The bridge code could issue a warning when adding
> > a bridge on top of a macvlan device, but there's no clean indication
> > that something is a macvlan device besides dev->rtnl_link_ops->kind
> > being "macvlan".
> 
> A mac-vlan can't really be PROMISC either since it only receives
> BCAST frames and packets destined for it's own MAC, so I can't see
> how it could work in a bridge....
> 

Would handling of frames for promiscuous macvlan interfaces be quite
similar to handling of incoming broadcast and multicast frames e.g.
for an incoming frame, walk through the list of macvlan interfaces (or
a separate list of promiscuous macvlan interfaces) that are currently in
promiscuous mode, and hand them a copy of the incoming frame?

> user-space brctl could check for the driver string returned by
> ethtool API to check if no one wants any additional cruft in
> the kernel...
> 
> 
> You might try using a pair of VETH interfaces to bridge between
> your host and virtual host.
> 

What I was fundamentally trying to achieve was to avoid using any more
than one physical interface on the box (excepting a separate
management interface) to do this testing. While I happened to have
another unused interface I could bridge this virtual host onto, in some
cases you might not. Conceptually when using them, it is very easy to
think of the macvlan interfaces as nothing very different to having
multiple physical interfaces sitting on the same LAN segment. In my
scenario, bridging only one of them for this specific case of a
virtual guest host seemed like quite a logical thing to do.

Would veth interfaces facilitate the sharing of a single physical
interface between bridged and non-bridged processes on the host?

Regards,
Mark.

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Ben Greear]

Mark Smith wrote:
> Hi,

> Would handling of frames for promiscuous macvlan interfaces be quite
> similar to handling of incoming broadcast and multicast frames e.g.
> for an incoming frame, walk through the list of macvlan interfaces (or
> a separate list of promiscuous macvlan interfaces) that are currently in
> promiscuous mode, and hand them a copy of the incoming frame?

That could probably work.

>> You might try using a pair of VETH interfaces to bridge between
>> your host and virtual host.
>>
> 
> What I was fundamentally trying to achieve was to avoid using any more
> than one physical interface on the box (excepting a separate
> management interface) to do this testing. While I happened to have
> another unused interface I could bridge this virtual host onto, in some
> cases you might not. Conceptually when using them, it is very easy to
> think of the macvlan interfaces as nothing very different to having
> multiple physical interfaces sitting on the same LAN segment. In my
> scenario, bridging only one of them for this specific case of a
> virtual guest host seemed like quite a logical thing to do.
> 
> Would veth interfaces facilitate the sharing of a single physical
> interface between bridged and non-bridged processes on the host?

A Veth pair is like two ports linked back to each other..what is tx'd on one
is rx'd on the other.

You could probably add eth0, veth1a, veth2a to a bridge,
then add macvlans on veth1b and have your virtual guest
talk to veth2b.

I do similar things with my redirdev devices, which are almost identical
to veths, so I think it will work.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Stephen Hemminger]

On Fri, 05 Dec 2008 16:08:09 -0800
Ben Greear <greearb@candelatech.com> wrote:

> Mark Smith wrote:
> > Hi,
> 
> > Would handling of frames for promiscuous macvlan interfaces be quite
> > similar to handling of incoming broadcast and multicast frames e.g.
> > for an incoming frame, walk through the list of macvlan interfaces (or
> > a separate list of promiscuous macvlan interfaces) that are currently in
> > promiscuous mode, and hand them a copy of the incoming frame?
> 
> That could probably work.
> 
> >> You might try using a pair of VETH interfaces to bridge between
> >> your host and virtual host.
> >>
> > 
> > What I was fundamentally trying to achieve was to avoid using any more
> > than one physical interface on the box (excepting a separate
> > management interface) to do this testing. While I happened to have
> > another unused interface I could bridge this virtual host onto, in some
> > cases you might not. Conceptually when using them, it is very easy to
> > think of the macvlan interfaces as nothing very different to having
> > multiple physical interfaces sitting on the same LAN segment. In my
> > scenario, bridging only one of them for this specific case of a
> > virtual guest host seemed like quite a logical thing to do.
> > 
> > Would veth interfaces facilitate the sharing of a single physical
> > interface between bridged and non-bridged processes on the host?
> 
> A Veth pair is like two ports linked back to each other..what is tx'd on one
> is rx'd on the other.
> 
> You could probably add eth0, veth1a, veth2a to a bridge,
> then add macvlans on veth1b and have your virtual guest
> talk to veth2b.
> 
> I do similar things with my redirdev devices, which are almost identical
> to veths, so I think it will work.
> 


Why bother, a macvlan is just a special case of a bridge, so why not
just bridge everything?

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Ben Greear]

Stephen Hemminger wrote:

> Why bother, a macvlan is just a special case of a bridge, so why not
> just bridge everything?

You can add multiple mac-vlans to a single interface, but you can only
add one bridge on top of that interface.

If you need lots of (virtual) interfaces, bridges won't work as far
as I can tell.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Herbert Xu]

Patrick McHardy <kaber@trash.net> wrote:
> 
>> I've had a look at the dev.c file in 2.6.27, and my very naive guess
>> is that as the handle_bridge() call is before the handle_macvlan() call,
>> because the incoming real physical interface is not part of the bridge,
>> the incoming unicast packet is being dropped, before the macvlan code
>> gets a look at it.
> 
> Yes, that seems correct.

Would it be possible to implement promiscuous mode for macvlans by
simply keeping a separate list of macvlan devices in promiscuous
mode and sending all inbound packets to them?

This should make bridging work, right?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Is it valid to add a macvlan virtual interface to a bridge? If so, there seems to be a bug with it.

[Patrick McHardy]

Herbert Xu wrote:
> Patrick McHardy <kaber@trash.net> wrote:
>>> I've had a look at the dev.c file in 2.6.27, and my very naive guess
>>> is that as the handle_bridge() call is before the handle_macvlan() call,
>>> because the incoming real physical interface is not part of the bridge,
>>> the incoming unicast packet is being dropped, before the macvlan code
>>> gets a look at it.
>> Yes, that seems correct.
> 
> Would it be possible to implement promiscuous mode for macvlans by
> simply keeping a separate list of macvlan devices in promiscuous
> mode and sending all inbound packets to them?
> 
> This should make bridging work, right?

I think that should work, at least if we also put the underlying device
in promiscous mode. It wouldn't really be *mac*vlan anymore though :)

I've put it on my TODO list, though I won't be able to take care
of this any time soon.