Does macvtap support host to guest communication?

Does macvtap support host to guest communication?

[Asias He]

Hi, folks

I am trying to use qemu/qemu-kvm with macvtap using following commands:

# ip link add link eth0 name v0 type macvtap mode {vepa,bridge,private}
# ip link set v0 address da:4e:17:88:42:b1 up
# idx=`ip link show v0 | grep mtu| awk -F":" '{print $1}'`
# kvm -net nic,macaddr=da:4e:17:88:42:b1 -net tap,fd=3 -hda
/home/asias/qemu-stuff/sid.img  3<>/dev/tap${idx}

I found that guest can access other hosts on the LAN except the host
where guest lives, and host where guest lives can not access guest.

My question is: Does macvtap support host(hypervisor host) to guest
communication?

-- 
Best Regards,
Asias He

Re: Does macvtap support host to guest communication?

[Arnd Bergmann]

On Monday 18 April 2011, Asias He wrote:
> 
> Hi, folks
> 
> I am trying to use qemu/qemu-kvm with macvtap using following commands:
> 
> # ip link add link eth0 name v0 type macvtap mode {vepa,bridge,private}
> # ip link set v0 address da:4e:17:88:42:b1 up
> # idx=`ip link show v0 | grep mtu| awk -F":" '{print $1}'`
> # kvm -net nic,macaddr=da:4e:17:88:42:b1 -net tap,fd=3 -hda
> /home/asias/qemu-stuff/sid.img  3<>/dev/tap${idx}
> 
> I found that guest can access other hosts on the LAN except the host
> where guest lives, and host where guest lives can not access guest.
> 
> My question is: Does macvtap support host(hypervisor host) to guest
> communication?
> 

You can communicate between macvtap and macvlan devices when they are in
bridge mode, but these devices cannot communicate with clients that
run on the underlying device.

Just add a macvlan device to your hardware interface and use that in
the host instead of running on the low-level device directly.

The other option is to use a vepa enabled bridge, but these are relatively
rare.

	Arnd

Re: Does macvtap support host to guest communication?

[Asias He]

On 04/18/2011 02:58 PM, Arnd Bergmann wrote:
> On Monday 18 April 2011, Asias He wrote:
>>
>> Hi, folks
>>
>> I am trying to use qemu/qemu-kvm with macvtap using following commands:
>>
>> # ip link add link eth0 name v0 type macvtap mode {vepa,bridge,private}
>> # ip link set v0 address da:4e:17:88:42:b1 up
>> # idx=`ip link show v0 | grep mtu| awk -F":" '{print $1}'`
>> # kvm -net nic,macaddr=da:4e:17:88:42:b1 -net tap,fd=3 -hda
>> /home/asias/qemu-stuff/sid.img  3<>/dev/tap${idx}
>>
>> I found that guest can access other hosts on the LAN except the host
>> where guest lives, and host where guest lives can not access guest.
>>
>> My question is: Does macvtap support host(hypervisor host) to guest
>> communication?
>>
> 
> You can communicate between macvtap and macvlan devices when they are in
> bridge mode, but these devices cannot communicate with clients that
> run on the underlying device.
> 
> Just add a macvlan device to your hardware interface and use that in
> the host instead of running on the low-level device directly.

Hi, Arnd

Works for me now. Thanks!

Here are some thoughts on the macvtap:

(1) Is it possible to add an interface to macvtap like /dev/net/tun,
eg, /dev/net/macvtap. Currently, it is hard to use macvtap programmatically.

(2) Adding another macvlan device(e.g., macvlan0) to the hardware
interface(e.g., eth0) and using it as the old eth0 make the process of
using macvtap complicate. One has to reconfigure the network. This is
not optimal from the user perspective. Is it possible to leave the
low-level device as is when using the macvtap device?

> 
> The other option is to use a vepa enabled bridge, but these are relatively
> rare.
> 
> 	Arnd
> 


-- 
Best Regards,
Asias He

Re: Does macvtap support host to guest communication?

[Arnd Bergmann]

On Monday 18 April 2011, Asias He wrote:
> (1) Is it possible to add an interface to macvtap like /dev/net/tun,
> eg, /dev/net/macvtap. Currently, it is hard to use macvtap programmatically.

I decided against having a multiplexor device because it makes permission
handling rather hard. One chardev per network interface makes it possible
to handle permissions in multiuser setups.

> (2) Adding another macvlan device(e.g., macvlan0) to the hardware
> interface(e.g., eth0) and using it as the old eth0 make the process of
> using macvtap complicate. One has to reconfigure the network. This is
> not optimal from the user perspective. Is it possible to leave the
> low-level device as is when using the macvtap device?

Only in VEPA mode. Note that a similar restriction applies when using
the bridge device, for the same technical reasons.

	Arnd

Re: Does macvtap support host to guest communication?

[Ingo Molnar]

* Arnd Bergmann <arnd@arndb.de> wrote:

> On Monday 18 April 2011, Asias He wrote:
> > (1) Is it possible to add an interface to macvtap like /dev/net/tun,
> > eg, /dev/net/macvtap. Currently, it is hard to use macvtap programmatically.
> 
> I decided against having a multiplexor device because it makes permission
> handling rather hard. One chardev per network interface makes it possible
> to handle permissions in multiuser setups.
> 
> > (2) Adding another macvlan device(e.g., macvlan0) to the hardware
> > interface(e.g., eth0) and using it as the old eth0 make the process of
> > using macvtap complicate. One has to reconfigure the network. This is
> > not optimal from the user perspective. Is it possible to leave the
> > low-level device as is when using the macvtap device?
> 
> Only in VEPA mode. Note that a similar restriction applies when using the 
> bridge device, for the same technical reasons.

Just to sum things up, our goal is to allow the tools/kvm/ unprivileged tool to 
provide TCP connectivity to Linux guests transparently, with the following 
parameters:

 - the kvm tool runs unprivileged - as ordinary user

 - without having to configure much (preferably zero configuration: without 
   having to configure anything) on the guest Linux side

 - multiple guests should just work without interfering with each other

 - the kvm tool wants to be stateless - i.e. it does not want to allocate or 
   manage host side devices - it just wants to provide the kind of TCP/IP 
   connectivity host unprivileged user-space has, to the guest. The tool wants 
   to be a generic tool with no global state, not a daemon.

So it wants to be a stateless, unprivileged and zero-configuration solution.

Is this possible with macvtap, and if yes, what kind of macvtap mode and usage 
would you recommend for that goal?

Thanks,

	Ingo

Re: Does macvtap support host to guest communication?

[Arnd Bergmann]

On Monday 18 April 2011, Ingo Molnar wrote:
> > Only in VEPA mode. Note that a similar restriction applies when using the 
> > bridge device, for the same technical reasons.
> 
> Just to sum things up, our goal is to allow the tools/kvm/ unprivileged tool to 
> provide TCP connectivity to Linux guests transparently, with the following 
> parameters:
> 
>  - the kvm tool runs unprivileged - as ordinary user
> 
>  - without having to configure much (preferably zero configuration: without 
>    having to configure anything) on the guest Linux side
> 
>  - multiple guests should just work without interfering with each other
> 
>  - the kvm tool wants to be stateless - i.e. it does not want to allocate or 
>    manage host side devices - it just wants to provide the kind of TCP/IP 
>    connectivity host unprivileged user-space has, to the guest. The tool wants 
>    to be a generic tool with no global state, not a daemon.
> 
> So it wants to be a stateless, unprivileged and zero-configuration solution.
>
> Is this possible with macvtap, and if yes, what kind of macvtap mode and usage 
> would you recommend for that goal?

With the above requirements, I would suggest using something like the the qemu
user networking. This is slower and does not allow servers to be present in
the guest, but those are not your goal as it seems.

The primary goals of macvtap are to allow efficient networking (zero-copy,
multi-queue, although we're not completely there yet) and proper security
abstractions.

If you want a guest to appear on the same network as the host, you can not
do that without privileges to manage the host network setup, and I guess that
will have to stay that way.

	Arnd

Re: Does macvtap support host to guest communication?

[Asias He]

On 04/18/2011 09:20 PM, Arnd Bergmann wrote:
> On Monday 18 April 2011, Ingo Molnar wrote:
>>> Only in VEPA mode. Note that a similar restriction applies when using the 
>>> bridge device, for the same technical reasons.
>>
>> Just to sum things up, our goal is to allow the tools/kvm/ unprivileged tool to 
>> provide TCP connectivity to Linux guests transparently, with the following 
>> parameters:
>>
>>  - the kvm tool runs unprivileged - as ordinary user
>>
>>  - without having to configure much (preferably zero configuration: without 
>>    having to configure anything) on the guest Linux side
>>
>>  - multiple guests should just work without interfering with each other
>>
>>  - the kvm tool wants to be stateless - i.e. it does not want to allocate or 
>>    manage host side devices - it just wants to provide the kind of TCP/IP 
>>    connectivity host unprivileged user-space has, to the guest. The tool wants 
>>    to be a generic tool with no global state, not a daemon.
>>
>> So it wants to be a stateless, unprivileged and zero-configuration solution.
>>
>> Is this possible with macvtap, and if yes, what kind of macvtap mode and usage 
>> would you recommend for that goal?
> 
> With the above requirements, I would suggest using something like the the qemu
> user networking. This is slower and does not allow servers to be present in
> the guest, but those are not your goal as it seems.
> 
> The primary goals of macvtap are to allow efficient networking (zero-copy,
> multi-queue, although we're not completely there yet) and proper security
> abstractions.
> 
> If you want a guest to appear on the same network as the host, you can not
> do that without privileges to manage the host network setup, and I guess that
> will have to stay that way.

We do need "guest appearing on the same network as the host" support as
well. The reason I am considering using macvatp instead of tap plus
brctl is that it simplifies the bridge configuration and it is more
efficient.

However, IMHO, the interface of macvtap is not user friendly, at least
for me. I have no idea about the technical reasons that make the
low-level device inaccessible. But if it is accessible, a lot of
configuration can be eliminated. I know virtualbox's bridge mode has
this kind of restriction, while VMware's bridge mode does not.

> 
> 	Arnd
> 


-- 
Best Regards,
Asias He

Re: Does macvtap support host to guest communication?

[Arnd Bergmann]

On Monday 18 April 2011, Asias He wrote:
> We do need "guest appearing on the same network as the host" support as
> well. The reason I am considering using macvatp instead of tap plus
> brctl is that it simplifies the bridge configuration and it is more
> efficient.

Right, you certainly don't need to consider tap/brctl any more.

> However, IMHO, the interface of macvtap is not user friendly, at least
> for me. I have no idea about the technical reasons that make the
> low-level device inaccessible. But if it is accessible, a lot of
> configuration can be eliminated. I know virtualbox's bridge mode has
> this kind of restriction, while VMware's bridge mode does not.

The main reason is that having a MAC address scan in the regular
networking core would make the common TX case where there is no
macvlan device more complex. Macvtap is derived from the plain
macvlan driver, which used to support only sending out to the
wire until I added the optional bridge mode.

If you want a regular device to be able to send to a macvlan
port, that would require at least these changes:

* Add an option to put a plain device into macvlan-bridge mode
* Add support for that option into iproute2
* Add a hook into dev_queue_xmit() to check for macvlan ports

	Arnd

Re: Does macvtap support host to guest communication?

[Asias He]

On 04/18/2011 11:05 PM, Arnd Bergmann wrote:
> On Monday 18 April 2011, Asias He wrote:
>> We do need "guest appearing on the same network as the host" support as
>> well. The reason I am considering using macvatp instead of tap plus
>> brctl is that it simplifies the bridge configuration and it is more
>> efficient.
> 
> Right, you certainly don't need to consider tap/brctl any more.
> 
>> However, IMHO, the interface of macvtap is not user friendly, at least
>> for me. I have no idea about the technical reasons that make the
>> low-level device inaccessible. But if it is accessible, a lot of
>> configuration can be eliminated. I know virtualbox's bridge mode has
>> this kind of restriction, while VMware's bridge mode does not.
> 
> The main reason is that having a MAC address scan in the regular
> networking core would make the common TX case where there is no
> macvlan device more complex. Macvtap is derived from the plain
> macvlan driver, which used to support only sending out to the
> wire until I added the optional bridge mode.
> 
> If you want a regular device to be able to send to a macvlan
> port, that would require at least these changes:
> 
> * Add an option to put a plain device into macvlan-bridge mode
> * Add support for that option into iproute2
> * Add a hook into dev_queue_xmit() to check for macvlan ports

Cool! Arnd, mind to add this feature to macvtap?

-- 
Best Regards,
Asias He

Re: Does macvtap support host to guest communication?

[Arnd Bergmann]

On Monday 18 April 2011, Asias He wrote:
> > 
> > If you want a regular device to be able to send to a macvlan
> > port, that would require at least these changes:
> > 
> > * Add an option to put a plain device into macvlan-bridge mode
> > * Add support for that option into iproute2
> > * Add a hook into dev_queue_xmit() to check for macvlan ports
> 
> Cool! Arnd, mind to add this feature to macvtap?

No, not after I just explained why I haven't done it before
and why it's so controversial.

Also, I have moved on to other projects and am no longer doing
active development of the macvtap driver. I'd be happy to
pass on the ownership to someone else and help him or her extend
it.

	Arnd

Re: Does macvtap support host to guest communication?

[Arnd Bergmann]

On Tuesday 19 April 2011, Ashish Saxena wrote:
> @Arnd, I would like to initiate the dig into into macvtap driver. Could you
> please provide me some pointers/docs to start with.

I dont' have any general recommendations.

I'd suggest you start reading the macvlan.c and macvtap.c source code and
come back when you have specific questions. 

	Arnd

Re: Does macvtap support host to guest communication?

[Michael S. Tsirkin]

On Tue, Apr 19, 2011 at 02:14:25PM +0200, Arnd Bergmann wrote:
> On Monday 18 April 2011, Asias He wrote:
> > > 
> > > If you want a regular device to be able to send to a macvlan
> > > port, that would require at least these changes:
> > > 
> > > * Add an option to put a plain device into macvlan-bridge mode
> > > * Add support for that option into iproute2
> > > * Add a hook into dev_queue_xmit() to check for macvlan ports
> > 
> > Cool! Arnd, mind to add this feature to macvtap?
> 
> No, not after I just explained why I haven't done it before
> and why it's so controversial.
> 
> Also, I have moved on to other projects and am no longer doing
> active development of the macvtap driver. I'd be happy to
> pass on the ownership to someone else and help him or her extend
> it.
> 
> 	Arnd

There's overlap between macvtap and some
ongoing work on vhost-net.  In particular there are efforts to add zero
copy and multiqueue support.  So if you like, apply something like the
below:

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

diff --git a/MAINTAINERS b/MAINTAINERS
index 649600c..28861a4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4034,6 +4034,13 @@ S:	Maintained
 F:	drivers/net/macvlan.c
 F:	include/linux/if_macvlan.h
 
+MACVTAP DRIVER
+M:	Michael S. Tsirkin <mst@redhat.com>
+M:	Arnd Bergmann <arnd@arndb.de>
+L:	netdev@vger.kernel.org
+S:	Maintained
+F:	drivers/net/macvtap.c
+
 MAN-PAGES: MANUAL PAGES FOR LINUX -- Sections 2, 3, 4, 5, and 7
 M:	Michael Kerrisk <mtk.manpages@gmail.com>
 W:	http://www.kernel.org/doc/man-pages

Re: Does macvtap support host to guest communication?

[Michael S. Tsirkin]

On Mon, Apr 18, 2011 at 02:01:14PM +0200, Ingo Molnar wrote:
>  - the kvm tool wants to be stateless - i.e. it does not want to allocate or 
>    manage host side devices - it just wants to provide the kind of TCP/IP 
>    connectivity host unprivileged user-space has, to the guest. The tool wants 
>    to be a generic tool with no global state, not a daemon.
> 
> So it wants to be a stateless, unprivileged and zero-configuration solution.
> 
> Is this possible with macvtap, and if yes, what kind of macvtap mode and usage 
> would you recommend for that goal?

Well current approaches really all boil down to a network device
in guest passing us raw packets. Typically this means the backend
deals with raw packets as well. For example,
macvtap injects raw packets into host device, That's a priveledged
operation. For tap you can setup NAT to reduce the priveledge.
This is at the moment far from stateless though.

I would love to see something that does NAT but does not
need priveledges or host state. qemu uses slirp to
do (most of) this in userspace but it just seems silly to
maintain yet another networking stack just for this. What do others think?

-- 
MST