[Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

Hi; this is a bug report I'm forwarding for somebody else.

The problem is that QEMU's tun/tap support in net/tap-linux.c only
supports the legacy /dev/net/tun interface (the filename is hardcoded).
If you created the tap device via macvtap, then this is the wrong
filename, but there's no code in QEMU to get the correct /dev/tapN
filename from the user-provided ifname=whatever option.

Talking on IRC, the answer suggested was that we ought to
do a SIOCGIFINDEX ioctl passing in the user-specified ifname
string, which then gives you the index N to construct the
/dev/tapN filename. There is probably complexity in working out
whether we should do that or use the legacy interface (or try both
always?) This is probably overall not a very large amount of code,
though.

The usual suggested workaround is to use the -netdev fd option, like
  fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
(which gets the shell to open the right /dev/tap device).
Unfortunately this isn't compatible with multi-queue support
because netdev complains
"ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
and vhostfds= are invalid with fd="
so you can't pass options like "queues=4"...

Detailed repro case below if required (uses aarch64 but none
of this is arch-specific as far as I know).

------------<Begin Reproducer>--------------

To reproduce, here are the steps I followed.

Create a tap device on the host
===============================

    # ip link add link eth0 name tap0 type macvtap mode bridge

To create the tap device the host kernel needs CONFIG_MACVTAP enabled.

You can check that the device is created by running

    $ ip a

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
UNKNOWN group default qlen 1
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
UP group default qlen 1000
        link/ether 02:a1:a2:a3:a4:a5 brd ff:ff:ff:ff:ff:ff
        inet 10.1.194.30/23 brd 10.1.195.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::a1:a2ff:fea3:a4a5/64 scope link
           valid_lft forever preferred_lft forever
    [...]
    8: tap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP group default qlen 500
        link/ether 5e:fc:d2:28:f2:f9 brd ff:ff:ff:ff:ff:ff
        inet6 fe80::5cfc:d2ff:fe28:f2f9/64 scope link
           valid_lft forever preferred_lft forever

Creating a tap device creates a character device. The character device
created is "/dev/tap" followed by the interface index (8 in our example above).

The interface index can also be discovered by reading a node in sysfs.

    $ cat /sys/class/net/tap0/ifindex
    8

Modify tap device permissions
=============================

By default, the device node is created with permissions root:root.

    $ ls -l /dev/tap8
    crw------- 1 root root 242, 1 Nov 30 11:57 /dev/tap8

Update the permissions so that qemu-system-aarch64 can access the node.

    # chgrp kvm /dev/tap8
    # chmod g+rw /dev/tap8

    $ ls -l /dev/tap8
    crw-rw---- 1 root kvm 242, 1 Nov 30 11:57 /dev/tap8

Run qemu
========
I am using a locally built qemu (tag: v2.8.0-rc2). When attempting to
start a guest I get the following complaint -

    $ qemu-system-aarch64 -M virt -m 4096 -cpu host -smp 4 -enable-kvm
-nographic \
         -pflash flash0.img -pflash flash1.img \
         -drive if=none,file=hda.qcow2,id=hd0" \
         -device virtio-blk-device,drive=hd0   \
         -netdev tap,id=eth0,script=,ifname=tap0,queues=4 \
         -device
virtio-net-pci,netdev=eth0,mq=on,vectors=12,mac=22:22:3a:57:07:ed

    qemu-system-aarch64: -netdev
tap,id=eth0,script=,ifname=tap0,queues=4: could not open /dev/net/tun:
No such file or directory

------------<End of Reproducer>--------------

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Daniel P. Berrange]

On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
> Hi; this is a bug report I'm forwarding for somebody else.
> 
> The problem is that QEMU's tun/tap support in net/tap-linux.c only
> supports the legacy /dev/net/tun interface (the filename is hardcoded).
> If you created the tap device via macvtap, then this is the wrong
> filename, but there's no code in QEMU to get the correct /dev/tapN
> filename from the user-provided ifname=whatever option.
> 
> Talking on IRC, the answer suggested was that we ought to
> do a SIOCGIFINDEX ioctl passing in the user-specified ifname
> string, which then gives you the index N to construct the
> /dev/tapN filename. There is probably complexity in working out
> whether we should do that or use the legacy interface (or try both
> always?) This is probably overall not a very large amount of code,
> though.
> 
> The usual suggested workaround is to use the -netdev fd option, like
>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> (which gets the shell to open the right /dev/tap device).
> Unfortunately this isn't compatible with multi-queue support
> because netdev complains
> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
> and vhostfds= are invalid with fd="
> so you can't pass options like "queues=4"...

FWIW you should be able to instead do

   fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)

note 'fds' plural, instead of 'fd'

It would be nicer to make it "just work" though when giving a device
name for macvtap

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
>> The usual suggested workaround is to use the -netdev fd option, like
>>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>> (which gets the shell to open the right /dev/tap device).
>> Unfortunately this isn't compatible with multi-queue support
>> because netdev complains
>> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
>> and vhostfds= are invalid with fd="
>> so you can't pass options like "queues=4"...
>
> FWIW you should be able to instead do
>
>    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>
> note 'fds' plural, instead of 'fd'

What's the difference between that and the "usual suggested workaround"
I described above that doesn't work with queues=... ?

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Daniel P. Berrange]

On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
> > On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
> >> The usual suggested workaround is to use the -netdev fd option, like
> >>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >> (which gets the shell to open the right /dev/tap device).
> >> Unfortunately this isn't compatible with multi-queue support
> >> because netdev complains
> >> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
> >> and vhostfds= are invalid with fd="
> >> so you can't pass options like "queues=4"...
> >
> > FWIW you should be able to instead do
> >
> >    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >
> > note 'fds' plural, instead of 'fd'
> 
> What's the difference between that and the "usual suggested workaround"
> I described above that doesn't work with queues=... ?

It just seems the 'queues' param always wants you to use 'fds' instead
of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
and 'fd' only takes a single FD.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>> > On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
>> >> The usual suggested workaround is to use the -netdev fd option, like
>> >>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>> >> (which gets the shell to open the right /dev/tap device).
>> >> Unfortunately this isn't compatible with multi-queue support
>> >> because netdev complains
>> >> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
>> >> and vhostfds= are invalid with fd="
>> >> so you can't pass options like "queues=4"...
>> >
>> > FWIW you should be able to instead do
>> >
>> >    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>> >
>> > note 'fds' plural, instead of 'fd'
>>
>> What's the difference between that and the "usual suggested workaround"
>> I described above that doesn't work with queues=... ?
>
> It just seems the 'queues' param always wants you to use 'fds' instead
> of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
> and 'fd' only takes a single FD.

Oh, I see. That seems a bit obscure :-)

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Daniel P. Berrange]

On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
> > On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
> >> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
> >> > On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
> >> >> The usual suggested workaround is to use the -netdev fd option, like
> >> >>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >> >> (which gets the shell to open the right /dev/tap device).
> >> >> Unfortunately this isn't compatible with multi-queue support
> >> >> because netdev complains
> >> >> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
> >> >> and vhostfds= are invalid with fd="
> >> >> so you can't pass options like "queues=4"...
> >> >
> >> > FWIW you should be able to instead do
> >> >
> >> >    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >> >
> >> > note 'fds' plural, instead of 'fd'
> >>
> >> What's the difference between that and the "usual suggested workaround"
> >> I described above that doesn't work with queues=... ?
> >
> > It just seems the 'queues' param always wants you to use 'fds' instead
> > of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
> > and 'fd' only takes a single FD.
> 
> Oh, I see. That seems a bit obscure :-)

And pointless, because QemuOpts would have allowed use of 'fd' multiple
times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
fine with multi-queue :-(

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 7 December 2016 at 14:24, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
>> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
>> > On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>> >> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>> >> > note 'fds' plural, instead of 'fd'
>> >>
>> >> What's the difference between that and the "usual suggested workaround"
>> >> I described above that doesn't work with queues=... ?
>> >
>> > It just seems the 'queues' param always wants you to use 'fds' instead
>> > of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
>> > and 'fd' only takes a single FD.
>>
>> Oh, I see. That seems a bit obscure :-)
>
> And pointless, because QemuOpts would have allowed use of 'fd' multiple
> times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
> fine with multi-queue :-(

I guess the best we could do now would be to make fd= and fds= synonyms
with both supporting either comma-separated or being specified
multiple times ?

It's particularly confusing in this case that fd= doesn't work with
queues=, because the user isn't trying to pass multiple fds, just
the one is fine.

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Jason Wang]

On 2016年12月07日 22:32, Peter Maydell wrote:
> On 7 December 2016 at 14:24, Daniel P. Berrange <berrange@redhat.com> wrote:
>> On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
>>> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
>>>> On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>>>>> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>>>>>> note 'fds' plural, instead of 'fd'
>>>>> What's the difference between that and the "usual suggested workaround"
>>>>> I described above that doesn't work with queues=... ?
>>>> It just seems the 'queues' param always wants you to use 'fds' instead
>>>> of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
>>>> and 'fd' only takes a single FD.
>>> Oh, I see. That seems a bit obscure :-)
>> And pointless, because QemuOpts would have allowed use of 'fd' multiple
>> times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
>> fine with multi-queue :-(
> I guess the best we could do now would be to make fd= and fds= synonyms
> with both supporting either comma-separated or being specified
> multiple times ?

When multiqueue were introduced, qemu does not support such kind of 
parameters. This maybe useful (or use fd[0],fd[1]), but I'm not sure 
this is really needed consider libvirt is in charge of doing such things 
now.

>
> It's particularly confusing in this case that fd= doesn't work with
> queues=, because the user isn't trying to pass multiple fds, just
> the one is fine.

Since fd can only accept on file descriptor, so it implies 1 queue. For 
"fds", specifying queues seems redundant, since qemu can count the #fds 
by itself.

Thanks

>
> thanks
> -- PMM
>

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 8 December 2016 at 01:52, Jason Wang <jasowang@redhat.com> wrote:
> On 2016年12月07日 22:32, Peter Maydell wrote:
>> It's particularly confusing in this case that fd= doesn't work with
>> queues=, because the user isn't trying to pass multiple fds, just
>> the one is fine.
>
>
> Since fd can only accept on file descriptor, so it implies 1 queue. For
> "fds", specifying queues seems redundant, since qemu can count the #fds by
> itself.

Opening /dev/net/tun like net/tap-linux.c does also only gives
you one file descriptor, but it still supports multiple queues,
unless I'm misunderstanding how this works (quite possible).

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Jason Wang]

On 2016年12月07日 22:24, Daniel P. Berrange wrote:
> On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
>> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
>>> On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>>>> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>>>>> On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
>>>>>> The usual suggested workaround is to use the -netdev fd option, like
>>>>>>    fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>>>>>> (which gets the shell to open the right /dev/tap device).
>>>>>> Unfortunately this isn't compatible with multi-queue support
>>>>>> because netdev complains
>>>>>> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
>>>>>> and vhostfds= are invalid with fd="
>>>>>> so you can't pass options like "queues=4"...
>>>>> FWIW you should be able to instead do
>>>>>
>>>>>     fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>>>>>
>>>>> note 'fds' plural, instead of 'fd'
>>>> What's the difference between that and the "usual suggested workaround"
>>>> I described above that doesn't work with queues=... ?
>>> It just seems the 'queues' param always wants you to use 'fds' instead
>>> of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
>>> and 'fd' only takes a single FD.
>> Oh, I see. That seems a bit obscure :-)
> And pointless, because QemuOpts would have allowed use of 'fd' multiple
> times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
> fine with multi-queue :-(
>
> Regards,
> Daniel

Yes, but not the time when multiqueue is introduced. I'm not sure how 
much value we can gain from this, consider libvirt has already use fds 
for years.

Thanks